CN112463804A - KDTree-based image database data processing method - Google Patents
KDTree-based image database data processing method Download PDFInfo
- Publication number
- CN112463804A CN112463804A CN202110139298.4A CN202110139298A CN112463804A CN 112463804 A CN112463804 A CN 112463804A CN 202110139298 A CN202110139298 A CN 202110139298A CN 112463804 A CN112463804 A CN 112463804A
- Authority
- CN
- China
- Prior art keywords
- word
- sensitivity
- words
- sensitive
- single character
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 230000035945 sensitivity Effects 0.000 claims abstract description 74
- 230000010354 integration Effects 0.000 claims abstract description 21
- 238000002372 labelling Methods 0.000 claims abstract description 21
- 238000000586 desensitisation Methods 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 20
- 238000001514 detection method Methods 0.000 claims abstract description 6
- 239000013598 vector Substances 0.000 claims description 40
- 238000005516 engineering process Methods 0.000 claims description 18
- 238000000034 method Methods 0.000 claims description 15
- 230000011218 segmentation Effects 0.000 claims description 14
- 238000010276 construction Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000012217 deletion Methods 0.000 claims description 4
- 230000037430 deletion Effects 0.000 claims description 4
- 230000009191 jumping Effects 0.000 claims description 3
- 238000009825 accumulation Methods 0.000 claims description 2
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 230000006872 improvement Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000001186 cumulative effect Effects 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/57—Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
- G06F21/577—Assessing vulnerabilities and evaluating computer system security
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
- G06F21/6254—Protecting personal data, e.g. for financial or medical purposes by anonymising data, e.g. decorrelating personal data from the owner's identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- Data Mining & Analysis (AREA)
- Medical Informatics (AREA)
- Remote Sensing (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an image database data processing method based on a KDTree, which comprises the following steps: step one, traversing and integrating map marking information based on a KDTree to obtain a marking set S = { S1, S2, …, sn }; secondly, sensitive information detection based on word similarity is carried out on the label set S, and sensitivity grading is carried out on the map label content; and thirdly, performing corresponding desensitization treatment according to the sensitivity level of the map labeling content. The invention utilizes the position information of map label, realizes the automatic processing of traversal and integration of label content in the geographic space data and desensitization of sensitive information, and overcomes the phenomena of fussy work, low efficiency, easy error and vulnerability under the existing manual processing.
Description
Technical Field
The invention relates to the field of geographic information data processing, in particular to a KDTree-based image database data processing method.
Background
In recent years, the rapid development of computer technology has promoted the development and progress of geographic information system technology. A Geographic Information System (GIS) relates to multiple disciplines such as geography, mapping, computer science and technology, takes a computer as a tool, takes geospatial data as a research object, integrates the unique visualization effect and the Geographic analysis function of a map with database operation, and provides decision Information for multiple industries and departments such as geography, planning and management.
Currently, with the development of mobile internet, the demand of people for services based on geographic information is increasing during daily trips. The wide application of various electronic maps brings about not only convenience for work and life of people, but also the problem of potential safety hazard. Among them, the security protection of map labeling information is a problem worthy of research. Some sensitive contents, such as national strategic resources, military banners, defense facilities, etc., may be included in the map annotation information. In this regard, countries have come up with a plurality of legal provisions and policies, such as "supplementary provisions for content representation of public maps" (trial implementation), "provisions for public representation of basic geographic information" (trial implementation), etc., which explicitly specify the contents that can be represented and cannot be represented in public maps, and thus enhance the work of geographic information security protection from the legal and policy level. At present, each organization unit adopts an intranet isolation technical means to ensure data security, and requires desensitization processing on sensitive data in an intranet before releasing geographic information to an extranet.
In a real scene, the labeled content on the map is stored in the form of an attribute table, in order to make the expression effect beautiful and complete, a part of the label is composed of a plurality of single characters, for example, a plurality of records composed of single characters in the attribute table represent a place name, and when a keyword is searched on the map, part of the labeled content is easy to miss. In this case, the existing desensitization processing work has to rely on manual inspection of the labeled content of each region in the map, and the examination, identification and processing are performed on the labeled content, which still has the problem of content omission, and the map region above the market level is large, the content is complex, the manual processing work is tedious, and the efficiency is low. Therefore, a method for integrating the labeling information in the map and performing desensitization processing based on the computer technology is urgently needed to be researched, and the method has an important significance for maintaining the safety of the geographic information.
For the labeled content composed of multiple single characters, the common division method is lexical analysis, i.e. converting a character sequence into a word sequence, segmenting a received string of continuous characters into single words, and then matching the obtained words with a sensitive word bank to further detect sensitive information. However, when conventional adding, deleting or modifying operations are performed on a plurality of single words contained in the label, the single words may be disorderly and repeatedly arranged in the attribute table, and the situation cannot be handled only by simple word segmentation.
Although the arrangement of the label content in the attribute table is irregular, each character in the map is associated with a position coordinate, and the label fields to be integrated have strong relevance in position distribution, such as closer arrangement, top to bottom, left to right, almost on a straight line, and the like. The invention utilizes the position information of map labeling to realize the traversal, integration and desensitization treatment of sensitive information of the labeled content in the geospatial data.
The noun explains: KDTree (k-dimensional tree): is a tree data structure that stores instance points in k-dimensional space for fast retrieval thereof.
jieba word segmentation: a very popular Chinese open source word segmentation packet has the characteristics of high performance, accuracy, expandability and the like, mainly supports python at present, and has related versions in other languages.
word2 vec: is a cluster of correlation models used to generate word vectors. These models are shallow, two-layer neural networks that are trained to reconstruct linguistic word text. The network is represented by words and the input words in adjacent positions are guessed, and the order of the words is unimportant under the assumption of the bag-of-words model in word2 vec. After training is completed, the word2vec model can be used to map each word to a vector, which can be used to represent word-to-word relationships, and the vector is a hidden layer of the neural network.
Disclosure of Invention
The invention provides an image database data processing method based on a KDTree (k-dimensional tree). The invention utilizes the position information of map label, realizes the automatic processing of traversal and integration of label content in the geographic space data and desensitization of sensitive information, and overcomes the phenomena of fussy work, low efficiency, easy error and vulnerability under the existing manual processing.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a KDTree-based image database data processing method comprises the following steps:
step one, traversing and integrating map labeling information based on KDTree to obtain a labeling set S = { S } formed by n labels1,s2,…,sn}; sn represents the nth label; the method specifically comprises the following steps:
1.1, extracting position coordinates and character contents of all single characters on a map in an attribute table of the map to form a data set, and constructing a KDTree according to a two-dimensional coordinate of each character;
1.2, arranging all the single characters according to the sequence of the ordinate from big to small to obtain a primary queue Q; creating a mark array vis [ ]) for recording whether each single character in the queue Q is processed or not, initializing the mark array to 0, and traversing the queue Q until the queue Q is empty;
1.3, processing the single characters in sequence according to the arrangement sequence of the single characters in the primary queue Q;
if the current single character point p is not processed, namely vis [ p ] =0, executing 1.4 steps, and juxtaposing vis [ p ] as 1;
if the current single sub-point p is processed, namely vis [ p ] =1, jumping to the next point in the queue Q;
1.4 searching points which are within a threshold value of [0, epsilon ] from the range of the current single character point p in the constructed KDTree to obtain a neighbor node set of the current single character point p, wherein epsilon represents a parameter of an integration range, and 1.5-2 times of the word width corresponding to the previous single character point p is taken; searching a single character point q meeting the integration condition with the current single character point p in the neighbor node set according to the sequence of the distance points p from near to far, if the single character point q is found successfully, replacing the current single character point with q, and juxtaposing vis [ q ] as 1; the single character point q is the point corresponding to the single character q in the KDTree;
1.5 repeating the step 1.4 until no single character point which can be integrated with the current single character point exists in the neighbor node set, and taking the integrated single character point as a label;
1.6 when no single character point which can be integrated with the current single character point exists in the neighbor node set, processing the next unprocessed single character according to the arrangement sequence of the single characters in the primary queue Q;
1.7 repeating the steps 1.3-1.6 to finish processing the single characters in the preliminary queue Q, and obtaining each label on the map.
Secondly, sensitive information detection based on word similarity is carried out on the label set S, and sensitivity grading is carried out on the map label content;
and thirdly, performing corresponding desensitization treatment according to the sensitivity level of the map labeling content.
In a further improvement, in step 1.4, the integration conditions are as follows:
the single character point q is not processed, namely vis [ q ] = 0;
in the first case, when the integrated field only contains one current single character point p, the current single character point p is integrated with a single character point q which is closest to the previous single character point p in the neighbor node set; when only a single character point p exists in the neighbor node set, the single character point p forms a label;
and secondly, when the integrated field contains two or more words, namely, when the field formed by a plurality of words is integrated with the single word point q, judging whether all the words in the new field s formed by the single word point q and the integrated field are in the same straight line and whether the range R of the array formed by the distance between every two adjacent words meets the following conditions:
wherein Len represents the number of the single characters contained in the new field s,indicates the new field siIndividual character point and jth individual character pointjThe distance of (a) to (b),andrespectively representing taking a maximum value and a minimum value; theta is 0.2-0.5 times of the width of the word in the new field s;
if all of the 2 integration conditions are satisfied, the fields formed by the plurality of words are integrated with the single word point q, and if at least one of the integration conditions is not satisfied, the fields formed by the plurality of words are not integrated with the single word point q.
In a further improvement, before integration, the interference of repeated words is firstly eliminated, if word frames corresponding to p and q are intersected and the contents of the words are the same, the p and q are the repeated words, and q is deleted from the attribute table to realize duplication elimination.
In a further improvement, for the horizontal distribution labeling in step 1.7, the single characters in the labeling are arranged from left to right in order from small to large on the abscissa.
In a further improvement, the second step includes the following steps:
2.1: performing word segmentation on the marked content by adopting a Chinese word segmentation technology:
aiming at each piece of labeled content si of the label set S, converting the labeled content si into a plurality of word vectors by adopting a Chinese word segmentation technology and a word vector construction technology; obtaining si = { a1, a2, …, am }, wherein a1 … am is m feature words obtained after division;
2.2: converting the characteristic words and the sensitive words into word vectors by adopting a word vector construction technology:
converting the characteristic words into word vectors by using word2vec, and recording the word vectors after j-th characteristic word Aj conversion as Aj; similarly, all sensitive words Bk in the sensitive word stock are converted into word vectors which are recorded as Bk; the similarity degree of the feature words and the sensitive words is quantized into the similarity degree of the feature word vectors and the sensitive word vectors, namely, the cosine value of an included angle of inner product spaces of the two vectors is taken as the similarity degree, the value range is [0,1], and the closer the similarity degree is to 1, the greater the similarity degree of the two words is;
2.3 sensitivity calculation of feature words:
in the sensitive word bank, each sensitive word ck corresponds to one sensitive level,The larger the value is, the higher the sensitivity degree of the sensitive word is; traversing the sensitive word stock, and defining the maximum sensitivity of the characteristic words aj for the characteristic words ajIs composed of
Wherein,representing a sensitive word bank, calculating the product of the similarity of each sensitive word vector and the feature word vector and the sensitivity level, and taking the maximum value to represent the maximum sensitivity of the feature words;
setting a threshold parameter theta whenWhen the value is larger than theta, the characteristic word aj has sensitivity, otherwise, the characteristic word aj is not considered as a sensitive word, the sensitivity is marked as 0, namely, the sensitivity of the characteristic word is
2.4 sensitivity calculation for annotated content:
in the formula, notationContaining m feature words, j represents a labelThe jth feature word of (1) is a labelSensitivity of (2) is a labelThe sum of the sensitivity accumulation of the contained m characteristic words; after the sensitivities of the feature words and the labeled contents are obtained, the labeled contents are divided into 4 levels of high sensitivity, medium sensitivity, low sensitivity and non-sensitivity according to the sensitivities.
In a further improvement, the third step includes the following steps:
3.1, constructing a white list of the geographic marking information, and adding the non-sensitive data into the white list every time when the non-sensitive data which is wrongly identified by the algorithm is manually found, so that the fault tolerance rate is improved;
3.2 after the sensitive data are screened by the white list, the marked content with high sensitive level is directly deleted; labeling the medium and low sensitivity levels, extracting the sensitivity characteristic words in the labels, randomly selecting a desensitization means of deletion, replacement and generalization for processing, then recalculating the desensitization labeled sensitivity, and completely deleting the corresponding labels if the desensitization labeled sensitivity does not meet the public requirement after iteration preset times; when the non-sensitive word with the maximum similarity to the current sensitive characteristic word is selected as the replacement in the selection replacement, the specific content of the label description is abstracted when the generalization operation is selected, so that the description range comprises more non-sensitive information.
The invention has the advantages that:
the invention utilizes the position information of map label, realizes the automatic processing of traversal and integration of label content in the geographic space data and desensitization of sensitive information, and overcomes the phenomena of fussy work, low efficiency, easy error and vulnerability under the existing manual processing.
Drawings
FIG. 1 is a flow chart of the present invention.
Detailed Description
Fig. 1 shows a KDTree-based image database data processing method, which includes the following steps:
(1) traversing and integrating map labeling information based on KDTree;
(2) sensitive information detection is carried out based on word similarity;
(3) desensitization is performed based on the sensitivity of the signature.
The specific contents are as follows:
(1) a map labeling information traversal and integration method based on a KDTree comprises the following steps:
the method utilizes the relevance of map labeling information on position distribution, considers points formed by all single characters, greedily considers that another point closest to the point of the current single character in a certain range can meet the condition that the two are integrated into a word, and considers other points in the range when the condition of integrating into the word cannot be met, and the specific flow is as follows:
the method comprises the following steps: and extracting the position coordinates and character contents of all the single characters on the map in the attribute table, and constructing a KDTree according to the two-dimensional coordinates of each character. KDTree is a data structure that partitions k-dimensional data space and is commonly used for range search and nearest neighbor search.
Step two: because people often read from top to bottom and from left to right, all the single characters are arranged in the descending order of the ordinate before the marked content is integrated, so that the subsequently accessed single characters are added into the currently integrated field in sequence to obtain a primary queue Q. A tag array vis is created, recording whether each word has been processed, initialized to 0. Traverse queue Q until the queue is empty:
step three: if the current point p is processed, namely vis [ p ] =1, jumping to the next point in the queue; if vis [ p ] =0, searching a point which is in a threshold value [0, epsilon ] from the range of the current single character point p in the KDTree to obtain an adjacent node set of the current single character point p, searching a single character point q which meets the integration condition with the current single character point p in the adjacent node set according to the sequence of the distance points p from near to far, if the finding is successful, replacing the current single character point with q, and juxtaposing vis [ q ] as 1; step four: repeating the third step until no single character point which can be integrated with the current single character point exists in the neighbor node set, taking the integrated single character point as a label, and processing the next unprocessed single character according to the arrangement sequence of the single characters in the preliminary queue Q;
step five: and repeating the third step to the fourth step until the single characters in the preliminary queue Q are all processed to obtain all the marks on the map.
In addition, when an unprocessed point is integrated into a processed field, it is necessary to satisfy that all words in the formed new field s are in the same straight line and the distance between every two adjacent words is close, that is, the position distribution on the map satisfies the condition of forming a label, otherwise, the integration is not performed. In the integration, the ordinate is almost unchanged, and the field with the larger abscissa change needs to be placed at the front end of the labeled content when the field with the smaller abscissa change is combined, namely, the field is arranged from left to right.
And after the queue Q is empty, all the single characters are processed, and the label set obtained after integration is recorded as S = { S = (S) }1,s2,…,sn}。
(2) A sensitive information detection method based on word similarity comprises the following steps:
for the label set S, in order to protect data security, sensitive information detection based on word similarity is required, and the method mainly comprises the following four steps:
the method comprises the following steps: performing word segmentation on the marked content by adopting a Chinese word segmentation technology:
first, for each piece of annotation content S of the annotation set SiThe Chinese word segmentation technology and the word vector construction technology are adopted to convert the Chinese word segmentation technology and the word vector construction technology into a plurality of word vectors. The method divides map annotation into a plurality of words by using jieba word segmentation to obtain si={a1,a2,…,am},a1…amM feature words obtained after division.
Step two: converting the characteristic words and the sensitive words into word vectors by adopting a word vector construction technology:
the invention uses word2vec to convert the feature words into word vectors, each feature word ajThe converted word vector is marked as Aj. Similarly, all the sensitive words b in the sensitive word bank are combinedkConverted into word vectors, denoted Bk. At the moment, the similarity degree of the feature words and the sensitive words can be quantized into the similarity degree of the feature word vectors and the sensitive word vectors, namely, the cosine value of the included angle of the inner product space of the two vectors is taken as the similarity degreeThe value range is [0,1]]The closer the similarity is to 1, the greater the degree of similarity between the two words.
Step three: sensitivity calculation of the characteristic words:
in the sensitive word bank, each sensitive word ckCorresponding to a sensitivity levelThe larger the value of L, the higher the sensitivity of the sensitive word. Traversing the sensitive word stock to the characteristic word ajDefining its maximum sensitivity as
Wherein,representing the sensitive word stock, calculating the product of the similarity of each sensitive word vector and the characteristic word vector and the sensitivity level, and taking the maximum value to represent the characteristic wordThe maximum sensitivity of (c).
Setting a threshold parameter theta whenWhen greater than theta, the feature wordHas sensitivity, otherwise is not considered as a sensitive word, and the sensitivity is marked as 0, namely a characteristic wordHas a sensitivity of
Step four: sensitivity calculation of the annotation content:
since a label can be composed of a plurality of feature words, the sensitivity of the label needs to be measured in consideration of the sensitivities of all the feature words. In addition, the characteristic words are often distributed near the tail of the annotation content, and the sensitivity of the characteristic words has a greater influence on the sensitivity of the annotation. For example, "the armed police hospital in Hunan province", although "armed police" is a sensitive word, the hospital is open to the public and does not belong to sensitive labeled contents. For example, the term "nuclear power plant", "military base", etc. is used to determine the sensitivity of the tag. Therefore, considering the location distribution of the feature words, the sensitivity of the label is defined as:
in the formula, notationContaining m feature words, j representsIs the jth feature word of the annotation,to its sensitivity, Is the cumulative sum of 1,2, …, m. After the sensitivities of the feature words and the labeled contents are obtained, the labeled contents can be divided into 4 levels of high, medium, low and non-sensitivity according to the sensitivities.
(3) A desensitization method based on feature word sensitivity.
After the sensitive data in the marked content is detected according to the method (2), the sensitive data needs to be processed according to the sensitive level.
The method comprises the following steps: for some labeled contents with insensitive data, because the labeled contents possibly contain characteristic words similar to the sensitive words and are mistakenly recognized as the sensitive data by the algorithm, a white list of the geographic labeling information can be constructed, and each time the non-sensitive data mistakenly recognized by the algorithm is manually discovered, the non-sensitive data is added into the white list, so that the fault tolerance rate of the model is continuously improved.
Step two: in order to ensure the availability of map data, it is not desirable to delete all sensitive data. After the sensitive data identified by the algorithm is screened by the white list, the sensitivity of the sensitive data can be reduced to a non-sensitive level by adopting some desensitization means so as to meet the requirement of external publication. For the labeled content with high sensitivity level, the security of the geographic information is very easy to threaten after the labeled content is leaked, so that a direct deletion measure is adopted. And (3) labeling the medium and low sensitivity levels, extracting the sensitive characteristic words in the medium and low sensitivity levels, randomly selecting desensitization means such as deletion, replacement and generalization to process, recalculating the labeling sensitivity, and deleting the label if the labeling sensitivity does not meet the public requirement after iteration for a certain number of times.
During replacement, the non-sensitive word with the maximum similarity to the current sensitive characteristic word is selected from the non-sensitive word stock to be replaced, and generalization operation is to abstract the concrete content of the label description so that the description range comprises more non-sensitive information, for example, the 'liberty military logistics base' is transformed into a 'warehouse' after generalization and replacement operation.
Claims (6)
1. A KDTree-based image database data processing method is characterized by comprising the following steps:
step one, traversing and integrating map labeling information based on KDTree to obtain a labeling set S = { S } formed by n labels1,s2,…,sn}; sn represents the nth label;
the method specifically comprises the following steps:
1.1, extracting position coordinates and character contents of all single characters on a map in an attribute table of the map to form a data set, and constructing a KDTree according to a two-dimensional coordinate of each character;
1.2, arranging all the single characters according to the sequence of the ordinate from big to small to obtain a primary queue Q; creating a mark array vis [ ]) for recording whether each single character in the queue Q is processed or not, initializing the mark array to 0, and traversing the queue Q until the queue Q is empty;
1.3, processing the single characters in sequence according to the arrangement sequence of the single characters in the primary queue Q;
if the current single character point p is not processed, namely vis [ p ] =0, executing 1.4 steps, and juxtaposing vis [ p ] as 1;
if the current single sub-point p is processed, namely vis [ p ] =1, jumping to the next point in the queue Q;
1.4 searching points which are within a threshold value of [0, epsilon ] from the range of the current single character point p in the constructed KDTree to obtain a neighbor node set of the current single character point p, wherein epsilon represents a parameter of an integration range, and 1.5-2 times of the word width corresponding to the previous single character point p is taken; searching a single character point q meeting the integration condition with the current single character point p in the neighbor node set according to the sequence of the distance points p from near to far, if the single character point q is found successfully, replacing the current single character point with q, and juxtaposing vis [ q ] as 1; the single character point q is the point corresponding to the single character q in the KDTree;
1.5 repeating the step 1.4 until no single character point which can be integrated with the current single character point exists in the neighbor node set, and taking the integrated single character point as a label;
1.6 when no single character point which can be integrated with the current single character point exists in the neighbor node set; processing the next unprocessed single character according to the arrangement sequence of the single characters in the preliminary queue Q;
1.7 repeating the steps 1.3-1.6 until the single characters in the preliminary queue Q are all processed; obtaining each label on the map;
secondly, sensitive information detection based on word similarity is carried out on the label set S, and sensitivity grading is carried out on the map label content;
and thirdly, performing corresponding desensitization treatment according to the sensitivity level of the map labeling content.
2. The KDTree-based image database data processing method according to claim 1, wherein in step 1.4, the integration condition is as follows:
the single character point q is not processed, namely vis [ q ] = 0;
in the first case, when the integrated field only contains one current single character point p, the current single character point p is integrated with a single character point q which is closest to the previous single character point p in the neighbor node set; when only a single character point p exists in the neighbor node set, the single character point p forms a label;
and secondly, when the integrated field contains two or more words, namely, when the field formed by a plurality of words is integrated with the single word point q, judging whether all the words in the new field s formed by the single word point q and the integrated field are in the same straight line and whether the range R of the array formed by the distance between every two adjacent words meets the following conditions:
wherein Len represents the number of the single characters contained in the new field s,indicates the new field siIndividual character point and jth individual character pointjThe distance of (a) to (b),andrespectively representing taking a maximum value and a minimum value; theta is 0.2-0.5 times of the width of the word in the new field s;
if all of the 2 integration conditions are satisfied, the fields formed by the plurality of words are integrated with the single word point q, and if at least one of the integration conditions is not satisfied, the fields formed by the plurality of words are not integrated with the single word point q.
3. The KDTree-based image database data processing method according to claim 2, wherein before integration, interference of duplicate words is first eliminated, if word frames corresponding to p and q intersect and the contents of the words are the same, then p and q are duplicate words, and q is deleted from the attribute table to realize deduplication.
4. The KDTree-based image database data processing method according to claim 1, wherein in step 1.7, for horizontally distributed labels, the individual characters in the label are arranged in order from small to large on the abscissa, and in order from left to right.
5. The KDTree-based image database data processing method of claim 1, wherein the second step comprises the steps of:
2.1: performing word segmentation on the marked content by adopting a Chinese word segmentation technology:
aiming at each piece of labeled content si of the label set S, converting the labeled content si into a plurality of word vectors by adopting a Chinese word segmentation technology and a word vector construction technology; obtaining si = { a1, a2, …, am }, wherein a1 … am is m feature words obtained after division;
2.2: converting the characteristic words and the sensitive words into word vectors by adopting a word vector construction technology:
converting the characteristic words into word vectors by using word2vec, and recording the word vectors after j-th characteristic word Aj conversion as Aj; similarly, all sensitive words Bk in the sensitive word stock are converted into word vectors which are recorded as Bk; the similarity degree of the feature words and the sensitive words is quantized into the similarity degree of the feature word vectors and the sensitive word vectors, namely, the cosine value of an included angle of inner product spaces of the two vectors is taken as the similarity degree, the value range is [0,1], and the closer the similarity degree is to 1, the greater the similarity degree of the two words is;
2.3 sensitivity calculation of feature words:
in the sensitive word bank, each sensitive word ck corresponds to one sensitive level,The larger the value is, the higher the sensitivity degree of the sensitive word is; traversing the sensitive word stock, and defining the maximum sensitivity of the characteristic words aj for the characteristic words ajIs composed of
Wherein,representing a sensitive word bank, calculating the product of the similarity of each sensitive word vector and the feature word vector and the sensitivity level, and taking the maximum value to represent the maximum sensitivity of the feature words;
setting a threshold parameter theta whenWhen the value is larger than theta, the characteristic word aj has sensitivity, otherwise, the characteristic word aj is not considered as a sensitive word, the sensitivity is marked as 0, namely, the sensitivity of the characteristic word is
2.4 sensitivity calculation for annotated content:
in the formula, notationContaining m feature words, j represents a labelThe jth feature word of (1) is a markNote thatSensitivity of (2) is a labelThe sum of the sensitivity accumulation of the contained m characteristic words; after the sensitivities of the feature words and the labeled contents are obtained, the labeled contents are divided into 4 levels of high sensitivity, medium sensitivity, low sensitivity and non-sensitivity according to the sensitivities.
6. The KDTree-based image database data processing method of claim 1, wherein the third step comprises the steps of:
3.1, constructing a white list of the geographic marking information, and adding the non-sensitive data into the white list every time when the non-sensitive data which is wrongly identified by the algorithm is manually found, so that the fault tolerance rate is improved;
3.2 after the sensitive data are screened by the white list, the marked content with high sensitive level is directly deleted; labeling the medium and low sensitivity levels, extracting the sensitivity characteristic words in the labels, randomly selecting a desensitization means of deletion, replacement and generalization for processing, then recalculating the desensitization labeled sensitivity, and completely deleting the corresponding labels if the desensitization labeled sensitivity does not meet the public requirement after iteration preset times; when the non-sensitive word with the maximum similarity to the current sensitive characteristic word is selected as the replacement in the selection replacement, the specific content of the label description is abstracted when the generalization operation is selected, so that the description range comprises more non-sensitive information.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110139298.4A CN112463804B (en) | 2021-02-02 | 2021-02-02 | KDTree-based image database data processing method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110139298.4A CN112463804B (en) | 2021-02-02 | 2021-02-02 | KDTree-based image database data processing method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112463804A true CN112463804A (en) | 2021-03-09 |
CN112463804B CN112463804B (en) | 2021-06-15 |
Family
ID=74802248
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110139298.4A Active CN112463804B (en) | 2021-02-02 | 2021-02-02 | KDTree-based image database data processing method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112463804B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115774769A (en) * | 2022-11-17 | 2023-03-10 | 北京中知智慧科技有限公司 | Sensitive word checking processing method and device |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE69725677D1 (en) * | 1996-10-25 | 2003-11-27 | Navigation Tech Corp | Device and method for storing geographic data on a physical storage medium |
US20150339848A1 (en) * | 2014-05-20 | 2015-11-26 | Here Global B.V. | Method and apparatus for generating a composite indexable linear data structure to permit selection of map elements based on linear elements |
CN106874415A (en) * | 2017-01-23 | 2017-06-20 | 国网山东省电力公司电力科学研究院 | Environmental sensitive area database construction method and server based on generalized information system |
US20170188067A1 (en) * | 2015-12-28 | 2017-06-29 | The Nielsen Company (Us), Llc | Methods and apparatus to perform identity matching across audience measurement systems |
CN108257119A (en) * | 2018-01-08 | 2018-07-06 | 浙江大学 | A kind of immediate offshore area floating harmful influence detection method for early warning based near ultraviolet image procossing |
CN109257385A (en) * | 2018-11-16 | 2019-01-22 | 重庆邮电大学 | A kind of location privacy protection strategy based on difference privacy |
CN109446288A (en) * | 2018-10-18 | 2019-03-08 | 重庆邮电大学 | One kind being based on the internet Spark concerning security matters map detection algorithm |
-
2021
- 2021-02-02 CN CN202110139298.4A patent/CN112463804B/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE69725677D1 (en) * | 1996-10-25 | 2003-11-27 | Navigation Tech Corp | Device and method for storing geographic data on a physical storage medium |
US20150339848A1 (en) * | 2014-05-20 | 2015-11-26 | Here Global B.V. | Method and apparatus for generating a composite indexable linear data structure to permit selection of map elements based on linear elements |
US20170188067A1 (en) * | 2015-12-28 | 2017-06-29 | The Nielsen Company (Us), Llc | Methods and apparatus to perform identity matching across audience measurement systems |
CN106874415A (en) * | 2017-01-23 | 2017-06-20 | 国网山东省电力公司电力科学研究院 | Environmental sensitive area database construction method and server based on generalized information system |
CN108257119A (en) * | 2018-01-08 | 2018-07-06 | 浙江大学 | A kind of immediate offshore area floating harmful influence detection method for early warning based near ultraviolet image procossing |
CN109446288A (en) * | 2018-10-18 | 2019-03-08 | 重庆邮电大学 | One kind being based on the internet Spark concerning security matters map detection algorithm |
CN109257385A (en) * | 2018-11-16 | 2019-01-22 | 重庆邮电大学 | A kind of location privacy protection strategy based on difference privacy |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115774769A (en) * | 2022-11-17 | 2023-03-10 | 北京中知智慧科技有限公司 | Sensitive word checking processing method and device |
Also Published As
Publication number | Publication date |
---|---|
CN112463804B (en) | 2021-06-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2812883B1 (en) | System and method for semantically annotating images | |
KR101377389B1 (en) | Visual and multi-dimensional search | |
CN110399515B (en) | Picture retrieval method, device and system | |
CN112256939B (en) | Text entity relation extraction method for chemical field | |
WO2023108980A1 (en) | Information push method and device based on text adversarial sample | |
MX2008013657A (en) | Annotation by search. | |
JP2006508446A (en) | Information storage and retrieval method | |
US11797705B1 (en) | Generative adversarial network for named entity recognition | |
CN106407445B (en) | A kind of unstructured data resource identification and localization method based on URL | |
US10831820B2 (en) | Content based image management and selection | |
Manandhar et al. | Learning structural similarity of user interface layouts using graph networks | |
CN111325030A (en) | Text label construction method and device, computer equipment and storage medium | |
CN108491543A (en) | Image search method, image storage method and image indexing system | |
Li et al. | An automatic approach for generating rich, linked geo-metadata from historical map images | |
CN112035723A (en) | Resource library determination method and device, storage medium and electronic device | |
CN112463804B (en) | KDTree-based image database data processing method | |
US11163761B2 (en) | Vector embedding models for relational tables with null or equivalent values | |
CA3012647A1 (en) | Content based image management and selection | |
Liaqat et al. | Applying uncertain frequent pattern mining to improve ranking of retrieved images | |
Larson et al. | Ranking and representation for geographic information retrieval | |
Zhu et al. | Using thesaurus to model keyblock-based image retrieval | |
Deniziak et al. | World wide web CBIR searching using query by approximate shapes | |
Wu et al. | Design of a Computer‐Based Legal Information Retrieval System | |
CN113392312A (en) | Information processing method and system and electronic equipment | |
Sumathy et al. | Image Retrieval and Analysis Using Text and Fuzzy Shape Features: Emerging Research and Opportunities: Emerging Research and Opportunities |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |