CN104200240A - Sketch retrieval method based on content adaptive Hash encoding - Google Patents
Sketch retrieval method based on content adaptive Hash encoding Download PDFInfo
- Publication number
- CN104200240A CN104200240A CN201410493545.0A CN201410493545A CN104200240A CN 104200240 A CN104200240 A CN 104200240A CN 201410493545 A CN201410493545 A CN 201410493545A CN 104200240 A CN104200240 A CN 104200240A
- Authority
- CN
- China
- Prior art keywords
- sketch
- feature
- window
- app
- hash
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a sketch retrieval method based on content adaptive Hash encoding. The method is characterized by including the steps: candidate windows are adaptively selected from sketches or outlines under retrieval according to their contents and are used for feature extraction, and information included in whole pictures are evenly distributed to the windows; significance of each feature window is detected by a key point based significance detection method; local visual features, significance and structural spatial features of the stretches or outlines are combined and complied into Hash feature codes by a locality sensitivity based Hash algorithm; the Hash feature codes of the sketches or outlines are indexed, Hamming distances of the Hash feature codes are calculated to measure similarity of the sketches, and highly similar ones of the sketches are returned to users. The method has the advantages that precision is higher, applicable range is wider and matching capacity is higher.
Description
Technical field
The present invention relates to image processing field, in particular a kind of sketch search method of content-based self-adaptation Hash coding.
Background technology
In recent years, utilize sketch (vision material information such as hand-drawing graphics, picture and three-dimensional model) to retrieve as input is the study hotspot of computer vision field always.This is because day by day universal along with touch-screen equipment, and the modes such as people prefer more to be made to use gesture, felt pen complete the input of various information and mutual with computing machine, and it is very simple that this mode can give expression to user's intention and operation better.Meanwhile, by the mode of cartographical sketching, carry out all kinds of retrieval tasks, this is for using the user of handheld input device (as Apple iPhone/iPad, Microsoft Surface and other all kinds of panel computers etc.) that very strong convenience is provided.
In all kinds of sketch interactive tasks, sketch occupies an important position in whole sketch interactive computing field from the matching problem of the different visual information carriers such as hand-drawing graphics, picture and 3D model, and this is also the basic research problems of the search method based on sketch equally.In order to obtain in retrieval tasks as far as possible with people's cognition, visual experience directly perceived result always, the similarity that need to have a kind of efficient algorithm can measure accurately and rapidly input sketch and be retrieved between information, the search method research based on sketch is exactly to be intended to address this problem.From actual value, on the one hand, the search method based on sketch and the application of Practical Project technology are closely related, not only have very strong field applicability, or the multiple main core technology based on sketch application.On the other hand, it also has very strong scientific research value, and it has explored the basic method that computer recognition is artificially created visual pattern.
In traditional retrieval tasks, a sketch is regarded as the set of a series of Freehandhand-drawing strokes conventionally, with this, represents the information such as skeleton, profile of an object.Yet, other appearance details that object has, contents such as color, texture has but inevitably been lost in being converted into the process of sketch.Due to this characteristic, the search method based on sketch is compared and is had very large difference with traditional searching algorithm based on picture.Does recently for a period of time, the research work carried out in the field attempt to answer a problem always: how can reasonably extract and compare the characteristic information containing in each sketch?
In the recent period, the method based on cutting apart (segmentation) is widely used, and is very effective in sketch retrieval and verified this class algorithm in identification field.The stroke that they conventionally can comprise sketch in the process realizing is cut apart, and with this, reduces computation complexity.Then, then the stroke after cutting apart is carried out to the extraction of the attributes such as topology, geometric configuration.Yet, for sketch, carry out accurate stroke and cut apart and be difficult to accomplish, especially, for the picture that comprises natural landscape or 3D model, it is more difficult that this step becomes.This is because the contour feature of this type of visual information has comprised a lot of noises conventionally, and carry out that perfect stroke cuts apart to them is almost an impossible mission.Thereby this shortcoming has greatly limited the class methods cut apart based on the stroke applicability when being used to based on sketch, picture or 3D model be retrieved just.
Other research work is referred from traditional picture retrieval algorithm, by provincial characteristics (patch), comes the similarity of comparison sketch to complete retrieval tasks.First, a sketch is divided into zonule one by one equably, then therefrom extracts different visual signature descriptors.Under normal conditions, these class methods cover the grid with region overlapping or the large window such as dense on whole sketch, with this, describe the feature distribution situation of this sketch.This is highly suitable for common picture, because a pictures has often comprised the minutias such as abundant color or texture, still, for the sparse contour images of this class of sketch that only comprises limited stroke, is inapplicable.In this case, the most information that sketch comprises all can cover the image-region the inside of several highly significants, and makes almost vacancy of remaining region.Due to the unbalanced phenomenon of this extreme, can produce many invalid Feature Descriptors, after during the characteristic similarity of these descriptors after participating in calculates, can greatly reduce the validity of calculating, and, before they being carried out to index at use hash algorithm, need it to carry out binaryzation, this can make only visual information be lost further, thereby allows situation become even worse.
Another problem of search method based on provincial characteristics is the computing method of similarity between sketch weaker, accurately not strict.A cartographical sketching has conventionally comprised the contents such as lines stroke miscellaneous rather than color or texture and has showed different objects, thereby sketch has very large difference with traditional picture both ways: otherness (due to the disappearance of these class very important visual information such as color or texture) between otherness in huge class (because each sketch drafting person more or less has the subjective understanding of oneself for same thing) and less class.Even this has just caused the sketch of same subject still to comprise a large amount of dissimilar provincial characteristicss.Therefore, " similar picture has often comprised a large amount of similar provincial characteristicss ", this concept being widely used in picture retrieval algorithm is inaccurate for the search method based on sketch, and it retrains too strict to the definition of similarity thereby becomes effective not.
Therefore,, in view of above analysis can be found, existing retrieval technique imperfection based on sketch, has yet to be improved and developed.The search method of the present invention's research based on provincial characteristics, because it is higher to compare its applicability of method of cutting apart based on stroke, and concentrates two problems that exist in the above-mentioned search method based on provincial characteristics that solved.
Summary of the invention
The object of the present invention is to provide a kind of sketch search method of content-based self-adaptation Hash coding, be intended to solve the existing sketch search method based on Hash coding and when extracting visual signature, do not take into account sketch self distribution of content feature, the method adopting can reduce the validity of calculating greatly, and the computing method to similarity between sketch are weaker, strict problem accurately not.
Technical scheme of the present invention is as follows: a kind of sketch search method of content-based self-adaptation Hash coding, and it comprises following concrete steps:
Steps A: based on outward appearance constraint and diversity, retrain, to the sketch being retrieved or profile diagram according to its content-adaptive choose candidate window for feature extraction, realize the information that whole image comprises and be distributed to equably in each window;
Step B: the conspicuousness of the conspicuousness detection method detected characteristics window based on key point;
Step C: the hash algorithm based on local sensitivity combines the local visual feature of sketch or profile diagram, conspicuousness and structure space feature, is compiled into feature Hash codes;
Step D: the feature Hash codes of sketch or profile diagram is carried out to index, measure the similarity between sketch by the Hamming distance between calculated characteristics Hash codes, and the high result of similarity is returned to user.
Described sketch search method, wherein, also can retrieve picture and 3D model, before picture and 3D model are retrieved, carry out pre-service to it, and they are changed into contour line picture.
Described sketch search method, wherein, becomes the method for contour line picture to be respectively with 3D model conversation in picture: for picture, want the conspicuousness detection algorithm of jointing edge extraction algorithm and picture to calculate the remarkable configuration figure of this picture; For 3D model, according to the matching algorithm based on visual angle, calculate the contour projection charts that model is corresponding.
Described sketch search method, wherein, for the candidate window Algorithms of Selecting of feature extraction: be first the grid of the sketch installation initialization n*n of input; A uniform sampling m*m initial seed from grid again; Then be the window calculation HOG feature h in each grid
i; Finally calculate overall HOG feature
Described sketch search method, wherein, described outward appearance constraint is denoted as C
app, be specifically expressed as: C
app(h) :=F
app(h)>=k
app* F
app(H)
Wherein,
f
appbe the target equation of outward appearance constraint, work as F
appcalculate the value of gained when higher, it represents that the visual signature information that this characteristic window comprises is more.
Described sketch search method, wherein, described diversity constraint is denoted as C
var, be specifically expressed as: C
var(h) :=F
var(h)≤k
var* F
var(H)
Wherein,
F
varthe target equation of diversity constraint, if F
varthe lower and F of value
apphigher, show that the value of each dimension in window feature vector h is all higher.
Described sketch search method, wherein, the concrete grammar of the conspicuousness of the conspicuousness detection method detected characteristics window based on key point is: first, use Harris-Laplace detecting device as sketch conspicuousness extracting tool, for each characteristic window w
i, define its conspicuousness k
ifor:
Wherein, Number (S
i) be illustrated in characteristic window w
ithe significant point number that middle use Harris-Laplace detecting device extracts; Area (w
i) be w
ithe number that comprises pixel.
Described sketch search method, wherein, the method that is compiled into feature Hash codes is: make f
ifor from characteristic window w
iin the proper vector that extracts first its two-value is turned to vector
; Then, follow the computation process of similar hash algorithm, corresponding according to each window
and k
ivalue calculates the feature Hash codes of this window; Then, sketch is divided into two respectively in the horizontal and vertical directions, obtains the locus of four separations, the candidate window being positioned on each locus is carried out respectively to Hash coding; Finally, thus by the Hash codes on four locus successively head and the tail splicings being obtained representing the feature Hash codes of whole sketch.
Described sketch search method, wherein, Feature Descriptor also can be selected Feature Descriptor or the local linear Gabor Feature Descriptor that yardstick is constant.
Described sketch search method, wherein, has selected without the HOG feature of normalized and has described the visual information that each subwindow comprises.
Beneficial effect of the present invention: the present invention is by the process extracting sketch or profile diagram feature taking the distribution of content situation of itself into account, all features are distributed in candidate window as far as possible equably, and the proper vector extracting more can characterize himself feature of single sketch.Compare the searching algorithm of cutting apart based on stroke, these features are subject to the impact of the factors such as the noise that comprises in picture, 3D model silhouette figure and texture less; Compare other search method based on Hash, algorithm proposed by the invention has higher precision, adaptability and stronger matching capacity widely.The application of this method can be generalized to other so that characteristic information is carried out to the research field that Hash is encoded to gordian technique, is of universal significance.
Accompanying drawing explanation
Fig. 1 is used the present invention in the result schematic diagram of the enterprising line retrieval of several data collection.
Fig. 2 a, 2b, 2c are sketch characteristic window Selection Strategy schematic diagram.
Fig. 3 a, 3b, 3c are the remarkable window calculation result schematic diagrams of sketch.
Fig. 4 is sketch provincial characteristics hash algorithm process flow diagram.
Fig. 5 calculates picture contour images intermediate result schematic diagram.
Fig. 6 a, 6b are the curve maps of the different assembly properties of comparison this method.
Fig. 7 a, 7b are the performance comparative graph of using algorithms of different to retrieve.
Embodiment
For making object of the present invention, technical scheme and advantage clearer, clear and definite, referring to accompanying drawing, developing simultaneously, the present invention is described in more detail for embodiment.
Searching algorithm based on sketch proposed by the invention mainly comprises following three ingredients: in conjunction with the own feature of input sketch, according to two class constraint conditions selected characteristic extraction adaptively window; The key point information comprising with window detects the conspicuousness of each characteristic window adaptively; Structural information above two category feature information with sketch combines, and by LSH algorithm, they is compiled into Hash codes for creating aspect indexing.Next, this instructions is divided into equally these several parts the present invention is illustrated.
As shown in Figure 2 a, a given sketch, is first divided into the uniform grid of n*n.Then, as shown in Figure 2 b, on the point of crossing of these grids, choose equably m*m point as the initial seed that produces all characteristic window, then, in sketch (x, y) locational Seed Points is defined as subwindow set Δ w (x, y, i), Δ w (x, y, i) represented all i circle windows that are looped around on seed.As shown in Figure 2 c, wherein black round dot represents the Seed Points of choosing, and around 4 the most contiguous subwindows of this Seed Points, is exactly the 1st circle window x around this Seed Points, is designated as Δ w (x, y, 1); Around the 1st circle window and with it the most contiguous all subwindows be exactly the 2nd circle window y around this Seed Points, be designated as Δ w (x, y, 2).
In order to give each initial seed the final window w (x that produces suitable size, y), just need iteratively Δ w (x, y, i) to be joined to w (x successively, y), until w (x, y) meets some requirements, constraint shows that its characteristic information comprising is sufficient, or as w (x, while y) becoming an illegal window, for example window has overflowed the edge of whole sketch or it become excessive (be greater than whole sketch 1/4th).
Therefore, constraint choose and design becomes and is even more important, this is directly connected to the quality of the characteristic window of finally choosing, if constraint is too strict, the characteristic information that will cause most window to comprise is not enough; Otherwise it is excessive that characteristic window can become, to such an extent as to a large amount of regions is repeatedly included in a plurality of windows, makes the characteristic information undue redundancy that becomes, and strengthened the calculated amount of a rear step characteristic similarity comparison.The present invention proposes following two kinds of effective constraints policies.
HOG (Histogram of Oriented Gradients due to image, gradient orientation histogram) feature has been widely used in the computer vision research fields such as object detection and image retrieval and has obtained good effect, therefore, this method has been selected without the HOG feature of normalized and has been described the visual information that each subwindow comprises.In the HOG of standard feature calculation process, often adopt normalized to remove the impact of shadow on image, and sketch is all the black-and-white two color images that consist of stroke and background and does not exist illumination to convert, thereby the not normalized HOG feature of the use precision that effect characteristics does not extract, and have benefited from the minimizing of calculated amount, accelerated to a certain extent arithmetic speed.Remember vectorial h={b
1, b
2... b
nbe the HOG feature extracting from window w (x, y), Δ h (x, y, i) be Δ w (x, y, i) the HOG proper vector of corresponding window set; Note H is the HOG feature H vector of whole sketch, because histogram has additive property, therefore when calculating the HOG proper vector h of all subwindows in sketch
iafter, can be by these values be added up and directly obtain the HOG histogram feature H of full figure, it is defined as
as the constraint of the outward appearance of the sketch characteristic window of giving a definition, be denoted as C
app, it is defined as follows:
C
app(h) :=F
app(h)>=k
app* F
app(H) formula 1
Wherein,
f
appbe the target equation of outward appearance constraint, it has calculated the average of HOG feature h in essence.Work as F
appcalculate the value of gained when higher, it represents that the visual signature information that this characteristic window comprises is more, on the contrary, and lower F
appvalue represents that whole window is almost empty.Obviously, constraint condition C
appguaranteed that each characteristic window need to have enough abundant visual signature information.
Second constraint that this method proposes is called diversity constraint, is designated as C
var, it is defined as follows:
C
var(h) :=F
var(h)>=k
var* F
var(H) formula 2
Wherein,
F
varbe the target equation of diversity constraint, it is actually the variance of HOG feature h.If F
varthe lower and F of value
apphigher, show that the value of each dimension in window feature vector h is all higher.Therefore, meet C
varthe window of constraint will comprise more diversified feature, rather than the straight line of single direction or line segment etc., and it is very useful that this multifarious feature has been proved to be in sketch search problem.
It should be noted that parameter k
appand k
varcontrolled the impact that sketch overall situation HOG feature H produces two kinds of constraint conditions.By specific experiment, find k
appand k
varbe set to respectively 0.8 and 1 and can obtain best retrieval performance.More than having defined after two class constraints, take each initial seed position is starting point, iteratively subwindow is around added to wherein until everywhere window all meets this two classes constraint, has so just completed choosing of feature candidate window.The algorithm of following above window Selection Strategy is designated as window considerations constraint Algorithms of Selecting, has summarized as follows the workflow of above-mentioned whole algorithm:
Step a: be the grid of input sketch initialization n*n;
Step b: a uniform sampling m*m initial seed from grid;
Step c: be the window calculation HOG feature h in each grid
i;
Steps d: calculate overall HOG feature H.
Specific algorithm is as follows:
Owing to having mentioned, to " iteratively " use " two constraints " produce candidate window above, so a kind of implementation method example of simple, intuitive is provided by false code here, helps user to understand and how to use two constraints.
It should be noted that constraint C
apptarget equation F
appcan being incremented calculate, that is to say that it meets equation:
F
app(h+ Δ h)=F
app(h)+F
app(Δ h) formula 3
Along with window ceaselessly increases, only need to be by the F of incremental portion subwindow institute character pair vector
appvalue joins in the result of a front iteration, and need on each iteration gained new window, not recalculate F
appvalue.Therefore, in each iterative process, verify whether characteristic window meets constraint C
appvery rapidly.Although F
varcan not being incremented calculate, but until each feature candidate window is meeting constraint condition C
appit does not need to be calculated before, so as a whole, the computation process of the whole characteristic window of this method is relatively fast.
In experiment, can see, no matter be Feature Descriptor, local linear Gabor Feature Descriptor (the Gabor Local Line-based Feature that selects yardstick constant, GALIF) or previously described HOG Feature Descriptor, use the present invention's characteristic window Selection Strategy proposed above all can improve significantly the precision of retrieval.
After completing the choosing of sketch characteristic window, this method has also proposed a kind of method that detects each window conspicuousness in input sketch.For sketch, characterize its local features, the conspicuousness extraction algorithm of use based on unique point, for example multiple dimensioned Gauss model, Hessian algorithm and Harris-Laplace detecting device, than using the conspicuousness extraction algorithm based on region to come more effectively.This is due to different from the picture that comprises continuum, and a sketch has comprised many lines that independently separate and point conventionally.Conspicuousness extraction algorithm based on unique point is designed to find significant point in image just, so be very suitable for detecting node or the flex point in sketch, this is also the key message place comprising in sketch.In the present invention, found through experiments Harris-Laplace detecting device and when detecting sketch key point, having better performance, therefore, use it as sketch conspicuousness extracting method of the present invention.
For each, belong to the candidate window w of W
i(wherein W is the result that algorithm 1 returns by mentioned earlier, is all characteristic window set of choosing), defining its conspicuousness is k
i, according to following formula, can be calculated:
Wherein, Number (S
i) be illustrated in characteristic window w
ithe significant point number that middle use Harris-Laplace detecting device extracts; Area (w
i) be w
ithe number that comprises pixel.Intuitively, can draw: when window is less and comprise more significant points, this window is just more remarkable.
It should be noted that the conspicuousness in order to prevent characteristic window is too responsive for its size, resolution, above formula adopts the secondary radical sign item of window Pixel-level area to characterize its size.Fig. 3 has shown to be the intermediate result of a sketch detected characteristics window conspicuousness, and wherein Fig. 3 a is the example of input sketch; Soft dot in Fig. 3 b is illustrated in all key points of using Harris-Laplace detecting device to find out in this sketch; Fig. 3 c has indicated by formula 4 and has calculated the most significant candidate feature window that front 3 the mutual area coverages that obtain are less than 20% with square frame.
The vision descriptor of calculated candidate characteristic window with and conspicuousness after, in order to make these information can effectively be used among similarity comparison and index, these contents organically need to be integrated.Measuring similarity algorithm based on Hash can be retrieved large-scale data set very effectively, guarantees high computing velocity simultaneously, thereby the present invention has used the hash algorithm of local sensitivity to encode to the feature extracting, and detailed process is as described below.This method is used HOG to describe algorithm as window feature in implementation procedure.
Make f
ifor from characteristic window w
iin the proper vector that extracts, by by f
iin the highest position of front 40% value be made as 1, remaining is set to-1, can be by f
itwo-value turns to vector
.It is pointed out that having benefited from two kinds of characteristic window that this algorithm proposes chooses constraint condition, and then effectively guaranteed that each window can comprise abundant characteristic information, make in binaryzation process, the loss of information is greatly reduced.Then, the present invention is with reference to the computation process of similar hash algorithm (Sim-hash), corresponding by each window
and k
ivalue calculates the feature Hash codes of this window.
In actual computation, by window conspicuousness k
ias
weight.The spatial information that sketch comprises has been proved to be a very useful category feature in sketch searching field.Therefore, for the spatial relationship of local feature in sketch is covered in feature Hash codes, first, sketch is divided into two respectively in the horizontal and vertical directions, obtains the locus of four separations, as shown in flow process in Fig. 4 (a).Then, according to method mentioned above, the candidate window being positioned on each locus is carried out respectively to Hash coding, whole process is as shown in the flow process in Fig. 4 (b)～(e).Finally, by the head and the tail splicing successively of the Hash codes on four locus, as shown in flow process (f), can obtain having represented the feature Hash codes of whole sketch.The contour images of given any two sketches, picture or 3D model projection, the Hamming distance between their feature Hash codes (Hamming Distance) is exactly its similarity.
Following part has been introduced specific implementation details of the present invention, is included in retrieval before to the concrete setting of the preprocessing process of picture and 3D model, parameter, characteristic index scheme and whole query script.
Sketch is a kind of black white image that comprises lines of outline normally, and picture has often comprised abundant color and texture miscellaneous, 3D model is a kind of dough sheet set in three dimensions, obviously, the feature of these three kinds of data is completely different, cannot directly they be compared, be retrieved.Therefore, between retrieving starts, in order to allow user using sketch as inputting retrieving image and 3D model, need to carry out pre-service to this two category information, they are changed into the contour images of class sketch.
For a given secondary picture, first use Canny or other image edge extraction algorithm to calculate the profile diagram E of picture in its entirety
c, but obviously, this contour images has inevitably comprised many false edges lines that produce due to background texture.For the body matter that finds picture to express (also i.e. the desired part being retrieved of this picture), need to use the conspicuousness detection algorithm of picture that this partial content is marked, and be S the remarkable seal of its correspondence.Then, this method has also been used maximum filter (Maximum Filter, MF) S has been done to a filtering and processed, so just can expand a little the marking area of picture, avoided causing the outline of picture body matter to be lost due to the segmentation errors of algorithm.Finally, according to following formula, just can calculate the remarkable configuration figure E of this picture, computation process as shown in Figure 5.
For a given 3D model, the coupling based on visual angle (View-based matching) algorithm that the present invention proposes in paper " Sketch-Based Shape Retrieval " according to people such as Mathias Eitz calculates the contour projection charts that model is corresponding.In computation process, in order to guarantee that projection extracts the stability of result, the present invention adopts Lloyd relaxed algorithm by loop iteration, is each 3D model approximately 14 projection visual angles of sampling out equably from the encirclement sphere of its correspondence.And enlightening outline line (Suggestive Contours) is used to extract suitable lines of outline from model projection figure.
Afterwards, no matter be picture remarkable configuration figure or the figure of the projected outline of 3D model calculating according to said method, all use a smallest square bounding box that their cuttings from baseline results out and zoom to the resolution of 160*160, are reduced to the impact that picture size size and deformation produce result for retrieval with this.Then, according to the computation process of algorithm 1, the profile diagram of input sketch, picture or model is divided into the grid of 80*80, therefrom uniform sampling goes out 15*15 seed.In each feature subwindow, the not normalized HOG feature histogram that calculating comprises 8 directions is as the h in constraint condition, thereby each proper vector has comprised 8 dimensions.All characteristic window have all been scaled to the block of pixels of 16*16 afterwards, so that next carry out Visual Feature Retrieval Process.Finally, because not normalized HOG feature histogram has before been calculated, after they are normalized, HOG feature just can be used to describe the proper vector f of each characteristic window.
A given sketch is arbitrarily as input, the profile diagram of all sketches, picture or model and the Hamming distance of feature Hash codes between it in measurement database, and ascending sequence, just can find sketch the most similar to this sketch in database, picture or 3D model.Because all Hash codes are all binaryzations, therefore only by simple displacement, and exclusive disjunction just can very rapidly calculate the Hamming distance between it, even characteristic is not carried out to index, can not have very high retrieval rate yet.In order further to accelerate retrieval performance, the multiple Index Algorithm (Fast Search in Hamming Space with Multi-Index Hashing) of Hamming space Hash proposing according to people such as Norouzi combines it with the present invention, characteristic has been carried out to index, so just made this method inquiry request each time can be completed in the time at sublinear.Summed up as follows the flow process of whole search method of the present invention:
First, for all pictures in database or 3D model produce suitable profile diagram E; Secondly according to constraint Algorithms of Selecting, be that all profile diagram E produce candidate feature window w; For each characteristic window w, calculate corresponding HOG feature f and conspicuousness k again; Then according to all HOG feature f in profile diagram E and conspicuousness k, be that each profile diagram E produces a feature Hash codes h; Be finally that all feature Hash codes h create index I.
According to the second step to the of retrieving four steps, for inputting sketch S, calculate its feature Hash codes h
s; From index I, calculate this Query Result R and return to user
In order to support and verify research method proposed by the invention and gordian technique, on three standard data sets that are widely used, this method has been carried out to Performance Ratio with the searching algorithm based on sketch of other up-to-date forefront respectively.Magic Sketch data set is set up by people such as Liang, has wherein altogether comprised 1100 sketches, and according to painted content, they are divided into respectively 55 classifications.It is reference that these sketches all be take UK PTO trademark database, refrigerator electric elements figure and the engineering drawing of MPEG-CE1 trademark image database, Britain trademark patent office, select wherein representational shape, by 10 people, drawn and obtained, this data set is used to verify the validity of each ingredient of algorithm that this method proposes.TU Berlin data set is constructed by people such as Eitz, has wherein comprised 31 kinds of different themes, each theme have an example sketch with and 40 corresponding test pictures.This data set is the validity when verifying that this method is used to the picture retrieval task based on sketch.PSB data set is set up by people such as Eitz, and its each sketch comprising is all easy to identification, and corresponding to the class 3D model in Princeton 3 d model library (Princeton Shape Benchmark, PSB).Therefore, this data set can be used for verifying validity when this method is used to the 3D model index task based on sketch.
In order to evaluate the performance of the retrieval technique based on sketch proposed by the invention, this method adopts recall ratio (Recall), precision ratio (Precision) and data set benchmark score (Benchmark Score) as Performance Evaluating Indexes.In order to evaluate more intuitively existing method, with the form (Precision-Recall curve, PR curve) of curve, provide the relation between recall ratio or precision ratio and the size of candidate's graphic result collection.Along with returning results increasing of number, recall ratio will increase gradually precision ratio and will progressively reduce, and this is that computing method by precision ratio and recall ratio are determined.When the window of return structure number increases, the denominator of precision ratio is larger, and precision ratio is lower; And for recall ratio, window is larger, the number of the correlated results returning is just more, and the molecule of recall ratio is just larger, and the value of recall ratio is just larger.Clearly, in recall ratio figure and precision ratio figure, the higher retrieval effectiveness of curve is better because in the situation that return to the candidate vector figure of same number, curve more Gao Ze represent corresponding recall ratio or precision ratio higher; Vice versa.Data set benchmark score is to be proposed in paper " Sketch-Based Image Retrieval:Benchmark and Bag-of-Features Descriptors " by people such as Mathias Eitz, is the another kind of index for evaluating data searching algorithm performance.This index is by after relatively under the same retrieve data collection of use, given inquiry is inputted, and the Kendall coefficient of rank correlation between the legitimate reading of the ranking result of searching algorithm gained and artificial mark judges the quality of this searching algorithm.The benchmark score of searching algorithm gained is higher, shows that the result for retrieval of this algorithm is got over the cognitive result close to the mankind.In addition, the present invention has also tested searching algorithm and has completed the time that primary retrieval task spends, to assess performance working time of proposed scheme.
For verify that this algorithm proposes for selecting two constraint C of characteristic window
app(formula 1) and C
varthe validity of (formula 2), first, the algorithm proposing in paper " Large Scale Sketch Based Image Retrieval Using Patch Hashing " according to Konstantinos Bozas has been realized an algorithm datum line that uses overlapping grid (Grid) to carry out feature extraction.Then, it is retrained to C with only using
appand the C of use constraint simultaneously
appand C
varthis algorithm compare.For the characteristic window conspicuousness k that verifies that this algorithm proposes
ivalidity, k in above algorithm is realized
iall be set to 1, and by all C of they same uses
app, C
varand k
irealization compare.Fig. 6 a has shown the retrieval performance that all kinds of algorithms are obtained on Magic Sketch data set as mentioned above, therefrom can find that two characteristic window selection constraints that this algorithm proposes are complementary, and, remove C
app, C
varand k
iin any one assembly all can make the retrieval performance of this algorithm decrease.Therefore the C that, this algorithm proposes
app, C
varand k
ibe indispensable, each ingredient all has its validity.
In addition, in order further to show the performance characteristics of this method, the searching algorithm realization of having used SIFT, GALIF and HOG Feature Descriptor to carry out on uniform grid after feature extraction is realized and being compared with the algorithm that uses them to carry out feature extraction on the feature extraction region of the selected taking-up of this algorithm respectively.The PR curve of Fig. 6 b has shown the comparative result of above-mentioned algorithm realization on Magic Sketch data set.From figure, can find no matter use which kind of Feature Descriptor, the result of carrying out feature extraction on the selected feature extraction region of this algorithm is all better than the result on uniform grid, thereby has proved the versatility of this method to different characteristic descriptor.It should be noted that, having benefited from GALIF Feature Descriptor has used multi-direction, multiple dimensioned feature sampling policy to substitute traditional histogram strategy sign visual signature, thereby can find from figure, in conjunction with this method and GALIF Feature Descriptor, can obtain best retrieval performance.But, owing to calculating GALIF Feature Descriptor, can bring huge time overhead, therefore, in order to guarantee the availability of algorithm, in the standard implementation of this method, still with HOG Feature Descriptor, calculate the visual signature of sketch.
Last in experiment, this algorithm has also been entered relatively with the searching algorithm based on sketch of other forefront.On TU Berlin data set, this algorithm is with making word bag model (Bag-of-Words, BW), critical shape (Key Shapes, KW) and min-hash (Min-hash, MH) searching algorithm compares, data set benchmark score is for passing judgment on the performance of searching algorithm, and result is as shown in table 1.In addition, on Magic Sketch data set, this algorithm is same based on there being the algorithm of inclined to one side SVM (BSVM), geometric space relation (Spatial Relations, SR) and geometric space contiguous (Spatial Proximity, SP) to compare; On PSB data set, this algorithm compares with the searching algorithm based on diffusion tensor (Diffusion Tensor, DT), grid SIFT (SIFT-Grid) and GALIF (GALIF-Grid).On these two data sets, all with the PR curve of standard, evaluate retrieval performance, result is respectively as shown in Fig. 7 a and Fig. 7 b.Above experimental result all shows that this method has higher retrieval precision than other searching algorithm, and performance is better.Referring to Fig. 1, shown respectively the result for retrieval example of this method on different pieces of information collection.
The benchmark score of table 1 algorithms of different
Searching algorithm based on sketch proposed by the invention, on Magic Sketch standard data set, approximately spends 1.87 seconds and completes sketch retrieval tasks one time.This result is actual recording on the desktop computer that has been equipped with the four core CPU of Intel 3.39GHz, 16GB internal memory, and code is used MATLAB programming and do not passed through parallel optimization.It should be noted that this method still can meet under the prerequisite of basic requirement of real-time, comparing other algorithm has higher retrieval precision.
Should be understood that, application of the present invention is not limited to above-mentioned giving an example, and for those of ordinary skills, can be improved according to the above description or convert, and all these improvement and conversion all should belong to the protection domain of claims of the present invention.
Claims (10)
1. a sketch search method for content-based self-adaptation Hash coding, is characterized in that, comprises following concrete steps:
Steps A: based on outward appearance constraint and diversity, retrain, to the sketch being retrieved or profile diagram according to its content-adaptive choose candidate window for feature extraction, realize the information that whole image comprises and be distributed to equably in each window;
Step B: the conspicuousness of the conspicuousness detection method detected characteristics window based on key point;
Step C: the hash algorithm based on local sensitivity combines the local visual feature of sketch or profile diagram, conspicuousness and structure space feature, is compiled into feature Hash codes;
Step D: the feature Hash codes of sketch or profile diagram is carried out to index, measure the similarity between sketch by the Hamming distance between calculated characteristics Hash codes, and the high result of similarity is returned to user.
2. sketch search method according to claim 1, is characterized in that, also can retrieve picture and 3D model, before picture and 3D model are retrieved, carry out pre-service to it, and they are changed into contour line picture.
3. sketch search method according to claim 2, it is characterized in that, become the method for contour line picture to be respectively with 3D model conversation in picture: for picture, want the conspicuousness detection algorithm of jointing edge extraction algorithm and picture to calculate the remarkable configuration figure of this picture; For 3D model, according to the matching algorithm based on visual angle, calculate the contour projection charts that model is corresponding.
4. sketch search method according to claim 1, is characterized in that, for the candidate window Algorithms of Selecting of feature extraction: be first the grid of the sketch installation initialization n*n of input; A uniform sampling m*m initial seed from grid again; Then be the window calculation HOG feature h in each grid
i; Finally calculate overall HOG feature
5. according to the sketch search method described in claim 1 or 3, it is characterized in that, described outward appearance constraint is denoted as C
app, be specifically expressed as: C
app(h) :=F
app(h)>=k
app* F
app(H)
Wherein,
f
appbe the target equation of outward appearance constraint, work as F
appcalculate the value of gained when higher, it represents that the visual signature information that this characteristic window comprises is more.
6. sketch search method according to claim 5, is characterized in that, described diversity constraint is denoted as C
var, be specifically expressed as: C
var(h) :=F
var(h)≤k
var* F
var(H)
Wherein,
f
varthe target equation of diversity constraint, if F
varthe lower and F of value
apphigher, show that the value of each dimension in window feature vector h is all higher.
7. sketch search method according to claim 1, it is characterized in that, the concrete grammar of the conspicuousness of the conspicuousness detection method detected characteristics window based on key point is: first, use Harris-Laplace detecting device as sketch conspicuousness extracting tool, for each characteristic window w
i, define its conspicuousness k
ifor:
Wherein, Number (S
i) be illustrated in characteristic window w
ithe significant point number that middle use Harris-Laplace detecting device extracts; Area (w
i) be w
ithe number that comprises pixel.
8. sketch search method according to claim 1, is characterized in that, the method that is compiled into feature Hash codes is: make f
ifor from characteristic window w
iin the proper vector that extracts first its two-value is turned to vector
then, follow the computation process of similar hash algorithm, corresponding according to each window
and k
ivalue calculates the feature Hash codes of this window; Then, sketch is divided into two respectively in the horizontal and vertical directions, obtains the locus of four separations, the candidate window being positioned on each locus is carried out respectively to Hash coding; Finally, thus by the Hash codes on four locus successively head and the tail splicings being obtained representing the feature Hash codes of whole sketch.
9. sketch search method according to claim 4, is characterized in that, Feature Descriptor also can be selected Feature Descriptor or the local linear Gabor Feature Descriptor that yardstick is constant.
10. sketch search method according to claim 4, is characterized in that, has selected without the HOG feature of normalized and has described the visual information that each subwindow comprises.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410493545.0A CN104200240B (en) | 2014-09-24 | 2014-09-24 | A kind of Sketch Searching method based on content-adaptive Hash coding |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410493545.0A CN104200240B (en) | 2014-09-24 | 2014-09-24 | A kind of Sketch Searching method based on content-adaptive Hash coding |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104200240A true CN104200240A (en) | 2014-12-10 |
| CN104200240B CN104200240B (en) | 2017-07-07 |
Family
ID=52085529
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410493545.0A Active CN104200240B (en) | 2014-09-24 | 2014-09-24 | A kind of Sketch Searching method based on content-adaptive Hash coding |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104200240B (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105069173A (en) * | 2015-09-10 | 2015-11-18 | 天津中科智能识别产业技术研究院有限公司 | Rapid image retrieval method based on supervised topology keeping hash |
| CN106126581A (en) * | 2016-06-20 | 2016-11-16 | 复旦大学 | Cartographical sketching image search method based on degree of depth study |
| CN106126585A (en) * | 2016-06-20 | 2016-11-16 | 北京航空航天大学 | Unmanned plane image search method based on quality grading with the combination of perception Hash feature |
| CN107402974A (en) * | 2017-07-01 | 2017-11-28 | 南京理工大学 | Sketch Searching method based on a variety of binary system HoG descriptors |
| CN108154155A (en) * | 2017-11-13 | 2018-06-12 | 合肥阿巴赛信息科技有限公司 | A kind of jewelry search method and system based on sketch |
| CN108681555A (en) * | 2018-04-08 | 2018-10-19 | 天津大学 | A kind of sketch image search method returned based on shape |
| CN108694411A (en) * | 2018-04-03 | 2018-10-23 | 南昌奇眸科技有限公司 | A method of identification similar image |
| CN108694406A (en) * | 2017-04-08 | 2018-10-23 | 大连万达集团股份有限公司 | The method compared for the X-Y scheme goodness of fit in engineering |
| CN108897746A (en) * | 2018-04-03 | 2018-11-27 | 南昌奇眸科技有限公司 | A kind of image search method |
| CN109033144A (en) * | 2018-06-11 | 2018-12-18 | 厦门大学 | Method for searching three-dimension model based on sketch |
| US10248664B1 (en) | 2018-07-02 | 2019-04-02 | Inception Institute Of Artificial Intelligence | Zero-shot sketch-based image retrieval techniques using neural networks for sketch-image recognition and retrieval |
| CN109960738A (en) * | 2019-03-15 | 2019-07-02 | 西安电子科技大学 | Extensive Remote Sensing Images search method based on depth confrontation Hash study |
| CN111528834A (en) * | 2020-03-25 | 2020-08-14 | 西安电子科技大学 | Real-time SAR image target detection system and method |
| CN112115292A (en) * | 2020-09-25 | 2020-12-22 | 海尔优家智能科技（北京）有限公司 | Picture searching method and device, storage medium and electronic device |
| CN113282775A (en) * | 2021-05-27 | 2021-08-20 | 上海垚亨电子商务有限公司 | Similar population expansion algorithm based on locality sensitive hashing algorithm |
| CN115114966A (en) * | 2022-08-29 | 2022-09-27 | 苏州魔视智能科技有限公司 | Method, device, equipment and storage medium for determining operation strategy of model |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101211356A (en) * | 2006-12-30 | 2008-07-02 | 中国科学院计算技术研究所 | Image inquiry method based on marking area |
| KR100848034B1 (en) * | 2007-03-23 | 2008-07-23 | 한양대학교 산학협력단 | Moment-based local descriptor using scale invariant feature |
| CN101576896A (en) * | 2008-05-09 | 2009-11-11 | 鸿富锦精密工业（深圳）有限公司 | Retrieval system and retrieval method for similar pictures |
-
2014
- 2014-09-24 CN CN201410493545.0A patent/CN104200240B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101211356A (en) * | 2006-12-30 | 2008-07-02 | 中国科学院计算技术研究所 | Image inquiry method based on marking area |
| KR100848034B1 (en) * | 2007-03-23 | 2008-07-23 | 한양대학교 산학협력단 | Moment-based local descriptor using scale invariant feature |
| CN101576896A (en) * | 2008-05-09 | 2009-11-11 | 鸿富锦精密工业（深圳）有限公司 | Retrieval system and retrieval method for similar pictures |
Non-Patent Citations (1)
| Title |
|---|
| 李勇: "基于内容的图像检索技术研究", 《中国博士学位论文全文数据库 信息科技辑》 * |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105069173B (en) * | 2015-09-10 | 2019-04-19 | 天津中科智能识别产业技术研究院有限公司 | The fast image retrieval method of Hash is kept based on the topology for having supervision |
| CN105069173A (en) * | 2015-09-10 | 2015-11-18 | 天津中科智能识别产业技术研究院有限公司 | Rapid image retrieval method based on supervised topology keeping hash |
| CN106126581A (en) * | 2016-06-20 | 2016-11-16 | 复旦大学 | Cartographical sketching image search method based on degree of depth study |
| CN106126585A (en) * | 2016-06-20 | 2016-11-16 | 北京航空航天大学 | Unmanned plane image search method based on quality grading with the combination of perception Hash feature |
| CN106126585B (en) * | 2016-06-20 | 2019-11-19 | 北京航空航天大学 | The unmanned plane image search method combined based on quality grading with perceived hash characteristics |
| CN106126581B (en) * | 2016-06-20 | 2019-07-05 | 复旦大学 | Cartographical sketching image search method based on deep learning |
| CN108694406A (en) * | 2017-04-08 | 2018-10-23 | 大连万达集团股份有限公司 | The method compared for the X-Y scheme goodness of fit in engineering |
| CN107402974A (en) * | 2017-07-01 | 2017-11-28 | 南京理工大学 | Sketch Searching method based on a variety of binary system HoG descriptors |
| CN107402974B (en) * | 2017-07-01 | 2021-01-26 | 南京理工大学 | Sketch retrieval method based on multiple binary HoG descriptors |
| CN108154155A (en) * | 2017-11-13 | 2018-06-12 | 合肥阿巴赛信息科技有限公司 | A kind of jewelry search method and system based on sketch |
| CN108897746A (en) * | 2018-04-03 | 2018-11-27 | 南昌奇眸科技有限公司 | A kind of image search method |
| CN108694411A (en) * | 2018-04-03 | 2018-10-23 | 南昌奇眸科技有限公司 | A method of identification similar image |
| CN108694411B (en) * | 2018-04-03 | 2022-02-25 | 南昌奇眸科技有限公司 | Method for identifying similar images |
| CN108897746B (en) * | 2018-04-03 | 2022-02-08 | 南昌奇眸科技有限公司 | Image retrieval method |
| CN108681555B (en) * | 2018-04-08 | 2019-08-02 | 天津大学 | A kind of sketch image search method returned based on shape |
| CN108681555A (en) * | 2018-04-08 | 2018-10-19 | 天津大学 | A kind of sketch image search method returned based on shape |
| CN109033144A (en) * | 2018-06-11 | 2018-12-18 | 厦门大学 | Method for searching three-dimension model based on sketch |
| CN109033144B (en) * | 2018-06-11 | 2021-10-22 | 厦门大学 | Three-dimensional model retrieval method based on sketch |
| US10248664B1 (en) | 2018-07-02 | 2019-04-02 | Inception Institute Of Artificial Intelligence | Zero-shot sketch-based image retrieval techniques using neural networks for sketch-image recognition and retrieval |
| CN109960738A (en) * | 2019-03-15 | 2019-07-02 | 西安电子科技大学 | Extensive Remote Sensing Images search method based on depth confrontation Hash study |
| CN109960738B (en) * | 2019-03-15 | 2020-12-08 | 西安电子科技大学 | Large-scale remote sensing image content retrieval method based on depth countermeasure hash learning |
| CN111528834B (en) * | 2020-03-25 | 2021-09-24 | 西安电子科技大学 | Real-time SAR image target detection system and method |
| CN111528834A (en) * | 2020-03-25 | 2020-08-14 | 西安电子科技大学 | Real-time SAR image target detection system and method |
| CN112115292A (en) * | 2020-09-25 | 2020-12-22 | 海尔优家智能科技（北京）有限公司 | Picture searching method and device, storage medium and electronic device |
| CN113282775A (en) * | 2021-05-27 | 2021-08-20 | 上海垚亨电子商务有限公司 | Similar population expansion algorithm based on locality sensitive hashing algorithm |
| CN113282775B (en) * | 2021-05-27 | 2023-10-03 | 上海焱祺华伟信息***技术有限公司 | Similar crowd expansion method based on local sensitive hash algorithm |
| CN115114966A (en) * | 2022-08-29 | 2022-09-27 | 苏州魔视智能科技有限公司 | Method, device, equipment and storage medium for determining operation strategy of model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104200240B (en) | 2017-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104200240A (en) | Sketch retrieval method based on content adaptive Hash encoding | |
| US10528620B2 (en) | Color sketch image searching | |
| Li et al. | A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries | |
| CN110738207A (en) | character detection method for fusing character area edge information in character image | |
| CN104063723B (en) | The stroke restoring method and device of the Off-line Handwritten Chinese | |
| Zhang et al. | Semantic classification of heterogeneous urban scenes using intrascene feature similarity and interscene semantic dependency | |
| CN105574063A (en) | Image retrieval method based on visual saliency | |
| CN105243139A (en) | Deep learning based three-dimensional model retrieval method and retrieval device thereof | |
| Yang et al. | Adaptive region matching for region‐based image retrieval by constructing region importance index | |
| CN104751463B (en) | A kind of threedimensional model optimal viewing angle choosing method based on sketch outline feature | |
| CN104899883A (en) | Indoor object cube detection method for depth image scene | |
| Seidl et al. | Automated classification of petroglyphs | |
| CN103870569A (en) | Colorful animal image retrieval method based on content and colorful animal image retrieval system based on content | |
| CN108959379A (en) | A kind of image of clothing search method of view-based access control model marking area and cartographical sketching | |
| CN105138672A (en) | Multi-feature fusion image retrieval method | |
| Biasotti et al. | Mathematical tools for shape analysis and description | |
| CN109408655A (en) | The freehand sketch retrieval method of incorporate voids convolution and multiple dimensioned sensing network | |
| Favreau et al. | Extracting geometric structures in images with delaunay point processes | |
| Xiao et al. | Sketch-based human motion retrieval via selected 2D geometric posture descriptor | |
| Manandhar et al. | Magic layouts: Structural prior for component detection in user interface designs | |
| CN105844299A (en) | Image classification method based on bag of words | |
| CN113723208B (en) | Three-dimensional object shape classification method based on canonical and other transformation conversion sub-neural network | |
| CN106570124B (en) | Remote sensing images semantic retrieving method and system based on object level correlation rule | |
| Bhatia et al. | A Model of Heteroassociative Memory: Deciphering Surprising Features and Locations. | |
| Saglam et al. | An efficient object extraction with graph-based image segmentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |