CN112800267B - Fine-granularity shoe print image retrieval method - Google Patents

Fine-granularity shoe print image retrieval method Download PDF

Info

Publication number
CN112800267B
CN112800267B CN202110152570.2A CN202110152570A CN112800267B CN 112800267 B CN112800267 B CN 112800267B CN 202110152570 A CN202110152570 A CN 202110152570A CN 112800267 B CN112800267 B CN 112800267B
Authority
CN
China
Prior art keywords
shoe
semantic
width
image
shoe print
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110152570.2A
Other languages
Chinese (zh)
Other versions
CN112800267A (en
Inventor
王新年
段硕古
王文卿
白桂欣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dalian Maritime University
Original Assignee
Dalian Maritime University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dalian Maritime University filed Critical Dalian Maritime University
Priority to CN202110152570.2A priority Critical patent/CN112800267B/en
Publication of CN112800267A publication Critical patent/CN112800267A/en
Application granted granted Critical
Publication of CN112800267B publication Critical patent/CN112800267B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/53—Querying
    • G06F16/538—Presentation of query results

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a fine-grained shoe print image retrieval method and a fine-grained shoe print image retrieval system, comprising the following steps: extracting shoe print attribute information; calculating the similarity of the shoe print attributes; calculating the similarity of the shoe print content; sorting score calculation combining the shoe print content information and the attribute information; and outputting the images in the dataset in descending order according to the sorting score to obtain a query result. By combining the shoe print attribute information, the shoe print content information and the shoe print semantic block space layout relation to calculate the similarity between shoe print images, the distinguishing property between the shoe print images with small difference is effectively increased, and therefore the fine-granularity shoe print image retrieval precision is improved.

Description

Fine-granularity shoe print image retrieval method
Technical Field
The invention relates to a fine-grained image retrieval method, in particular to a retrieval method for suspects shoe marks with the same pattern and different sizes.
Background
The current shoe print retrieval method mainly comprises retrieval methods based on global apparent features, local area features and key point features. The shoe print search algorithm based on the global apparent features mainly takes the whole shoe print image as input, and extracts features of the shoe print image, such as Fourier spectrum features, gbaor features, depth features and the like. Pradeep M.Patil takes the sole pattern as a texture image, firstly adopts a Randon transformation method to calculate the deflection angle of the sole pattern, and corrects the deflection angle; secondly, 8 Gabor filters in different directions are constructed, and filtering of patterns is carried out; then selecting 4 Gabor feature graphs with the maximum energy value, and dividing each feature graph into 8 multiplied by 8 small blocks which are not overlapped with each other; calculating a variance of each sub-block to represent the sub-block; this is used as a feature of the tread pattern. The shoe print searching algorithm based on the local area features mainly extracts a part of interested areas in the shoe print, and extracts the features of the interested areas. In 2015, wang et al proposed a shoe print search algorithm based on wavelet fourier melin features. The algorithm firstly divides the shoe print image into a forefoot part and a heel part, and respectively gives different weights to the two parts to extract the characteristics. The specific method comprises the following steps: firstly, respectively performing wavelet transformation on two parts of shoe print images; secondly, carrying out primary Fourier transform on the image after wavelet transform; and then performing polar coordinate transformation on the amplitude obtained after the primary Fourier transformation, and performing the primary Fourier transformation on the amplitude to obtain the frequency spectrum characteristic with the invariance of the rotation translation scale. The retrieval method based on the key point feature can also be called as the feature retrieval of the interesting point feature, and the key point is a feature in essence and represents a specific physical area or a spatial relation node on the image. ALMAADEED et al propose that multi-scale Harris and Hessian are subjected to key point detection to obtain scale invariance characteristics, the detected characteristics are subjected to SIFT (scale invariant feature transform) to obtain rotation invariance, and finally the characteristics extracted by the two detection methods are fused at a score level.
At present, shoe print search based on global features, shoe print search based on local features and shoe print search based on key point features have achieved certain results, but the existing method mainly considers the similarity between patterns, namely, shoe print images with similar patterns as the query image are arranged at the front of the search result, but the sequence of sizes among the same patterns in the search result is not considered. The size is used as an important attribute of the shoe print, and the returned results of the similar patterns are arranged in a descending order according to a size similar program of the query graph, so that the shoe print image with the common attribute with the search image can be more accurately found, and the case handling efficiency of public security personnel is improved.
Reference document:
Yan K,Lu L,Summers R M.Unsupervised Body Part Regression via Spatially Self-ordering Convolutional Neural Networks[J].2017.
Disclosure of Invention
According to the technical problem set forth above, a fine-grained shoe print image retrieval method is provided. The invention mainly utilizes a fine-grained shoe print image retrieval method, which is characterized by comprising the following steps:
S1: extracting shoe print attribute information;
s2: calculating the similarity of the shoe print attributes;
S3: calculating the similarity of the shoe print content;
S4: sorting score calculation combining the shoe print content information and the attribute information;
S5: and outputting the images in the dataset in descending order according to the sorting score to obtain a retrieval result.
Further, step S1 further includes the steps of:
s11: extracting attribute information from the single shoe print;
S111: shoe print image integrity judgment: obtaining an image area contained in a minimum circumscribed rectangle of a single shoe print image A, marking the image area as P A, binarizing P A, marking points on a pattern as 1, and marking background points as 0; calculating the duty ratio of the shoe print image A, and marking as O A, wherein O A is the ratio of the sum of the number of points marked as 1 in P A to the number of pixel points of the whole image; if O A is smaller than the threshold value z, the shoe print image is considered to have the missing of the sole or the heel;
s112: calculating the length, width and heel width of the shoe mark;
a: if the shoe print image is complete, taking the height of P A as the shoe length H A and the width as the shoe width W A; the shoe print image contained in P A is printed according to the following ratio of 3:2, dividing the shoe sole into a shoe sole and a shoe heel, solving the minimum circumscribing rectangle of the shoe heel part, and defining the width of the minimum circumscribing rectangle of the shoe heel part as the shoe heel width w A; the authenticity judgment is carried out on the shoe length, the shoe width and the heel width of the shoe print image, and the attribute information is updated;
b: if the shoe print image has a missing sole or heel, defining a shoe length H A =0, a shoe width W A =0 and a heel width W A =0;
S12: extracting attribute information from the shoe prints in the data set g= { G 1,...,Gk,...,GN }, k=1..n according to a single shoe print attribute information extraction method, and forming a shoe print attribute information matrix M:
further, step S112 further includes the steps of:
S1121: let r 1＝WA/wA, if r 1 is less than 1.28, consider that both sides of the sole are incomplete, update the width of the shoe to W A＝1.28×wA; if r 1 is more than 1.33, considering the two sides of the heel as incomplete, and updating the heel width w A＝WA/1.33;
s1122: let r 2＝HA/WA, if r 2 is less than 2.9, then consider the tip or heel to be incomplete, update the shoe length H A＝3.0×wA; if r 2 > 3.1, it is considered that the updated shoe width W A in step S1121 is still smaller than the actual shoe width, the shoe width W A＝HA/3.0 is updated again, and whether r 1 > 1.33 is judged, if yes, the heel width W A＝WA/1.33 is updated.
Further, step S2 further includes the steps of:
S21: extracting attribute information of the query image Q according to a single shoe print attribute information extraction method, respectively marking the length of the shoe as H Q, the width of the shoe as W Q and the width of the heel as W Q;
s22: calculating attribute information similarity scores S p (k) between the query image Q and the library map G k;
a: if there is a shoe length, shoe width, and heel width of the shoe print between the query map Q and the library map G k being 0, the attribute information similarity score S p (k) =0 of the query map Q and the library map G k;
b: if the shoe length, shoe width and heel width of the shoe marks in the query graph Q and the library graph G k are all not 0, the following calculation is performed:
(1) Calculating a difference dis i (k) between each attribute of the query image Q and the library map G k, i=1, 2,3;
(2) Calculating similarity scores s i (k) between the query image Q and each attribute of the library graph G k, i=1, 2,3;
wherein Δd 1,Δd2,Δd3 represents the difference of the shoe length, the shoe width and the heel width when the sizes differ by one size;
(3) Calculating an overall attribute similarity score S P (k) of the query image Q and the library map G k;
If it is The overall attribute similarity score of the query image Q and the library map G k is thenWherein/>The weight of the similarity scores of the three attribute information is calculated; α i is constant, α i =1 when dis i (k) > 0, otherwise α i＝0;si (k) is three attribute information similarities;
If it is The overall attribute similarity score of the query image Q and the library map G k is thenWherein/>The weight of the similarity scores of the three attribute information is calculated; s i (k) is three attribute information similarity.
Further, step S3 further includes the steps of:
S31: constructing a semantic block sample matrix;
S32: constructing a spatial layout relation of the semantic blocks in the query graph;
s33: constructing a spatial layout relation of the semantic blocks in the image set g= { G 1,...,Gk,...,GN }, k=1.
S34: calculating a spatial layout similarity score S D (k) and a pattern similarity score S F (k) of the query graph Q and the library graph G k;
S35: a content similarity score S c(k)Sc(k)＝η×SD(k)+(1-η)×SF (k) is calculated between the query graph Q and the library graph G k, where η is greater than or equal to 0.5.
Further, step S31 further includes the steps of:
S311: defining the minimum circumscribed rectangle of the query image Q as a semantic block P 1 and obtaining the height thereof Width/>And the y coordinate/>, in Q, of its upper left cornerAnd the y coordinate/>, of the lower right corner in Q
S312: trisecting the semantic blocks P 1 in the vertical direction, taking the uppermost and lowermost parts to obtain semantic blocks P 2,P5, and obtaining the height of each semantic blockWidth/>Y coordinate/>, of its upper left corner in QAnd the y coordinate of the lower right corner in QWherein i=2, 5;
S313: halving the semantic blocks P 2、P5 in the horizontal direction to obtain semantic blocks P 3,P4 and P 6,P7, and obtaining the height of each semantic block Width/>Y coordinate/>, of its upper left corner in QAnd the y coordinate/>, of the lower right corner in QWherein i=3, 4,6,7;
S314: horizontally overturning the semantic block P i selected in the steps S311-S313 to obtain an overturned semantic block T i, wherein i=1, & gt, and 7, and constructing a semantic block sample matrix
S315: binarizing the image content in each semantic block, marking the point on the pattern as 1, and marking the background point as 0; the duty cycle of each semantic block was calculated and noted as D i, i=1,..7, where D i is the ratio of the sum of the number of pixels marked 1 in the ith semantic block to its area.
Further, the step S32 further includes the steps of:
S321: calculate the vertical distance V Q(Pi,Pi+3 between the semantic blocks in Q), i=2, 3,4, where V Q(Pi,Pi+3) is the difference between the y-coordinate values of the upper left corner of semantic block P i+3 and semantic block P i, i.e.
S322: updating y coordinate values of the upper left corner and the lower right corner of the semantic block in Q, namelyWherein lambda is E [0, H Q), and/>
S323: calculating the occupancy ratio R i,j of semantic blocks in the query image Q, i=1,.. 7,j =1, 2; wherein R i,1 is the y-coordinate value of the upper left corner of the ith semantic blockThe ratio to the height of the query image Q, i.e./>R i,2 is the y coordinate value/>, of the lower right corner of the ith semantic blockThe ratio to the height of the query image Q, i.e./>H Q is the high of query image Q.
Further, step S33 further includes the steps of:
S331: according to the duty ratio R i,j of the ith semantic block in the query image Q, primarily determining the y coordinate value B k,i of the upper left corner in G k and the y coordinate value N k,i of the lower right corner in G k; wherein the method comprises the steps of Height for database image G k;
S332: according to the coordinates of the left upper corner and the right lower corner which are preliminarily determined by the ith semantic block in the database image G k, a corresponding matching area J k,i is intercepted in the database image G k;
S333: similarity of E (1,1),E(2,1) and J k,1 was calculated separately If/>Q and G k are symmetrical about the y-axis, and semantic blocks E (2,i) and G k are selected for matching and similarity calculation; on the contrary, the semantic blocks E (1,i) and G k are used for matching and similarity calculation, wherein i=2,..7;
s334: traversing the semantic block selected in step S333, performing the following operations:
a: carrying out normalized cross-correlation operation on the semantic blocks E (j,i) and the corresponding matching areas J k,i intercepted by the semantic blocks E (j,i) in G k respectively to obtain a response diagram M k,i corresponding to each semantic block, wherein j=1, 2; i=2.. 7, preparing a base material;
b: taking the y coordinate of the maximum value point in F k,i as the y coordinate C k,i of the central point of the semantic block E (j,i) in J k,i, recording the maximum value in F k,i as beta i, and taking the maximum value as the pattern similarity score of the ith semantic block and the library graph G k;
calculating the y coordinate u k,i of the upper left corner of the semantic block E (j,i) in J k,i, wherein H i is the height of the ith semantic block;
s335: calculating the y-coordinate U k,i of the upper left corner of the semantic block E (j,i) in G k, where U k,i＝uk,i+Uk,i, i=2,..7;
S336: calculate the vertical distance V k(E(j,i),E(j,i+3) between the semantic blocks in G k), where V k(E(j,i),E(j,i+3)) is the difference between the y coordinate values of the upper left corner of semantic block E (j,i+3) and semantic block E (j,i) in G k, i.e. V k(E(j,i),E(j,i+3))＝Uk,i+3-Uk,i, i=2, 3,4.
Further, step S34 further includes the steps of:
S341: calculating the sum O i,i+3 of the duty ratios of the semantic blocks E (j,i+3) and E (j,i) in Q, i=3, 4;
S342: calculate the distance difference Dis k(Pi,Pi+3 of the semantic block distance in the library map G k and the query map Q), where Disk(Pi,Pi+3)＝|VQ(Pi,Pi+3)-Vk(E(j,i),E(j,i+3))|,i＝2,3,4;
S343: calculating a spatial layout similarity score S d(Pi,Pi+3 between the semantic blocks in the query graph Q and the library graph G k), wherein
S344: calculating a pattern similarity score S f(Pi,Pi+3 between the semantic blocks in the query graph Q and the library graph G k, wherein S f(Pi,Pi+3)＝βi+βi+3, i=2, 3,4;
S345: giving different weights gamma m with m=1, 2,3 to the similarity scores;
a: the semantic inter-block spatial layout similarity score S d(P2,P5) and the pattern similarity score S f(P2,P5), wherein γ 1 is a constant value, and γ 1 =0.5;
b: according to the sum O i,i+3 of the duty ratios of the ith and the (i+3) th semantic blocks as a spatial layout similarity score S d(Pi,Pi+3 between the semantic blocks, i=3, 4 and a pattern similarity score S f(Pi,Pi+3), i=3, 4 is endowed with different weights gamma m, m=2, 3; if O 3,6≥O4,7, then gamma 2＞γ3, otherwise gamma 2＜γ3; and γ 2+γ3 =0.5;
S346: a spatial layout similarity score S D is calculated:
SD(k)＝γ1×Sd(P2,P5)+γ2×Sd(P3,P6)+γ3×Sd(P4,P7);
s347: calculating a pattern similarity score S F:
SF(k)＝γ1×Sf(P2,P5)+γ2×Sf(P3,P6)+γ3×Sf(P4,P7).
Further, according to the similarity of the information of the shoe print attribute and the similarity of the content information calculated by the query graph Q and each of the data sets in the steps S2 and S3, a ranking score of the query graph and the graph in the data sets is calculated:
Where k is the identity of the shoe print image in the dataset.
Compared with the prior art, the invention has the following advantages:
The method not only utilizes the pattern similarity information to carry out shoe print retrieval, but also considers the space position information of the patterns to carry out shoe print retrieval, namely if the semantic block selected in the query graph is positioned in the half sole area, the position of the most similar semantic block matched in the library graph is the heel area, and the space similarity score corresponding to the most similar semantic block is smaller, so that the final similarity score is reduced, and the influence of the similar patterns on the retrieval result can be effectively weakened.
In the shoe print searching process, the method considers the size attribute information among the same patterns, and can enable police officers to more quickly and accurately determine the identity of the suspected person and improve the efficiency of handling cases by arranging the library patterns which are similar to the query patterns in the same pattern and have similar sizes in front of the returned results.
The method of the invention is convenient for public security personnel to effectively manage the shoe marks by measuring the size attribute information between the shoe marks, and provides a reasonable and effective arrangement mode for the public security personnel to establish a suspected shoe mark database.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to the drawings without inventive effort to a person skilled in the art.
FIG. 1 is a schematic overall flow chart of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
As shown in fig. 1, the invention provides a fine-grained shoe print image retrieval method, which is characterized by comprising the following steps:
step S1: and extracting shoe print attribute information. Further, the step S1 further includes the following steps:
step S11: extracting attribute information from the single shoe print;
S111: shoe print image integrity judgment: obtaining an image area contained in a minimum circumscribed rectangle of a single shoe print image A, marking the image area as P A, binarizing P A, marking points on a pattern as 1, and marking background points as 0; calculating the duty ratio of the shoe print image A, and marking as O A, wherein O A is the ratio of the sum of the number of points marked as 1 in P A to the number of pixel points of the whole image; if O A is smaller than the threshold value z, the shoe print image is considered to have the missing of the sole or the heel; step S112: calculating the length, width and heel width of the shoe mark;
a: if the shoe print image is complete, taking the height of P A as the shoe length H A and the width as the shoe width W A; the shoe print image contained in P A is printed according to the following ratio of 3:2, dividing the shoe sole into a shoe sole and a shoe heel, solving the minimum circumscribing rectangle of the shoe heel part, and defining the width of the minimum circumscribing rectangle of the shoe heel part as the shoe heel width w A; the authenticity judgment is carried out on the shoe length, the shoe width and the heel width of the shoe print image, and the attribute information is updated;
b: if the shoe print image has a missing sole or heel, defining a shoe length H A =0, a shoe width W A =0 and a heel width W A =0;
further, step S112 further includes the steps of:
Step S1121: let r 1＝WA/wA, if r 1 is less than 1.28, consider that both sides of the sole are incomplete, update the width of the shoe to W A＝1.28×wA; if r 1 is more than 1.33, considering the two sides of the heel as incomplete, and updating the heel width w A＝WA/1.33;
Step S1122: let r 2＝HA/WA, if r 2 is less than 2.9, then consider the tip or heel to be incomplete, update the shoe length H A＝3.0×wA; if r 2 > 3.1, it is considered that the updated shoe width W A in step S1121 is still smaller than the actual shoe width, the shoe width W A＝HA/3.0 is updated again, and whether r 1 > 1.33 is judged, if yes, the heel width W A＝WA/1.33 is updated. As a preferred embodiment, the complete sole includes a toe, the calculated length being smaller than the actual length if the toe of the sole is incomplete, and the calculated width being smaller than the actual width if both sides of the sole are incomplete.
Step S12: extracting attribute information from the shoe prints in the data set g= { G 1,...,Gk,...,GN }, k=1..n according to a single shoe print attribute information extraction method, and forming a shoe print attribute information matrix M:
Further, after extracting the shoe print attribute information, step S2 is performed: and calculating the similarity of the shoe print attributes.
Step S21: extracting attribute information of the query image Q according to a single shoe print attribute information extraction method, respectively marking the length of the shoe as H Q, the width of the shoe as W Q and the width of the heel as W Q;
Step S22: calculating attribute information similarity scores S p (k) between the query image Q and the library map G k;
a: if there is a shoe length, shoe width, and heel width of the shoe print between the query map Q and the library map G k being 0, the attribute information similarity score S p (k) =0 of the query map Q and the library map G k;
b: if the shoe length, shoe width and heel width of the shoe marks in the query graph Q and the library graph G k are all not 0, the following calculation is performed:
(1) Calculating a difference dis i (k) between each attribute of the query image Q and the library map G k, i=1, 2,3; here, the suspect shoe print database image library and the database are the same database.
(2) Calculating similarity scores s i (k) between the query image Q and each attribute of the library graph G k, i=1, 2,3; as a preferred embodiment, the present application is characterized by a long shoe length, a wide shoe width, and a wide heel.
Wherein Δd 1,Δd2,Δd3 represents the difference of the shoe length, the shoe width and the heel width when the sizes differ by one size;
(3) An overall attribute similarity score S P (k) is calculated for the query image Q and the library map G k. There are three types of attribute information for a shoe print image: the length, width and heel width of the shoe are respectively provided with a similarity score, but the weights of the three attribute similarity are different, so that the three attribute similarity distributions are combined together to calculate the overall attribute similarity score
If it isThe overall attribute similarity score of the query image Q and the library map G k is thenWherein/>The weight of the similarity scores of the three attribute information is calculated; α i is constant, α i =1 when dis i (k) > 0, otherwise α i＝0;si (k) is three attribute information similarities;
If it is The overall attribute similarity score of the query image Q and the library map G k is thenWherein/>The weight of the similarity scores of the three attribute information is calculated; s i (k) is three attribute information similarity.
As a preferred embodiment, the present application further includes step S3: and calculating the similarity of the shoe print content.
Step S31: a semantic block sample matrix is constructed.
In a preferred embodiment, in the present application, step S31 further includes the steps of:
step S311: defining the minimum circumscribed rectangle of the query image Q as a semantic block P 1 and obtaining the height thereof Width of (L)And the y coordinate/>, in Q, of its upper left cornerAnd the y coordinate/>, of the lower right corner in Q
Step S312: trisecting the semantic blocks P 1 in the vertical direction, taking the uppermost and lowermost parts to obtain semantic blocks P 2,P5, and obtaining the height of each semantic blockWidth/>Y coordinate/>, of its upper left corner in QAnd the y coordinate/>, of the lower right corner in QWherein i=2, 5;
Step S313: halving the semantic blocks P 2、P5 in the horizontal direction to obtain semantic blocks P 3,P4 and P 6,P7, and obtaining the height of each semantic block Width/>Y coordinate/>, of its upper left corner in QAnd the y coordinate of the lower right corner in QWherein i=3, 4,6,7;
Step S314: horizontally overturning the semantic block P i selected in the steps S311-S313 to obtain an overturned semantic block T i, wherein i=1, & gt, and 7, and constructing a semantic block sample matrix
Step S315: binarizing the image content in each semantic block, marking the point on the pattern as 1, and marking the background point as 0; the duty cycle of each semantic block was calculated and noted as D i, i=1,..7, where D i is the ratio of the sum of the number of pixels marked 1 in the ith semantic block to its area.
Step S32: the spatial layout relation of the semantic blocks in the query graph is constructed.
Step S321: calculate the vertical distance V Q(Pi,Pi+3 between the semantic blocks in Q), i=2, 3,4, where V Q(Pi,Pi+3) is the difference between the y-coordinate values of the upper left corner of semantic block P i+3 and semantic block P i, i.e.
Step S322: updating y coordinate values of the upper left corner and the lower right corner of the semantic block in Q, namelyWherein lambda is E [0, H Q), and/>
Step S323: calculating the occupancy ratio R i,j of semantic blocks in the query image Q, i=1,.. 7,j =1, 2; wherein R i,1 is the y-coordinate value of the upper left corner of the ith semantic blockThe ratio to the height of the query image Q, i.e./>R i,2 is the y coordinate value/>, of the lower right corner of the ith semantic blockThe ratio to the height of the query image Q, i.e./>H Q is the high of query image Q.
Step S33: building spatial layout relations of semantic blocks in the image set g= { G 1,...,Gk,...,GN }, k=1.
Step S331: according to the duty ratio R i,j of the ith semantic block in the query image Q, primarily determining the y coordinate value B k,i of the upper left corner in G k and the y coordinate value N k,i of the lower right corner in G k; wherein the method comprises the steps ofHeight for database image G k;
Step S332: according to the coordinates of the left upper corner and the right lower corner which are preliminarily determined by the ith semantic block in the database image G k, a corresponding matching area J k,i is intercepted in the database image G k;
Step S333: similarity of E (1,1),E(2,1) and J k,1 was calculated separately If/>Q and G k are symmetrical about the y-axis, and semantic blocks E (2,i) and G k are selected for matching and similarity calculation; on the contrary, the semantic blocks E (1,i) and G k are used for matching and similarity calculation, wherein i=2,..7;
Step S334: traversing the semantic block selected in step S333, performing the following operations:
a: carrying out normalized cross-correlation operation on the semantic blocks E (j,i) and the corresponding matching areas J k,i intercepted by the semantic blocks E (j,i) in G k respectively to obtain a response diagram M k,i corresponding to each semantic block, wherein j=1, 2; i=2.. 7, preparing a base material;
b: taking the y coordinate of the maximum value point in F k,i as the y coordinate C k,i of the central point of the semantic block E (j,i) in J k,i, recording the maximum value in F k,i as beta i, and taking the maximum value as the pattern similarity score of the ith semantic block and the library graph G k;
calculating the y coordinate u k,i of the upper left corner of the semantic block E (j,i) in J k,i, wherein H i is the height of the ith semantic block;
Step S335: calculating the y-coordinate U k,i of the upper left corner of the semantic block E (j,i) in G k, where U k,i＝uk,i+Uk,i, i=2,..7;
Step S336: calculate the vertical distance V k(E(j,i),E(j,i+3) between the semantic blocks in G k), where V k(E(j,i),E(j,i+3)) is the difference between the y coordinate values of the upper left corner of semantic block E (j,i+3) and semantic block E (j,i) in G k, i.e. V k(E(j,i),E(j,i+3))＝Uk,i+3-Uk,i, i=2, 3,4.
Step S34: calculating a spatial layout similarity score S D (k) and a pattern similarity score S F (k) of the query graph Q and the library graph G k
Step S35: a content similarity score S c(k)Sc(k)＝η×SD(k)+(1-η)×SF (k) is calculated between the query graph Q and the library graph G k, where η is greater than or equal to 0.5.
Calculating the ranking scores of the query graph and the data set graph according to the similarity of the information of the shoe print attribute and the similarity of the information of the content calculated by the query graph Q and each of the data sets in the steps S2 and S3:
Where k is the identity of the shoe print image in the dataset.
Step S4: and (5) calculating the sorting score by combining the shoe print content information and the attribute information.
Step S34 further includes the steps of:
S341: calculating the sum O i,i+3 of the duty ratios of the semantic blocks E (j,i+3) and E (j,i) in Q, i=3, 4;
S342: calculate the distance difference Dis k(Pi,Pi+3 of the semantic block distance in the library map G k and the query map Q), where Disk(Pi,Pi+3)＝|VQ(Pi,Pi+3)-Vk(E(j,i),E(j,i+3))|,i＝2,3,4;
S343: calculating a spatial layout similarity score S d(Pi,Pi+3 between the semantic blocks in the query graph Q and the library graph G k), wherein
S344: calculating a pattern similarity score S f(Pi,Pi+3 between the semantic blocks in the query graph Q and the library graph G k, wherein S f(Pi,Pi+3)＝βi+βi+3, i=2, 3,4;
S345: giving different weights gamma m with m=1, 2,3 to the similarity scores;
a: the semantic inter-block spatial layout similarity score S d(P2,P5) and the pattern similarity score S f(P2,P5), wherein γ 1 is a constant value, and γ 1 =0.5;
b: according to the sum O i,i+3 of the duty ratios of the ith and the (i+3) th semantic blocks as a spatial layout similarity score S d(Pi,Pi+3 between the semantic blocks, i=3, 4 and a pattern similarity score S f(Pi,Pi+3), i=3, 4 is endowed with different weights gamma m, m=2, 3; if O 3,6≥O4,7, then gamma 2＞γ3, otherwise gamma 2＜γ3; and γ 2+γ3 =0.5;
S346: a spatial layout similarity score S D is calculated:
SD(k)＝γ1×Sd(P2,P5)+γ2×Sd(P3,P6)+γ3×Sd(P4,P7);
s347: calculating a pattern similarity score S F:
SF(k)＝γ1×Sf(P2,P5)+γ2×Sf(P3,P6)+γ3×Sf(P4,P7).
step S5: and outputting the images in the dataset in descending order according to the sorting score to obtain a retrieval result.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments. In the foregoing embodiments of the present invention, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.

Claims (9)

1. A fine-grained shoe print image retrieval method, characterized by comprising the following steps:
S1: extracting shoe print attribute information; step S1 further comprises the steps of:
s11: extracting attribute information from the single shoe print;
S111: shoe print image integrity judgment: obtaining an image area contained in a minimum circumscribed rectangle of a single shoe print image A, marking the image area as P A, binarizing P A, marking points on a pattern as 1, and marking background points as 0; calculating the duty ratio of the shoe print image A, and marking as O A, wherein O A is the ratio of the sum of the number of points marked as 1 in P A to the number of pixel points of the whole image; if O A is smaller than the threshold value z, the shoe print image is considered to have the missing of the sole or the heel;
s112: calculating the length, width and heel width of the shoe mark;
a: if the shoe print image is complete, taking the height of P A as the shoe length H A and the width as the shoe width W A; the shoe print image contained in P A is printed according to the following ratio of 3:2, dividing the shoe sole into a shoe sole and a shoe heel, solving the minimum circumscribing rectangle of the shoe heel part, and defining the width of the minimum circumscribing rectangle of the shoe heel part as the shoe heel width w A; the authenticity judgment is carried out on the shoe length, the shoe width and the heel width of the shoe print image, and the attribute information is updated;
b: if the shoe print image has a missing sole or heel, defining a shoe length H A =0, a shoe width W A =0 and a heel width W A =0;
S12: according to the single shoe print attribute information extraction method, extracting attribute information from the shoe print in the data set g= { G 1,...,GkGN }, k=, 1, and forming a shoe print attribute information matrix M:
s2: calculating the similarity of the shoe print attributes;
S3: calculating the similarity of the shoe print content;
S4: sorting score calculation combining the shoe print content information and the attribute information;
S5: and outputting the images in the dataset in descending order according to the sorting score to obtain a retrieval result.
2. The fine-grained shoe print image retrieval method according to claim 1, characterized in that: step S112 further includes the steps of:
s1121: let r 1＝WA/wA, if r 1 is less than 1.28, consider that both sides of the sole are incomplete, update the shoe width to W A＝1.28×wA; if r 1 is more than 1.33, considering the two sides of the heel as incomplete, and updating the heel width w a＝WA/1.33;
S1122: let r 2＝HA/WA, if r 2 is less than 2.9, then consider the tip or heel to be incomplete, update the shoe length H A＝3.0×wA; if r 2 >3.1, it is considered that the updated shoe width W A of step S1121 is still smaller than the actual shoe width, the shoe width W A＝HA/3.0 is updated again, and it is judged whether r 1 >1.33, if yes, the heel width W A＝WA/1.33 is updated.
3. The fine-grained shoe print image retrieval method according to claim 1, characterized in that: step S2 further comprises the steps of:
S21: extracting attribute information of the query image Q according to a single shoe print attribute information extraction method, respectively marking the length of the shoe as H Q, the width of the shoe as W Q and the width of the heel as W Q;
s22: calculating attribute information similarity scores S p (k) between the query image Q and the library map G k;
a: if there is a shoe length, shoe width, and heel width of the shoe print between the query map Q and the library map G k being 0, the attribute information similarity score S p (k) =0 of the query map Q and the library map G k;
b: if the shoe length, shoe width and heel width of the shoe marks in the query graph Q and the library graph G k are all not 0, the following calculation is performed:
(1) Calculating a difference dis i (k) between each attribute of the query image Q and the library map G k, i=1, 2,3;
(2) Calculating similarity scores s i (k) between the query image Q and each attribute of the library graph G k, i=1, 2,3;
Wherein Δd 1,Δd2,Δd3 represents the difference of the shoe length, the shoe width and the heel width when the sizes differ by one size;
(3) Calculating an overall attribute similarity score S P (k) of the query image Q and the library map G k;
If it is The overall attribute similarity score of the query image Q and the library map G k is/>Wherein/> The weight of the similarity scores of the three attribute information is calculated; α i is constant, α i =1 when dis i (k) >0, otherwise α i＝0;si (k) is three attribute information similarities;
If it is The overall attribute similarity score of the query image Q and the library map G k is/>Wherein/> The weight of the similarity scores of the three attribute information is calculated; s i (k) is three attribute information similarity.
4. The fine-grained shoe print image retrieval method according to claim 1, characterized in that: step S3 further comprises the steps of:
S31: constructing a semantic block sample matrix;
S32: constructing a spatial layout relation of the semantic blocks in the query graph;
s33: constructing a spatial layout relation of the semantic blocks in the image set g= { G 1,...,Gk,...,GN }, k=1.
S34: calculating a spatial layout similarity score S D (k) and a pattern similarity score S F (k) of the query graph Q and the library graph G k;
S35: a content similarity score S c(k)Sc(k)＝η×SD(k)+(1-η)×SF (k) is calculated between the query graph Q and the library graph G k, where η is greater than or equal to 0.5.
5. The fine-grained shoe print image retrieval method according to claim 4, wherein: step S31 further includes the steps of:
S311: defining the minimum circumscribed rectangle of the query image Q as a semantic block P 1 and obtaining the height thereof Width/>And the y coordinate/>, in Q, of its upper left cornerAnd the y coordinate/>, of the lower right corner in Q
S312: trisecting the semantic blocks P 1 in the vertical direction, taking the uppermost and lowermost parts to obtain semantic blocks P 2,P5, and obtaining the height of each semantic blockWidth/>Y coordinate/>, of its upper left corner in QAnd the y coordinate of the lower right corner in QWherein i=2, 5;
S313: halving the semantic blocks P 2、P5 in the horizontal direction to obtain semantic blocks P 3,P4 and P 6,P7, and obtaining the height of each semantic block Width/>Y coordinate/>, of its upper left corner in QAnd the y coordinate/>, of the lower right corner in QWherein i=3, 4,6,7;
S314: horizontally overturning the semantic block P i selected in the steps S311-S313 to obtain an overturned semantic block T i, wherein i=1, & gt, and 7, and constructing a semantic block sample matrix
S315: binarizing the image content in each semantic block, marking the point on the pattern as 1, and marking the background point as 0; the duty cycle of each semantic block was calculated and noted as D i, i=1,..7, where D i is the ratio of the sum of the number of pixels marked 1 in the ith semantic block to its area.
6. The fine-grained shoe print image retrieval method according to claim 4, wherein: the step S32 further has the steps of:
S321: calculate the vertical distance V Q(Pi,Pi+3 between the semantic blocks in Q), i=2, 3,4, where V Q(Pi,Pi+3) is the difference between the y-coordinate values of the upper left corner of semantic block P i+3 and semantic block P i, i.e.
S322: updating y coordinate values of the upper left corner and the lower right corner of the semantic block in Q, namely Wherein lambda is E [0, H Q), and/>
S323: calculating the occupancy ratio R i,j of semantic blocks in the query image Q, i=1,.. 7,j =1, 2; wherein R i,1 is the y-coordinate value of the upper left corner of the ith semantic blockThe ratio to the height of the query image Q, i.e./> R i,2 is the y coordinate value/>, of the lower right corner of the ith semantic blockThe ratio to the height of the query image Q, i.e./>H Q is the high of query image Q.
7. The fine-grained shoe print image retrieval method according to claim 4, wherein: step S33 further includes the steps of:
S331: according to the duty ratio R i,j of the ith semantic block in the query image Q, preliminarily determining the y coordinate value of the upper left corner in G k And the y-coordinate value/>, in G k, of its lower right corner pointWherein/> Height for database image G k;
S332: according to the coordinates of the left upper corner and the right lower corner which are preliminarily determined by the ith semantic block in the database image G k, a corresponding matching area J k,i is intercepted in the database image G k;
S333: similarity of E (1,1),E(2,1) and J k,1 was calculated separately If/>Q and G k are symmetrical about the y-axis, and semantic blocks E (2,i) and G k are selected for matching and similarity calculation; on the contrary, the semantic blocks E (1,i) and G k are used for matching and similarity calculation, wherein i=2,..7;
s334: traversing the semantic block selected in step S333, performing the following operations:
a: carrying out normalized cross-correlation operation on the semantic blocks E (j,i) and the corresponding matching areas J k,i intercepted by the semantic blocks E (j,i) in G k respectively to obtain a response diagram F k,i corresponding to each semantic block, wherein j=1, 2; i=2.. 7, preparing a base material;
b: taking the y coordinate of the maximum value point in F k,i as the y coordinate C k,i of the central point of the semantic block E (j,i) in J k,i, recording the maximum value in F k,i as beta i, and taking the maximum value as the pattern similarity score of the ith semantic block and the library graph G k;
calculating the y coordinate u k,i of the upper left corner of the semantic block E (j,i) in J k,i, wherein H i is the height of the ith semantic block;
S335: calculating the y coordinate of the upper left corner of the semantic block E (j,i) in G k Wherein/>
S336: calculate the vertical distance between semantic blocks V k(E(j,i),E(j,i+3) in G k, where V k(E(j,i),E(j,i+3)) is the difference between the y-coordinate values of the upper left corner of semantic block E (j,i+3) and semantic block E (j,i) in G k, i.e.
8. The fine-grained shoe print image retrieval method according to claim 4, wherein: step S34 further includes the steps of:
S341: calculating the sum O i,i+3 of the duty ratios of the semantic blocks E (j,i+3) and E (j,i) in Q, i=3, 4;
S342: calculate the distance difference Dis k(Pi,Pi+3 of the semantic block distance in the library map G k and the query map Q), where Disk(Pi,Pi+3)＝|VQ(Pi,Pi+3)-Vk(E(j,i),E(j,i+3))|,i＝2,3,4;
S343: calculating a spatial layout similarity score S d(Pi,Pi+3 between the semantic blocks in the query graph Q and the library graph G k), wherein
S344: calculating a pattern similarity score S f(Pi,Pi+3 between the semantic blocks in the query graph Q and the library graph G k, wherein S f(Pi,Pi+3)＝βi+βi+3, i=2, 3,4;
S345: giving different weights gamma m with m=1, 2,3 to the similarity scores;
a: the semantic inter-block spatial layout similarity score S d(P2,P5) and the pattern similarity score S f(P2,P5), wherein γ 1 is a constant value, and γ 1 =0.5;
b: according to the sum O i,i+3 of the duty ratios of the ith and the (i+3) th semantic blocks as a spatial layout similarity score S d(Pi,Pi+3 between the semantic blocks, i=3, 4 and a pattern similarity score S f(Pi,Pi+3), i=3, 4 is endowed with different weights gamma m, m=2, 3; if O 3,6≥O4,7, then gamma 2>γ3, otherwise gamma 2<γ3; and γ 2+γ3 =0.5;
S346: a spatial layout similarity score S D is calculated:
SD(k)＝γ1×Sd(P2,P5)+γ2×Sd(P3,P6)+γ3×Sd(P4,P7);
s347: calculating a pattern similarity score S F:
SF(k)＝γ1×Sf(P2,P5)+γ2×Sf(P3,P6)+γ3×Sf(P4,P7).
9. the fine-grained shoe print image retrieval method according to claim 1, characterized in that:
Calculating the ranking scores of the query graph and the data set graph according to the similarity of the information of the shoe print attribute and the similarity of the information of the content calculated by the query graph Q and each of the data sets in the steps S2 and S3:
Where k is the identity of the shoe print image in the dataset.
CN202110152570.2A 2021-02-03 2021-02-03 Fine-granularity shoe print image retrieval method Active CN112800267B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110152570.2A CN112800267B (en) 2021-02-03 2021-02-03 Fine-granularity shoe print image retrieval method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110152570.2A CN112800267B (en) 2021-02-03 2021-02-03 Fine-granularity shoe print image retrieval method

Publications (2)

Publication Number Publication Date
CN112800267A CN112800267A (en) 2021-05-14
CN112800267B true CN112800267B (en) 2024-06-11

Family

ID=75814048

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110152570.2A Active CN112800267B (en) 2021-02-03 2021-02-03 Fine-granularity shoe print image retrieval method

Country Status (1)

Country Link
CN (1) CN112800267B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115795081B (en) * 2023-01-20 2023-04-18 安徽大学 Cross-domain incomplete footprint image retrieval system based on multi-channel fusion

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20170043308A (en) * 2015-10-13 2017-04-21 충북대학교 산학협력단 Method for identificating Person on the basis gait data
CN106776950A (en) * 2016-12-02 2017-05-31 大连海事大学 A kind of field shoe impression mark decorative pattern image search method based on expertise guiding
CN106951906A (en) * 2017-03-22 2017-07-14 重庆市公安局刑事警察总队 The comprehensive analysis method that shoe sole print various dimensions are classified with identification
CN106970968A (en) * 2017-03-22 2017-07-21 重庆市公安局刑事警察总队 The classification of shoe sole print various dimensions looks into a yard method with identification
KR20170095062A (en) * 2016-02-12 2017-08-22 대한민국(관리부서: 행정자치부 국립과학수사연구원장) A Method Of Providing For Searching Footprint And The System Practiced The Method
WO2017168125A1 (en) * 2016-03-31 2017-10-05 Queen Mary University Of London Sketch based search methods
KR101833986B1 (en) * 2016-10-05 2018-03-02 김현철 Shoes component with image film and manufacturing method thereof
US10043109B1 (en) * 2017-01-23 2018-08-07 A9.Com, Inc. Attribute similarity-based search
CN109191481A (en) * 2018-08-03 2019-01-11 辽宁师范大学 Shoes watermark image Fast Threshold dividing method under bad conditions of exposure
CN110188222A (en) * 2019-06-03 2019-08-30 大连海事大学 Shoes based on the semantic filter in part and bridge joint similarity print search method

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20170043308A (en) * 2015-10-13 2017-04-21 충북대학교 산학협력단 Method for identificating Person on the basis gait data
KR20170095062A (en) * 2016-02-12 2017-08-22 대한민국(관리부서: 행정자치부 국립과학수사연구원장) A Method Of Providing For Searching Footprint And The System Practiced The Method
WO2017168125A1 (en) * 2016-03-31 2017-10-05 Queen Mary University Of London Sketch based search methods
KR101833986B1 (en) * 2016-10-05 2018-03-02 김현철 Shoes component with image film and manufacturing method thereof
CN106776950A (en) * 2016-12-02 2017-05-31 大连海事大学 A kind of field shoe impression mark decorative pattern image search method based on expertise guiding
US10043109B1 (en) * 2017-01-23 2018-08-07 A9.Com, Inc. Attribute similarity-based search
CN106951906A (en) * 2017-03-22 2017-07-14 重庆市公安局刑事警察总队 The comprehensive analysis method that shoe sole print various dimensions are classified with identification
CN106970968A (en) * 2017-03-22 2017-07-21 重庆市公安局刑事警察总队 The classification of shoe sole print various dimensions looks into a yard method with identification
CN109191481A (en) * 2018-08-03 2019-01-11 辽宁师范大学 Shoes watermark image Fast Threshold dividing method under bad conditions of exposure
CN110188222A (en) * 2019-06-03 2019-08-30 大连海事大学 Shoes based on the semantic filter in part and bridge joint similarity print search method

Also Published As

Publication number Publication date
CN112800267A (en) 2021-05-14

Similar Documents

Publication Publication Date Title
Pun et al. A two-stage localization for copy-move forgery detection
Liu et al. Automatic grape bunch detection in vineyards with an SVM classifier
Ouyang et al. Copy-move forgery detection based on deep learning
Galvez-Lopez et al. Real-time loop detection with bags of binary words
Tao et al. Unsupervised detection of built-up areas from multiple high-resolution remote sensing images
Jung Detecting building changes from multitemporal aerial stereopairs
CN109583305A (en) A kind of advanced method that the vehicle based on critical component identification and fine grit classification identifies again
CN1322471C (en) Comparing patterns
CN104850822B (en) Leaf identification method under simple background based on multi-feature fusion
CN104077605A (en) Pedestrian search and recognition method based on color topological structure
CN104966081B (en) Spine image-recognizing method
CN104751475B (en) A kind of characteristic point Optimum Matching method towards still image Object identifying
CN109271932A (en) Pedestrian based on color-match recognition methods again
Almaadeed et al. Partial shoeprint retrieval using multiple point-of-interest detectors and SIFT descriptors
KR101386513B1 (en) Method for recognizing human action using weighted bag of features histogram based on spatial pyramid and random forest classifier
CN106874421A (en) Image search method based on self adaptation rectangular window
CN112800267B (en) Fine-granularity shoe print image retrieval method
Romero et al. Enhanced local binary covariance matrices (ELBCM) for texture analysis and object tracking
US20100209008A1 (en) method and a system for organizing an image database
Pavlou et al. Automatic extraction and classification of footwear patterns
Li et al. Research of shoeprint image stream retrival algorithm with scale-invariance feature transform
Bakić et al. Inria IMEDIA2's participation at ImageCLEF 2012 plant identification task
He et al. Discovering visual element evolutions for historical document dating
Wei et al. The use of scale‐invariance feature transform approach to recognize and retrieve incomplete shoeprints
Li et al. A combined feature representation of deep feature and hand-crafted features for person re-identification

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant