CN114612985A - Portrait query method, system, device and storage medium - Google Patents
Portrait query method, system, device and storage medium Download PDFInfo
- Publication number
- CN114612985A CN114612985A CN202210264194.0A CN202210264194A CN114612985A CN 114612985 A CN114612985 A CN 114612985A CN 202210264194 A CN202210264194 A CN 202210264194A CN 114612985 A CN114612985 A CN 114612985A
- Authority
- CN
- China
- Prior art keywords
- retrieval
- portrait
- database
- person
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000003860 storage Methods 0.000 title claims abstract description 12
- 230000002708 enhancing effect Effects 0.000 claims abstract description 8
- 238000001514 detection method Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 claims description 4
- 230000009467 reduction Effects 0.000 claims description 3
- 230000000392 somatic effect Effects 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims description 2
- 238000012163 sequencing technique Methods 0.000 claims description 2
- 239000000126 substance Substances 0.000 claims description 2
- 230000001965 increasing effect Effects 0.000 abstract description 6
- 230000008859 change Effects 0.000 description 5
- 238000013461 design Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000009826 distribution Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000007717 exclusion Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000001174 ascending effect Effects 0.000 description 1
- -1 carrier Substances 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000000306 component Substances 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004615 ingredient Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/73—Querying
- G06F16/732—Query formulation
- G06F16/7335—Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/73—Querying
- G06F16/735—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/73—Querying
- G06F16/738—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Mathematical Physics (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a portrait query method, a system, equipment and a storage medium, wherein a new query framework is designed, face information is further mined, constraints among people query results are increased, the influence of people reloading is reduced, and the accuracy of portrait query is improved; specifically, the method comprises the following steps: 1) during face retrieval, repeated iterative retrieval is carried out by using the faces with high confidence coefficient in the retrieved database, face retrieval performance is further mined, a new query set is obtained, and the number of retrieved reliable samples is increased while accuracy is guaranteed. 2) By increasing the constraint among the character query results, the database to be retrieved is reduced, and the retrieval performance is further improved. 3) By enhancing the body appearance characteristics by using KNN characteristic expansion, the retrieval result is improved, the Top-lambda similarity is designed, and the influence of different dresses on the characteristic similarity is eliminated as much as possible.
Description
Technical Field
The invention relates to the technical field of portrait query, in particular to a portrait query method, a system, equipment and a storage medium.
Background
The portrait query refers to a person being retrieved from a database based on only one portrait of the person that contains only face information, wherein the faces of the person's pictures in the database may not be visible. At present, the research on human retrieval in academic circles and industrial circles is mainly face recognition and pedestrian re-recognition, and these two problems are usually considered separately, however, in a higher level, both of these two tasks are actually completing the same thing, namely, confirming the Identity (ID) of a person, and only the information utilized is different, one utilizes the face and one utilizes the body information. The method limits the available information on the face or body information, which greatly limits the generalization of the algorithm, and the portrait query is just an application which needs to utilize the face and body information at the same time and has more application prospect.
In many cases, only one portrait picture of a certain person can be obtained, for example, the person wants to find out the trail of the person in the monitoring of the whole city according to one portrait or portrait of the person; for another example, the film and television industry wants to directly search all the pictures of a certain actor appearing in a movie. In these scenarios, the front face of a person is sometimes not visible in a database to be searched (such as a surveillance video or a movie picture), and the dress, the environment, and even the age of the same person may change. At the moment, the method is obviously insufficient for searching by only dressing a human face or a body, how to design a set of effective portrait query system, and simultaneously, the human face and the body information are utilized, so that the portrait query system is more suitable for a real application scene, and meanwhile, the method has greater challenge.
The portrait query system mainly comprises a face retrieval part and a body retrieval part. The face retrieval needs to find images of the same person containing faces as much as possible in a database on the premise of ensuring the accuracy of the portrait picture, and the body retrieval uses the body dressing and other characteristics of the samples to perform secondary retrieval, so as to recall more picture results of the person containing faces or not containing faces. The existing method generally focuses on designing a better Convolutional Neural Network (CNN) for extracting more robust human face and body features, and neglects the design of the overall retrieval process. In fact, a good search frame design can often achieve the effect of getting twice the result with half the effort, and on the basis of using the same characteristics, better performance is obtained.
Based on the above introduction, the prior art mainly has the following technical problems:
1) in the face retrieval part, the existing method only uses a query sample (face portrait to be queried) to directly query in a database, however, the face data distribution of the query sample and the face data distribution of images in the database are different, which causes performance degradation. Face retrieval is the basis of the whole portrait query, and if the potential of face recognition cannot be fully utilized, the performance of the whole system is reduced.
2) The existing method directly searches from the whole database, and the search results of different query samples are not restricted, and the mutual exclusivity of the character identities is not considered, namely, the same character does not belong to two identities at the same time, so that the search system can not optimize the search results of a single query sample from the global information, and the search results have certain limitations.
3) In the body retrieval part, the existing method does not make special design for appearance interference caused by person changing, and directly uses body dresses of all results obtained by the first-step face retrieval to carry out secondary retrieval. However, the change of the dress of the person greatly affects the appearance, and the use of the dress without distinction for the body retrieval will greatly limit the performance of the secondary retrieval.
Disclosure of Invention
The invention aims to provide a portrait query method, a system, equipment and a storage medium, which can improve the retrieval efficiency and the accuracy of portrait query.
The purpose of the invention is realized by the following technical scheme:
a portrait query method, comprising:
respectively extracting face features of all the person portraits to be inquired in the video to be inquired, and respectively extracting the face features and body features of the images in the database;
performing iterative retrieval on the current portrait of the person to be queried in the video to be queried by using the face features to obtain a face matching result; when the current retrieval is carried out, retrieving images with the average human face feature similarity of all images in the retrieval set exceeding a threshold value in the database by using the query set obtained in the last retrieval, combining the obtained retrieval result of the current retrieval and the query set obtained in the last retrieval to be used as the query set obtained at the current time, and determining the human face matching result of the portrait of the person to be queried through the query set obtained at the last time; when the search is carried out for the first time, the query set only contains the portrait of the current person to be queried in the video to be queried, and the threshold value is gradually reduced when the search is carried out for each time;
reducing the database by using the face matching result of the person portrait which is not to be inquired currently and the images of all other persons which are in the same frame with all the images in the face matching result of the person portrait to be inquired currently to obtain a reduced database;
and respectively enhancing body characteristics of the images in the reduced database and the face matching result, taking the face matching result as a query database, and retrieving the reduced database by using the similarity of the enhanced body characteristics to obtain the retrieval result of the person portrait to be queried currently.
A portrait query system, comprising:
the characteristic extraction unit is used for respectively extracting human face characteristics of all the person portraits to be inquired in the video to be inquired and respectively extracting the human face characteristics and the body characteristics of the images in the database;
the face iterative retrieval and face matching result acquisition unit is used for carrying out iterative retrieval on the current person portrait to be queried in the video to be queried by using the face characteristics to obtain a face matching result; when the current retrieval is carried out, retrieving images with the average human face feature similarity of all images in the retrieval set exceeding a threshold value in the database by using the query set obtained in the last retrieval, combining the obtained retrieval result of the current retrieval and the query set obtained in the last retrieval to be used as the query set obtained at the current time, and determining the human face matching result of the portrait of the person to be queried through the query set obtained at the last time; when the search is carried out for the first time, the query set only contains the portrait of the current person to be queried in the video to be queried, and the threshold value is gradually reduced when the search is carried out for each time;
the database reduction unit is used for reducing the database by using the face matching result of the person portrait which is not to be inquired currently and the images of all other persons of which all the images are in the same frame in the face matching result of the person portrait to be inquired currently to obtain a reduced database;
and the body retrieval and joint retrieval result generation unit is used for respectively enhancing body characteristics of the images in the reduced database and the face matching result, taking the face matching result as a query database, and retrieving the reduced database by using the similarity of the enhanced appearance characteristics to obtain the retrieval result of the portrait of the person to be queried currently.
A processing device, comprising: one or more processors; a memory for storing one or more programs;
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the aforementioned methods.
A readable storage medium, storing a computer program which, when executed by a processor, implements the aforementioned method.
According to the technical scheme provided by the invention, a new query framework is designed, the face information is further mined, the constraint among the figure query results is increased, the influence of figure change is reduced, and the accuracy of portrait query is improved; specifically, the method comprises the following steps: 1) when the face is searched, the high-confidence face in the searched database is used for carrying out repeated iterative search, the face searching performance is further mined, a new query set is obtained, and the number of the searched reliable samples is increased while the accuracy is ensured. 2) By increasing the constraint among the character query results, the base library to be retrieved is reduced, and the retrieval performance is further improved. 3) By enhancing the appearance characteristics of the body, the influence of different dresses on the similarity of the characteristics is eliminated as much as possible.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
Fig. 1 is a flowchart of a portrait query method according to an embodiment of the present invention;
fig. 2 is a schematic diagram illustrating calculation of Top- λ similarity between two different database images and a query library according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a portrait query system according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a processing apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
The terms that may be used herein are first described as follows:
the terms "comprising," "including," "containing," "having," or other similar terms of meaning should be construed as non-exclusive inclusions. For example: including a feature (e.g., material, component, ingredient, carrier, formulation, material, dimension, part, component, mechanism, device, process, procedure, method, reaction condition, processing condition, parameter, algorithm, signal, data, product, or article of manufacture), is to be construed as including not only the particular feature explicitly listed but also other features not explicitly listed as such which are known in the art.
The following describes a portrait query method, system, device and storage medium provided by the present invention in detail. Details which are not described in detail in the embodiments of the invention belong to the prior art which is known to the person skilled in the art. Those not specifically mentioned in the examples of the present invention were carried out according to the conventional conditions in the art or conditions suggested by the manufacturer.
Example one
In order to solve the three technical problems in the prior art, the embodiment of the invention provides a portrait query method, which adopts a more effective query flow frame, fully excavates face information, and enables results of different portrait of people to be queried to be mutually constrained and further optimized. Meanwhile, during body retrieval, special design is made for the figure change problem, the influence of figure change on body retrieval is reduced as much as possible, and a better and more robust portrait query result is obtained. As shown in fig. 1, a flowchart of a portrait query method provided in an embodiment of the present invention mainly includes the following steps:
In the embodiment of the invention, for a video to be retrieved, the video is assumed to contain M person portraits to be inquired
Let the database include N images, note
Wherein q isiRepresenting the i-th person's portrait to be queried, glRepresenting the ith image in the database.
In the embodiment of the invention, the face detection is respectively carried out on M person portraits to be inquired through a face detection network, and corresponding face features are extracted from all face detection results through a face feature extraction network; considering that all the people to be queried are queried from the database, all the images in the database are also processed respectively through the same network to obtain the face features. In addition, body detection is carried out on all images in the database through a body detection network, and corresponding body characteristics are extracted from all body detection results through a body characteristic extraction network. These facial features and body features are stored for use in subsequent processes. The four networks are Convolutional Neural Networks (CNN).
Step 2, carrying out iterative retrieval on the current portrait of the person to be queried in the video to be queried by using the face characteristics to obtain a face matching result; when the current retrieval is carried out, retrieving images with the average human face feature similarity of all images in the retrieval set exceeding a threshold value in the database by using the query set obtained in the last retrieval, combining the obtained retrieval result of the current retrieval and the query set obtained in the last retrieval to be used as the query set obtained at the current time, and determining the human face matching result of the portrait of the person to be queried through the query set obtained at the last time; and when the search is carried out for the first time, the query set only contains the portrait of the current person to be queried in the video to be queried, and the threshold value is gradually reduced when the search is carried out for each time.
In the embodiment of the invention, the iterative retrieval frequency is set as T, the threshold value of each retrieval is gradually reduced, and the threshold value in the iterative retrieval is represented as follows: th (h)1≥th2≥…≥thTWhere th denotes a threshold, T ═ 1,2, …, T.
The image q of the ith person to be inquirediAs the portrait of the person currently to be queried. At first retrieval, query set
Searching the database for the person portrait q to be inquirediThe similarity of the face features exceeds a threshold th1Is marked as { g ] as the first search result1,g2,…,gnN represents the number of the images retrieved for the first time, each g represents the image retrieved from the database, and the retrieval result for the first time and the i-th person portrait to be queried q are comparediMerging to obtain a query set
At the time of second retrieval, a query set obtained from the first retrieval is retrieved from the database
The average face feature similarity of all the images exceeds the threshold value th2As a second retrieval result; combining the second search result with the query set
Merging to obtain a query set
By analogy, the search is carried out for T times in an iteration mode, and the obtained query set is utilized
Determining face matching result Mi：
Taking the current retrieval as the t-th retrieval as an example, the last retrieval is obtainedQuery set notation
And calculating the image with the average human face feature similarity of the images in the database and all the images in the query set exceeding a threshold value according to the following formula:
wherein the content of the first and second substances,
representing a set of queries obtained from a previous search
Q represents the set of queries obtained from the last search
Of (e.g., when t is 2, { q }i,g1,g2,…,gnEach image in the } is calculated as q-substituted as above), gjRepresenting the jth image in the database; s (-) is a face feature similarity calculation function,
representing the jth image g in the database calculated at the time of the tth searchjQuery set obtained from last retrieval
Average face feature similarity of all images in the image.
Passing threshold thtThe screened t-th retrieval result is combined with the query set obtained by the last retrieval
Merging sets of queries obtained as the t-th time
In the embodiment of the invention, the retrieval result obtained by each iterative retrieval is an image from a database, and the images are arranged in a descending order according to the average human face feature similarity of all the images in the query set; and when merging, placing the retrieval result obtained by the current retrieval at the tail of the query set obtained by the last retrieval.
And 3, reducing the database by using the face matching result of the person portrait not to be inquired currently and the images of all other persons in the same frame with all the images in the face matching result of the person portrait to be inquired currently to obtain a reduced database.
The principle of the step is as follows: based on two assumptions: 1) exclusion Constraint (EC), if the face matches the result MiBelonging to the person portrait q currently to be queried with high confidenceiThen face matching result MiWill belong to other person portraits q to be inquiredjThe confidence of (j ≠ i) will be low; 2) Intra-Frame Constraint (IFC), and MiOther persons with the same video frame in the middle image cannot belong to the person portrait q to be inquirediThe upper right of fig. 1 illustrates the principle of intra-frame constraint, i.e. there may be multiple people simultaneously within the same image, if the first person already belongs to qiThen others who appear simultaneously with it will not likely belong to qiAnd the subsequent retrieval sorting should not be participated (actually, the retrieval can be put at the end of the sorting result). Specifically, the person portrait q to be inquired is calculatediIs ranked as a result ofiAccording to exclusion constraints, MiShould be at LiForemost of (A), and Mj(j ≠ i) then should be at LiTo the rearmost. At the same time, according to intra constraints, with MiAll images in the sameThe images of all other persons in one frame are marked as FiThen it should be on list LiAnd finally.
Based on the above principle, the final reduced database can be represented by the following set:
wherein, the i-th person portrait q to be inquirediAs the portrait of the person currently to be inquired, MjRepresenting a person portrait q not currently to be queriedjJ is 1,2, M, j is not equal to i, and M represents the number of the person portraits to be inquired; fiRepresenting the person portrait q to be inquirediFace matching result MiAll images of other people in the same frame.
Through the restriction of the two constraints, compared with the original face retrieval, only face information is used, but obvious performance improvement can be obtained.
And 4, respectively enhancing body characteristics of the reduced database and the images in the face matching result, taking the face matching result as a query library, and retrieving the reduced database by using the similarity of the enhanced body characteristics to obtain the retrieval result of the portrait of the person to be queried currently.
Somatic feature enhancement can be enhanced using KNN feature expansion, including: for each body feature, the enhanced body feature is obtained by performing weighted fusion on the K neighbor feature of the body feature, so that the enhanced body feature has robustness; by doing so, a query library and a reduced database U are obtainediAll images in (a) enhance the somatic features.
Then, for a single image u in the reduced database, respectively calculating the similarity of the enhanced body features of all the images in the query library, selecting Top- λ similarity (that is, λ highest similarities, and λ specific number can be set according to actual conditions or experience), and selecting Top- λ similarity from the similarity, where λ is the highest similarity, and λ specific number can be set according to actual conditions or experienceTaking the mean of the lambda highest similarities as the similarity of the single image u and the body features of the query library can reduce MiInfluence on body retrieval when the middle body dresses are different.
And after calculating the similarity between all the images in the reduced database and the body characteristics of the query library, sequencing all the images in the reduced database according to the similarity of the body characteristics, placing the images in the face matching result, and then placing the reduced images at the tail to form the final retrieval result of the person portrait to be queried. The reduced images are sorted according to the original sequence of the reduced images in the database, namely, the reduced images are sorted according to the ascending order of the sizes of the original corner marks of the images.
Fig. 2 shows a schematic diagram of calculating Top- λ similarity of the body of two different database images and the query library, where λ is set to 3, that is, 3 highest similarities are selected, and the 3 highest similarity means is calculated as the similarity of the body features of a single image and the query library, and the similarity calculated in this way is called Top- λ similarity and is distinguished from the conventional average similarity.
It should be noted that each image referred to in fig. 1 and fig. 2 is from an existing public data set, and therefore, there is no problem of face privacy; in addition, the image content, the number of search results of each part, and the content referred to in fig. 2 are all exemplified.
Compared with the traditional scheme, the scheme provided by the embodiment of the invention effectively excavates the face information in the base database, utilizes the mutual exclusivity of the character identities, reduces the database scale and improves the retrieval efficiency and performance. In addition, by reasonably designing the body retrieval scheme, the influence of body dress on body retrieval is reduced, and a more excellent and robust portrait query result is finally obtained.
Example two
The invention also provides a portrait query system, which is implemented mainly based on the method provided by the foregoing embodiment, as shown in fig. 3, the system mainly includes:
the characteristic extraction unit is used for respectively extracting the face characteristics of all the person portraits to be inquired in the video to be inquired and respectively extracting the face characteristics and the body characteristics of the images in the database;
the face iterative retrieval and face matching result acquisition unit is used for carrying out iterative retrieval on the current person portrait to be queried in the video to be queried by using the face characteristics to obtain a face matching result; when the current retrieval is carried out, retrieving images with the average human face feature similarity of all images in the retrieval set exceeding a threshold value in the database by using the query set obtained in the last retrieval, combining the obtained retrieval result of the current retrieval and the query set obtained in the last retrieval to be used as the query set obtained at the current time, and determining the human face matching result of the portrait of the person to be queried through the query set obtained at the last time; when the search is carried out for the first time, the query set only contains the portrait of the current person to be queried in the video to be queried, and the threshold value is gradually reduced when the search is carried out for each time;
the database reduction unit is used for reducing the database by using the face matching result of the person portrait which is not to be inquired currently and the images of all other persons of which all the images are in the same frame in the face matching result of the person portrait to be inquired currently to obtain a reduced database;
and the body retrieval and joint retrieval result generation unit is used for respectively enhancing body characteristics of the images in the reduced database and the face matching result, taking the face matching result as a query database, and retrieving the reduced database by using the similarity of the enhanced appearance characteristics to obtain the retrieval result of the portrait of the person to be queried currently.
It will be clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be performed by different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to perform all or part of the above described functions.
The main technical details of each unit in the system are already described in detail in the first embodiment, and thus are not described again.
EXAMPLE III
The present invention also provides a processing apparatus, as shown in fig. 4, which mainly includes: one or more processors; a memory for storing one or more programs; wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the methods provided by the foregoing embodiments.
Further, the processing device further comprises at least one input device and at least one output device; in the processing device, a processor, a memory, an input device and an output device are connected through a bus.
In the embodiment of the present invention, the specific types of the memory, the input device, and the output device are not limited; for example:
the input device can be a touch screen, an image acquisition device, a physical button or a mouse and the like;
the output device may be a display terminal;
the Memory may be a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as a disk Memory.
Example four
The present invention also provides a readable storage medium storing a computer program which, when executed by a processor, implements the method provided by the foregoing embodiments.
The readable storage medium in the embodiment of the present invention may be provided in the foregoing processing device as a computer readable storage medium, for example, as a memory in the processing device. The readable storage medium may be various media that can store program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.
Claims (10)
1. A portrait query method, comprising:
respectively extracting face features of all the person portraits to be inquired in the video to be inquired, and respectively extracting the face features and body features of the images in the database;
performing iterative retrieval on the current portrait of the person to be queried in the video to be queried by using the face features to obtain a face matching result; when the current retrieval is carried out, retrieving images with the average human face feature similarity of all images in the retrieval set exceeding a threshold value in the database by using the query set obtained in the last retrieval, combining the obtained retrieval result of the current retrieval and the query set obtained in the last retrieval to be used as the query set obtained at the current time, and determining the human face matching result of the portrait of the person to be queried through the query set obtained at the last time; when the search is carried out for the first time, the query set only contains the portrait of the current person to be queried in the video to be queried, and the threshold value is gradually reduced when the search is carried out for each time;
reducing the database by using the face matching result of the person portrait which is not to be inquired currently and the images of all other persons which are in the same frame with all the images in the face matching result of the person portrait to be inquired currently to obtain a reduced database;
and respectively enhancing body characteristics of the images in the reduced database and the face matching result, taking the face matching result as a query database, and retrieving the reduced database by using the similarity of the enhanced body characteristics to obtain the retrieval result of the person portrait to be queried currently.
2. The portrait query method according to claim 1, wherein the extracting of the face features of all the portrait of the person to be queried in the video to be queried respectively, and the extracting of the face features and the body features of the images in the database respectively comprises:
for a video to be retrieved, the video comprises M person portraits to be inquired
Let the database include N images, note
Wherein q isiRepresenting the i-th person's portrait to be queried, glRepresenting the ith image in the database;
respectively carrying out face detection on the M person portraits to be inquired through a face detection network, and extracting corresponding face features from all face detection results through a face feature extraction network;
respectively carrying out face detection on the N images in the database through a face detection network, and extracting corresponding face features from all face detection results through a face feature extraction network; and respectively carrying out body detection on the N images in the database through a body detection network, and extracting corresponding body characteristics from all body detection results through a body characteristic extraction network.
3. The portrait query method according to claim 1, wherein the performing iterative retrieval on the portrait of the current person to be queried in the video to be queried by using the face features to obtain the face matching result comprises:
setting the iterative retrieval frequency as T, gradually reducing the threshold value of each retrieval, wherein the threshold value during the iterative retrieval represents that: th (th)1≥th2≥…≥thTWherein th represents a threshold, T1, 2.., T;
the image q of the ith person to be inquirediAs the portrait of the person to be inquired currently, the first retrieval is carried out to inquire the set
Searching the database for the person portrait q to be inquirediThe similarity of the face features exceeds a threshold th1Is marked as { g ] as the first search result1，g2，…，gnN represents the number of the images retrieved for the first time, each g represents the image retrieved from the database, and the retrieval result for the first time and the i-th person portrait to be queried q are comparediMerging to obtain a query set
At the time of second retrieval, a query set obtained from the first retrieval is retrieved from the database
The average face feature similarity of all the images exceeds the threshold value th2As a second retrieval result; combining the second search result with the query set
Merging to obtain a query set
By analogy, the search is carried out for T times in an iteration mode, and the obtained query set is utilized
Determining face matching result Mi：
The retrieval results obtained by each iterative retrieval are images from the database and are arranged in a descending order according to the average human face feature similarity of all the images in the query set; and when merging, placing the retrieval result obtained by the current retrieval at the tail of the query set obtained by the last retrieval.
4. The portrait query method according to claim 1 or 3, wherein in the current retrieval, images with an average face feature similarity exceeding a threshold with respect to all images in the query set are retrieved from the database by using the query set obtained in the previous retrieval, and the merging of the obtained current retrieval result and the query set obtained in the previous retrieval as the current query set comprises:
the image q of the ith person to be inquirediAs the portrait of the person to be queried currently, the current retrieval is recorded as the tth time, and the query set obtained by the last retrieval is recorded as the tth time
And calculating the image with the average human face feature similarity of the images in the database and all the images in the query set exceeding a threshold value according to the following formula:
wherein the content of the first and second substances,
representing a set of queries obtained from a previous search
Q represents the set of queries obtained from the last search
A single image of (1), gjRepresenting the jth image in the database; s (-) is a face feature similarity calculation function,
in a database representing the calculation at the time of the t-th searchJ (th) image gjQuery set obtained by last retrieval
Average face feature similarity of all the images;
passing threshold thtThe screened t-th retrieval result is combined with the query set obtained by the last retrieval
Merging sets of queries obtained as the t-th time
5. The portrait query method according to claim 1 or 3, wherein the database is reduced by using the face matching result of the person portrait not currently to be queried and the images of all other persons in the same frame with all the images in the face matching result of the person portrait currently to be queried, and the obtained reduced database is represented as:
wherein, the i-th person portrait q to be inquirediAs the portrait of the person currently to be inquired, MjRepresenting a person portrait q not currently to be queriedjThe face matching result of (1), (2) ·, M, j ≠ i, where M denotes the number of person portraits to be queried; fiRepresenting the person portrait q to the ith inquired personiFace matching result MiAll images of other people in the same frame.
6. The portrait query method according to claim 1 or 2, wherein the performing of the somatic feature enhancement on the images in the reduced database and the face matching result respectively comprises:
the image q of the ith person to be inquirediAs the portrait of the person to be inquired currently, the corresponding face matching result M is usediAs a query library, the corresponding reduced database is recorded as Ui；
Using the KNN feature expansion to enhance the body features, including: and for each body feature, performing weighted fusion on the K adjacent features of the body feature to obtain the enhanced body feature.
7. The portrait query method according to claim 6, wherein the face matching result is used as a query library, the similarity of the enhanced body features is used for searching in the reduced database, and the obtaining of the search result of the portrait of the person to be queried comprises:
respectively calculating the similarity of the single image u in the reduced database and the enhanced body characteristics of all images in the query library, selecting Top-lambda similarity, namely lambda highest similarity, from the similarity, and taking the mean value of the lambda highest similarities as the body characteristic similarity of the single image u and the query library;
and after calculating the similarity between all the images in the reduced database and the body characteristics of the query database, sequencing all the images in the reduced database according to the similarity of the body characteristics, placing the images in the face matching result, and then placing the images in the reduced database at the tail according to the original sequence to form the final retrieval result of the person portrait to be queried currently.
8. A portrait query system, implemented based on the method of any one of claims 1 to 7, the system comprising:
the characteristic extraction unit is used for respectively extracting the face characteristics of all the person portraits to be inquired in the video to be inquired and respectively extracting the face characteristics and the body characteristics of the images in the database;
the face iterative retrieval and face matching result acquisition unit is used for carrying out iterative retrieval on the current person portrait to be queried in the video to be queried by using the face characteristics to obtain a face matching result; when the current retrieval is carried out, retrieving images with the average human face feature similarity of all images in the retrieval set exceeding a threshold value in the database by using the query set obtained in the last retrieval, combining the obtained retrieval result of the current retrieval and the query set obtained in the last retrieval to be used as the query set obtained at the current time, and determining the human face matching result of the portrait of the person to be queried through the query set obtained at the last time; when the search is carried out for the first time, the query set only contains the portrait of the current person to be queried in the video to be queried, and the threshold value is gradually reduced when the search is carried out each time;
the database reduction unit is used for reducing the database by utilizing the face matching result of the person portrait which is not to be inquired currently and the images of all other persons of which all the images are in the same frame in the face matching result of the person portrait to be inquired currently to obtain a reduced database;
and the body retrieval and joint retrieval result generation unit is used for respectively enhancing body characteristics of the images in the reduced database and the face matching result, taking the face matching result as a query database, and retrieving the reduced database by using the similarity of the enhanced appearance characteristics to obtain the retrieval result of the portrait of the person to be queried currently.
9. A processing device, comprising: one or more processors; a memory for storing one or more programs;
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.
10. A readable storage medium, storing a computer program, characterized in that the computer program, when executed by a processor, implements the method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210264194.0A CN114612985B (en) | 2022-03-17 | 2022-03-17 | Portrait query method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210264194.0A CN114612985B (en) | 2022-03-17 | 2022-03-17 | Portrait query method, system, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114612985A true CN114612985A (en) | 2022-06-10 |
| CN114612985B CN114612985B (en) | 2024-04-02 |
Family
ID=81865260
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210264194.0A Active CN114612985B (en) | 2022-03-17 | 2022-03-17 | Portrait query method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114612985B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040073543A1 (en) * | 2002-10-14 | 2004-04-15 | Samsung Electronics Co., Ltd. | Image retrieval method and apparatus using iterative matching |
| CN109766470A (en) * | 2019-01-15 | 2019-05-17 | 北京旷视科技有限公司 | Image search method, device and processing equipment |
| CN110532432A (en) * | 2019-08-21 | 2019-12-03 | 深圳供电局有限公司 | A kind of personage's trajectory retrieval method and its system, computer readable storage medium |
| CN111126249A (en) * | 2019-12-20 | 2020-05-08 | 深圳久凌软件技术有限公司 | Pedestrian re-identification method and device combining big data and Bayes |
-
2022
- 2022-03-17 CN CN202210264194.0A patent/CN114612985B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040073543A1 (en) * | 2002-10-14 | 2004-04-15 | Samsung Electronics Co., Ltd. | Image retrieval method and apparatus using iterative matching |
| CN109766470A (en) * | 2019-01-15 | 2019-05-17 | 北京旷视科技有限公司 | Image search method, device and processing equipment |
| CN110532432A (en) * | 2019-08-21 | 2019-12-03 | 深圳供电局有限公司 | A kind of personage's trajectory retrieval method and its system, computer readable storage medium |
| CN111126249A (en) * | 2019-12-20 | 2020-05-08 | 深圳久凌软件技术有限公司 | Pedestrian re-identification method and device combining big data and Bayes |
Non-Patent Citations (2)
| Title |
|---|
| 张皓;吴建鑫;: "基于深度特征的无监督图像检索研究综述", 计算机研究与发展, no. 09, 15 September 2018 (2018-09-15) * |
| 王太峰;袁平波;荚济民;俞能海;: "基于新闻环境的人物肖像检索", 山东大学学报(理学版), no. 03, 30 June 2006 (2006-06-30) * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114612985B (en) | 2024-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Cong et al. | Going from RGB to RGBD saliency: A depth-guided transformation model | |
| Wang et al. | Effective multi-query expansions: Collaborative deep networks for robust landmark retrieval | |
| Wu et al. | A novel clustering method for static video summarization | |
| Cong et al. | Towards scalable summarization of consumer videos via sparse dictionary selection | |
| EP2124159B1 (en) | Image learning, automatic annotation, retrieval method, and device | |
| Xu et al. | Geolocalized modeling for dish recognition | |
| Kuanar et al. | Video key frame extraction through dynamic Delaunay clustering with a structural constraint | |
| CN105100894B (en) | Face automatic labeling method and system | |
| Panda et al. | Diversity-aware multi-video summarization | |
| CN103207898B (en) | A kind of similar face method for quickly retrieving based on local sensitivity Hash | |
| US9626585B2 (en) | Composition modeling for photo retrieval through geometric image segmentation | |
| KR101711677B1 (en) | Clustering videos by location | |
| JP2001155169A (en) | Method and system for dividing, classifying and summarizing video image | |
| CN109255357B (en) | RGBD image collaborative saliency detection method | |
| CN106991373A (en) | A kind of copy video detecting method based on deep learning and graph theory | |
| Liu et al. | Iterative local-global collaboration learning towards one-shot video person re-identification | |
| CN113051984A (en) | Video copy detection method and apparatus, storage medium, and electronic apparatus | |
| CN111159475B (en) | Pedestrian re-identification path generation method based on multi-camera video image | |
| Zhang et al. | Bioinspired scene classification by deep active learning with remote sensing applications | |
| Wu et al. | Deep spatiotemporal LSTM network with temporal pattern feature for 3D human action recognition | |
| CN103745223A (en) | Face detection method and apparatus | |
| CN109241315B (en) | Rapid face retrieval method based on deep learning | |
| Zhang et al. | Referring camouflaged object detection | |
| Zheng et al. | Multi-spectral vehicle re-identification with cross-directional consistency network and a high-quality benchmark | |
| Gao et al. | Occluded person re-identification based on feature fusion and sparse reconstruction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |