CN112836077A - Method, system, device and storage medium for searching specific person - Google Patents
Method, system, device and storage medium for searching specific person Download PDFInfo
- Publication number
- CN112836077A CN112836077A CN202110082869.5A CN202110082869A CN112836077A CN 112836077 A CN112836077 A CN 112836077A CN 202110082869 A CN202110082869 A CN 202110082869A CN 112836077 A CN112836077 A CN 112836077A
- Authority
- CN
- China
- Prior art keywords
- face
- image
- gallery
- feature
- features
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000001514 detection method Methods 0.000 claims abstract description 52
- 238000000605 extraction Methods 0.000 claims description 14
- 230000001815 facial effect Effects 0.000 claims description 14
- 238000003491 array Methods 0.000 claims description 11
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 238000002372 labelling Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 description 8
- 239000013598 vector Substances 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 208000029152 Small face Diseases 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 235000021110 pickles Nutrition 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/51—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Library & Information Science (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a specific character retrieval method, a system, equipment and a storage medium, wherein the method comprises the following steps: acquiring a gallery image, and carrying out face detection on the gallery image to obtain a face area image of the gallery image; extracting the face characteristics of the face region image of the gallery image to obtain the face characteristics of the gallery image; establishing a face feature library based on the face features of the image library, and establishing an index of the face feature library; acquiring a specific figure image, and extracting the face characteristics of the specific figure image; and inquiring the gallery images similar to the human face features of the specific character images based on the indexes of the human face feature library. The face retrieval method and the face retrieval system improve the recall rate of face retrieval, avoid missing detection and improve the retrieval speed.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, a system, a device, and a storage medium for retrieving a specific character.
Background
With the continuous development of internet technology, face search is more and more widely applied. The existing face search mainly aims at social networks, crawls blogs on the network and pictures in an electronic photo album by using a crawler, and then searches. The problems of the prior art are as follows: in the prior art, the speed and the performance of face search are difficult to balance, the performance is more concerned about the retrieval of specific persons, and the performance mainly refers to recall rate call and precision rate precision, wherein the recall rate call is more important and the retrieval efficiency can be ignored. Meanwhile, the technical scheme is heavier due to different purposes. The use scene of the prior art mainly aims at personal photos uploaded by a user, the scene is relatively simple, particularly, the number of the personal photos, stickers and murals is small, and meanwhile, the recall rate is influenced by small faces. In addition, in the prior art, a specific person interface is generally used for searching for a specific person, specific person identification is separately carried out on each picture, and a method for directly detecting the whole image library is lacked.
Disclosure of Invention
In view of the problems in the prior art, an object of the present invention is to provide a method, a system, a device and a storage medium for retrieving a specific person, which can improve the recall rate of face retrieval, avoid missing retrieval and improve the retrieval speed.
The embodiment of the invention provides a specific character retrieval method, which comprises the following steps:
acquiring a gallery image, and carrying out face detection on the gallery image to obtain a face area image of the gallery image;
extracting the face characteristics of the face region image of the gallery image to obtain the face characteristics of the gallery image;
establishing a face feature library based on the face features of the image library, and establishing an index of the face feature library;
acquiring a specific figure image, and extracting the face characteristics of the specific figure image;
and inquiring the gallery images similar to the human face features of the specific character images based on the indexes of the human face feature library.
In some embodiments, the acquiring a gallery image comprises:
downloading the gallery image based on the url of the gallery image;
judging whether the downloaded gallery image has an edge smaller than a preset pixel;
if yes, the gallery image is enlarged, and the side smaller than the preset pixel is zoomed to be equal to the preset pixel.
In some embodiments, the performing face detection on the gallery image includes performing face detection on the gallery image by using a retinafec detection algorithm, and a score threshold of the retinafec detection algorithm is 0.05.
In some embodiments, after obtaining the face region image of the gallery image, the method further includes the following steps:
and carrying out affine transformation on the face area images, aligning the face area images, and scaling the face area images to a specified size.
In some embodiments, creating a face feature library based on the face features of the gallery images includes the steps of:
and respectively storing the detected face features and the url of the corresponding gallery image into a feature array and a url list, and storing the corresponding relation between the serial number of the face features of the feature array and the serial number of the url list.
In some embodiments, the step of storing the detected face features and the url of the corresponding gallery image in a feature array and a url list respectively includes the following steps:
in the process of detecting the gallery images, storing each detected face feature and the url of the corresponding gallery image into a tuple, and forming a tuple list based on arrays stored by all face features of each gallery image;
storing all the human face features in each gallery image into an image feature file, wherein the image feature file is named by url base64 codes of the corresponding gallery image;
after all the gallery images are detected, the face features and the urls of the tuples in all the tuple lists are respectively taken out, the taken face features are spliced into a feature array, the taken urls are spliced into a url number list, and the corresponding relation between the serial numbers of the face features of the feature array and the serial numbers of the urls in the url list is stored.
In some embodiments, the step of establishing the index of the face feature library comprises the following steps:
and taking the feature array as the input of the IVFPQ algorithm of the faiss, and establishing and storing indexes for all the human face features in the feature array.
In some embodiments, querying a gallery image similar to the facial features of the specific character image based on the index of the facial feature library comprises the following steps:
using an interface of an IVFPQ algorithm of faiss, inquiring the index of the face feature library through the face features of the specific character image, and acquiring the serial numbers of the first number of face features with the highest similarity with the face features of the specific character image;
and acquiring the url corresponding to the url list based on the acquired sequence number of the face features, and taking the url as an output face detection result.
In some embodiments, after obtaining the sequence numbers of the first number of facial features with the highest similarity to the facial features of the specific person image, the method further includes performing an extended query by using the following steps:
taking the serial numbers of the human face features with the highest similarity with the human face features of the specific person image as the serial numbers of the human face features obtained by the first detection;
acquiring a second number of face features with the highest similarity with the face features of the specific person image, wherein the second number is less than or equal to the first number;
taking the face features with the highest similarity in the second quantity as input of the second detection, reusing an interface of an IVFPQ algorithm of the faiss to obtain the serial numbers of the face features with the highest similarity with the face features input in the second detection, and taking the serial numbers of the face features with the second quantity and the first quantity as the serial numbers of the face features obtained in the second detection;
and acquiring the url corresponding to the url list based on the acquired sequence number of the face feature, wherein the acquiring includes acquiring the url corresponding to the url list based on the sequence numbers of the face feature acquired by the first detection and the second detection.
In some embodiments, after the querying the gallery images similar to the facial features of the specific character image, the method further includes the following steps:
re-ranking the similar gallery images using pedestrian re-retrieval based on K-intersection neighbors.
In some embodiments, after the querying the gallery images similar to the facial features of the specific character image, the method further includes the following steps:
and manually detecting whether the image corresponding to the url in the face detection result comprises the face of the specific person by adopting a via image labeling tool.
The embodiment of the invention also provides a specific character retrieval system, which is used for realizing the specific character retrieval method and is characterized by comprising the following steps:
the image acquisition module is used for acquiring a gallery image;
the image library establishing module is used for carrying out face detection on the image of the image library to obtain a face area image of the image library, carrying out face feature extraction on the face area image of the image library to obtain face features of the image library, establishing a face feature library based on the face features of the image library and establishing an index of the face feature library;
the characteristic extraction module is used for acquiring a specific figure image and extracting the face characteristic of the specific figure image;
and the face retrieval module is used for inquiring the gallery images similar to the face features of the specific figure images based on the indexes of the face feature library.
An embodiment of the present invention further provides a specific character retrieval device, including:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the particular persona retrieval method via execution of the executable instructions.
The embodiment of the invention also provides a computer readable storage medium for storing a program, and the program realizes the steps of the specific character retrieval method when being executed by a processor.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
The specific character retrieval method, the system, the equipment and the storage medium have the following beneficial effects:
the invention divides the specific character search into two parts of library establishment and library searching. Firstly, a gallery image is collected and a face area image is extracted, after face features are extracted, a face feature library and an index are established, so that the step of establishing a library is realized, and then the library is searched based on the face features of the specific figure image. Therefore, the invention can realize the comprehensive retrieval of the image based on the whole gallery, improve the recall rate of face retrieval, avoid missing detection, improve the detection speed and ensure the operation efficiency under the conditions of complex scene and super-large data set.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
FIG. 1 is a flow chart of a method for retrieving a specific character according to an embodiment of the present invention;
FIG. 2 is a flow chart of the library creation in the method for retrieving a specific character according to an embodiment of the present invention;
FIG. 3 is a flow chart of the database search in the method for retrieving a specific character according to an embodiment of the present invention;
FIG. 4 is a block diagram of a specific character retrieval system according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a specific character retrieval apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the steps. For example, some steps may be decomposed, and some steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
As shown in fig. 1, an embodiment of the present invention provides a method for retrieving a specific character, including the following steps:
s100: acquiring a gallery image, and carrying out face detection on the gallery image to obtain a face area image of the gallery image;
s200: extracting the face characteristics of the face region image of the gallery image to obtain the face characteristics of the gallery image;
s300: establishing a face feature library based on the face features of the image library, and establishing an index of the face feature library;
s400: acquiring a specific figure image, and extracting the face characteristics of the specific figure image;
s500: and inquiring the gallery images similar to the human face features of the specific character images based on the indexes of the human face feature library.
In the specific character searching method, the specific character searching is divided into two parts of library building and library searching. Firstly, a gallery image is collected and a face region image is extracted through the step S100, after face features are extracted through the step S200, a face feature library and an index are established through the step S300, so that the step of establishing a library is realized, and then, the library is searched based on the face features of the specific character image through the steps S400 and S500. Therefore, the invention can realize the comprehensive retrieval of the image based on the whole gallery, improve the recall rate of face retrieval, avoid missing detection, improve the detection speed and ensure the operation efficiency under the conditions of complex scene and super-large data set.
In this embodiment, the step S100: collecting a gallery image, comprising the steps of:
downloading the gallery image based on url (Uniform Resource Locator) of the gallery image;
judging whether the downloaded gallery image has an edge smaller than a preset pixel;
if yes, the gallery image is enlarged, and the side smaller than the preset pixel is enlarged to be equal to the preset pixel;
if not, the downloaded gallery image does not need to be adjusted.
For example, for each image in the gallery, the image is first downloaded using the url of the image, the dimensions of the downloaded image are viewed, and if a dimension in the height and width is less than 1000 pixels, the image is scaled to have its minimum edge equal to 1000. Here, the value of the preset pixel may be set and adjusted as needed, and is not limited to the example. This embodiment is used to improve the accuracy of the face detection algorithm for small faces (e.g., 10 x 10) by appropriately magnifying the input gallery images when building the gallery.
In this embodiment, the face detection is performed on the gallery image, and includes performing face detection on the gallery image by using a retinafec detection algorithm, where a score threshold of the retinafec detection algorithm is 0.05, and the like. The invention can detect more human faces as far as possible by setting a smaller score threshold.
As shown in fig. 2, in this embodiment, after obtaining the face region image of the gallery image in step S100, the method further includes the following steps:
performing affine transformation on the face region image, aligning the face region image, and scaling the face region image to a specified size, for example, uniformly scaling 112 × 112, but the invention is not limited thereto.
In the embodiment, the feature extraction is performed on the aligned and zoomed face region images by adopting an ArcFace algorithm to obtain the face features of each face region image. In other alternative embodiments, other feature extraction algorithms may also be employed, and are not limited to the arcface algorithm.
Therefore, in this embodiment, in order to improve the library building speed, the library building process is divided into two parts: face detection, face alignment and face feature extraction. The two parts are processed in sequence, and batch reasoning and multiple processes can be used to improve the library building speed.
In this embodiment, the step S300: establishing a face feature library based on the face features of the gallery images, comprising the following steps:
and respectively storing the detected face features and the url of the corresponding gallery image into a feature array and a url list, and storing the corresponding relation between the serial number of the face features of the feature array and the serial number of the url list.
Further, the step of respectively storing the detected face features and the url of the corresponding gallery image in a feature array and a url list includes the following steps:
in the process of detecting the gallery images, storing each detected face feature and the url of the corresponding gallery image into a tuple, and forming a tuple list based on arrays stored by all face features of each gallery image;
storing all the human face features in each gallery image into an image feature file, wherein the image feature file is named by url base64 codes of the corresponding gallery image;
for example, each face feature is a list of type float32 with 512 dimensions long, each face stores a tuple array of (url, feature), wherein url is url of an image, feature is a face feature, and the face feature of an image is an array list of tags. Then all the face features in one image are stored as a pickle format file on a hard disk, namely an image feature file, and the name of the pickle file is the name of the url coded by base 64. If the picture does not have the face, the picture is stored as a folder file of an empty list.
After all the gallery images are detected, respectively taking out the face features and the urls of the arrays in all the array lists, splicing the taken-out face features into feature arrays, splicing the taken-out urls into url number lists, and storing the corresponding relation between the serial numbers of the face features of the feature arrays and the serial numbers of the urls of the url lists;
for example, after all the face features in all the images are extracted, url and face feature features of a tuple array in all the list are respectively taken out, url is converted and spliced into url list through pinckles, the face feature features are spliced into a numy array, namely a feature array, the size of the feature array is (n,512), n is the number of faces, the one-to-one corresponding sequence of url and the face feature features is unchanged, and url is conveniently acquired according to the sequence number of the face features during subsequent query.
In this embodiment, the step S300: establishing an index of the face feature library, comprising the following steps:
and taking the feature array as the input of the IVFPQ algorithm of the faiss, and establishing and storing indexes for all the human face features in the feature array.
For example, the IVFPQ algorithm of the faiss library uses a numpy array of the face features as input, trains all face features to build an index, and then stores the index for subsequent query.
In this embodiment, the specific person image obtained in step S400 may be a single image of the specific person, or multiple images of the same specific person from different angles. The face feature extraction method in the step S400 of extracting the face feature of the specific character image may be a face feature extraction method in library construction, that is, firstly, retinaface is used to perform face detection to obtain a face region image, and then, arcfacace is used to extract the face feature in the face region image, but the invention is not limited thereto.
In this embodiment, the step S500: based on the index of the face feature library, inquiring a gallery image similar to the face feature of the specific character image, comprising the following steps:
using an interface of an IVFPQ algorithm of faiss, inquiring the index of the face feature library through the face features of the specific character image, and acquiring the serial numbers of the first number of face features with the highest similarity with the face features of the specific character image; the similarity can be determined by adopting a Euclidean distance calculation mode, and the smaller the Euclidean distance between two characteristic vectors is, the higher the similarity between the two characteristic vectors is;
and acquiring the url corresponding to the url list based on the acquired sequence number of the face features, and taking the url as an output face detection result.
As shown in fig. 3, for an input specific person image, a similar procedure to the library construction is used to extract feature vectors of a face in the image. It is assumed that there is only one face in the input specific person image, and the same method can be applied to a plurality of faces in other embodiments. Using an interface provided by the fuzzy IVFPQ algorithm, searching indexes through the face feature vectors, finding the sequence numbers of the first m feature vectors closest to the Euclidean distance of the query feature vectors and the corresponding Euclidean distances, and then converting the sequence numbers into corresponding urls, wherein m corresponds to the first number.
In this embodiment, after obtaining the sequence numbers of the first number of facial features having the highest similarity to the facial features of the specific person image, the method further includes performing an extended query by using the following steps:
taking the serial numbers of the human face features with the highest similarity with the human face features of the specific person image as the serial numbers of the human face features obtained by the first detection;
acquiring a second number of face features with the highest similarity with the face features of the specific person image, wherein the second number is less than or equal to the first number;
taking the face features with the highest similarity in the second quantity as input of the second detection, reusing an interface of an IVFPQ algorithm of the faiss to obtain the serial numbers of the face features with the highest similarity with the face features input in the second detection, and taking the serial numbers of the face features with the second quantity and the first quantity as the serial numbers of the face features obtained in the second detection;
and acquiring the url corresponding to the url list based on the acquired sequence number of the face feature, wherein the acquiring includes acquiring the url corresponding to the url list based on the sequence numbers of the face feature acquired by the first detection and the second detection.
In practical application, a plurality of different images of the same person can be used for simultaneously inquiring so as to improve the recall rate. It is also assumed that there is only one face in the input image. Each input image returns m urls and their euclidean distances dist. And re-indexing the first 3 eigenvectors with the minimum Euclidean distance in the m approximate faces, and returning 3 x m results. Here, 3 corresponds to the second number, but the present invention is not limited thereto. In other embodiments, the second number may be selected to be other values. After re-retrieval, 4 × m results are returned for each face feature. Assuming that there are p different photos of the same person, the query will return p × 4 × m results, sort the results by euclidean distance dist, and output only the corresponding url.
In order to improve the library searching speed, the invention can manually carry out block operation, for example, a human face feature library is randomly divided into 10 blocks, each block uses IVFPQ to establish index and feature query, meanwhile, the barrel parameter nlist of the IVFPQ algorithm is adjusted to be maximum, and a smaller query barrel number nprobe is set during retrieval.
In this embodiment, after querying the gallery image similar to the face feature of the specific character image, the method further includes the following steps:
reordering (Re-rank) the similar gallery images based on K-intersection neighbors using pedestrian Re-search (ReiD).
Therefore, the invention can input a plurality of different images of a specific character during library check, including images of different angles such as age, posture, expression, illumination and the like, simultaneously can re-search the query result of each query image, and finally re-sequence the query results and the re-search results of the plurality of images, thereby improving the recall rate and the search efficiency of the search. The similarity calculation is described by taking the euclidean distance as an example, and in other alternative embodiments, other similarity calculation methods, such as cosine similarity, may also be adopted, and all of them fall within the protection scope of the present invention.
In this embodiment, after querying the gallery image similar to the face feature of the specific character image, the method further includes the following steps:
and manually detecting whether the image corresponding to the url in the face detection result comprises the face of the specific person by adopting a via image labeling tool.
As shown in fig. 4, an embodiment of the present invention further provides a specific character retrieval system, which is configured to implement the specific character retrieval method, where the system includes:
the image acquisition module M100 is used for acquiring a gallery image;
a gallery establishing module M200, configured to perform face detection on the gallery image to obtain a face area image of the gallery image, perform face feature extraction on the face area image of the gallery image to obtain a face feature of the gallery image, establish a face feature gallery based on the face feature of the gallery image, and establish an index of the face feature gallery;
the feature extraction module M300 is used for acquiring a specific character image and extracting the face features of the specific character image;
and the face retrieval module M400 is configured to query a gallery image similar to the face feature of the specific character image based on the index of the face feature library.
In the specific character retrieval system, the specific character retrieval is divided into two parts of library establishment and library searching. Firstly, a gallery image is acquired by the image acquisition module M100, a face region image is extracted by the gallery establishing module M200, after face features are extracted by the gallery establishing module M200, a face feature library and an index are established by the gallery establishing module M200, so that the step of establishing a library is realized, and then the library is searched by the feature extraction module M300 and the face retrieval module M400 based on the face features of the specific figure image. Therefore, the invention can realize the comprehensive retrieval of the image based on the whole gallery, improve the recall rate of face retrieval, avoid missing detection, improve the detection speed and ensure the operation efficiency under the conditions of complex scene and super-large data set.
In this embodiment, the image capturing module M100 capturing the gallery image includes downloading the gallery image based on url of the gallery image; judging whether the downloaded gallery image has an edge smaller than a preset pixel; if yes, scaling the edge smaller than the preset pixel to be equal to the preset pixel; if not, the downloaded gallery image does not need to be adjusted.
In this embodiment, the gallery establishing module M200 is further configured to, after obtaining the face region images of the gallery images, perform affine transformation on the face region images to align the face region images, and scale the face region images to a specified size, and then perform feature extraction on the aligned and scaled face region images by using an arcface algorithm to obtain the face features of each face region image.
In this embodiment, the gallery creation module M200 creating a face feature library based on the face features of the gallery images includes: in the process of detecting the gallery images, storing each detected face feature and the url of the corresponding gallery image into an array, and forming an array list based on the arrays stored by all the face features of each gallery image; storing all the human face features in each gallery image into an image feature file, wherein the image feature file is named by the url mapping value of the corresponding gallery image; after all the gallery images are detected, the face features and the urls of the arrays in all the array lists are taken out respectively, the taken face features are spliced into feature arrays, the taken urls are spliced into url number lists, and the corresponding relation between the serial numbers of the face features of the feature arrays and the serial numbers of the urls in the url lists is stored.
In this embodiment, the gallery creation module M200 creates an index of the face feature library, including: and taking the feature array as the input of the IVFPQ algorithm of the faiss, and establishing and storing indexes for all the human face features in the feature array.
In this embodiment, the feature extraction module M300 may extract the facial features of a specific person by using the method of extracting the facial features by using the gallery creation module M200, but the invention is not limited thereto.
In this embodiment, the face retrieval module M400 queries a gallery image similar to the face feature of the specific character image, including: using an interface of an IVFPQ algorithm of faiss, inquiring the index of the face feature library through the face features of the specific character image, and acquiring the serial numbers of the first number of face features with the highest similarity with the face features of the specific character image; the similarity can be determined by adopting a Euclidean distance calculation mode, and the smaller the Euclidean distance between two characteristic vectors is, the higher the similarity between the two characteristic vectors is; and acquiring the url corresponding to the url list based on the acquired sequence number of the face features, and taking the url as an output face detection result.
The embodiment of the invention also provides specific character retrieval equipment, which comprises a processor; a memory having stored therein executable instructions of the processor; wherein the processor is configured to perform the steps of the particular persona retrieval method via execution of the executable instructions.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 5. The electronic device 600 shown in fig. 5 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 that connects the various system components (including the storage unit 620 and the processing unit 610), a display unit 640, and the like.
Wherein the storage unit stores program code executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention described in the above-mentioned specific person retrieval method section of the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
In the specific character retrieval apparatus, the program in the memory implements the steps of the specific character retrieval method when executed by the processor, and therefore, the computer storage medium can also obtain the technical effects of the specific character retrieval method described above.
The embodiment of the invention also provides a computer readable storage medium for storing a program, and the program realizes the steps of the specific character retrieval method when being executed by a processor. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above-mentioned part of the description of the specific character retrieval method when said program product is carried out on said terminal device.
Referring to fig. 6, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be executed on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The program in the computer storage medium implements the steps of the specific character retrieval method when executed by the processor, and therefore, the computer storage medium can also achieve the technical effects of the specific character retrieval method described above.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (14)
1. A specific character retrieval method is characterized by comprising the following steps:
acquiring a gallery image, and carrying out face detection on the gallery image to obtain a face area image of the gallery image;
extracting the face characteristics of the face region image of the gallery image to obtain the face characteristics of the gallery image;
establishing a face feature library based on the face features of the image library, and establishing an index of the face feature library;
acquiring a specific figure image, and extracting the face characteristics of the specific figure image;
and inquiring the gallery images similar to the human face features of the specific character images based on the indexes of the human face feature library.
2. The specific character retrieval method according to claim 1, wherein the capturing of the gallery image comprises the steps of:
downloading the gallery image based on the url of the gallery image;
judging whether the downloaded gallery image has an edge smaller than a preset pixel;
if yes, the gallery image is enlarged, and the side smaller than the preset pixel is zoomed to be equal to the preset pixel.
3. The method of claim 1, wherein the detecting the face of the gallery image comprises detecting the face of the gallery image using a retinaface detection algorithm, and the score threshold of the retinaface detection algorithm is 0.05.
4. The method for retrieving a specific person as claimed in claim 1, further comprising the following steps after obtaining the face region image of the gallery image:
and carrying out affine transformation on the face area images, aligning the face area images, and scaling the face area images to a specified size.
5. The method for retrieving a specific person as claimed in claim 1, wherein the step of creating a face feature library based on the face features of the images in the library comprises the steps of:
and respectively storing the detected face features and the url of the corresponding gallery image into a feature array and a url list, and storing the corresponding relation between the serial number of the face features of the feature array and the serial number of the url list.
6. The method of claim 5, wherein the step of storing the detected face features and the url of the corresponding gallery image in a feature array and a url list respectively comprises the steps of:
in the process of detecting the gallery images, storing each detected face feature and the url of the corresponding gallery image into a tuple, and forming a tuple list based on arrays stored by all face features of each gallery image;
storing all the human face features in each gallery image into an image feature file, wherein the image feature file is named by the urlbase64 code of the corresponding gallery image;
after all the gallery images are detected, the face features and the urls of the tuples in all the tuple lists are respectively taken out, the taken face features are spliced into a feature array, the taken urls are spliced into a url number list, and the corresponding relation between the serial numbers of the face features of the feature array and the serial numbers of the urls in the url list is stored.
7. The method of claim 5, wherein the step of creating an index of the face feature library comprises the steps of:
and taking the feature array as the input of the IVFPQ algorithm of the faiss, and establishing and storing indexes for all the human face features in the feature array.
8. The specific character retrieval method as claimed in claim 7, wherein the step of searching a gallery image similar to the facial feature of the specific character image based on the index of the facial feature library comprises the steps of:
using an interface of an IVFPQ algorithm of faiss, inquiring the index of the face feature library through the face features of the specific character image, and acquiring the serial numbers of the first number of face features with the highest similarity with the face features of the specific character image;
and acquiring the url corresponding to the url list based on the acquired sequence number of the face features, and taking the url as an output face detection result.
9. The method of claim 8, further comprising performing an extended query by using the following steps after obtaining the sequence numbers of the first number of facial features having the highest similarity to the facial features of the specific character image:
taking the serial numbers of the human face features with the highest similarity with the human face features of the specific person image as the serial numbers of the human face features obtained by the first detection;
acquiring a second number of face features with the highest similarity with the face features of the specific person image, wherein the second number is less than or equal to the first number;
taking the face features with the highest similarity in the second quantity as input of the second detection, reusing an interface of an IVFPQ algorithm of the faiss to obtain the serial numbers of the face features with the highest similarity with the face features input in the second detection, and taking the serial numbers of the face features with the second quantity and the first quantity as the serial numbers of the face features obtained in the second detection;
and acquiring the url corresponding to the url list based on the acquired sequence number of the face feature, wherein the acquiring includes acquiring the url corresponding to the url list based on the sequence numbers of the face feature acquired by the first detection and the second detection.
10. The specific character retrieval method as claimed in claim 8, wherein after said searching for the gallery image similar to the face feature of the specific character image, further comprising the steps of:
re-ranking the similar gallery images using pedestrian re-retrieval based on K-intersection neighbors.
11. The specific character retrieval method as claimed in claim 8, wherein after said searching for the gallery image similar to the face feature of the specific character image, further comprising the steps of:
and manually detecting whether the image corresponding to the url in the face detection result comprises the face of the specific person by adopting a via image labeling tool.
12. A specific character retrieval system for implementing the specific character retrieval method according to any one of claims 1 to 11, the system comprising:
the image acquisition module is used for acquiring a gallery image;
the image library establishing module is used for carrying out face detection on the image of the image library to obtain a face area image of the image library, carrying out face feature extraction on the face area image of the image library to obtain face features of the image library, establishing a face feature library based on the face features of the image library and establishing an index of the face feature library;
the characteristic extraction module is used for acquiring a specific figure image and extracting the face characteristic of the specific figure image;
and the face retrieval module is used for inquiring the gallery images similar to the face features of the specific figure images based on the indexes of the face feature library.
13. A specific person retrieval device characterized by comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the specific character retrieval method of any one of claims 1 to 11 via execution of the executable instructions.
14. A computer-readable storage medium storing a program, wherein the program when executed by a processor implements the steps of the specific character retrieval method of any one of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110082869.5A CN112836077B (en) | 2021-01-21 | 2021-01-21 | Specific person retrieval method, system, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110082869.5A CN112836077B (en) | 2021-01-21 | 2021-01-21 | Specific person retrieval method, system, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112836077A true CN112836077A (en) | 2021-05-25 |
| CN112836077B CN112836077B (en) | 2024-03-12 |
Family
ID=75929380
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110082869.5A Active CN112836077B (en) | 2021-01-21 | 2021-01-21 | Specific person retrieval method, system, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112836077B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115599791A (en) * | 2022-11-15 | 2023-01-13 | 以萨技术股份有限公司（Cn） | Milvus database parameter determination method, device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101211365A (en) * | 2006-12-28 | 2008-07-02 | 国际商业机器公司 | Method and system for building search index |
| US20160132720A1 (en) * | 2014-11-07 | 2016-05-12 | Noblis, Inc. | Vector-based face recognition algorithm and image search system |
| CN109241325A (en) * | 2018-09-11 | 2019-01-18 | 武汉魅瞳科技有限公司 | A kind of extensive face retrieval method and apparatus based on depth characteristic |
| CN111709303A (en) * | 2020-05-21 | 2020-09-25 | 北京明略软件***有限公司 | Face image recognition method and device |
| CN111753923A (en) * | 2020-07-02 | 2020-10-09 | 携程计算机技术（上海）有限公司 | Intelligent photo album clustering method, system, equipment and storage medium based on human face |
-
2021
- 2021-01-21 CN CN202110082869.5A patent/CN112836077B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101211365A (en) * | 2006-12-28 | 2008-07-02 | 国际商业机器公司 | Method and system for building search index |
| US20160132720A1 (en) * | 2014-11-07 | 2016-05-12 | Noblis, Inc. | Vector-based face recognition algorithm and image search system |
| CN109241325A (en) * | 2018-09-11 | 2019-01-18 | 武汉魅瞳科技有限公司 | A kind of extensive face retrieval method and apparatus based on depth characteristic |
| CN111709303A (en) * | 2020-05-21 | 2020-09-25 | 北京明略软件***有限公司 | Face image recognition method and device |
| CN111753923A (en) * | 2020-07-02 | 2020-10-09 | 携程计算机技术（上海）有限公司 | Intelligent photo album clustering method, system, equipment and storage medium based on human face |
Non-Patent Citations (1)
| Title |
|---|
| 王成济 等: "一种多层特征融合的人脸检测方法", 《智能***学报》, vol. 13, no. 1, pages 138 - 146 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115599791A (en) * | 2022-11-15 | 2023-01-13 | 以萨技术股份有限公司（Cn） | Milvus database parameter determination method, device and storage medium |
| CN115599791B (en) * | 2022-11-15 | 2023-03-10 | 以萨技术股份有限公司 | Milvus database parameter determination method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112836077B (en) | 2024-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10268703B1 (en) | System and method for associating images with semantic entities | |
| US9372920B2 (en) | Identifying textual terms in response to a visual query | |
| US20070237426A1 (en) | Generating search results based on duplicate image detection | |
| US20120072410A1 (en) | Image Search by Interactive Sketching and Tagging | |
| US20110106805A1 (en) | Method and system for searching multilingual documents | |
| CN113434716B (en) | Cross-modal information retrieval method and device | |
| CN114153795A (en) | Method and device for intelligently calling electronic archive, electronic equipment and storage medium | |
| JP6615225B2 (en) | Image processing apparatus, image search interface display apparatus, and image search interface display method | |
| CN108062422B (en) | Sorting method, intelligent terminal, system and storage medium for paging query | |
| CN116662839A (en) | Associated big data cluster analysis method and device based on multidimensional intelligent acquisition | |
| Al-Jubouri | Content-based image retrieval: Survey | |
| CN111126054A (en) | Method, device, storage medium and electronic equipment for determining similar texts | |
| CN112836077A (en) | Method, system, device and storage medium for searching specific person | |
| CN112287144B (en) | Picture retrieval method, equipment and storage medium | |
| JP2002007413A (en) | Image retrieving device | |
| CN111651625A (en) | Image retrieval method, image retrieval device, electronic equipment and storage medium | |
| CN111783786A (en) | Picture identification method and system, electronic equipment and storage medium | |
| CN115409041B (en) | Unstructured data extraction method, device, equipment and storage medium | |
| CN112199531B (en) | Cross-modal retrieval method and device based on hash algorithm and neighborhood graph | |
| CN111444159B (en) | Refined data processing method, device, electronic equipment and storage medium | |
| CN113986823A (en) | Picture archiving method, device, medium and equipment for communication machine room | |
| CN113486148A (en) | PDF file conversion method and device, electronic equipment and computer readable medium | |
| CN112328743A (en) | Code searching method and device, readable storage medium and electronic equipment | |
| JP2000231560A (en) | Automatic document classification system | |
| CN117573916B (en) | Retrieval method, device and storage medium for image text of marine unmanned aerial vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |