CN114549502A - Method and device for evaluating face quality, electronic equipment and storage medium - Google Patents
Method and device for evaluating face quality, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114549502A CN114549502A CN202210191456.5A CN202210191456A CN114549502A CN 114549502 A CN114549502 A CN 114549502A CN 202210191456 A CN202210191456 A CN 202210191456A CN 114549502 A CN114549502 A CN 114549502A
- Authority
- CN
- China
- Prior art keywords
- face
- quality score
- face image
- image
- quality
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000013441 quality evaluation Methods 0.000 claims abstract description 81
- 238000001303 quality assessment method Methods 0.000 claims abstract description 27
- 239000013598 vector Substances 0.000 claims description 56
- 238000012549 training Methods 0.000 claims description 27
- 238000000605 extraction Methods 0.000 claims description 21
- 238000002372 labelling Methods 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 11
- 238000011156 evaluation Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 9
- 238000012935 Averaging Methods 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000005286 illumination Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013434 data augmentation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The present disclosure provides a method, an apparatus, an electronic device and a storage medium for face quality assessment, wherein the method comprises: acquiring a first face image; extracting the characteristics of the first face image by using a face recognition network to obtain Gaussian distribution characteristics for representing the uncertainty of the first face image; and determining a quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image. The Gaussian distribution characteristics in the method are fused with the relevant uncertainty characteristics of the images, compared with the traditional mode of determining the image quality by artificially stipulated logic, the method can more accurately evaluate the sample quality based on the predicted uncertainty of each image, is not interfered and restricted by factors such as scene environment and the like, has higher robustness, and thus meets the requirements of each application scene.
Description
Technical Field
The present disclosure relates to the field of image data processing technologies, and in particular, to a method and an apparatus for evaluating human face quality, an electronic device, and a storage medium.
Background
With the rapid development of Artificial Intelligence (AI), face recognition technology is widely used in various fields. In order to ensure the real-time performance and accuracy of face recognition, a face image with better quality is often selected from an acquired video to perform a recognition task.
In the traditional human face image quality screening process, human face quality evaluation is involved, the human face images are respectively scored mainly through a plurality of fixed dimensions such as illumination, ambiguity, human face pose and the like, and then the integral scores of the images are determined through integration of all scoring results. However, different scoring criteria are usually set to cope with different application scenarios. For example, for two scenes, namely day and night, the scoring criteria corresponding to the dimension of illumination are different, so that the existing quality assessment is not robust and is difficult to adapt to the requirements of each application scene.
Disclosure of Invention
The embodiment of the disclosure at least provides a method and a device for evaluating the quality of a human face, electronic equipment and a storage medium.
In a first aspect, an embodiment of the present disclosure provides a method for evaluating face quality, where the method includes:
acquiring a first face image;
extracting the features of the first face image by using a face recognition network to obtain Gaussian distribution features for representing the uncertainty of the first face image;
and determining a quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image.
By adopting the method for evaluating the face quality, under the condition that the first face image is obtained, the face recognition network can be used for extracting the characteristics of the face image, so that the quality evaluation result of the sample can be determined based on the obtained Gaussian distribution characteristics, mainly because the Gaussian distribution characteristics can represent the uncertainty of the sample, the quality of the sample with high uncertainty is poorer, and conversely, the quality of the sample with high certainty is higher. The Gaussian distribution characteristics in the method are fused with the relevant uncertainty characteristics of the images, compared with the traditional mode of determining the image quality by artificially stipulated logic, the method can more accurately evaluate the sample quality based on the predicted uncertainty of each image, is not interfered and restricted by factors such as scene environment and the like, has higher robustness, and thus meets the requirements of each application scene.
In one possible embodiment, the quality evaluation result includes a quality score, and the number of the first face images is plural; determining a quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image, wherein the determining comprises:
determining an initial quality score of each first face image based on Gaussian distribution characteristics respectively obtained by the plurality of first face images;
determining an average quality score corresponding to each first face image and a maximum quality score and a minimum quality score corresponding to each face category label pre-labeled for each first face image based on the initial quality score of each first face image;
and adjusting the initial quality score of each first face image under each face class label based on the maximum quality score and the minimum quality score corresponding to each face class label and the average quality score to obtain the adjusted quality score of each first face image.
Here, based on the distribution characteristic of the gaussian distribution feature, here, the initial quality score of the first face image may be adjusted based on the maximum quality score and the minimum quality score corresponding to the face category label, so that the adjusted quality score is distributed more uniformly, and overall quality evaluation is facilitated.
In a possible implementation manner, before the adjusting the initial quality score of each first face image under each face class label based on the maximum quality score and the minimum quality score corresponding to each face class label and the average quality score, the method further includes:
for each face class label, determining whether a first difference between the maximum quality score corresponding to the face class label and the average quality score is greater than a second difference between the average quality score and the minimum quality score corresponding to the face class label;
in response to the first difference being greater than the second difference, adjusting the minimum quality score corresponding to the face class label based on the maximum quality score corresponding to the face class label and the average quality score to obtain an adjusted minimum quality score corresponding to the face class label;
and in response to the first difference being less than or equal to the second difference, adjusting the maximum quality score corresponding to the face class label based on the minimum quality score corresponding to the face class label and the average quality score to obtain the adjusted maximum quality score corresponding to the face class label.
In a possible implementation manner, the adjusting the initial quality score of each first face image under each face category label based on the maximum quality score and the minimum quality score corresponding to each face category label and the average quality score includes:
determining a third difference value between the initial quality score of each first face image under the face class label and the minimum quality score corresponding to the face class label and a fourth difference value between the maximum quality score and the minimum quality score corresponding to the face class label aiming at each face class label;
and performing ratio operation on the third difference and the fourth difference to determine the adjusted quality score of each first face image under the face type label.
Here, the quality score may be adjusted based on the proximity between the third difference and the fourth difference, further making the adjusted quality score more uniform, thereby facilitating subsequent network training.
In one possible implementation, after determining the quality evaluation result of the first face image, the method further includes:
and taking the first face image as input data of a face quality evaluation network to be trained, taking a quality evaluation result of the first face image as comparison supervision data of an output result of the face quality evaluation network to be trained, and training the face quality evaluation network to be trained to obtain the trained face quality evaluation network.
In a possible implementation manner, before the taking the first face image as input data of a face quality assessment network to be trained and taking a quality assessment result of the first face image as comparison supervision data of an output result of the face quality assessment network to be trained, the method further includes:
and under the condition that the network training speed is determined to be smaller than a preset threshold value, carrying out gray processing on the first face image to obtain a processed first face image.
Here, the first face image may be subjected to gray scale processing first, so as to obtain a processed first face image with a smaller amount of calculation required, so as to achieve a higher network training speed.
In a possible implementation manner, after obtaining the trained face quality assessment network, the method further includes:
acquiring an image to be evaluated;
inputting the image to be evaluated into a trained human face quality evaluation network to obtain a quality score of the image to be evaluated;
and responding to the fact that the quality score of the image to be evaluated is larger than a preset threshold value, and performing face comparison on the image to be evaluated to obtain a face comparison result.
Here, the face comparison is performed only when it is determined that the quality score of the image to be evaluated is relatively high, thereby realizing faster and safer face recognition.
In one possible embodiment, the face recognition network comprises an extraction layer and a classification layer; the face recognition network is trained according to the following steps:
acquiring each second face image;
for each second face image in the plurality of second face images, performing feature extraction on the second face image through an extraction layer of the face recognition network to obtain an image feature vector and a Gaussian distribution feature vector for representing uncertainty of the second face image;
sampling the image characteristic vector and the Gaussian distribution characteristic vector under the same dimension to obtain sampled image characteristics;
classifying the sampled image features through a classification layer of the face recognition network to obtain a classification result, and comparing the classification result with a pre-labeling result of the second face image;
and responding to the comparison result indicating that the classification result is inconsistent with the pre-labeling result, adjusting the network parameter value of the face recognition network, and performing the next round of network training until the obtained comparison result indicates that the classification result is consistent with the pre-labeling result.
In one possible implementation, the gaussian distribution features include a gaussian variance vector consistent with a dimension of an image feature vector of the first face image; determining a quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image, wherein the determining comprises:
averaging all vector values included in a Gaussian variance vector obtained by the first face image to obtain an average vector value;
and taking the average vector value as a quality evaluation result of the first face image.
The gaussian distribution feature can be a gaussian variance vector corresponding to the image feature vector, each dimension of the vector can represent the deviation condition of the corresponding dimension of the image feature vector, and a more accurate quality evaluation result can be obtained by averaging vector values.
In a second aspect, an embodiment of the present disclosure further provides an apparatus for evaluating a face quality, where the apparatus includes:
the acquisition module is used for acquiring a first face image;
the extraction module is used for extracting the characteristics of the first face image by using a face recognition network to obtain Gaussian distribution characteristics for representing the uncertainty of the first face image;
and the evaluation module is used for determining the quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image.
In a third aspect, an embodiment of the present disclosure further provides an electronic device, including: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating via the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the steps of the method of face quality assessment according to the first aspect and any of its various embodiments.
In a fourth aspect, the disclosed embodiments also provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs the steps of the method for face quality assessment according to the first aspect and any of its various embodiments.
For the description of the effects of the above-mentioned apparatus, electronic device, and computer-readable storage medium for face quality assessment, reference is made to the above description of the method for face quality assessment, which is not repeated herein.
In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly described below, and the drawings herein incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the technical solutions of the present disclosure. It is appreciated that the following drawings depict only certain embodiments of the disclosure and are therefore not to be considered limiting of its scope, for those skilled in the art will be able to derive additional related drawings therefrom without the benefit of the inventive faculty.
Fig. 1 shows a flowchart of a method for face quality assessment provided by an embodiment of the present disclosure;
fig. 2 is a schematic diagram illustrating an apparatus for face quality assessment provided by an embodiment of the present disclosure;
fig. 3 shows a schematic diagram of an electronic device provided by an embodiment of the present disclosure.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions in the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure, as generally described and illustrated in the figures herein, could be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure, presented in the figures, is not intended to limit the scope of the claimed disclosure, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the disclosure without making creative efforts, shall fall within the protection scope of the disclosure.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
The term "and/or" herein merely describes an associative relationship, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.
Research shows that face recognition is one of the most widely applied biometric identification technologies at present, is a common algorithm in the fields of security and the like, and is widely applied to many scenes, such as people and card comparison, access control and person search.
However, a complete face recognition system is not simple in overall flow, after a face image is acquired, face detection is firstly performed, quality judgment is performed after a face frame is obtained, face alignment is performed after a low-quality face image is filtered, and finally, the face image is input into a recognition system for face recognition. The quality evaluation is very important for the whole system and directly influences the subsequent recognition effect of the recognition system.
The quality of the face images obtained by image detection is uneven, and in view of judging efficiency, if the quality model is not sensitive enough to the low-quality images, the low-quality face images transmitted to the recognition model are easy to cause unnecessary false recognition or rejection recognition; if the high-quality face image is too severe, the face image is screened out with high probability, and the recognition efficiency is also low.
The existing face quality evaluation method mainly scores face images through a plurality of fixed dimensions, such as illumination, ambiguity, face pose and the like, and then determines the integral scores of the images through the respective scoring results. However, different scoring criteria may need to be set for different application scenarios in this manner, for example, for two scenarios, namely day and night, the scoring criteria corresponding to the dimension of illumination are different, so that the robustness of the existing quality assessment is not high, and the existing quality assessment cannot meet the requirements of each application scenario.
Based on the above research, the present disclosure provides a scheme for face quality evaluation based on gaussian distribution feature extraction to determine a quality evaluation result, so as to improve the robustness of quality evaluation on the premise of ensuring high accuracy evaluation.
To facilitate understanding of the present embodiment, first, a method for evaluating a face quality disclosed in the embodiments of the present disclosure is described in detail, where an execution subject of the method for evaluating a face quality provided in the embodiments of the present disclosure is generally an electronic device with certain computing capability, and the electronic device includes, for example: a terminal device, which may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle mounted device, a wearable device, or a server or other processing device. In some possible implementations, the method for face quality assessment may be implemented by a processor calling computer readable instructions stored in a memory.
Referring to fig. 1, which is a flowchart of a method for evaluating face quality provided in the embodiment of the present disclosure, the method includes steps S101 to S103, where:
s101: acquiring a first face image;
s102: extracting the characteristics of the first face image by using a face recognition network to obtain Gaussian distribution characteristics for representing the uncertainty of the first face image;
s103: and determining a quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image.
In order to facilitate understanding of the method for evaluating the face quality provided by the embodiment of the present disclosure, first, an application scenario of the method is briefly described below. The method for evaluating the face quality in the embodiment of the disclosure can be mainly applied to the field of face recognition, for example, can be applied to application scenarios such as witness comparison, entrance guard passing, and criminal search monitoring, and the face with a better quality evaluation result can be applied to each application scenario to realize high-quality face recognition. Besides, the method can be separately applied to the related fields needing face quality evaluation, and is not particularly limited herein.
The evaluation robustness of a face quality evaluation mode realized based on a scoring mode of a plurality of fixed dimensions in the related technology is not high, and the requirements of each application scene cannot be met. Based on this, the embodiment of the present disclosure provides a method for evaluating face quality based on gaussian distribution feature extraction, so as to improve robustness of face quality evaluation.
The first face image may be any image that needs to be subjected to face quality evaluation, and for example, may be a face image captured by a camera included in an access control system. The first face image may be one or multiple, and may be, for example, several face images captured continuously for any target person.
For any first face image, feature extraction may be performed on the first face image by using a face recognition network, and the extracted features are gaussian distribution features used for representing uncertainty of the first face image, that is, the face recognition network in the embodiment of the present disclosure may map the first face image into one gaussian distribution in a high-dimensional space.
Since the gaussian distribution features in the embodiment of the present disclosure may represent the quality of the image to some extent, the quality evaluation result of the first face image may be determined based on the gaussian distribution features obtained from the first face image.
In a particular application, the gaussian distribution features may be gaussian variance vectors that are consistent with dimensions of an image feature vector of the first face image. Each dimension of the image feature vector can represent an image feature of a corresponding dimension, and a vector value of a gaussian variance vector of the corresponding dimension can be used for representing a deviation value of the image feature deviating from the corresponding dimension, wherein the smaller the deviation value is, the higher the image quality is to some extent, and conversely, the larger the deviation value is, the lower the sample quality is to some extent. Here, by averaging the vector values included in the gaussian variance vector obtained for the first face image, an average vector value can be obtained, and the obtained average vector value is taken as a quality evaluation result, so that the determined quality evaluation result is closer to the true situation.
Considering the key role of the training of the face recognition network to the gaussian distribution feature extraction, the following can specifically describe the training process related to the face recognition network, including the following steps:
step one, acquiring each second face image;
secondly, extracting the features of each second face image in the plurality of second face images through an extraction layer of a face recognition network to obtain an image feature vector and a Gaussian distribution feature vector for representing the uncertainty of the second face image;
sampling the image characteristic vector and the Gaussian distribution characteristic vector under the same dimension to obtain sampled image characteristics;
classifying the sampled image features through a classification layer of the face recognition network to obtain a classification result, and comparing the classification result with a pre-labeling result of the second face image;
and step five, responding to the fact that the comparison result indicates that the classification result is inconsistent with the pre-labeling result, adjusting the network parameter value of the face recognition network, and performing the next round of network training until the obtained comparison result indicates that the classification result is consistent with the pre-labeling result.
The face recognition network comprises an extraction layer with two output branches, which are used for predicting an image feature vector mu in a d dimension and a Gaussian distribution feature vector sigma respectively. In the training stage, the image feature vector and the gaussian distribution feature vector adopt the feature vector with the same dimension, so that the image feature vector and the gaussian distribution feature vector can be sampled in the same dimension to obtain the sampled image feature, for example, the image feature value and the gaussian distribution feature value in the second dimension can be simultaneously sampled, so as to analyze the image quality from a single feature dimension.
Under the condition of a classification layer included in the sampled image characteristic face recognition network, iterative training of the network can be realized based on a comparison result between an obtained classification result and a pre-labeling result of a second face image until the training is ended, and the trained face recognition network is obtained.
In the embodiment of the disclosure, in order to determine the quality evaluation result of each first face image at the same reference level, the initial quality score of the first face image may be determined first, and then the adjustment may be performed according to the overall quality score condition, so as to obtain a more accurate adjusted quality score. The method can be realized by the following steps:
step one, determining an initial quality score of each first face image based on Gaussian distribution characteristics respectively obtained by a plurality of first face images;
determining an average quality score corresponding to each first face image and a maximum quality score and a minimum quality score corresponding to each face category label pre-labeled for each first face image based on the initial quality score of each first face image;
and thirdly, adjusting the initial quality score of each first face image under each face class label based on the maximum quality score and the minimum quality score corresponding to each face class label and the average quality score to obtain the adjusted quality score of the first face image.
Here, the average quality score corresponding to each first face image, and the maximum quality score and the minimum quality score corresponding to each face category label pre-labeled for each first face image may be determined based on the initial quality score of each first face image, and then the initial quality score of each first face image under each face category label may be adjusted based on the several quality scores.
However, the average quality score may be determined by averaging the initial quality scores of the respective first face images, considering that the average quality score reflects the score of the entire sample. The maximum quality score and the minimum quality score corresponding to each face category label may correspond to scores of corresponding category labels, and may be selected from initial quality scores of respective first face images belonging to the face category label, that is, the maximum and minimum quality scores in one category label correspond to each other. The method of obtaining the average quality score may be, for example, a method of removing the maximum value and the minimum value and then averaging the values, and is not limited herein.
In the process of adjusting each face category label, a third difference between the initial quality score of each first face image under the face category label and the minimum quality score corresponding to the face category label and a fourth difference between the maximum quality score and the minimum quality score corresponding to the face category label may be determined, and then a ratio operation may be performed on the third difference and the fourth difference to determine an adjusted quality score of each first face image under the face category label.
Here, the quality scores of the first face images may be adjusted to be within the quality distribution range where the minimum quality score and the maximum quality score are located, so that the quality evaluation is at the same reference level, and under-fitting of subsequent network training, which may be caused by a relatively concentrated quality score corresponding to one category label, may also be reduced.
Before the quality score is adjusted, the deviation condition of the overall score can be determined under each face class label, and overall distribution adjustment is further realized based on the deviation condition. Here, it may be determined whether a first difference between the maximum quality score and the average quality score corresponding to the face class label is greater than a second difference between the average quality score and the minimum quality score corresponding to the face class label, and based on this determination, it may be determined whether the face class label deviates from the minimum quality score as a whole or deviates from the maximum quality score as a whole.
When it is determined that the whole is deviated from the maximum quality score, that is, the first difference is greater than the second difference, the minimum quality score corresponding to the face category label may be adjusted based on the maximum quality score corresponding to the face category label and the average quality score, and then the quality scores of the first face images in the face category label are adjusted by the adjusted minimum quality score.
Similarly, when it is determined that the minimum quality score deviates, that is, the first difference is smaller than the second difference, the maximum quality score corresponding to the face category label may be adjusted based on the minimum quality score corresponding to the face category label and the average quality score, and then the quality scores of the first face images in the face category label are adjusted by the adjusted maximum quality score.
To facilitate further understanding of the above adjustment process regarding the quality score, a specific description may be provided below in conjunction with the formula. Here, a plurality of first face images may be associated, and the quality score corresponding to the ith personal face category label may be set to siThe determined average mass score is smThe maximum quality score and the minimum quality score in the ith personal face category label are respectively si maxAnd si min. Here, the offset may be performed using the following equation:
if si max-sm>sm-si minThen si min＝2*sm-si max；
If si max-sm≤sm-si minThen si max＝2*sm-si min。
Wherein,
it can be known that, in the embodiment of the present disclosure, the feature of each first face image is not a predicted vector but a gaussian distribution, the variance σ of each image distribution can be regarded as the uncertainty of this sample, and the fundamental initial quality score a can be obtained by calculating the harmonic mean of σ. And then, carrying out distribution adjustment on the initial quality score A, wherein the distribution adjustment comprises normalization and stretching to artificial visual distribution, and obtaining an adjusted quality score B.
In the embodiment of the present disclosure, in the case of determining the quality evaluation result of each first face image, training on a face quality evaluation network may be performed. Here, the first face image may be used as input data of a face quality evaluation network to be trained, the quality evaluation result of the first face image may be used as comparison supervision data of an output result of the face quality evaluation network to be trained, and the face quality evaluation network to be trained may be trained to obtain a trained face quality evaluation network.
In a specific application, the adopted quality evaluation result may be the initial quality score a of the above basis, or may be the quality score B after distribution adjustment.
The face quality evaluation network training may be a correspondence between the first face image and the quality score, so that under the condition that the face quality evaluation network is obtained through training, the quality evaluation of any image to be evaluated can be realized.
In the embodiment of the disclosure, after each first face image generates an image score pair, a convolutional neural network may be used as a face quality evaluation network to train a regression task, the network inputs an image of h × w, and the number of channels may be determined according to the scale of the network.
In specific applications, in some application scenarios with high real-time requirements, such as automatic driving, a network is often required to be constructed and updated more quickly, and in such a case, a clear requirement is imposed on the network training speed. In the embodiment of the disclosure, under the condition that it is determined that the network training speed is less than the preset threshold, the gray processing may be performed on the first face image, so that the processed first face image is used as input data of the network.
In addition, in the embodiment of the present disclosure, RGB may also be used as an input, a trunk layer in the network may use a small-scale network such as mobilenetv2, and output as a single numerical value, the numerical value may obtain a prediction score after passing through a sigmoid function, and the prediction score may be compared with a corresponding quality score in an image score pair, so as to determine a loss function value, and further support training of the network.
In practical application scenarios, the training network and data can be fine-tuned for special cases of some specific scenarios. For example, a security testimony compares the concern about motion blur and the concern about the occlusion of ornaments, and by adding such data or enriching a data augmentation scheme, the network can be made more sensitive to the low-quality situation.
In the embodiment of the disclosure, under the condition that a face quality evaluation network is obtained through training, the rapid quality evaluation of an image to be evaluated can be realized based on the evaluation network, and then subsequent face comparison is facilitated, which can be specifically realized through the following steps:
step one, obtaining an image to be evaluated;
inputting the image to be evaluated into a trained human face quality evaluation network to obtain the quality score of the image to be evaluated;
and thirdly, performing face comparison on the image to be evaluated in response to the fact that the quality score of the image to be evaluated is larger than a preset threshold value, and obtaining a face comparison result.
Here, when the quality score of the image to be evaluated is obtained based on the face quality evaluation network, it may be determined whether the quality score of the image to be evaluated is greater than a preset threshold, and when it is determined that the quality score is higher, face comparison may be performed to verify related information of the face.
In practical application, the face quality evaluation network can be directly embedded into a face recognition process and used for screening out low-quality images in a sequence image or directly selecting the highest-quality image and transmitting the highest-quality image to a subsequent recognition model; and the method can also be used for ensuring high-quality business scenes when the human face is put in storage.
It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.
Based on the same inventive concept, the embodiment of the present disclosure further provides a device for face quality evaluation corresponding to the method for face quality evaluation, and as the principle of solving the problem of the device in the embodiment of the present disclosure is similar to the method for face quality evaluation in the embodiment of the present disclosure, the implementation of the device may refer to the implementation of the method, and repeated details are not repeated.
Referring to fig. 2, a schematic diagram of an apparatus for evaluating face quality according to an embodiment of the present disclosure is shown, where the apparatus includes: an acquisition module 201, an extraction module 202 and an evaluation module 203; wherein,
an obtaining module 201, configured to obtain a first face image;
the extraction module 202 is configured to perform feature extraction on the first face image by using a face recognition network to obtain a gaussian distribution feature for representing uncertainty of the first face image;
and the evaluation module 203 is configured to determine a quality evaluation result of the first face image based on the gaussian distribution characteristics obtained by the first face image.
By adopting the device for evaluating the quality of the human face, under the condition that the first human face image is obtained, the characteristic extraction can be carried out on the human face image by utilizing the human face recognition network, so that the quality evaluation result of the sample can be determined based on the obtained Gaussian distribution characteristic, mainly because the Gaussian distribution characteristic can represent the uncertainty of the sample, the quality of the sample with high uncertainty is poorer, and conversely, the quality of the sample with high certainty is higher. The Gaussian distribution characteristics in the method are fused with the relevant uncertainty characteristics of the images, compared with the traditional mode of determining the image quality by artificially stipulated logic, the method can more accurately evaluate the sample quality based on the predicted uncertainty of each image, is not interfered and restricted by factors such as scene environment and the like, has higher robustness, and thus meets the requirements of each application scene.
In one possible embodiment, the quality evaluation result includes a quality score, and the number of the first face images is multiple; the evaluation module 203 is configured to determine a quality evaluation result of the first face image based on the gaussian distribution feature obtained by the first face image according to the following steps:
determining an initial quality score of each first face image based on Gaussian distribution characteristics respectively obtained by the plurality of first face images;
determining an average quality score corresponding to each first face image and a maximum quality score and a minimum quality score corresponding to each face category label pre-labeled for each first face image based on the initial quality score of each first face image;
and adjusting the initial quality score of each first face image under each face class label based on the maximum quality score and the minimum quality score corresponding to each face class label and the average quality score to obtain the adjusted quality score of each first face image.
In a possible implementation, the evaluation module 203 is further configured to:
determining whether a first difference value between the maximum quality score and the average quality score corresponding to each face class label is larger than a second difference value between the average quality score and the minimum quality score corresponding to the face class label or not for each face class label before adjusting the initial quality score of each first face image under each face class label based on the maximum quality score and the minimum quality score corresponding to each face class label and the average quality score; responding to the first difference value being larger than the second difference value, and adjusting the minimum quality score corresponding to the face class label based on the maximum quality score and the average quality score corresponding to the face class label to obtain the adjusted minimum quality score corresponding to the face class label; and in response to the first difference being less than or equal to the second difference, adjusting the maximum quality score corresponding to the face class label based on the minimum quality score and the average quality score corresponding to the face class label to obtain the adjusted maximum quality score corresponding to the face class label.
In one possible implementation, the evaluation module 203 is configured to adjust the initial quality score of each first face image under each face class label based on the maximum quality score and the minimum quality score corresponding to each face class label and the average quality score according to the following steps:
determining a third difference value between the initial quality score of each first face image under each face class label and the minimum quality score corresponding to the face class label and a fourth difference value between the maximum quality score and the minimum quality score corresponding to the face class label;
and performing ratio operation on the third difference and the fourth difference to determine the adjusted quality score of each first face image under the face type label.
In a possible embodiment, the above apparatus further comprises:
the training module 204 is configured to, after determining a quality evaluation result of the first face image, take the first face image as input data of a face quality evaluation network to be trained, take the quality evaluation result of the first face image as comparison supervision data of an output result of the face quality evaluation network to be trained, train the face quality evaluation network to be trained, and obtain a trained face quality evaluation network.
In a possible implementation manner, the training module 204 is further configured to use the first face image as input data of a to-be-trained face quality assessment network, and perform gray processing on the first face image under the condition that it is determined that a network training speed is less than a preset threshold before a quality assessment result of the first face image is used as comparison supervision data of an output result of the to-be-trained face quality assessment network, so as to obtain a processed first face image.
In a possible embodiment, the above apparatus further comprises:
the recognition module 205 is configured to obtain an image to be evaluated after the trained face quality evaluation network is obtained; inputting the image to be evaluated into a trained human face quality evaluation network to obtain the quality score of the image to be evaluated; and performing face comparison on the image to be evaluated in response to the fact that the quality score of the image to be evaluated is larger than a preset threshold value, and obtaining a face comparison result.
In one possible embodiment, the face recognition network comprises an extraction layer and a classification layer; the extracting module 202 is configured to train a face recognition network according to the following steps:
acquiring each second face image;
for each second face image in the plurality of second face images, performing feature extraction on the second face image through an extraction layer of a face recognition network to obtain an image feature vector and a Gaussian distribution feature vector for representing uncertainty of the second face image;
sampling the image characteristic vector and the Gaussian distribution characteristic vector under the same dimension to obtain the sampled image characteristics;
classifying the sampled image features through a classification layer of a face recognition network to obtain a classification result, and comparing the classification result with a pre-labeling result of a second face image;
and responding to the fact that the comparison result indicates that the classification result is inconsistent with the pre-labeling result, adjusting the network parameter value of the face recognition network, and performing the next round of network training until the obtained comparison result indicates that the classification result is consistent with the pre-labeling result.
In one possible embodiment, the gaussian distribution features include a gaussian variance vector consistent with a dimension of an image feature vector of the first face image; the evaluation module 203 is configured to determine a quality evaluation result of the first face image based on the gaussian distribution feature obtained by the first face image according to the following steps:
averaging all vector values included in a Gaussian variance vector obtained from the first face image to obtain an average vector value;
and taking the average vector value as a quality evaluation result of the first face image.
The description of the processing flow of each module in the device and the interaction flow between the modules may refer to the related description in the above method embodiments, and will not be described in detail here.
An embodiment of the present disclosure further provides an electronic device, as shown in fig. 3, which is a schematic structural diagram of the electronic device provided in the embodiment of the present disclosure, and the electronic device includes: a processor 301, a memory 302, and a bus 303. The memory 302 stores machine-readable instructions executable by the processor 301 (for example, execution instructions corresponding to the obtaining module 201, the extracting module 202, the evaluating module 203, and the like in the apparatus in fig. 2), when the electronic device is operated, the processor 301 and the memory 302 communicate through the bus 303, and when the machine-readable instructions are executed by the processor 301, the following processes are performed:
acquiring a first face image;
extracting the characteristics of the first face image by using a face recognition network to obtain Gaussian distribution characteristics for representing the uncertainty of the first face image;
and determining a quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image.
The disclosed embodiment also provides a computer readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to perform the steps of the method for evaluating the face quality in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.
The embodiments of the present disclosure also provide a computer program product, where the computer program product carries a program code, and instructions included in the program code may be used to execute the steps of the method for evaluating human face quality in the foregoing method embodiments, which may be referred to specifically for the foregoing method embodiments, and are not described herein again.
The computer program product may be implemented by hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing an electronic device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
Finally, it should be noted that: the above-mentioned embodiments are merely specific embodiments of the present disclosure, which are used for illustrating the technical solutions of the present disclosure and not for limiting the same, and the scope of the present disclosure is not limited thereto, and although the present disclosure is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: those skilled in the art can still make modifications or changes to the embodiments described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope of the disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present disclosure, and should be construed as being included therein. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims (12)
1. A method for face quality assessment, the method comprising:
acquiring a first face image;
extracting the features of the first face image by using a face recognition network to obtain Gaussian distribution features for representing the uncertainty of the first face image;
and determining a quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image.
2. The method according to claim 1, wherein the quality evaluation result includes a quality score, and the number of the first face images is plural; determining a quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image, wherein the determining comprises:
determining an initial quality score of each first face image based on Gaussian distribution characteristics respectively obtained by the plurality of first face images;
determining an average quality score corresponding to each first face image and a maximum quality score and a minimum quality score corresponding to each face category label pre-labeled for each first face image based on the initial quality score of each first face image;
and adjusting the initial quality score of each first face image under each face class label based on the maximum quality score and the minimum quality score corresponding to each face class label and the average quality score to obtain the adjusted quality score of each first face image.
3. The method of claim 2, wherein before adjusting the initial quality score of each first face image under each face class label based on the maximum quality score and the minimum quality score corresponding to each face class label and the average quality score, the method further comprises:
for each face class label, determining whether a first difference between the maximum quality score corresponding to the face class label and the average quality score is greater than a second difference between the average quality score and the minimum quality score corresponding to the face class label;
in response to the first difference being greater than the second difference, adjusting the minimum quality score corresponding to the face class label based on the maximum quality score corresponding to the face class label and the average quality score to obtain an adjusted minimum quality score corresponding to the face class label;
and in response to the first difference being less than or equal to the second difference, adjusting the maximum quality score corresponding to the face class label based on the minimum quality score corresponding to the face class label and the average quality score to obtain the adjusted maximum quality score corresponding to the face class label.
4. The method according to claim 2 or 3, wherein the adjusting the initial quality score of each first face image under each face class label based on the maximum quality score and the minimum quality score corresponding to each face class label and the average quality score comprises:
determining a third difference value between the initial quality score of each first face image under the face class label and the minimum quality score corresponding to the face class label and a fourth difference value between the maximum quality score and the minimum quality score corresponding to the face class label aiming at each face class label;
and performing ratio operation on the third difference and the fourth difference to determine the adjusted quality score of each first face image under the face type label.
5. The method according to any one of claims 1 to 4, wherein after determining the quality assessment result of the first face image, the method further comprises:
and taking the first face image as input data of a face quality evaluation network to be trained, taking a quality evaluation result of the first face image as comparison supervision data of an output result of the face quality evaluation network to be trained, and training the face quality evaluation network to be trained to obtain the trained face quality evaluation network.
6. The method according to claim 5, wherein before the first face image is used as input data of a face quality assessment network to be trained, and the quality assessment result of the first face image is used as comparison supervision data of an output result of the face quality assessment network to be trained, the method further comprises:
and under the condition that the network training speed is determined to be smaller than a preset threshold value, carrying out gray processing on the first face image to obtain a processed first face image.
7. The method according to claim 5 or 6, wherein after obtaining the trained face quality assessment network, the method further comprises:
acquiring an image to be evaluated;
inputting the image to be evaluated into a trained human face quality evaluation network to obtain a quality score of the image to be evaluated;
and responding to the fact that the quality score of the image to be evaluated is larger than a preset threshold value, and performing face comparison on the image to be evaluated to obtain a face comparison result.
8. The method of any one of claims 1 to 7, wherein the face recognition network comprises an extraction layer and a classification layer; the face recognition network is trained according to the following steps:
acquiring each second face image;
for each second face image in the plurality of second face images, performing feature extraction on the second face image through an extraction layer of the face recognition network to obtain an image feature vector and a Gaussian distribution feature vector for representing uncertainty of the second face image;
sampling the image characteristic vector and the Gaussian distribution characteristic vector under the same dimension to obtain sampled image characteristics;
classifying the sampled image features through a classification layer of the face recognition network to obtain a classification result, and comparing the classification result with a pre-labeling result of the second face image;
and responding to the comparison result indicating that the classification result is inconsistent with the pre-labeling result, adjusting the network parameter value of the face recognition network, and performing the next round of network training until the obtained comparison result indicates that the classification result is consistent with the pre-labeling result.
9. The method of any of claims 1 to 8, wherein the Gaussian distribution feature comprises a Gaussian variance vector consistent with a dimension of an image feature vector of the first face image; determining a quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image, wherein the determining comprises:
averaging all vector values included in a Gaussian variance vector obtained by the first face image to obtain an average vector value;
and taking the average vector value as a quality evaluation result of the first face image.
10. An apparatus for face quality assessment, the apparatus comprising:
the acquisition module is used for acquiring a first face image;
the extraction module is used for extracting the characteristics of the first face image by using a face recognition network to obtain Gaussian distribution characteristics for representing the uncertainty of the first face image;
and the evaluation module is used for determining the quality evaluation result of the first face image based on the Gaussian distribution characteristics obtained by the first face image.
11. An electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating via the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the steps of the method of face quality assessment according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, performs the steps of the method for face quality assessment according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210191456.5A CN114549502A (en) | 2022-02-28 | 2022-02-28 | Method and device for evaluating face quality, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210191456.5A CN114549502A (en) | 2022-02-28 | 2022-02-28 | Method and device for evaluating face quality, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114549502A true CN114549502A (en) | 2022-05-27 |
Family
ID=81661713
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210191456.5A Pending CN114549502A (en) | 2022-02-28 | 2022-02-28 | Method and device for evaluating face quality, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114549502A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114998978A (en) * | 2022-07-29 | 2022-09-02 | 杭州魔点科技有限公司 | Method and system for analyzing quality of face image |
-
2022
- 2022-02-28 CN CN202210191456.5A patent/CN114549502A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114998978A (en) * | 2022-07-29 | 2022-09-02 | 杭州魔点科技有限公司 | Method and system for analyzing quality of face image |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109697416B (en) | Video data processing method and related device | |
| CN107153817B (en) | Pedestrian re-identification data labeling method and device | |
| CN110795595B (en) | Video structured storage method, device, equipment and medium based on edge calculation | |
| CN111814902A (en) | Target detection model training method, target identification method, device and medium | |
| CN111738357B (en) | Junk picture identification method, device and equipment | |
| CN111461165A (en) | Image recognition method, recognition model training method, related device and equipment | |
| CN109522925B (en) | Image recognition method, device and storage medium | |
| CN111935479B (en) | Target image determination method and device, computer equipment and storage medium | |
| CN111582150A (en) | Method and device for evaluating face quality and computer storage medium | |
| CN110929785A (en) | Data classification method and device, terminal equipment and readable storage medium | |
| CN111401196A (en) | Method, computer device and computer readable storage medium for self-adaptive face clustering in limited space | |
| CN113283368B (en) | Model training method, face attribute analysis method, device and medium | |
| CN111291887A (en) | Neural network training method, image recognition method, device and electronic equipment | |
| CN110555439A (en) | identification recognition method, training method and device of model thereof and electronic system | |
| CN112215831B (en) | Method and system for evaluating quality of face image | |
| CN111694954B (en) | Image classification method and device and electronic equipment | |
| CN113269010B (en) | Training method and related device for human face living body detection model | |
| CN116543261A (en) | Model training method for image recognition, image recognition method device and medium | |
| CN117155706A (en) | Network abnormal behavior detection method and system | |
| CN114549502A (en) | Method and device for evaluating face quality, electronic equipment and storage medium | |
| CN113240092A (en) | Neural network training and face recognition method, device, equipment and storage medium | |
| Borah | Detecting Background Dynamic Scenes using Naive Bayes Classifier Analysis Compared to CNN Analysis | |
| CN112214639B (en) | Video screening method, video screening device and terminal equipment | |
| CN114782721A (en) | Image processing and target detection method, device, equipment and storage medium | |
| CN114445916A (en) | Living body detection method, terminal device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |