CN113435273B - Data augmentation method, data augmentation device, electronic device, and storage medium - Google Patents
Data augmentation method, data augmentation device, electronic device, and storage medium Download PDFInfo
- Publication number
- CN113435273B CN113435273B CN202110658676.XA CN202110658676A CN113435273B CN 113435273 B CN113435273 B CN 113435273B CN 202110658676 A CN202110658676 A CN 202110658676A CN 113435273 B CN113435273 B CN 113435273B
- Authority
- CN
- China
- Prior art keywords
- data
- face
- mask
- color
- depth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The embodiment of the invention relates to the field of data processing, and discloses a data augmentation method, a data augmentation device, electronic equipment and a storage medium. The data augmentation method comprises the following steps: acquiring mask template data and a face data set to be augmented, wherein the mask template data image comprises color mask template data and depth mask template data; and processing the color mask template data, the depth mask template data and the face data set to be augmented by using a preset face reconstruction network to generate an augmented face data set. The face data obtained by the method has a real effect and strong augmentation performance.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a data augmentation method and apparatus, an electronic device, and a storage medium.
Background
For face recognition, large-scale face data is the most important, a wearing mask face recognition mainly depends on a deep learning network to train and match data to obtain a stable recognition model, a large amount of wearing mask face data need to be input in the early stage of model training, and considering that the actual lack of a large amount of real wearing mask face data, the current wearing mask face recognition generally adopts the augmented wearing mask face data to be used for actual training.
However, the existing method for expanding the face data of the mask is as follows: the mask color mask template is pasted to the face by using the face key points, but the mask pasting effect by using the face key points is rough and not real, and the mask pasting by using the face key points is only suitable for augmenting two-dimensional face data of the mask, and cannot be augmented on face depth data, so that the augmentation performance of the face data of the mask is poor.
Disclosure of Invention
The embodiment of the invention aims to provide a data augmentation method, a data augmentation device, electronic equipment and a storage medium, which can acquire color mask template data and depth mask template data by using a face reconstruction network and augment the data to original face data, so that the acquired augmented face data has a real effect and strong augmentation performance.
An embodiment of the present invention provides a data augmentation method, including: acquiring mask template data and a face data set to be augmented, wherein the mask template data comprises color mask template data and depth mask template data; and processing the color mask template data, the depth mask template data and the face data set to be augmented by using a preset face reconstruction network to generate an augmented face data set.
An embodiment of the present invention also provides a data augmentation apparatus, including: the mask template data acquisition module is used for acquiring mask template data and a face data set to be augmented, wherein the mask template data comprises color mask template data and depth mask template data; and the augmentation module is used for processing the color mask template data, the depth mask template data and the face data set to be augmented by using a preset face reconstruction network to generate an augmented face data set.
An embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data augmentation method described above.
An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to implement the data augmentation method described above when executed by a processor.
According to the embodiment of the invention, in the process of face data augmentation, the color mask template data and the depth mask template data are acquired, and then the face reconstruction network is utilized to augment the color mask template data and the depth mask template data to the face data to be augmented to generate augmented face data, so that the acquired augmented face data is real in effect and strong in augmentation performance, and the technical problems of rough and unreal face data and poor augmentation performance of masks caused by the adoption of a face key point pasting mask method in the prior art are solved.
In addition, the data augmentation method according to an embodiment of the present invention includes: acquiring a sample data set, wherein the sample data set comprises multi-view face color data and multi-view face depth data; segmenting the multi-view face color data to obtain multi-view mask data; reconstructing the multi-view face color data, the multi-view face depth data and the multi-view mask data by using the face reconstruction network to obtain multi-view color data, multi-view point cloud data and multi-view mask data; fusing the multi-view color data and the multi-view mask data to obtain the color mask template data; and fusing the multi-view point cloud data and the multi-view mask data to obtain the depth mask template data. According to the technical scheme provided by the invention, the multi-view-angle face color data and the face depth data are processed, and the multi-view-angle data are fused into one data, so that the acquired color mask template data and depth mask template data are more complete and more accord with the actual face effect.
In addition, the data augmentation method according to an embodiment of the present invention further includes, before reconstructing the multi-view face color data, the multi-view face depth data, and the multi-view mask data by using the face reconstruction network: and carrying out filtering processing and hole filling processing on the multi-view face depth data. According to the technical scheme provided by the invention, the quality of the face depth data can be improved by preprocessing the face depth data, so that the acquired depth mask template data is more real and complete.
In addition, according to the data amplification method provided by the embodiment of the invention, the face data set to be amplified comprises face color data to be amplified and face depth data to be amplified; processing the color mask template data, the depth mask template data and the face data set to be augmented by using the face reconstruction network to generate an augmented face data set, wherein the augmented face data set comprises: reconstructing the color data of the face to be augmented and the depth data of the face to be augmented by using the face reconstruction network to obtain the color data to be augmented and the depth data to be augmented; fusing the color data to be augmented and the color mask template data to obtain color mask data; fusing the depth data to be augmented and the depth mask template data to obtain depth mask data; and respectively rendering the color mask data and the depth mask data to the color data of the face to be amplified and the depth data of the face to be amplified to generate the amplified face data set, wherein the amplified face data set comprises the color data of the face to be amplified and the depth data of the face to be amplified. According to the technical scheme provided by the invention, the color data to be amplified and the color mask template data, the depth data to be amplified and the depth mask template data are fused respectively, so that the amplification of the amplified human face data obtained by the method is realized.
In addition, the data augmentation method provided by the embodiment of the present invention further includes, before the fusing the color data to be augmented and the color mask template data to obtain the color mask data: carrying out color space conversion processing on the color data to be amplified and the color mask template data to obtain color conversion data and color mask conversion data; performing linear processing on the brightness of the color mask conversion data according to the brightness of the color conversion data; carrying out linear processing on the saturation of the color mask conversion data according to the saturation of the color conversion data; and performing color adding mode conversion processing on the color conversion data and the color mask conversion data after linear processing to obtain the color data to be amplified and linear color mask template data. The technical scheme provided by the invention can ensure that the brightness and the saturation of the color data to be amplified and the color mask template data are kept consistent, so that the amplified face data obtained by the invention is more real.
In addition, the data augmentation method according to the embodiment of the present invention, where the obtaining of the depth mask data by fusing the depth data to be augmented and the depth mask template data includes: performing networking processing on the depth mask template data to obtain boundary point data of the depth mask template data; processing boundary point data of the depth mask template data by using a preset rigid loss function and a preset non-rigid loss function; and fusing the processed boundary point data of the depth mask template data and the depth data to be augmented to obtain the depth mask data. According to the technical scheme provided by the invention, the depth data to be augmented and the depth mask template data are fused more closely by using rigid registration and non-rigid deformation, so that the augmented face data acquired by the method is more real.
In addition, the data augmentation method provided by the embodiment of the present invention further includes, before processing the color mask template data, the depth mask template data, and the face data set to be augmented by using a preset face reconstruction network: acquiring colorful face training data, and performing mask augmentation processing on the colorful face training data to acquire mask face training data; and training the preset face reconstruction network by using the colorful face training data and the mask face training data. According to the technical scheme provided by the invention, the face reconstruction network is trained before the face reconstruction network is used, so that the performance of the obtained face reconstruction network is better, the reconstruction effect on the face color data and the face depth data is better, and the obtained mask template data is more real.
Drawings
One or more embodiments are illustrated by way of example in the accompanying drawings, which correspond to the figures in which like reference numerals refer to similar elements and which are not to scale unless otherwise specified.
FIG. 1 is a first flowchart of a data augmentation method according to an embodiment of the present invention;
FIG. 2 is a second flowchart of a data augmentation method according to an embodiment of the present invention;
FIG. 3 is a flow chart diagram three of a data augmentation method according to an embodiment of the present invention;
FIG. 4 is a fourth flowchart of a data augmentation method according to an embodiment of the present invention;
FIG. 5 is a flow chart diagram of a fifth method of data augmentation according to an embodiment of the present invention;
FIG. 6 is a sixth flowchart of a data augmentation method according to an embodiment of the present invention;
FIG. 7 is a block diagram of a data augmentation device in accordance with an embodiment of the present invention;
fig. 8 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments. The following embodiments are divided for convenience of description, and should not constitute any limitation to the specific implementation manner of the present invention, and the embodiments may be mutually incorporated and referred to without contradiction.
The embodiment of the invention relates to a data augmentation method, as shown in fig. 1, specifically comprising:
Specifically, the acquired face data set to be augmented is of a three-primary-color plus depth type and comprises face color data without a mask and face depth data without the mask, and the acquired mask template data can be acquired from a buffering result by a face reconstruction network or acquired according to an acquired sample data set by the face reconstruction network.
And 102, processing the color mask template data, the depth mask template data and the face data set to be augmented by using a preset face reconstruction network to generate an augmented face data set.
In particular, because the acquired data set of the face to be augmented also comprises the color data of the face to be augmented and the depth data of the face to be augmented, therefore, a face reconstruction network is needed to reconstruct the color data of the face to be augmented and the depth data of the face to be augmented to obtain the color data to be augmented and the depth data to be augmented, then the face reconstruction network fuses the corresponding relationship of each pixel of the color data to be augmented and the color mask template data and the corresponding relationship of each pixel of the depth data to be augmented and the depth mask template data to obtain the color mask data and the depth mask data, finally the face reconstruction network performs rendering processing according to the corresponding relationship of the color mask data and the color data of the face to be augmented and the corresponding relationship of the depth data of the face to be augmented and the depth mask vision to generate mask face data, and the generated augmented face data set also comprises the color data of the face to be augmented and the depth data of the face to be augmented.
It should be noted here that: the face reconstruction network may be a three-dimensional face reconstruction network or a two-dimensional face reconstruction network, and when the face reconstruction network is a three-dimensional face reconstruction network, the acquired mask template data and the augmented face data are both three-dimensional, and when the face reconstruction network is a two-dimensional face reconstruction network, the acquired mask template data and the augmented face data are both two-dimensional.
According to the embodiment of the invention, in the process of face data augmentation, the color mask template data and the depth mask template data are acquired, and then the face reconstruction network is utilized to augment the color mask template data and the depth mask template data to the face data to be augmented to generate augmented face data, so that the acquired augmented face data is real in effect and strong in augmentation performance, and the technical problems of rough and unreal face data and poor augmentation performance of masks caused by the adoption of a face key point pasting mask method in the prior art are solved.
The embodiment of the invention relates to a data augmentation method, as shown in fig. 2, specifically comprising:
Specifically, the acquired sample data set is of a three-primary-color plus depth type and comprises multi-view face color data and multi-view face depth data, the multi-view face color data refers to face color data acquired at different shooting angles, the different shooting angles can be angles such as a front face, a left face, a right face and a 45-degree side face, the multi-view face depth data is the same, the multi-view face color data and the multi-view face depth data are face data of a mask, the type of the mask can be divided into three types of a complete nose shielding type, a half nose shielding type and a chin shielding type, and a group of sample data set can comprise multi-view face color data and multi-view face depth data of various mask wearing types; the face data set to be augmented is also of a three-primary-color plus depth type and comprises face color data without a mask and face depth data without the mask.
It should be noted here that: the face color data and the face depth data are correspondingly present, one face data must simultaneously acquire the face color data and the face depth data, the face color data and the face depth data are aligned in a pixel level, otherwise, the face depth data needs to be aligned to the face color data according to camera parameters, and the mask data on the sample data in one sample data set preferably comprises various types of data of a disposable medical mask, an activated carbon mask, an N95 mask and a cotton cloth mask.
Specifically, after the multi-view face color data is acquired, the face color data of each view needs to be segmented to obtain a mask part on each face color data, and the segmentation can be processed by a special segmentation model or a segmentation network, or can be processed by a face reconstruction network by calling third-party matting software.
And 203, reconstructing the multi-view face color data, the multi-view face depth data and the multi-view mask data by using a preset face reconstruction network to obtain the multi-view color data, the multi-view point cloud data and the multi-view mask data.
Specifically, the multi-view face color data includes front face color data, left face color data, and right face color data; the multi-view face depth data comprises front face depth data, left face depth data and right face depth data; the multi-view mask data includes face mask data, left face mask data, and right face mask data as an example for explanation; inputting the data into a human face reconstruction network for reconstruction processing to obtain face color data, left face color data, right face color data, face point cloud data, left face point cloud data, right face point cloud data, face mask data, left face mask data and right face mask data; the mask data comprises color data, point cloud data and mask data, wherein each pixel value of the color data is obtained according to color texture data of each face area on the face color data, each pixel value of the point cloud data is obtained according to a three-dimensional point cloud coordinate value of the face area on the face depth data, the three-dimensional point cloud coordinate value is obtained by converting the face depth data according to camera parameters, and each pixel value of the mask data indicates whether the mask area is a mask area (can be represented by 0 and 1).
And 204, fusing the multi-view color data and the multi-view mask data to obtain color mask template data. Specifically, color data is obtained by fusing the front face color data, the left face color data, and the right face color data according to the respective visible regions, mask data is obtained by fusing the front face mask data, the left face mask data, and the right face mask data according to the respective visible regions, and then color mask template data is obtained by respectively increasing the color data according to the pixel value indicating whether the mask region is present or not on the mask data.
And step 205, fusing the multi-view point cloud data and the multi-view mask data to obtain depth mask template data. Specifically, rigid transformation from left face point cloud data and right face point cloud data to front face point cloud data is calculated, then coordinate transformation is carried out on three-dimensional coordinates on each pixel of the left face point cloud data and the right face point cloud data, so that the three-dimensional coordinates are in the same coordinate system with the coordinates of the front face point cloud data, finally the front face point cloud data, the left face point cloud data and the right face point cloud data are fused according to respective visible regions to obtain the point cloud data, mask data of a front face mask, mask data of a left face mask and mask data of a right face mask are fused according to respective visible regions to obtain the mask data, pixel values used for representing whether the mask data are mask regions are respectively expanded on the point cloud data, and accordingly depth mask template data are obtained.
It should be noted here that the color mask template data and the depth mask template data may be stored and reused, and when the color mask template data and the depth mask template data are stored, the color data and the mask data are required to be stored as unit8 type data, and the point cloud data and the mask data are required to be stored as unit16 type data after being quantized. The face reconstruction network used by the invention can be a three-dimensional face reconstruction network or a two-dimensional face reconstruction network, and when the three-dimensional face reconstruction network is used, the obtained multi-view color data, multi-view point cloud data and multi-view mask data are all three-dimensional; when the two-dimensional face reconstruction network is used, the acquired multi-view color data, multi-view point cloud data and multi-view mask data are all two-dimensional.
And step 206, processing the color mask template data, the depth mask template data and the face data set to be augmented by using the face reconstruction network to generate an augmented face data set.
Specifically, this step is substantially the same as step 102 in the embodiment of the present invention, and is not described herein again.
According to the embodiment of the invention, on the basis of the beneficial effects brought by other embodiments, the multi-view face color data and the multi-view face depth data can be processed, and the multi-view data is fused into one data, so that the acquired color mask template data and the depth mask template data are more complete, and the actual effect of the face is better met.
The embodiment of the invention relates to a data augmentation method, as shown in fig. 3, specifically comprising:
Specifically, this step is substantially the same as step 301 in the embodiment of the present invention, and is not described herein again.
And step 302, performing filtering processing and hole filling processing on the multi-view face depth data.
Specifically, after the face depth data is acquired, the face depth data may be subjected to smoothing filtering processing by using a filtering algorithm to eliminate some noises on the face depth data, or the face depth data may be subjected to hole filling processing by using a hole filling algorithm to make the acquired face depth data more complete, or in addition, the face depth data may be subjected to other algorithms capable of improving data quality, which is not limited herein.
And step 303, segmenting the multi-view face color data to obtain multi-view mask data.
Specifically, this step is substantially the same as step 202 in the embodiment of the present invention, and is not described herein again.
And 304, reconstructing the multi-view face color data, the multi-view face depth data and the multi-view mask data by using a face reconstruction network to obtain the multi-view color data, the multi-view point cloud data and the multi-view mask data.
Specifically, this step is substantially the same as step 203 in the embodiment of the present invention, and is not described herein again.
And 305, fusing the multi-view color data and the multi-view mask data to obtain color mask template data.
Specifically, this step is substantially the same as step 204 in the embodiment of the present invention, and is not described herein again.
And step 306, fusing the multi-view point cloud data and the multi-view mask data to obtain depth mask template data.
Specifically, this step is substantially the same as step 205 in the embodiment of the present invention, and is not described herein again.
And 307, processing the color mask template data, the depth mask template data and the face data set to be augmented by using a face reconstruction network to generate an augmented face data set.
Specifically, this step is substantially the same as step 102 in the embodiment of the present invention, and is not described herein again.
According to the embodiment of the invention, on the basis of the beneficial effects brought by other embodiments, the quality of the face depth data can be improved by preprocessing the face depth data, so that the depth mask template data acquired by the invention is more real and complete.
The embodiment of the invention relates to a data augmentation method, as shown in fig. 4, specifically comprising:
Specifically, this step is substantially the same as step 101 in the embodiment of the present invention, and is not described herein again.
And 402, reconstructing the color data of the face to be augmented and the depth data of the face to be augmented by using a preset face reconstruction network to obtain the color data to be augmented and the depth data to be augmented.
Specifically, the acquired face data set to be augmented is also of a three-primary-color plus depth type and comprises face color data to be augmented and face depth data to be augmented, and the face color data to be augmented and the face depth data to be augmented are processed through a face reconstruction network to obtain the color data to be augmented and the depth data to be augmented.
And step 403, performing fusion processing on the color data to be augmented and the color mask template data to obtain color mask data, and performing fusion processing on the depth data to be augmented and the depth mask template data to obtain depth mask data.
Specifically, before the fusion processing of the color data to be augmented and the color mask template data, the color space HSV conversion processing needs to be performed on the color data to be augmented and the color mask template data, the original three primary colors RGB space is converted into the hue saturation brightness HSV space, and since the brightness and the saturation of the color conversion data and the color mask conversion data are inconsistent, and the two are fused under the inconsistent condition, the obtained fusion data is "disjointed", so that the brightness of the color conversion data needs to be taken as a standard, and the brightness of the color mask conversion data is linearly processedIn other words, the saturation of the color mask conversion data is required to be subjected to linear processing by taking the saturation of the color conversion data as a standard, so that the brightness and the saturation of the color conversion data and the color mask conversion data are kept consistent; after the brightness and the saturation of the color conversion data and the color mask conversion data are consistent, the color conversion data of hue saturation brightness HSV type and the color mask conversion data need to be subjected to RGB conversion processing in a color adding mode again, so that the original types of the color conversion data and the color mask conversion data are restored, and the fusion processing is convenient to perform; fusing color data to be amplified and color mask template data according to a fusion function to obtain color mask data, wherein the fusion function is UVMAPfuseThe expression of (a) is:
UVMAPfuse＝UVMAPface×(1–Imask)+UVMAPmask×Imask
wherein, UVMAPfuseData on color mask, UVMAPfaceTo be augmented color data, UVMAPmaskRepresenting color mask template data, ImaskMask template mask data.
When the depth mask data is acquired by fusing the depth mask template data and the depth mask template data, rigid transformation and non-rigid transformation are required to be performed on the depth mask template data by using a human face reconstruction network, and then the depth mask template data subjected to transformation and the depth mask template data to be augmented are fused to acquire the depth mask data.
And 404, respectively rendering the color mask data and the depth mask data to the color data of the face to be augmented and the depth data of the face to be augmented to generate an augmented face data set, wherein the augmented face data set comprises the color data of the face and the augmented depth data of the face.
Specifically, a face reconstruction network is used for rendering according to the corresponding relation between the color mask data and the color data of the face to be augmented and the corresponding relation between the depth data of the face to be augmented and the depth mask data of the face to be augmented to generate an augmented face data set, and the generated augmented face data set also comprises the face augmented color data and the face augmented depth data.
According to the embodiment of the invention, on the basis of the beneficial effects brought by other embodiments, the color data, the color mask template data, the depth data and the depth mask template data can be fused respectively, so that the augmentative human face data set obtained by the method is augmented; the brightness and the saturation of the color data to be amplified and the color mask template data are kept consistent, so that the mask face data acquired by the method is more real.
The embodiment of the invention relates to a data augmentation method, as shown in fig. 5, specifically comprising:
Specifically, this step is substantially the same as step 401 in the embodiment of the present invention, and is not described herein again.
And 502, reconstructing the color data of the face to be augmented and the depth data of the face to be augmented by using a preset face reconstruction network to obtain the color data to be augmented and the depth data to be augmented.
Specifically, this step is substantially the same as step 402 in the embodiment of the present invention, and is not described herein again.
And 503, fusing the color data to be augmented and the color mask template data to obtain color mask data.
Specifically, this step is substantially the same as the method for acquiring the color mask data mentioned in step 403 of the embodiment of the present invention, and is not repeated herein.
And step 504, performing networking processing on the depth mask data to obtain boundary point data of the depth mask data.
Specifically, the depth mask template data needs to be gridded to determine grid boundary point data of the depth mask template data, and then the depth data to be augmented and the depth mask template data at the same pixel position can be directly used as target data and source data because the data have semantic information.
And 505, processing the boundary point data of the depth mask template data by using a preset rigid loss function and a preset non-rigid loss function.
Specifically, rigid transformation with specifications is firstly carried out on the depth mask template data by using a face reconstruction network according to a rigid LOSS function, then non-rigid transformation is carried out on the depth mask template data after rigid transformation according to a non-rigid LOSS function, and finally fusion processing is carried out on the depth mask template data after two times of transformation and the depth template data to be amplified, wherein a rigid LOSS function LOSSrigid_transformThe expression of (scale, R, T) is as follows:
LOSSrigid_transform(scale,R,T)＝|scale*T*Vsource+T–Vtarget|2
wherein scale represents scaling, R represents a rotation matrix, T represents a three-dimensional translation, and V representssourcePoint cloud coordinate value, V, of original depth mask template datatargetAnd point cloud coordinate values of the deformed depth mask template data.
LOSS of non-rigidity function LOSSnon_rigid_deformThe expression of (a) is as follows:
LOSSnon_rigid_deform＝LOSSlap(Vdeform)+LOSScorres(Vdeform)
among them, LOSSlap(Vdeform)＝Σi|Laplace(Vdeform)–Laplace(Vori)|2，
Among them, LOSScorres(Vdeform)＝ΣV∈BoundPointSet|Vdeform–Laplace(Vtarget)|2，
Among them, LOSSlap(Vdeform) Showing that the laplace coordinates before and after the deformation of the mask template data are consistent, LOSScorres(Vdeform) Representing that the boundary point coordinates after the mask template data deformation are close to the coordinates on the depth data to be augmented, and Laplace () representing the Laplace seat of the data gridValue, VoriPoint cloud coordinate value V representing data of depth mask template after rigid transformationdeformAnd expressing a point cloud coordinate value of the depth mask template data of non-rigid transformation to be solved, and V belongs to BoundPointSet to express grid boundary point data only considering the depth mask template data.
And step 506, fusing the boundary point data of the processed depth mask template data and the depth data to be augmented to obtain the depth mask data.
Specifically, the depth mask data is obtained by performing fusion processing on each point cloud coordinate value of the boundary point data of the processed depth mask template data and the point cloud coordinate value of the depth patch data to be augmented.
And 507, respectively rendering the color mask data and the depth mask data to the color data of the face to be augmented and the depth data of the face to be augmented to generate an augmented face data set, wherein the augmented face data set comprises the color data of the face and the augmented depth data of the face.
Specifically, this step is substantially the same as step 405 in the embodiment of the present invention, and is not described herein again.
According to the embodiment of the invention, on the basis of the beneficial effects brought by other embodiments, the depth data to be augmented and the depth mask template data can be fused more closely by using rigid registration and non-rigid deformation, so that the augmented face data acquired by the method is more real.
The embodiment of the invention relates to a data augmentation method, as shown in fig. 6, specifically comprising:
Specifically, after color face training data of RGB three primary colors are acquired, a base line Baseline and mask method is adopted to augment the simulation mask on the color face training data, and mask face training data are acquired.
Specifically, color face training data and mask face training data are input into a face reconstruction network, and the face reconstruction network is trained by using a reconstruction LOSS function, wherein the expression of the reconstruction LOSS function LOSS is as follows:
LOSS＝|RecNet(Imasked)–BaselineRecNet(I)|+|RecNet(I)–BaselineRecNet(I)|
wherein I represents color face training data, ImaskedRepresenting mask face training data, RecNet (I), after a simulated mask is added by a Baseline plus mask methodmasked) Representing face reconstruction network to mask face training data ImaskedReconstructing a result; RecNet (I) represents the reconstruction result of the network to be trained on the colorful face data I, BaselineRecNet (I) represents the face reconstruction network on the mask face training data ImaskedAnd (5) reconstructing a result.
Specifically, this step is substantially the same as step 101 in the embodiment of the present invention, and is not described herein again.
And step 604, processing the color mask template data, the depth mask template data and the face data set to be augmented by using a face reconstruction network to generate an augmented face data set.
Specifically, this step is substantially the same as step 102 in the embodiment of the present invention, and is not described herein again.
The embodiment of the invention can train the face reconstruction network before using the face reconstruction network on the basis of the beneficial effects brought by other embodiments, so that the performance of the acquired face reconstruction network is better, the reconstruction effect on the face color data and the face depth data is better, and the acquired mask template data is more real.
The embodiment of the present invention relates to a data amplification device, as shown in fig. 7, specifically including:
the template module 701 is configured to obtain mask template data and a face data set to be augmented, where the mask template data includes color mask template data and depth mask template data.
And the augmentation module 702 is configured to process the color mask template data, the depth mask template data and the face data set to be augmented by using a preset face reconstruction network, so as to generate an augmented face data set.
It should be noted that each module referred to in this embodiment is a logical module, and in practical applications, one logical unit may be one physical unit, may be a part of one physical unit, and may be implemented by a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, elements that are not so closely related to solving the technical problems proposed by the present invention are not introduced in the present embodiment, but this does not indicate that other elements are not present in the present embodiment.
An embodiment of the present invention relates to an electronic device, as shown in fig. 8, including: at least one processor 801; and a memory 802 communicatively coupled to the at least one processor 801; the memory 802 stores instructions executable by the at least one processor 801 to enable the at least one processor 801 to perform any of the data augmentation methods of the present invention.
Where the memory and processor are connected by a bus, the bus may comprise any number of interconnected buses and bridges, the buses connecting together one or more of the various circuits of the processor and the memory. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor is transmitted over a wireless medium via an antenna, which further receives the data and transmits the data to the processor.
The processor is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And the memory may be used to store data used by the processor in performing operations. The present invention relates to a computer-readable storage medium storing a computer program. The computer program realizes the above-described method embodiments when executed by a processor.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.
Claims (9)
1. A method of data augmentation, the method comprising:
acquiring mask template data and a face data set to be augmented, wherein the mask template data comprises color mask template data and depth mask template data;
processing the color mask template data, the depth mask template data and the face data set to be augmented by using a preset face reconstruction network to generate an augmented face data set;
wherein, it includes to acquire gauze mask template data:
acquiring a sample data set, wherein the sample data set comprises multi-view face color data and multi-view face depth data; segmenting the multi-view face color data to obtain multi-view mask data;
reconstructing the multi-view face color data, the multi-view face depth data and the multi-view mask data by using the face reconstruction network to obtain multi-view color data, multi-view point cloud data and multi-view mask data;
fusing the multi-view color data and the multi-view mask data to obtain the color mask template data; and fusing the multi-view point cloud data and the multi-view mask data to obtain the depth mask template data.
2. The data augmentation method of claim 1, wherein the reconstructing the multi-view face color data, the multi-view face depth data, and the multi-view mask data using the face reconstruction network further comprises: and carrying out filtering processing and hole filling processing on the multi-view face depth data.
3. The data augmentation method of claim 1, wherein the face data set to be augmented comprises face color data to be augmented and face depth data to be augmented;
processing the color mask template data, the depth mask template data and the face data set to be augmented by using the face reconstruction network to generate an augmented face data set, wherein the augmented face data set comprises:
reconstructing the color data of the face to be augmented and the depth data of the face to be augmented by using the face reconstruction network to obtain the color data to be augmented and the depth data to be augmented;
fusing the color data to be augmented and the color mask template data to obtain color mask data;
fusing the depth data to be augmented and the depth mask template data to obtain depth mask data;
and respectively rendering the color mask data and the depth mask data to the color data of the face to be amplified and the depth data of the face to be amplified to generate the amplified face data set, wherein the amplified face data set comprises the color data of the face to be amplified and the depth data of the face to be amplified.
4. The data augmentation method according to claim 3, wherein before the fusing the color data to be augmented and the color mask template data to obtain color mask data, the method further comprises:
carrying out color space conversion processing on the color data to be amplified and the color mask template data to obtain color conversion data and color mask conversion data;
performing linear processing on the brightness of the color mask conversion data according to the brightness of the color conversion data;
carrying out linear processing on the saturation of the color mask conversion data according to the saturation of the color conversion data;
and performing color adding mode conversion processing on the color conversion data and the color mask conversion data after linear processing to obtain the color data to be amplified and linear color mask template data.
5. The data augmentation method according to claim 3, wherein the obtaining of the depth mask data by fusing the depth data to be augmented and the depth mask template data comprises:
performing networking processing on the depth mask template data to obtain boundary point data of the depth mask template data;
processing boundary point data of the depth mask template data by using a preset rigid loss function and a preset non-rigid loss function;
and fusing the processed boundary point data of the depth mask template data and the depth data to be augmented to obtain the depth mask data.
6. The data augmentation method according to claim 1, wherein the processing the color mask template data, the depth mask template data and the face data set to be augmented by using a preset face reconstruction network further comprises: acquiring colorful face training data, and performing mask augmentation processing on the colorful face training data to acquire mask face training data; and training the preset face reconstruction network by using the colorful face training data and the mask face training data.
7. A data augmentation apparatus, the apparatus comprising:
the mask template data acquisition module is used for acquiring mask template data and a face data set to be augmented, wherein the mask template data comprises color mask template data and depth mask template data;
the augmentation module is used for processing the color mask template data, the depth mask template data and the face data set to be augmented by using a preset face reconstruction network to generate an augmentation face data set;
wherein, it includes to acquire gauze mask template data:
acquiring a sample data set, wherein the sample data set comprises multi-view face color data and multi-view face depth data; segmenting the multi-view face color data to obtain multi-view mask data;
reconstructing the multi-view face color data, the multi-view face depth data and the multi-view mask data by using the face reconstruction network to obtain multi-view color data, multi-view point cloud data and multi-view mask data;
fusing the multi-view color data and the multi-view mask data to obtain the color mask template data; and fusing the multi-view point cloud data and the multi-view mask data to obtain the depth mask template data.
8. An electronic device, comprising: at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data augmentation method of any one of claims 1 to 6.
9. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the data augmentation method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110658676.XA CN113435273B (en) | 2021-06-15 | 2021-06-15 | Data augmentation method, data augmentation device, electronic device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110658676.XA CN113435273B (en) | 2021-06-15 | 2021-06-15 | Data augmentation method, data augmentation device, electronic device, and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113435273A CN113435273A (en) | 2021-09-24 |
| CN113435273B true CN113435273B (en) | 2022-03-25 |
Family
ID=77755912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110658676.XA Active CN113435273B (en) | 2021-06-15 | 2021-06-15 | Data augmentation method, data augmentation device, electronic device, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113435273B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113837105A (en) * | 2021-09-26 | 2021-12-24 | 北京的卢深视科技有限公司 | Face recognition method, face recognition system, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108776983A (en) * | 2018-05-31 | 2018-11-09 | 北京市商汤科技开发有限公司 | Based on the facial reconstruction method and device, equipment, medium, product for rebuilding network |
| CN108921070A (en) * | 2018-06-22 | 2018-11-30 | 北京旷视科技有限公司 | Image processing method, model training method and corresponding intrument |
| CN111401133A (en) * | 2020-02-19 | 2020-07-10 | 北京三快在线科技有限公司 | Target data augmentation method, device, electronic device and readable storage medium |
| CN111914630A (en) * | 2020-06-19 | 2020-11-10 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for generating training data for face recognition |
| CN111914629A (en) * | 2020-06-19 | 2020-11-10 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for generating training data for face recognition |
| CN111932439A (en) * | 2020-06-28 | 2020-11-13 | 深圳市捷顺科技实业股份有限公司 | Method and related device for generating face image of mask |
| CN112487923A (en) * | 2020-11-25 | 2021-03-12 | 奥比中光科技集团股份有限公司 | Method and system for acquiring training data of human face head posture |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109872379B (en) * | 2017-12-05 | 2022-12-02 | 富士通株式会社 | Data processing apparatus and method |
| CN111832745B (en) * | 2020-06-12 | 2023-08-01 | 北京百度网讯科技有限公司 | Data augmentation method and device and electronic equipment |
| CN111881770B (en) * | 2020-07-06 | 2024-05-31 | 上海序言泽网络科技有限公司 | Face recognition method and system |
| CN112232199A (en) * | 2020-10-15 | 2021-01-15 | 燕山大学 | Wearing mask detection method based on deep learning |
-
2021
- 2021-06-15 CN CN202110658676.XA patent/CN113435273B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108776983A (en) * | 2018-05-31 | 2018-11-09 | 北京市商汤科技开发有限公司 | Based on the facial reconstruction method and device, equipment, medium, product for rebuilding network |
| CN108921070A (en) * | 2018-06-22 | 2018-11-30 | 北京旷视科技有限公司 | Image processing method, model training method and corresponding intrument |
| CN111401133A (en) * | 2020-02-19 | 2020-07-10 | 北京三快在线科技有限公司 | Target data augmentation method, device, electronic device and readable storage medium |
| CN111914630A (en) * | 2020-06-19 | 2020-11-10 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for generating training data for face recognition |
| CN111914629A (en) * | 2020-06-19 | 2020-11-10 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for generating training data for face recognition |
| CN111932439A (en) * | 2020-06-28 | 2020-11-13 | 深圳市捷顺科技实业股份有限公司 | Method and related device for generating face image of mask |
| CN112487923A (en) * | 2020-11-25 | 2021-03-12 | 奥比中光科技集团股份有限公司 | Method and system for acquiring training data of human face head posture |
Non-Patent Citations (2)
| Title |
|---|
| SAR Image Data Enhancement Method Based on EMD;Haoran R.等;《2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA)》;20200901;第1137-1141页 * |
| 基于改进后的SSD人脸口罩佩戴识别方法研究与典型应用;刘子龙 等;《中国安全防范技术与应用》;20210430;第29-34页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113435273A (en) | 2021-09-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3839879B1 (en) | Facial image processing method and apparatus, image device, and storage medium | |
| CN111383232B (en) | Matting method, matting device, terminal equipment and computer readable storage medium | |
| CN108124489B (en) | Information processing method, apparatus, cloud processing device and computer program product | |
| CN111784818A (en) | Method and device for generating three-dimensional human body model and computer readable storage medium | |
| US11450068B2 (en) | Method and device for processing image, and storage medium using 3D model, 2D coordinates, and morphing parameter | |
| Ahmed et al. | Deep event stereo leveraged by event-to-image translation | |
| CN113435273B (en) | Data augmentation method, data augmentation device, electronic device, and storage medium | |
| CN110443839A (en) | A kind of skeleton model spatial registration method and device | |
| CN114972634A (en) | Multi-view three-dimensional deformable human face reconstruction method based on feature voxel fusion | |
| CN117593178A (en) | Virtual fitting method based on feature guidance | |
| CN117274605A (en) | Method and device for extracting water area outline from photo shot by unmanned aerial vehicle | |
| CN112966390A (en) | Method and apparatus for processing clothes based on dual three-dimensional distance fields | |
| CN115222895B (en) | Image generation method, device, equipment and storage medium | |
| CN112464924A (en) | Method and device for constructing training set | |
| CN115393241A (en) | Medical image enhancement method and device, electronic equipment and readable storage medium | |
| CN116342385A (en) | Training method and device for text image super-resolution network and storage medium | |
| CN109859306A (en) | A method of extracting manikin in the slave photo based on machine learning | |
| CN112818820B (en) | Image generation model training method, image generation device and electronic equipment | |
| CN113496468B (en) | Depth image restoration method, device and storage medium | |
| US20220277586A1 (en) | Modeling method, device, and system for three-dimensional head model, and storage medium | |
| CN113781653A (en) | Object model generation method and device, electronic equipment and storage medium | |
| CN114581389A (en) | Point cloud quality analysis method based on three-dimensional edge similarity characteristics | |
| JP2023502813A (en) | Model training method, image processing and registration method, apparatus, device, medium | |
| CN113781372A (en) | Deep learning-based opera facial makeup generation method and system | |
| CN208101077U (en) | A kind of 3D printing system based on VR |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230411 Address after: 230091 room 611-217, R & D center building, China (Hefei) international intelligent voice Industrial Park, 3333 Xiyou Road, high tech Zone, Hefei, Anhui Province Patentee after: Hefei lushenshi Technology Co.,Ltd. Address before: 100083 room 3032, North B, bungalow, building 2, A5 Xueyuan Road, Haidian District, Beijing Patentee before: BEIJING DILUSENSE TECHNOLOGY CO.,LTD. Patentee before: Hefei lushenshi Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right |