CN111681162A - Defect sample generation method and device, electronic equipment and storage medium - Google Patents
Defect sample generation method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111681162A CN111681162A CN202010526870.8A CN202010526870A CN111681162A CN 111681162 A CN111681162 A CN 111681162A CN 202010526870 A CN202010526870 A CN 202010526870A CN 111681162 A CN111681162 A CN 111681162A
- Authority
- CN
- China
- Prior art keywords
- image
- defect
- workpiece
- processed
- style migration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000007547 defect Effects 0.000 title claims abstract description 220
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000003860 storage Methods 0.000 title claims abstract description 10
- 230000005012 migration Effects 0.000 claims abstract description 91
- 238000013508 migration Methods 0.000 claims abstract description 91
- 230000002950 deficient Effects 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000012549 training Methods 0.000 claims description 31
- 230000006870 function Effects 0.000 claims description 24
- 230000004913 activation Effects 0.000 claims description 17
- 230000008859 change Effects 0.000 claims description 9
- 239000000463 material Substances 0.000 abstract description 7
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 9
- 238000004519 manufacturing process Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 5
- 238000013473 artificial intelligence Methods 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- XEEYBQQBJWHFJM-UHFFFAOYSA-N Iron Chemical compound [Fe] XEEYBQQBJWHFJM-UHFFFAOYSA-N 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 230000001788 irregular Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000006748 scratching Methods 0.000 description 2
- 230000002393 scratching effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 229910000831 Steel Inorganic materials 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 229910052742 iron Inorganic materials 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000000053 physical method Methods 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The application provides a defect sample generation method, a defect sample generation device, electronic equipment and a storage medium. The method comprises the following steps: acquiring an image of a workpiece to be processed and an image of the workpiece with defects; the image of the defective workpiece includes a first defective region; randomly determining a second defect area in the image of the workpiece to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting a new workpiece image to be processed into the style migration model for style migration processing to obtain a target image output by the style migration model; including the generated defects in the target image. According to the method and the device, the second defect area is randomly determined in the image of the workpiece to be processed to simulate the defect, and the position, the shape and the size of the second defect area are random, so that more kinds of defects can be obtained, and then the authenticity of the simulated defect is improved by utilizing the histogram matching and the style migration method. The defects are not required to be manually manufactured on the workpiece, and manpower and material resources are greatly reduced.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a defect sample generation method and apparatus, an electronic device, and a storage medium.
Background
With the rapid development of artificial intelligence technology, the artificial intelligence technology represented by deep learning has penetrated into the aspects of society, taking manufacturing industry as an example, and the surface defect detection technology based on the artificial intelligence technology is greatly improving the production efficiency and the production quality of the manufacturing industry, so that the manufacturing industry in China can improve the product quality competitiveness while reducing the production cost, and promote the gradual change of China from the large manufacturing country to the strong manufacturing country.
When the artificial intelligence technology represented by the neural network is used for surface defect quality inspection, a large number of workpiece samples with defects need to be collected generally, and the neural network can be well fitted and generalized through learning a large number of workpieces with defects, so that the intelligent defect detection function is completed, and the artificial completion of appearance quality inspection is replaced.
But generally defective workpieces are very limited. Therefore, a large number of workpieces with defects need to be imitated, so that the requirement of a neural network algorithm is met, and finally, a network model with the performance reaching the standard can be obtained for production and use.
At present, the method for simulating the defects is a manual counterfeiting method, and the manual counterfeiting is to simulate and generate a sample with the defects by using a tool and a physical method. However, this method consumes a lot of manpower and material resources, and the number of the counterfeited defects is limited.
Disclosure of Invention
An object of the embodiments of the present application is to provide a defect sample generation method, apparatus, electronic device, and storage medium, so as to solve the problems that a large amount of manpower and material resources are consumed for counterfeit defects and the number of counterfeit defects is limited in the prior art.
In a first aspect, an embodiment of the present application provides a defect sample generation method, including: acquiring an image of a workpiece to be processed and an image of the workpiece with defects; wherein the defective workpiece image includes a first defective region; randomly determining a second defect area in the workpiece image to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defect is included in the target image.
According to the embodiment of the application, the second defect area is randomly determined in the workpiece image to be processed to simulate the defects, and the positions, the shapes and the sizes of the second defect area are random, so that more kinds of defects can be obtained, then the color and the gray level of the defect area to be simulated are consistent with those of real defects in a histogram matching mode, the authenticity of the simulated defects is improved, and the texture of the simulated defect area in the workpiece image to be processed is consistent with that of a background area by using a style migration method. The defects are not required to be manually manufactured on the workpiece, and manpower and material resources are greatly reduced.
Further, the randomly determining a second defect area in the workpiece image to be processed includes: randomly determining a point from the workpiece image to be processed as a reference point; randomly selecting one defect type from preset defect types as a target defect type; converting the defects corresponding to the target defect type to obtain the converted defects; and generating the second defect area on the workpiece image to be processed by taking the reference point as the central point of the transformed defect.
The embodiment of the application randomly determines the second defect area in the image of the workpiece to be processed to simulate the defect, and the position, the shape and the size of the second defect area are random, so that the simulated defect is more diversified.
Further, the histogram matching the first defect area and the second defect area to obtain a new workpiece image to be processed includes: obtaining a new workpiece image to be processed according to a function y ← S [ L' ] + histmtch (S [ L ], H [ M ]); wherein y is the new workpiece image to be processed, S is the workpiece image to be processed, L is the second defective region, L' is, H is the workpiece image with a defect, and M is the first defective region.
The embodiment of the application ensures that the defect formed on the second defect area keeps consistent with the real defect in color and gray scale through histogram matching, thereby improving the authenticity of the imitated defect.
Further, before inputting the new workpiece image to be processed into the style migration model for style migration processing, the method further comprises: acquiring a plurality of training samples, wherein the training samples comprise original workpiece images and noise images obtained after noise processing is carried out on the original workpiece images; inputting the noise image into a style migration model to be trained to obtain a predicted image output by the style migration model to be trained; and optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image to obtain the trained style migration model.
According to the embodiment of the application, the style migration model is trained, the obtained style migration model can process the workpiece image to be processed, so that the texture of the imitated defect area in the processed workpiece image to be processed is consistent with that of the background area, and the authenticity of the defect sample obtained by imitation is improved.
Further, the obtaining a plurality of training samples comprises: acquiring at least one original workpiece image, performing scratch removal on any area on each original workpiece image, and adding random noise into the scratched area to obtain a plurality of noise images corresponding to the original workpiece images.
Further, the optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image includes: calculating according to the predicted image and the original workpiece image to obtain corresponding content loss, style loss, histogram loss and full-image change loss; and optimizing parameters in the style migration model to be trained according to the content loss, the style loss, the histogram loss and the full-map change loss.
According to the method and the device, the style migration model is optimized through content loss, style loss, histogram loss and full-map change loss from multiple factors, and therefore the obtained style migration model can improve the authenticity of a defect sample obtained through imitation.
Further, the style migration model comprises a first convolution unit, a residual convolution module, a second convolution unit and an up-sampling module which are connected in sequence; wherein the first convolution unit and the second convolution unit each include a plurality of convolution modules; the residual convolution module is used for splicing the feature map output by each convolution module in the first convolution unit and the feature map output by the residual convolution module and inputting the spliced feature maps into the second convolution unit; the upsampling module includes a tanh activation function.
According to the embodiment of the application, the combination of different perception visual field characteristics before and after the detection is ensured through the residual convolution module, and the performance of the convolution network is enhanced.
In a second aspect, an embodiment of the present application provides a defect sample generating apparatus, including: the image obtaining module is used for obtaining an image of a workpiece to be processed and an image of the workpiece with the defect; wherein the defective workpiece image includes a first defective region; the area determining module is used for randomly determining a second defect area in the workpiece image to be processed; the histogram matching module is used for performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; the style migration module is used for inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defect is included in the target image.
In a third aspect, an embodiment of the present application provides an electronic device, including: the system comprises a processor, a memory and a bus, wherein the processor and the memory are communicated with each other through the bus; the memory stores program instructions executable by the processor, the processor being capable of performing the method of the first aspect when invoked by the program instructions.
In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, including: the non-transitory computer readable storage medium stores computer instructions that cause the computer to perform the method of the first aspect.
Additional features and advantages of the present application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a schematic flow chart of a defect sample generation method according to an embodiment of the present disclosure;
FIG. 2 provides another defect sample generation flow diagram for an embodiment of the present application;
FIG. 3 is a diagram of a style migration model architecture provided in an embodiment of the present application;
FIG. 4 is a schematic diagram of a style migration module training process according to an embodiment of the present application;
FIG. 5 is a schematic structural diagram of an apparatus according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
It can be understood that the style migration model training method and the defect sample generation method provided by the embodiment of the present application may be applied to a terminal device (also referred to as an electronic device) and a server; the terminal device may be a smart phone, a tablet computer, a Personal Digital Assistant (PDA), or the like; the server may specifically be an application server, and may also be a Web server. In addition, both the model training method and the defect generation method can be executed by the same terminal device, and can also be executed by different terminal devices.
For convenience of understanding, in the technical solution provided in the embodiment of the present application, an application scenario of the model training method and the defect generating method provided in the embodiment of the present application is described below by taking a terminal device as an execution subject.
Fig. 1 is a schematic flow chart of a defect sample generation method provided in an embodiment of the present application, and as shown in fig. 1, the method includes:
step 101: acquiring an image of a workpiece to be processed and an image of the workpiece with defects; wherein the defective workpiece image includes a first defective region.
The size of the workpiece image to be processed can be the same as or different from that of the workpiece image with the defect. The image of the workpiece with the defect can be obtained by image acquisition of a real workpiece with the defect, or can be generated by the defect generation method. The image of the workpiece with the defect can be in color or in gray scale, and if the image is in color, the image can be subjected to gray scale conversion and converted into a gray scale image. When the image of the workpiece with the defects is selected, the first defect area can be a circle, an irregular polygon, a gap and the like, and the position and the size of the first defect area in the workpiece are not limited. The image of the workpiece to be processed can be obtained by image acquisition of the workpiece without defects, or can be a workpiece image obtained from a network.
Step 102: and randomly determining a second defect area in the workpiece image to be processed.
Wherein, an area can be randomly selected as the second defect area in the workpiece image to be processed according to a preset algorithm. It should be noted that the random determination means that the position, size, and shape of the second defective region in the image of the workpiece to be processed are random. The size and shape of the second defective region may be different from those of the first defective region, and may be the same.
Step 103: and performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed.
Histogram matching, also called histogram normalization, is to transform the image histogram using the histogram of a standard image as a standard so that the histograms of both images are identical and approximate, thereby making both images have similar hue and contrast. The principle of histogram matching is to equalize both histograms to become the same normalized uniform histogram, and to perform the inverse operation of equalization on the reference image using the uniform histogram as a medium.
The embodiment of the application performs histogram matching on the gray histogram of the second defect area according to the gray histogram of the first defect area, and the process is as follows: and respectively calculating the gray level histograms of the workpiece image to be processed and the workpiece image with the defect, then respectively calculating the accumulation results on the gray level histograms, and changing the distribution of the gray level histogram of the second defect area to be as close as possible to the distribution of the gray level histogram of the first defect area, thereby obtaining a new second defect area. And scratching the second defect area in the workpiece image to be processed, and filling the new second defect area in the original second defect area of the workpiece image to be processed to obtain a new workpiece image to be processed. The histogram matching can be expressed by the following functions:
y←S[L']+histmatch(S[L],H[M])
y is the new workpiece image to be processed, S is the workpiece image to be processed, L is the second defective region, L' is, H is the workpiece image with defects, and M is the first defective region.
Step 104: inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defect is included in the target image.
The style migration model is obtained by pre-training, and the specific structure and training process thereof are described in the following embodiments. And inputting a new workpiece image to be processed into the style migration model, processing the input new workpiece image to be processed by the style migration model, and outputting a target image, wherein the defect part in the output target image can be kept consistent with the original workpiece image to be processed without defects on the whole, so that the imitated defect area can be naturally fused into the background pattern. The style migration model may be represented by the following function:
y'←S[L']+TransferNet(y)[L]
and y 'is the new workpiece image to be processed, S is the workpiece image to be processed, L is the second defect area, and L' is the second defect area.
FIG. 2 is a flow chart of another defect sample generation method according to an embodiment of the present application, as shown in FIG. 2, a defect-free sample (workpiece image to be processed) is obtained first, and a region, i.e., a second defect region, is randomly set on the defect-free sample. And performing histogram matching on the second defect area and the real defects on the workpiece image with the defects to obtain a new workpiece image to be processed. And inputting the new workpiece image to be processed into a style migration model, and outputting a target image by the style migration model so as to obtain a generated defect sample, wherein the texture of the defect in the new workpiece image to be processed is consistent with the texture of other areas.
According to the embodiment of the application, the second defect area is randomly determined in the workpiece image to be processed to simulate the defects, and the positions, the shapes and the sizes of the second defect area are random, so that more kinds of defects can be obtained, then the color and the gray level of the defect area to be simulated are consistent with those of real defects in a histogram matching mode, the authenticity of the simulated defects is improved, and the texture of the simulated defect area in the workpiece image to be processed is consistent with that of a background area by using a style migration method. The defects are not required to be manually manufactured on the workpiece, and manpower and material resources are greatly reduced.
On the basis of the above embodiment, the randomly determining a second defect area in the workpiece image to be processed includes:
randomly determining a point from the workpiece image to be processed as a reference point;
randomly selecting one defect type from preset defect types as a target defect type;
converting the defects corresponding to the target defect type to obtain the converted defects;
and generating the second defect area on the workpiece image to be processed by taking the reference point as the central point of the transformed defect.
In a specific implementation process, the terminal device stores defects of multiple defect types in advance, for example: circular, irregular polygonal, slit-type, etc. When the second defect area is randomly determined from the workpiece image to be processed, a point on the workpiece image to be processed can be randomly found as a reference point, which is also the center point of the second defect area. Then, a defect type is determined from the plurality of defect types as a target defect type, and the defects of the target defect type are transformed, wherein the transformation can be stretching, shrinking, rotating, rippling and the like, and the transformed defects are obtained.
And after the transformed defect is obtained, generating a second defect area on the workpiece to be processed by taking the reference point as the central point of the transformed defect.
In another implementation process, the terminal device may further directly determine a target defect type, set the defect corresponding to the target defect type at any position of the workpiece to be processed, and then perform transformation operations such as scaling, rotating, rippling and the like on the defect, so as to obtain an image of the workpiece to be processed with the second defect region.
The embodiment of the application randomly determines the second defect area in the image of the workpiece to be processed to simulate the defect, and the position, the shape and the size of the second defect area are random, so that the simulated defect is more diversified.
On the basis of the embodiment, the style migration model comprises a first convolution unit, a residual convolution module, a second convolution unit and an up-sampling module which are connected in sequence; wherein the first convolution unit and the second convolution unit each include a plurality of convolution modules;
the residual convolution module is used for splicing the feature map output by each convolution module in the first convolution unit and the feature map output by the residual convolution module and inputting the spliced feature maps into the second convolution unit;
the upsampling module includes a tanh activation function.
In a specific implementation process, fig. 3 is a structure diagram of a style migration model provided in an embodiment of the present application, and as shown in fig. 3, the first convolution unit may include three convolution modules, which are a first convolution module, a second convolution module, and a third convolution module, respectively.
The first convolution module comprises a convolution layer with an input channel of 3, an output channel of 32, a convolution kernel of 9 and a step length of 1, an example regularization layer and an additional activation layer with a Relu function as an activation function; the first convolution module can output a first profile.
The second convolution module comprises a convolution layer with an input channel of 32, an output channel of 64, a convolution kernel of 3 and a step length of 2, an example regularization layer and an additional activation layer with a Relu function as an activation function; the second convolution module can output a second signature.
The third convolution module comprises a convolution layer with an input channel of 64, an output channel of 128, a convolution kernel of 3 and a step length of 2, an example regularization layer and an additional activation layer with a Relu function as an activation function; the third convolution module can output a third signature.
The input channel of the residual convolution module is 4, the output channels are 128, the convolution kernel is 3, the step length is 1, and the output is the fourth feature map.
The second convolution unit may include three convolution modules, i.e., a fourth convolution module, a fifth convolution module, and a sixth convolution module.
The fourth convolution module comprises a convolution layer with an input channel of 256, an output channel of 128, a convolution kernel of 1 and a step length of 1, an up-sampling convolution layer with an input channel of 128, an output channel of 64 and a step length of 2 times of 1, an example regularization layer and an activation layer with a Relu function as an activation function.
The fifth convolution module includes a convolution layer with 128 input channels, 64 output channels, 1 convolution kernel and 1 step size, an upsampled convolution layer with 2 times of 1 input channel, 64 output channel and 32 step size, an example regularization layer, and an additional activation layer with a Relu function as an activation function.
The sixth convolution module includes a convolution layer with an input channel of 64, an output channel of 32, a convolution kernel of 1 and a step size of 1, a convolution layer with an input channel of 64, an output channel of 32, a step size of 1 and a convolution kernel of 9, an example regularization layer, and an activation layer with a Relu function as an activation function.
The upsampling module includes a tanh activation function, the output of which is the target image.
In the process of processing the input image by using the style migration model, the data input into the fourth convolution module is formed by splicing the first feature map, the second feature map, the third feature map and the fourth feature map. The combination of different perception visual field characteristics before and after can be ensured, and the performance of the convolution network is enhanced.
On the basis of the foregoing embodiment, fig. 4 is a schematic diagram of a style migration module training flow provided in the embodiment of the present application, and as shown in fig. 4, the style migration module training flow includes:
step 401: the method comprises the steps of obtaining a plurality of training samples, wherein the training samples comprise original workpiece images and noise images obtained after noise processing is carried out on the original workpiece images.
The training sample comprises an original workpiece image and a noise image obtained after the original image is subjected to noise processing. The noise image is the input of the model, and the original workpiece image is the label. The original workpiece image can be obtained by image acquisition of a real defect-free workpiece, and can also be obtained from the network. The type of the workpiece may be the same according to the type of the workpiece actually generating the defect, and may be, for example, a steel plate, an iron plate, or the like, or may be a workpiece made of another material or having another shape.
The noise image may be obtained by:
a region is randomly determined in the original workpiece image, and it can be understood that the determination manner of the region may be consistent with the determination manner of the second defective region in the above embodiment, and details are not described here. After the area is determined, the area is subjected to matting processing, and random noise is added into the matting area of the original workpiece image to obtain a noise image. It will be appreciated that one raw workpiece image may generate multiple noisy images, for example: the method can remove regions with different shapes and sizes in the original workpiece image and add the same noise; regions with different shapes and sizes can be scratched out from the original workpiece image, and different types of noise can be added; the method can also be used for scratching the regions with different shapes and sizes in the original workpiece image and adding different types of noise.
Step 402: and inputting the noise image into a style migration model to be trained, and obtaining a predicted image output by the style migration model to be trained.
Step 403: and optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image to obtain the trained style migration model.
In a specific implementation process, model losses corresponding to the style transition model are calculated according to the predicted image output by the style transition model and the corresponding original workpiece image, the model losses include content losses, style losses, histogram losses and full-image change losses, and the total loss can be obtained through each loss value and the corresponding weight. And optimizing parameters in the style migration model by a back propagation method by utilizing the total loss.
It should be noted that the training of the style migration model may be implemented by multiple iterations, and in addition, when it is determined whether the model training satisfies the training end condition, the first model may be verified by using a test sample, where an obtaining manner of the test sample is consistent with that of the training sample, and the test sample also includes an original workpiece image and a noise image. The first model is a model obtained by performing one round of training optimization on the style migration model by using a plurality of training samples; specifically, the terminal device inputs a touch mode in a test sample into a first model, and processes an input noise image by using the first model to obtain a corresponding predicted image; and then, calculating accuracy according to the original workpiece image in the test sample and the predicted image output by the first model, and when the prediction accuracy is greater than a preset threshold, determining that the model performance of the first model can meet the requirement, and at the moment, generating a style migration model according to the parameters and the model structure of the first model.
It should be understood that the preset threshold may be set according to actual situations, and the preset threshold is not specifically limited in the embodiment of the present application.
In addition, when judging whether the style migration model meets the training end condition, the iteration times can be preset, and when the iteration times in the training process reach the preset iteration times, the training can be stopped.
It should be understood that the number of iterations may be set according to historical experience, which is not particularly limited in the embodiments of the present application.
According to the embodiment of the application, the style migration model is trained, the obtained style migration model can process the workpiece image to be processed, so that the texture of the imitated defect area in the processed workpiece image to be processed is consistent with that of the background area, and the authenticity of the defect sample obtained by imitation is improved.
Fig. 5 is a schematic structural diagram of an apparatus provided in an embodiment of the present application, where the apparatus may be a module, a program segment, or code on an electronic device. It should be understood that the apparatus corresponds to the above-mentioned embodiment of the method of fig. 1, and can perform various steps related to the embodiment of the method of fig. 1, and the specific functions of the apparatus can be referred to the description above, and the detailed description is appropriately omitted here to avoid redundancy. The device includes: an image acquisition module 501, a region determination module 502, a histogram matching module 503, and a lattice migration module 504, wherein:
the image obtaining module 501 is used for obtaining an image of a workpiece to be processed and an image of the workpiece with a defect; wherein the defective workpiece image includes a first defective region; the region determining module 502 is configured to randomly determine a second defect region in the workpiece image to be processed; the histogram matching module 503 is configured to perform histogram matching on the first defect region and the second defect region to obtain a new workpiece image to be processed; the style migration module 504 is configured to input the new workpiece image to be processed into a style migration model for style migration processing, so as to obtain a target image output by the style migration model; wherein the generated defect is included in the target image.
On the basis of the foregoing embodiment, the area determining module 502 is specifically configured to:
randomly determining a point from the workpiece image to be processed as a reference point;
randomly selecting one defect type from preset defect types as a target defect type;
converting the defects corresponding to the target defect type to obtain the converted defects;
and generating the second defect area on the workpiece image to be processed by taking the reference point as the central point of the transformed defect.
On the basis of the foregoing embodiment, the histogram matching module 503 is specifically configured to:
obtaining a new workpiece image to be processed according to a function y ← S [ L' ] + histmtch (S [ L ], H [ M ]);
wherein y is the new workpiece image to be processed, S is the workpiece image to be processed, L is the second defective region, L' is, H is the workpiece image with a defect, and M is the first defective region.
On the basis of the above embodiment, the apparatus further includes a training module configured to:
acquiring a plurality of training samples, wherein the training samples comprise original workpiece images and noise images obtained after noise processing is carried out on the original workpiece images;
inputting the noise image into a style migration model to be trained to obtain a predicted image output by the style migration model to be trained;
and optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image to obtain the trained style migration model.
On the basis of the above embodiment, the training module is specifically configured to:
and acquiring at least one original workpiece image, performing scratch removal on any area on each original workpiece image, and adding random noise into the deducted area to obtain a plurality of noise images corresponding to the original workpiece images.
On the basis of the above embodiment, the training module is specifically configured to:
calculating according to the predicted image and the original workpiece image to obtain corresponding content loss, style loss, histogram loss and full-image change loss;
and optimizing parameters in the style migration model to be trained according to the content loss, the style loss, the histogram loss and the full-map change loss.
On the basis of the embodiment, the style migration model comprises a first convolution unit, a residual convolution module, a second convolution unit and an up-sampling module which are connected in sequence; wherein the first convolution unit and the second convolution unit each include a plurality of convolution modules;
the residual convolution module is used for splicing the feature map output by each convolution module in the first convolution unit and the feature map output by the residual convolution module and inputting the spliced feature maps into the second convolution unit;
the upsampling module includes a tanh activation function.
In summary, the embodiment of the present application randomly determines the second defect area in the workpiece image to be processed to simulate the defect, and the position, shape and size of the second defect area are random, so that a wider variety of defects can be obtained, and then the color and gray scale of the defect area to be simulated are consistent with those of the real defect in a histogram matching manner, so as to improve the reality of the simulated defect, and then the texture of the simulated defect area in the workpiece image to be processed is consistent with that of the background area by using the style migration method. The defects are not required to be manually manufactured on the workpiece, and manpower and material resources are greatly reduced.
Fig. 6 is a schematic structural diagram of an entity of an electronic device provided in an embodiment of the present application, and as shown in fig. 6, the electronic device includes: a processor (processor)601, a memory (memory)602, and a bus 603; wherein the content of the first and second substances,
the processor 601 and the memory 602 communicate with each other through the bus 603;
the processor 601 is configured to call program instructions in the memory 602 to perform the methods provided by the above-mentioned method embodiments, for example, including: acquiring an image of a workpiece to be processed and an image of the workpiece with defects; wherein the defective workpiece image includes a first defective region; randomly determining a second defect area in the workpiece image to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defect is included in the target image.
The processor 601 may be an integrated circuit chip having signal processing capabilities. The processor 601 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. Which may implement or perform the various methods, steps, and logic blocks disclosed in the embodiments of the present application. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The Memory 602 may include, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Read Only Memory (EPROM), an Electrically Erasable Read Only Memory (EEPROM), and the like.
The present embodiment discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method provided by the above-mentioned method embodiments, for example, comprising: acquiring an image of a workpiece to be processed and an image of the workpiece with defects; wherein the defective workpiece image includes a first defective region; randomly determining a second defect area in the workpiece image to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defect is included in the target image.
The present embodiments provide a non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the methods provided by the above method embodiments, for example, including: acquiring an image of a workpiece to be processed and an image of the workpiece with defects; wherein the defective workpiece image includes a first defective region; randomly determining a second defect area in the workpiece image to be processed; performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed; inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defect is included in the target image.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
Furthermore, the functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A defect sample generation method, comprising:
acquiring an image of a workpiece to be processed and an image of the workpiece with defects; wherein the defective workpiece image includes a first defective region;
randomly determining a second defect area in the workpiece image to be processed;
performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed;
inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defect is included in the target image.
2. The method of claim 1, wherein randomly determining a second defect region in the image of the workpiece to be processed comprises:
randomly determining a point from the workpiece image to be processed as a reference point;
randomly selecting one defect type from preset defect types as a target defect type;
converting the defects corresponding to the target defect type to obtain converted defects;
and generating the second defect area on the workpiece image to be processed by taking the reference point as the central point of the transformed defect.
3. The method of claim 1, wherein histogram matching the first defect region with the second defect region to obtain a new workpiece image to be processed comprises:
obtaining a new workpiece image to be processed according to a function y ← S [ L' ] + histmtch (S [ L ], H [ M ]);
wherein y is the new workpiece image to be processed, S is the workpiece image to be processed, L is the second defective region, L' is, H is the workpiece image with a defect, and M is the first defective region.
4. The method of claim 1, wherein prior to inputting the new workpiece image to be processed into a style migration model for style migration processing, the method further comprises:
acquiring a plurality of training samples, wherein the training samples comprise original workpiece images and noise images obtained after noise processing is carried out on the original workpiece images;
inputting the noise image into a style migration model to be trained to obtain a predicted image output by the style migration model to be trained;
and optimizing parameters in the style migration model to be trained according to the predicted image and the original workpiece image to obtain the trained style migration model.
5. The method of claim 4, wherein the obtaining the plurality of training samples comprises:
and acquiring at least one original workpiece image, performing scratch removal on any area on each original workpiece image, and adding random noise into the deducted area to obtain a plurality of noise images corresponding to the original workpiece images.
6. The method according to claim 4, wherein the optimizing parameters in the style migration model to be trained according to the predicted image and the raw workpiece image comprises:
calculating according to the predicted image and the original workpiece image to obtain corresponding content loss, style loss, histogram loss and full-image change loss;
and optimizing parameters in the style migration model to be trained according to the content loss, the style loss, the histogram loss and the full-map change loss.
7. The method according to any one of claims 1 to 6, wherein the style migration model comprises a first convolution unit, a residual convolution module, a second convolution unit and an upsampling module which are connected in sequence; wherein the first convolution unit and the second convolution unit each include a plurality of convolution modules;
the residual convolution module is used for splicing the feature map output by each convolution module in the first convolution unit and the feature map output by the residual convolution module and inputting the spliced feature maps into the second convolution unit;
the upsampling module includes a tanh activation function.
8. A defect sample generation apparatus, comprising:
the image obtaining module is used for obtaining an image of a workpiece to be processed and an image of the workpiece with the defect; wherein the defective workpiece image includes a first defective region;
the area determining module is used for randomly determining a second defect area in the workpiece image to be processed;
the histogram matching module is used for performing histogram matching on the first defect area and the second defect area to obtain a new workpiece image to be processed;
the style migration module is used for inputting the new workpiece image to be processed into a style migration model for style migration processing to obtain a target image output by the style migration model; wherein the generated defect is included in the target image.
9. An electronic device, comprising: a processor, a memory, and a bus, wherein,
the processor and the memory are communicated with each other through the bus;
the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of any one of claims 1-7.
10. A non-transitory computer-readable storage medium storing computer instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010526870.8A CN111681162B (en) | 2020-06-09 | 2020-06-09 | Defect sample generation method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010526870.8A CN111681162B (en) | 2020-06-09 | 2020-06-09 | Defect sample generation method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111681162A true CN111681162A (en) | 2020-09-18 |
| CN111681162B CN111681162B (en) | 2023-09-01 |
Family
ID=72435297
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010526870.8A Active CN111681162B (en) | 2020-06-09 | 2020-06-09 | Defect sample generation method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111681162B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111862096A (en) * | 2020-09-23 | 2020-10-30 | 平安科技（深圳）有限公司 | Image segmentation method and device, electronic equipment and storage medium |
| CN112884758A (en) * | 2021-03-12 | 2021-06-01 | 国网四川省电力公司电力科学研究院 | Defective insulator sample generation method and system based on style migration method |
| CN113808011A (en) * | 2021-09-30 | 2021-12-17 | 深圳万兴软件有限公司 | Feature fusion based style migration method and device and related components thereof |
| CN114332086A (en) * | 2022-03-14 | 2022-04-12 | 启东市固德防水布有限公司 | Textile defect detection method and system based on style migration and artificial intelligence |
| CN114445309A (en) * | 2020-10-20 | 2022-05-06 | 斗山重工业建设有限公司 | Defect image generation method for depth learning and system for defect image generation method for depth learning |
| CN115861312A (en) * | 2023-02-24 | 2023-03-28 | 季华实验室 | OLED dry film defect detection method based on style migration positive sample generation |
| CN116030038A (en) * | 2023-02-23 | 2023-04-28 | 季华实验室 | Unsupervised OLED defect detection method based on defect generation |
| CN116091873A (en) * | 2023-04-10 | 2023-05-09 | 宁德时代新能源科技股份有限公司 | Image generation method, device, electronic equipment and storage medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2043593A1 (en) * | 1990-06-12 | 1991-12-13 | Henry Spalding Baird | Methods and apparatus for image analysis |
| CN104091327A (en) * | 2014-06-19 | 2014-10-08 | 华南理工大学 | Method and system for generating dendritic shrinkage porosity defect simulation image of casting |
| US20170139572A1 (en) * | 2015-11-17 | 2017-05-18 | Adobe Systems Incorporated | Image Color and Tone Style Transfer |
| CN107358636A (en) * | 2017-06-16 | 2017-11-17 | 华南理工大学 | A kind of rarefaction defect image generating method based on textures synthesis |
| CN108447054A (en) * | 2018-03-22 | 2018-08-24 | 北京木业邦科技有限公司 | Defects in timber sample acquiring method, device, electronic equipment and storage medium |
| CN109190620A (en) * | 2018-09-03 | 2019-01-11 | 苏州科达科技股份有限公司 | License plate sample generating method, system, equipment and storage medium |
| CN109800697A (en) * | 2019-01-09 | 2019-05-24 | 国网浙江省电力有限公司舟山供电公司 | Transformer target detection and open defect recognition methods based on VGG-net Style Transfer |
| CN110675359A (en) * | 2019-06-29 | 2020-01-10 | 创新奇智(南京)科技有限公司 | Defect sample generation method and system for steel coil surface and electronic equipment |
| CN110689477A (en) * | 2019-09-07 | 2020-01-14 | 创新奇智(重庆)科技有限公司 | Universal flaw image simulation method |
| CN110706308A (en) * | 2019-09-07 | 2020-01-17 | 创新奇智(成都)科技有限公司 | GAN-based steel coil end face edge loss artificial sample generation method |
| WO2020048242A1 (en) * | 2018-09-04 | 2020-03-12 | 阿里巴巴集团控股有限公司 | Method and apparatus for generating vehicle damage image based on gan network |
| CN111127309A (en) * | 2019-12-12 | 2020-05-08 | 杭州格像科技有限公司 | Portrait style transfer model training method, portrait style transfer method and device |
-
2020
- 2020-06-09 CN CN202010526870.8A patent/CN111681162B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2043593A1 (en) * | 1990-06-12 | 1991-12-13 | Henry Spalding Baird | Methods and apparatus for image analysis |
| CN104091327A (en) * | 2014-06-19 | 2014-10-08 | 华南理工大学 | Method and system for generating dendritic shrinkage porosity defect simulation image of casting |
| US20170139572A1 (en) * | 2015-11-17 | 2017-05-18 | Adobe Systems Incorporated | Image Color and Tone Style Transfer |
| CN107358636A (en) * | 2017-06-16 | 2017-11-17 | 华南理工大学 | A kind of rarefaction defect image generating method based on textures synthesis |
| CN108447054A (en) * | 2018-03-22 | 2018-08-24 | 北京木业邦科技有限公司 | Defects in timber sample acquiring method, device, electronic equipment and storage medium |
| CN109190620A (en) * | 2018-09-03 | 2019-01-11 | 苏州科达科技股份有限公司 | License plate sample generating method, system, equipment and storage medium |
| WO2020048242A1 (en) * | 2018-09-04 | 2020-03-12 | 阿里巴巴集团控股有限公司 | Method and apparatus for generating vehicle damage image based on gan network |
| CN109800697A (en) * | 2019-01-09 | 2019-05-24 | 国网浙江省电力有限公司舟山供电公司 | Transformer target detection and open defect recognition methods based on VGG-net Style Transfer |
| CN110675359A (en) * | 2019-06-29 | 2020-01-10 | 创新奇智(南京)科技有限公司 | Defect sample generation method and system for steel coil surface and electronic equipment |
| CN110689477A (en) * | 2019-09-07 | 2020-01-14 | 创新奇智(重庆)科技有限公司 | Universal flaw image simulation method |
| CN110706308A (en) * | 2019-09-07 | 2020-01-17 | 创新奇智(成都)科技有限公司 | GAN-based steel coil end face edge loss artificial sample generation method |
| CN111127309A (en) * | 2019-12-12 | 2020-05-08 | 杭州格像科技有限公司 | Portrait style transfer model training method, portrait style transfer method and device |
Non-Patent Citations (2)
| Title |
|---|
| 位一鸣;童力;罗麟;杨珊;: "基于卷积神经网络的主变压器外观缺陷检测方法", 浙江电力, no. 04 * |
| 林嵩: "面向表面缺陷检测的小样本机器学习方法研究", 中国优秀硕士学位论文全文数据库 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111862096A (en) * | 2020-09-23 | 2020-10-30 | 平安科技（深圳）有限公司 | Image segmentation method and device, electronic equipment and storage medium |
| CN111862096B (en) * | 2020-09-23 | 2021-06-18 | 平安科技（深圳）有限公司 | Image segmentation method and device, electronic equipment and storage medium |
| CN114445309A (en) * | 2020-10-20 | 2022-05-06 | 斗山重工业建设有限公司 | Defect image generation method for depth learning and system for defect image generation method for depth learning |
| CN112884758A (en) * | 2021-03-12 | 2021-06-01 | 国网四川省电力公司电力科学研究院 | Defective insulator sample generation method and system based on style migration method |
| CN113808011A (en) * | 2021-09-30 | 2021-12-17 | 深圳万兴软件有限公司 | Feature fusion based style migration method and device and related components thereof |
| CN113808011B (en) * | 2021-09-30 | 2023-08-11 | 深圳万兴软件有限公司 | Style migration method and device based on feature fusion and related components thereof |
| CN114332086A (en) * | 2022-03-14 | 2022-04-12 | 启东市固德防水布有限公司 | Textile defect detection method and system based on style migration and artificial intelligence |
| CN114332086B (en) * | 2022-03-14 | 2022-05-13 | 启东市固德防水布有限公司 | Textile defect detection method and system based on style migration and artificial intelligence |
| CN116030038A (en) * | 2023-02-23 | 2023-04-28 | 季华实验室 | Unsupervised OLED defect detection method based on defect generation |
| CN115861312A (en) * | 2023-02-24 | 2023-03-28 | 季华实验室 | OLED dry film defect detection method based on style migration positive sample generation |
| CN116091873A (en) * | 2023-04-10 | 2023-05-09 | 宁德时代新能源科技股份有限公司 | Image generation method, device, electronic equipment and storage medium |
| CN116091873B (en) * | 2023-04-10 | 2023-11-28 | 宁德时代新能源科技股份有限公司 | Image generation method, device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111681162B (en) | 2023-09-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111681162B (en) | Defect sample generation method and device, electronic equipment and storage medium | |
| CN106875381B (en) | Mobile phone shell defect detection method based on deep learning | |
| CN107169956B (en) | Color woven fabric defect detection method based on convolutional neural network | |
| CN106023098B (en) | Image mending method based on the more dictionary learnings of tensor structure and sparse coding | |
| CN114528950B (en) | Destroying method and system for identifying type of confidential medium based on three-dimensional point cloud | |
| CN115984662B (en) | Multi-mode data pre-training and identifying method, device, equipment and medium | |
| CN111127454A (en) | Method and system for generating industrial defect sample based on deep learning | |
| CN114581646A (en) | Text recognition method and device, electronic equipment and storage medium | |
| CN111325728B (en) | Product defect detection method, device, equipment and storage medium | |
| CN115761225A (en) | Image annotation method based on neural network interpretability | |
| CN117197763A (en) | Road crack detection method and system based on cross attention guide feature alignment network | |
| CN114359269A (en) | Virtual food box defect generation method and system based on neural network | |
| CN112800851B (en) | Water body contour automatic extraction method and system based on full convolution neuron network | |
| CN117011274A (en) | Automatic glass bottle detection system and method thereof | |
| He et al. | Unsupervised defect inspection algorithm based on cascaded GAN with edge repair feature fusion | |
| CN115587989B (en) | Workpiece CT image defect detection segmentation method and system | |
| CN117076997A (en) | User electricity larceny detection method and system | |
| CN115631301A (en) | Soil-rock mixture image three-dimensional reconstruction method based on improved full convolution neural network | |
| CN115063679A (en) | Pavement quality assessment method based on deep learning | |
| CN108198173A (en) | A kind of online test method, device and the terminal device in distress in concrete region | |
| CN113901947A (en) | Intelligent identification method for tire surface flaws under small sample | |
| CN113052798A (en) | Screen aging detection model training method and screen aging detection method | |
| CN116109627B (en) | Defect detection method, device and medium based on migration learning and small sample learning | |
| CN112070137B (en) | Training data set generation method, target object detection method and related equipment | |
| CN117495711B (en) | Image mark removing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |