CN113033656B - Interactive hole detection data expansion method based on generation countermeasure network - Google Patents

Interactive hole detection data expansion method based on generation countermeasure network Download PDF

Info

Publication number
CN113033656B
CN113033656B CN202110311132.6A CN202110311132A CN113033656B CN 113033656 B CN113033656 B CN 113033656B CN 202110311132 A CN202110311132 A CN 202110311132A CN 113033656 B CN113033656 B CN 113033656B
Authority
CN
China
Prior art keywords
network
defect
hole detection
image
shape
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110311132.6A
Other languages
Chinese (zh)
Other versions
CN113033656A (en
Inventor
王洪建
黄睿
薛明宏
张宁
段博坤
邢艳
彭洪健
陈望
马孝汶
叶清池
陈宇竹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Civil Aviation University of China
Xiamen Airlines Co Ltd
Original Assignee
Civil Aviation University of China
Xiamen Airlines Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Civil Aviation University of China, Xiamen Airlines Co Ltd filed Critical Civil Aviation University of China
Priority to CN202110311132.6A priority Critical patent/CN113033656B/en
Publication of CN113033656A publication Critical patent/CN113033656A/en
Application granted granted Critical
Publication of CN113033656B publication Critical patent/CN113033656B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an interactive hole detection data expansion method based on a generated countermeasure network, which comprises the following steps: classifying the defect images according to defect types, storing the defect images in corresponding folders, constructing an aeroengine hole detection image data set, and constructing and generating an countermeasure network model structure based on depth convolution; training to generate an countermeasure network model, and acquiring one or more engine hole detection defect image generators for generating different defects; constructing a P network, inputting shape information in a training sample, encoding the shape information into hidden vectors identified by an engine hole detection defect image generator, and training the P network by using the characteristics of a fourth convolution layer of an AlexNet model; based on the hidden vector and the engine hole detection defect image generator, acquiring a generated image with a specified shape, and testing through a trained P network; and constructing an interactive basic frame, dividing defects by using an adaptive threshold value, and correcting edge marks of a fusion area by using a poisson fusion algorithm.

Description

Interactive hole detection data expansion method based on generation countermeasure network
Technical Field
The invention relates to the field of data expansion, in particular to an interactive hole detection data expansion method based on a generated countermeasure network.
Background
Data expansion (Data Augmentation) is one of the most practical methods to enhance model performance without increasing computational cost during the deep neural network training phase. Data extensions can be divided into two categories: the method is to use a traditional data transformation method, for example, a random clipping algorithm introduced in the document [1] can make the scale information of a target to be detected more insensitive to a certain extent, thereby improving the recognition effect of the network on small objects; the random erasing algorithm introduced in the document [2] can achieve the purpose of similar random clipping; the geometric transformation algorithm comprises classical data expansion algorithms such as translation, scaling, affine transformation, perspective transformation and the like, and is one of the most popular data expansion methods; the color space conversion rule introduced in the document [3] can simulate different illumination and color temperature environments in the forms of color dithering and the like, so that the robustness of the model to the image colors is improved; the neural style conversion rule described in document [4] can change the style, texture, and other characteristics of an image. The reasonable use of the methods can achieve the purposes of rapidly expanding the data set and enhancing the robustness of the model to a certain extent.
However, for the target detection task, these methods only change the expression form of the image, and do not increase the proportion of the target to be detected in the sample space, and cannot solve the problem of uneven distribution of the target to be detected in the data set. Another type of data expansion method is therefore required: a data resampling method. The data resampling method is characterized in that before or during model training, the sampling frequency of the categories with more samples is reduced, and the sampling frequency of the categories with less samples is increased, so that the number of the samples of the categories is maintained at a relatively balanced level. The document [5] improves the accuracy of the face detection model by a bidirectional resampling technique. Current data resampling methods focus mainly on the synthesis of new instances. For example, in the image mixing method proposed in document [6], discrete sample space is continuously changed by interpolation, so that smoothness in the neighborhood is improved; the feature space expansion method proposed in document [7] obtains more reasonable synthesized data by applying data transformation to a feature layer; document [8], document [9] then generate some more realistic samples for deep learning tasks by using the method of GAN (generation of antagonism network) to generate images. These methods can generate new extended images, but the newly generated extended images have no pixel-level labels, can only be manually annotated again, or are used only for classification tasks of the images.
The pixel-level tag information is important in the object detection task. Document [10] proposes a method for performing random synthesis by using a scene image and a target instance with a tag, but experiments find that if context information of a scene is ignored, whether the target instance can appear at a specific position in the scene is not considered, and only the target instance and the image are randomly combined, so that the aim of improving the detection precision cannot be achieved, and even the performance of a model may be reduced.
Reference to the literature
[1]Liu W,Anguelov D,Erhan D,et al.SSD:Single shot multibox detector[C]//European conference on computer vision.Springer,Cham,2016:21-37.
[2]Zhong Z,Zheng L,Kang G,et al.Random Erasing Data Augmentation[C]//AAAI.2020:13001-13008.
[3]Shorten C,Khoshgoftaar T M.A survey on image data augmentation for deep learning[J].Journal of Big Data,2019,6(1):60.
[4]Jing Y,Yang Y,Feng Z,et al.Neural style transfer:A review[J].IEEE transactions on visualization and computer graphics,2019.
[5] Li Wenhui, jiang Yuanyuan, wang Ying, etc. A face recognition algorithm based on resampling bi-directional 2DLDA fusion [ J ]. Electronic journal 2011,39 (11): 2526-2533.
[6]Zhang H,Cisse M,Dauphin Y N,et al.mixup:Beyond Empirical Risk Minimization[C]//International Conference on Learning Representations.2018.
[7]DeVries T,Taylor G W.Dataset augmentation in feature space[J].arXiv preprint arXiv:1702.05538,2017.
[8]Antoniou A,Storkey A,Edwards H.Data augmentation generative adversarial networks[J].arXiv preprint arXiv:1711.04340,2017.
[9]Wang Y X,Girshick R,Hebert M,et al.Low-shot learning from imaginary data[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.2018:7278-7286.
[10]Dvornik N,Mairal J,Schmid C.Modeling visual context is key to augmenting object detection datasets[C]//Proceedings of the European Conference on Computer Vision(ECCV).2018:364-380.
[11]Steiner B,Devito Z,Chintala S,et al.PyTorch:An Imperative Style,High-Performance Deep Learning Library[C].neural information processing systems,2019:8026-8037.
[12]Deng J,Dong W,Socher R,et al.ImageNet:A large-scale hierarchical image database[C].computer vision and pattern recognition,2009:248-255.
[13]Krizhevsky A,Sutskever I,Hinton G E,et al.ImageNet Classification with Deep Convolutional Neural Networks[C].neural information processing systems,2012:1097-1105.
[14]Radford A,Metz L,Chintala S,et al.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[C].international conference on learning representations,2016.
[15]Canny J.A Computational Approach to Edge Detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1986,8(6):679-698.
[16]Perez P,Gangnet M,Blake A,et al.Poisson image editing[C].international conference on computer graphics and interactive techniques,2003,22(3):313-318.
[17]Land E H,Mccann J J.Lightness and Retinex Theory[J].Journal of the Optical Society of America,1971,61(1):1-11.
Disclosure of Invention
The invention provides an interactive hole detection data expansion method based on a generated countermeasure network, which provides construction and training of the generated countermeasure network and a P network, and adopts an interactive interface to select the type, shape and position of a generated defect by a user; encoding the shape information into hidden vectors using the trained P-network model; generating defects of a specified shape by generating an antagonistic network generator decoding; dividing defects in the generated image by adopting a mode of manually adjusting a threshold value; the fusion edges are modified using a poisson fusion algorithm, as described in detail below:
an interactive hole detection data expansion method based on generation of an antagonism network, the method comprising:
classifying the defect images according to defect types, storing the defect images in corresponding folders, constructing an aeroengine hole detection image data set, and constructing and generating an countermeasure network model structure based on depth convolution;
training to generate an countermeasure network model, and acquiring one or more engine hole detection defect image generators for generating different defects;
constructing a P network, inputting shape information in a training sample, encoding the shape information into hidden vectors identified by an engine hole detection defect image generator, and training the P network by using the characteristics of a fourth convolution layer of an AlexNet model;
based on the hidden vector, an engine hole detection defect image generator, acquiring a generated image with a specified shape, and testing through a trained P network;
and constructing an interactive basic frame, dividing defects by using an adaptive threshold value, and correcting edge marks of a fusion area by using a poisson fusion algorithm.
The defect types are divided into four types of cracks, ablation, abrasion and coating loss, and models are respectively trained and generated for each type of defect.
Further, the P network includes 5 convolution layers, where the first convolution layer is configured to convolve a 64×64×3 picture into a 32×32×128 tensor; the second convolution layer is configured to convolve a tensor of 32×32×128 with a tensor of 16×16×256, the third convolution layer is configured to convolve a tensor of 16×16×256 with a tensor of 8×8×512, and the fourth convolution layer is configured to convolve a tensor of 8×8×512 with a tensor of 4×4×1024; the last convolution layer is used to convolve the 4 x 1024 tensors into a 100-dimensional vector.
The shape information in the input training sample is encoded into hidden vectors recognized by an engine hole detection defect image generator, and the training of the P network by using the characteristics of the fourth convolution layer of the AlexNet model is specifically as follows:
extracting shape information of the training sample, and constructing an image pair according to the training sample and the corresponding shape information; inputting shape information into P i A network encoded into hidden vectors;
the hidden vector is decoded by an engine hole defect detection image generator to generate a defect image; loading an AlexNet model trained based on an Imagenet data set, and extracting the characteristics of a conv4 layer from the generated defect image; extracting conv4 layer characteristics from training samples corresponding to the shape information;
calculating the minimum mean square error of two conv4 layer features as P i A loss function of the network model; shape information as P i And the input of the network outputs the hidden vector as the input of the generator to obtain a generated image with a specified shape, and an objective function is obtained.
Further, the objective function is specifically:
wherein C is the conv4 layer of AlexNetSign, G i Generator, z i As the hidden vector, the vector is a vector of the hidden vector,for training samples, θ Pi Is P i Parameters to be updated in the network.
The test by using the trained P network specifically comprises the following steps:
input shape I to be specified shape Inputting trained P i Generating hidden vectors z in a network i To hidden vector z i A fine noise disturbance N (z) is added,
D fake ＝G i (P i (I shape )+N(z))
wherein D is fake To generate a defect image, P i (I shape ) Representation of the utilization P i Network predicted input shape I shape N (z) represents a fine noise conforming to a gaussian distribution.
Further, the steps of constructing an interactive basic framework and using an adaptive threshold segmentation defect are specifically as follows:
selecting a corresponding defect generator, acquiring strokes input by a user in real time, and extracting the strokes as input shapes; through P i Network encodes input strokes into z i Reuse of G i Network pair z i Decoding to generate a defect image D fake ；
And then D is carried out fake The method is fused to the strokes drawn by the user, so as to generate specified type defects which are similar to the strokes in shape and the same in position; by aligning z i Generating a plurality of defect images which are similar in shape and have fine gaps by using different noise disturbance;
and changing the low threshold tau in Canny in real time, and adjusting the segmentation results of different defect images by setting the high threshold.
Wherein the method further comprises: and constructing an interactive interface based on the Pyqt5 framework.
The technical scheme provided by the invention has the beneficial effects that:
1. according to the invention, an countermeasure network is constructed and generated, a hole detection image data set is used as a training sample, and the distribution of aeroengine defect images is learned, so that an engine hole detection defect image generator which can generate specified types of defects and has realistic effects is obtained;
2. the invention constructs a P network model for encoding the shape information of the input image into hidden vectors; decoding the hidden vector by using an engine hole detection defect image generator to generate a defect similar to the shape of the input image;
3. the invention carries out tiny noise disturbance on the hidden vector obtained by the P network model coding, and can obtain a plurality of defect images with similar shapes and different details;
4. the method adopts a self-adaptive threshold value mode to segment the generated defects, adopts a poisson fusion algorithm to correct the edges of the fusion areas, generates a new engine image without fusion marks, ensures that the defect generation position accords with a defect generation mechanism, can automatically generate marking information, and ensures that the generated engine data set has excellent performance through professional detection;
5. the invention designs the interactive data expansion module, combines the respective advantages of the generation countermeasure network, the P network and the Poisson fusion algorithm, and ensures that the generated image meets the requirement of project development.
Drawings
Fig. 1 is a schematic diagram of a training flow of a P-network model according to the present invention;
FIG. 2 is a schematic diagram of a generating countermeasure network used in the present invention;
FIG. 3 is a diagram showing a structure of a P network model according to the present invention;
FIG. 4 is a schematic flow chart of an interactive data expansion module according to the present invention;
FIG. 5 is a schematic diagram of an interactive interface according to the present invention;
FIG. 6 is a schematic diagram of the segmentation result and the expanded image thereof under different thresholds obtained by adjusting the segmentation threshold in real time;
fig. 7 is a comparison chart of an aeroengine hole detection image generated by the method provided by the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in further detail below.
The method provides an interactive hole detection data expansion method based on the generation of an countermeasure network aiming at the defect detection problem in the hole detection image of the aeroengine, and the target to be detected can be generated on the background of the hole detection image of the aeroengine and is more similar to a real image.
Example 1
An interactive hole detection data expansion method based on generation of an antagonism network, see fig. 1 and 4, comprises the following steps:
1. constructing an aeroengine hole detection image training set, and training to generate an countermeasure network
The method specifically comprises the following steps: reading an engine hole detection image with marking information, cutting out defect information from the engine hole detection image, respectively storing the defect information into a defect image only containing the defect information and a defect image containing background information (namely 2 defect types), classifying the two types of defect images according to the defect types again, and storing the defect images under different folders to construct an aeroengine hole detection image training set.
In the embodiment of the invention, the defects of the aeroengine are divided into four types of cracks, ablation, abrasion and coating loss, and the model is independently trained and generated for each type of defects. When the crack generation model is trained, considering the characteristic of larger aspect ratio of the crack defects, cutting the longer side by taking the shorter side as a reference, cutting the long side into a plurality of square images, and manually screening to reserve 550 crack defect images as a training set. If the length and width of the rest defects are relatively close, the consistency of the length and width is ensured by directly adopting a scaling mode, 902 images of ablation, 206 images of abrasion and 1712 images of coating loss are reserved after screening. The embodiments of the present invention will be described by taking the above numerical values as examples, and the embodiments of the present invention are not limited to these specific implementations.
In the embodiment of the invention, the training flow of the generated countermeasure network is referred to a training GAN part in fig. 1, the generated countermeasure network model structure adopts deep convolution proposed in reference [14] to generate a countermeasure network model DCGAN, and the model structure comprises a generation network part and a discrimination network part. The generating network comprises 5 deconvolution layers, a first deconvolution layer for deconvolving the 100-dimensional hidden vector into a tensor of 4 x 1024, a second deconvolution layer for deconvolving the tensor of 4 x 1024 into a tensor of 8 x 512, the third deconvolution layer is used for deconvolving the tensor of 8 x 512 into the tensor of 16 x 256, the fourth deconvolution layer is used for deconvolving the tensor of 16 x 256 into the tensor of 32 x 128, the last deconvolution layer was used to deconvolve a 32 x 128 tensor to a 64 x 3 picture. The generating network may generate the hidden vector of 100 dimensions into a 64×64×3 picture through the deconvolution layer.
The method comprises the steps of judging a network structure, wherein the network structure is opposite to a generated network structure, a model structure of the network structure comprises 5 convolution layers, and a first convolution layer is used for convolving a 64 x 3 picture into a 32 x 128 tensor; the second convolution layer is used to convolve a tensor of 32 x 128 to a tensor of 16 x 256, the third convolution layer is used to convolve a tensor of 16 x 256 to a tensor of 8 x 512, the fourth convolution layer is configured to convolve the tensor of 8×8×512 into a tensor of 4×4×1024; the last convolution layer is used to convolve the 4 x 1024 tensors into a 2-dimensional vector. The discrimination network may compress an input picture with a size of 64×64×3 into a 2-dimensional vector through a convolution layer, and use a first dimension in the 2-dimensional vector to represent a probability that the input picture is a real picture, and a second dimension to represent a probability that the input picture is a false picture (a generated picture of the generation network). The network structure of DCGAN is shown in fig. 2.
Embodiments of the invention use a Pytorch framework [11] And adjusting training parameters, and using the engine hole detection image training set as a training sample to learn the distribution of the aeroengine defect images. And respectively training 1000 epochs for the four defect data sets (each epoch is used for completely training all training data once) to obtain 4 engine hole detection defect image generators which can generate specified types of defects (namely cracks, ablations, abrasion and coating loss) and have realistic effects.
2. Building and training P-networks
In the embodiment of the invention, the training flow of the P network is shown in fig. 1. Wherein the structure of the P network portion includes 5 convolution layers, the first convolution layer is configured to convolve a 64×64×3 picture into a 32×32×128 tensor; the second convolution layer is used to convolve a tensor of 32 x 128 to a tensor of 16 x 256, the third convolution layer is used to convolve a tensor of 16 x 256 to a tensor of 8 x 512, the fourth convolution layer is configured to convolve the tensor of 8×8×512 into a tensor of 4×4×1024; the last convolution layer is used to convolve the 4 x 1024 tensors into a 100-dimensional vector. The P network structure is shown in fig. 3. Before training the P network, training the P network to generate an countermeasure network is completed to obtain one or more engine hole detection defect image generators G which can generate different defects and have realistic effects i 。
Engine hole detection defect image generator G i After training is completed, the shape information of the training sample in the first step is extractedAnd according to training sample->And shape information corresponding to training samples +.>Constructing an image pair->Since the generator G is needed in the P network training process i Participating, different generators may train out different P networks, thus P i Network and generator G i One-to-one correspondence; shape information +.>Input P i Network encoded into hidden vector z i 。
Formalized representation is as follows:
in the method, in the process of the invention,is P i Parameters to be updated in the network, < >>Shape information for training samples.
Wherein the hidden vector z i Image generator G for detecting defects through engine holes i Generating a defect image G after decoding i (z); loading is based on Imagenet dataset [12] Trained AlexNet [13] The model is used for extracting the characteristics of the conv4 layer from the generated defect image; and then to the shape informationCorresponding training sample->Conv4 layer features are also extracted; calculating L2 loss (least mean square error) of two conv4 layer features as P i Loss function of network model. Shape information->As P i Input to the network, which outputs the hidden vector z i As generator G i Can obtain a generated image of a specified shape.
The objective function is as follows:
wherein C is the conv4 layer feature of AlexNet. Training 4 different P's using 4 different defect generators, respectively i Network, each P i After 1000 epochs are trained on the network, the shape information can be accurately encoded into hidden vectors on the training set corresponding to the defects.
3. Testing using trained P-networks
Referring to fig. 4, part of the P-network test flow in the embodiment of the present invention will specify the input shape I shape Inputting trained P i Generating hidden vectors z in a network i To hidden vector z i Adding fine noise disturbance N (z) achieves the aim that each input can generate defect images with similar shapes and different details.
The process can be formalized as: d (D) fake ＝G i (P i (I shape )+N(z)) (3)
Wherein D is fake To generate a defect image, P i (I shape ) Representation of the utilization P i Network predicted input shape I shape Is represented by a hidden vector of (a). N (z) represents a fine noise conforming to a Gaussian distribution.
4. Constructing an interactive base framework and partitioning defects using adaptive thresholds
In the embodiment of the invention, referring to fig. 4 for the interactive flow, referring to fig. 5 for the interactive interface, before the interactive program is run, the defect image generator G should be ensured first i Shape information coding model P i Training is completed.
Adding event functions to the interactive interface, acquiring the type of the defect which the user wants to generate in real time, and selecting a corresponding defect generator G i The method comprises the steps of carrying out a first treatment on the surface of the The interactive interface allows the user to draw strokes, and can acquire the strokes input by the user in real time and extract the strokes as input shapes I shape (i.e., a specified shape entered by the user through the interactive interface); clicking on the update button after drawing is completed can be done by P i Network encodes input strokes into z i Reuse of G i Network pair z i Decoding to generate a defect image D fake The method comprises the steps of carrying out a first treatment on the surface of the And then D is carried out fake The method is fused to the strokes drawn by the user, so as to generate specified type defects which are similar to the strokes in shape and the same in position; at the same time by to z i Different defect images with similar shapes and fine gaps are generated by using different noise disturbance, are displayed on the right side of the main interface, and can be moved to the main interface for viewing by clicking a corresponding small window on the right side.
Canny 1986 can be used for the generated defect imageAnnual proposed Canny edge detection algorithm [15] Performing edge extraction on the generated defect image; then connecting the closer edges through morphological operation, and searching the maximum connected domain of the edge information as a true value of the corresponding defect outline in the marking information of the subsequent pixel level; the Canny edge detection algorithm has two thresholds, namely a high threshold and a low threshold, important information can be missed due to the fact that the set threshold is too high, the threshold is too low, the branch information can be seen to be important, and a general threshold applicable to all images is difficult to give, so that a user can change the low threshold tau in the Canny algorithm in real time, and the high threshold is set to be 3 tau, the segmentation results of images with different defects are adjusted, and a proper segmentation contour D is obtained mask 。D mask ＝Canny(D fake ,τ) (4)
Finally, clicking the save button will save the current main interface image and automatically generate a matching annotation information file for the new sample.
5. Correction of fusion area edge marks using poisson fusion algorithm
The poisson fusion algorithm is Perez et al (2003) [16] The proposed method utilizes gradient information to perform natural interpolation on the boundary of the region to be fused. The method uses psychologist Land (1971) in paper [17 ]]The principle proposed in (a) is that the gradient of gradual change in the image is smoothed by the restriction of Laplace operator, so as to reduce the trace left by fusing two images.
Poisson fusion algorithm makes background image I new At the edge of the region to be fused according to the defect image D fake Gradient information generation and background image I new Similar pixels to achieve a smoothing effect; d (D) fake Is defined by the edge region of (a)By the profile information D in the fourth step mask Obtained.
The process may be formalized as:
wherein the method comprises the steps of，I new In order to obtain a new image after the fusion,representing edge regions to fused imagesA poisson fusion algorithm is performed, (x, y) being the center coordinates of the user input stroke.
6. Network training and testing
Based on Pytorch deep learning network framework, training by using the countermeasure network model and the data set generated in the first step, and obtaining an engine hole detection defect image generator with vivid generation effect; and training by using the P network model and the data set introduced in the second step to obtain the P network model. The P-network model, in combination with the engine hole defect image generator, may generate a defect image of a specified shape.
Constructing an interactive interface based on a Pyqt5 framework (the framework is an open source library in a python development environment and is used for constructing a visual interface, which is well known in the art), wherein the interactive interface is shown in fig. 5 and comprises: the system comprises a main interface interaction area, a small window area, a defect type selection area, a segmentation threshold adjustment area and a functional area;
displaying a background image through a main interface interaction area, and acquiring stroke information input by a user; the small window area is used for displaying other generated defect expansion results except the main interface; the defect type selection area is used for selecting the defect type which the user wants to generate; the segmentation threshold adjustment area is used for manually adjusting a segmentation threshold and segmenting the generated defects to obtain pixel-level labeling information; the function area selects different background images by clicking the buttons of ' last and ' next ', the strokes input by a user in the main interface area are converted into corresponding defect information by clicking the button of ' update ', the main interface expansion image is stored under a designated folder by clicking the button of ' save ', and the annotation information of the expansion image is generated at the same time.
The user selects the defect to be generated through the interactive interface; outlining the shape of the defect to be generated at the position of the defect to be generated by using a mouse; encoding the stroke shape input P network into hidden vectors; inputting the hidden vector into an engine hole detection defect image generator for decoding to generate a defect with a specified shape; the segmentation threshold is adjusted by the segmentation threshold adjustment area to obtain a more accurate defect segmentation result, and the segmentation result pairs with different thresholds are shown in fig. 6; and fusing the generated defect image with the background image, correcting the segmented defect boundary by using a poisson fusion algorithm, and reducing fusion marks to obtain the engine hole detection image with the new defect. The generated image obtained by the embodiment of the invention is shown in fig. 7.
In summary, the embodiment of the invention uses the interactive data expansion method based on the generation of the countermeasure network, and encodes the input shape information by constructing and training the P network to obtain the hidden vector; decoding by an engine hole defect detection image generator to generate a defect image with a specified shape; and the defect image is segmented by using a method for adjusting the segmentation threshold value, and the problem of obvious image fusion edge trace is solved by using a poisson fusion technology, so that various requirements in practical application are met.
Example 2
The feasibility of the solution in example 1 is verified in conjunction with fig. 6 and 7, as described in detail below:
building and training a generated countermeasure network model and a P network model according to the network structure and the training flow shown in fig. 1, fig. 2 and fig. 3; according to FIG. 4, the process and interface shown in FIG. 5 builds an interactive frame to obtain the defect type, defect position and defect shape information selected by the user in real time; encoding the shape information through a P network model, and then generating a defect image with a specified shape through generating model decoding; adjusting the segmentation result of the defect image by manually adjusting the segmentation threshold; and fusing the generated defect image into the background image, and performing poisson correction on the edge area of the fused defect image to eliminate the fusion trace.
In the interactive interface of fig. 5, the main interface is a background image, and a user can draw strokes on the main interface; the first small window in the five small windows on the right side represents an unexpanded original image, and the remaining four small windows represent a plurality of expanded defect images with similar stroke shapes but different details; a defect type selection area is arranged below the defect type selection area in sequence and is used for selecting the defect type to be generated; the segmentation threshold adjustment area is used for adjusting the segmentation threshold of the generated defect, and the segmentation threshold adjustment area has a value of 0-255; the storage button is used for storing the expanded image of the main interface and automatically generating the annotation information of the expanded image; an update button for updating the strokes drawn by the user to defects of the specified type; the last button and the next button select other hole detection images under the current folder.
From fig. 6, it can be seen that in the embodiment of the present invention, different segmentation results can be obtained on the generated defect image by manually adjusting the segmentation threshold, and a relatively accurate segmentation result can be obtained by manual fine tuning, so as to provide a reference for automatically generating annotation information.
From fig. 7, it can be seen that the data expansion result obtained by the embodiment of the present invention has no obvious fusion trace, and the sample availability is high. In fig. 7, the first column is an original image, the second column and the fourth column are images generated by the embodiment of the present invention, and the third column and the fifth column are labeling information automatically generated by the embodiment. As can be seen from the experimental comparison result of FIG. 7, compared with the existing method for expanding the data of the whole image, the method provided by the embodiment of the invention can increase the detection information on the fixed background to generate a new training sample, and the boundary of the fusion area of the new training sample is smoother and more natural.
The embodiment of the invention does not limit the types of other devices except the types of the devices, so long as the devices can complete the functions.
Those skilled in the art will appreciate that the drawings are schematic representations of only one preferred embodiment, and that the above-described embodiment numbers are merely for illustration purposes and do not represent advantages or disadvantages of the embodiments.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims (7)

1. An interactive hole detection data expansion method based on generation of an antagonism network, which is characterized in that the method comprises the following steps:
classifying the defect images according to defect types, storing the defect images in corresponding folders, constructing an aeroengine hole detection image data set, and constructing and generating an countermeasure network model structure based on depth convolution;
training to generate an countermeasure network model, and acquiring one or more engine hole detection defect image generators for generating different defects;
constructing a P network, inputting shape information in a training sample, encoding the shape information into hidden vectors identified by an engine hole detection defect image generator, and training the P network by using the characteristics of a fourth convolution layer of an AlexNet model;
based on the hidden vector and the engine hole detection defect image generator, acquiring a generated image with a specified shape, and testing through a trained P network;
constructing an interactive basic frame, dividing defects by using a self-adaptive threshold value, and correcting edge marks of a fusion area by using a poisson fusion algorithm;
the shape information in the input training sample is encoded into hidden vectors recognized by an engine hole detection defect image generator, and the training of the P network by using the characteristics of the fourth convolution layer of the AlexNet model is specifically as follows:
extracting shape information of the training sample, and constructing an image pair according to the training sample and the corresponding shape information; inputting shape information into P i A network encoded into hidden vectors;
the hidden vector is decoded by an engine hole defect detection image generator to generate a defect image; loading an AlexNet model trained based on an Imagenet data set, and extracting the characteristics of a conv4 layer from the generated defect image; extracting conv4 layer characteristics from training samples corresponding to the shape information;
calculating the minimum mean square error of two conv4 layer features as P i A loss function of the network model; shape information as P i An input of the network, which outputs the hidden vector as an input of the generator, to obtain a raw shape of the specified shapeImaging and obtaining an objective function.
2. The method of claim 1, wherein the defect types are classified into four types of cracks, ablations, abrasions, and coating losses, and the model is trained separately for each type of defect.
3. The method of claim 1, wherein the P-network comprises 5 convolution layers, the first convolution layer being configured to convolve a 64 x 3 picture to a 32 x 128 tensor; the second convolution layer is configured to convolve a tensor of 32×32×128 with a tensor of 16×16×256, the third convolution layer is configured to convolve a tensor of 16×16×256 with a tensor of 8×8×512, and the fourth convolution layer is configured to convolve a tensor of 8×8×512 with a tensor of 4×4×1024; the last convolution layer is used to convolve the 4 x 1024 tensors into a 100-dimensional vector.
4. The interactive hole detection data expansion method based on the generation of the countermeasure network according to claim 1, wherein the objective function is specifically:
wherein C is conv4 layer characteristics of AlexNet, G i Generator, z i As the hidden vector, the vector is a vector of the hidden vector,for training samples, < >>Is P i Parameters to be updated in the network.
5. The interactive hole detection data expansion method based on the generation of the countermeasure network according to claim 1, wherein the test by the trained P network is specifically:
input shape I to be specified shape Inputting trained P i Generating hidden vectors z in a network i To hidden vector z i A fine noise disturbance N (z) is added,
D fake ＝G i (P i (I shape )+N(z))
wherein D is fake To generate a defect image, P i (I shape ) Representation of the utilization P i Network predicted input shape I shape N (z) represents a fine noise conforming to a gaussian distribution.
6. The method for expanding interactive hole detection data based on the generation of the countermeasure network according to claim 1, wherein the steps of constructing an interactive basic framework and using the adaptive threshold segmentation defect are specifically as follows:
selecting the corresponding P i Network and defect generator G i A network for acquiring strokes input by a user in real time and extracting the strokes as input shapes; through P i Network encodes input strokes into z i Reuse of G i Network pair z i Decoding to generate a defect image D fake ；
And then D is carried out fake Fusing the defects to the strokes drawn by the user to generate specified type defects which are similar to the strokes in shape and the same in position; by aligning z i Generating a plurality of defect images which are similar in shape and have fine gaps by using different noise disturbance;
and changing the low threshold tau in Canny in real time, and adjusting the segmentation results of different defect images by setting the high threshold.
7. An interactive hole detection data extension method based on generating an antagonism network of claim 1, further comprising: and constructing an interactive interface based on the Pyqt5 framework.
CN202110311132.6A 2021-03-24 2021-03-24 Interactive hole detection data expansion method based on generation countermeasure network Active CN113033656B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110311132.6A CN113033656B (en) 2021-03-24 2021-03-24 Interactive hole detection data expansion method based on generation countermeasure network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110311132.6A CN113033656B (en) 2021-03-24 2021-03-24 Interactive hole detection data expansion method based on generation countermeasure network

Publications (2)

Publication Number Publication Date
CN113033656A CN113033656A (en) 2021-06-25
CN113033656B true CN113033656B (en) 2023-12-26

Family

ID=76473089

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110311132.6A Active CN113033656B (en) 2021-03-24 2021-03-24 Interactive hole detection data expansion method based on generation countermeasure network

Country Status (1)

Country Link
CN (1) CN113033656B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107679465A (en) * 2017-09-20 2018-02-09 上海交通大学 A kind of pedestrian's weight identification data generation and extending method based on generation network
CN108769993A (en) * 2018-05-15 2018-11-06 南京邮电大学 Based on the communication network abnormal user detection method for generating confrontation network
CN110533606A (en) * 2019-07-30 2019-12-03 中国民航大学 Safety check X-ray contraband image data Enhancement Method based on production confrontation network
EP3576020A1 (en) * 2018-05-30 2019-12-04 Siemens Healthcare GmbH Methods for generating synthetic training data and for training deep learning algorithms for tumor lesion characterization, method and system for tumor lesion characterization, computer program and electronically readable storage medium
CN111145116A (en) * 2019-12-23 2020-05-12 哈尔滨工程大学 Sea surface rainy day image sample augmentation method based on generation of countermeasure network
CN111199531A (en) * 2019-12-27 2020-05-26 中国民航大学 Interactive data expansion method based on Poisson image fusion and image stylization
CN111415316A (en) * 2020-03-18 2020-07-14 山西安数智能科技有限公司 Defect data synthesis algorithm based on generation of countermeasure network

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107679465A (en) * 2017-09-20 2018-02-09 上海交通大学 A kind of pedestrian's weight identification data generation and extending method based on generation network
CN108769993A (en) * 2018-05-15 2018-11-06 南京邮电大学 Based on the communication network abnormal user detection method for generating confrontation network
EP3576020A1 (en) * 2018-05-30 2019-12-04 Siemens Healthcare GmbH Methods for generating synthetic training data and for training deep learning algorithms for tumor lesion characterization, method and system for tumor lesion characterization, computer program and electronically readable storage medium
CN110533606A (en) * 2019-07-30 2019-12-03 中国民航大学 Safety check X-ray contraband image data Enhancement Method based on production confrontation network
CN111145116A (en) * 2019-12-23 2020-05-12 哈尔滨工程大学 Sea surface rainy day image sample augmentation method based on generation of countermeasure network
CN111199531A (en) * 2019-12-27 2020-05-26 中国民航大学 Interactive data expansion method based on Poisson image fusion and image stylization
CN111415316A (en) * 2020-03-18 2020-07-14 山西安数智能科技有限公司 Defect data synthesis algorithm based on generation of countermeasure network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于风格迁移的交互式航空发动机孔探图像扩展方法;樊玮等;基于风格迁移的交互式航空发动机孔探图像扩展方法;第40卷(第12期);第3631-3636页 *

Also Published As

Publication number Publication date
CN113033656A (en) 2021-06-25

Similar Documents

Publication Publication Date Title
Jiang et al. Scfont: Structure-guided chinese font generation via deep stacked networks
Jam et al. A comprehensive review of past and present image inpainting methods
Zhang et al. Cross-domain correspondence learning for exemplar-based image translation
CN109299274B (en) Natural scene text detection method based on full convolution neural network
Zhang et al. Shadow remover: Image shadow removal based on illumination recovering optimization
CN108376244B (en) Method for identifying text font in natural scene picture
US8655069B2 (en) Updating image segmentation following user input
CN112784736B (en) Character interaction behavior recognition method based on multi-modal feature fusion
CN111126127B (en) High-resolution remote sensing image classification method guided by multi-level spatial context characteristics
CN111951381B (en) Three-dimensional face reconstruction system based on single face picture
CN110413816A (en) Colored sketches picture search
CN108595558B (en) Image annotation method based on data equalization strategy and multi-feature fusion
Qin et al. Automatic skin and hair masking using fully convolutional networks
Shi et al. Deep line art video colorization with a few references
Chen et al. A review of image and video colorization: From analogies to deep learning
Yao et al. Manga vectorization and manipulation with procedural simple screentone
CN113393546B (en) Fashion clothing image generation method based on clothing type and texture pattern control
Zhang et al. EXCOL: An EXtract-and-COmplete layering approach to cartoon animation reusing
Zhang et al. Imageadmixture: Putting together dissimilar objects from groups
Fu et al. Fast accurate and automatic brushstroke extraction
Zhang et al. Deep exemplar-based color transfer for 3d model
Lu et al. Real-time video stylization using object flows
Lin et al. Video stylization: painterly rendering and optimization with content extraction
CN113033656B (en) Interactive hole detection data expansion method based on generation countermeasure network
Zhang et al. A broad generative network for two-stage image outpainting

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant