CN113096037B - Deep learning-based repairing method for wheel pair bar images - Google Patents
Deep learning-based repairing method for wheel pair bar images Download PDFInfo
- Publication number
- CN113096037B CN113096037B CN202110350173.6A CN202110350173A CN113096037B CN 113096037 B CN113096037 B CN 113096037B CN 202110350173 A CN202110350173 A CN 202110350173A CN 113096037 B CN113096037 B CN 113096037B
- Authority
- CN
- China
- Prior art keywords
- feature
- image
- layer
- repair
- pconv
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000013135 deep learning Methods 0.000 title claims abstract description 9
- 230000008439 repair process Effects 0.000 claims abstract description 56
- 230000004927 fusion Effects 0.000 claims abstract description 42
- 230000006870 function Effects 0.000 claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 16
- 230000000750 progressive effect Effects 0.000 claims abstract description 8
- 125000004122 cyclic group Chemical group 0.000 claims description 21
- 238000013507 mapping Methods 0.000 claims description 15
- 239000013598 vector Substances 0.000 claims description 15
- 238000005259 measurement Methods 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 10
- 238000005457 optimization Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 6
- 230000004913 activation Effects 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 3
- 238000012549 training Methods 0.000 claims description 3
- 230000001902 propagating effect Effects 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 14
- 238000010586 diagram Methods 0.000 description 15
- 239000011159 matrix material Substances 0.000 description 5
- 238000013136 deep learning model Methods 0.000 description 4
- 230000006872 improvement Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a method for repairing a wheel pair bar image based on deep learning. The method comprises the following steps: acquiring a wheel pair bar image to be repaired, and inputting the wheel pair bar image to be repaired into a circulation network; in a circulation network, performing circulation progressive restoration processing on a light bar image of a wheel to be restored by utilizing a soft coding Pconv layer and an asymmetric similarity module; and carrying out feature fusion on the multiple repair feature images output by the circulation network to obtain a fusion feature image, calculating a difference value between the fusion feature image and the true image by utilizing a loss function and combining an MS-SSIM loss item, and adjusting the fusion feature image according to the difference value to obtain the repaired wheel pair bar image. The method of the embodiment of the invention can effectively repair the light spot and the local fracture of the multi-line laser light bar image of the wheel set, and can accurately restore the light bar of the fracture area; the invention has good repairing effect in the light spot area and can meet the precision requirement in the current practical industrial environment.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a method for repairing a wheel pair light bar image based on deep learning.
Background
RFR-Net (Recurrent Feature Reasoning Network, cyclic feature inference network) is a cyclic feature inference network, the structure of which is schematically shown in FIG. 1. The convolution mode of the network mainly adopts Pconv (Partial convolution ) and common convolution, wherein each network is provided with one Pconv layer at the head and the tail, and the middle is a main circulation network structure. The cyclic network structure comprises a multi-layer cyclic structure, wherein each layer of cyclic structure starts with two Pconv layers for repairing images, and a core part of the cyclic network structure, namely a coding and decoding sub-network is connected. The coding and decoding sub-network is composed of normal convolution downsampling and upsampling, jump connection is arranged between the downsampling and the corresponding upsampling (shown by a dotted arrow in the figure), and besides, a similarity mapping module is embedded between the downsampling and the upsampling, and the similarity mapping module is responsible for optimizing the precision of the image restoration features. The network has multiple layers of loops, each layer of loops outputs an intermediate feature map, and the multiple intermediate feature maps are subjected to feature fusion through an average value taking strategy and serve as input of the next common convolution.
The loss function of RFR-Net combines three loss terms.
Absolute difference term: and (3) calculating the absolute difference of pixel values between the restored image and the truth image, and specifically calculating the L1 distance average value of the pixel values at the corresponding positions of the two images.
Style loss term: the loss term for keeping the style of the constrained image consistent is characterized in that the repaired image and the truth image are respectively input into a VGG-16 network (Visual Geometry Group-16, 16-layer network) and output at a plurality of pooling layers, and an L1 distance average value of the corresponding position characteristic value, namely the style loss term, is calculated between the characteristic diagram obtained by inputting the repaired image and the corresponding characteristic diagram obtained by inputting the truth image.
Perceptual loss term: the method is also a loss term for keeping the style of the constrained image consistent, but the angle of realization is different from the style loss term, and the difference is that after the output of the pooling layer, the gram matrix of the output characteristic diagram is calculated, and the average value of the L1 distance between the gram matrix of the output of the restored image and the corresponding gram matrix of the output of the true image is calculated.
The image restoration method based on the RFR-Net network in the prior art has the following defects: the core similarity mapping module of RFR-Net adopts cosine distance as a measurement standard of similarity between different characteristic points, and the cosine distance is more suitable for color images. However, the pair of light bar images are gray images, the values of the R, G, B three components are the same, so that cosine angles between different middle characteristic points tend to be 0, and the meaning of similarity measurement is small. In addition, the loss function of RFR-Net only takes into account the difference in pixel values and style between the output image and the truth image, while the rotation has significant structural characteristics for the stripe image, which is a not insignificant challenge for the current loss function design of RFR-Net.
Disclosure of Invention
The embodiment of the invention provides a method for repairing a wheel pair light bar image based on deep learning, which aims to overcome the problems in the prior art.
In order to achieve the above purpose, the present invention adopts the following technical scheme.
A method for repairing a pair of light bar images based on deep learning comprises the following steps:
acquiring a wheel pair bar image to be repaired, and inputting the wheel pair bar image to be repaired into a circulation network;
in a circulation network, performing circulation progressive restoration processing on a light bar image of a wheel to be restored by utilizing a soft coding Pconv layer and an asymmetric similarity module;
and carrying out feature fusion on the multiple repair feature images output by the circulation network to obtain a fusion feature image, calculating a difference value between the fusion feature image and the true image by utilizing a loss function and combining an MS-SSIM loss item, and adjusting the fusion feature image according to the difference value to obtain the repaired wheel pair bar image.
Preferably, in the cyclic network, the cyclic progressive restoration processing is performed on the stripe image of the wheel to be restored by using a soft coding Pconv layer and an asymmetric similarity module, including:
each layer in the cyclic network comprises a soft coding Pconv layer and an asymmetric similarity module, the soft coding Pconv layer in the first layer is used for receiving an input round bar image to be repaired, the soft coding Pconv layer is used for progressively repairing the bar image to be repaired by using a mask image and outputting a staged repairing feature image, the asymmetric similarity module in the first layer is used for carrying out multi-segment downsampling on the repairing feature image output by the soft coding Pconv layer, then carrying out similarity measurement calculation on target feature points and source feature points, carrying out optimization processing on the repairing feature image according to the similarity measurement, and outputting the optimized repairing feature image;
and then, inputting the final output repair feature map of the previous layer of cycle into the soft coding Pconv layer of each cycle, executing repair processing, outputting the next staged repair feature map, optimizing the repair feature map output by the soft coding Pconv layer by the asymmetric similarity module of each cycle, and outputting the optimized repair feature map.
Preferably, the feature fusion is performed on the multiple repair feature graphs output by the circulation network to obtain a fusion feature graph, a difference value between the fusion feature graph and a true image is calculated by combining a loss function with an MS-SSIM loss term, the fusion feature graph is adjusted according to the difference value, and a repaired round bar image is obtained, which comprises:
and carrying out feature fusion on the multiple repair feature images circularly output by the circulation network to obtain a fusion feature image, after carrying out up-sampling operation on the fusion feature image, calculating a difference value between the fusion feature image and a truth image by utilizing a loss function and combining an MS-SSIM loss item, reversely propagating and updating network parameters according to the difference value, carrying out feature fusion on the multiple repair feature images by utilizing the updated network parameters, and recalculating the difference between the fusion feature image and the truth image, and carrying out the processing process circularly until the difference between the fusion feature image and the truth image is smaller than a set threshold value to obtain a final repaired round bar image.
Preferably, the first layer cyclic asymmetric similarity module performs similarity measurement calculation of the target feature point and the source feature point after performing multi-segment downsampling on the repair feature map output by the soft coding Pconv layer, including:
after the first layer cyclic asymmetric similarity module carries out multi-segment downsampling on the repair feature map output by the soft coding Pconv layer, the feature vector corresponding to the target feature point is normalized, the mode of the feature vector corresponding to the source feature point is kept unchanged, and the formula for calculating the similarity measurement of the target feature point and the source feature point is as follows:
wherein (x, y) is a target feature point on the feature map, (x ', y') is a source feature point in the feature map, f x,y And f x’,y’ Is the feature vector of the feature points (x, y) and (x ', y') corresponding in the channel dimension of the feature map, sim represents the feature similarity between the two feature points, and sim represents the inner product of the two feature vectors;
considering the smoothness of the restoration result of the image of the strip of the wheel, the final similarity value takes the average value of the similarity of all the characteristic points and the source characteristic points (x ', y') in the k x k region taking (x, y) as the center point, and the specific formula is as follows:
similarity mapping score for layer i loops ’i x,y Processing sim by softmax function ’i x,y,x’,y’ Obtaining a final similarity mapping score obtained by an ith layer i x,y Mapping result score including current loop computation ’i x,y Similarity map score with the previous layer i-1 x,y The specific formula is as follows:
where λ is the hyper-parameter of the formula.
Preferably, the soft-coded Pconv layer convolutions kernel k m Iteratively updated during training of the depth model and employing a convolution kernel filter M with learning c ＝M⊙k m ；
The soft coding Pconv layer adopts an improved mask updating strategy g M (M c The formula is as follows:
g M (M c )＝(ReLU(M c )) α
wherein ReLU is a nonlinear activation function, α is a super-parameter, α is not less than 0, g when α=0 M (M c ) Is degenerated to f M (M c )；
The soft coding Pconv layer adopts an asymmetric Gaussian shape function g M (M c ) As a characteristic scaling factor, the specific formula is as follows:
wherein a, gamma l ,γ r μ is a hyper-parameter in the formula, mc is calculated in a), exp represents an exponential function based on e.
Preferably, the MS-SSIM loss term constrains the difference between the repair image and the truth image in terms of brightness, contrast, and structure, and contains M resolutions, and the MS-SSIM specific formula is as follows:
wherein l M (x, y) represents the luminance component of the M-th layer, c i (x, y) represents the contrast component of the ith layer, s i (x, y) represents the structural component of the ith layer, α M Representing the luminance component super-parameter, beta of the Mth layer i Super-parameters, gamma, representing the contrast component of the ith layer i Representing structural component superparameters of the ith layer;
a) The luminance component formula is as follows:
C 1 is a super parameter of a formula, and assuming that a pixel region x and a pixel region y are respectively selected at the same position of the repair image and the truth image, μx and μy represent average values of the two pixel regions, and the specific formula is as follows:
wherein N is the number of pixel points in the pixel area, and x i And y j Pixel values representing the x and y regions, respectively;
b) The contrast component formula is as follows:
wherein C is 2 Is a hyper-parameter of a formula, sigma x and sigma y respectively represent the contrast of the repair image and the truth image selection areas x and y, and specifically calculates the standard deviation of the value and the average value in the areas, and the formula is as follows:
wherein N represents the number of points in the x and y regions;
c) The structural component formula is as follows:
wherein C is 3 Is a hyper-parameter of the formula, σx and σy still represent the contrast of the x and y regions, σ xy The formula of (2) is as follows:
according to the technical scheme provided by the embodiment of the invention, the method can be used for effectively repairing the light spots and the local fracture of the multi-line laser light bar image of the wheel set, and accurately restoring the light bar of the fracture area; the invention has good repairing effect in the light spot area and can meet the precision requirement in the current practical industrial environment.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a block diagram of an RFR-Net network of the prior art;
FIG. 2 is a block diagram of a deep learning model based on a cyclic network structure according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an implementation principle of an asymmetric similarity module according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a repairing effect of a wheel pair bar image according to an embodiment of the present invention, wherein the upper 4 graphs are repairing effects of a broken area, and the lower 4 graphs are repairing effects of a light spot area.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only for explaining the present invention and are not to be construed as limiting the present invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or coupled. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
For the purpose of facilitating an understanding of the embodiments of the invention, reference will now be made to the drawings of several specific embodiments illustrated in the drawings and in no way should be taken to limit the embodiments of the invention.
The embodiment of the invention provides a deep learning model based on a circulating network structure, and provides a method for repairing a pair of light bar images based on deep learning based on the deep learning model. The structure of the deep learning model based on the cyclic network structure in the embodiment of the invention is shown in fig. 2, has essential differences with RFR-Net, and comprises the following modules:
step 10: and acquiring a wheel pair bar image to be repaired through a camera and a laser, and inputting the wheel pair bar image to be repaired and a mask image for repairing into a circulation network.
The mask image is used for identifying the range of the area to be repaired of the light bar image, the mask image is a binary image, 1 indicates that the corresponding area is an existing area, and 0 indicates that the corresponding area is the area to be repaired.
Step 20: in a circulation network, a soft coding Pconv layer and an asymmetric similarity module are utilized to carry out circulation progressive restoration on a light bar image of a wheel to be restored.
And the soft coding Pconv layer at the beginning of the network uses the mask image to carry out progressive restoration on the optical stripe image to be restored and outputs a stepwise restoration characteristic diagram. The soft coding Pconv layer of the first layer of the cycle receives the repair feature map output by the soft coding Pconv layer started by the network, and then the soft coding Pconv layer of each cycle inputs the repair feature map finally output by the previous layer of the cycle, and executes repair to output the next staged repair feature map.
The asymmetric similarity module is responsible for optimizing the repair result of the soft coding Pconv layer after the soft coding Pconv layer of each layer cycle. The input of the method is an output characteristic diagram of the soft coding Pconv layer output of the same layer of cycle after multi-section downsampling, the dissymmetry similarity module carries out similarity measurement calculation of target characteristic points and source characteristic points on the repair characteristic diagram, the repair characteristic diagram is optimized according to the similarity measurement, and the output of the repair characteristic diagram is the optimized repair characteristic diagram with the same resolution as the input.
Step 30: combining MS-SSIM penalty terms with penalty functions
After the execution of the cyclic network is completed, feature fusion is carried out on a plurality of repair feature graphs obtained through the cyclic process to obtain a fusion feature graph, after up-sampling operation is carried out on the fusion feature graph, a loss function is utilized to combine with an MS-SSIM loss item to calculate the difference value between the fusion feature graph and a true value image from multiple aspects, and network parameters are reversely propagated and updated according to the difference value. And then, carrying out feature fusion on the multiple repair feature images again by utilizing the updated network parameters, and calculating the difference between the fusion feature images and the truth images again, and circularly executing the processing process until the difference between the fusion feature images and the truth images is smaller than a set threshold value, so as to obtain the final repaired round bar image.
1) Asymmetric similarity module
Fig. 3 is a schematic diagram of an implementation principle of an asymmetric similarity module according to an embodiment of the present invention. As described above, RFR-Net uses cosine distance to perform similarity measurement in the core similarity mapping module, and the similarity mapping module is used to calculate the similarity between all the target feature points and the source feature points, so as to optimize the feature values corresponding to the target feature points. The specific formula is shown below.
Wherein (x, y) is a target feature point on the feature map, (x ', y') is a source feature point in the feature map, f x,y And f x’,y’ Is a feature vector of feature points (x, y) and (x ', y') corresponding in the channel dimension of the feature map, sim represents feature similarity between the two feature points, ||.
The similarity measurement of cosine distance is improved by the module, and the improvement scheme can improve the optimization effect of the similarity mapping module on the target feature points. The new similarity metric formula is shown below:
the cosine distance is obtained by normalizing all the two feature vectors, dividing the feature vectors by the own module to change the module of the feature vectors into 1, and the improvement of the asymmetric similarity module relative to the cosine distance is obtained by normalizing the feature vectors corresponding to the target feature points, wherein the module of the feature vectors corresponding to the source feature points is kept unchanged. Therefore, when cosine angles between the target feature point and the source feature points are not greatly different, the similarity can be distinguished by using modes of different source feature points, and the problem that the cosine distance has smaller effect in the gray image is solved.
The formula calculates the similarity between the corresponding feature vectors of the target feature points (x, y) and the source feature points (x ', y'), and takes the average value of the similarity between all the feature points and the source feature points (x ', y') in a k x k region taking (x, y) as a central point into consideration of the smoothness of the restoration result of the strip image of the wheel. The specific formula is as follows:
because the network of the present invention is a cyclic network structure, it has multiple layers of loops. Similarity mapping (noted score) for layer i loops ’i x,y ) Processing sim by softmax function ’i x,y,x’,y’ Acquisition, to integrate the similarity maps obtained for the different loops, the final similarity map obtained by the ith layer (denoted score i x,y ) Mapping result score including not only current loop calculation ’i x,y Also contains the similarity map (named score) of the previous layer (i-1) i-1 x,y ). The specific formula is as follows:
where λ is the hyper-parameter of the formula.
2) Combining soft coded Pconv layers
As described above, the original Pconv layer uses more artificially set, fixed parameters on the forward propagation and mask area update formulas and thus does not work well. The embodiment of the invention utilizes the soft-coded Pconv layer to carry out image restoration, combines the soft-coded Pconv layer to carry out a plurality of optimization strategies based on the original Pconv layer, and the optimization strategies can promote the capability of restoring the image of the whole network, and the optimization contents are as follows:
a) K for improved fixation 1/n A filter matrix.
Original k 1/n The values of the filter matrix are all 1/n and are fixed, and therefore have no learning. The filter after soft coding Pconv layer modification is defined as k m Is a convolution kernel filter with learning in deep learning, M is improved c ＝M⊙k m 。
k m Is with k 1/n Convolution kernels of equal size whose values are in depth modelIs updated iteratively during the training process of (a). M ≡k m Representing convolution operations, i.e. M and k m Point-to-point multiply sums.
B) Update strategy for improving mask area update
The original mask area update uses f M (M c ) Is updated with the policy. For example, improvement A) M c Is shown in the formula of (1), M is improved c With convolution kernel filters, the present invention is intended to be directed to higher M c The higher the value is, the more updated the value is, and thus the M is calculated c And adding a nonlinear activation function mapping ReLU as a final weight value, wherein the ReLU monotonically increases in the range of the positive value definition domain. The modified mask update policy is defined as g M (M c ) The formula is as follows:
g M (M c )＝(ReLU(M c )) α
wherein ReLU is a nonlinear activation function for enhancing learning and feature representation capabilities of the network (because of M c The calculation itself is linear and the features represent insufficient power). Alpha is a super parameter, alpha is not less than 0, g when alpha=0 M (M c ) Is degenerated to f M (M c )。g M (M c ) Is present to update the value of the mask area. Because the mask value does not have a negative value, a ReLU activation function is employed; in addition to the purpose of highlighting M c The higher value region corresponds to the feature and employs the index α. f (f) M (M c ) Is a method for updating a mask map, i.e. a feature map for identifying repair areas as described before in the text. The mask map is updated each time the soft code Pconv layer performs convolution, and the updated formula adopts f M (M c )。
C) Improving feature scaling factors in forward propagation
As previously described, the original PCnv layer forward propagation employs monotonically varying g M (M c ) Feature scaling factor whose calculation strategy considers scaling factor as M c Is monotonically decreasing. The improved characteristic scaling factor calculation strategy in the invention considers that the scaling is performed for the round bar imageThe factor is not M c Monotonically changing, but with intermediate extremum points, the invention therefore improves the characteristic scaling factor to an asymmetric gaussian shape function g M (M c ) The specific formula is as follows:
wherein a, gamma l ,γ r μ is a hyper-parameter in the formula, mc is calculated in a), exp represents an exponential function based on e.
Because the nature of image restoration is that the feature information of the existing region or the restored region is processed to obtain the feature of the region to be restored, the specific gravity of the existing region or the feature of the restored region in the processing process is necessarily light and heavy. g M (M c ) The effect is to more accurately describe the specific gravity corresponding to the features of the existing region and the different locations within the repaired region.
3) The loss function incorporates MS-SSIM (Multi-Scale Structural Similarity ) loss terms.
Besides the absolute difference term, the style loss term and the perception loss term of the pixel values, the invention combines an MS-SSIM loss term, which can ensure that the structural characteristics of the wheel pair light bars are restored more accurately and consider the problems of different image resolutions. MS-SSIM constrains the difference between the repair image and the truth image from three aspects (brightness, contrast, and structure, respectively) and contains M resolutions that are downsampled 2 times from the original resolution, iterated M-1 times. The MS-SSIM has the following specific formula:
wherein l M (x, y) represents the luminance component of the M-th layer, c i (x, y) represents the contrast component of the ith layer, s i (x, y) represents the structural component of the i-th layer. Alpha M Representing the luminance component super-parameter, beta of the Mth layer i Super-parameters, gamma, representing the contrast component of the ith layer i Representing structural component superparameters of the i-th layer.
A) The luminance component formula is as follows:
C 1 is a hyper-parameter of the formula. Assuming that the same positions of the repair image and the truth image respectively select one pixel region x and y, μx and μy represent the average value of the two pixel regions, and the specific formula is as follows:
wherein N is the number of pixel points in the pixel area, and x i And y j Representing pixel values for the x and y regions, respectively.
B) The contrast component formula is as follows:
wherein C is 2 Is a hyper-parameter of the formula. σx and σy represent the contrast of the repair image and the truth image selection areas x and y, respectively, and the standard deviation of the value and average value (luminance) in the specific calculation area is as follows:
where N represents the number of points in the x and y regions.
C) The structural component formula is as follows:
wherein C is 3 Is a hyper-parameter of the formula. Sigma x and sigma y still represent the contrast of the x and y regions, sigma xy The formula of (2) is as follows:
the intelligent fault detection system is used for the train wheel set, and the intelligent fault detection system is used for collecting the images of the train wheel set through the combination of a camera and a laser. However, in a practical industrial collection environment, the collected image can generate a large number of light spot areas and light bar fracture areas due to the influence of external light rays, and the aim of the invention is to repair the problems and achieve higher repair accuracy. 1) Firstly, removing a light spot area of a light bar with high precision and recovering the original shape of the light bar; 2) And then repairing the image fracture with high precision, and restoring the due light bar shape of the fracture area.
In summary, the light spot area repair and the light stripe fracture area repair effects of the embodiment of the invention in the wheel pair light stripe image are very good, as shown in fig. 4. The upper 4 graphs are fracture zone repair effects and the lower 4 graphs are spot zone repair effects. Left 1 is original image, left 2 is image of mask region (region to be repaired), right 2 is repair effect of RFR-Net model, and right 1 is repair effect of the present invention. The comparison shows that the invention has stronger fracture repair capability and can accurately restore the light bar in the fracture area; the repairing effect of the invention in the light spot area is good, the light bar in the light spot area repaired by RFR-Net is still broken, and the model of the invention can accurately remove the light spot and restore the shape of the light bar.
In general, the invention surpasses the current mainstream model in the aspects of light spot repair and local fracture repair on the wheel set multi-line laser light bar image, and can meet the precision requirement in the current actual industrial environment.
Those of ordinary skill in the art will appreciate that: the drawing is a schematic diagram of one embodiment and the modules or flows in the drawing are not necessarily required to practice the invention.
From the above description of embodiments, it will be apparent to those skilled in the art that the present invention may be implemented in software plus a necessary general hardware platform. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments or some parts of the embodiments of the present invention.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for apparatus or system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, with reference to the description of method embodiments in part. The apparatus and system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
The present invention is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present invention are intended to be included in the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.
Claims (2)
1. A method for repairing a pair of light bar images based on deep learning is characterized by comprising the following steps:
acquiring a wheel pair bar image to be repaired, and inputting the wheel pair bar image to be repaired into a circulation network;
in a circulation network, performing circulation progressive restoration processing on a light bar image of a wheel to be restored by utilizing a soft coding Pconv layer and an asymmetric similarity module;
feature fusion is carried out on a plurality of repair feature images output by the circulation network to obtain a fusion feature image, a loss function is utilized to combine with an MS-SSIM loss item to calculate a difference value between the fusion feature image and a true value image, and the fusion feature image is adjusted according to the difference value to obtain a repaired wheel pair bar image;
the feature fusion is carried out on the multiple repair feature graphs output by the circulation network to obtain a fusion feature graph, a difference value between the fusion feature graph and a true value image is calculated by combining a loss function with an MS-SSIM loss term, the fusion feature graph is adjusted according to the difference value, and a repaired round bar image is obtained, and the method comprises the following steps:
performing feature fusion on the multiple repair feature images circularly output by the circulation network to obtain a fusion feature image, performing up-sampling operation on the fusion feature image, calculating a difference value between the fusion feature image and a truth image by using a loss function and combining an MS-SSIM loss item, reversely propagating and updating network parameters according to the difference value, re-performing feature fusion on the multiple repair feature images by using the updated network parameters, and re-calculating the difference between the fusion feature image and the truth image, and performing the processing process circularly until the difference between the fusion feature image and the truth image is smaller than a set threshold value to obtain a final repaired round bar image;
the first layer of cyclic asymmetric similarity module performs similarity measurement calculation of target feature points and source feature points after performing multi-segment downsampling on a repair feature map output by a soft coding Pconv layer, and the method comprises the following steps:
after the first layer cyclic asymmetric similarity module carries out multi-segment downsampling on the repair feature map output by the soft coding Pconv layer, the feature vector corresponding to the target feature point is normalized, the mode of the feature vector corresponding to the source feature point is kept unchanged, and the formula for calculating the similarity measurement of the target feature point and the source feature point is as follows:
wherein (x, y) is a target feature point on the feature map, (x ', y') is a source feature point in the feature map, f x,y And f x’,y’ Is the feature vector of the feature points (x, y) and (x ', y') corresponding in the channel dimension of the feature map, sim represents the feature similarity between the two feature points, and sim represents the inner product of the two feature vectors;
considering the smoothness of the restoration result of the image of the strip of the wheel, the final similarity value takes the average value of the similarity of all the characteristic points and the source characteristic points (x ', y') in the k x k region taking (x, y) as the center point, and the specific formula is as follows:
similarity mapping score for layer i loops ’i x,y Processing sim by softmax function ’i x,y,x’,y’ Obtaining a final similarity mapping score obtained by an ith layer i x,y Mapping result score including current loop computation ’i x,y Similarity map score with the previous layer i-1 x,y The specific formula is as follows:
wherein λ is the hyper-parameter of the formula;
the convolution kernel k of the soft coding Pconv layer m Iteratively updated during training of the depth model and employing a convolution kernel filter M with learning c ＝M⊙k m ；
The soft coding Pconv layer adopts an improved mask updating strategy g M (M c ) The formula is as follows:
g M (M c )＝(ReLU(M c )) α
wherein ReLU is a nonlinear activation function, α is a super-parameter, α is not less than 0, g when α=0 M (M c ) Is degenerated to f M (M c )；
The soft coding Pconv layer adopts an asymmetric Gaussian shape function g M (M c ) As a characteristic scaling factor, the specific formula is as follows:
wherein a, gamma l ,γ r Mu is the hyper-parameter in the formula, M c Is calculated in A), exp represents an exponential function based on e;
the MS-SSIM loss term constrains the difference between the repair image and the truth image in terms of brightness, contrast and structure, and contains M resolutions, and the specific formula of the MS-SSIM is as follows:
wherein l M (x, y) represents the luminance component of the M-th layer, c i (x, y) represents the contrast component of the ith layer, s i (x, y) represents the structural component of the ith layer, α M Representing the luminance component super-parameter, beta of the Mth layer i Super-parameters, gamma, representing the contrast component of the ith layer i Representing structural component superparameters of the ith layer;
a) The luminance component formula is as follows:
C 1 is of the formulaSuper-parameters, mu, assuming that the same positions of the repair image and the truth image are respectively selected from one pixel region x and one pixel region y x Sum mu y Represents the average value of two pixel areas, and the specific formula is as follows:
wherein N is the number of pixel points in the pixel area, and x i And y j Pixel values representing the x and y regions, respectively;
b) The contrast component formula is as follows:
wherein C is 2 Is the hyper-parameter of the formula, sigma x Sum sigma y The contrast of the selected areas x and y representing the repair image and the truth image respectively, and the standard deviation of the values and average values in the areas are calculated specifically, the formula is as follows:
wherein N represents the number of points in the x and y regions;
c) The structural component formula is as follows:
wherein C is 3 Is the hyper-parameter of the formula, sigma x Sum sigma y Still representing the contrast of the x and y regions, σ xy The formula of (2) is as follows:
2. the method of claim 1, wherein the performing, in the cyclic network, cyclic progressive restoration processing on the stripe image of the wheel to be restored by using the soft coding Pconv layer and the asymmetric similarity module comprises:
each layer in the cyclic network comprises a soft coding Pconv layer and an asymmetric similarity module, the soft coding Pconv layer in the first layer is used for receiving an input round bar image to be repaired, the soft coding Pconv layer is used for progressively repairing the bar image to be repaired by using a mask image and outputting a staged repairing feature image, the asymmetric similarity module in the first layer is used for carrying out multi-segment downsampling on the repairing feature image output by the soft coding Pconv layer, then carrying out similarity measurement calculation on target feature points and source feature points, carrying out optimization processing on the repairing feature image according to the similarity measurement, and outputting the optimized repairing feature image;
and then, inputting the final output repair feature map of the previous layer of cycle into the soft coding Pconv layer of each cycle, executing repair processing, outputting the next staged repair feature map, and optimizing the repair feature map output by the soft coding Pconv layer by the asymmetric similarity module of each cycle to output the optimized repair feature map.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110350173.6A CN113096037B (en) | 2021-03-31 | 2021-03-31 | Deep learning-based repairing method for wheel pair bar images |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110350173.6A CN113096037B (en) | 2021-03-31 | 2021-03-31 | Deep learning-based repairing method for wheel pair bar images |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113096037A CN113096037A (en) | 2021-07-09 |
| CN113096037B true CN113096037B (en) | 2023-08-22 |
Family
ID=76672068
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110350173.6A Active CN113096037B (en) | 2021-03-31 | 2021-03-31 | Deep learning-based repairing method for wheel pair bar images |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113096037B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107945140A (en) * | 2017-12-20 | 2018-04-20 | 中国科学院深圳先进技术研究院 | A kind of image repair method, device and equipment |
| CN108986058A (en) * | 2018-06-22 | 2018-12-11 | 华东师范大学 | The image interfusion method of lightness Consistency Learning |
| CN111028167A (en) * | 2019-12-05 | 2020-04-17 | 广州市久邦数码科技有限公司 | Image restoration method based on deep learning |
| CN111242874A (en) * | 2020-02-11 | 2020-06-05 | 北京百度网讯科技有限公司 | Image restoration method and device, electronic equipment and storage medium |
| WO2020118902A1 (en) * | 2018-12-14 | 2020-06-18 | 深圳市华星光电半导体显示技术有限公司 | Image processing method and image processing system |
| CN111325679A (en) * | 2020-01-08 | 2020-06-23 | 深圳深知未来智能有限公司 | Method for enhancing dark light image from Raw to Raw |
| CN111402149A (en) * | 2020-03-06 | 2020-07-10 | 四川大学 | Fringe pattern restoration method based on convolutional neural network denoising regularization |
| CN112581392A (en) * | 2020-12-15 | 2021-03-30 | 中山大学 | Image exposure correction method, system and storage medium based on bidirectional illumination estimation and fusion restoration |
-
2021
- 2021-03-31 CN CN202110350173.6A patent/CN113096037B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107945140A (en) * | 2017-12-20 | 2018-04-20 | 中国科学院深圳先进技术研究院 | A kind of image repair method, device and equipment |
| CN108986058A (en) * | 2018-06-22 | 2018-12-11 | 华东师范大学 | The image interfusion method of lightness Consistency Learning |
| WO2020118902A1 (en) * | 2018-12-14 | 2020-06-18 | 深圳市华星光电半导体显示技术有限公司 | Image processing method and image processing system |
| CN111028167A (en) * | 2019-12-05 | 2020-04-17 | 广州市久邦数码科技有限公司 | Image restoration method based on deep learning |
| CN111325679A (en) * | 2020-01-08 | 2020-06-23 | 深圳深知未来智能有限公司 | Method for enhancing dark light image from Raw to Raw |
| CN111242874A (en) * | 2020-02-11 | 2020-06-05 | 北京百度网讯科技有限公司 | Image restoration method and device, electronic equipment and storage medium |
| CN111402149A (en) * | 2020-03-06 | 2020-07-10 | 四川大学 | Fringe pattern restoration method based on convolutional neural network denoising regularization |
| CN112581392A (en) * | 2020-12-15 | 2021-03-30 | 中山大学 | Image exposure correction method, system and storage medium based on bidirectional illumination estimation and fusion restoration |
Non-Patent Citations (1)
| Title |
|---|
| A simple deep learning based image illumination correction method for paintings;Suranjan Goswami 等;ELSEVIER;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113096037A (en) | 2021-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lee et al. | Mu-net: Multi-scale U-net for two-photon microscopy image denoising and restoration | |
| CN110599500B (en) | Tumor region segmentation method and system of liver CT image based on cascaded full convolution network | |
| CN111696110B (en) | Scene segmentation method and system | |
| CN110070531B (en) | Model training method for detecting fundus picture, and fundus picture detection method and device | |
| CN113807355A (en) | Image semantic segmentation method based on coding and decoding structure | |
| CN112712472A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN112365514A (en) | Semantic segmentation method based on improved PSPNet | |
| CN113160058B (en) | Thermal imaging super-resolution reconstruction method combined with segmentation map and storage medium | |
| Ye et al. | Depth super-resolution with deep edge-inference network and edge-guided depth filling | |
| CN115564649B (en) | Image super-resolution reconstruction method, device and equipment | |
| Feng et al. | SSN: A stair-shape network for real-time polyp segmentation in colonoscopy images | |
| CN116309648A (en) | Medical image segmentation model construction method based on multi-attention fusion | |
| CN113554665A (en) | Blood vessel segmentation method and device | |
| CN115272437A (en) | Image depth estimation method and device based on global and local features | |
| US11783454B2 (en) | Saliency map generation method and image processing system using the same | |
| CN113516604B (en) | Image restoration method | |
| CN117197627B (en) | Multi-mode image fusion method based on high-order degradation model | |
| Wu et al. | Multi-focus image fusion: Transformer and shallow feature attention matters | |
| CN113096037B (en) | Deep learning-based repairing method for wheel pair bar images | |
| CN117351487A (en) | Medical image segmentation method and system for fusing adjacent area and edge information | |
| Chacon-Murguia et al. | Moving object detection in video sequences based on a two-frame temporal information CNN | |
| CN112750110A (en) | Evaluation system for evaluating lung lesion based on neural network and related products | |
| CN117036380A (en) | Brain tumor segmentation method based on cascade transducer | |
| CN115100731B (en) | Quality evaluation model training method and device, electronic equipment and storage medium | |
| CN116934972A (en) | Three-dimensional human body reconstruction method based on double-flow network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |