WO2021134874A1 - Training method for deep residual network for removing a moire pattern of two-dimensional code - Google Patents
Training method for deep residual network for removing a moire pattern of two-dimensional code Download PDFInfo
- Publication number
- WO2021134874A1 WO2021134874A1 PCT/CN2020/076819 CN2020076819W WO2021134874A1 WO 2021134874 A1 WO2021134874 A1 WO 2021134874A1 CN 2020076819 W CN2020076819 W CN 2020076819W WO 2021134874 A1 WO2021134874 A1 WO 2021134874A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- dimensional code
- code image
- moiré
- residual
- Prior art date
Links
- 238000012549 training Methods 0.000 title claims abstract description 45
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000012545 processing Methods 0.000 claims abstract description 17
- 230000004927 fusion Effects 0.000 claims abstract description 12
- 238000005070 sampling Methods 0.000 claims abstract description 12
- 238000007781 pre-processing Methods 0.000 claims abstract description 10
- 238000000746 purification Methods 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 21
- 230000004913 activation Effects 0.000 claims description 17
- 230000009466 transformation Effects 0.000 claims description 14
- 238000013508 migration Methods 0.000 claims description 9
- 230000005012 migration Effects 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 5
- 241001270131 Agaricus moelleri Species 0.000 claims description 3
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims description 3
- 230000015572 biosynthetic process Effects 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 230000005855 radiation Effects 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 15
- 238000005516 engineering process Methods 0.000 description 10
- 230000000694 effects Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 238000000354 decomposition reaction Methods 0.000 description 3
- 238000013526 transfer learning Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06K19/00—Record carriers for use with machines and with at least a part designed to carry digital markings
- G06K19/06—Record carriers for use with machines and with at least a part designed to carry digital markings characterised by the kind of the digital marking, e.g. shape, nature, code
- G06K19/06009—Record carriers for use with machines and with at least a part designed to carry digital markings characterised by the kind of the digital marking, e.g. shape, nature, code with optically detectable marking
- G06K19/06037—Record carriers for use with machines and with at least a part designed to carry digital markings characterised by the kind of the digital marking, e.g. shape, nature, code with optically detectable marking multi-dimensional coding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
- G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
- G06K7/14—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation using light without selection of wavelength, e.g. sensing reflected white light
- G06K7/1404—Methods for optical code recognition
- G06K7/1408—Methods for optical code recognition the method being specifically adapted for the type of code
- G06K7/1417—2D bar codes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06K—GRAPHICAL DATA READING; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
- G06K7/00—Methods or arrangements for sensing record carriers, e.g. for reading patterns
- G06K7/10—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation
- G06K7/14—Methods or arrangements for sensing record carriers, e.g. for reading patterns by electromagnetic radiation, e.g. optical sensing; by corpuscular radiation using light without selection of wavelength, e.g. sensing reflected white light
- G06K7/1404—Methods for optical code recognition
- G06K7/146—Methods for optical code recognition the method including quality enhancement steps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Definitions
- the present disclosure generally relates to a training method of a deep residual network for removing moiré in a two-dimensional code.
- Multiphase component layer decomposition technology is used to remove the moiré in the image captured by the camera; the wavelet domain filtering method is used to remove the moiré in the scanned image; The image restoration method realizes the moiré removal.
- LDPC Multiphase component layer decomposition technology
- Multi-phase component layer decomposition technology is used to solve the problem of moiré removal, the removal effect is not obvious and the details of the original image will be smoothed, that is, the restored image has a certain degree of distortion.
- image restoration methods such as reference denoising adaptive encoder to remove moiré, while removing the moiré in the high-frequency range, the high-frequency components of the original image will also be removed. Because the image loses high-frequency information, there will also be a certain amount of blur. None of the above technologies can completely avoid the moiré removal without affecting the original image content.
- the present invention aims to provide a training method for a deep residual network that can effectively remove moiré in a two-dimensional code.
- the present disclosure provides a method for training a deep residual network for removing moiré in a two-dimensional code, which includes: preparing an original two-dimensional code image with moiré; preparing to process the original two-dimensional code
- the network device includes a preprocessing module, a first residual module, a second residual module, and a third residual module; the original two-dimensional code image is input into the preprocessing module, and the original
- the two-dimensional code image is blurred to increase the pixels in the two-dimensional code area in the simulated two-dimensional code image, and then down-sampling is performed to form a pre-processed image with reduced image; then the pre-processed image is input to the first residual image
- the difference module performs up-sampling processing to form a first output image whose image size is enlarged to the size of the original two-dimensional code image; then the first output image is input to the second residual module to form a restoration of the first output image.
- the residual module performs a purification process to form a moiré-removed image.
- the pre-processed image is processed by the first residual module, the second residual module and the third residual module in the deep residual network, which can conveniently and effectively remove the moire in the original two-dimensional code image. Pattern.
- the first residual module includes a convolution kernel connected in sequence with a size of 3 ⁇ 3 and 64
- the third convolutional layer with a convolution kernel size of 3 ⁇ 3 and 256 feature maps, a fourth convolution layer with a convolution kernel size of 1 ⁇ 1 and 3 feature maps, and a tanh activation layer can be removed more effectively.
- the second residual module includes 10 successively connected layers consisting of a convolution kernel with a size of 5 ⁇ 5.
- the convolutional layer with 64 feature maps and the second ReLU activation layer are concatenated into the first combined layer and 10 layers are activated by the convolutional layer with a convolution kernel size of 3 ⁇ 3 and 64 feature maps and the third ReLU
- the second combination layer formed by series connection. In this way, it is possible to restore the image information lost in the first output image.
- the third residual module includes a convolution kernel connected in sequence with a size of 3 ⁇ 3 and 128
- the original two-dimensional code image is a synthesized simulated two-dimensional code image.
- the original two-dimensional code image can be generated more conveniently.
- the formation of the simulated two-dimensional code image includes the following steps: re-sampling the input image to generate a A mosaic composed of RGB pixels and displayed on the display; random projection transformation is used to simulate the different relative positions and directions between the display and the camera; the radiation distortion function is used to simulate the distortion of the camera lens; a flat top is used Gaussian filter to simulate anti-aliasing filtering; re-sampling the input image to simulate the input of the camera sensor; adding Gaussian noise to the input image to simulate sensor noise; demosaicing processing; using a denoising filter for processing Denoising processing; compressing the input image; and outputting a decompressed image to form the simulated two-dimensional code image.
- the simulated two-dimensional code image can be generated more conveniently.
- the two-dimensional code image on the display corresponding to the input image uses the same The projection transformation and the lens distortion function are processed. As a result, it is possible to easily obtain the two-dimensional code image on the display corresponding to the input image one-to-one.
- the mean square error function is used.
- M and N are the height and width of the simulated two-dimensional code image
- H(G(I)) is the two-dimensional code image output by the deep residual network
- J is and The two-dimensional code image on the display corresponding to the simulated two-dimensional code image is stored, and the model and parameters of the deep residual network when the training loss is the least are stored. Therefore, the deep residual network can be effectively trained.
- the training method of the deep residual network for removing the moiré of the two-dimensional code involved in the present disclosure optionally, it also includes a real two-dimensional code image with moiré, based on the minimum training loss.
- the model and parameters of the deep residual network and use the real two-dimensional code image to perform a migration learning operation on the deep residual network. As a result, the deep residual network can be trained more effectively.
- the angle transformation is performed on the real two-dimensional code image to obtain the same image as the real two-dimensional code image.
- a corresponding two-dimensional code image on the display In this way, it is possible to obtain the two-dimensional code image on the display corresponding to the real two-dimensional code image one-to-one.
- the pre-processed image is processed by the first residual module, the second residual module and the third residual module in the deep residual network, which can conveniently and effectively remove the moire in the original two-dimensional code image. Pattern.
- FIG. 1 is a schematic diagram showing the modules of the deep residual network involved in this embodiment.
- FIG. 2 is a schematic diagram showing the specific structure of the deep residual network involved in this embodiment.
- FIG. 3 is a schematic flowchart showing a method for training a deep residual network for removing moiré in a two-dimensional code according to this embodiment.
- FIG. 4(a) is a schematic diagram showing the synthesized simulated two-dimensional code image related to this embodiment
- FIG. 4(b) is a diagram showing the de-moiré image related to FIG. 4(a) related to this embodiment. .
- FIG. 5(a) is a schematic diagram showing a captured image according to this embodiment
- FIG. 5(b) is a schematic diagram showing a reconstructed image according to this embodiment
- FIG. 5(c) is a schematic diagram showing the image The embodiment relates to the de-moiré image of FIG. 5(a).
- Network device preprocessing module...10, first residual module...20, second residual module...30, first combination layer...31, second combination layer...32, third residual module...40.
- FIG. 1 is a schematic diagram showing a module of a network device 1 related to this embodiment.
- FIG. 2 is a schematic diagram showing a specific structure of the network device 1 according to this embodiment.
- FIG. 3 is a schematic flowchart showing a training method for removing moiré from a two-dimensional code based on the network device 1 according to this embodiment.
- the present disclosure designs a training method of a deep residual network based on the two-dimensional code moiré removal of the network device 1.
- the network device 1 of the present disclosure may include a preprocessing module 10, a first residual module 20, a second residual module 30, and a third residual module 40, the preprocessing module 10, the first residual module The module 20, the second residual module 30, and the third residual module 40 are connected in series in sequence.
- the training method of the deep residual network for removing the moiré of the two-dimensional code of the present disclosure includes the following steps: preparing an original two-dimensional code image I with moiré (step S100); preparing a network device for processing the original two-dimensional code ,
- the network device includes a preprocessing module, a first residual module, a second residual module, and a third residual module (step S200);
- the original two-dimensional code image I is input into the preprocessing module 10
- the original two-dimensional code image I perform blur processing to increase the pixels in the QR code area in the simulated two-dimensional code image, and then perform down-sampling processing to form a reduced image preprocessed image I'(step S300); then input the preprocessed image I'to the first
- the residual module 20 performs up-sampling processing to form a first output image whose image size is enlarged to the size of the original two-dimensional code image I (step S400); then the first output image is input to the second residual module 30 to form a restored first
- step S500 Output the second output image of the missing image information of the image (step S500); and then perform feature fusion of the second output image and the original two-dimensional code image I to form a feature fusion image, and then input the feature fusion image to the third residual module 40 performs a purification process to form a moiré-removed image (step S600).
- the first residual module 20, the second residual module 30, and the third residual module 40 in the network device 1 process the preprocessed image I', which can easily and effectively remove the original two-dimensional code. Moiré in image I.
- the lost image information of the first output image may be, for example, the lack of pixel points, the loss of pixel coordinates, and the like.
- the first residual module 20 may include a first convolutional layer with a convolution kernel size of 3 ⁇ 3 and 64 feature maps, a first ReLU (modified linear unit) activation layer, and a concatenated 16 residual blocks, a second convolutional layer with a convolution kernel size of 3 ⁇ 3 and 64 feature maps, a batch normalization layer (Batch Norm), a convolution kernel size of 3 ⁇ 3 and 256 features
- the first residual module 20 can be used to remove moiré in the pre-processed image I′ that is significantly different from the pre-processed image I′, such as some colored band-shaped moiré.
- the size of the convolution kernel and the number of feature maps in the first residual module 20 are not fixed, and can be adjusted according to different network devices 1 to be trained, and there is no limitation here.
- the second residual module 30 may include 10 successively connected layers, a first combination layer composed of a convolutional layer with a convolution kernel size of 5 ⁇ 5 and 64 feature maps and a second ReLU activation layer in series.
- the 31 and 10 layers are a second combination layer 32 composed of a convolutional layer with a convolution kernel size of 3 ⁇ 3 and 64 feature maps and a third ReLU activation layer in series.
- each layer of the 10-layer first combination layer 31 can be composed of a convolutional layer with a size of 5 ⁇ 5 and 64 feature maps and a second ReLU activation layer in series, and then 10 layers
- each layer of the 10-layer second combination layer 32 can be composed of a convolutional layer with a size of 3 ⁇ 3 and 64 feature maps and a layer
- the third ReLU activation layer is connected in series, and then each of the 10 layers is connected in series to form the second combined layer 32.
- the second residual module 30 can be used to restore the lost detail information of the first output image.
- the size of the convolution kernels and the number of feature maps in the first combination layer 31 and the second combination layer 32 are not fixed, and can be adjusted according to the network device 1 to be trained, and there is no limitation here.
- the third residual module 40 may include a fifth convolution layer with a convolution kernel size of 3 ⁇ 3 and 128 feature maps, a fourth ReLU activation layer, and a convolution kernel size of 3 ⁇ 3. 3 and a sixth convolutional layer with 3 feature maps.
- the input to the third residual module 40 is the second output image and the original two-dimensional code image I, that is, the second output image and the original two-dimensional code image I are feature-fused and input to the third residual module 40.
- two convolutional layers can be used to purify the second output image and the original two-dimensional code image I to further remove some comparative rules and moiré that is not easily separated from the original two-dimensional code image I.
- the final moiré-removed de-moiré image is obtained.
- FIG. 4(a) is a schematic diagram showing a synthesized pseudo two-dimensional code image according to this embodiment
- FIG. 4(b) is a diagram showing a de-moiré image related to figure (a) according to this embodiment.
- the original two-dimensional code image I is a synthesized simulated two-dimensional code image. As a result, the original two-dimensional code image I can be generated more conveniently.
- the formation of a simulated two-dimensional code image includes the following steps: re-sampling the input image to generate a mosaic composed of RGB pixels, and display it on the display; random projection transformation to simulate the difference between the display and the camera The relative position and direction of the camera; use the radiation distortion function to simulate the distortion of the camera lens; use a flat-top Gaussian filter to simulate anti-aliasing filtering; resample the input image to simulate the input of the camera sensor; add Gaussian noise to the input image To simulate sensor noise; demosaicing; using denoising filters for denoising; compressing the input image; and outputting the decompressed image to form a simulated two-dimensional code image.
- the simulated two-dimensional code image can be generated more conveniently.
- bayer CFA may be used to resample the input image to simulate the input of the camera sensor.
- the input image may be compressed in JPEG, TIFF, RAW, and other formats.
- the two-dimensional code image on the display corresponding to the input image can be processed using the same projection transformation and lens distortion function. Since it is necessary to simulate a one-to-one correspondence between the two-dimensional code image and the two-dimensional code image on the display to train the network device 1, the same projection transformation and lens distortion function can be used to process the two-dimensional code image on the display .
- the mean square error function can be used Train the network device 1 as a loss function, where M and N are the height and width of the simulated two-dimensional code image, H(G(I')) is the two-dimensional code image output by the network device 1, and J is the same as the simulated two-dimensional code image Corresponding to the two-dimensional code image on the display, and save the model and parameters of the network device 1 when the training loss is the smallest. As a result, the network device 1 can be effectively trained.
- the loss function can also be implemented by a loss function, or a cross-entropy loss function, an exponential loss function, or a Hinge loss function (SVM), which can be selected according to actual conditions in specific applications, and there is no limitation here.
- a loss function or a cross-entropy loss function, an exponential loss function, or a Hinge loss function (SVM), which can be selected according to actual conditions in specific applications, and there is no limitation here.
- SVM Hinge loss function
- the simulated two-dimensional code image can be input to the network device 1 as the original image I for training, so as to obtain the corresponding moiré image (as shown in FIG. 4(b)).
- the network device 1 As a result, it is possible to reduce the difficulty caused by a real two-dimensional code image using a large amount of data.
- FIG. 5(a) is a schematic diagram showing a captured image according to this embodiment
- FIG. 5(b) is a schematic diagram showing a reconstructed image according to this embodiment
- FIG. 5(c) is a schematic diagram showing the image The embodiment relates to the de-moiré image of FIG. 5(a).
- the network device 1 model proposed in the present disclosure can be divided into two stages: (a) A large number of simulated two-dimensional code images with synthetic moiré are used to pair the network. Carry out pre-training (refer to the training of the simulated two-dimensional code image as above), so that the network can play the performance of removing moiré for the two-dimensional code. (b) Considering the characteristics of the moiré in the real environment, a relatively small amount of real-photographed real two-dimensional code images with moiré can be used in training for migration learning (Fine-tune) based on the network device 1, so that the network can target the real image The moiré pattern has a more ideal removal effect. The network device 1 trained in two stages can achieve good de-moiring performance on real-photographed two-dimensional code images with moiré.
- the present disclosure may also include a real two-dimensional code image with moiré, based on the model and parameters of the network device 1 when the training loss is minimal, and use the real two-dimensional code image to perform the network device 1 Transfer learning operation.
- the network device 1 can be trained more effectively.
- the migration learning operation is detailed below.
- the network After pre-training the network with the simulated two-dimensional code image, the network has the function of removing moiré, but the simulated two-dimensional code image cannot fully simulate the situation in the real environment, so it is necessary to use the real two-dimensional code image captured
- the network is fine-tuned to enable the network to exert practical effects while enhancing the robustness of the network.
- the model and parameters at the time of the minimum loss during the pre-training process are saved, and on this basis, the real two-dimensional code image with moiré is used to perform the migration learning operation on the network.
- the network device 1 Since the network device 1 is a supervised learning method, the input and output images of the network should have no other difference except whether there is moiré or not. However, it is difficult to find the matching label for the real two-dimensional code image that is actually shot, so it can be used
- the fixed modules in the two-dimensional code: the positioning pattern, the calibration pattern and the detection pattern are used for angular transformation of the captured two-dimensional code image so that it can correspond one-to-one with the two-dimensional code image on the display.
- the reconstructed image (as shown in Figure 5(b)) after the angle transformation is used as the network input of the migration learning stage, and the corresponding two-dimensional code image on the screen is used as the network output.
- the transfer learning process still uses the mean square error (MSE) function as the loss function. After the transfer learning operation of the real data (real two-dimensional code image), the network can have a more ideal removal effect on the moiré in the real shooting environment.
- MSE mean square error
- the batchsize can be 4, the number of iterations can be 12 ⁇ 10 5 times, the learning rate can be 10 -5 , and the optimizer can be Adam.
- the deep learning framework tensorflow can be used to implement the network device 1.
- the training and testing environment of the present disclosure may be a server equipped with NVIDIA Tesla P100 GPU and Intel Xeon E5-2695 v4 CPU.
- the Dell U2414H display can be used to display different QR code images, and three smart phones (Apple iPhone 8plus, Huawei MI 8, Meizu m1 metal, etc.) can be used to capture the QR code images.
- Fig. 5(a) in some examples, 10,000 real moiré real two-dimensional code images can be collected to perform migration learning operations on the network device 1, so that the network can perform better demolition The effect of the pattern.
- the corresponding reconstructed image is shown in Figure 5(b).
- a real two-dimensional code image taken by a mobile phone and other devices can be used as input, and the reconstructed image can be obtained by performing perspective transformation, angle transformation, and other operations to obtain a reconstructed image, which can be used as the input of the network device 1 for migration learning .
- the trained network device 1 can have a more ideal removal effect on the two-dimensional code image taken in the real environment.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Electromagnetism (AREA)
- Toxicology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Image Processing (AREA)
Abstract
A training method for a deep residual network for removing a moire pattern of a two-dimensional code, comprising: preparing an original two-dimensional code image having a moire pattern (S100); inputting the original two-dimensional code image into a preprocessing module, and then performing down-sampling processing to form a zoomed-out preprocessed image (S300); inputting the preprocessed image into a first residual module for up-sampling processing to form a first output image of which the image size is zoomed in to the size of the original two-dimensional code image (S400); inputting the first output image into a second residual module to form a second output image for recovering lost image information of the first output image (S500); and performing feature fusion on the second output image and the original two-dimensional code image to form a feature fusion image, and inputting the feature fusion image into a third residual module to perform purification processing to form a moire pattern-removed image in which a moire pattern is removed (S600). Therefore, the moire pattern in an original two-dimensional code image can be removed more effectively.
Description
本公开大体涉及一种二维码去摩尔纹的深度残差网络的训练方法。The present disclosure generally relates to a training method of a deep residual network for removing moiré in a two-dimensional code.
目前,去除图像中的摩尔纹常用到三种方法,采用多相分量的层分解技术(LDPC),来针对相机拍摄所得图像去摩尔纹;采用小波域滤波的方法对扫描图像去摩尔纹;采用图像还原方法实现去摩尔纹。At present, there are three commonly used methods to remove moiré in images. Multiphase component layer decomposition technology (LDPC) is used to remove the moiré in the image captured by the camera; the wavelet domain filtering method is used to remove the moiré in the scanned image; The image restoration method realizes the moiré removal.
上述三种方法,都可以不同程度实现去除图像中的摩尔纹,但同时也存在以下三方面的局限性:The above three methods can remove the moiré in the image to varying degrees, but they also have the following three limitations:
(1)降低了去摩尔纹技术在通用性上的优势。采用多相分量的层分解技可实现相机拍摄图像上摩尔纹的去除。但实际上这项技术只能去除少量的摩尔纹干扰,不能对存在大规模摩尔纹干扰的图像中发挥明显作用,尤其不能对摩尔纹彩色带起作用,且最终会平滑图像细节。采用小波域滤波可对扫描图像去摩尔纹,但针对的摩尔纹必须是网状结构的。采用传统的图像还原方法或者对DnCNN(去噪卷积神经网络)进行改进实现去摩尔纹,由于这些模型并非专门针对去摩尔纹问题进行设计,只能实现次优的性能。实际上的摩尔纹是各向异性且随机变化的,上述技术只能针对特定的摩尔纹式样解决去摩尔纹问题,存在一定的局限性。(1) Reduce the versatility of the moiré removal technology. The use of multi-phase component layer decomposition technology can achieve the removal of moiré on the image taken by the camera. But in fact, this technology can only remove a small amount of moiré interference, and cannot play a significant role in images with large-scale moiré interference, especially for moiré color bands, and will eventually smooth image details. The wavelet domain filtering can remove the moiré in the scanned image, but the moiré must have a mesh structure. Using traditional image restoration methods or improving DnCNN (Denoising Convolutional Neural Network) to achieve moiré removal, because these models are not specifically designed for moiré removal problems, they can only achieve sub-optimal performance. In fact, the moiré pattern is anisotropic and varies randomly. The above-mentioned technology can only solve the problem of removing the moiré pattern for a specific moiré pattern, which has certain limitations.
(2)无法彻底避免对原图像造成的影响。采用多相分量的层分解技术解决去摩尔纹问题,去除效果不明显且会平滑原图像的细节,即恢复出来的图像存在一定的失真。采用图像还原方法例如参考去噪的自适应编码器进行去摩尔纹,在去除高频带范围的摩尔纹的同时,也会去除原图像的高频成分。由于图像丢失高频信息,也会存在一定的模糊。上述技术均无法彻底避免在去摩尔纹的同时不对原图像内容造成影响。(2) The impact on the original image cannot be completely avoided. Multi-phase component layer decomposition technology is used to solve the problem of moiré removal, the removal effect is not obvious and the details of the original image will be smoothed, that is, the restored image has a certain degree of distortion. Using image restoration methods such as reference denoising adaptive encoder to remove moiré, while removing the moiré in the high-frequency range, the high-frequency components of the original image will also be removed. Because the image loses high-frequency information, there will also be a certain amount of blur. None of the above technologies can completely avoid the moiré removal without affecting the original image content.
(3)增加了使用复杂度。由于摩尔纹是各向异性,且非均匀的。 对于一副摩尔纹多样的图像,我们得通过多种现有技术的结合,使它们分别针对不同的摩尔纹式样发挥作用。这种做法增加了去摩尔纹问题的复杂度,需要多个步骤才能实现去摩尔纹,且无法实现较好的去除效果。(3) Increased use complexity. Because the moiré is anisotropic and non-uniform. For an image with various moiré patterns, we have to combine a variety of existing technologies to make them work for different moiré patterns. This approach increases the complexity of the moiré removal problem, requires multiple steps to achieve the moiré removal, and cannot achieve a better removal effect.
发明内容Summary of the invention
本发明有鉴于上述现有的状况,其目的在于提供一种能够有效去除二维码中摩尔纹的深度残差网络的训练方法。In view of the above-mentioned existing conditions, the present invention aims to provide a training method for a deep residual network that can effectively remove moiré in a two-dimensional code.
为此,本公开提供一种二维码去摩尔纹的深度残差网络的训练方法,其包括:准备带有摩尔纹的原始二维码图像;准备用于对所述原始二维码进行处理的网络装置,所述网络装置包括预处理模块、第一残差模块、第二残差模块以及第三残差模块;将所述原始二维码图像输入所述预处理模块,对所述原始二维码图像进行模糊处理以增加仿真二维码图像中二维码区域的像素,接着进行下采样处理以形成图像缩小的预处理图像;接着将所述预处理图像输入至所述第一残差模块进行上采样处理以形成图像大小放大至所述原始二维码图像大小的第一输出图像;接着将所述第一输出图像输入至所述第二残差模块以形成恢复所述第一输出图像丢失的图像信息的第二输出图像;并且接着将所述第二输出图像和所述原始二维码图像进行特征融合以形成特征融合图像,接着将所述特征融合图像输入所述第三残差模块进行提纯处理以形成去除摩尔纹的去摩尔纹图像。To this end, the present disclosure provides a method for training a deep residual network for removing moiré in a two-dimensional code, which includes: preparing an original two-dimensional code image with moiré; preparing to process the original two-dimensional code The network device includes a preprocessing module, a first residual module, a second residual module, and a third residual module; the original two-dimensional code image is input into the preprocessing module, and the original The two-dimensional code image is blurred to increase the pixels in the two-dimensional code area in the simulated two-dimensional code image, and then down-sampling is performed to form a pre-processed image with reduced image; then the pre-processed image is input to the first residual image The difference module performs up-sampling processing to form a first output image whose image size is enlarged to the size of the original two-dimensional code image; then the first output image is input to the second residual module to form a restoration of the first output image. Output the second output image of the missing image information of the image; and then perform feature fusion of the second output image and the original two-dimensional code image to form a feature fusion image, and then input the feature fusion image into the third The residual module performs a purification process to form a moiré-removed image.
在本公开中,通过深度残差网络中的第一残差模块、第二残差模块和第三残差模块对预处理图像进行处理,能够方便且有效的去除原始二维码图像中的摩尔纹。In the present disclosure, the pre-processed image is processed by the first residual module, the second residual module and the third residual module in the deep residual network, which can conveniently and effectively remove the moire in the original two-dimensional code image. Pattern.
另外,在本公开所涉及的二维码去摩尔纹的深度残差网络的训练方法中,可选地,所述第一残差模块包括依次连接的卷积核大小为3×3且具有64个特征映射的第一卷积层、第一ReLU激活层、串联的16个残差块、卷积核大小为3×3且具有64个特征映射的第二卷积层、批归一化层、卷积核大小为3×3且具有256个特征映射的第三卷积层、卷积核大小为1×1且具有3个特征映射的第四卷积层以及tanh激活层。由此,能够更有效地去除预处理图像中的摩尔纹。In addition, in the training method of the deep residual network for removing the moiré of the two-dimensional code involved in the present disclosure, optionally, the first residual module includes a convolution kernel connected in sequence with a size of 3×3 and 64 The first convolutional layer with feature maps, the first ReLU activation layer, the 16 residual blocks in series, the second convolutional layer with 64 feature maps and the batch normalization layer with a convolution kernel size of 3×3 , The third convolutional layer with a convolution kernel size of 3×3 and 256 feature maps, a fourth convolution layer with a convolution kernel size of 1×1 and 3 feature maps, and a tanh activation layer. As a result, the moiré in the preprocessed image can be removed more effectively.
另外,在本公开所涉及的二维码去摩尔纹的深度残差网络的训练方法中,可选地,所述第二残差模块包括依次连接的10层由卷积核大小为5×5且具有64个特征映射的卷积层和第二ReLU激活层串联成的第一组合层以及10层由卷积核大小为3×3且具有64个特征映射的卷积层和第三ReLU激活层串联成的第二组合层。由此,能够回复第一输出图像丢失的图像信息。In addition, in the training method of the deep residual network for removing the moiré of the two-dimensional code involved in the present disclosure, optionally, the second residual module includes 10 successively connected layers consisting of a convolution kernel with a size of 5×5. And the convolutional layer with 64 feature maps and the second ReLU activation layer are concatenated into the first combined layer and 10 layers are activated by the convolutional layer with a convolution kernel size of 3×3 and 64 feature maps and the third ReLU The second combination layer formed by series connection. In this way, it is possible to restore the image information lost in the first output image.
另外,在本公开所涉及的二维码去摩尔纹的深度残差网络的训练方法中,可选地,所述第三残差模块包括依次连接的卷积核大小为3×3且具有128个特征映射的第五卷积层、第四ReLU激活层、卷积核大小为3×3且具有3个特征映射的第六卷积层。由此,能够有效的去除第二输出图像中的摩尔纹。In addition, in the training method of the deep residual network for removing the moiré of the two-dimensional code involved in the present disclosure, optionally, the third residual module includes a convolution kernel connected in sequence with a size of 3×3 and 128 The fifth convolutional layer with three feature maps, the fourth ReLU activation layer, and the sixth convolutional layer with a convolution kernel size of 3×3 and three feature maps. Thus, the moiré in the second output image can be effectively removed.
另外,在本公开所涉及的二维码去摩尔纹的深度残差网络的训练方法中,可选地,所述原始二维码图像为合成的仿真二维码图像。由此,能够更方便地生成原始二维码图像。In addition, in the training method of the deep residual network for removing the moiré of the two-dimensional code involved in the present disclosure, optionally, the original two-dimensional code image is a synthesized simulated two-dimensional code image. As a result, the original two-dimensional code image can be generated more conveniently.
另外,在本公开所涉及的二维码去摩尔纹的深度残差网络的训练方法中,可选地,所述仿真二维码图像的形成包括如下步骤:对输入图像重新采样以生成一个由RGB像素组成的嵌合体,并显示在显示器上;随机作投影变换来模拟所述显示器和相机之间不同的相对位置和方向;使用放射失真函数来模拟所述相机的镜头的失真;采用平顶高斯滤波器来模拟抗混叠滤波;对所述输入图像重新采样以模拟所述相机传感器的输入;在所述输入图像上增加高斯噪声来模拟传感器噪声;去马赛克处理;采用去噪滤波器进行去噪处理;压缩所述输入图像;以及输出解压缩图像以形成所述仿真二维码图像。由此,能够更方便的生成仿真二维码图像。In addition, in the training method of the deep residual network for removing the moiré of the two-dimensional code involved in the present disclosure, optionally, the formation of the simulated two-dimensional code image includes the following steps: re-sampling the input image to generate a A mosaic composed of RGB pixels and displayed on the display; random projection transformation is used to simulate the different relative positions and directions between the display and the camera; the radiation distortion function is used to simulate the distortion of the camera lens; a flat top is used Gaussian filter to simulate anti-aliasing filtering; re-sampling the input image to simulate the input of the camera sensor; adding Gaussian noise to the input image to simulate sensor noise; demosaicing processing; using a denoising filter for processing Denoising processing; compressing the input image; and outputting a decompressed image to form the simulated two-dimensional code image. As a result, the simulated two-dimensional code image can be generated more conveniently.
另外,在本公开所涉及的二维码去摩尔纹的深度残差网络的训练方法中,可选地,与所述输入图像相对应的所述显示器上的二维码图像使用相同的所述投影变换和所述镜头失真函数进行处理。由此,能够方便得到与输入图像一一对应的显示器上的二维码图像。In addition, in the training method of the deep residual network for removing the moiré of the two-dimensional code involved in the present disclosure, optionally, the two-dimensional code image on the display corresponding to the input image uses the same The projection transformation and the lens distortion function are processed. As a result, it is possible to easily obtain the two-dimensional code image on the display corresponding to the input image one-to-one.
另外,在本公开所涉及的二维码去摩尔纹的深度残差网络的训练方法中,可选地,采用均方差函数
作为损失函数训 练所述深度残差网络,其中M和N是所述仿真二维码图像的高和宽,H(G(I))是深度残差网络输出的二维码图像,J是与所述仿真二维码图像相对应的显示器上的二维码图像,且保存训练的损失最小时的所述深度残差网络的模型和参数。由此,能够有效的训练该深度残差网络。
In addition, in the training method of the deep residual network for removing the moiré pattern of the two-dimensional code involved in the present disclosure, optionally, the mean square error function is used Train the deep residual network as a loss function, where M and N are the height and width of the simulated two-dimensional code image, H(G(I)) is the two-dimensional code image output by the deep residual network, and J is and The two-dimensional code image on the display corresponding to the simulated two-dimensional code image is stored, and the model and parameters of the deep residual network when the training loss is the least are stored. Therefore, the deep residual network can be effectively trained.
另外,在本公开所涉及的二维码去摩尔纹的深度残差网络的训练方法中,可选地,还包括带有摩尔纹的真实二维码图像,基于训练的损失最小时的所述深度残差网络的模型和参数,并利用所述真实二维码图像对所述深度残差网络进行迁移学习操作。由此,能够更有效地训练该深度残差网络。In addition, in the training method of the deep residual network for removing the moiré of the two-dimensional code involved in the present disclosure, optionally, it also includes a real two-dimensional code image with moiré, based on the minimum training loss. The model and parameters of the deep residual network, and use the real two-dimensional code image to perform a migration learning operation on the deep residual network. As a result, the deep residual network can be trained more effectively.
另外,在本公开所涉及的二维码去摩尔纹的深度残差网络的训练方法中,可选地,对所述真实二维码图像进行角度变换,得到与所述真实二维码图像一一对应的显示器上的二维码图像。由此,能够得到与真实二维码图像一一对应的显示器上的二维码图像。In addition, in the training method of the deep residual network for removing the moiré of the two-dimensional code involved in the present disclosure, optionally, the angle transformation is performed on the real two-dimensional code image to obtain the same image as the real two-dimensional code image. A corresponding two-dimensional code image on the display. In this way, it is possible to obtain the two-dimensional code image on the display corresponding to the real two-dimensional code image one-to-one.
在本公开中,通过深度残差网络中的第一残差模块、第二残差模块和第三残差模块对预处理图像进行处理,能够方便且有效的去除原始二维码图像中的摩尔纹。In the present disclosure, the pre-processed image is processed by the first residual module, the second residual module and the third residual module in the deep residual network, which can conveniently and effectively remove the moire in the original two-dimensional code image. Pattern.
现在将仅通过参考附图的例子进一步详细地解释本公开的实施例,其中:The embodiments of the present disclosure will now be explained in further detail only by referring to the examples of the accompanying drawings, in which:
图1是示出了本实施方式所涉及的深度残差网络的模块示意图。FIG. 1 is a schematic diagram showing the modules of the deep residual network involved in this embodiment.
图2是示出了本实施方式所涉及的深度残差网络的具体结构示意图。FIG. 2 is a schematic diagram showing the specific structure of the deep residual network involved in this embodiment.
图3是示出了本实施方式所涉及的二维码去摩尔纹的深度残差网络的训练方法的流程示意图。FIG. 3 is a schematic flowchart showing a method for training a deep residual network for removing moiré in a two-dimensional code according to this embodiment.
图4(a)是示出了本实施方式所涉及的合成的仿真二维码图像的示意图,图4(b)是示出了本实施方式所涉及的关于图4(a)去摩尔纹图像。FIG. 4(a) is a schematic diagram showing the synthesized simulated two-dimensional code image related to this embodiment, and FIG. 4(b) is a diagram showing the de-moiré image related to FIG. 4(a) related to this embodiment. .
图5(a)是示出了本实施方式所涉及的拍摄图像的示意图,图5(b)是示出了本实施方式所涉及的重建图像的示意图,图5(c)是示出了本实施方式所涉及的关于图5(a)的去摩尔纹图像。FIG. 5(a) is a schematic diagram showing a captured image according to this embodiment, FIG. 5(b) is a schematic diagram showing a reconstructed image according to this embodiment, and FIG. 5(c) is a schematic diagram showing the image The embodiment relates to the de-moiré image of FIG. 5(a).
符号说明:Symbol Description:
网络装置…1,预处理模块…10,第一残差模块…20,第二残差模块…30,第一组合层…31,第二组合层…32,第三残差模块…40。Network device...1, preprocessing module...10, first residual module...20, second residual module...30, first combination layer...31, second combination layer...32, third residual module...40.
以下,参考附图,详细地说明本公开的优选实施方式。在下面的说明中,对于相同的部件赋予相同的符号,省略重复的说明。另外,附图只是示意性的图,部件相互之间的尺寸的比例或者部件的形状等可以与实际的不同。Hereinafter, with reference to the drawings, preferred embodiments of the present disclosure will be described in detail. In the following description, the same symbols are assigned to the same components, and repeated descriptions are omitted. In addition, the drawings are only schematic diagrams, and the ratio of dimensions between components or the shapes of components may be different from actual ones.
图1是示出了本实施方式所涉及的网络装置1的模块示意图。图2是示出了本实施方式所涉及的网络装置1的具体结构示意图。图3是示出了本实施方式所涉及的基于网络装置1的二维码去摩尔纹的训练方法的流程示意图。FIG. 1 is a schematic diagram showing a module of a network device 1 related to this embodiment. FIG. 2 is a schematic diagram showing a specific structure of the network device 1 according to this embodiment. FIG. 3 is a schematic flowchart showing a training method for removing moiré from a two-dimensional code based on the network device 1 according to this embodiment.
采用数字相机或智能手机等对光电显示器(或显示屏)进行拍摄可方便我们记录和传输数据,然而,由于相机传感器和显示器设备之间的像素格子不能完全匹配,拍摄所得的图像往往存在摩尔纹样式的图样干扰。现有的去摩尔纹技术大都针对自然图像进行处理,且效果不是特别理想。基于图像二维码构建显示-拍摄通信信道是一个热点问题,然而摩尔纹失真严重阻碍了该类通信***的有效性。经调研发现目前尚未有针对二维码提出的去摩尔纹技术。Using digital cameras or smartphones to take photos of photoelectric displays (or display screens) can facilitate us to record and transmit data. However, because the pixel grid between the camera sensor and the display device cannot be completely matched, the captured images often have moiré patterns. Pattern interference. The existing de-moiring technologies mostly process natural images, and the effect is not particularly ideal. It is a hot issue to construct a display-shooting communication channel based on the image two-dimensional code, but the moiré distortion seriously hinders the effectiveness of this type of communication system. After investigation, it is found that there is no moiré removal technology proposed for two-dimensional codes.
针对上述问题,本公开设计了一个基于网络装置1的二维码去摩尔纹的深度残差网络的训练方法。参照图1至图3,本公开的网络装置1可以包括预处理模块10、第一残差模块20、第二残差模块30以及第三残差模块40,预处理模块10、第一残差模块20、第二残差模块30以及第三残差模块40依次串联。In view of the above-mentioned problems, the present disclosure designs a training method of a deep residual network based on the two-dimensional code moiré removal of the network device 1. 1 to 3, the network device 1 of the present disclosure may include a preprocessing module 10, a first residual module 20, a second residual module 30, and a third residual module 40, the preprocessing module 10, the first residual module The module 20, the second residual module 30, and the third residual module 40 are connected in series in sequence.
本公开二维码去摩尔纹的深度残差网络的训练方法包括如下步骤:准备带有摩尔纹的原始二维码图像I(步骤S100);准备用于对原始二维码进行处理的网络装置,该网络装置包括预处理模块、第一残差模块、第二残差模块以及第三残差模块(步骤S200);将原始二维码图像I输入预处理模块10,对原始二维码图像I进行模糊处理以增加仿真二维码图像中二维码区域的像素,接着进行下采样处理以形成图像缩小 的预处理图像I'(步骤S300);接着将预处理图像I'输入至第一残差模块20进行上采样处理以形成图像大小放大至原始二维码图像I大小的第一输出图像(步骤S400);接着将第一输出图像输入至第二残差模块30以形成恢复第一输出图像丢失的图像信息的第二输出图像(步骤S500);并且接着将第二输出图像和原始二维码图像I进行特征融合以形成特征融合图像,接着将特征融合图像输入第三残差模块40进行提纯处理以形成去除摩尔纹的去摩尔纹图像(步骤S600)。The training method of the deep residual network for removing the moiré of the two-dimensional code of the present disclosure includes the following steps: preparing an original two-dimensional code image I with moiré (step S100); preparing a network device for processing the original two-dimensional code , The network device includes a preprocessing module, a first residual module, a second residual module, and a third residual module (step S200); the original two-dimensional code image I is input into the preprocessing module 10, and the original two-dimensional code image I perform blur processing to increase the pixels in the QR code area in the simulated two-dimensional code image, and then perform down-sampling processing to form a reduced image preprocessed image I'(step S300); then input the preprocessed image I'to the first The residual module 20 performs up-sampling processing to form a first output image whose image size is enlarged to the size of the original two-dimensional code image I (step S400); then the first output image is input to the second residual module 30 to form a restored first output image. Output the second output image of the missing image information of the image (step S500); and then perform feature fusion of the second output image and the original two-dimensional code image I to form a feature fusion image, and then input the feature fusion image to the third residual module 40 performs a purification process to form a moiré-removed image (step S600).
在本公开中,通过网络装置1中的第一残差模块20、第二残差模块30和第三残差模块40对预处理图像I'进行处理,能够方便且有效的去除原始二维码图像I中的摩尔纹。In the present disclosure, the first residual module 20, the second residual module 30, and the third residual module 40 in the network device 1 process the preprocessed image I', which can easily and effectively remove the original two-dimensional code. Moiré in image I.
在一些示例中,第一输出图像丢失的图像信息例如可以是像素点的缺失、像素点坐标的丢失等。In some examples, the lost image information of the first output image may be, for example, the lack of pixel points, the loss of pixel coordinates, and the like.
在一些示例中,第一残差模块20可以包括依次连接的卷积核大小为3×3且具有64个特征映射的第一卷积层、第一ReLU(修正线性单元)激活层、串联的16个残差块、卷积核大小为3×3且具有64个特征映射的第二卷积层、批归一化层(Batch Norm)、卷积核大小为3×3且具有256个特征映射的第三卷积层、卷积核大小为1×1且具有3个特征映射的第四卷积层以及tanh(双曲正切)激活层。在这种情况下,可以利用第一残差模块20去除预处理图像I'中与预处理图像I'差异明显的摩尔纹,例如一些彩色的带状摩尔纹。当然,第一残差模块20中的卷积核的大小和特征映射的个数并不固定,可以根据训练的网络装置1的不同进行调整,此处不做限制。In some examples, the first residual module 20 may include a first convolutional layer with a convolution kernel size of 3×3 and 64 feature maps, a first ReLU (modified linear unit) activation layer, and a concatenated 16 residual blocks, a second convolutional layer with a convolution kernel size of 3×3 and 64 feature maps, a batch normalization layer (Batch Norm), a convolution kernel size of 3×3 and 256 features The mapped third convolutional layer, the fourth convolutional layer with the size of the convolution kernel of 1×1 and 3 feature maps, and the tanh (hyperbolic tangent) activation layer. In this case, the first residual module 20 can be used to remove moiré in the pre-processed image I′ that is significantly different from the pre-processed image I′, such as some colored band-shaped moiré. Of course, the size of the convolution kernel and the number of feature maps in the first residual module 20 are not fixed, and can be adjusted according to different network devices 1 to be trained, and there is no limitation here.
在一些示例中,第二残差模块30可以包括依次连接的10层由卷积核大小为5×5且具有64个特征映射的卷积层和第二ReLU激活层串联成的第一组合层31以及10层由卷积核大小为3×3且具有64个特征映射的卷积层和第三ReLU激活层串联成的第二组合层32。也即,10层第一组合层31的每层都可以由一层大小为5×5且具有64个特征映射的卷积层和一层第二ReLU激活层串联而成,然后再将10层中的每一层串联起来形成第一组合层31;同样的,10层第二组合层32的每层都可以由一层大小为3×3且具有64个特征映射的卷积层和一层第三ReLU激活层串联而成,然后再将10层中的每一层串联起来形成 第二组合层32。在这种情况下,可以利用第二残差模块30来恢复第一输出图像丢失的细节信息。当然,第一组合层31和第二组合层32中卷积核的大小和特征映射的个数并不固定,可以根据训练的网络装置1的不同进行调整,此处不做限制。In some examples, the second residual module 30 may include 10 successively connected layers, a first combination layer composed of a convolutional layer with a convolution kernel size of 5×5 and 64 feature maps and a second ReLU activation layer in series. The 31 and 10 layers are a second combination layer 32 composed of a convolutional layer with a convolution kernel size of 3×3 and 64 feature maps and a third ReLU activation layer in series. That is, each layer of the 10-layer first combination layer 31 can be composed of a convolutional layer with a size of 5×5 and 64 feature maps and a second ReLU activation layer in series, and then 10 layers In the same way, each layer of the 10-layer second combination layer 32 can be composed of a convolutional layer with a size of 3×3 and 64 feature maps and a layer The third ReLU activation layer is connected in series, and then each of the 10 layers is connected in series to form the second combined layer 32. In this case, the second residual module 30 can be used to restore the lost detail information of the first output image. Of course, the size of the convolution kernels and the number of feature maps in the first combination layer 31 and the second combination layer 32 are not fixed, and can be adjusted according to the network device 1 to be trained, and there is no limitation here.
在一些示例中,第三残差模块40可以包括依次连接的卷积核大小为3×3且具有128个特征映射的第五卷积层、第四ReLU激活层、卷积核大小为3×3且具有3个特征映射的第六卷积层。由此,能够有效的去除第二输出图像中的摩尔纹。输入第三残差模块40的为第二输出图像和原始二维码图像I,也即将第二输出图像和原始二维码图像I进行特征融合并输入第三残差模块40。在这种情况下,可以使用两个卷积层对第二输出图像和原始二维码图像I进行提纯处理,以进一步去除一些比较规律以及和原始二维码图像I不易分离的摩尔纹,以得到最终的去除摩尔纹的去摩尔纹图像。In some examples, the third residual module 40 may include a fifth convolution layer with a convolution kernel size of 3×3 and 128 feature maps, a fourth ReLU activation layer, and a convolution kernel size of 3×3. 3 and a sixth convolutional layer with 3 feature maps. Thus, the moiré in the second output image can be effectively removed. The input to the third residual module 40 is the second output image and the original two-dimensional code image I, that is, the second output image and the original two-dimensional code image I are feature-fused and input to the third residual module 40. In this case, two convolutional layers can be used to purify the second output image and the original two-dimensional code image I to further remove some comparative rules and moiré that is not easily separated from the original two-dimensional code image I. The final moiré-removed de-moiré image is obtained.
图4(a)是示出了本实施方式所涉及的合成的仿真二维码图像的示意图,图4(b)是示出了本实施方式所涉及的关于图(a)去摩尔纹图像。FIG. 4(a) is a schematic diagram showing a synthesized pseudo two-dimensional code image according to this embodiment, and FIG. 4(b) is a diagram showing a de-moiré image related to figure (a) according to this embodiment.
参照图4(a)和图4(b),在一些示例中,原始二维码图像I为合成的仿真二维码图像。由此,能够更方便地生成原始二维码图像I。4(a) and 4(b), in some examples, the original two-dimensional code image I is a synthesized simulated two-dimensional code image. As a result, the original two-dimensional code image I can be generated more conveniently.
可以理解的是,为了使本公开网络装置1更加精确地识别真实环境中的摩尔纹,理想情况下我们只能使用真实拍摄的图像和与之相对应的显示器上的图像构成的图像对来训练该网络装置1。然而获取在空间位置上完全匹配的图像对是比较困难的,网络很容易将不匹配的边缘错误识别成摩尔纹,而且真实拍摄的图像存在许多共同的问题:镜头失真、相机抖动等,这些都会严重影响到图像对的对齐。考虑到使用真实图像构建高质量的大型图像集的难度,我们采用带有真实摩尔纹的仿真二维码图像来训练网络。为了更加准确地模拟摩尔纹生成过程,我们严格执行从图像显示在LCD显示器上到使用相机拍摄图像再到相机内部的数字处理整个流程。It is understandable that, in order to make the network device 1 of the present disclosure more accurately recognize the moiré in the real environment, ideally we can only use the image pair composed of the real shot image and the image on the corresponding display for training. The network device1. However, it is more difficult to obtain a pair of images that exactly match in space. It is easy for the network to misidentify unmatched edges as moiré, and there are many common problems in real images: lens distortion, camera shake, etc. Seriously affect the alignment of the image pair. Considering the difficulty of using real images to construct high-quality large image sets, we use simulated two-dimensional code images with real moiré to train the network. In order to simulate the moiré generation process more accurately, we strictly implement the entire process from the image display on the LCD monitor to the use of the camera to capture the image and then to the digital processing inside the camera.
在一些实例中,仿真二维码图像的形成包括如下步骤:对输入图像重新采样以生成一个由RGB像素组成的嵌合体,并显示在显示器上;随机作投影变换来模拟显示器和相机之间不同的相对位置和方向;使 用放射失真函数来模拟相机的镜头的失真;采用平顶高斯滤波器来模拟抗混叠滤波;对输入图像重新采样以模拟相机传感器的输入;在输入图像上增加高斯噪声来模拟传感器噪声;去马赛克处理;采用去噪滤波器进行去噪处理;压缩输入图像;以及输出解压缩图像以形成仿真二维码图像。由此,能够更方便的生成仿真二维码图像。In some instances, the formation of a simulated two-dimensional code image includes the following steps: re-sampling the input image to generate a mosaic composed of RGB pixels, and display it on the display; random projection transformation to simulate the difference between the display and the camera The relative position and direction of the camera; use the radiation distortion function to simulate the distortion of the camera lens; use a flat-top Gaussian filter to simulate anti-aliasing filtering; resample the input image to simulate the input of the camera sensor; add Gaussian noise to the input image To simulate sensor noise; demosaicing; using denoising filters for denoising; compressing the input image; and outputting the decompressed image to form a simulated two-dimensional code image. As a result, the simulated two-dimensional code image can be generated more conveniently.
在一些示例中,可以采用bayer CFA对输入图像重新采样以模拟相机传感器的输入。In some examples, bayer CFA may be used to resample the input image to simulate the input of the camera sensor.
另外,在一些示例中,可以采用JPEG、TIFF、RAW等格式压缩输入图像。In addition, in some examples, the input image may be compressed in JPEG, TIFF, RAW, and other formats.
在一些示例中,与输入图像相对应的显示器上的二维码图像可以使用相同的投影变换和镜头失真函数进行处理。由于需要仿真二维码图像与显示器上的二维码图像一一对应的构成图像对来训练该网络装置1,因此可以使用相同的投影变换和镜头失真函数对显示器上的二维码图像进行处理。In some examples, the two-dimensional code image on the display corresponding to the input image can be processed using the same projection transformation and lens distortion function. Since it is necessary to simulate a one-to-one correspondence between the two-dimensional code image and the two-dimensional code image on the display to train the network device 1, the same projection transformation and lens distortion function can be used to process the two-dimensional code image on the display .
在一些示例中,可以采用均方差函数
作为损失函数训练网络装置1,其中M和N是仿真二维码图像的高和宽,H(G(I'))是网络装置1输出的二维码图像,J是与仿真二维码图像相对应的显示器上的二维码图像,且保存训练的损失(loss)最小时的网络装置1的模型和参数。由此,能够有效的训练该网络装置1。
In some examples, the mean square error function can be used Train the network device 1 as a loss function, where M and N are the height and width of the simulated two-dimensional code image, H(G(I')) is the two-dimensional code image output by the network device 1, and J is the same as the simulated two-dimensional code image Corresponding to the two-dimensional code image on the display, and save the model and parameters of the network device 1 when the training loss is the smallest. As a result, the network device 1 can be effectively trained.
在一些示例中,损失函数也可以用损失函数也可以用交叉熵损失函数、指数损失函数、Hinge损失函数(SVM)来实现,在具体应用中可以根据实际情况进行选择,此处不做限制。In some examples, the loss function can also be implemented by a loss function, or a cross-entropy loss function, an exponential loss function, or a Hinge loss function (SVM), which can be selected according to actual conditions in specific applications, and there is no limitation here.
在本实施方式中,如上所述,可以将仿真二维码图像作为原始图像I输入该网络装置1进行训练,以得到相应的去摩尔纹图像(如图4(b)所示)。由此,能够减小使用大量数据的真实二维码图像带来的难度。In this embodiment, as described above, the simulated two-dimensional code image can be input to the network device 1 as the original image I for training, so as to obtain the corresponding moiré image (as shown in FIG. 4(b)). As a result, it is possible to reduce the difficulty caused by a real two-dimensional code image using a large amount of data.
图5(a)是示出了本实施方式所涉及的拍摄图像的示意图,图5(b)是示出了本实施方式所涉及的重建图像的示意图,图5(c)是示出了本实施方式所涉及的关于图5(a)的去摩尔纹图像。FIG. 5(a) is a schematic diagram showing a captured image according to this embodiment, FIG. 5(b) is a schematic diagram showing a reconstructed image according to this embodiment, and FIG. 5(c) is a schematic diagram showing the image The embodiment relates to the de-moiré image of FIG. 5(a).
在一些示例中,为了实现二维码的去摩尔纹,本公开提出的网络装置1模型可以分成两个阶段进行:(a)采用大量仿真生成的带合成摩尔纹的仿真二维码图像对网络进行预训练(参照如上对仿真二维码图像的训练),使网络可以针对二维码发挥去摩尔纹的性能。(b)考虑真实环境中摩尔纹的特性,训练中可以采用相对少量的真实拍摄的带摩尔纹的真实二维码图像基于网络装置1进行迁移学习(Fine-tune),使网络针对真实图像上的摩尔纹有更加理想的去除效果。经过两个阶段训练的网络装置1,可以对带摩尔纹的真实拍摄的二维码图像实现良好的去摩尔纹性能。In some examples, in order to achieve the moiré removal of the two-dimensional code, the network device 1 model proposed in the present disclosure can be divided into two stages: (a) A large number of simulated two-dimensional code images with synthetic moiré are used to pair the network. Carry out pre-training (refer to the training of the simulated two-dimensional code image as above), so that the network can play the performance of removing moiré for the two-dimensional code. (b) Considering the characteristics of the moiré in the real environment, a relatively small amount of real-photographed real two-dimensional code images with moiré can be used in training for migration learning (Fine-tune) based on the network device 1, so that the network can target the real image The moiré pattern has a more ideal removal effect. The network device 1 trained in two stages can achieve good de-moiring performance on real-photographed two-dimensional code images with moiré.
因此,在一些示例中,本公开还可以包括带有摩尔纹的真实二维码图像,基于训练的损失最小时的网络装置1的模型和参数,并利用真实二维码图像对网络装置1进行迁移学习操作。由此,能够更有效地训练该网络装置1。Therefore, in some examples, the present disclosure may also include a real two-dimensional code image with moiré, based on the model and parameters of the network device 1 when the training loss is minimal, and use the real two-dimensional code image to perform the network device 1 Transfer learning operation. As a result, the network device 1 can be trained more effectively.
以下详述迁移学习操作。The migration learning operation is detailed below.
使用仿真二维码图像对网络进行预训练后,网络具有去除摩尔纹的功能,但仿真二维码图像并不能完全模拟出真实环境中的情况,所以需要利用真实拍摄的真实二维码图像对网络进行微调,使网络能够发挥实际效果,同时增强网络的鲁棒性。对预训练过程中损失最小时的模型和参数进行保存,在此基础上,利用真实拍摄带摩尔纹的真实二维码图像对网络进行迁移学习操作。After pre-training the network with the simulated two-dimensional code image, the network has the function of removing moiré, but the simulated two-dimensional code image cannot fully simulate the situation in the real environment, so it is necessary to use the real two-dimensional code image captured The network is fine-tuned to enable the network to exert practical effects while enhancing the robustness of the network. The model and parameters at the time of the minimum loss during the pre-training process are saved, and on this basis, the real two-dimensional code image with moiré is used to perform the migration learning operation on the network.
由于该网络装置1是有监督学习的方式,网络的输入输出图像应当除了有无摩尔纹之外无其他差别,但真实拍摄的真实二维码图像很难找到与之匹配的标签,所以可以利用二维码中固定的模块:定位图样、校准图样和探测图样对拍摄的二维码图像进行角度变换,使之可以与显示器上的二维码图像一一对应。经过角度变换后的重建图像(如图5(b)所示)作为迁移学习阶段的网络输入,而与之相对应的屏幕上的二维码图像作为网络输出。迁移学习过程仍然使用均方误差(MSE)函数作为损失函数,网络经过真实数据(真实二维码图像)的迁移学习操作,可以对真实拍摄环境中的摩尔纹有更加理想的去除效果。Since the network device 1 is a supervised learning method, the input and output images of the network should have no other difference except whether there is moiré or not. However, it is difficult to find the matching label for the real two-dimensional code image that is actually shot, so it can be used The fixed modules in the two-dimensional code: the positioning pattern, the calibration pattern and the detection pattern are used for angular transformation of the captured two-dimensional code image so that it can correspond one-to-one with the two-dimensional code image on the display. The reconstructed image (as shown in Figure 5(b)) after the angle transformation is used as the network input of the migration learning stage, and the corresponding two-dimensional code image on the screen is used as the network output. The transfer learning process still uses the mean square error (MSE) function as the loss function. After the transfer learning operation of the real data (real two-dimensional code image), the network can have a more ideal removal effect on the moiré in the real shooting environment.
在一些示例中,在具体的训练参数设置为batchsize可以为4,迭代次数可以为12×10
5次,学习率可以为10
-5,优化器可以为Adam。
In some examples, when the specific training parameters are set to batchsize, the batchsize can be 4, the number of iterations can be 12×10 5 times, the learning rate can be 10 -5 , and the optimizer can be Adam.
在预训练阶段,可以利用MATLAB对展示在显示器上600张不同的二维码进行摩尔纹仿真处理以以生成60000张512×512带有摩尔纹的仿真二维码图像,其中可以将50000张用于训练,10000张用于测试。In the pre-training stage, you can use MATLAB to perform moiré simulation processing on 600 different two-dimensional codes displayed on the display to generate 60,000 512×512 simulated two-dimensional code images with moiré, of which 50,000 can be used For training, 10,000 sheets are used for testing.
在一些示例中,可以利用深度学习框架tensorflow来实现网络装置1。本公开的训练和测试环境可以为搭载NVIDIA Tesla P100GPU和Intel Xeon E5-2695 v4CPU的服务器。In some examples, the deep learning framework tensorflow can be used to implement the network device 1. The training and testing environment of the present disclosure may be a server equipped with NVIDIA Tesla P100 GPU and Intel Xeon E5-2695 v4 CPU.
在一些示例中,可以使用Dell U2414H显示器显示不同的二维码图像,并可以采用三台智能手机(苹果iPhone 8plus、小米MI 8、魅族m1 metal等)对二维码图像进行拍摄。In some examples, the Dell U2414H display can be used to display different QR code images, and three smart phones (Apple iPhone 8plus, Xiaomi MI 8, Meizu m1 metal, etc.) can be used to capture the QR code images.
参照图5(a),在一些示例中,在迁移学习阶段,可以采集10000张真实的带摩尔纹的真实二维码图像对网络装置1进行迁移学习操作,使网络可以发挥更好的去摩尔纹的效果。可以从手机的版本2到版本6每个版本各采集2000张,利用手机从不同角度,不同距离对屏幕上同一张二维码图像拍摄20次。接着可以对拍摄的10000张真实二维码图像进行重建,根据二维码中固定的模块:定位图样,校准图样和三个探测图样进行角度变换,得到与屏幕上显示的二维码图像一一对应的重建图像,如图5(b)所示。Referring to Fig. 5(a), in some examples, in the migration learning stage, 10,000 real moiré real two-dimensional code images can be collected to perform migration learning operations on the network device 1, so that the network can perform better demolition The effect of the pattern. You can collect 2000 images from each version of the mobile phone from version 2 to version 6, and use the mobile phone to take the same QR code image on the screen 20 times from different angles and different distances. Then you can reconstruct the 10,000 real two-dimensional code images taken, and perform angle transformations based on the fixed modules in the two-dimensional code: positioning pattern, calibration pattern and three detection patterns, and get one-to-one with the two-dimensional code image displayed on the screen. The corresponding reconstructed image is shown in Figure 5(b).
另外,再次参照图1,可以将手机等设备拍摄的真实二维码图像作为输入,并经过透视变换、角度变换等操作进行图像的重建得到重建图像,以作为该网络装置1的输入进行迁移学习。在这种情况下,可以使该训练好的网络装置1对真实环境中拍摄好的二维码图像有更加理想的去除效果。In addition, referring again to FIG. 1, a real two-dimensional code image taken by a mobile phone and other devices can be used as input, and the reconstructed image can be obtained by performing perspective transformation, angle transformation, and other operations to obtain a reconstructed image, which can be used as the input of the network device 1 for migration learning . In this case, the trained network device 1 can have a more ideal removal effect on the two-dimensional code image taken in the real environment.
虽然以上结合附图和实施例对本公开进行了具体说明,但是可以理解,上述说明不以任何形式限制本公开。本领域技术人员在不偏离本公开的实质精神和范围的情况下可以根据需要对本公开进行变形和变化,这些变形和变化均落入本公开的范围内。Although the present disclosure has been specifically described above with reference to the drawings and embodiments, it can be understood that the foregoing description does not limit the present disclosure in any form. Those skilled in the art can make modifications and changes to the present disclosure as needed without departing from the essential spirit and scope of the present disclosure, and these modifications and changes fall within the scope of the present disclosure.
Claims (10)
- 一种二维码去摩尔纹的深度残差网络的训练方法,其特征在于,A training method of a deep residual network for removing moiré patterns from two-dimensional codes, which is characterized in that:包括:include:准备带有摩尔纹的原始二维码图像;Prepare the original QR code image with moiré patterns;准备用于对所述原始二维码进行处理的网络装置,所述网络装置包括预处理模块、第一残差模块、第二残差模块以及第三残差模块;Preparing a network device for processing the original two-dimensional code, the network device including a preprocessing module, a first residual module, a second residual module, and a third residual module;将所述原始二维码图像输入所述预处理模块,对所述原始二维码图像进行模糊处理以增加仿真二维码图像中二维码区域的像素,接着进行下采样处理以形成图像缩小的预处理图像;The original two-dimensional code image is input to the preprocessing module, the original two-dimensional code image is blurred to increase the pixels in the two-dimensional code area in the simulated two-dimensional code image, and then down-sampling is performed to form an image reduction Preprocessed image;将所述预处理图像输入至所述第一残差模块进行上采样处理以形成图像大小放大至所述原始二维码图像大小的第一输出图像;Inputting the preprocessed image to the first residual module for up-sampling processing to form a first output image whose image size is enlarged to the size of the original two-dimensional code image;将所述第一输出图像输入至所述第二残差模块以形成恢复所述第一输出图像丢失的图像信息的第二输出图像;并且Inputting the first output image to the second residual module to form a second output image that restores the image information lost in the first output image; and将所述第二输出图像和所述原始二维码图像进行特征融合以形成特征融合图像,接着将所述特征融合图像输入所述第三残差模块进行提纯处理以形成去除摩尔纹的去摩尔纹图像。Perform feature fusion of the second output image and the original two-dimensional code image to form a feature fusion image, and then input the feature fusion image to the third residual module for purification processing to form moiré removal Pattern image.
- 如权利要求1所述的训练方法,其特征在于,The training method according to claim 1, wherein:所述第一残差模块包括依次连接的卷积核大小为3×3且具有64个特征映射的第一卷积层、第一ReLU激活层、串联的16个残差块、卷积核大小为3×3且具有64个特征映射的第二卷积层、批归一化层、卷积核大小为3×3且具有256个特征映射的第三卷积层、卷积核大小为1×1且具有3个特征映射的第四卷积层以及tanh激活层。The first residual module includes a first convolution layer with a convolution kernel size of 3×3 and 64 feature maps, a first ReLU activation layer, 16 residual blocks connected in series, and a convolution kernel size. The second convolutional layer that is 3×3 and has 64 feature maps, the batch normalization layer, the third convolutional layer with a convolution kernel size of 3×3 and 256 feature maps, and the convolution kernel size is 1. ×1 The fourth convolutional layer with 3 feature maps and the tanh activation layer.
- 如权利要求1所述的训练方法,其特征在于,The training method according to claim 1, wherein:所述第二残差模块包括依次连接的10层由卷积核大小为5×5且具有64个特征映射的卷积层和第二ReLU激活层串联成的第一组合层以及10层由卷积核大小为3×3且具有64个特征映射的卷积层和第三ReLU激活层串联成的第二组合层。The second residual module includes 10 successively connected first combination layers composed of a convolutional layer with a convolution kernel size of 5×5 and 64 feature maps and a second ReLU activation layer, and 10 layers consisting of a convolutional layer and a second ReLU activation layer. The second combination layer is a convolutional layer with a size of 3×3 and 64 feature maps and a third ReLU activation layer in series.
- 如权利要求1所述的训练方法,其特征在于,The training method according to claim 1, wherein:所述第三残差模块包括依次连接的卷积核大小为3×3且具有128个特征映射的第五卷积层、第四ReLU激活层、卷积核大小为3×3且具有3个特征映射的第六卷积层。The third residual module includes a fifth convolutional layer with a convolution kernel size of 3×3 and 128 feature maps, a fourth ReLU activation layer, a convolution kernel size of 3×3 and 3 The sixth convolutional layer of feature maps.
- 如权利要求1所述的训练方法,其特征在于,The training method according to claim 1, wherein:所述原始二维码图像为合成的仿真二维码图像。The original two-dimensional code image is a synthesized simulated two-dimensional code image.
- 如权利要求5所述的训练方法,其特征在于,The training method according to claim 5, wherein:所述仿真二维码图像的形成包括如下步骤:The formation of the simulated two-dimensional code image includes the following steps:对输入图像重新采样以生成一个由RGB像素组成的嵌合体,并显示在显示器上;Resample the input image to generate a mosaic composed of RGB pixels and display it on the monitor;随机作投影变换来模拟所述显示器和相机之间不同的相对位置和方向;Randomly perform projection transformation to simulate different relative positions and directions between the display and the camera;使用放射失真函数来模拟所述相机的镜头的失真;Using a radiation distortion function to simulate the distortion of the lens of the camera;采用平顶高斯滤波器来模拟抗混叠滤波;Use flat-top Gaussian filter to simulate anti-aliasing filtering;对所述输入图像重新采样以模拟所述相机传感器的输入;Re-sampling the input image to simulate the input of the camera sensor;在所述输入图像上增加高斯噪声来模拟传感器噪声;Adding Gaussian noise to the input image to simulate sensor noise;去马赛克处理;Demosaicing采用去噪滤波器进行去噪处理;Use denoising filter for denoising processing;压缩所述输入图像;以及Compress the input image; and输出解压缩图像以形成所述仿真二维码图像。The decompressed image is output to form the simulated two-dimensional code image.
- 如权利要求6所述的训练方法,其特征在于,The training method according to claim 6, wherein:与所述输入图像相对应的所述显示器上的二维码图像使用相同的所述投影变换和所述镜头失真函数进行处理。The two-dimensional code image on the display corresponding to the input image is processed using the same projection transformation and lens distortion function.
- 如权利要求1所述的训练方法,其特征在于,The training method according to claim 1, wherein:采用均方差函数 作为损失函数训练所述深度残差网络,其中M和N是所述仿真二维码图像的高和宽,H(G(I))是深度残差网络输出的二维码图像,J是与所述仿真二维码图像相对应的显示器上的二维码图像,且保存训练的损失最小时的所述深度残差网络的模型和参数。 Using the mean square error function Train the deep residual network as a loss function, where M and N are the height and width of the simulated two-dimensional code image, H(G(I)) is the two-dimensional code image output by the deep residual network, and J is and The two-dimensional code image on the display corresponding to the simulated two-dimensional code image is stored, and the model and parameters of the deep residual network when the training loss is the least are stored.
- 如权利要求8所述的训练方法,其特征在于,The training method according to claim 8, wherein:还包括带有摩尔纹的真实二维码图像,基于训练的损失最小时的所述深度残差网络的模型和参数,并利用所述真实二维码图像对所述深度残差网络进行迁移学习操作。It also includes a real two-dimensional code image with moiré, based on the model and parameters of the deep residual network when the training loss is minimal, and uses the real two-dimensional code image to perform migration learning on the deep residual network operating.
- 如权利要求8所述的训练方法,其特征在于,The training method according to claim 8, wherein:对所述真实二维码图像进行角度变换,得到与所述真实二维码图像一一对应的显示器上的二维码图像。Performing an angle transformation on the real two-dimensional code image to obtain a two-dimensional code image on the display corresponding to the real two-dimensional code image one-to-one.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911414600 | 2019-12-31 | ||
CN201911414600.1 | 2019-12-31 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021134874A1 true WO2021134874A1 (en) | 2021-07-08 |
Family
ID=71185725
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/076819 WO2021134874A1 (en) | 2019-12-31 | 2020-02-26 | Training method for deep residual network for removing a moire pattern of two-dimensional code |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN111340729B (en) |
WO (1) | WO2021134874A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113723515A (en) * | 2021-08-31 | 2021-11-30 | 平安科技(深圳)有限公司 | Moire pattern recognition method, device, equipment and medium based on image recognition |
CN114693558A (en) * | 2022-03-31 | 2022-07-01 | 福州大学 | Image Moire removing method and system based on progressive fusion multi-scale strategy |
CN114881888A (en) * | 2022-06-10 | 2022-08-09 | 福州大学 | Video Moire removing method based on linear sparse attention transducer |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112465092B (en) * | 2020-10-29 | 2023-03-03 | 深圳大学 | Two-dimensional code sample generation method and device, server and storage medium |
CN112712467B (en) * | 2021-01-11 | 2022-11-11 | 郑州科技学院 | Image processing method based on computer vision and color filter array |
CN113066027B (en) * | 2021-03-31 | 2022-06-28 | 天津大学 | Screen shot image moire removing method facing to Raw domain |
CN114418846A (en) * | 2021-12-30 | 2022-04-29 | 深圳市安健科技股份有限公司 | Method for removing moire of grid and terminal |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107358575A (en) * | 2017-06-08 | 2017-11-17 | 清华大学 | A kind of single image super resolution ratio reconstruction method based on depth residual error network |
US20180268533A1 (en) * | 2017-03-14 | 2018-09-20 | Adobe Systems Incorporated | Digital Image Defect Identification and Correction |
CN109345449A (en) * | 2018-07-17 | 2019-02-15 | 西安交通大学 | A kind of image super-resolution based on converged network and remove non-homogeneous blur method |
CN109993698A (en) * | 2019-03-29 | 2019-07-09 | 西安工程大学 | A kind of single image super-resolution texture Enhancement Method based on generation confrontation network |
CN110287969A (en) * | 2019-06-14 | 2019-09-27 | 大连理工大学 | Mole text image binaryzation system based on figure residual error attention network |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106709875B (en) * | 2016-12-30 | 2020-02-18 | 北京工业大学 | Compressed low-resolution image restoration method based on joint depth network |
CN109949235A (en) * | 2019-02-26 | 2019-06-28 | 浙江工业大学 | A kind of chest x-ray piece denoising method based on depth convolutional neural networks |
CN110009615B (en) * | 2019-03-31 | 2020-06-19 | 深圳大学 | Image corner detection method and detection device |
CN110020684B (en) * | 2019-04-08 | 2021-01-29 | 西南石油大学 | Image denoising method based on residual convolution self-coding network |
CN110599388B (en) * | 2019-08-26 | 2021-10-15 | 华中科技大学 | Blind robust digital watermark embedding and detecting method based on positioning point assistance |
-
2020
- 2020-02-26 CN CN202010120759.9A patent/CN111340729B/en active Active
- 2020-02-26 WO PCT/CN2020/076819 patent/WO2021134874A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180268533A1 (en) * | 2017-03-14 | 2018-09-20 | Adobe Systems Incorporated | Digital Image Defect Identification and Correction |
CN107358575A (en) * | 2017-06-08 | 2017-11-17 | 清华大学 | A kind of single image super resolution ratio reconstruction method based on depth residual error network |
CN109345449A (en) * | 2018-07-17 | 2019-02-15 | 西安交通大学 | A kind of image super-resolution based on converged network and remove non-homogeneous blur method |
CN109993698A (en) * | 2019-03-29 | 2019-07-09 | 西安工程大学 | A kind of single image super-resolution texture Enhancement Method based on generation confrontation network |
CN110287969A (en) * | 2019-06-14 | 2019-09-27 | 大连理工大学 | Mole text image binaryzation system based on figure residual error attention network |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113723515A (en) * | 2021-08-31 | 2021-11-30 | 平安科技(深圳)有限公司 | Moire pattern recognition method, device, equipment and medium based on image recognition |
CN113723515B (en) * | 2021-08-31 | 2023-08-18 | 平安科技(深圳)有限公司 | Moire pattern recognition method, device, equipment and medium based on image recognition |
CN114693558A (en) * | 2022-03-31 | 2022-07-01 | 福州大学 | Image Moire removing method and system based on progressive fusion multi-scale strategy |
CN114881888A (en) * | 2022-06-10 | 2022-08-09 | 福州大学 | Video Moire removing method based on linear sparse attention transducer |
Also Published As
Publication number | Publication date |
---|---|
CN111340729A (en) | 2020-06-26 |
CN111340729B (en) | 2023-04-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021134874A1 (en) | Training method for deep residual network for removing a moire pattern of two-dimensional code | |
Syu et al. | Learning deep convolutional networks for demosaicing | |
Liu et al. | Demoir\'eing of Camera-Captured Screen Images Using Deep Convolutional Neural Network | |
CN112750076B (en) | Light field multi-view image super-resolution reconstruction method based on deep learning | |
CN110992244B (en) | Picture generation method, system, device and storage medium with mole pattern | |
JPWO2009150882A1 (en) | Image alignment processing device, area expansion processing device, and image quality improvement processing device | |
CN103841298B (en) | Video image stabilization method based on color constant and geometry invariant features | |
CN108846818B (en) | Moire pattern removing method, device, terminal and computer readable storage medium | |
CN114881888A (en) | Video Moire removing method based on linear sparse attention transducer | |
CN107680043B (en) | Single image super-resolution output method based on graph model | |
CN113066027B (en) | Screen shot image moire removing method facing to Raw domain | |
CN114693558A (en) | Image Moire removing method and system based on progressive fusion multi-scale strategy | |
Yang et al. | Designing display pixel layouts for under-panel cameras | |
WO2016047893A1 (en) | Fisheye lens image correction method and apparatus therefor | |
CN110874827A (en) | Turbulent image restoration method and device, terminal equipment and computer readable medium | |
CN110634103A (en) | Image demosaicing method based on generation of countermeasure network | |
CN115937794A (en) | Small target object detection method and device, electronic equipment and storage medium | |
Zhang et al. | Deep spatial adaptive network for real image demosaicing | |
JP6236908B2 (en) | Imaging apparatus, imaging system, and imaging method | |
Cristóbal et al. | Superresolution imaging: a survey of current techniques | |
CN102129671B (en) | Remote sensing image upsampling method based on POCS (projections onto convex sets) and complex wavelet packet field | |
CN105956530A (en) | Image correction method and image correction device | |
CN113066023B (en) | SAR image speckle removing method based on self-calibration convolutional neural network | |
Chopade et al. | Single and multi frame image super-resolution and its performance analysis: a comprehensive survey | |
CN115018707B (en) | Image amplification method and device based on bicubic fitting of nine curved surface pieces |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20909937 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 03.11.2022) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20909937 Country of ref document: EP Kind code of ref document: A1 |