WO2022047625A1 - Image processing method and system, and computer storage medium - Google Patents

Image processing method and system, and computer storage medium Download PDF

Info

Publication number
WO2022047625A1
WO2022047625A1 PCT/CN2020/112870 CN2020112870W WO2022047625A1 WO 2022047625 A1 WO2022047625 A1 WO 2022047625A1 CN 2020112870 W CN2020112870 W CN 2020112870W WO 2022047625 A1 WO2022047625 A1 WO 2022047625A1
Authority
WO
WIPO (PCT)
Prior art keywords
layer
network
discriminator
generator
image
Prior art date
Application number
PCT/CN2020/112870
Other languages
French (fr)
Chinese (zh)
Inventor
郑海荣
刘新
张娜
胡战利
薛恒志
梁栋
Original Assignee
深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳先进技术研究院 filed Critical 深圳先进技术研究院
Priority to PCT/CN2020/112870 priority Critical patent/WO2022047625A1/en
Publication of WO2022047625A1 publication Critical patent/WO2022047625A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology

Definitions

  • the present application relates to the technical field of image processing, and in particular, to an image processing method, system and storage medium for convolutional neural networks.
  • Magnetic resonance imaging (MRI) scanning is currently a non-invasive high-resolution imaging technique that plays an important role in current clinical diagnosis and scientific research, as it can reveal the three-dimensional, internal details and structures of human tissues and organs.
  • noise can easily affect the quality of the image, especially when high speed and high resolution are required, local blurred areas (such as white, shadow, distortion, etc.) will be generated on the image.
  • Noise in MRI not only reduces imaging quality but also reduces clinical diagnostic accuracy; it also negatively affects the reliability of subsequent analysis tasks such as registration, segmentation, and detection, such as the formation of several Slice (pixel block or pixel strip) as the sampling input of the processing model.
  • the slice partially or completely covers the blurred area, the sampling slice cannot be used as input data.
  • the data for the under-sampling part needs to be filled by the processing model according to other sampling data. or perfect. Therefore, effective noise reduction algorithms are necessary for further magnetic resonance analysis, and have important scientific significance and application prospects in the field of medical diagnosis.
  • 3D magnetic resonance image denoising technology based on multi-channel residual learning of convolutional neural network
  • a 10-layer convolutional neural network layer is designed, and the architecture of VGG network is adopted to reduce noise and residual learning strategy.
  • This deep learning method exhibits robust denoising performance on both the maximum peak signal-to-noise ratio and the global structural similarity index evaluation metrics.
  • noise suppression and image structure preservation it retains more details of the image and effectively removes the noise of 3D magnetic resonance images.
  • a convolutional neural network technique for magnetic resonance image denoising is designed under the framework of deep learning of a convolutional neural network, which uses a set of convolutions to separate image features from noise.
  • the network adopts an encoder-decoder structure, which not only preserves the salient features of the image, but also ignores unnecessary features.
  • the network is trained end-to-end using a residual learning scheme.
  • the performance of the proposed CNN is qualitatively and quantitatively tested on one simulated and four real magnetic resonance datasets. Extensive experimental results show that the network can effectively remove noise from MRI images without losing key image details.
  • the image denoised by the convolutional neural network is prone to lose the edge details of the image, and also has defects such as excessive network parameters, huge consumption of computing resources and slow processing speed.
  • the present application provides an image processing method, which adopts the following technical solutions:
  • An image processing method comprising:
  • a generator training step that extracts noisy image data from the training set as input images to train generator network parameters by reducing the cycle consistency loss function so that the output images of the generator network are the same as the noise-free images in the training set. The difference is reduced; the generator network includes an attention mechanism module for enhancing the contrast of edge details of the input image;
  • the discriminator training step the parameters of the discriminator network are trained by reducing the discriminator loss function by inputting the output image of the trained generator network and the noise-free image respectively to the discriminator network, so that the output of the discriminator network indicates the discriminator network. whether the input to the generator network is the output image of the trained generator network or an indication of the noise-free image;
  • the generator training step and the discriminator training step are repeated with different noise images to obtain the final generator network parameters and discriminator network parameters by minimizing the cycle consistency loss function and the discriminator loss function.
  • a multi-layer deep convolutional neural network adversarial network model is constructed.
  • the parameters are reduced to 1/1000 or even 1/10,000, which greatly reduces the calculation factor.
  • the complexity of the image processing speed is accelerated, and the implementability is provided from the technical level.
  • the attention mechanism to extract the feature map for the detailed information such as edges in the image and introduce the deconvolution process during the convolution process, the edge detail information in the image is preserved, so as to improve the mapping relationship between the noisy image and the real image , reducing image distortion and loss of image edge features.
  • the adversarial network can denoise the noisy 3D MRI images and achieve high-quality MR images that meet the diagnostic requirements of doctors.
  • the quality of the denoised image is better than the image obtained after denoising with the original convolutional neural network.
  • the application provides an image processing system, which includes:
  • the memory stores computer-readable codes, and executes the above-mentioned image processing method when the computer-readable codes are executed by the processor.
  • the above image processing method is presented in the form of computer readable code and stored in the memory, and when the system processor runs the computer readable code in the memory, the steps of the above image processing method are executed to obtain improved image processing. speed and optimize the effect of image edge information.
  • the present application provides a computer storage medium, which stores computer-readable codes, and executes the above-mentioned image processing method when the computer-readable codes are executed.
  • the above image processing method is presented in the form of computer readable code and stored on a computer storage medium, and when the processor runs the computer code on the medium, the steps of the above image processing method are executed to obtain an enhanced image Process speed and optimize the effect of image edge information.
  • the present application includes at least one of the following beneficial technical effects:
  • the application adopts an adversarial network generated based on the attention mechanism, which solves the problem of serious loss of edge information features in the target image due to the decline of the original generation adversarial network mapping ability;
  • the mean square error loss is introduced into the least squares loss function, which solves the problem of losing image details due to the over-smoothing of the image caused by the noise reduction of the CNN network, and avoids the disadvantage of over-smoothing the image caused by a single adversarial loss function. More image edge detail is preserved.
  • FIG. 1 is a flowchart of an image processing method according to an embodiment of the present application.
  • Figure 2 is an architectural diagram of a generator network according to an embodiment of the present application.
  • FIG. 3 is an architectural diagram of an attention mechanism module according to an embodiment of the present application.
  • FIG. 4 is an architectural diagram of a discriminator network according to an embodiment of the present application.
  • FIG. 5 is a flowchart of an image processing method according to another embodiment of the present application.
  • an image processing method includes the following steps:
  • the generator training step extracting noise image data from the training set as input images to train the generator network parameters by reducing the cycle consistency loss function, so that the output image of the generator network and the training set are noise-free Differences in images are reduced.
  • the generator network includes an attention mechanism module for enhancing the contrast of edge details of the input image.
  • the cycle consistency loss function may represent the degree of difference between the output image of the generator network and the noise-free image based on the discriminator network parameters.
  • S20, the discriminator training step, the output image and the noise-free image of the trained generator network are respectively input to the discriminator network to train the parameters of the discriminator network by reducing the loss function of the discriminator, so that the output of the discriminator network indicates Whether the input to the discriminator network is the output image of the trained generator network or an indication of the noise-free image.
  • a discriminator loss function may represent how well the output image of the generator network corresponds to the noise-free image.
  • the implementation principle of the above image processing method is as follows: by introducing a new convolutional neural network system architecture, the traditional training strategy is replaced by the adversarial network method, and the noisy image is used as input to allow artificially generated detailed information through deep learning to fill in due to noise. resulting image defects.
  • the adversarial network uses two convolutional neural network systems, namely: a so-called “generator”, which is a type of generator network; and a so-called “discriminator” network, which is For assessing the quality of images with magnified contrast.
  • the "discriminator” receives as input the noise image and the real image, and outputs a number such as -1 or 1.
  • the "discriminator” considers the noisy image to correspond to the original real image content (enhancing contrast). If the output is -1, the “discriminator” considers the noisy image to be the boosted output of the generator network.
  • the goal of training the generator is to maximize the output of the "discriminator” so that it becomes as realistic as possible.
  • the “discriminator” is trained to accurately distinguish between the original enhanced contrast content and the boosted content. The two networks alternate training steps to compete with each other and obtain the best parameters.
  • the image processing method is described in detail by taking 3D MRI human body image data as an example, wherein the training set contains a noisy image x, contains several samples ⁇ x 1 , x 2 , x 3 ,..., x n ⁇ and a noise-free image y contains several The sample ⁇ y 1 , y 2 , y 3 ,...,y n ⁇ , the noise-free image y can be understood as the real image obtained by removing the noise from the noise image.
  • the noise image sample x 1 is first decomposed or sliced to obtain several sub-band images as input data, and the first convolutional layer in the network constructed by the generator extracts the input data to obtain the feature representation, and the feature representation is usually is the feature matrix.
  • the features extracted here may be contrast, resolution, grayscale, and the like.
  • the correlation between the sub-band images can be established in the coding area, and then the attention mechanism can insert or superimpose the edge information of the image into the deconvolution of the decoding area through the correlation between the sub-band images. layer, which ultimately enables the output image to retain more edge information.
  • the first layer obtained by the feature matrix pooling operation is grouped into one layer, and finally the first layer of activation function is obtained by continuous pooling of the nonlinear enhancement operator.
  • the first layer of attention mechanism module is the self-attention map obtained by extracting the feature map from the input data to capture the multiple features in x 1 that can reflect the image edge detail information, and then transform and combine them.
  • the first layer of activation function layer is used as input data for feature extraction to obtain feature representation to obtain the second layer of convolutional layer.
  • the convolution operation of the second layer is roughly the same as that of the first layer, except that the number of convolution kernels is increased and more features are extracted.
  • the second layer of attention mechanism module dimension is the self-attention map obtained by extracting features from the first layer of convolution layer, the first layer of batch normalization layer and the first layer of activation function layer, respectively, for transformation and combination. Same as above, complete the third and fourth layers of convolution to obtain the fourth activation function layer and obtain the self-attention map of the third layer of attention mechanism module and the fourth layer of attention mechanism module, respectively.
  • the activation function layer, batch normalization layer and deconvolution layer are obtained sequentially based on the fourth layer activation function layer by layer-by-layer decoding relative to the convolution reverse operation, and then the fourth layer
  • the deconvolution layer is logically superposed with the self-attention map of the fourth layer attention mechanism module to obtain the fourth layer deconvolution layer of edge enhancement.
  • the deconvolution operation is performed relative to the third layer convolution with the enhanced fourth layer deconvolution layer as the input, as above, and the third layer, the second layer, and the first layer deconvolution layer of edge enhancement are completed in turn, Output image x 11 after fitting the edge-enhanced first deconvolution layer.
  • the above-mentioned output image x 11 filtered by the generator and the real image y 1 are respectively input into the discriminator, and each input is sequentially processed by four layers of convolution filtering, and then activated after being connected to the hidden layer by the first fully connected layer.
  • the function layer is processed nonlinearly, and finally the fully connected layer and the activation function layer with a unit of 1 are output to complete a round of iteration.
  • the next iteration can repeat the above steps with x 2 as the input image.
  • the generator network and the discriminator network can be trained alternately.
  • alternate training is performed in the order of generator training, discriminator training, generator training, and discriminator training steps, where one generator training step and one discriminator training step are referred to as sequential iterations.
  • the generator training step and the discriminator training step are exchanged in order, that is, the training is performed alternately in the order of the discriminator training step, the generator training step, the discriminator training step, and the generator training step, wherein one of the discriminators The training step and one generator training step are called successive iterations.
  • Both the discriminator network and the generator network can take the form of a convolutional neural network, and both have various parameters of the convolutional application network.
  • the parameters of the generator network may include the weights of the filters of each convolutional layer, the paranoia of each activation function layer, and the reinforcement parameters of each attention mechanism module;
  • the parameters of the discriminator network may include the paranoia of each activation function layer, the The weights of the filters of the convolutional layers and the degradation parameters of the fully connected layers.
  • the parameters of the generator network and the parameters of the discriminator network can be preset or randomly given values.
  • the training of the generator network is based on the training results of the discriminator network (that is, the training results of the parameters of the gradienter network), and the training of the discriminator network requires the use of the generator network.
  • the output image, so the training of the discriminator network is based on the training results of the generator network (that is, the training results of the parameters of the generator network), this way is called "adversarial", that is, the generator network and the discriminator network fight against each other.
  • This approach allows two adversarial networks to compete and continuously improve on each iteration based on the better and better results of the other network to train with better and better parameters.
  • the cycle consistency loss function in the generator training step can be obtained from the generator loss function and the discriminator loss function.
  • the cycle consistency loss function can be composed of two parts, where the first part is based on the mean square error output between the output image of the generator network and the noise-free image, and the second part is based on the output image of the generator network through all output of the discriminator network.
  • a mean squared error loss is added to the cycle consistency loss function to avoid the disadvantage of smooth image transition caused by a single adversarial loss function, thereby preserving more details of the image.
  • an MRI noise reduction network based on the least squares loss function generation anti-network based on 3D attention mechanism can be added to use 3D attention least squares for high-noise MRI images
  • Generative Adversarial Networks filter the image to arrive at a medical image that can be diagnosed by a doctor.
  • the least squares adversarial loss in order to improve the mapping ability between noisy images and real images and the training process of the network, can be expressed as:
  • G is the generator, where L LSGAN (G) represents the loss function of the generator, L LSGAN (D) is the loss function of the discriminator, and P x (x) and P y (y) represent the noise data and real label data, respectively Distribution; x represents noise data, y represents real label data, G(x) is the result of the generator output when noise image data is used as input, D(G(x)) is the discriminator when G(x) is used as input The probability of the output, G(y) is the probability of the discriminator output when the real label data is used as input, and IE represents the loss calculation function.
  • the mean square error function is:
  • d, w, and h are the depth, width and height of the extracted feature map, respectively;
  • L 3D a-LSGAN ⁇ 1 L mse + ⁇ 2 L LSGAN (G)
  • ⁇ 1 and ⁇ 2 are empirical parameters used to balance different ratios, which are set values; according to experience, we set ⁇ 1 and ⁇ 2 to be 1 and 0.0001, respectively.
  • the generator construction step is to construct a generator of a multi-layer deep convolutional neural network based on the U-Net network structure, the generator includes a skip-connected encoder-decoder network, and the skip connection structure of the U-Net network structure is added with self-
  • the attention mechanism is used to transfer the edge detail image information of the encoded region to the corresponding decoding region.
  • the edge detail image information here may refer to edge information, detail information and the like of an image. After processing the noisy image through this step, the details of some darker areas can also be clearly seen, such as the outline of the organ, the folds of the folds, the distribution network of the trachea, etc., which helps doctors to make correct analysis and diagnosis.
  • the number of parameters is reduced to one thousandth or even ten thousandths. One of them, thereby greatly reducing the complexity of the calculation factor, speeding up the image processing speed, and providing practicability from the technical level.
  • the discriminator construction step constructs the discriminator of the multi-layer deep convolutional neural network based on the generator network.
  • the MRI denoising network based on the least squares generative anti-network of 3D attention mechanism is as follows: the generator includes an encoding network formed by a multi-layer convolutional architecture, The decoding network and multi-layer self-attention mechanism module formed by the product architecture; each layer of convolution architecture corresponds to one layer of deconvolution architecture and one layer of attention mechanism module;
  • Each layer of convolutional architecture includes: convolutional layer, batch normalization layer and activation function layer;
  • Each layer of deconvolution architecture includes: deconvolution layer, batch normalization layer and activation function layer.
  • each layer of attention mechanism module may include: a first feature map extracted based on a convolutional layer of a corresponding layer convolutional architecture, a batch normalization layer extracted based on a corresponding layered convolutional architecture The second feature map of and the third feature map extracted based on the activation function layer of the corresponding layer convolution architecture.
  • the attention map can be obtained by transposing the third feature map and multiplying the second feature map by the softmax activation function; and then multiplying the first feature map and the attention map to obtain the self-attention feature map.
  • the image After being processed by the attention mechanism module, the image can obtain more detailed information and pass it to the decoding area through skip connections.
  • the above-mentioned first feature map, second feature map, and third feature map are related to parameters such as the length, width, and number of feature channels of the image.
  • the step of obtaining the output image in the generator training step S10 may include:
  • the convolution step during the convolution operation of each layer, the image data with noise in the training set is randomly cut into pieces and then used as input for feature extraction to obtain a convolution layer; the convolution layer is pooled to obtain a batch normalization layer, and the batch The normalization layer obtains the activation function layer through nonlinear combination of functions;
  • the deconvolution step in the deconvolution operation of each layer, the deconvolution layer is added to the self-attention feature map obtained by the self-attention mechanism module corresponding to the layer, and then a pooling operation is performed to obtain a batch normalization layer , and then activate the batch normalization layer through the activation function layer and output it;
  • the output image is obtained after all layers of convolution and deconvolution are completed.
  • the discriminator may comprise a convolutional architecture and fully connected layers with the same number of layers as the generator;
  • Each layer of convolutional architecture includes: convolution layer, batch normalization layer and activation function layer.
  • the step of obtaining the output indication in the discriminator training step S20 may include:
  • the convolution step taking the enhanced image output by the generator into blocks and then using it as an input to perform feature extraction to obtain a convolutional layer; pooling the convolutional layer to obtain a batch normalization layer, and performing a non-decoding process on the batch normalization layer through a function Linear combination to obtain activation function layer;
  • connection step is to non-linearly combine the features obtained by completing the convolution operations of all layers through the fully connected layer, and when the loss function of the discriminator is close to 1, it is determined that the input of the discriminator network is the generated after training.
  • the step of constructing the discriminator and before the training of the generator it may further include:
  • an encoder-decoder network similar to U-net network with skip connections is used as the generator network. All convolutions are performed by a 3D convolutional layer that processes 3D data, in which an attention mechanism is added to the skip connection part to transfer the detailed image information of the encoding area to the corresponding decoding area, so that the decoding network can transfer the detailed image information to the corresponding decoding area. back to the image.
  • the generator network contains a total of 8 layers, including 4 convolutional layers and 4 deconvolutional layers. Each layer contains a 3D convolutional layer, a batch normalization layer, and an activation function layer.
  • the size of the used convolution kernels is all 3 ⁇ 3 ⁇ 3 pixels, and the number of convolution kernels can be, for example, 32, 64, 128, 256, 128, 64, and 32.
  • the convolution strides are all 1 pixel.
  • the discriminator consists of four convolutional layers and two fully connected layers (including a first fully connected layer and a second fully connected layer). Each convolutional layer is followed by a batch normalization layer and a LeakeyRelu activation function layer. After the four-layer convolutional layer is connected to the first fully connected layer, the output unit of the first fully connected layer is 1024 values, followed by the LeakeyRelu activation function. The second fully connected layer is a fully connected layer with an output unit of 1 value and a LeakeyRelu activation function layer.
  • the discriminator and generator use the same convolution kernel size, both 3 ⁇ 3 ⁇ 3 pixels, and the number of convolution kernels in each layer is 32, 64, 128, 256.
  • the first convolutional layer outputs 32 feature maps
  • the second convolutional layer outputs 64 feature maps
  • the third convolutional layer outputs 128 feature maps
  • the fourth convolutional layer outputs 256 feature maps.
  • the MRI image with noise and the MRI image without noise are randomly divided into 3D pixel blocks and the corresponding noise-free MRI image is used as the input and label of the adversarial network for training, and the attention mechanism considers the difference between the blocks. Correlated information and can pass the important information of the coding region (convolutional layer) to the corresponding decoding part (deconvolutional layer) through skip connections.
  • the training of network parameters is completed, the training network is obtained, and the mapping relationship G from the MRI image with noise to the MRI image without noise is obtained at the same time.
  • denoise the noisy MRI images through the trained 3D attention least squares generative adversarial network to obtain denoised images that meet the doctor's diagnostic requirements.
  • the 3D attention least squares generative adversarial network can denoise the noisy 3D MRI images and achieve high-quality MR images that meet the diagnostic requirements of doctors. Edge details in the image can also be preserved by using an attention mechanism during encoding and decoding.
  • the method according to the present application can also be applied to image noise reduction in the fields of 3D SPECT images, low-dose 3D CT images, and low-count 3DPET after appropriate transformation.
  • the present application also provides an image processing system for a convolutional neural network, comprising: a processor and a memory; the memory stores computer-readable codes, and the processor executes the aforementioned image processing method when running the computer-readable codes any of the.
  • the implementation principle of the image processing system is as follows: the above-mentioned image processing method is presented in the form of computer-readable code and stored in the memory, and when the system processor runs the computer-readable code in the memory, the steps of the above-mentioned image processing method are executed to obtain Improve image processing speed and optimize the effect of image edge information.
  • the present application also provides a computer storage medium storing computer-readable codes, and the processor executes any one of the above image processing methods when running the computer-readable codes.
  • the implementation principle of the computer storage medium is as follows: the above-mentioned image processing method is presented in the form of computer-readable codes and stored on the computer storage medium, and when the processor runs the computer code on the medium, the steps of the above-mentioned image processing method are executed to Get the effect of speeding up image processing and optimizing image edge information.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

An image processing method and system, and a computer storage medium. The method comprises a generator training step: inputting a noise image to train network parameters by means of a loss function, so as to reduce the difference between an output image and a noiseless image (S10); a discriminator training step: respectively inputting the output image and the noiseless image of a trained generator network into a discriminator network to train network parameters by reducing a loss function, so that an output indicates whether an input is the output image or the noiseless image of the trained generator network (S20); and repeating the generator and discriminator training steps by using different noise images so as to obtain the final parameters of the generator and discriminator networks by minimizing the loss function (S30). The method has the effects of accelerating image processing and optimizing image edge information.

Description

一种图像处理方法、***和计算机存储介质An image processing method, system and computer storage medium 技术领域technical field
本申请涉及图像处理技术领域，尤其是涉及一种用于卷积神经网络的图像处理方法、***和存储介质。The present application relates to the technical field of image processing, and in particular, to an image processing method, system and storage medium for convolutional neural networks.
背景技术Background technique
目前磁共振成像(MRI)扫描是一种无创性的高分辨率成像技术，在当前的临床诊断和科学研究中发挥着重要作用，因为它能揭示人体组织器官的三维、内部细节和结构。然而，在图像采集过程中，噪声很容易影响图像的质量，特别是在需要高速、高分辨率的情况下，会在图像上产生局部模糊区域(例如白色、阴影、失真等)。磁共振成像中的噪声不仅会降低成像质量，而且会降低临床诊断精度；也会对后续分析任务(如配准、分割和检测)的可靠性产生负面影响，例如在对图像进行分割后形成若干切片(像素块或像素条)，以作为处理模型的采样输入，在切片局部或全部覆盖模糊区域时，该采样切片不能作为输入数据，针对欠采样部分的数据需要通过处理模型根据其他采样数据填充或完善。因此，有效的降噪算法对于进一步的磁共振分析是必要的，同时对于医疗诊断领域具有重要的科学意义和应用前景。Magnetic resonance imaging (MRI) scanning is currently a non-invasive high-resolution imaging technique that plays an important role in current clinical diagnosis and scientific research, as it can reveal the three-dimensional, internal details and structures of human tissues and organs. However, in the process of image acquisition, noise can easily affect the quality of the image, especially when high speed and high resolution are required, local blurred areas (such as white, shadow, distortion, etc.) will be generated on the image. Noise in MRI not only reduces imaging quality but also reduces clinical diagnostic accuracy; it also negatively affects the reliability of subsequent analysis tasks such as registration, segmentation, and detection, such as the formation of several Slice (pixel block or pixel strip) as the sampling input of the processing model. When the slice partially or completely covers the blurred area, the sampling slice cannot be used as input data. The data for the under-sampling part needs to be filled by the processing model according to other sampling data. or perfect. Therefore, effective noise reduction algorithms are necessary for further magnetic resonance analysis, and have important scientific significance and application prospects in the field of medical diagnosis.
相关技术一，基于卷积神经网络多通道残差学习的三维磁共振图像去噪技术设计了10层的卷积神经网络层，采用了VGG网络的架构来降低噪音及残差学习的策略。这种深度学习方法在最大峰值信噪比和全局结构相似性指数评价指标上，都表现了稳健的去噪性能。在噪声抑制方面和图像结构保留方面，保留图像的更多细节，有效的去除三维磁共振图像的噪声。Related technology 1, 3D magnetic resonance image denoising technology based on multi-channel residual learning of convolutional neural network, a 10-layer convolutional neural network layer is designed, and the architecture of VGG network is adopted to reduce noise and residual learning strategy. This deep learning method exhibits robust denoising performance on both the maximum peak signal-to-noise ratio and the global structural similarity index evaluation metrics. In terms of noise suppression and image structure preservation, it retains more details of the image and effectively removes the noise of 3D magnetic resonance images.
相关技术二，一种用于磁共振图像去噪的卷积神经网络技术设计了在卷积神经网络深度学习的框架下，该网络使用一组卷积将图像特征与噪声分离。该网络采用了编解码结构，既保留了图像的显著特征，又忽略了不必要的特征。利用残差学习方案对网络进行端到端的训练。在一个模拟和四个真实的磁共振数据集上对所提出的CNN的性能进行了定性和定量的测试。大量的实验结果表明，该网络在不丢失关键图像细节的情况下，能有效地去除MRI图像的噪声。Related Art Two, a convolutional neural network technique for magnetic resonance image denoising is designed under the framework of deep learning of a convolutional neural network, which uses a set of convolutions to separate image features from noise. The network adopts an encoder-decoder structure, which not only preserves the salient features of the image, but also ignores unnecessary features. The network is trained end-to-end using a residual learning scheme. The performance of the proposed CNN is qualitatively and quantitatively tested on one simulated and four real magnetic resonance datasets. Extensive experimental results show that the network can effectively remove noise from MRI images without losing key image details.
然而，上述相关技术存在以下缺陷：However, the above-mentioned related technologies have the following drawbacks:
相关技术一中，在卷积神经网络框架下执行图像降噪时，为了提高图像质量，必须通过增加网络中的层数或构建复杂的网络结构例如Resnet-101等来提取许多抽象特征。然而，在该情况下，网络的参数将增加到数亿，图像处理过程中运算因子复杂，需要巨大的计算资源支持才能达到正常的计算速度；但在实际应用中难以实现这点。In the related art, when image noise reduction is performed under the convolutional neural network framework, in order to improve the image quality, many abstract features must be extracted by increasing the number of layers in the network or constructing a complex network structure such as Resnet-101. However, in this case, the parameters of the network will increase to hundreds of millions, and the computational factors in the image processing process are complex, requiring huge computing resources to achieve normal computing speed; however, it is difficult to achieve this in practical applications.
相关技术二中，卷积神经网络降噪后的图像易丢失图像边缘细节，且同样存在网络参数过多计算资源消耗巨大、处理速度慢等缺陷。In the second related art, the image denoised by the convolutional neural network is prone to lose the edge details of the image, and also has defects such as excessive network parameters, huge consumption of computing resources and slow processing speed.
发明内容SUMMARY OF THE INVENTION
为了加快图像处理速度并优化图像边缘信息，第一方面，本申请提供一种图像处理方法，采用如下的技术方案：In order to speed up image processing and optimize image edge information, in the first aspect, the present application provides an image processing method, which adopts the following technical solutions:
一种图像处理方法，包括：An image processing method, comprising:
生成器训练步骤，从训练集中提取噪声图像数据作为输入图像以通过减小循环一致性损失函数来训练生成器网络参数，以使所述生成器网络的输出图像与所述训练集中无噪声图像的差异减小；所述生成器网络包括用 于提升输入图像的边缘细节对比度的注意力机制模块；A generator training step that extracts noisy image data from the training set as input images to train generator network parameters by reducing the cycle consistency loss function so that the output images of the generator network are the same as the noise-free images in the training set. The difference is reduced; the generator network includes an attention mechanism module for enhancing the contrast of edge details of the input image;
鉴别器训练步骤，通过训练后的生成器网络的输出图像和无噪声图像分别输入鉴别器网络以通过减小鉴别器损失函数来训练鉴别器网络的参数，使得鉴别器网络的输出指示所述鉴别器网络的输入是所述训练后的生成器网络的输出图像还是指示所述无噪声图像；The discriminator training step, the parameters of the discriminator network are trained by reducing the discriminator loss function by inputting the output image of the trained generator network and the noise-free image respectively to the discriminator network, so that the output of the discriminator network indicates the discriminator network. whether the input to the generator network is the output image of the trained generator network or an indication of the noise-free image;
采用不同的噪声图像重复所述生成器训练步骤和所述鉴别器训练步骤以通过最小化循环一致性损失函数和鉴别器损失函数得到最终的生成器网络的参数和鉴别器网络的参数。The generator training step and the discriminator training step are repeated with different noise images to obtain the final generator network parameters and discriminator network parameters by minimizing the cycle consistency loss function and the discriminator loss function.
通过采用上述技术方案，构建了多层深度卷积神经网络的对抗网络模型，与相关技术中采用的Resnet-101等网络相比参数缩减到千分之一甚至万分之一，大大降低计算因子的复杂度，加快了图像处理速度，从技术层面提供了可实施性。通过使用注意力机制在卷积过程中对图像中的边缘等细节信息提取特征图并引入反卷积过程，从而保留了图像中的边缘细节信息，以提高噪声图像与真实图像之间的映射关系，降低图像失真和图像边缘特征的丢失。By adopting the above technical solutions, a multi-layer deep convolutional neural network adversarial network model is constructed. Compared with the Resnet-101 and other networks used in related technologies, the parameters are reduced to 1/1000 or even 1/10,000, which greatly reduces the calculation factor. The complexity of the image processing speed is accelerated, and the implementability is provided from the technical level. By using the attention mechanism to extract the feature map for the detailed information such as edges in the image and introduce the deconvolution process during the convolution process, the edge detail information in the image is preserved, so as to improve the mapping relationship between the noisy image and the real image , reducing image distortion and loss of image edge features.
此外，对抗网络可以将带噪声的3D MRI图像进行降噪，并达到符合医生诊断要求的高质量MR图像。降噪图像的质量优于采用原始的卷积神经网络进行降噪之后获得的图像。In addition, the adversarial network can denoise the noisy 3D MRI images and achieve high-quality MR images that meet the diagnostic requirements of doctors. The quality of the denoised image is better than the image obtained after denoising with the original convolutional neural network.
第二方面，本申请提供一种图像处理***，其包括：In a second aspect, the application provides an image processing system, which includes:
处理器；processor;
存储器，存储有计算机可读代码，在计算机可读代码被处理器运行时执行上述的图像处理方法。The memory stores computer-readable codes, and executes the above-mentioned image processing method when the computer-readable codes are executed by the processor.
通过采用上述技术方案，将上述图像处理方法以计算机可读代码的形式呈现并存储于存储器内，在***处理器运行存储器内的计算机可读代码时，执行上述图像处理方法的步骤获得提升图像处理速度并优化图像边缘信息的效果。By adopting the above technical solution, the above image processing method is presented in the form of computer readable code and stored in the memory, and when the system processor runs the computer readable code in the memory, the steps of the above image processing method are executed to obtain improved image processing. speed and optimize the effect of image edge information.
第三方面，本申请提供一种计算机存储介质，其存储有计算机可读代码，在所述计算机可读代码被运行时执行上述的图像处理方法。In a third aspect, the present application provides a computer storage medium, which stores computer-readable codes, and executes the above-mentioned image processing method when the computer-readable codes are executed.
通过采用上述技术方案，将上述图像处理方法以计算机可读代码的形式呈现并存储于计算机存储介质上，在处理器运行该介质上的计算机代码时，执行上述图像处理方法的步骤以获得提升图像处理速度并优化图像边缘信息的效果。By adopting the above technical solution, the above image processing method is presented in the form of computer readable code and stored on a computer storage medium, and when the processor runs the computer code on the medium, the steps of the above image processing method are executed to obtain an enhanced image Process speed and optimize the effect of image edge information.
综上所述，本申请包括以下至少一种有益技术效果：To sum up, the present application includes at least one of the following beneficial technical effects:
1.在快速成像情况下，可以实现高速MRI成像并可以获取高质量的扫描图像；1. In the case of fast imaging, high-speed MRI imaging can be achieved and high-quality scanning images can be obtained;
2.本申请采用了基于注意力机制生成的对抗网络，解决了因原始生成对抗网络映射能力下降而导致目标图像中边缘信息特征丢失严重的问题；2. The application adopts an adversarial network generated based on the attention mechanism, which solves the problem of serious loss of edge information features in the target image due to the decline of the original generation adversarial network mapping ability;
3.在最小二乘损失函数中引入了均方误差损失，解决了因CNN网络降噪所带来的图像过于平滑导致失去图像细节的问题，避免单一对抗损失函数带来图像过于平滑的缺点，保留了更多的图像边缘细节。3. The mean square error loss is introduced into the least squares loss function, which solves the problem of losing image details due to the over-smoothing of the image caused by the noise reduction of the CNN network, and avoids the disadvantage of over-smoothing the image caused by a single adversarial loss function. More image edge detail is preserved.
附图说明Description of drawings
图1是根据本申请一个实施方案的图像处理方法的流程图。FIG. 1 is a flowchart of an image processing method according to an embodiment of the present application.
图2是根据本申请一个实施方案的生成器网络的构架图。Figure 2 is an architectural diagram of a generator network according to an embodiment of the present application.
图3是根据本申请一个实施方案的注意力机制模块的构架图。FIG. 3 is an architectural diagram of an attention mechanism module according to an embodiment of the present application.
图4是根据本申请一个实施方案的鉴别器网络的构架图。FIG. 4 is an architectural diagram of a discriminator network according to an embodiment of the present application.
图5是根据本申请另一实施方案的图像处理方法的流程图。FIG. 5 is a flowchart of an image processing method according to another embodiment of the present application.
具体实施方式detailed description
以下结合附图1-4及实施例对本申请作进一步详细说明。The present application will be further described in detail below with reference to the accompanying drawings 1-4 and the embodiments.
本申请本领域的普通技术人员将会理解，提供以下具体实施例的目的仅仅是对本申请的解释，使本领域的普通技术人员能够更好地理解本申请的原理，而无意于对本申请作出任何限制。Those of ordinary skill in the art of the present application will understand that the following specific examples are provided for the purpose of explaining the present application only, so that those of ordinary skill in the art can better understand the principles of the present application, and are not intended to make any changes to the present application. limit.
在本申请中，当使用术语“和/或”时，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，如无特殊说明，一般表示前后关联对象是一种“或”的关系。In this application, when the term "and/or" is used, it is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, and there are simultaneously A and B, there are three cases of B alone. In addition, the character "/" in this text, unless otherwise specified, generally indicates that the related objects before and after are an "or" relationship.
参照图1，一种图像处理方法包括以下步骤：1, an image processing method includes the following steps:
S10，生成器训练步骤，从训练集中提取噪声图像数据作为输入图像以通过减小循环一致性损失函数来训练生成器网络参数，以使所述生成器网络的输出图像与所述训练集中无噪声图像的差异减小。在一个实施方案中，生成器网络包括用于提升输入图像的边缘细节对比度的注意力机制模块。循环一致性损失函数可以表示基于鉴别器网络参数所述生成器网络的输出图像与所述无噪声图像之间差异的程度。S10, the generator training step, extracting noise image data from the training set as input images to train the generator network parameters by reducing the cycle consistency loss function, so that the output image of the generator network and the training set are noise-free Differences in images are reduced. In one embodiment, the generator network includes an attention mechanism module for enhancing the contrast of edge details of the input image. The cycle consistency loss function may represent the degree of difference between the output image of the generator network and the noise-free image based on the discriminator network parameters.
S20，鉴别器训练步骤，通过训练后的生成器网络的输出图像和无噪声图像分别输入鉴别器网络以通过减小鉴别器的损失函数来训练鉴别器网络的参数，使得鉴别器网络的输出指示所述鉴别器网络的输入是所述训练后的生成器网络的输出图像还是指示所述无噪声图像。鉴别器损失函数可以 表示所述生成器网络的输出图像与所述无噪声图像对应的程度。S20, the discriminator training step, the output image and the noise-free image of the trained generator network are respectively input to the discriminator network to train the parameters of the discriminator network by reducing the loss function of the discriminator, so that the output of the discriminator network indicates Whether the input to the discriminator network is the output image of the trained generator network or an indication of the noise-free image. A discriminator loss function may represent how well the output image of the generator network corresponds to the noise-free image.
S30，采用不同的噪声图像重复所述生成器训练步骤和所述鉴别器训练步骤以通过最小化循环一致性损失函数得到最终的生成器网络的参数和鉴别器网络的参数。S30, repeating the generator training step and the discriminator training step using different noise images to obtain final parameters of the generator network and parameters of the discriminator network by minimizing the cycle consistency loss function.
上述图像处理方法的实施原理为：通过引入一种新的卷积神经网络***架构，利用对抗网络方法取代传统训练策略，将噪声图像作为输入，以允许通过深度学习人工产生细节信息以填补由于噪声导致的图像缺陷。在一个实施方案中，对抗网络使用两个卷积神经网络***，即：所谓的“生成器(generator)”，它是一种生成器网络；和所谓的“鉴别器(discriminator)”网络，用于评估放大了对比度的图像的质量。在一个实施方案中，“鉴别器”接收噪声图像和真实图像作为输入，并输出例如为-1或1的数字。如果输出为1，则“鉴别器”认为噪声图像对应于原始真实图像内容(强化对比度)。如果输出为-1，则“鉴别器”会认为噪声图像是生成器网络的提升后的输出。训练生成器的目的是最大化“鉴别器”的输出，以使该输出变得尽量真实。通过对“鉴别器”进行训练，能够准确地区分原始强化对比度内容和提升后的内容。两个网络交替地进行训练步骤，从而相互竞争，并获得最佳参数。The implementation principle of the above image processing method is as follows: by introducing a new convolutional neural network system architecture, the traditional training strategy is replaced by the adversarial network method, and the noisy image is used as input to allow artificially generated detailed information through deep learning to fill in due to noise. resulting image defects. In one embodiment, the adversarial network uses two convolutional neural network systems, namely: a so-called "generator", which is a type of generator network; and a so-called "discriminator" network, which is For assessing the quality of images with magnified contrast. In one embodiment, the "discriminator" receives as input the noise image and the real image, and outputs a number such as -1 or 1. If the output is 1, the "discriminator" considers the noisy image to correspond to the original real image content (enhancing contrast). If the output is -1, the "discriminator" considers the noisy image to be the boosted output of the generator network. The goal of training the generator is to maximize the output of the "discriminator" so that it becomes as realistic as possible. The "discriminator" is trained to accurately distinguish between the original enhanced contrast content and the boosted content. The two networks alternate training steps to compete with each other and obtain the best parameters.
下面以3D MRI人体图像数据为例对图像处理方法进行详细说明，其中训练集包含噪声图像x，包含若干样本{x 1,x 2,x 3,…,x n}和无噪声图像y包含若干样本{y 1,y 2,y 3,…,y n}，无噪声图像y可以理解为噪声图像去除噪声后获得的真实图像。参见图2，首先将噪声图像样本x 1分解或切片获得若干子带图像作为输入数据，通过生成器构建的网络中的第一层卷积层将输入数据进行特征提取获得特征表示，特征表示通常为特征矩阵。在一些 实施方案中，此处提取的特征可以是对比度、分辨率、灰度等。通过这些特征的提取，在编码区可以建立子带图像之间的关联性，进而经注意力机制能够通过子带图像之间的关联性将图像的边缘信息***或叠加到解码区的反卷积层，最终使得输出图像能够保留更多的边缘信息。然后，将特征矩阵池化操作获得第一层批归一层，最后经非线性增强算子持续池化获得第一层激活函数层。第一层注意力机制模块为从输入数据中提取特征图以捕获x 1中能体现图像边缘细节信息的多个特征进行变换组合后获得的自注意力图。以第一层激活函数层为输入数据进行特征提取获得特征表示获得第二层卷积层。第二层的卷积操作与第一层大致相同，不同之处在于卷积核的数量增加，提取的特征更多。第二层注意力机制模块维从第一层卷积层、第一层批归一层和第一层激活函数层分别提取特征进行变换组合后获得的自注意力图。同上，完成第三层和第四层卷积获得第四激活函数层并分别获得第三层注意力机制模块和第四层注意力机制模块的自注意力图。 The image processing method is described in detail by taking 3D MRI human body image data as an example, wherein the training set contains a noisy image x, contains several samples {x 1 , x 2 , x 3 ,..., x n } and a noise-free image y contains several The sample {y 1 , y 2 , y 3 ,...,y n }, the noise-free image y can be understood as the real image obtained by removing the noise from the noise image. Referring to Figure 2, the noise image sample x 1 is first decomposed or sliced to obtain several sub-band images as input data, and the first convolutional layer in the network constructed by the generator extracts the input data to obtain the feature representation, and the feature representation is usually is the feature matrix. In some embodiments, the features extracted here may be contrast, resolution, grayscale, and the like. Through the extraction of these features, the correlation between the sub-band images can be established in the coding area, and then the attention mechanism can insert or superimpose the edge information of the image into the deconvolution of the decoding area through the correlation between the sub-band images. layer, which ultimately enables the output image to retain more edge information. Then, the first layer obtained by the feature matrix pooling operation is grouped into one layer, and finally the first layer of activation function is obtained by continuous pooling of the nonlinear enhancement operator. The first layer of attention mechanism module is the self-attention map obtained by extracting the feature map from the input data to capture the multiple features in x 1 that can reflect the image edge detail information, and then transform and combine them. The first layer of activation function layer is used as input data for feature extraction to obtain feature representation to obtain the second layer of convolutional layer. The convolution operation of the second layer is roughly the same as that of the first layer, except that the number of convolution kernels is increased and more features are extracted. The second layer of attention mechanism module dimension is the self-attention map obtained by extracting features from the first layer of convolution layer, the first layer of batch normalization layer and the first layer of activation function layer, respectively, for transformation and combination. Same as above, complete the third and fourth layers of convolution to obtain the fourth activation function layer and obtain the self-attention map of the third layer of attention mechanism module and the fourth layer of attention mechanism module, respectively.
在反卷积操作过程中，首先基于第四层激活函数层逐层解码相对于卷积反向操作顺次获得激活函数层、批归一化层和反卷积层，然后再将第四层反卷积层与第四层注意力机制模块的自注意力图逻辑叠加获得边缘增强的第四层反卷积层。然后，以增强的第四层反卷积层作为输入相对于第三层卷积进行反卷积操作，同上，依次完成边缘增强的第三层、第二层、第一层反卷积层，将边缘增强的第一层反卷积层拟合后输出图像x 11。 In the process of deconvolution operation, the activation function layer, batch normalization layer and deconvolution layer are obtained sequentially based on the fourth layer activation function layer by layer-by-layer decoding relative to the convolution reverse operation, and then the fourth layer The deconvolution layer is logically superposed with the self-attention map of the fourth layer attention mechanism module to obtain the fourth layer deconvolution layer of edge enhancement. Then, the deconvolution operation is performed relative to the third layer convolution with the enhanced fourth layer deconvolution layer as the input, as above, and the third layer, the second layer, and the first layer deconvolution layer of edge enhancement are completed in turn, Output image x 11 after fitting the edge-enhanced first deconvolution layer.
将上述经生成器滤波后的输出图像x 11和真实图像y 1分别输入鉴别器，每个输入均依次进行四层卷积滤波处理后经第一个全连接层与隐含层连接后经激活函数层非线性处理，最后输出单位为1的全连接层和激活函数层， 完成一轮迭代。 The above-mentioned output image x 11 filtered by the generator and the real image y 1 are respectively input into the discriminator, and each input is sequentially processed by four layers of convolution filtering, and then activated after being connected to the hidden layer by the first fully connected layer. The function layer is processed nonlinearly, and finally the fully connected layer and the activation function layer with a unit of 1 are output to complete a round of iteration.
下一轮迭代可以以x 2作为输入图像重复上述步骤。 The next iteration can repeat the above steps with x 2 as the input image.
可以对生成器网络和鉴别器网络交替训练。在一个实施方案中中按照生成器训练、鉴别器训练、生成器训练、鉴别器训练步骤的顺序进行交替训练，其中一个生成器训练步骤和一个鉴别器训练步骤称为依次迭代。在另一实施方案中，生成器训练步骤和鉴别器训练步骤交换顺序，即按照鉴别器训练步骤、生成器训练步骤、鉴别器训练步骤、生成器训练步骤的顺序进行交替训练，其中一个鉴别器训练步骤和一个生成器训练步骤称为依次迭代。The generator network and the discriminator network can be trained alternately. In one embodiment, alternate training is performed in the order of generator training, discriminator training, generator training, and discriminator training steps, where one generator training step and one discriminator training step are referred to as sequential iterations. In another embodiment, the generator training step and the discriminator training step are exchanged in order, that is, the training is performed alternately in the order of the discriminator training step, the generator training step, the discriminator training step, and the generator training step, wherein one of the discriminators The training step and one generator training step are called successive iterations.
鉴别器网络与生成器网络都可以采取卷积神经网络的形式，均具有卷积申请网络的各个参数。例如，生成器网络的参数可以包括各卷积层的过滤器的权重、各激活函数层的偏执和各注意力机制模块的强化参数；鉴别器网络的参数可以包括各激活函数层的偏执、各卷积层的过滤器的权重和全连接层的降级参数。在初始化时，生成器网络的参数和鉴别器网络的参数可以是预设值或随机给定值。Both the discriminator network and the generator network can take the form of a convolutional neural network, and both have various parameters of the convolutional application network. For example, the parameters of the generator network may include the weights of the filters of each convolutional layer, the paranoia of each activation function layer, and the reinforcement parameters of each attention mechanism module; the parameters of the discriminator network may include the paranoia of each activation function layer, the The weights of the filters of the convolutional layers and the degradation parameters of the fully connected layers. At initialization, the parameters of the generator network and the parameters of the discriminator network can be preset or randomly given values.
由于生成器损失函数基于鉴别器网络的参数，因此生成器网络的训练基于鉴别器网络的训练结果(即渐变器网络的参数的训练结果)，而鉴别器网络的训练需要用到生成器网络的输出图像，因此鉴别器网络的训练又基于生成器网络的训练结果(即生成器网络的参数的训练结果)，这种方式称为“对抗”，即生成器网络和鉴别器网络相互对抗。这种方式使得两个相互对抗的网络在每次迭代中基于另一网络的越来越好的结果而进行竞争和不断改进，以训练得到越来越优异的参数。Since the generator loss function is based on the parameters of the discriminator network, the training of the generator network is based on the training results of the discriminator network (that is, the training results of the parameters of the gradienter network), and the training of the discriminator network requires the use of the generator network. The output image, so the training of the discriminator network is based on the training results of the generator network (that is, the training results of the parameters of the generator network), this way is called "adversarial", that is, the generator network and the discriminator network fight against each other. This approach allows two adversarial networks to compete and continuously improve on each iteration based on the better and better results of the other network to train with better and better parameters.
优选地，在一个实施方案中，所述生成器训练步骤中的循环一致性损失函数可以根据生成器损失函数和鉴别器损失函数获得。例如，循环一致性损失函数可由两部分构成，其中第一部分基于所述生成器网络的输出图像与无噪声图像之间的均方误差输出，第二部分基于所述生成器网络的输出图像经过所述鉴别器网络的输出。在一个实施方案中，在循环一致性损失函数中加入均方误差损失，以避免单一对抗损失函数带来图像过渡平滑的缺点，从而保留图像更多的细节。Preferably, in one embodiment, the cycle consistency loss function in the generator training step can be obtained from the generator loss function and the discriminator loss function. For example, the cycle consistency loss function can be composed of two parts, where the first part is based on the mean square error output between the output image of the generator network and the noise-free image, and the second part is based on the output image of the generator network through all output of the discriminator network. In one embodiment, a mean squared error loss is added to the cycle consistency loss function to avoid the disadvantage of smooth image transition caused by a single adversarial loss function, thereby preserving more details of the image.
为了解决因噪声带来的图像质量差的问题，可以增加基于3D注意力机制的最小二乘损失函数生成抗网络的MRI降噪网络，以通过对高噪声的MRI图像使用3D注意力最小二乘生成对抗网络对图像进行滤波达到能供医生诊断的医学图像。In order to solve the problem of poor image quality caused by noise, an MRI noise reduction network based on the least squares loss function generation anti-network based on 3D attention mechanism can be added to use 3D attention least squares for high-noise MRI images Generative Adversarial Networks filter the image to arrive at a medical image that can be diagnosed by a doctor.
优选地，在根据一个实施方案的3D注意力最小二乘生成对抗网络框架中，为提高噪声图像与真实图像之间的映射能力和网络的训练过程，最小二乘对抗损失可以表示为：Preferably, in the 3D attention least squares generative adversarial network framework according to one embodiment, in order to improve the mapping ability between noisy images and real images and the training process of the network, the least squares adversarial loss can be expressed as:
Figure PCTCN2020112870-appb-000001
Figure PCTCN2020112870-appb-000001
Figure PCTCN2020112870-appb-000002
Figure PCTCN2020112870-appb-000002
G为生成器，其中L LSGAN(G)表示生成器的损失函数，L LSGAN(D)为鉴别器的损失函数，P x(x)和P y(y)分别表示噪声数据和真实的标签数据分布；x表示噪声数据、y表示真实的标签数据，G(x)为以噪声图像数据作为输入时生成器输出的结果、D(G(x))为以G(x)作为输入时鉴别器输出的概率、G(y)为以真实的标签数据作为输入时鉴别器输出的概率、IE表示损失计算函数。噪声数据与真实数据之间的差异越大，D(G(x))指示为训练后的生成器输出的图像，L LSGAN(G)越大，噪声数据与真实数据之间的差异越小，D(G(x))指 示为真实数据，L LSGAN(G)越接近0。 G is the generator, where L LSGAN (G) represents the loss function of the generator, L LSGAN (D) is the loss function of the discriminator, and P x (x) and P y (y) represent the noise data and real label data, respectively Distribution; x represents noise data, y represents real label data, G(x) is the result of the generator output when noise image data is used as input, D(G(x)) is the discriminator when G(x) is used as input The probability of the output, G(y) is the probability of the discriminator output when the real label data is used as input, and IE represents the loss calculation function. The larger the difference between the noisy data and the real data, D(G(x)) is indicated as the image output by the trained generator, the larger the L LSGAN (G), the smaller the difference between the noisy data and the real data, D(G(x)) is indicated as real data, the closer L LSGAN (G) is to 0.
为了避免单一的对抗损失导致的图像降噪后的图像平滑，丢失图像细节，可以在原有对抗损失基础上增加均方误差函数，均方差函数为：In order to avoid the smoothing of the image after noise reduction and the loss of image details caused by a single adversarial loss, the mean square error function can be added on the basis of the original adversarial loss. The mean square error function is:
Figure PCTCN2020112870-appb-000003
Figure PCTCN2020112870-appb-000003
其中，d、w、h分别为提取特征图的深度，宽度和高度；Among them, d, w, and h are the depth, width and height of the extracted feature map, respectively;
最终将所述循环一致性损失函数定义为：Finally, the cycle consistency loss function is defined as:
L 3D a-LSGAN＝λ 1L mse+λ 2L LSGAN(G) L 3D a-LSGAN =λ 1 L mse +λ 2 L LSGAN (G)
其中λ 1和λ 2是用于平衡不同比例的经验参数，为设定值；根据经验，我们将λ 1和λ 2分别设置为1和0.0001。 Among them, λ 1 and λ 2 are empirical parameters used to balance different ratios, which are set values; according to experience, we set λ 1 and λ 2 to be 1 and 0.0001, respectively.
参见图5，在一个优选方案中，所述生成器训练步骤之前还包括：Referring to Fig. 5, in a preferred solution, before the generator training step, it further includes:
S01，生成器构建步骤，基于U-Net网络结构构建多层深度卷积神经网络的生成器，所述生成器包括跳跃连接的编码解码网络，在U-Net网络结构的跳跃连接结构中加入自注意力机制以将编码区域边缘细节图像信息传递到对应的解码区域。S01, the generator construction step is to construct a generator of a multi-layer deep convolutional neural network based on the U-Net network structure, the generator includes a skip-connected encoder-decoder network, and the skip connection structure of the U-Net network structure is added with self- The attention mechanism is used to transfer the edge detail image information of the encoded region to the corresponding decoding region.
此处的边缘细节图像信息可以指图像的边缘信息、细节信息等。通过该步骤处理噪声图像后，一些较暗区域的细节也能清楚地看到，例如器官的轮廓、褶皱的折痕、气管的分布网络等，从而有助于医生做出正确的分析和诊断。The edge detail image information here may refer to edge information, detail information and the like of an image. After processing the noisy image through this step, the details of some darker areas can also be clearly seen, such as the outline of the organ, the folds of the folds, the distribution network of the trachea, etc., which helps doctors to make correct analysis and diagnosis.
与相关技术中采用的Resnet-101等网络相比，在基于U-Net等网络结构构建的多层深度卷积神经网络的对抗网络模型中，将参数数量缩减到其千分之一甚至万分之一，从而大幅降低计算因子的复杂度，加快图像处理速度，从技术层面提供了可实施性。Compared with the Resnet-101 and other networks used in related technologies, in the adversarial network model of the multi-layer deep convolutional neural network constructed based on the network structure such as U-Net, the number of parameters is reduced to one thousandth or even ten thousandths. One of them, thereby greatly reducing the complexity of the calculation factor, speeding up the image processing speed, and providing practicability from the technical level.
S02，鉴别器构建步骤，基于生成器网络构建多层深度卷积神经网络的 鉴别器。S02, the discriminator construction step, constructs the discriminator of the multi-layer deep convolutional neural network based on the generator network.
参见图2，在一个实施方案中，基于3D注意力机制的最小二乘生成抗网络的MRI降噪网络如下：所述生成器包括由多层卷积构架形成的编码网络、由多层反卷积构架形成的解码网络和多层自注意力机制模块；每层卷积构架与一层反卷积构架、一层注意力机制模块一一对应；Referring to FIG. 2, in one embodiment, the MRI denoising network based on the least squares generative anti-network of 3D attention mechanism is as follows: the generator includes an encoding network formed by a multi-layer convolutional architecture, The decoding network and multi-layer self-attention mechanism module formed by the product architecture; each layer of convolution architecture corresponds to one layer of deconvolution architecture and one layer of attention mechanism module;
每层卷积构架均包括：卷积层、批归一化层和激活函数层；Each layer of convolutional architecture includes: convolutional layer, batch normalization layer and activation function layer;
每层反卷积构架均包括：反卷积层、批归一化层和激活函数层。Each layer of deconvolution architecture includes: deconvolution layer, batch normalization layer and activation function layer.
参见图3，在一个实施方案中，每层注意力机制模块均可包括：基于对应层卷积构架的卷积层提取的第一特征图、基于对应层卷积构架的批归一化层提取的第二特征图和基于对应层卷积构架的激活函数层提取的第三特征图。在一个实施方案中，可以将第三特征图转置后与第二特征图相乘后经softmax激活函数得到注意力图；再将第一特征图与注意力图相乘获得自注意力特征图。Referring to FIG. 3, in one embodiment, each layer of attention mechanism module may include: a first feature map extracted based on a convolutional layer of a corresponding layer convolutional architecture, a batch normalization layer extracted based on a corresponding layered convolutional architecture The second feature map of and the third feature map extracted based on the activation function layer of the corresponding layer convolution architecture. In one embodiment, the attention map can be obtained by transposing the third feature map and multiplying the second feature map by the softmax activation function; and then multiplying the first feature map and the attention map to obtain the self-attention feature map.
经过注意力机制模块的处理，图像可获得更多的细节信息，并通过跳跃连接传递到解码区域。上述第一特征图、第二特征图、第三特征图与图像的长、宽和特征通道数等参数有关。After being processed by the attention mechanism module, the image can obtain more detailed information and pass it to the decoding area through skip connections. The above-mentioned first feature map, second feature map, and third feature map are related to parameters such as the length, width, and number of feature channels of the image.
优选地，生成器训练步骤S10中获得输出图像的步骤可以包括：Preferably, the step of obtaining the output image in the generator training step S10 may include:
S101，卷积步骤，每层卷积操作时将训练集中带噪声的图像数据随机切块后作为输入进行特征提取获得卷积层；将卷积层进行池化获得批归一化层，将批归一化层通过函数进行非线性组合获得激活函数层；S101, the convolution step, during the convolution operation of each layer, the image data with noise in the training set is randomly cut into pieces and then used as input for feature extraction to obtain a convolution layer; the convolution layer is pooled to obtain a batch normalization layer, and the batch The normalization layer obtains the activation function layer through nonlinear combination of functions;
S102，反卷积步骤，每层反卷积操作时将反卷积层与通过该层对应的自注意力机制模块获得的自注意力特征图相加后进行池化操作获得批归一 化层，再将批归一化层通过激活函数层激活后输出；S102, the deconvolution step, in the deconvolution operation of each layer, the deconvolution layer is added to the self-attention feature map obtained by the self-attention mechanism module corresponding to the layer, and then a pooling operation is performed to obtain a batch normalization layer , and then activate the batch normalization layer through the activation function layer and output it;
S103，输出步骤，所有层卷积和反卷积均完成后输出获得输出图像。S103, in the output step, the output image is obtained after all layers of convolution and deconvolution are completed.
优选地，鉴别器可包括层数与生成器相同的卷积构架和全连接层；Preferably, the discriminator may comprise a convolutional architecture and fully connected layers with the same number of layers as the generator;
每层卷积构架均包括：卷积层、批归一化层和激活函数层。Each layer of convolutional architecture includes: convolution layer, batch normalization layer and activation function layer.
优选地，鉴别器训练步骤S20中获得输出指示的步骤可以包括：Preferably, the step of obtaining the output indication in the discriminator training step S20 may include:
S201，卷积步骤，以生成器输出的增强图像切块后作为输入进行特征提取获得卷积层；将卷积层进行池化获得批归一化层，将批归一化层通过函数进行非线性组合获得激活函数层；S201, the convolution step, taking the enhanced image output by the generator into blocks and then using it as an input to perform feature extraction to obtain a convolutional layer; pooling the convolutional layer to obtain a batch normalization layer, and performing a non-decoding process on the batch normalization layer through a function Linear combination to obtain activation function layer;
S202，连接步骤，通过全连接层对完成所有层卷积操作获得的特征进行非线性组合，在鉴别器的损失函数接近于1时确定为所述鉴别器网络的输入是所述训练后的生成器网络的输出图像；在鉴别器的损失函数接近于0时确定为所述鉴别器网络的输入是所述无噪声图像。S202 , the connection step is to non-linearly combine the features obtained by completing the convolution operations of all layers through the fully connected layer, and when the loss function of the discriminator is close to 1, it is determined that the input of the discriminator network is the generated after training. The output image of the discriminator network; when the loss function of the discriminator is close to 0, it is determined that the input of the discriminator network is the noise-free image.
优选地，在鉴别器构建步骤之后、生成器训练之前还可包括：Preferably, after the step of constructing the discriminator and before the training of the generator, it may further include:
S03，采用Adam优化算法对由生成器构建步骤构建的生成器和由鉴别器构建步骤构建的鉴别器共同形成的对抗网络进行优化。S03, using the Adam optimization algorithm to optimize the adversarial network jointly formed by the generator constructed by the generator construction step and the discriminator constructed by the discriminator construction step.
参见图2，在一个实施方案中，采用是带有跳跃连接的类似于U-net网络的编码解码网络作为生成器网络。所有的卷积都由处理3D数据的3D卷积层进行，其中在跳跃连接部分增加了注意力机制，以将编码区域的细节图像信息传递到对应的解码区域，从而使得解码网络将细节图像信息恢复到图像当中。在一个实施方案中，生成器网络一共包含8层，包括4个卷积层和4个反卷积层。每层包含3D卷积层、批归一化层和激活函数层。所使用的卷积核大小都为3×3×3像素，卷积核数量可以为例如32、64、128、 256、128、64、32。在一个实施方案中，卷积步长都为1个像素。Referring to FIG. 2, in one embodiment, an encoder-decoder network similar to U-net network with skip connections is used as the generator network. All convolutions are performed by a 3D convolutional layer that processes 3D data, in which an attention mechanism is added to the skip connection part to transfer the detailed image information of the encoding area to the corresponding decoding area, so that the decoding network can transfer the detailed image information to the corresponding decoding area. back to the image. In one embodiment, the generator network contains a total of 8 layers, including 4 convolutional layers and 4 deconvolutional layers. Each layer contains a 3D convolutional layer, a batch normalization layer, and an activation function layer. The size of the used convolution kernels is all 3×3×3 pixels, and the number of convolution kernels can be, for example, 32, 64, 128, 256, 128, 64, and 32. In one embodiment, the convolution strides are all 1 pixel.
参见图4，在一个实施方案中，鉴别器由四个卷积层和两个完全连接层(包括第一全连接层和第二全连接层)组成。每个卷积层后面都包含一个批归一化层和LeakeyRelu激活函数层。在四层卷积层之后连接有第一个全连接层，其中第一个全连接层的输出单元为1024个数值，后面为LeakeyRelu激活函数。第二个全连接层为一个输出单元为1个数值的全连接层和LeakeyRelu激活函数层。鉴别器和生成器使用的卷积核大小一致，都为3×3×3像素，并且每层中的卷积核数为32、64、128、256。第一个卷积层输出32张特征图，第二个卷积层输出64张特征图，第三个卷积层输出128张特征图，第四个卷积层输出256张特征图。Referring to Figure 4, in one embodiment, the discriminator consists of four convolutional layers and two fully connected layers (including a first fully connected layer and a second fully connected layer). Each convolutional layer is followed by a batch normalization layer and a LeakeyRelu activation function layer. After the four-layer convolutional layer is connected to the first fully connected layer, the output unit of the first fully connected layer is 1024 values, followed by the LeakeyRelu activation function. The second fully connected layer is a fully connected layer with an output unit of 1 value and a LeakeyRelu activation function layer. The discriminator and generator use the same convolution kernel size, both 3×3×3 pixels, and the number of convolution kernels in each layer is 32, 64, 128, 256. The first convolutional layer outputs 32 feature maps, the second convolutional layer outputs 64 feature maps, the third convolutional layer outputs 128 feature maps, and the fourth convolutional layer outputs 256 feature maps.
将带有噪声的MRI图像和不带噪声的MRI图像进行随机切块，得到3D像素块和对应的无噪声MRI图像作为对抗网络的输入和标签进行训练，注意力机制考虑了切块之间的关联信息并能够将编码区域(卷积层)重要的信息通过跳跃连接传递给对应的解码部分(反卷积层)。在达到收敛设定的条件时完成网络参数的训练，获得训练网络，同时得到由带噪声的MRI图像到无噪声的MRI图像映射关系G。最后，将从带噪声的MRI图像通过训练好的3D注意力最小二乘生成对抗网络进行降噪，得到符合医生诊断要求的降噪图像。The MRI image with noise and the MRI image without noise are randomly divided into 3D pixel blocks and the corresponding noise-free MRI image is used as the input and label of the adversarial network for training, and the attention mechanism considers the difference between the blocks. Correlated information and can pass the important information of the coding region (convolutional layer) to the corresponding decoding part (deconvolutional layer) through skip connections. When the set condition of convergence is reached, the training of network parameters is completed, the training network is obtained, and the mapping relationship G from the MRI image with noise to the MRI image without noise is obtained at the same time. Finally, denoise the noisy MRI images through the trained 3D attention least squares generative adversarial network to obtain denoised images that meet the doctor's diagnostic requirements.
3D注意力最小二乘生成对抗网络可以将带噪声的3D MRI图像进行降噪，并达到符合医生诊断要求的高质量MR图像。在编码和解码时，还可以通过使用注意力机制保留图像之中的边缘细节信息。The 3D attention least squares generative adversarial network can denoise the noisy 3D MRI images and achieve high-quality MR images that meet the diagnostic requirements of doctors. Edge details in the image can also be preserved by using an attention mechanism during encoding and decoding.
需要说明的是，除应用于MRI图像降噪之外，根据本申请的方法经适 当变换后也可应用于3D SPECT图像、低剂量3D CT图像和低计数3DPET等领域的图像降噪。It should be noted that, in addition to being applied to MRI image noise reduction, the method according to the present application can also be applied to image noise reduction in the fields of 3D SPECT images, low-dose 3D CT images, and low-count 3DPET after appropriate transformation.
本申请还提供了一种用于卷积神经网络的图像处理***，其包括：处理器和存储器；存储器存储了计算机可读代码，所述处理器在运行计算机可读代码时执行前述图像处理方法中的任一种。The present application also provides an image processing system for a convolutional neural network, comprising: a processor and a memory; the memory stores computer-readable codes, and the processor executes the aforementioned image processing method when running the computer-readable codes any of the.
该图像处理***的实施原理为：将上述图像处理方法以计算机可读代码的形式呈现并存储于存储器内，在***处理器运行存储器内的计算机可读代码时，执行上述图像处理方法的步骤获得提升图像处理速度并优化图像边缘信息的效果。The implementation principle of the image processing system is as follows: the above-mentioned image processing method is presented in the form of computer-readable code and stored in the memory, and when the system processor runs the computer-readable code in the memory, the steps of the above-mentioned image processing method are executed to obtain Improve image processing speed and optimize the effect of image edge information.
本申请还提供了一种计算机存储介质，其存储有计算机可读代码，所述处理器在运行所述计算机可读代码时执行上述图像处理方法中的任一种。The present application also provides a computer storage medium storing computer-readable codes, and the processor executes any one of the above image processing methods when running the computer-readable codes.
该计算机存储介质的实施原理为：将上述图像处理方法以计算机可读代码的形式呈现并存储于计算机存储介质上，在处理器运行该介质上的计算机代码时，执行上述图像处理方法的步骤以获得提升图像处理速度并优化图像边缘信息的效果。The implementation principle of the computer storage medium is as follows: the above-mentioned image processing method is presented in the form of computer-readable codes and stored on the computer storage medium, and when the processor runs the computer code on the medium, the steps of the above-mentioned image processing method are executed to Get the effect of speeding up image processing and optimizing image edge information.
以上均为本申请的较佳实施例，并非依此限制本申请的保护范围，故：凡依本申请的结构、形状、原理所做的等效变化，均应涵盖于本申请的保护范围之内。The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Therefore: all equivalent changes made according to the structure, shape and principle of the present application should be covered within the scope of the present application. Inside.

Claims (10)

  1. 一种图像处理方法，其特征在于，包括：An image processing method, comprising:
    生成器训练步骤，从训练集中提取噪声图像数据作为输入图像以通过减小循环一致性损失函数来训练生成器网络参数，以使所述生成器网络的输出图像与所述训练集中无噪声图像的差异减小；所述生成器网络包括用于提升输入图像的边缘细节对比度的注意力机制模块；A generator training step that extracts noisy image data from the training set as input images to train generator network parameters by reducing the cycle consistency loss function so that the output images of the generator network are the same as the noise-free images in the training set. The difference is reduced; the generator network includes an attention mechanism module for enhancing the contrast of edge details of the input image;
    鉴别器训练步骤，通过训练后的生成器网络的输出图像和无噪声图像分别输入鉴别器网络以通过减小鉴别器的损失函数来训练鉴别器网络的参数，使得鉴别器网络的输出指示所述鉴别器网络的输入是所述训练后的生成器网络的输出图像还是指示所述无噪声图像；The discriminator training step, the output image of the trained generator network and the noise-free image are respectively input to the discriminator network to train the parameters of the discriminator network by reducing the loss function of the discriminator such that the output of the discriminator network indicates the whether the input to the discriminator network is the output image of the trained generator network or an indication of the noise-free image;
    采用不同的噪声图像重复所述生成器训练步骤和所述鉴别器训练步骤以通过最小化循环一致性损失函数和鉴别器损失函数得到最终的生成器网络的参数和鉴别器网络的参数。The generator training step and the discriminator training step are repeated with different noise images to obtain the final generator network parameters and discriminator network parameters by minimizing the cycle consistency loss function and the discriminator loss function.
  2. 根据权利要求1所述的图像处理方法，其特征在于，所述循环一致性损失函数由两部分构成，其中第一部分基于所述生成器网络的输出图像与无噪声图像之间的均方误差输出，其中第二部分基于所述生成器网络的输出图像经过所述鉴别器网络的输出。The image processing method according to claim 1, wherein the cycle consistency loss function consists of two parts, wherein the first part is based on the mean square error output between the output image of the generator network and the noise-free image , wherein the second part is based on the output of the generator network through the output of the discriminator network.
  3. 根据权利要求2所述的图像处理方法，其特征在于，所述循环一致性损失函数为：The image processing method according to claim 2, wherein the cycle consistency loss function is:
    L 3D a-LSGAN＝λ 1L mse+λ 2L LSGAN(G) L 3D a-LSGAN =λ 1 L mse +λ 2 L LSGAN (G)
    其中λ 1和λ 2是用于平衡不同比例的经验参数，为设定值； where λ 1 and λ 2 are empirical parameters used to balance different ratios, which are set values;
    Figure PCTCN2020112870-appb-100001
    Figure PCTCN2020112870-appb-100001
    其中，d、w、h分别为提取特征图的深度，宽度和高度；Among them, d, w, and h are the depth, width and height of the extracted feature map, respectively;
    Figure PCTCN2020112870-appb-100002
    Figure PCTCN2020112870-appb-100002
    Figure PCTCN2020112870-appb-100003
    Figure PCTCN2020112870-appb-100003
    G为生成器，其中L LSGAN(G)表示生成器的损失函数，L LSGAN(D)为鉴别器的损失函数，P x(x)和P y(y)分别表示噪声数据和真实的标签数据分布；x表示噪声数据、y表示真实的标签数据，G(x)为以噪声图像数据作为输入时生成器输出的结果、D(G(x))为以G(x)作为输入时鉴别器输出的概率、G(y)为以真实的标签数据作为输入时鉴别器输出的概率。 G is the generator, where L LSGAN (G) represents the loss function of the generator, L LSGAN (D) is the loss function of the discriminator, and P x (x) and P y (y) represent the noise data and real label data, respectively Distribution; x represents noise data, y represents real label data, G(x) is the result of the generator output when noise image data is used as input, D(G(x)) is the discriminator when G(x) is used as input The output probability, G(y), is the probability of the discriminator output when the real label data is used as input.
  4. 根据权利要求1～3任一项所述的图像处理方法，其特征在于，所述生成器训练步骤之前还包括：The image processing method according to any one of claims 1 to 3, wherein before the generator training step further comprises:
    生成器构建步骤，基于U-Net网络结构构建多层深度卷积神经网络的生成器，所述生成器包括跳跃连接的编码解码网络，在U-Net网络结构的跳跃连接结构中加入所述生成器网络以将编码区域边缘细节图像信息传递到对应的解码区域；The generator construction step is to construct a generator of a multi-layer deep convolutional neural network based on the U-Net network structure, the generator includes a skip-connected encoding-decoding network, and the generation The encoder network is used to transfer the edge detail image information of the encoding region to the corresponding decoding region;
    鉴别器构建步骤，基于生成器网络构建多层深度卷积神经网络的鉴别器。The discriminator construction step, builds the discriminator of the multi-layer deep convolutional neural network based on the generator network.
  5. 根据权利要求4所述的图像处理方法，其特征在于，所述生成器包括由多层卷积构架形成的编码网络、由多层反卷积构架形成的解码网络和多层注意力机制模块；每层卷积构架与一层反卷积构架、一层注意力机制模块一一对应；The image processing method according to claim 4, wherein the generator comprises an encoding network formed by a multi-layer convolution framework, a decoding network formed by a multi-layer deconvolution framework, and a multi-layer attention mechanism module; Each layer of convolutional architecture has a one-to-one correspondence with a layer of deconvolution architecture and a layer of attention mechanism modules;
    每层卷积构架均包括：卷积层、批归一化层和激活函数层；Each layer of convolutional architecture includes: convolutional layer, batch normalization layer and activation function layer;
    每层反卷积构架均包括：反卷积层、批归一化层和激活函数层；Each layer of deconvolution architecture includes: deconvolution layer, batch normalization layer and activation function layer;
    每层注意力机制模块均包括：基于对应层卷积构架的卷积层提取的第一特征图、基于对应层卷积构架的批归一化层提取的第二特征图和基于对应层卷积构架的激活函数层提取的第三特征图；Each layer of attention mechanism module includes: the first feature map extracted based on the convolutional layer of the corresponding layer convolutional architecture, the second feature map extracted based on the batch normalization layer of the corresponding layered convolutional architecture, and the second feature map extracted based on the corresponding layer convolutional layer The third feature map extracted by the activation function layer of the architecture;
    其中将第三特征图转置后与第二特征图相乘后经softmax激活函数得到注意力图；再将第一特征图与注意力图相乘获得自注意力特征图。The attention map is obtained by transposing the third feature map and multiplying the second feature map by the softmax activation function; and then multiplying the first feature map and the attention map to obtain the self-attention feature map.
  6. 根据权利要求5所述的图像处理方法，其特征在于，所述生成器训练步骤中获得输出图像的步骤包括：The image processing method according to claim 5, wherein the step of obtaining an output image in the generator training step comprises:
    卷积步骤，每层卷积操作时将训练集中带噪声的图像数据随机切块后作为输入进行特征提取获得卷积层；将卷积层进行池化获得批归一化层，将批归一化层通过函数进行非线性组合获得激活函数层；In the convolution step, during the convolution operation of each layer, the noisy image data in the training set is randomly cut into pieces and used as input for feature extraction to obtain a convolution layer; the convolution layer is pooled to obtain a batch normalization layer, and the batch normalization layer is obtained. The transformation layer obtains the activation function layer through nonlinear combination of functions;
    反卷积步骤，每层反卷积操作时将反卷积层与通过该层对应的自注意力机制模块获得的自注意力特征图相加后进行池化操作获得批归一化层，再将批归一化层通过激活函数层激活后输出；In the deconvolution step, in the deconvolution operation of each layer, the deconvolution layer is added to the self-attention feature map obtained by the self-attention mechanism module corresponding to the layer, and then the pooling operation is performed to obtain a batch normalization layer, and then The batch normalization layer is activated by the activation function layer and output;
    输出步骤，所有层卷积和反卷积均完成后输出获得输出图像。In the output step, the output image is obtained after all layers of convolution and deconvolution are completed.
  7. 根据权利要求4所述的图像处理方法，其特征在于，所述鉴别器包括层数与生成器相同的卷积构架和全连接层；The image processing method according to claim 4, wherein the discriminator comprises a convolutional framework and a fully connected layer with the same number of layers as the generator;
    每层卷积构架均包括：卷积层、批归一化层和激活函数层；Each layer of convolutional architecture includes: convolutional layer, batch normalization layer and activation function layer;
    所述鉴别器训练步骤中获得输出指示的步骤包括：The step of obtaining an output indication in the discriminator training step includes:
    卷积步骤，以生成器输出的增强图像切块后作为输入进行特征提取获得卷积层；将卷积层进行池化获得批归一化层，将批归一化层通过函数进行非线性组合获得激活函数层；In the convolution step, the enhanced image output by the generator is cut into pieces and used as input for feature extraction to obtain a convolution layer; the convolution layer is pooled to obtain a batch normalization layer, and the batch normalization layer is nonlinearly combined through a function Get the activation function layer;
    连接步骤，通过全连接层对完成所有层卷积操作获得的特征进行非线性组合，在鉴别器的损失函数接近于1时确定为所述鉴别器网络的输入是所述训练后的生成器网络的输出图像；在鉴别器的损失函数接近于0时确定为所述鉴别器网络的输入是所述无噪声图像。In the connection step, the features obtained by completing the convolution operations of all layers are nonlinearly combined through the fully connected layer, and when the loss function of the discriminator is close to 1, it is determined that the input of the discriminator network is the trained generator network. The output image of ; it is determined that the input of the discriminator network is the noise-free image when the loss function of the discriminator is close to 0.
  8. 根据权利要求4所述的图像处理方法，其特征在于，在所述鉴别器构建步骤之后还包括：The image processing method according to claim 4, characterized in that, after the step of constructing the discriminator, it further comprises:
    采用Adam优化算法对由生成器构建步骤构建的生成器和由鉴别器构建步骤构建的鉴别器共同形成的对抗网络进行优化。The adversarial network formed by the generator constructed by the generator construction step and the discriminator constructed by the discriminator construction step is optimized using the Adam optimization algorithm.
  9. 一种用于卷积神经网络的图像处理***，包括：An image processing system for convolutional neural networks, comprising:
    处理器；processor;
    存储器，存储了计算机可读代码，在计算机可读代码被处理器运行时执行如权利要求1～8中任一项所述的图像处理方法的步骤。The memory stores computer readable codes, and when the computer readable codes are executed by the processor, the steps of the image processing method according to any one of claims 1 to 8 are executed.
  10. 一种计算机存储介质，存储了计算机可读代码，所述计算机可读代码被运行时执行如权利要求1～9中任一项所述的图像处理方法的步骤。A computer storage medium storing computer-readable codes, when the computer-readable codes are executed, the steps of the image processing method according to any one of claims 1 to 9 are executed.
PCT/CN2020/112870 2020-09-01 2020-09-01 Image processing method and system, and computer storage medium WO2022047625A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/112870 WO2022047625A1 (en) 2020-09-01 2020-09-01 Image processing method and system, and computer storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/112870 WO2022047625A1 (en) 2020-09-01 2020-09-01 Image processing method and system, and computer storage medium

Publications (1)

Publication Number Publication Date
WO2022047625A1 true WO2022047625A1 (en) 2022-03-10

Family

ID=80492111

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/112870 WO2022047625A1 (en) 2020-09-01 2020-09-01 Image processing method and system, and computer storage medium

Country Status (1)

Country Link
WO (1) WO2022047625A1 (en)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114757847A (en) * 2022-04-24 2022-07-15 汕头市超声仪器研究所股份有限公司 Multi-information extraction extended U-Net and application method thereof in low-dose X-ray imaging
CN114842384A (en) * 2022-04-30 2022-08-02 南京邮电大学 6G-oriented tactile modal signal reconstruction method
CN114943322A (en) * 2022-04-11 2022-08-26 山东大学 Automatic generation method and system from layout to scene image based on deep learning
CN115115544A (en) * 2022-06-22 2022-09-27 哈尔滨理工大学 Irradiation environment image noise reduction method based on attention mechanism GAN
CN115272679A (en) * 2022-08-08 2022-11-01 北京理工大学 Method, device, terminal and storage medium for identifying geothermal favorable area
US11540798B2 (en) 2019-08-30 2023-01-03 The Research Foundation For The State University Of New York Dilated convolutional neural network system and method for positron emission tomography (PET) image denoising
CN115620113A (en) * 2022-12-20 2023-01-17 成都理工大学 Elastic wave vector separation method for generating countermeasure network based on deep convolution
CN115721318A (en) * 2022-11-24 2023-03-03 西安工程大学 Electrocardiosignal noise reduction processing method, system, equipment and storage medium
CN115860054A (en) * 2022-07-21 2023-03-28 广州工商学院 Sparse codebook multiple access coding and decoding system based on generation countermeasure network
CN115984106A (en) * 2022-12-12 2023-04-18 武汉大学 Line scanning image super-resolution method based on bilateral generation countermeasure network
CN116051662A (en) * 2023-03-31 2023-05-02 腾讯科技（深圳）有限公司 Image processing method, device, equipment and medium
CN116051562A (en) * 2023-03-31 2023-05-02 北京大学 Metal 3D printing quality prediction method based on depth generation network
CN116168352A (en) * 2023-04-26 2023-05-26 成都睿瞳科技有限责任公司 Power grid obstacle recognition processing method and system based on image processing
CN116245856A (en) * 2023-03-15 2023-06-09 石鹏飞 Image fitting processing system and method for numerical conversion
CN116843696A (en) * 2023-04-27 2023-10-03 山东省人工智能研究院 Cardiac MRI (magnetic resonance imaging) segmentation method based on feature similarity and super-parameter convolution attention
CN116930884A (en) * 2023-09-15 2023-10-24 西安电子科技大学 SAR deception jamming template generation and jamming method based on optical SAR image conversion
WO2023221421A1 (en) * 2022-05-19 2023-11-23 浙江大学 Image and segmentation label generation model of tree structure data and application
CN117409192A (en) * 2023-12-14 2024-01-16 武汉大学 Data enhancement-based infrared small target detection method and device
CN117409376A (en) * 2023-12-15 2024-01-16 南京中鑫智电科技有限公司 Infrared online monitoring method and system for high-voltage sleeve
CN117437514A (en) * 2023-12-22 2024-01-23 南昌航空大学 Colposcope image mode conversion method based on CycleGan

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106127702A (en) * 2016-06-17 2016-11-16 兰州理工大学 A kind of image mist elimination algorithm based on degree of depth study
CN106296692A (en) * 2016-08-11 2017-01-04 深圳市未来媒体技术研究院 Image significance detection method based on antagonism network
US20170031920A1 (en) * 2015-07-31 2017-02-02 RCRDCLUB Corporation Evaluating performance of recommender system
CN107122826A (en) * 2017-05-08 2017-09-01 京东方科技集团股份有限公司 Processing method and system and storage medium for convolutional neural networks

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170031920A1 (en) * 2015-07-31 2017-02-02 RCRDCLUB Corporation Evaluating performance of recommender system
CN106127702A (en) * 2016-06-17 2016-11-16 兰州理工大学 A kind of image mist elimination algorithm based on degree of depth study
CN106296692A (en) * 2016-08-11 2017-01-04 深圳市未来媒体技术研究院 Image significance detection method based on antagonism network
CN107122826A (en) * 2017-05-08 2017-09-01 京东方科技集团股份有限公司 Processing method and system and storage medium for convolutional neural networks

Cited By (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11540798B2 (en) 2019-08-30 2023-01-03 The Research Foundation For The State University Of New York Dilated convolutional neural network system and method for positron emission tomography (PET) image denoising
CN114943322A (en) * 2022-04-11 2022-08-26 山东大学 Automatic generation method and system from layout to scene image based on deep learning
CN114757847A (en) * 2022-04-24 2022-07-15 汕头市超声仪器研究所股份有限公司 Multi-information extraction extended U-Net and application method thereof in low-dose X-ray imaging
CN114842384B (en) * 2022-04-30 2024-05-31 南京邮电大学 6G-oriented haptic mode signal reconstruction method
CN114842384A (en) * 2022-04-30 2022-08-02 南京邮电大学 6G-oriented tactile modal signal reconstruction method
WO2023221421A1 (en) * 2022-05-19 2023-11-23 浙江大学 Image and segmentation label generation model of tree structure data and application
CN115115544A (en) * 2022-06-22 2022-09-27 哈尔滨理工大学 Irradiation environment image noise reduction method based on attention mechanism GAN
CN115860054A (en) * 2022-07-21 2023-03-28 广州工商学院 Sparse codebook multiple access coding and decoding system based on generation countermeasure network
CN115860054B (en) * 2022-07-21 2023-09-26 广州工商学院 Sparse codebook multiple access coding and decoding system based on generation countermeasure network
CN115272679A (en) * 2022-08-08 2022-11-01 北京理工大学 Method, device, terminal and storage medium for identifying geothermal favorable area
CN115272679B (en) * 2022-08-08 2024-03-19 北京理工大学 Identification method, device, terminal and storage medium for geothermal advantageous zone
CN115721318A (en) * 2022-11-24 2023-03-03 西安工程大学 Electrocardiosignal noise reduction processing method, system, equipment and storage medium
CN115984106A (en) * 2022-12-12 2023-04-18 武汉大学 Line scanning image super-resolution method based on bilateral generation countermeasure network
CN115984106B (en) * 2022-12-12 2024-04-02 武汉大学 Line scanning image super-resolution method based on bilateral generation countermeasure network
CN115620113B (en) * 2022-12-20 2023-04-07 成都理工大学 Elastic wave vector separation method for generating countermeasure network based on deep convolution
CN115620113A (en) * 2022-12-20 2023-01-17 成都理工大学 Elastic wave vector separation method for generating countermeasure network based on deep convolution
CN116245856A (en) * 2023-03-15 2023-06-09 石鹏飞 Image fitting processing system and method for numerical conversion
CN116245856B (en) * 2023-03-15 2023-09-22 湖北华中电力科技开发有限责任公司 Image fitting processing system and method for numerical conversion
CN116051562A (en) * 2023-03-31 2023-05-02 北京大学 Metal 3D printing quality prediction method based on depth generation network
CN116051562B (en) * 2023-03-31 2023-06-13 北京大学 Metal 3D printing quality prediction method based on depth generation network
CN116051662A (en) * 2023-03-31 2023-05-02 腾讯科技（深圳）有限公司 Image processing method, device, equipment and medium
CN116168352B (en) * 2023-04-26 2023-06-27 成都睿瞳科技有限责任公司 Power grid obstacle recognition processing method and system based on image processing
CN116168352A (en) * 2023-04-26 2023-05-26 成都睿瞳科技有限责任公司 Power grid obstacle recognition processing method and system based on image processing
CN116843696A (en) * 2023-04-27 2023-10-03 山东省人工智能研究院 Cardiac MRI (magnetic resonance imaging) segmentation method based on feature similarity and super-parameter convolution attention
CN116843696B (en) * 2023-04-27 2024-04-09 山东省人工智能研究院 Cardiac MRI (magnetic resonance imaging) segmentation method based on feature similarity and super-parameter convolution attention
CN116930884B (en) * 2023-09-15 2023-12-26 西安电子科技大学 SAR deception jamming template generation and jamming method based on optical SAR image conversion
CN116930884A (en) * 2023-09-15 2023-10-24 西安电子科技大学 SAR deception jamming template generation and jamming method based on optical SAR image conversion
CN117409192A (en) * 2023-12-14 2024-01-16 武汉大学 Data enhancement-based infrared small target detection method and device
CN117409192B (en) * 2023-12-14 2024-03-08 武汉大学 Data enhancement-based infrared small target detection method and device
CN117409376A (en) * 2023-12-15 2024-01-16 南京中鑫智电科技有限公司 Infrared online monitoring method and system for high-voltage sleeve
CN117409376B (en) * 2023-12-15 2024-05-10 南京中鑫智电科技有限公司 Infrared online monitoring method and system for high-voltage sleeve
CN117437514A (en) * 2023-12-22 2024-01-23 南昌航空大学 Colposcope image mode conversion method based on CycleGan
CN117437514B (en) * 2023-12-22 2024-04-05 南昌航空大学 Colposcope image mode conversion method based on CycleGan

Similar Documents

Publication Publication Date Title
WO2022047625A1 (en) Image processing method and system, and computer storage medium
CN112085677B (en) Image processing method, system and computer storage medium
CN110889852B (en) Liver segmentation method based on residual error-attention deep neural network
Jiang et al. Low-dose CT lung images denoising based on multiscale parallel convolution neural network
CN111242956A (en) U-Net-based ultrasonic fetal heart and fetal lung deep learning joint segmentation method
CN113191968B (en) Method for establishing three-dimensional ultrasonic image blind denoising model and application thereof
CN111899165A (en) Multi-task image reconstruction convolution network model based on functional module
CN114943721A (en) Neck ultrasonic image segmentation method based on improved U-Net network
CN111091575A (en) Medical image segmentation method based on reinforcement learning method
CN110570394A (en) medical image segmentation method, device, equipment and storage medium
CN114972362A (en) Medical image automatic segmentation method and system based on RMAU-Net network
CN112634192A (en) Cascaded U-N Net brain tumor segmentation method combined with wavelet transformation
CN116342444A (en) Dual-channel multi-mode image fusion method and fusion imaging terminal
CN116645283A (en) Low-dose CT image denoising method based on self-supervision perceptual loss multi-scale convolutional neural network
CN112070752A (en) Method, device and storage medium for segmenting auricle of medical image
CN114187181A (en) Double-path lung CT image super-resolution method based on residual information refining
CN116503506B (en) Image reconstruction method, system, device and storage medium
Sander et al. Autoencoding low-resolution MRI for semantically smooth interpolation of anisotropic MRI
CN116563554A (en) Low-dose CT image denoising method based on hybrid characterization learning
Zhou et al. High-resolution hierarchical adversarial learning for OCT speckle noise reduction
Cheng et al. CAIR: Combining integrated attention with iterative optimization learning for sparse-view CT reconstruction
CN116188352A (en) Pulmonary nodule segmentation method based on enhanced edge features
CN114387257A (en) Segmentation method, system, device and medium for lung lobe region in lung image
CN114862982A (en) Hybrid domain unsupervised finite angle CT reconstruction method based on generation countermeasure network
CN114219738A (en) Single-image multi-scale super-resolution reconstruction network structure and method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20951873

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20951873

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 09/08/2023)