CN111462041B - Image processing method, device, equipment and storage medium - Google Patents

Image processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN111462041B
CN111462041B CN202010137174.8A CN202010137174A CN111462041B CN 111462041 B CN111462041 B CN 111462041B CN 202010137174 A CN202010137174 A CN 202010137174A CN 111462041 B CN111462041 B CN 111462041B
Authority
CN
China
Prior art keywords
convolution
image
characteristic information
level
subunit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010137174.8A
Other languages
Chinese (zh)
Other versions
CN111462041A (en
Inventor
张朴
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Neusoft Medical Systems Co Ltd
Beijing Neusoft Medical Equipment Co Ltd
Original Assignee
Neusoft Medical Systems Co Ltd
Beijing Neusoft Medical Equipment Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Neusoft Medical Systems Co Ltd, Beijing Neusoft Medical Equipment Co Ltd filed Critical Neusoft Medical Systems Co Ltd
Priority to CN202010137174.8A priority Critical patent/CN111462041B/en
Publication of CN111462041A publication Critical patent/CN111462041A/en
Application granted granted Critical
Publication of CN111462041B publication Critical patent/CN111462041B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G06T3/4076—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution using the original low-resolution images to iteratively correct the high-resolution images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30101—Blood vessel; Artery; Vein; Vascular

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

The present disclosure provides an image processing method, apparatus, device, and storage medium, where an angiographic image is processed by using a pre-trained neural network, where the neural network includes a convolution unit, N cascaded convolution units, and an upsampling convolution unit; the method comprises the following steps: inputting the input image into a convolution unit, and acquiring initial characteristic information of the input image, wherein the input image is one frame in an angiography image sequence; inputting the initial characteristic information into a first-stage convolution circulation unit to obtain intermediate characteristic information output by each stage convolution circulation unit; adding the initial characteristic information and the intermediate characteristic information output by the nth-stage convolution circulation unit to obtain combined characteristic information; and inputting the combined characteristic information into an up-sampling convolution unit, performing up-sampling convolution operation on the combined characteristic information, and obtaining a target reconstructed image according to the obtained up-sampling result, wherein N is a positive integer.

Description

Image processing method, device, equipment and storage medium
Technical Field
The disclosure relates to the technical field of medical equipment, and in particular relates to an image processing method, an image processing device, an image processing equipment and a storage medium.
Background
Angiography (Digital Substraction Angiography, DSA for short) is a technique in which blood vessels are visualized with a contrast agent and X-rays, thereby generating a sequence of angiographic images; the technology is used for diagnosing and treating vascular diseases, and because the placement of fine accessories such as vascular stents, spring coils and the like is often involved in the treatment process, the details are required to be ensured to be clear and complete in the image so as to provide accurate basis for doctors to formulate treatment schemes. Obtaining high resolution angiographic images requires long, focused X-ray radiation, which is clearly detrimental to the health of the subject (i.e. patient). Therefore, it is necessary to obtain high-resolution angiographic images while avoiding the influence on the health of the subject during angiographic procedures as much as possible.
Disclosure of Invention
The present disclosure provides an image processing method, apparatus, device, and storage medium.
Specifically, the present disclosure is implemented by the following technical scheme:
in a first aspect, an image processing method is provided, and an angiographic image is processed by using a pre-trained neural network, wherein the neural network comprises a convolution unit, N cascaded convolution circulation units and an up-sampling convolution unit; the method comprises the following steps: inputting the input image into a convolution unit, and acquiring initial characteristic information of the input image, wherein the input image is one frame in an angiography image sequence; inputting the initial characteristic information into a first-stage convolution circulation unit to obtain intermediate characteristic information output by each stage convolution circulation unit; adding the initial characteristic information and the intermediate characteristic information output by the nth-stage convolution circulation unit to obtain combined characteristic information; and inputting the combined characteristic information into an up-sampling convolution unit, performing up-sampling convolution operation on the combined characteristic information, and obtaining a target reconstructed image according to the obtained up-sampling result, wherein N is a positive integer.
Optionally, the convolution unit includes a first convolution subunit and a second convolution subunit; the inputting the input image to a convolution unit, obtaining initial characteristic information of the input image, including: inputting the input image to the first convolution subunit to obtain first initial characteristic information; and inputting the cascaded multiple pieces of characteristic information into the second convolution subunit to obtain second initial characteristic information, wherein the cascaded multiple pieces of characteristic information comprise the first initial characteristic information and the second initial characteristic information output by the second convolution subunit for the previous frame of the input image.
Optionally, the convolution cyclic unit comprises a first convolution cyclic subunit and a second convolution cyclic subunit, wherein each level of the first convolution cyclic subunit is input to the next level of the convolution cyclic subunit; the step of inputting the initial characteristic information into a first-stage convolution circulation unit to obtain intermediate characteristic information output by each stage convolution circulation unit, comprises the following steps: inputting the initial characteristic information into a first-stage convolution circulation subunit to obtain first intermediate characteristic information output by each stage convolution circulation subunit; inputting the cascaded multiple pieces of characteristic information into a second convolution circulation subunit in each level of convolution circulation units to obtain second intermediate characteristic information, wherein for an ith level of second convolution circulation subunit, the cascaded multiple pieces of characteristic information comprise first intermediate characteristic information output by an ith level of first convolution circulation subunit for the input image, second intermediate characteristic information output by an ith-1 level of second convolution circulation subunit for the input image and second intermediate characteristic information output by the ith level of second convolution circulation subunit for a previous frame of the input image, wherein i is a positive integer less than or equal to N.
Optionally, the neural network further comprises an M-level up-sampling convolution subunit; performing up-sampling convolution operation on the combined characteristic information, and obtaining a target reconstructed image according to the obtained up-sampling result comprises the following steps: inputting the combined characteristic information to a j-th level up-sampling convolution subunit, and outputting a j-th level up-sampling result; obtaining a j-th level reconstruction image according to a j-th level up-sampling result, and obtaining the combined characteristic information by taking the j-th level reconstruction image as an input image; adding the j-th level up-sampling result and the combination characteristic information, and inputting the added result to a j+1-th level up-sampling convolution subunit; and obtaining a target reconstruction image according to an M-th up-sampling result output by the M-th up-sampling convolution subunit, wherein M is a positive integer, and j is a positive integer smaller than or equal to M-1.
Optionally, before acquiring the initial feature information of the input image, the processing method further includes: inputting a first resolution image of a pre-obtained image training set to the neural network, wherein the image training set comprises a sequence of angiographic image pairs, each comprising a first resolution image and a second resolution image for the same angiographic image, the second resolution image comprising a second resolution image of each stage corresponding to the magnification of the up-sampling convolution unit of each stage, the first resolution image having a resolution lower than the second resolution image; obtaining each level of reconstructed images corresponding to each level of up-sampling convolution units in the neural network; determining a network loss value according to each level of reconstructed image corresponding to the first resolution image and each level of second resolution image corresponding to the first resolution image; and adjusting network parameters of the neural network based on the network loss value.
Optionally, the determining the network loss value according to each level of reconstructed image corresponding to the first resolution image and each level of second resolution image corresponding to the first resolution image includes: determining loss values of each level according to each level of output image corresponding to the first resolution image and the second resolution image corresponding to the first resolution image; determining network loss values according to the weights corresponding to the reconstructed images at all levels and the loss values at all levels respectively; the adjusting the network parameters of the neural network based on the network loss value includes: and stopping the adjustment of the network parameters of the neural network when the network loss value is smaller than a preset loss value threshold value, and/or stopping the adjustment of the network parameters of the neural network when the adjustment times exceed a preset times threshold value.
Optionally, in inputting a first resolution image in a pre-obtained image training set to the neural network, the processing method further includes: acquiring a second resolution image sequence, wherein the second resolution images in the sequence comprise all levels of second resolution images corresponding to the amplification factors of all levels of up-sampling convolution units; performing low resolution processing on a second resolution image corresponding to the amplification factor of the highest-level up-sampling convolution unit to obtain a corresponding first resolution image, wherein the second resolution image and the corresponding first resolution image form an angiogram image pair; the angiographic image pairs are arranged according to the order of the second-resolution image sequence to form an image training set.
Optionally, the performing low resolution processing on the second resolution image corresponding to the magnification of the highest-level up-sampling convolution unit to obtain a corresponding first resolution image includes: blurring the second resolution image using a gaussian kernel function; and performing downsampling processing on the blurred second resolution image by using a bicubic interpolation method to obtain a corresponding first resolution image.
In a second aspect, an image processing apparatus is provided for processing an angiographic image using a pre-trained neural network, the neural network including a convolution unit, N cascaded convolution units, and an upsampling convolution unit; the device comprises: the first module is used for acquiring initial characteristic information of an input image, and the input image is one frame in an angiography image sequence; the intermediate feature acquisition module is used for inputting the initial feature information into the first-stage convolution circulation unit to obtain intermediate feature information output by each stage convolution circulation unit; the third module is used for adding the initial characteristic information and the intermediate characteristic information output by the nth-level convolution circulation unit to obtain combined characteristic information; and the reconstruction module is used for carrying out up-sampling convolution operation on the combined characteristic information and obtaining a target reconstruction image according to the obtained up-sampling result, wherein N is a positive integer.
Optionally, the convolution unit includes a first convolution subunit and a second convolution subunit; the initial feature acquisition unit is specifically configured to: inputting the input image to the first convolution subunit to obtain first initial characteristic information; and inputting the cascaded multiple pieces of characteristic information into the second convolution subunit to obtain second initial characteristic information, wherein the cascaded multiple pieces of characteristic information comprise the first initial characteristic information and the second initial characteristic information output by the second convolution subunit for the previous frame of the input image.
Optionally, the convolution cyclic unit comprises a first convolution cyclic subunit and a second convolution cyclic subunit, wherein each level of the first convolution cyclic subunit is input to the next level of the convolution cyclic subunit; the intermediate feature acquisition unit is specifically configured to: inputting the initial characteristic information into a first-stage convolution circulation subunit to obtain first intermediate characteristic information output by each stage convolution circulation subunit; inputting the cascaded multiple pieces of characteristic information into a second convolution circulation subunit in each level of convolution circulation units to obtain second intermediate characteristic information, wherein for an ith level of second convolution circulation subunit, the cascaded multiple pieces of characteristic information comprise first intermediate characteristic information output by an ith level of first convolution circulation subunit for the input image, second intermediate characteristic information output by an ith-1 level of second convolution circulation subunit for the input image and second intermediate characteristic information output by the ith level of second convolution circulation subunit for a previous frame of the input image, wherein i is a positive integer less than or equal to N.
Optionally, before acquiring the initial feature information of the input image, the convolution unit includes a first convolution subunit and a second convolution subunit; the inputting the input image to a convolution unit, obtaining initial characteristic information of the input image, including: inputting the input image to the first convolution subunit to obtain first initial characteristic information; and inputting the cascaded multiple pieces of characteristic information into the second convolution subunit to obtain second initial characteristic information, wherein the cascaded multiple pieces of characteristic information comprise the first initial characteristic information and the second initial characteristic information output by the second convolution subunit for the previous frame of the input image.
Optionally, the neural network further comprises an M-level up-sampling convolution subunit; the reconstruction unit is specifically configured to: inputting the combined characteristic information to a j-th level up-sampling convolution subunit, and outputting a j-th level up-sampling result; obtaining a j-th level reconstruction image according to a j-th level up-sampling result, and obtaining the combined characteristic information by taking the j-th level reconstruction image as an input image; adding the j-th level up-sampling result and the combination characteristic information, and inputting the added result to a j+1-th level up-sampling convolution subunit; and obtaining a target reconstruction image according to an M-th up-sampling result output by the M-th up-sampling convolution subunit, wherein M is a positive integer, and j is a positive integer smaller than or equal to M-1.
Optionally, the apparatus further comprises a training unit for: inputting a first resolution image of a pre-obtained image training set to the neural network, wherein the image training set comprises a sequence of angiographic image pairs, each comprising a first resolution image and a second resolution image for the same angiographic image, the second resolution image comprising a second resolution image of each stage corresponding to the magnification of the up-sampling convolution unit of each stage, the first resolution image having a resolution lower than the second resolution image; obtaining each level of reconstructed images corresponding to each level of up-sampling convolution units in the neural network; determining a network loss value according to each level of reconstructed image corresponding to the first resolution image and each level of second resolution image corresponding to the first resolution image; and adjusting network parameters of the neural network based on the network loss value.
Optionally, the training unit is specifically configured to, when determining the network loss value according to each level of reconstructed image corresponding to the first resolution image and each level of second resolution image corresponding to the first resolution image: determining loss values of each level according to each level of output image corresponding to the first resolution image and the second resolution image corresponding to the first resolution image; determining network loss values according to the weights corresponding to the reconstructed images at all levels and the loss values at all levels respectively; the adjusting the network parameters of the neural network based on the network loss value includes: and stopping the adjustment of the network parameters of the neural network when the network loss value is smaller than a preset loss value threshold value, and/or stopping the adjustment of the network parameters of the neural network when the adjustment times exceed a preset times threshold value.
Optionally, the apparatus further includes a training set acquisition unit for inputting a first resolution image in a pre-acquired image training set into the neural network, and the processing method further includes: acquiring a second resolution image sequence, wherein the second resolution images in the sequence comprise all levels of second resolution images corresponding to the amplification factors of all levels of up-sampling convolution units; performing low resolution processing on a second resolution image corresponding to the amplification factor of the highest-level up-sampling convolution unit to obtain a corresponding first resolution image, wherein the second resolution image and the corresponding first resolution image form an angiogram image pair; the angiographic image pairs are arranged according to the order of the second-resolution image sequence to form an image training set.
Optionally, the training set obtaining unit is configured to, when performing low resolution processing on the second resolution image corresponding to the magnification of the highest level upsampling convolution unit to obtain a corresponding first resolution image, specifically: blurring the second resolution image using a gaussian kernel function; and performing downsampling processing on the blurred second resolution image by using a bicubic interpolation method to obtain a corresponding first resolution image.
In a third aspect, there is provided an electronic device comprising a memory for storing computer instructions executable on a processor for image processing based on any of the methods described above when the computer instructions are executed.
In a fourth aspect, a computer readable storage medium is provided, having stored thereon a computer program which when executed by a processor implements any of the methods described above.
The technical scheme provided by the embodiment of the specification can comprise the following beneficial effects:
in the embodiment of the disclosure, by acquiring initial characteristic information of an input image and intermediate characteristic information output by each stage of convolution circulation units, then performing up-sampling convolution operation on the initial characteristic information and combined characteristic information output by the final stage of convolution circulation units, obtaining a target reconstructed image according to an obtained up-sampling result, extracting deeper characteristics by means of residual connection and series connection of a plurality of groups of convolution circulation units, fusing shallow characteristics with deep characteristics, and improving resolution of the input image, so that details in the image are clearer and more complete.
Drawings
FIG. 1 is a flowchart of an image processing method shown in an exemplary embodiment of the present disclosure;
FIG. 2 is a schematic diagram of an image processing method shown in an exemplary embodiment of the present disclosure;
FIG. 3 is a schematic diagram of a convolution loop unit shown in an exemplary embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a method of acquiring initial feature information according to an exemplary embodiment of the present disclosure;
FIG. 5 is a schematic diagram of a method of acquiring intermediate feature information according to an exemplary embodiment of the present disclosure;
FIG. 6 is a schematic diagram of an upsampling convolution method according to an exemplary embodiment of the present disclosure;
FIG. 7 is a flow chart of an upsampling convolution method shown in an exemplary embodiment of the present disclosure;
FIG. 8 is a flowchart illustrating training a neural network according to an exemplary embodiment of the present disclosure;
FIG. 9 is a flowchart illustrating constructing a training set of images in accordance with an exemplary embodiment of the present disclosure;
fig. 10 is a schematic structural view of an image processing apparatus shown in an exemplary embodiment of the present disclosure;
fig. 11 is a schematic structural view of an apparatus shown in an exemplary embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present disclosure. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
Angiography (Digital Substraction Angiography, DSA for short) is a technique in which blood vessels are visualized with a contrast agent and X-rays, thereby generating a sequence of angiographic images; the technology is used for diagnosing and treating vascular diseases, and because the placement of fine accessories such as vascular stents, spring coils and the like is often involved in the treatment process, the details are required to be ensured to be clear and complete in the image so as to provide accurate basis for doctors to formulate treatment schemes. Obtaining an angiographic image with higher resolution requires time and focused X-ray irradiation, which is clearly detrimental to the health of the subject (i.e. patient). Therefore, it is necessary to obtain high-resolution angiographic images while avoiding the influence on the health of the subject during angiographic procedures as much as possible.
First, the present disclosure provides an image processing method that performs super-resolution processing on an angiographic image sequence, thereby improving the resolution of the angiographic image. Adverse effects on the health of the subject (i.e., patient) can be minimized during angiography, but the angiographic images obtained in this way often do not clearly show details in the images and do not meet the requirements of doctor diagnosis and treatment for clarity. The problem of low image resolution is solved by the processing method, and a specific method can use the processing method based on bilinear interpolation, bicubic interpolation and other interpolation, and the method is visual and easy to realize, but the reconstructed image is smoother, and high-frequency information such as image edges or details is easy to lose; reconstruction-based processing methods, including frequency domain methods, convex set projection methods, maximum a posteriori probability methods, etc., may also be used, which typically require the addition of complex prior knowledge as constraints to preserve more detail, but when magnification increases, performance tends to decrease and there is greater time consumption; a learning-based processing method can be used, the method generally uses machine learning to obtain a mapping relation between a low-resolution image and a corresponding high-resolution image, in the method, a mode of improving the resolution based on a single image only considers spatial correlation, a mode of improving the resolution based on an image sequence (such as video) is used for carrying out sub-pixel level motion supplementation by using the multi-frame image, a high-resolution image sequence is reconstructed on the basis of image alignment, and a super-resolution effect is not ideal.
The angiography image sequence has strong relativity between adjacent frames, and contains complex information and more fine structures, so that the super-resolution processing requirement on the low-resolution image is high, the above methods are easy to generate a phenomenon of excessive smoothness when the resolution is increased by a large multiple, and the common method for performing motion compensation on the multi-frame image not only can increase the calculation difficulty, but also can not clearly display details such as a bracket, a spring ring and the like when the resolution is increased due to insufficient information utilization between the adjacent frames.
Based on this, referring to fig. 1, the disclosure further provides an image processing method, which processes an angiographic image by using a pre-trained neural network, where the neural network includes a convolution unit, N cascaded convolution units, and an upsampling convolution unit; the method comprises the following steps 101-104.
In step 101, the input image is input to a convolution unit, and initial feature information of the input image is acquired, where the input image is one frame in an angiographic image sequence.
In the embodiments of the present disclosure, the input image is an angiographic directly obtained image, and due to its low resolution, it is not suitable for direct application in diagnosis and treatment in some cases, that is, the object of the treatment method, and the resolution thereof needs to be improved by the method, so as to meet the requirements of diagnosis and treatment.
In the embodiment of the disclosure, the processing method aims to improve the resolution of the input image, so as to obtain an image with higher resolution, so that the input image is a low-resolution image, and the output image is a super-resolution image.
Referring to the schematic diagram of the image processing method shown in fig. 2, an input image of a low resolution image is input to the convolution unit 210, and initial feature information F of the low resolution image can be obtained 0 。
In one example, the convolution element may comprise a single layer of convolution, such as a convolution of size 3 x 64.
In step 102, the initial feature information is input to a first-stage convolution circulation unit, so as to obtain intermediate feature information output by each stage convolution circulation unit.
For N cascade convolution circulation units, the characteristic information extracted by each stage of convolution circulation unit is input to the convolution circulation unit of the next stage.
As shown in fig. 2, the initial characteristic information F output by the convolution unit 210 0 Is input to the level 1 convolution unit 221, and the level 1 convolution unit 221 extracts level 1 intermediate feature information F of the input image 1 The method comprises the steps of carrying out a first treatment on the surface of the Level 1 intermediate feature information F 1 Is input to the level 2 convolution unit 222, and the level 2 convolution unit 222 extracts the level 2 of the input image Inter-feature information F 2 By the same way, each stage of convolution circulation unit outputs second characteristic information F m (m is an integer smaller than N), the nth stage convolution circulation unit outputs nth stage intermediate feature information F N 。
In one example, the structure of each convolution loop unit may be as shown in fig. 3. Each convolution circulation unit comprises a plurality of groups of convolution layers connected in a mode of combining dense connection and residual connection, the sizes of the convolution layers are 3 x 64, each layer receives the characteristics of all the previous layers as input, the characteristic multiplexing is increased, and the extracted characteristics of the last layer are added with the input pixel by pixel to obtain the output of the convolution circulation unit.
In step 103, the initial characteristic information and the intermediate characteristic information output by the nth stage convolution circulation unit are added to obtain combined characteristic information.
Initial characteristic information F obtained by characteristic extraction operation of convolution unit of input image 0 Is shallow characteristic information, and after convolution operation processing of a multi-level convolution cyclic unit, intermediate characteristic information F is obtained N Is deep characteristic information obtained by combining the initial characteristic information F 0 And intermediate characteristic information F N Adding to obtain combined characteristic information F C And fusing the shallow characteristic information and the deep characteristic information.
In step 104, the combined characteristic information is input to an upsampling convolution unit, upsampling convolution operation is performed on the combined characteristic information, and a target reconstructed image is obtained according to the obtained upsampling result.
By upsampling convolution operation on the combined feature information, higher resolution feature information is obtained, and by upsampling results, a target reconstructed image with higher resolution than the image input image is obtained. For the up-sampling result, the target reconstructed image may be obtained, for example, by a convolution kernel of 1*1.
In the embodiment of the disclosure, by acquiring initial characteristic information of an input image and intermediate characteristic information output by each stage of convolution circulation units, then performing up-sampling convolution operation on the initial characteristic information and combined characteristic information output by the final stage of convolution circulation units, obtaining a target reconstructed image according to an obtained up-sampling result, extracting deeper characteristics by means of residual connection and series connection of a plurality of groups of convolution circulation units, fusing shallow characteristics with deep characteristics, and improving resolution of the input image, so that details in the image are clearer and more complete.
In the neural network structure shown in fig. 2, when the feature information input by each convolution loop unit is the feature information for the input image, that is, the current time image, such a neural network may be referred to as a single-frame neural network. In the case where the features input by the respective convolution loop units are related not only to the feature information of the current time image but also to the feature information of the previous time image, such a neural network may be referred to as a fusion neural network. The fused neural network is described in detail below in conjunction with fig. 4 and 5.
Fig. 4 shows a schematic diagram of a method for acquiring initial characteristic information by using a fused neural network. As shown in fig. 4, the convolution unit 400 includes a first convolution subunit 410 and a second convolution subunit 420, where each of the first convolution subunit 410 and the second convolution subunit 420 may include one or more convolution layers.
The process of acquiring initial feature information from an input image is as follows: first, an input image is input to the first convolution subunit 410 to obtain first initial feature information F 01 The method comprises the steps of carrying out a first treatment on the surface of the Next, inputting the concatenated multiple feature information into the second convolution subunit 420 to obtain second initial feature information F 02 The plurality of feature information of the cascade includes first initial feature information F 01 And a second convolution subunit 420 outputs second initial feature information F for a previous frame of the input image 02 . That is, the first initial characteristic information F output from the first convolution subunit 410 01 And the second initial characteristic information F of the previous frame 02 After concatenation, input to a second convolution subunit 420, the second convolution subunit 420 outputs second initial feature information F for the current frame 02 . First initial characteristic information F 01 And a second initiationFeature information F 02 Initial characteristic information F forming convolution unit 400 0 。
For each stage of convolution circulation unit, the first intermediate characteristic information output by the first convolution circulation subunit and the second intermediate characteristic information output by the second convolution circulation subunit form intermediate characteristic information output by the stage of convolution circulation unit.
In the embodiment of the disclosure, the initial characteristic information output by the convolution unit is fused with the characteristic information of the previous frame image, and the time correlation of the angiography image sequence is utilized, so that the resolution of the target reconstructed image is improved.
Fig. 5 shows a schematic diagram of a method for acquiring intermediate feature information by using a fused neural network. Each convolution cyclic unit includes a first convolution cyclic subunit and a second convolution cyclic subunit, and each stage of the first convolution cyclic subunit is input to the next stage of the convolution cyclic subunit.
The process of obtaining intermediate feature information from the initial feature information is as follows:
the feature information is input to the first stage convolution sub-unit 511, resulting in first intermediate feature information output by each stage convolution sub-unit. In one example, the initial characteristic information F 0 First initial characteristic information F of (a) 01 Is input to the first-stage convolution subunit 511, and the first-stage convolution subunit 511 outputs first intermediate feature information F 11 The first stage convolution sub-unit 512 outputs first intermediate feature information F 12 And so on, the first-stage convolution subunit 51N outputs first intermediate feature information F 1N 。
Inputting the cascaded multiple pieces of characteristic information into a second convolution circulation subunit in each level of convolution circulation units to obtain second intermediate characteristic information, wherein for an ith level of second convolution circulation subunit, the cascaded multiple pieces of characteristic information comprise first intermediate characteristic information output by an ith level of first convolution circulation subunit for the input image, second intermediate characteristic information output by an ith-1 level of second convolution circulation subunit for the input image and second intermediate characteristic information output by the ith level of second convolution circulation subunit for a previous frame of the input image, wherein i is a positive integer less than or equal to N.
Taking the 2 nd level second convolution subunit as an example, the plurality of feature information input in cascade includes first intermediate feature information F output by the 2 nd level first convolution subunit for the input image, i.e. the current frame 11 Second intermediate feature information F output by the 1 st level second convolution sub-unit for the current frame 21 And the second intermediate feature information F output by the 2 nd-stage second convolution sub-unit for the previous frame 21 。
In the embodiment of the disclosure, the intermediate feature information output by each stage of convolution circulation unit is fused with the feature information of the previous frame image and the initial feature information of the corresponding stage, so that the time correlation and the space correlation of the angiography image sequence are utilized, the resolution of the target reconstructed image is improved, and details in the target reconstructed image are clearer and more complete.
In some embodiments of the present disclosure, the upsampling convolution unit may be constructed using a sub-pixel convolution layer, for example, an M-level upsampling convolution subunit may be used to construct the upsampling convolution unit, where for scenes with greater magnification, such as 4 times, 8 times, etc., a final super resolution image may be obtained in a step-wise manner.
Fig. 6 shows a schematic diagram of a method of upsampling convolution. As shown in fig. 6, the up-sampling convolution unit includes M up- sampling convolution subunits 611, 612, … M. Fig. 7 shows a flow chart of an upsampling convolution method.
In step 701, the combined characteristic information is input to a j-th level up-sampling convolution subunit, and a j-th level up-sampling result is output;
in step 702, a j-th reconstructed image is obtained according to the j-th upsampling result, and the j-th reconstructed image is used as an input image to obtain the combined characteristic information;
In step 703, adding the j-th up-sampling convolution unit and the combination feature information, and inputting the added combination to the j+1-th up-sampling convolution unit;
in step 704, a target reconstructed image is obtained according to the M-th up-sampling result output by the M-th up-sampling convolution unit, where M is a positive integer and j is a positive integer less than or equal to M-1.
That is, for the level 1 upsampling subunit 611, the combined feature information F obtained for the input image is input c0 Performing image reconstruction according to the output upsampling result to obtain a reconstructed image P1, and inputting the reconstructed image P1 as an input image into a neural network to obtain combined characteristic information F for the reconstructed image P1 c1 . Next, the up-sampling result output by the level 1 up-sampling subunit 611, and the combined feature information F for the reconstructed image P1 c1 The input-added combination is input to the level 2 upsampling subunit 612 and so on until the level M upsampling convolution subunit outputs a level M upsampling result, and a target reconstructed image is obtained from the upsampling result.
In the disclosed embodiment, the upsampling process of each stage of the upsampling convolution subunit is a process that increases the resolution of the input image, and each stage of the upsampling convolution subunit is a process that multiplies the resolution of the input image by one. The reconstructed image of the last stage (i.e., the M-th stage) has the highest resolution and the corresponding magnification is the largest, and is thus determined as the target reconstructed image. The 4 times magnification is taken as an example, the 1 st level up-sampling convolution subunit obtains a 2 times amplified super-division image, then the super-division image is used as the input of the 2 nd level up-sampling convolution subunit, the residual error network structure before lamination and the output of the up-sampling convolution subunit are used for obtaining the 4 times amplified super-division image, and the defect that the super-division image is too smooth due to direct 4 times amplification is overcome.
According to the method and the device, the residual network structure and the structures of the plurality of up-sampling convolution subunits are combined, the purpose of amplifying the resolution of the input image step by step can be achieved, the network performance of the neural network is effectively improved, and the super-resolution image which fully utilizes space-time information and keeps details can be reconstructed.
Referring to fig. 8, the disclosure further provides a method for pre-training the neural network, which specifically includes steps 801 to 804.
In step 801, one first resolution image of a pre-obtained image training set is input to the neural network, wherein the image training set comprises a sequence of angiographic image pairs, each comprising a first resolution image and a second resolution image for the same vessel, the second resolution image comprising second resolution images of stages corresponding to the magnification of up-sampling convolution units of stages, the resolution of the first resolution image being lower than the resolution of the second resolution image.
The first resolution image and the second resolution image may be referred to as a low resolution image and the second resolution image may be referred to as a high resolution image (real image) because of the size relationship of the resolutions. The neural network is trained such that, upon input of the first resolution image to the neural network, an image having a resolution approaching that of the second resolution image can be output.
In step 802, reconstructed images of each stage corresponding to each stage up-sampling convolution unit in the neural network are obtained.
As described above, in the image processing process, the up-sampling convolution units at each stage can output each stage of reconstructed image, and in the training process, the neural network performs the same input-output operation, so that each stage of reconstructed image corresponding to the first resolution image can also be output.
In step 803, a network loss value is determined from each level of reconstructed image corresponding to the first resolution image and the second resolution image corresponding to the first resolution image.
In this step, the resolution of each stage of reconstructed image is gradually increased by a multiple, so the resolution is gradually increased, and the resolution is more and more close to the resolution of the second resolution image, so the network loss value can be determined by the resolution difference between each stage of reconstructed image and each stage of second resolution image.
In one example, first, corresponding loss values of each level are determined according to each level of output image corresponding to the first resolution image and the second resolution image corresponding to the first resolution image; and respectively determining network loss values according to the weights corresponding to the reconstructed images of all levels and the loss values of all levels.
In the disclosed embodiment, the loss function employed is as shown in equation (1):
L total ＝λ 1 L 1 +λ 2 L 2 +…λ M L M (1)
wherein L is 1 、L 2 、…、L M Lambda is the loss of the convolution unit of each level of previous sample 1 、λ 2 、…、λ M Each stage loses the corresponding weight, which can be specifically set according to the required resolution of the target reconstructed image.
And calculating the loss value of each level of reconstructed image and the corresponding second resolution image, and supervising the reconstruction result of each up-sampling convolution unit, so that the error between each level of resolution image and each level of second resolution image is reduced.
In step 804, network parameters of the neural network are adjusted based on the network loss value.
In this step, the deviation of the network parameters of the neural network can be fed back through the network loss value, and the difference between the reconstructed images at each level and the second resolution images at each level is gradually reduced by gradually minimizing the network loss value through the adjustment of the network parameters.
In one example, the adjustment of the network parameters of the neural network is stopped when the network loss value is less than a preset loss value threshold, and/or the adjustment of the network parameters of the neural network is stopped when the number of adjustments exceeds a preset number of times threshold. The learning rate and the iteration number are set at the beginning of training, and the weights and the deviations of each layer and each level are initialized randomly, so that the accuracy of the loss function reaches the preset requirement when the network loss value is smaller than the preset loss value threshold, and the maximum iteration number is reached when the adjustment number exceeds the preset number threshold, the training can be ended under both conditions, and the trained model is saved.
Compared with the traditional method, the training method can directly learn the mapping relation between the low-resolution image and the high-resolution image, is an end-to-end learning method, uses the neural network which is trained to perform image processing, and can obtain better superdivision effect than the traditional method.
Referring to fig. 9, the disclosure further provides a method for constructing an image training set, which specifically includes steps 901 to 903.
In step 901, a sequence of second resolution images is acquired, the second resolution images in the sequence comprising respective stages of second resolution images corresponding to the magnification of respective stages of up-sampling convolution units.
For example, for a two-stage up-sampling convolution unit, the first stage up-sampling convolution unit corresponds to a magnification of 2 and the second stage up-sampling convolution unit corresponds to a magnification of 4, then the second resolution images in the sequence comprise two stages of second resolution images corresponding to 2 times magnification and 4 times magnification, respectively.
In this step, the second resolution image sequence may be obtained by collecting a clinically high resolution angiographic image sequence, which is taken by an angiographic apparatus under small-focus X-rays and long-term irradiation conditions, and which is required to contain at least one of a spring coil, a stent, a fine blood vessel, etc.
In step 902, a second resolution image corresponding to the magnification of the highest level up-sampling convolution unit is subjected to low resolution processing, so as to obtain a corresponding first resolution image, where the second resolution image and the corresponding first resolution image form an angiogram image pair.
For the above example, one angiographic image pair is formed including a second-resolution image corresponding to 4-fold magnification, a second-resolution image corresponding to 2-fold magnification, and a corresponding first-resolution image obtained by subjecting the second-resolution image corresponding to 4-fold magnification to low-resolution processing.
In this step, the second resolution image is processed with low resolution, and the obtained first resolution image should be as close to the low resolution image actually photographed clinically as possible, so that the image performance close to the training process can be obtained during prediction. The first resolution image obtained by the low resolution processing of the second resolution image corresponds to the first resolution image, and the contents contained in the first resolution image and the second resolution image are identical, and the difference is only that the resolutions are different, so that the first resolution image is taken as an input image, if a target reconstruction image output by the neural network is close to the second resolution image, the processing capacity of the neural network is high, the processing result is reliable, if the resolution difference distance from the second resolution image is large, the processing capacity of the neural network is poor, the processing result is unreliable, and the network parameters of the neural network need to be adjusted.
In one example, the low resolution processing of the first resolution image is performed as follows: first, performing low resolution processing on a second resolution image corresponding to an amplification factor of a highest-level up-sampling convolution unit using a gaussian kernel function; and then performing downsampling processing on the second resolution image by using a bicubic interpolation method to obtain a corresponding first resolution image.
In step 903, the angiographic image pairs are arranged according to the order of the second-resolution image sequence, forming an image training set.
Because each first resolution image forms a corresponding angiographic image pair, the original first resolution image sequence corresponds to form an angiographic image pair sequence, and the angiographic image pair sequence is used for training the neural network, namely, the first resolution image in each angiographic image pair is used as an input image, and the corresponding second resolution image is used as a target image, so that the angiographic image pair sequence is an image training set. Part or all of the angiogram image pairs are selected for training in the neural network training process.
In a second aspect, as shown in fig. 10, there is provided an image processing apparatus for processing an angiographic image using a pre-trained neural network, the neural network including a convolution unit, N cascaded convolution units, and an upsampling convolution unit; the device comprises: the first module is used for acquiring initial characteristic information of an input image, and the input image is one frame in an angiography image sequence; the intermediate feature acquisition module is used for inputting the initial feature information into the first-stage convolution circulation unit to obtain intermediate feature information output by each stage convolution circulation unit; the third module is used for adding the initial characteristic information and the intermediate characteristic information output by the nth-level convolution circulation unit to obtain combined characteristic information; and the reconstruction module is used for carrying out up-sampling convolution operation on the combined characteristic information and obtaining a target reconstruction image according to the obtained up-sampling result, wherein N is a positive integer.
In some embodiments, the convolution unit includes a first convolution subunit and a second convolution subunit; the initial feature acquisition unit is specifically configured to: inputting the input image to the first convolution subunit to obtain first initial characteristic information; and inputting the cascaded multiple pieces of characteristic information into the second convolution subunit to obtain second initial characteristic information, wherein the cascaded multiple pieces of characteristic information comprise the first initial characteristic information and the second initial characteristic information output by the second convolution subunit for the previous frame of the input image.
In some embodiments, the convolution loop unit comprises a first convolution loop subunit and a second convolution loop subunit, wherein each level of the first convolution loop subunit is input to the next level of the convolution loop subunit; the intermediate feature acquisition unit is specifically configured to: inputting the initial characteristic information into a first-stage convolution circulation subunit to obtain first intermediate characteristic information output by each stage convolution circulation subunit; inputting the cascaded multiple pieces of characteristic information into a second convolution circulation subunit in each level of convolution circulation units to obtain second intermediate characteristic information, wherein for an ith level of second convolution circulation subunit, the cascaded multiple pieces of characteristic information comprise first intermediate characteristic information output by an ith level of first convolution circulation subunit for the input image, second intermediate characteristic information output by an ith-1 level of second convolution circulation subunit for the input image and second intermediate characteristic information output by the ith level of second convolution circulation subunit for a previous frame of the input image, wherein i is a positive integer less than or equal to N.
In some embodiments, the convolution unit includes a first convolution subunit and a second convolution subunit prior to acquiring initial feature information of the input image; the inputting the input image to a convolution unit, obtaining initial characteristic information of the input image, including: inputting the input image to the first convolution subunit to obtain first initial characteristic information; and inputting the cascaded multiple pieces of characteristic information into the second convolution subunit to obtain second initial characteristic information, wherein the cascaded multiple pieces of characteristic information comprise the first initial characteristic information and the second initial characteristic information output by the second convolution subunit for the previous frame of the input image.
In some embodiments, the neural network further comprises an M-level upsampling convolution subunit; the reconstruction unit is specifically configured to: inputting the combined characteristic information to a j-th level up-sampling convolution subunit, and outputting a j-th level up-sampling result; obtaining a j-th level reconstruction image according to a j-th level up-sampling result, and obtaining the combined characteristic information by taking the j-th level reconstruction image as an input image; adding the j-th level up-sampling result and the combination characteristic information, and inputting the added result to a j+1-th level up-sampling convolution subunit; and obtaining a target reconstruction image according to an M-th up-sampling result output by the M-th up-sampling convolution subunit, wherein M is a positive integer, and j is a positive integer smaller than or equal to M-1.
In some embodiments, the apparatus further comprises a training unit for: inputting a first resolution image of a pre-obtained image training set to the neural network, wherein the image training set comprises a sequence of angiographic image pairs, each comprising a first resolution image and a second resolution image for the same angiographic image, the second resolution image comprising a second resolution image of each stage corresponding to the magnification of the up-sampling convolution unit of each stage, the first resolution image having a resolution lower than the second resolution image; obtaining each level of reconstructed images corresponding to each level of up-sampling convolution units in the neural network; determining a network loss value according to each level of reconstructed image corresponding to the first resolution image and each level of second resolution image corresponding to the first resolution image; and adjusting network parameters of the neural network based on the network loss value.
In some embodiments, the training unit is configured to, when configured to determine the network loss value from each level of reconstructed image corresponding to the first resolution image and each level of second resolution image corresponding to the first resolution image, specifically: determining loss values of each level according to each level of output image corresponding to the first resolution image and the second resolution image corresponding to the first resolution image; determining network loss values according to the weights corresponding to the reconstructed images at all levels and the loss values at all levels respectively; the adjusting the network parameters of the neural network based on the network loss value includes: and stopping the adjustment of the network parameters of the neural network when the network loss value is smaller than a preset loss value threshold value, and/or stopping the adjustment of the network parameters of the neural network when the adjustment times exceed a preset times threshold value.
In some embodiments, the apparatus further comprises a training set acquisition unit for inputting a first resolution image in a pre-acquired image training set into the neural network, and the processing method further comprises: acquiring a second resolution image sequence, wherein the second resolution images in the sequence comprise all levels of second resolution images corresponding to the amplification factors of all levels of up-sampling convolution units; performing low resolution processing on a second resolution image corresponding to the amplification factor of the highest-level up-sampling convolution unit to obtain a corresponding first resolution image, wherein the second resolution image and the corresponding first resolution image form an angiogram image pair; the angiographic image pairs are arranged according to the order of the second-resolution image sequence to form an image training set.
In some embodiments, the training set obtaining unit is specifically configured to, when configured to perform low resolution processing on the second resolution image corresponding to the magnification of the highest level upsampling convolution unit: blurring the second resolution image using a gaussian kernel function; and performing downsampling processing on the blurred second resolution image by using a bicubic interpolation method to obtain a corresponding first resolution image.
In a third aspect, as shown in fig. 11, there is provided an apparatus comprising a memory for storing computer instructions executable on a processor for image processing based on any of the above methods when the computer instructions are executed.
In a fourth aspect, a computer readable storage medium is provided, having stored thereon a computer program which when executed by a processor implements any of the methods described above.
Embodiments of the subject matter and the functional operations described in this specification can be implemented in: digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and structural equivalents thereof, or a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions may be encoded on a manually-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode and transmit information to suitable receiver apparatus for execution by data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
Computers suitable for executing computer programs include, for example, general purpose and/or special purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read only memory and/or a random access memory. The essential elements of a computer include a central processing unit for carrying out or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks, etc. However, a computer does not have to have such a device. Furthermore, the computer may be embedded in another device, such as a mobile phone, a Personal Digital Assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device such as a Universal Serial Bus (USB) flash drive, to name a few.
Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disk or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features of specific embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. On the other hand, the various features described in the individual embodiments may also be implemented separately in the various embodiments or in any suitable subcombination. Furthermore, although features may be acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, although operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Furthermore, the processes depicted in the accompanying drawings are not necessarily required to be in the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.
The foregoing description of the preferred embodiments of the present disclosure is not intended to limit the disclosure, but rather to cover all modifications, equivalents, improvements and alternatives falling within the spirit and principles of the present disclosure.

Claims (11)

1. An image processing method is characterized in that an angiographic image is processed by utilizing a pre-trained neural network, wherein the neural network comprises a convolution unit, N cascade convolution circulation units and an up-sampling convolution unit; the method comprises the following steps:
inputting an input image into a convolution unit, and acquiring initial characteristic information of the input image, wherein the input image is one frame in an angiography image sequence;
inputting the initial characteristic information into a first-stage convolution circulation unit to obtain intermediate characteristic information output by each stage convolution circulation unit;
adding the initial characteristic information and the intermediate characteristic information output by the nth-stage convolution circulation unit to obtain combined characteristic information;
and inputting the combined characteristic information into an up-sampling convolution unit, performing up-sampling convolution operation on the combined characteristic information, and obtaining a target reconstructed image according to the obtained up-sampling result, wherein N is a positive integer.
2. The method of claim 1, wherein the convolution unit comprises a first convolution subunit and a second convolution subunit;
the inputting the input image to a convolution unit, obtaining initial characteristic information of the input image, including:
Inputting the input image to the first convolution subunit to obtain first initial characteristic information;
and inputting the cascaded multiple pieces of characteristic information into the second convolution subunit to obtain second initial characteristic information, wherein the cascaded multiple pieces of characteristic information comprise the first initial characteristic information and the second initial characteristic information output by the second convolution subunit for the previous frame of the input image.
3. The method of claim 1, wherein the convolution cyclic unit comprises a first convolution cyclic subunit and a second convolution cyclic subunit, wherein each stage of the first convolution cyclic subunit is input to a next stage of the convolution cyclic subunit;
the step of inputting the initial characteristic information into a first-stage convolution circulation unit to obtain intermediate characteristic information output by each stage convolution circulation unit, comprises the following steps:
inputting the initial characteristic information into a first-stage convolution circulation subunit to obtain first intermediate characteristic information output by each stage convolution circulation subunit;
inputting the cascaded multiple pieces of characteristic information into a second convolution circulation subunit in each level of convolution circulation units to obtain second intermediate characteristic information, wherein for an ith level of second convolution circulation subunit, the cascaded multiple pieces of characteristic information comprise first intermediate characteristic information output by an ith level of first convolution circulation subunit for the input image, second intermediate characteristic information output by an ith-1 level of second convolution circulation subunit for the input image and second intermediate characteristic information output by the ith level of second convolution circulation subunit for a previous frame of the input image, wherein i is a positive integer less than or equal to N.
4. A method according to any one of claims 1 to 3, wherein the neural network further comprises an M-level up-sampling convolution subunit; the step of inputting the combined characteristic information into an up-sampling convolution unit, performing up-sampling convolution operation on the combined characteristic information, and obtaining a target reconstructed image according to the obtained up-sampling result comprises the following steps:
inputting the combined characteristic information to a j-th level up-sampling convolution subunit, and outputting a j-th level up-sampling result;
obtaining a j-th level reconstruction image according to a j-th level up-sampling result, and obtaining the combined characteristic information by taking the j-th level reconstruction image as an input image;
adding the j-th level up-sampling result and the combination characteristic information, and inputting the added result to a j+1-th level up-sampling convolution subunit;
and obtaining a target reconstruction image according to an M-th up-sampling result output by the M-th up-sampling convolution subunit, wherein M is a positive integer, and j is a positive integer smaller than or equal to M-1.
5. The method of claim 4, wherein the processing method further comprises:
before acquiring initial feature information of an input image, inputting a first resolution image in a pre-acquired image training set to the neural network, wherein the image training set comprises a sequence of angiographic image pairs, each angiographic image pair comprises a first resolution image and a second resolution image for the same angiographic image, the second resolution image comprises each stage of second resolution image corresponding to the magnification of each up-sampling convolution unit, and the resolution of the first resolution image is lower than that of the second resolution image;
Obtaining each level of reconstructed images corresponding to each level of up-sampling convolution units in the neural network;
determining a network loss value according to each level of reconstructed image corresponding to the first resolution image and each level of second resolution image corresponding to the first resolution image;
and adjusting network parameters of the neural network based on the network loss value.
6. The method of claim 5, wherein determining the network loss value from each level of reconstructed image corresponding to the first resolution image and each level of second resolution image corresponding to the first resolution image comprises:
determining loss values of each level according to each level of output image corresponding to the first resolution image and the second resolution image corresponding to the first resolution image;
determining network loss values according to the weights corresponding to the reconstructed images at all levels and the loss values at all levels respectively;
the adjusting the network parameters of the neural network based on the network loss value includes:
and stopping the adjustment of the network parameters of the neural network when the network loss value is smaller than a preset loss value threshold value, and/or stopping the adjustment of the network parameters of the neural network when the adjustment times exceed a preset times threshold value.
7. The method of claim 5 or 6, wherein a first resolution image in a training set of pre-acquired images is input to the neural network, the processing method further comprising:
acquiring a second resolution image sequence, wherein the second resolution images in the sequence comprise all levels of second resolution images corresponding to the amplification factors of all levels of up-sampling convolution units;
performing low resolution processing on a second resolution image corresponding to the amplification factor of the highest-level up-sampling convolution unit to obtain a corresponding first resolution image, wherein the second resolution image and the corresponding first resolution image form an angiogram image pair;
the angiographic image pairs are arranged according to the order of the second-resolution image sequence to form an image training set.
8. The method of claim 7, wherein low resolution processing the second resolution image corresponding to the magnification of the highest level up-sampling convolution unit to obtain a corresponding first resolution image, comprises:
blurring the second resolution image using a gaussian kernel function;
and performing downsampling processing on the blurred second resolution image by using a bicubic interpolation method to obtain a corresponding first resolution image.
9. An image processing device is characterized in that angiographic images are processed by a pre-trained neural network, and the neural network comprises a convolution unit, N cascade convolution circulation units and an up-sampling convolution unit; the device comprises:
the initial characteristic acquisition module is used for acquiring initial characteristic information of an input image, wherein the input image is one frame in an angiography image sequence;
the intermediate feature acquisition module is used for inputting the initial feature information into the first-stage convolution circulation unit to obtain intermediate feature information output by each stage convolution circulation unit;
the combined characteristic acquisition module is used for adding the initial characteristic information and the intermediate characteristic information output by the nth-level convolution circulation unit to obtain combined characteristic information;
and the reconstruction module is used for carrying out up-sampling convolution operation on the combined characteristic information and obtaining a target reconstruction image according to the obtained up-sampling result, wherein N is a positive integer.
10. An electronic device comprising a memory, a processor for storing computer instructions executable on the processor for performing image processing based on the method of any one of claims 1 to 8 when the computer instructions are executed.
11. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method of any one of claims 1 to 8.
CN202010137174.8A 2020-03-02 2020-03-02 Image processing method, device, equipment and storage medium Active CN111462041B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010137174.8A CN111462041B (en) 2020-03-02 2020-03-02 Image processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010137174.8A CN111462041B (en) 2020-03-02 2020-03-02 Image processing method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN111462041A CN111462041A (en) 2020-07-28
CN111462041B true CN111462041B (en) 2023-05-05

Family

ID=71684156

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010137174.8A Active CN111462041B (en) 2020-03-02 2020-03-02 Image processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111462041B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116525121B (en) * 2023-07-05 2023-09-26 昆明同心医联科技有限公司 Method for establishing primary spring coil recommendation model of embolic aneurysm and application of primary spring coil recommendation model

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017034682A (en) * 2016-09-08 2017-02-09 株式会社ニコン Image generation apparatus, image processing apparatus, image generation method, and image processing program
CN108376386A (en) * 2018-03-23 2018-08-07 深圳天琴医疗科技有限公司 A kind of construction method and device of the super-resolution model of image
CN109272452A (en) * 2018-08-30 2019-01-25 北京大学 Learn the method for super-resolution network in wavelet field jointly based on bloc framework subband
CN109345449A (en) * 2018-07-17 2019-02-15 西安交通大学 A kind of image super-resolution based on converged network and remove non-homogeneous blur method
CN109741260A (en) * 2018-12-29 2019-05-10 天津大学 A kind of efficient super-resolution method based on depth back projection network
CN109886871A (en) * 2019-01-07 2019-06-14 国家新闻出版广电总局广播科学研究院 The image super-resolution method merged based on channel attention mechanism and multilayer feature
CN110097611A (en) * 2019-04-28 2019-08-06 上海联影智能医疗科技有限公司 Image rebuilding method, device, equipment and storage medium
CN110717856A (en) * 2019-09-03 2020-01-21 天津大学 Super-resolution reconstruction algorithm for medical imaging

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7639289B2 (en) * 2006-05-08 2009-12-29 Mitsubishi Electric Research Laboratories, Inc. Increasing object resolutions from a motion-blurred image

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017034682A (en) * 2016-09-08 2017-02-09 株式会社ニコン Image generation apparatus, image processing apparatus, image generation method, and image processing program
CN108376386A (en) * 2018-03-23 2018-08-07 深圳天琴医疗科技有限公司 A kind of construction method and device of the super-resolution model of image
CN109345449A (en) * 2018-07-17 2019-02-15 西安交通大学 A kind of image super-resolution based on converged network and remove non-homogeneous blur method
CN109272452A (en) * 2018-08-30 2019-01-25 北京大学 Learn the method for super-resolution network in wavelet field jointly based on bloc framework subband
CN109741260A (en) * 2018-12-29 2019-05-10 天津大学 A kind of efficient super-resolution method based on depth back projection network
CN109886871A (en) * 2019-01-07 2019-06-14 国家新闻出版广电总局广播科学研究院 The image super-resolution method merged based on channel attention mechanism and multilayer feature
CN110097611A (en) * 2019-04-28 2019-08-06 上海联影智能医疗科技有限公司 Image rebuilding method, device, equipment and storage medium
CN110717856A (en) * 2019-09-03 2020-01-21 天津大学 Super-resolution reconstruction algorithm for medical imaging

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Bayesian resolution enhancement of compressed video;C.A. Segall 等;《IEEE Transactions on Image Processing》;20040607;第13卷(第7期);898-911 *
Deeply-Recursive Convolutional Network for Image Super-Resolution;Jiwon Kim 等;《2016 IEEE Conference on Computer Vision and Pattern Recognition》;20161212;1637-1645 *
功能型复合深度网络的图像超分辨率重建;唐家军 等;《计算机科学与探索》;20191125;1368-1379 *
图像超分辨率技术的回顾与展望;刘颖 等;《计算机科学与探索》;20191218;181-199 *

Also Published As

Publication number Publication date
CN111462041A (en) 2020-07-28

Similar Documents

Publication Publication Date Title
CN110599421B (en) Model training method, video fuzzy frame conversion method, device and storage medium
US20200349712A1 (en) Method and system for machine learning based segmentation of contrast filled coronary artery vessels on medical images
CN112489154A (en) MRI motion artifact correction method for generating countermeasure network based on local optimization
US11360180B2 (en) Methods, systems, and computer readable media for using a trained adversarial network for performing retrospective magnetic resonance imaging (MRI) artifact correction
Upadhyay et al. A mixed-supervision multilevel gan framework for image quality enhancement
CN111462041B (en) Image processing method, device, equipment and storage medium
CN116563916A (en) Attention fusion-based cyclic face super-resolution method and system
CN106981046B (en) Single image super resolution ratio reconstruction method based on multi-gradient constrained regression
Xin et al. Video face super-resolution with motion-adaptive feedback cell
Wang et al. GREnet: gradually recurrent network with curriculum learning for 2-D medical image segmentation
Lu et al. Two-stage self-supervised cycle-consistency transformer network for reducing slice gap in MR images
Liu et al. Ultrasound super resolution using vision transformer with convolution projection operation
Sander et al. Autoencoding low-resolution MRI for semantically smooth interpolation of anisotropic MRI
Feng et al. Deep multi-modal aggregation network for MR image reconstruction with auxiliary modality
Ni et al. SSCA‐Net: Simultaneous Self‐and Channel‐Attention Neural Network for Multiscale Structure‐Preserving Vessel Segmentation
CN112991220A (en) Method for correcting image artifacts by convolutional neural network based on multiple constraints
CN111223046B (en) Image super-resolution reconstruction method and device
Tran et al. Deep learning-based inpainting for chest X-ray image
EP4070236A1 (en) Medical image synthesis for motion correction using generative adversarial networks
Park et al. Blended-transfer learning for compressed-sensing cardiac CINE MRI
CN112419175A (en) Weight-sharing dual-region generation countermeasure network and image generation method thereof
Cui et al. RestoreDet: Degradation equivariant representation for object detection in low resolution images
Kuang et al. Bea-net: body and edge aware network with multi-scale short-term concatenation for medical image segmentation
CN110570355B (en) Multi-scale automatic focusing super-resolution processing system and method
Su et al. AngioMoCo: Learning-Based Motion Correction in Cerebral Digital Subtraction Angiography

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant