CN116823611A - Multi-focus image-based referenced super-resolution method - Google Patents

Multi-focus image-based referenced super-resolution method Download PDF

Info

Publication number
CN116823611A
CN116823611A CN202310782600.7A CN202310782600A CN116823611A CN 116823611 A CN116823611 A CN 116823611A CN 202310782600 A CN202310782600 A CN 202310782600A CN 116823611 A CN116823611 A CN 116823611A
Authority
CN
China
Prior art keywords
focus
resolution
image
super
focus image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310782600.7A
Other languages
Chinese (zh)
Inventor
季佳宇
贾爽
陈泽宇
沈全成
郇钲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Spaceflight Electronic and Communication Equipment Research Institute
Original Assignee
Shanghai Spaceflight Electronic and Communication Equipment Research Institute
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Spaceflight Electronic and Communication Equipment Research Institute filed Critical Shanghai Spaceflight Electronic and Communication Equipment Research Institute
Priority to CN202310782600.7A priority Critical patent/CN116823611A/en
Publication of CN116823611A publication Critical patent/CN116823611A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)

Abstract

The application provides a multi-focus image-based referenced super-resolution method, which solves the problem of the referenced super-resolution image quality improvement method of the image after multi-focus image fusion. In the application, a plurality of multi-focus images are fused into a full-focus image. And then taking the original multi-focus image as a reference, and performing reference image super-division on the full-focus image. And finally, combining the multi-focus fusion method with the referenced super-resolution method, training to obtain a final algorithm structure, and realizing the super-resolution of the multi-focus fusion image. The application uses the acquired original multi-focus image as a reference image, and provides a solution for further improving the resolution of the fused image. The application can be applied to super-resolution of multi-focus fusion images.

Description

Multi-focus image-based referenced super-resolution method
Technical Field
The application belongs to the field of quality improvement of acquired digital images, and particularly relates to a multi-focus image-based referenced super-resolution method.
Background
In a large number of electronic image applications, it is often desirable to obtain high resolution images. High resolution means that the pixel density in the image is high, which can provide more details that are essential in many practical applications. For example, high resolution medical images are very helpful for doctors to make a correct diagnosis; similar objects are easily distinguished from the similarity using high resolution satellite images; the performance of pattern recognition in computer vision is greatly improved if high resolution images can be provided. Digital image sensors have been widely used to capture digital images since the seventies of the last century. Although the performance of these sensors has been greatly improved over many years, consumer demand for higher resolution and clarity has not been met. Because of the limitation of manufacturing cost and device size, the depth of field and resolution of the image acquired by the sensor are smaller, the full-focus image of the scene with large depth of field cannot be directly acquired, and the quality of the acquired image is more influenced by the smaller resolution. Therefore, there is an urgent need to propose a super-resolution method based on multi-focus images to solve the above problems.
Disclosure of Invention
The application aims to solve the problems that a shot picture cannot be fully focused when a scene with a large depth of field is shot and the quality of an acquired image is poor due to insufficient resolution of a camera, and provides a reference super-resolution method based on a degree-focused image, which aims to enlarge the depth of field of the acquired image and improve the resolution of the image.
The technical scheme adopted for solving the technical problems is as follows: a referenced super-resolution method based on multi-focus images, the method comprising the steps of:
firstly, shooting a plurality of multi-focus images of a scene with a large depth of field by a camera, and fusing the multi-focus images into a full-focus clear image by using a deep learning algorithm;
step two, for the fused full-focus image, in order to improve the resolution, using a plurality of multi-focus images obtained by shooting as references, extracting relevant features in the full-focus image, fusing the features into corresponding positions, and obtaining a reference super-resolution image by using a deep learning method;
and thirdly, jointly training the multi-focus fusion method and the referenced super-resolution method to obtain a final algorithm structure, and further improving the super-resolution effect.
Further, the first step includes the following sub-steps:
(1.1) carrying out Gaussian blur and homography conversion on a saliency target detection data set DUTS to manufacture a data set for training a multi-focus image fusion algorithm;
(1.2) training a multi-focus fusion deep learning network algorithm by using the manufactured data set;
and (1.3) shooting a plurality of multi-focus images of a scene with a large depth of field by using a handheld camera, and fusing the multi-focus images into a full-focus clear image by using a trained network.
Further, in the step (1.2), the multi-focus fusion algorithm is selected from: UFA and IFCNN.
Further, the second step includes the following sub-steps:
(2.1) downsampling the data set generated in step (1.1) as a data set trained with a reference super-resolution algorithm;
(2.2) respectively extracting the features of the reference image and the full-focus image by using a VGG network, calculating the correlation between the features, and finding out the position and the feature value of the most similar feature in the reference image and the feature in each position in the full-focus image;
(2.3) training a non-reference super-resolution algorithm network using the data set generated in step (2.1);
and (2.4) merging the reference image features extracted in the step (2.2) into the non-reference super-resolution network in the step (2.3) to obtain a final reference super-resolution network.
Further, in the step (2.3), the non-reference super-resolution algorithm is selected from: EDSR, SRCNN, ESPCN.
Further, the third step includes the following sub-steps:
(3.1) taking the output of the multi-focus fusion network as the input with reference super resolution, connecting two networks, and constructing an end-to-end network structure;
and (3.2) performing fine tuning training on the unified network structure constructed in the step (3.1) by using the data set generated in the step (2.1) to obtain the final parameters of the multi-focus image-based referenced super-resolution network algorithm.
Further, the unified network structure constructed in the step (3.1) uses the parameters obtained by training in the step one and the step two.
Compared with the prior art, the application has the advantages that:
1. the application provides a data set trained by a multi-focus fusion algorithm constructed by Gaussian blur and homography based on a significant target detection data set, solves the problem of the lack of training data, and is more consistent with the data of a real scene;
2. the application uses the original multi-focus image as a reference, helps the fused full-focus image to further improve the resolution, fully utilizes the information of the data, and has better super-resolution effect than that of a single image.
3. The application provides a solution from multi-focus image acquisition to full-focus image super-resolution complete, fully utilizes the original image acquired by the camera, solves the problem of insufficient depth of field and resolution of the camera, and improves the imaging quality of the image.
Drawings
FIG. 1 is an overall flow of the multi-focus image-based referenced super-resolution method of the present application;
FIG. 2 is a dataset for training a multi-focus fusion algorithm made in accordance with the present application;
fig. 3 is a block diagram of a reference super resolution module.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of methods consistent with aspects of the application as detailed in the accompanying claims.
As shown in fig. 1, the present application provides a multi-focus image-based super-resolution method for a reference image, comprising the steps of:
firstly, shooting a plurality of multi-focus images of a scene with a large depth of field by a camera, and fusing the multi-focus images into a full-focus clear image by using a deep learning algorithm;
step two, for the fused full-focus image, in order to improve the resolution, using a plurality of multi-focus images obtained by shooting as references, extracting relevant features in the full-focus image, fusing the features into corresponding positions, and obtaining a reference super-resolution image by using a deep learning method;
and thirdly, jointly training the multi-focus fusion method and the referenced super-resolution method to obtain a final algorithm structure, and further improving the super-resolution effect.
In one embodiment of the present application, the first step includes the following sub-steps:
(1.1) As shown in FIG. 2, the application carries out Gaussian blur and homography conversion processing on the salient object detection data set DUTS to manufacture a data set for training a multi-focus image fusion algorithm. The saliency target detection dataset DUTS is used as the original dataset, and the saliency target mask of the first column in FIG. 2 is used as the foreground-background boundary. The foreground and background are processed separately using gaussian blur to obtain images of columns 2 and 4. And then simulating shaking when the image is actually shot by using homography transformation to obtain the image of the 3 rd and 5 th columns. The original image of column 6 is taken as the full focus image.
(1.2) training a multi-focus fusion deep learning network algorithm by using the manufactured data set. The multi-focus fusion deep learning algorithm is selected from: UFA and IFCNN.
And (1.3) shooting a plurality of multi-focus images of a scene with a large depth of field by using a handheld camera, and fusing the multi-focus images into a full-focus clear image by using a trained network. Here, the Fusion method is represented by Fusion, and the multi-focus image is obtained after processing:
I Focus ＝Fusion(I MF )
in an embodiment of the present application, the second step includes the following sub-steps:
(2.1) downsampling the data set generated in step (1.1) as a data set trained with a reference super-resolution algorithm;
(2.2) as shown in fig. 3, the trained VGG network is used to extract the features of the reference image and the full-focus image, calculate the correlation between the features, and find the position and feature value of the reference image, where the features are most similar to the features of each position in the full-focus image. Here, I is used Ref And I Focus Representing the reference image and the full focus image respectively, the feature is extracted by using VGG to obtain:
F Ref ＝VGG(I Ref )
F Focus ＝VGG(I Focus )
calculating correlation between features, finding F Ref Intermediate and F Focus The most similar feature at each location and moves to a location on the corresponding full focus feature map, expressed as:
F RefTrans ＝Transform(F Focus ,I Ref )
(2.3) training a non-reference super-resolution algorithm network using the data set generated in step (2.1) as in the lower two rows of the non-reference super-resolution network framework in fig. 3. The non-reference super-resolution algorithm here is selected from: EDSR, SRCNN, ESPCN. And (3) using SR-MainNet to represent a selected non-reference super-resolution network structure, and performing super-division on the reference map and the full-focus map to obtain the target image:
I RefSr ＝SR-MainNet(I Ref )
I FocusSr ＝SR-MainNet(I Focus )
(2.4) merging the reference image features extracted in the step (2.2) into the non-reference super-resolution network in the step (2.3) to obtain the final reference super-resolutionA rate network. Because the main network structure with reference superdivision is also SR-MainNet, the SR-MainNet is used mo Representing the corrected reference super-resolution network, and obtaining the following result after the reference super-resolution of the full-focus image:
I FocusRefSr ＝SR-MainNet mo (I Focus ，F RefTrans )
in an embodiment of the present application, the third step includes the following sub-steps:
(3.1) connecting two networks by taking the output of the multi-focus fusion network as the input with reference super resolution, and constructing an end-to-end network structure, wherein the network after connection is initialized by using the network parameters of the first step and the second step;
and (3.2) performing fine tuning training on the unified network structure constructed in the step (3.1) by using the data set generated in the step (2.1) to obtain the final parameters of the multi-focus image-based referenced super-resolution network algorithm.
The above examples of the present application are only for illustrating the technical aspects of the present application in detail, and are not limiting to the embodiments of the present application. Other variations and modifications of the above description will be apparent to those of ordinary skill in the art, and it is not intended to be exhaustive of all embodiments, all of which are within the scope of the application.

Claims (7)

1. A multi-focus image-based referenced super-resolution method, comprising:
firstly, shooting a plurality of multi-focus images of a scene with a large depth of field by a camera, and fusing the multi-focus images into a full-focus image by using a deep learning algorithm;
step two, for the fused full-focus image, taking a plurality of multi-focus images obtained by shooting as references to improve the resolution of the fused full-focus image, extracting relevant features in the full-focus image, fusing the features into corresponding positions, and obtaining a referenced super-resolution image by using a deep learning method;
and thirdly, jointly training the multi-focus fusion method and the referenced super-resolution method to obtain a final algorithm structure.
2. The multi-focus image-based referenced super-resolution method of claim 1, wherein: the first step comprises the following steps:
(1.1) carrying out Gaussian blur and homography conversion processing on a salient object detection data set DUTS to manufacture a first data set for training a multi-focus image fusion algorithm;
(1.2) training a multi-focus fused deep learning network algorithm using the first data set;
and (1.3) shooting a plurality of multi-focus images on a scene with a large depth of field by using a handheld camera, and fusing the multi-focus images into a full-focus image by using a trained multi-focus fused deep learning network algorithm.
3. The multi-focus image-based referenced super-resolution method of claim 2, wherein: in the step (1.2), the multi-focus fusion algorithm is selected from: UFA and/or IFCNN.
4. The multi-focus image-based referenced super-resolution method of claim 2, wherein: the second step comprises the following steps:
(2.1) downsampling the first data set generated in step (1.1) as a second data set trained with a reference super-resolution algorithm;
(2.2) respectively extracting the features of the reference image and the full-focus image by using a VGG network, calculating the correlation between the features, and finding out the position and the feature value of the most similar feature in the reference image and the feature in each position in the full-focus image;
(2.3) training a non-reference super-resolution algorithm network using the second data set generated in step (2.1);
and (2.4) merging the position and the characteristic value of the most similar characteristic to each position characteristic in the full-focus image in the reference image found in the step (2.2) into the non-reference super-resolution network in the step (2.3) to obtain a final reference super-resolution network.
5. The multi-focus image-based referenced super-resolution method of claim 4, wherein: in the step (2.3), the non-reference super-resolution algorithm is selected from: EDSR, SRCNN and/or ESPCN.
6. A multi-focus image based referenced super-resolution method as claimed in claim 3, wherein: the third step comprises the following steps:
(3.1) taking the output of the multi-focus fusion network as the input with reference super resolution, connecting two networks, and constructing an end-to-end network structure;
and (3.2) performing fine tuning training on the unified network structure constructed in the step (3.1) by using the second data set generated in the step (2.1) to obtain final parameters of the multi-focus image-based referenced super-resolution network algorithm.
7. The multi-focus image-based referenced super-resolution method of claim 6, wherein: the unified network structure constructed in the step (3.1) uses the parameters obtained by training in the step one and the step two.
CN202310782600.7A 2023-06-29 2023-06-29 Multi-focus image-based referenced super-resolution method Pending CN116823611A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310782600.7A CN116823611A (en) 2023-06-29 2023-06-29 Multi-focus image-based referenced super-resolution method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310782600.7A CN116823611A (en) 2023-06-29 2023-06-29 Multi-focus image-based referenced super-resolution method

Publications (1)

Publication Number Publication Date
CN116823611A true CN116823611A (en) 2023-09-29

Family

ID=88128827

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310782600.7A Pending CN116823611A (en) 2023-06-29 2023-06-29 Multi-focus image-based referenced super-resolution method

Country Status (1)

Country Link
CN (1) CN116823611A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117372274A (en) * 2023-10-31 2024-01-09 珠海横琴圣澳云智科技有限公司 Scanned image refocusing method, apparatus, electronic device and storage medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117372274A (en) * 2023-10-31 2024-01-09 珠海横琴圣澳云智科技有限公司 Scanned image refocusing method, apparatus, electronic device and storage medium

Similar Documents

Publication Publication Date Title
Wang et al. Light field image super-resolution using deformable convolution
CN110428366B (en) Image processing method and device, electronic equipment and computer readable storage medium
CN109190581B (en) Image sequence target detection and identification method
Zheng et al. Learning Cross-scale Correspondence and Patch-based Synthesis for Reference-based Super-Resolution.
CN110599400B (en) EPI-based light field image super-resolution method
US20220148297A1 (en) Image fusion method based on fourier spectrum extraction
CN111915484A (en) Reference image guiding super-resolution method based on dense matching and self-adaptive fusion
CN109614933B (en) Motion segmentation method based on deterministic fitting
WO2012068902A1 (en) Method and system for enhancing text image clarity
CN108668069B (en) Image background blurring method and device
CN116823611A (en) Multi-focus image-based referenced super-resolution method
CN112085031A (en) Target detection method and system
Wang et al. Learning an epipolar shift compensation for light field image super-resolution
CN112651911A (en) High dynamic range imaging generation method based on polarization image
CN117541629B (en) Infrared image and visible light image registration fusion method based on wearable helmet
Feng et al. Deep video super-resolution using hybrid imaging system
Mehta et al. Gated multi-resolution transfer network for burst restoration and enhancement
Lai et al. Hyperspectral Image Super Resolution With Real Unaligned RGB Guidance
CN115830064B (en) Weak and small target tracking method and device based on infrared pulse signals
CN116469172A (en) Bone behavior recognition video frame extraction method and system under multiple time scales
CN116402908A (en) Dense light field image reconstruction method based on heterogeneous imaging
CN116208812A (en) Video frame inserting method and system based on stereo event and intensity camera
Alam et al. Light field extraction from a conventional camera
CN113538241B (en) Super-resolution image generation method for scene text recognition
Čadík et al. Automated outdoor depth-map generation and alignment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination