WO2022042124A1 - 超分辨率图像重建方法、装置、计算机设备和存储介质 - Google Patents
超分辨率图像重建方法、装置、计算机设备和存储介质 Download PDFInfo
- Publication number
- WO2022042124A1 WO2022042124A1 PCT/CN2021/106647 CN2021106647W WO2022042124A1 WO 2022042124 A1 WO2022042124 A1 WO 2022042124A1 CN 2021106647 W CN2021106647 W CN 2021106647W WO 2022042124 A1 WO2022042124 A1 WO 2022042124A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- processed
- feature information
- filter
- pixel
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000001914 filtration Methods 0.000 claims abstract description 69
- 238000012545 processing Methods 0.000 claims description 33
- 238000004590 computer program Methods 0.000 claims description 29
- 238000000605 extraction Methods 0.000 claims description 10
- 230000008521 reorganization Effects 0.000 claims description 10
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 230000004927 fusion Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 8
- 239000000284 extract Substances 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 6
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
Definitions
- the present application relates to a super-resolution image reconstruction method, apparatus, computer equipment and storage medium.
- Image super-resolution aims to reconstruct a high-resolution image from a single low-resolution image.
- this technology has been widely used in high-definition images, surveillance video and other fields, and has made great progress.
- the prior art is based on deep learning methods, which can be roughly divided into two categories: one is to directly learn the mapping from low-resolution images to high-resolution images, and the other is to perform pixel adaptive filtering.
- a first aspect of the present application provides a super-resolution image reconstruction method, the method comprising:
- the corresponding preset filters are respectively combined to obtain a target filter corresponding to each of the pixel positions in the to-be-processed image
- the pixel information of the corresponding pixel positions in the initial resolution image is filtered by the target filter, so as to obtain a super-resolution image of the to-be-processed image.
- a second aspect of the present application provides a method for denoising and decompressing JPEG images, comprising:
- the corresponding preset filters are respectively combined to obtain a target filter corresponding to each of the pixel positions in the to-be-processed image
- the pixel information of the corresponding pixel positions in the initial resolution image is filtered through the target filter, so as to obtain a super-resolution image of the to-be-processed image.
- a third aspect of the present application provides an apparatus for super-resolution image reconstruction, the apparatus comprising:
- an image acquisition module for acquiring the initial resolution image of the image to be processed
- a feature extraction module configured to extract image feature information of the image to be processed, and determine a plurality of filtering parameters corresponding to each pixel position in the image to be processed according to the image feature information;
- a filter combination module configured to combine the corresponding preset filters according to the plurality of filtering parameters corresponding to the pixel positions, to obtain the target filter corresponding to each of the pixel positions in the image to be processed; as well as
- An image reconstruction module configured to perform filtering processing on pixel information of corresponding pixel positions in the initial resolution image through the target filter to obtain a super-resolution image of the to-be-processed image.
- a fourth aspect of the present application provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
- the corresponding preset filters are respectively combined to obtain a target filter corresponding to each of the pixel positions in the to-be-processed image
- the pixel information of the corresponding pixel positions in the initial resolution image is filtered by the target filter, so as to obtain a super-resolution image of the to-be-processed image.
- a fifth aspect of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
- the corresponding preset filters are respectively combined to obtain a target filter corresponding to each of the pixel positions in the to-be-processed image
- the pixel information of the corresponding pixel positions in the initial resolution image is filtered through the target filter, so as to obtain a super-resolution image of the to-be-processed image.
- FIG. 1 is an application environment diagram of a super-resolution image reconstruction method in one embodiment.
- FIG. 2 is a schematic flowchart of a super-resolution image reconstruction method in one embodiment.
- FIG. 3 is a schematic flowchart of extracting image feature information of an image to be processed through a predefined network in one embodiment.
- FIG. 4 is a schematic diagram of a predefined filter parameter information table in one embodiment.
- FIG. 5 is a schematic flowchart of a method for obtaining a Gaussian filter in an embodiment.
- FIG. 6 is a schematic flowchart of obtaining a target filter in one embodiment.
- FIG. 7 is a schematic flowchart of filtering processing in one embodiment.
- FIG. 8 is a schematic flowchart of a step of extracting image feature information of an image to be processed in one embodiment.
- FIG. 9 is a schematic flowchart of a step of extracting first image feature information of an image to be processed by using a residual network in one embodiment.
- FIG. 10 is a structural block diagram of an apparatus for super-resolution image reconstruction in one embodiment.
- Figure 11 is a diagram of the internal structure of a computer device in one embodiment.
- the super-resolution image reconstruction method provided in this application can be applied to the application environment shown in FIG. 1 .
- the terminal 11 communicates with the server 12 through the network.
- the server 12 acquires the image to be processed sent by the terminal 11 through the network; the server 12 acquires the initial resolution image of the image to be processed; the server 12 extracts the image feature information of the image to be processed, and determines the position of each pixel in the image to be processed according to the image feature information.
- the corresponding multiple filtering parameters the server 12 respectively combines the corresponding preset filters according to the multiple filtering parameters corresponding to the respective pixel positions, and obtains the target filter corresponding to each pixel position in the image to be processed; the server 12 passes the target filter The filter performs filtering processing on the pixel information of the corresponding pixel positions in the initial resolution image to obtain a super-resolution image of the to-be-processed image, and the server 12 returns the super-resolution image of the to-be-processed image to the terminal 11 to complete the super-resolution Image reconstruction.
- the terminal 11 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 12 can be implemented by an independent server or a server cluster composed of multiple servers.
- a super-resolution image reconstruction method is provided, and the method is applied to the server 12 in FIG. 1 as an example for description, including the following steps:
- Step 21 Acquire an image of an initial resolution of the image to be processed.
- the image to be processed is a single low-resolution image (LR, Low Resolution, low resolution);
- the initial resolution image is an initial high-resolution image obtained by performing simple interpolation processing on a single low-resolution image (HR, High Resolution, High Resolution).
- the terminal sends a super-resolution image reconstruction request to the server, and the request carries a low-resolution image to be processed for super-resolution image reconstruction, which may be in the form of a single image or a data set composed of multiple images;
- the server verifies After the super-resolution image reconstruction request sent by the terminal, the low-resolution to-be-processed images are sequentially extracted from the super-resolution image reconstruction request; in addition, the server obtains the image by obtaining the storage address of the image and the to-be-processed image from the request. A list of images, the server obtains images from the storage address provided by the terminal according to the image list.
- the server After acquiring the low-resolution image to be processed for super-resolution image reconstruction, the server performs preliminary processing on the low-resolution image to be processed through an interpolation algorithm to obtain a corresponding initial resolution image.
- the interpolation algorithm can use Nearest Neighbour Interpolation, Bilinear Interpolation, Bicubic Interpolation, etc.
- the resolution of the initial resolution image obtained by the interpolation processing of the image to be processed is higher than the resolution of the image to be processed, and the super-resolution image result can be obtained after the initial resolution image is filtered by the target filter obtained subsequently.
- the server obtains the to-be-processed image from the terminal, and performs preliminary processing on the to-be-processed image to obtain the initial resolution image, which can be directly used in the later filtering processing, thereby improving the efficiency of super-resolution image reconstruction.
- Step 22 Extract the image feature information of the image to be processed, and determine a plurality of filtering parameters corresponding to each pixel position in the image to be processed according to the image feature information.
- the image feature information of the image to be processed is extracted through a predefined network as shown in FIG. 3 , and a plurality of filtering parameters corresponding to each pixel position in the image to be processed are further determined according to the image feature information.
- the predefined network can be further divided into three parts, namely the local fusion part, the pixel reorganization part and the convolution part.
- the local fusion part is composed of multiple local fusion units (LFB, Local Fusion Block, local fusion block for efficient residual learning), each local fusion unit can extract the image features of the image to be processed, and perform deep residual learning.
- LLB Local Fusion Block
- the structure of the local fusion unit is shown in the lower part of the expansion box in Figure 3, which is mainly composed of two branches; the first branch passes through multiple residual modules (RB modules, Residual Block), and the outputs of multiple residual modules are connected in At the same time, after a convolutional layer (Conv), it is multiplied with the parameter ⁇ 2 ; the other branch is directly multiplied with the parameter ⁇ 1 ; finally, the results of the two branches are added together as the output of a local fusion unit.
- Conv convolutional layer
- the pixel shuffling unit is composed of a pixel shuffle, and can convert low-resolution image features into high-resolution image features.
- the convolution part can perform a convolution operation. After regressing the linear combination coefficients, the linear combination coefficients are output as multiple filtering parameters corresponding to each pixel position in the image to be processed; the three-dimensional sizes of the output results are Hs ⁇ Ws ⁇ L, where H is the image height, W is the image width, s is the magnification, and L is the number of preset filters.
- the image feature information of the image to be processed is extracted and processed through a predefined network including a local fusion part, a pixel reorganization part and a convolution part, and the corresponding filtering parameters are further obtained after the image feature information is obtained.
- the speed is increased, the model framework is easy to optimize, and the overall efficiency of super-resolution image reconstruction is improved.
- Step 23 Combine the corresponding preset filters according to the multiple filtering parameters corresponding to the pixel positions to obtain target filters corresponding to the pixel positions in the image to be processed.
- the predefined filter parameter information table in this application that is, the predefined filter dictionary
- the information table contains a plurality of preset filters, and the number of them corresponds to each pixel position
- the number of multiple filter parameters is the same.
- the predefined filter parameter information table is composed of two types of filters, namely Gaussian and Gaussian difference, and the Gaussian difference is obtained by subtracting two Gaussian filters.
- Figure 4 details the corresponding coefficients for each filter, where ⁇ contains three values 1.0, 0.6 and 0.2, r means the ratio of the short axis to the long axis ( ⁇ 1 divided by ⁇ 2 ), there are 1.0, 0.8, 0.6, 0.4, 0.2 these values, ⁇ is the angle of rotation, there are 0 degrees, 30 degrees, 60 degrees, 90 degrees, 120 degrees, 150 degrees.
- FIG. 6 it is a schematic diagram of combining respective preset filters to obtain a target filter.
- the server may combine preset filters corresponding to multiple filter parameters corresponding to each pixel position according to a predefined filter parameter information table, so as to obtain a target filter corresponding to each pixel position.
- the filtering parameters and the preset filter are correspondingly combined to obtain the target filter corresponding to each pixel position in the to-be-processed image; High-resolution images; the number of models is small, the calculation speed is accelerated, the model framework is easy to optimize, and the overall efficiency of super-resolution image reconstruction is improved.
- Step 24 Perform filtering processing on the pixel information of the corresponding pixel positions in the initial resolution image by the target filter to obtain a super-resolution image of the image to be processed.
- filtering is a common processing method in image processing; through the steps corresponding to Figure 6, different filter combinations corresponding to each pixel can be obtained, and these filters are applied to the initial resolution obtained by interpolation processing. After the image, the final result can be obtained.
- the obtained initial resolution image of the image to be processed is used as input, and the target filter corresponding to each pixel position in the image to be processed is used as the filtering processing device.
- the super-resolution image HR-Y can be obtained as the super-resolution image reconstruction result.
- a good image restoration effect can be obtained through one-time filtering.
- the number of models is small, the calculation speed is accelerated, the model framework is easy to optimize, and the overall efficiency of super-resolution image reconstruction is improved.
- the above-mentioned super-resolution image reconstruction method includes: acquiring an initial resolution image of an image to be processed; extracting image feature information of the image to be processed, and determining a plurality of filtering parameters corresponding to each pixel position in the image to be processed according to the image feature information; According to a plurality of filtering parameters corresponding to each pixel position, the corresponding preset filters are respectively combined to obtain the target filter corresponding to each pixel position in the image to be processed; The pixel information of the pixel position is filtered to obtain a super-resolution image of the image to be processed.
- a target filter is obtained by combining multiple filtering parameters corresponding to each pixel position of the image to be processed and a preset filter, and then a super-resolution image can be obtained by filtering the initial resolution image of the image to be processed by using the target filter.
- the application provides a new super-resolution image reconstruction strategy, which can obtain a good image restoration effect with only one filtering.
- the number of models is small, the calculation speed is significantly improved, the entire model framework is easy to optimize, and the overall improvement of super-resolution is improved. Efficiency of high-resolution image reconstruction.
- the image feature information of the image to be processed is extracted, including:
- Step 81 extracting the first image feature information of the image to be processed through a residual network
- Step 82 Perform pixel reorganization on the first image feature information to obtain second image feature information corresponding to the first image feature information, as the image feature information of the to-be-processed image; the image resolution corresponding to the second image feature information is higher than the first image feature information.
- the image resolution corresponding to the image feature information is higher than the first image feature information.
- the local fusion unit is equivalent to a residual network, which may include one or more cascaded residual blocks, concatenated splicing layers, convolutional layers in series, channel separation modules, etc.
- the input of the difference block is connected to the output of the feature extraction; the feature fusion module extracts the feature image input by the residual block and the output feature image of the last level residual block, and fuses these feature images of different levels and outputs them to the next residual block. poor network until iterative training is complete.
- the pixel reorganization module After the pixel reorganization module obtains the deep features extracted from the image block by the multi-layer residual network, the feature image is reorganized to improve the resolution of the image; then a convolution layer can be used to regress the linear combination coefficients to obtain the Process multiple filtering parameters corresponding to each pixel position in the image.
- the first image feature information is obtained through the residual network, and then the first image feature information is reorganized into the second image feature information through pixel reorganization, and the complete image feature information of the image to be processed has been obtained, and good image restoration can be obtained.
- the number of models is small, the calculation speed is accelerated, the model framework is easy to optimize, and the overall efficiency of super-resolution image reconstruction is improved.
- the first image feature information of the to-be-processed image is extracted through a residual network, including:
- Step 91 dividing the to-be-processed image into multiple image blocks of the same size
- Step 92 Extract feature information of a plurality of image blocks through a plurality of residual branches included in the residual network, respectively, as the first image feature information of the to-be-processed image.
- the to-be-processed image is processed into blocks, and cropped into image blocks with the same pixel size; the image blocks are respectively input into the residual network, and after feature extraction is performed through the residual network, the image blocks with and multiple image blocks are obtained respectively.
- feature information as the first image feature information of the image to be processed.
- the image is divided into blocks, which improves the processing speed of the image.
- obtaining the initial resolution image of the image to be processed includes: obtaining the image to be processed; performing linear interpolation processing on the image to be processed, and using the image after linear interpolation processing as the initial resolution image.
- the server obtains the to-be-processed image, and performs linear interpolation processing on the to-be-processed image, such as enlarging the to-be-processed image by a bicubic interpolation method, to obtain an image of the initial resolution of the to-be-processed image.
- a preliminary high-resolution image can be obtained through simple linear interpolation, less data is used, fewer model units, simple configuration, and faster running speed.
- multiple filtering parameters corresponding to each pixel position in the image to be processed are determined, including: performing convolution processing on the image feature information to obtain each pixel position in the to-be-processed image Corresponding combination coefficients; the combination coefficients include multiple filter parameters corresponding to corresponding pixel positions.
- a combination coefficient corresponding to each pixel position in the image to be processed can be obtained; the combination coefficient is composed of a plurality of filter parameters corresponding to each pixel position, and its three-dimensional size is Hs, Ws, L, where H is the image height, W is the image width, s is the magnification, and L is the number of preset filters.
- the combined coefficients are regressed to obtain multiple filter parameters corresponding to each pixel position in the image to be processed; Combining to obtain the target filter, and then using the target filter to filter the initial resolution image of the image to be processed to obtain a super-resolution image, only one filtering can obtain a good image restoration effect, and the number of models is small. The calculation speed is accelerated, the model framework is easy to optimize, and the overall efficiency of super-resolution image reconstruction is improved.
- the corresponding preset filters are respectively combined to obtain the target filter corresponding to each pixel position in the image to be processed, including:
- the predefined filter parameter information table is a predefined dictionary; assuming that there are L filters in a dictionary, each pixel position of the picture will be extracted through the image feature information, and then different L coefficients will be obtained. The coefficients are multiplied by the corresponding preset filters in the predefined filter parameter information table, and after summing the product results, the final filter corresponding to each pixel can be obtained, which is used as the position of each pixel in the image to be processed. the corresponding target filter.
- a target filter is obtained by combining multiple filter parameters corresponding to each pixel position of the image to be processed with a preset filter, and then the target filter is used to filter the initial resolution image of the image to be processed to obtain super-resolution Image; only one filtering can get a good image restoration effect.
- the number of models is small, the calculation speed is accelerated, the model framework is easy to optimize, and the overall efficiency of super-resolution image reconstruction is improved.
- the target filter is calculated in the following manner:
- F i is a single target filter
- D is a predefined filter parameter information table
- L is the number of preset filters in the predefined filter parameter information table
- ⁇ is a filtering parameter.
- each coefficient is multiplied by the corresponding preset filter in the predefined filter parameter information table, and after the multiplication results are summed, the final filter corresponding to each pixel can be obtained, which is used as the image corresponding to the image to be processed.
- the target filter is obtained by combining multiple filter parameters corresponding to each pixel position of the image to be processed and a preset filter, which improves the efficiency of super-resolution image reconstruction as a whole.
- the super-resolution image reconstruction method provided in this application can also be used for image denoising and image de-JPEG compression.
- the specific process of image de-noising and image de-JPEG compression is consistent with the above-mentioned super-resolution image reconstruction method, and only needs to remove the step of performing linear interpolation on the image to be processed, and the step of performing pixel reorganization on the feature information of the first image is not required; Because no upsampling is required for image denoising and image de-JPEG compression, the input and output sizes are the same.
- steps in the flowcharts of FIGS. 2 and 8-9 are sequentially displayed according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIGS. 2 and 8-9 may include multiple steps or multiple stages. These steps or stages are not necessarily executed and completed at the same time, but may be executed at different times. These steps or stages are not necessarily completed at the same time. The order of execution of the steps is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the steps or stages in the other steps.
- a super-resolution image reconstruction apparatus including: an image acquisition module 101, a feature extraction module 102, a filter combination module 103 and an image reconstruction module 104, wherein:
- An image acquisition module 101 configured to acquire an image of an initial resolution of an image to be processed
- a feature extraction module 102 configured to extract image feature information of the to-be-processed image, and determine a plurality of filtering parameters corresponding to each pixel position in the to-be-processed image according to the image feature information;
- the filter combination module 103 is configured to combine the corresponding preset filters according to the multiple filtering parameters corresponding to the pixel positions, to obtain the target filter corresponding to the pixel positions in the to-be-processed image. ;
- the image reconstruction module 104 is configured to perform filtering processing on the pixel information of the corresponding pixel positions in the initial resolution image through the target filter to obtain a super-resolution image of the to-be-processed image.
- the feature extraction module 102 is further configured to extract the first image feature information of the to-be-processed image through a residual network; perform pixel reorganization on the first image feature information to obtain the first image feature information
- the corresponding second image feature information is used as the image feature information of the to-be-processed image; the image resolution corresponding to the second image feature information is higher than the image resolution corresponding to the first image feature information.
- the feature extraction module 102 is further configured to divide the to-be-processed image into multiple image blocks of the same size; through multiple residual branches included in the residual network, extract multiple The feature information of the image block is used as the first image feature information of the to-be-processed image.
- the feature extraction module 102 is further configured to acquire the to-be-processed image; perform linear interpolation processing on the to-be-processed image, and use the linearly interpolated image as the initial resolution image.
- the feature extraction module 102 is further configured to perform convolution processing on the image feature information to obtain a combination coefficient corresponding to each pixel position in the to-be-processed image; the combination coefficient includes multiple filter parameters.
- the filter combination module 103 is further configured to determine, from a predefined filter parameter information table, a plurality of preset filters corresponding to the filtering parameters; The preset filter is linearly weighted to obtain the target filter corresponding to each of the pixel positions in the to-be-processed image.
- the filter combination module 103 is further configured to obtain the target filter by calculating in the following manner:
- the F i is the single target filter
- D is the predefined filter parameter information table
- the L is the number of the preset filters in the predefined filter parameter information table
- the ⁇ is the filtering parameter.
- each module in the above-mentioned super-resolution image reconstruction apparatus may be implemented in whole or in part by software, hardware, and combinations thereof.
- the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
- a computer device is provided, and the computer device may be a server, and its internal structure diagram may be as shown in FIG. 11 .
- the computer device includes a processor, memory, and a network interface connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities.
- the memory of the computer device includes a non-volatile storage medium, an internal memory.
- the nonvolatile storage medium stores an operating system, a computer program, and a database.
- the internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium.
- the computer device's database is used to store super-resolution image reconstruction data.
- the network interface of the computer device is used to communicate with an external terminal through a network connection.
- the computer program when executed by a processor, implements a super-resolution image reconstruction method.
- FIG. 11 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.
- a computer device including a memory and a processor, a computer program is stored in the memory, and the processor implements the following steps when executing the computer program:
- the pixel information of the corresponding pixel positions in the initial resolution image is filtered by the target filter, so as to obtain a super-resolution image of the to-be-processed image.
- the processor further implements the following steps when executing the computer program: extracting first image feature information of the to-be-processed image through a residual network; performing pixel reorganization on the first image feature information to obtain the first image feature information.
- the second image feature information corresponding to the first image feature information is used as the image feature information of the to-be-processed image; and the image resolution corresponding to the second image feature information is higher than the image resolution corresponding to the first image feature information .
- the processor further implements the following steps when executing the computer program: dividing the image to be processed into a plurality of image blocks of the same size; and using a plurality of residual branches included in the residual network, The feature information of a plurality of the image blocks is respectively extracted as the first image feature information of the to-be-processed image.
- the processor when the processor executes the computer program, the following steps are further implemented: acquiring the image to be processed; and performing linear interpolation processing on the image to be processed, and using the image after linear interpolation processing as the initial resolution image .
- the processor further implements the following steps when executing the computer program: performing convolution processing on the image feature information to obtain a combination coefficient corresponding to each pixel position in the to-be-processed image; the combination coefficient includes corresponding pixels Multiple filter parameters corresponding to the position.
- the processor further implements the following steps when executing the computer program: from a predefined filter parameter information table, determining a plurality of preset filters corresponding to the filtering parameters;
- the preset filter corresponding to the filtering parameter is linearly weighted to obtain the target filter corresponding to each of the pixel positions in the image to be processed.
- the processor also implements the following steps when executing the computer program: calculating the target filter in the following manner:
- the F i is the single target filter
- D is the predefined filter parameter information table
- the L is the number of the preset filters in the predefined filter parameter information table
- the ⁇ is the filtering parameter.
- a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
- the corresponding preset filters are respectively combined to obtain a target filter corresponding to each of the pixel positions in the to-be-processed image
- the pixel information of the corresponding pixel positions in the initial resolution image is filtered through the target filter, so as to obtain a super-resolution image of the to-be-processed image.
- the following steps are further implemented: extracting first image feature information of the to-be-processed image through a residual network; performing pixel reorganization on the first image feature information to obtain the The second image feature information corresponding to the first image feature information is used as the image feature information of the image to be processed; and the image resolution corresponding to the second image feature information is higher than the image resolution corresponding to the first image feature information. Rate.
- the computer program further implements the following steps when executed by the processor: dividing the image to be processed into a plurality of image blocks of the same size; and passing a plurality of residual branches included in the residual network , and extract the feature information of a plurality of the image blocks respectively, as the first image feature information of the to-be-processed image.
- the following steps are further implemented: acquiring the image to be processed; and performing linear interpolation processing on the image to be processed, and using the image after linear interpolation processing as the initial resolution image.
- the following steps are further implemented: performing convolution processing on the image feature information to obtain a combination coefficient corresponding to each pixel position in the to-be-processed image; the combination coefficient includes corresponding Multiple filter parameters corresponding to pixel positions.
- the computer program further implements the following steps when executed by the processor: from a predefined filter parameter information table, determining a plurality of preset filters corresponding to the filtering parameters;
- the preset filter corresponding to the filtering parameter is linearly weighted to obtain the target filter corresponding to each of the pixel positions in the image to be processed.
- the target filter is obtained by calculating in the following manner:
- the F i is the single target filter
- D is the predefined filter parameter information table
- the L is the number of the preset filters in the predefined filter parameter information table
- the ⁇ is the filtering parameter.
- any reference to memory, storage, database or other media used in the various embodiments provided in this application may include at least one of non-volatile and volatile memory.
- Non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, or optical memory, and the like.
- Volatile memory may include random access memory (RAM) or external cache memory.
- the RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
一种超分辨率图像重建方法、装置、计算机设备和存储介质,方法包括:获取待处理图像的初始分辨率图像;提取待处理图像的图像特征信息,根据图像特征信息确定出与待处理图像中各个像素位置对应的多个滤波参数;按照各个像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到待处理图像中各个像素位置对应的目标滤波器;以及通过目标滤波器对初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到待处理图像的超分辨率图像。
Description
相关申请交叉引用
本申请要求2020年08月25日递交的、标题为“超分辨率图像重建方法、装置、计算机设备和存储介质”、申请号为2020108629572的中国申请,其公开内容通过引用全部结合在本申请中。
本申请涉及一种超分辨率图像重建方法、装置、计算机设备和存储介质。
图像超分辨率旨在从单张低分辨率的图像重建出高分辨率的图像。在过去的十几年里,这项技术被广泛地应用于高清图像、监控视频等领域,取得了长足的发展。
现有技术是通过基于深度学习的方法,大致可以分为两类:一类是直接学习从低分辨率图像到高分辨率图像的映射,二是进行像素自适应滤波。
发明内容
根据多个实施例,本申请第一方面提供一种超分辨率图像重建方法,所述方法包括:
获取待处理图像的初始分辨率图像;
提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;
按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器;以及
通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到所述待处理图像的超分辨率图像。
根据多个实施例,本申请第二方面提供一种对图像进行去噪和去JPEG压缩的方法,包括:
获取待处理图像的初始分辨率图像;
提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;
按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到与所述待处理图像中各个所述像素位置对应的目标滤波器;以及
通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波 处理,得到所述待处理图像的超分辨率图像。
根据多个实施例,本申请第三方面提供一种超分辨率图像重建装置,所述装置包括:
图像获取模块,用于获取待处理图像的初始分辨率图像;
特征提取模块,用于提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;
滤波器组合模块,用于按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器;以及
图像重建模块,用于通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到所述待处理图像的超分辨率图像。
根据多个实施例,本申请第四方面提供一种计算机设备,包括存储器和处理器,所述存储器存储有计算机程序,所述处理器执行所述计算机程序时实现以下步骤:
获取待处理图像的初始分辨率图像;
提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;
按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器;以及
通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到所述待处理图像的超分辨率图像。
根据多个实施例,本申请第五方面提供一种计算机可读存储介质,其上存储有计算机程序,所述计算机程序被处理器执行时实现以下步骤:
获取待处理图像的初始分辨率图像;
提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;
按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器;以及
通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波 处理,得到所述待处理图像的超分辨率图像。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请的其它特征和优点将从说明书、附图以及权利要求书变得明显。
为了更清楚地说明本申请实施例或传统技术中的技术方案,下面将对实施例或传统技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。
图1为一个实施例中超分辨率图像重建方法的应用环境图。
图2为一个实施例中超分辨率图像重建方法的流程示意图。
图3为一个实施例中通过预定义网络提取待处理图像的图像特征信息的流程示意图。
图4为一个实施例中预定义的滤波器参数信息表的示意图。
图5为一个实施例中高斯滤波器的获得方式的流程示意图。
图6为一个实施例中得到目标滤波器的流程示意图。
图7为一个实施例中滤波处理的流程示意图。
图8为一个实施例中提取待处理图像的图像特征信息步骤的流程示意图。
图9为一个实施例中通过残差网络提取待处理图像的第一图像特征信息步骤的流程示意图。
图10为一个实施例中超分辨率图像重建装置的结构框图。
图11为一个实施例中计算机设备的内部结构图。
现有的基于深度学习对图像进行重建的方法虽然有着较好的图像恢复效果,但模型复杂度高,计算力需求大,推理速度慢;因此,现有的超分辨率图像重建方法的效率还较低。
为了使本申请的目的、技术方案及优点更加清楚明白,以下结合附图及实施例,对本申请进行进一步详细说明。应当理解,此处描述的具体实施例仅仅用以解释本申请,并不用于限定本申请。
本申请提供的超分辨率图像重建方法,可以应用于如图1所示的应用环境中。其中,终端11通过网络与服务器12进行通信。服务器12获取终端11通过网络发送的待处理图像;服务器12获取待处理图像的初始分辨率图像;服务器12提取待处理图像的图像特征信息,根据图像特征信息确定出与待处理图像中各个像素位置对应的多个滤波参数;服务器12按照各个像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到待处理图像中各个像素位置对应的目标滤波器;服务器12通过目标滤波器对初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到待处理图像的超分辨率图像,服务器12将得到待处理图像的超分辨率图像返回至终端11,完成超分辨率图像的重建。 其中,终端11可以但不限于是各种个人计算机、笔记本电脑、智能手机、平板电脑和便携式可穿戴设备,服务器12可以用独立的服务器或者是多个服务器组成的服务器集群来实现。
在一个实施例中,如图2所示,提供了一种超分辨率图像重建方法,以该方法应用于图1中的服务器12为例进行说明,包括以下步骤:
步骤21,获取待处理图像的初始分辨率图像。
其中,待处理图像是单张低分辨率的图像(LR,Low Resolution,低分辨率);初始分辨率图像是将单张低分辨率的图像进行简单的插值处理后得到的初始高分辨率图像(HR,High Resolution,高分辨率)。
具体地,终端向服务器发送超分辨率图像重建请求,请求中携带有待进行超分辨率图像重建处理的低分辨率的图像,可以是单张也可以是多张图像组成的数据集形式;服务器验证终端发送的超分辨率图像重建请求后,依次从超分辨率图像重建请求中提取低分辨率的待处理图像;另外,服务器获取图像的方式还可以是从请求中获取图像的存储地址以及待处理图像的列表,服务器根据图像列表从终端提供的存储地址中获取图像。
服务器获取到待进行超分辨率图像重建处理的低分辨率的图像后,通过插值算法对低分辨率的待处理图像进行初步处理,得到对应的初始分辨率图像。插值算法可以采用最近邻插值法(Nearest Neighbour Interpolation)、双线性插值法(Bilinear Interpolation)以及双三次插值(Bicubic Interpolation)等。待处理图像经过插值处理后得到的初始分辨率图像的分辨率高于待处理图像的分辨率,初始分辨率图像利用后续得到的目标滤波器进行滤波后即可得到超分辨率图像结果。
本步骤服务器从终端处获取待处理图像,并对待处理图像进行初步的处理后得到初始分辨率图像,能够直接用于后期的滤波处理,提升了超分辨率图像重建的效率。
步骤22,提取待处理图像的图像特征信息,根据图像特征信息确定出与待处理图像中各个像素位置对应的多个滤波参数。
具体地,通过如图3所示的预定义网络提取待处理图像的图像特征信息,并进一步根据图像特征信息确定出与待处理图像中各个像素位置对应的多个滤波参数。
该预定义网络可以进一步分为三个部分,分别是局部融合部、像素重组部以及卷积部。
其中,局部融合部由多个局部融合单元(LFB,Local Fusion Block,高效残差学习的局部融合块)组成,每个局部融合单元能够提取待处理图像的图像特征,并进行深度的残差学习;局部融合单元结构如图3下部分拓展框所示,主要由两个分支组成;其中第一个分支经过多个残差模块(RB模块,Residual Block),多个残差模块的输出连接在一起,经过一个卷积层(Conv)后与参数δ
2相乘;另外一个分支由直接与参数δ
1相乘;最后将两个分支的结果进行相加处理,作为一个局部融合单元的输出结果。
像素重组部由像素重组部件(Pixelshuffle)构成,能够将低分辨率的图像特征转换为高分辨率的图像特征。
卷积部能够执行卷积操作,回归线性组合系数后,输出线性组合系数作为与待处理图像中各个像素位置对应的多个滤波参数;输出结果的三维大小分别是Hs×Ws×L,其中H为图片高度,W为图片宽度,s为放大倍数,L为预设滤波器数量。
本步骤通过包含有局部融合部、像素重组部以及卷积部的预定义网络,对待处理图像的图像特征信息进行提取处理,得到图像特征信息后进一步获取对应的滤波参数,模型数量较少,计算速度加快,模型框架易于优化,整体提高了超分辨率图像重建的效率。
步骤23,按照各个像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到待处理图像中各个像素位置对应的目标滤波器。
其中,如图4所示为本申请中的预定义的滤波器参数信息表,也即预定义滤波器字典,该信息表中包含多个预设滤波器,并且其个数与各个像素位置对应的多个滤波参数个数相同。该预定义的滤波器参数信息表由两种类型的滤波器构成,分别是高斯和高斯差,高斯差是有两个高斯滤波器相减处理得到。
如图5所示,是高斯滤波器的获得方式:
1)由一个标准的圆形滤波器,通过控制长轴(σ
1)和短轴(σ
2)的大小,使之变成一个椭圆;2)通过对椭圆进行逆时针旋转(旋转度数为θ),得到倾斜了一定角度的椭圆;3)然同时调整椭圆的长短轴(σ
1与σ
2同时与系数γ相乘),对其进行放缩,得到最后的结果。
图4详细描述了每个滤波器对应的系数,其中γ包含三个值1.0,0.6和0.2,r的含义是短轴与长轴的比值(σ
1除以σ
2),有1.0,0.8,0.6,0.4,0.2这几个值,θ是旋转 的角度,有0度,30度,60度,90度,120度,150度。
具体地,如图6所示,是将对应的各个预设滤波器进行组合并得到目标滤波器的示意图。
服务器可以根据预定义的滤波器参数信息表将各个像素位置对应的多个滤波参数对应的预设滤波器进行组合,以得到各个像素位置对应的目标滤波器。
本步骤根据像素位置,将滤波参数与预设滤波器进行对应组合,得到待处理图像中各个像素位置对应的目标滤波器;利用目标滤波器对待处理图像的初始分辨率图像进行滤波即可得到超分辨率图像;本模型数量较少,计算速度加快,模型框架易于优化,整体提高了超分辨率图像重建的效率。
步骤24,通过目标滤波器对初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到待处理图像的超分辨率图像。
具体地,滤波是图像处理中常见的一种处理方法;通过图6对应的步骤可以得到与每个像素点都对应的不同滤波器组合,将这些滤波器作用在经过插值处理得到的初始分辨率图像后,可得到最终的结果。
如图7的滤波处理示意图所示,将获取到的待处理图像的初始分辨率图像作为输入,利用与待处理图像中各个像素位置对应的目标滤波器作为滤波处理装置,通过单次滤波后即可得到超分辨率图像HR-Y,作为超分辨率图像重建结果。
本实施例通过一次滤波就可以得到很好的图像恢复效果,同时模型数量较少,计算速度加快,模型框架易于优化,整体提高了超分辨率图像重建的效率。
上述超分辨率图像重建方法,包括:获取待处理图像的初始分辨率图像;提取待处理图像的图像特征信息,根据图像特征信息确定出与待处理图像中各个像素位置对应的多个滤波参数;按照各个像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到待处理图像中各个像素位置对应的目标滤波器;以及通过目标滤波器对初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到待处理图像的超分辨率图像。本申请通过将待处理图像各个像素位置对应的多个滤波参数与预设滤波器进行组合得到目标滤波器,再利用目标滤波器对待处理图像的初始分辨率图像进行滤波即可得到超分辨率图像;本申请提供一种新的超分辨率图像重建策略,仅需一次滤波就可以得到很好的图像 恢复效果,同时模型数量较少,计算速度明显提升,整个模型框架易于优化,整体提高了超分辨率图像重建的效率。
在一个实施例中,如图8所示,上述步骤22,提取待处理图像的图像特征信息,包括:
步骤81,通过残差网络提取待处理图像的第一图像特征信息;
步骤82,对第一图像特征信息进行像素重组,得到第一图像特征信息对应的第二图像特征信息,作为待处理图像的图像特征信息;第二图像特征信息对应的图像分辨率高于第一图像特征信息对应的图像分辨率。
具体地,如图3所示,局部融合单元相当于一个残差网络,其中可以包括一个或多个级联的残差块、串联的拼接层、串联的卷积层、通道分离模块等,残差块的输入与特征提取的输出相连;特征融合模块提取残差块输入的特征图像和最后一级残差块的输出特征图像,并将这些不同层级的特征图像进行融合后输出至下一残差网络,直至迭代训练完成。
像素重组模块获取多层残差网络从图像块中提取到的深层特征后,将特征图像进行重组,提升图像的分辨率;之后可采用一个卷积层,用于回归线性组合系数后得到与待处理图像中各个像素位置对应的多个滤波参数。
本实施例通过残差网络获取第一图像特征信息,再通过像素重组将第一图像特征信息重组为第二图像特征信息,已得到待处理图像完整的图像特征信息,可以得到很好的图像恢复效果,同时模型数量较少,计算速度加快,模型框架易于优化,整体提高了超分辨率图像重建的效率。
在一个实施例中,如图9所示,上述步骤71,通过残差网络提取待处理图像的第一图像特征信息,包括:
步骤91,将待处理图像分为多个大小相同的图像分块;
步骤92,通过残差网络中包含的多个残差分支,分别提取多个图像分块的特征信息,作为待处理图像的第一图像特征信息。
具体地,将待处理图像进行分块处理,分别裁剪为像素大小尺寸相同的图像块;将图像块分别输入残差网络,通过残差网络进行特征提取后分别得到与、多个图像分块的特征信息,以此作为待处理图像的第一图像特征信息。本实施例将图像进行分块,提高了图像 的处理运行速度。
在一个实施例中,上述步骤22,获取待处理图像的初始分辨率图像,包括:获取待处理图像;对待处理图像进行线性插值处理,将线性插值处理后的图像作为初始分辨率图像。
具体地,服务器获取到待处理图像,通过对待处理图像进行线性插值处理,例如通过双三次插值方法放大待处理图像,可得到待处理图像的初始分辨率图像。本实施例通过简单的线性插值即可得到初步的高分辨率图像,使用的数据较少,模型单元较少,配置简单,运行速度较快。
在一个实施例中,上述步骤21,根据图像特征信息确定出与待处理图像中各个像素位置对应的多个滤波参数,包括:对图像特征信息进行卷积处理,得到待处理图像中各个像素位置对应的组合系数;组合系数包括相应像素位置对应的多个滤波参数。
具体地,通过卷积处理,可得到与待处理图像中各个像素位置对应的组合系数;组合系数由各个像素位置对应的多个滤波参数组成,其三维大小分别是Hs,Ws,L,其中H为图片高度,W为图片宽度,s为放大倍数,L为预设滤波器的数量。本实施例通过简单的卷积,使得组合系数回归,得到与待处理图像中各个像素位置对应的多个滤波参数;之后通过将待处理图像各个像素位置对应的多个滤波参数与预设滤波器进行组合得到目标滤波器,再利用目标滤波器对待处理图像的初始分辨率图像进行滤波即可得到超分辨率图像,仅需一次滤波就可以得到很好的图像恢复效果,同时模型数量较少,计算速度加快,模型框架易于优化,整体提高了超分辨率图像重建的效率。
在一个实施例中,按照各个像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到待处理图像中各个像素位置对应的目标滤波器,包括:
从预定义的滤波器参数信息表中,确定出多个与滤波参数对应的预设滤波器;将多个与滤波参数对应的预设滤波器进行线性加权,得到待处理图像中各个像素位置对应的目标滤波器。
具体地,预定义的滤波器参数信息表即为预定义字典;假设一个字典中一共有L个滤波器,图片的每个像素位置都会通过图像特征信息提取后,得到不同的L个系数,每个系数乘上预定义的滤波器参数信息表内对应的预设滤波器,并将乘积结果进行求和后,即可 得到每个像素对应的最终滤波器,作为与待处理图像中各个像素位置对应的目标滤波器。本实施例通过将待处理图像各个像素位置对应的多个滤波参数与预设滤波器进行组合得到目标滤波器,再利用目标滤波器对待处理图像的初始分辨率图像进行滤波即可得到超分辨率图像;仅需一次滤波就可以得到很好的图像恢复效果,同时模型数量较少,计算速度加快,模型框架易于优化,整体提高了超分辨率图像重建的效率。
在一个实施例中,目标滤波器,通过下述方式计算得到:
其中,F
i为单个目标滤波器;D为预定义的滤波器参数信息表;L为预定义的滤波器参数信息表中预设滤波器的数量;Φ为滤波参数。
具体地,每个系数乘上预定义的滤波器参数信息表内对应的预设滤波器,并将乘积结果进行求和后,即可得到每个像素对应的最终滤波器,作为与待处理图像中各个像素位置对应的目标滤波器。本实施例通过将待处理图像各个像素位置对应的多个滤波参数与预设滤波器进行组合得到目标滤波器,整体提高了超分辨率图像重建的效率。
在一个实施例中,本申请所提供的超分辨率图像重建方法,还可以用于图像去噪以及图像去JPEG压缩。
具体地,图像去噪和图像去JPEG压缩的具体流程跟上述超分辨率图像重建方法一致,仅需去掉对待处理图像进行线性插值的步骤,以及无需对第一图像特征信息进行像素重组的步骤;因为图像去噪和图像去JPEG压缩时不需要进行上采样,输入输出大小相同。
应该理解的是,虽然图2、8-9的流程图中的各个步骤按照箭头的指示依次显示,但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明,这些步骤的执行并没有严格的顺序限制,这些步骤可以以其它的顺序执行。而且,图2、8-9中的至少一部分步骤可以包括多个步骤或者多个阶段,这些步骤或者阶段并不必然是在同一时刻执行完成,而是可以在不同的时刻执行,这些步骤或者阶段的执行顺序也不必然是依次进行,而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。
在一个实施例中,如图10所示,提供了一种超分辨率图像重建装置,包括:图像获取模块101、特征提取模块102、滤波器组合模块103和图像重建模块104,其中:
图像获取模块101,用于获取待处理图像的初始分辨率图像;
特征提取模块102,用于提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;
滤波器组合模块103,用于按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器;
图像重建模块104,用于通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到所述待处理图像的超分辨率图像。
在一个实施例中,特征提取模块102还用于通过残差网络提取所述待处理图像的第一图像特征信息;对所述第一图像特征信息进行像素重组,得到所述第一图像特征信息对应的第二图像特征信息,作为所述待处理图像的图像特征信息;所述第二图像特征信息对应的图像分辨率高于所述第一图像特征信息对应的图像分辨率。
在一个实施例中,特征提取模块102还用于将所述待处理图像分为多个大小相同的图像分块;通过所述残差网络中包含的多个残差分支,分别提取多个所述图像分块的特征信息,作为所述待处理图像的第一图像特征信息。
在一个实施例中,特征提取模块102还用于获取所述待处理图像;对所述待处理图像进行线性插值处理,将线性插值处理后的图像作为所述初始分辨率图像。
在一个实施例中,特征提取模块102还用于对所述图像特征信息进行卷积处理,得到所述待处理图像中各个像素位置对应的组合系数;所述组合系数包括相应像素位置对应的多个滤波参数。
在一个实施例中,滤波器组合模块103还用于从预定义的滤波器参数信息表中,确定出多个与所述滤波参数对应的预设滤波器;将多个与所述滤波参数对应的预设滤波器进行线性加权,得到所述待处理图像中各个所述像素位置对应的目标滤波器。
在一个实施例中,滤波器组合模块103还用于通过下述方式计算得到目标滤波器:
其中,所述F
i为单个所述目标滤波器;D为所述预定义的滤波器参数信息表;所述L为所述预定义的滤波器参数信息表中所述预设滤波器的数量;所述Φ为所述滤波参数。
关于超分辨率图像重建装置的具体限定可以参见上文中对于超分辨率图像重建方法的限定,在此不再赘述。上述超分辨率图像重建装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中,也可以以软件形式存储于计算机设备中的存储器中,以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中,提供了一种计算机设备,该计算机设备可以是服务器,其内部结构图可以如图11所示。该计算机设备包括通过***总线连接的处理器、存储器和网络接口。其中,该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作***、计算机程序和数据库。该内存储器为非易失性存储介质中的操作***和计算机程序的运行提供环境。该计算机设备的数据库用于存储超分辨率图像重建数据。该计算机设备的网络接口用于与外部的终端通过网络连接通信。该计算机程序被处理器执行时以实现一种超分辨率图像重建方法。
本领域技术人员可以理解,图11中示出的结构,仅仅是与本申请方案相关的部分结构的框图,并不构成对本申请方案所应用于其上的计算机设备的限定,具体的计算机设备可以包括比图中所示更多或更少的部件,或者组合某些部件,或者具有不同的部件布置。
在一个实施例中,提供了一种计算机设备,包括存储器和处理器,存储器中存储有计算机程序,该处理器执行计算机程序时实现以下步骤:
获取待处理图像的初始分辨率图像;
提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;
按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器;
通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到所述待处理图像的超分辨率图像。
在一个实施例中,处理器执行计算机程序时还实现以下步骤:通过残差网络提取所述待处理图像的第一图像特征信息;对所述第一图像特征信息进行像素重组,得到所述第一图像特征信息对应的第二图像特征信息,作为所述待处理图像的图像特征信息;并且所述 第二图像特征信息对应的图像分辨率高于所述第一图像特征信息对应的图像分辨率。
在一个实施例中,处理器执行计算机程序时还实现以下步骤:将所述待处理图像分为多个大小相同的图像分块;以及通过所述残差网络中包含的多个残差分支,分别提取多个所述图像分块的特征信息,作为所述待处理图像的第一图像特征信息。
在一个实施例中,处理器执行计算机程序时还实现以下步骤:获取所述待处理图像;以及对所述待处理图像进行线性插值处理,将线性插值处理后的图像作为所述初始分辨率图像。
在一个实施例中,处理器执行计算机程序时还实现以下步骤:对所述图像特征信息进行卷积处理,得到所述待处理图像中各个像素位置对应的组合系数;所述组合系数包括相应像素位置对应的多个滤波参数。
在一个实施例中,处理器执行计算机程序时还实现以下步骤:从预定义的滤波器参数信息表中,确定出多个与所述滤波参数对应的预设滤波器;以及将多个与所述滤波参数对应的预设滤波器进行线性加权,得到所述待处理图像中各个所述像素位置对应的目标滤波器。
在一个实施例中,处理器执行计算机程序时还实现以下步骤:通过下述方式计算得到目标滤波器:
其中,所述F
i为单个所述目标滤波器;D为所述预定义的滤波器参数信息表;所述L为所述预定义的滤波器参数信息表中所述预设滤波器的数量;所述Φ为所述滤波参数。
在一个实施例中,提供了一种计算机可读存储介质,其上存储有计算机程序,计算机程序被处理器执行时实现以下步骤:
获取待处理图像的初始分辨率图像;
提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;
按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器;以及
通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波 处理,得到所述待处理图像的超分辨率图像。
在一个实施例中,计算机程序被处理器执行时还实现以下步骤:通过残差网络提取所述待处理图像的第一图像特征信息;对所述第一图像特征信息进行像素重组,得到所述第一图像特征信息对应的第二图像特征信息,作为所述待处理图像的图像特征信息;并且所述第二图像特征信息对应的图像分辨率高于所述第一图像特征信息对应的图像分辨率。
在一个实施例中,计算机程序被处理器执行时还实现以下步骤:将所述待处理图像分为多个大小相同的图像分块;以及通过所述残差网络中包含的多个残差分支,分别提取多个所述图像分块的特征信息,作为所述待处理图像的第一图像特征信息。
在一个实施例中,计算机程序被处理器执行时还实现以下步骤:获取所述待处理图像;以及对所述待处理图像进行线性插值处理,将线性插值处理后的图像作为所述初始分辨率图像。
在一个实施例中,计算机程序被处理器执行时还实现以下步骤:对所述图像特征信息进行卷积处理,得到所述待处理图像中各个像素位置对应的组合系数;所述组合系数包括相应像素位置对应的多个滤波参数。
在一个实施例中,计算机程序被处理器执行时还实现以下步骤:从预定义的滤波器参数信息表中,确定出多个与所述滤波参数对应的预设滤波器;以及将多个与所述滤波参数对应的预设滤波器进行线性加权,得到所述待处理图像中各个所述像素位置对应的目标滤波器。
在一个实施例中,计算机程序被处理器执行时还实现以下步骤:通过下述方式计算得到目标滤波器:
其中,所述F
i为单个所述目标滤波器;D为所述预定义的滤波器参数信息表;所述L为所述预定义的滤波器参数信息表中所述预设滤波器的数量;所述Φ为所述滤波参数。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关的硬件来完成,上述的计算机程序可存储于一非易失性计算机可读取存储介质中,该计算机程序在执行时,可包括如上述各方法的实施例的流程。其中,本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用,均 可包括非易失性和易失性存储器中的至少一种。非易失性存储器可包括只读存储器(Read-Only Memory,ROM)、磁带、软盘、闪存或光存储器等。易失性存储器可包括随机存取存储器(Random Access Memory,RAM)或外部高速缓冲存储器。作为说明而非局限,RAM可以是多种形式,比如静态随机存取存储器(Static Random Access Memory,SRAM)或动态随机存取存储器(Dynamic Random Access Memory,DRAM)等。
以上实施例的各技术特征可以进行任意的组合,为使描述简洁,未对上述实施例中的各个技术特征所有可能的组合都进行描述,然而,只要这些技术特征的组合不存在矛盾,都应当认为是本说明书记载的范围。
以上实施例仅表达了本申请的几种实施方式,其描述较为具体和详细,但并不能因此而理解为对发明专利范围的限制。应当指出的是,对于本领域的普通技术人员来说,在不脱离本申请构思的前提下,还可以做出若干变形和改进,这些都属于本申请的保护范围。因此,本申请专利的保护范围应以所附权利要求为准。
Claims (15)
- 一种超分辨率图像重建方法,包括:获取待处理图像的初始分辨率图像;提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到与所述待处理图像中各个所述像素位置对应的目标滤波器;以及通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到所述待处理图像的超分辨率图像。
- 根据权利要求1所述的方法,其中所述提取所述待处理图像的图像特征信息,包括:通过残差网络提取所述待处理图像的第一图像特征信息;以及对所述第一图像特征信息进行像素重组,得到所述第一图像特征信息对应的第二图像特征信息,作为所述待处理图像的图像特征信息;所述第二图像特征信息对应的图像分辨率高于所述第一图像特征信息对应的图像分辨率。
- 根据权利要求2所述的方法,其中所述通过残差网络提取所述待处理图像的第一图像特征信息,包括:将所述待处理图像分为多个大小相同的图像分块;以及通过所述残差网络中包含的多个残差分支,分别提取多个所述图像分块的特征信息,作为所述待处理图像的第一图像特征信息。
- 根据权利要求1所述的方法,其中所述获取待处理图像的初始分辨率图像,包括:获取所述待处理图像;以及对所述待处理图像进行线性插值处理,将线性插值处理后的图像作为所述初始分辨率图像。
- 根据权利要求1所述的方法,其中所述根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数,包括:对所述图像特征信息进行卷积处理,得到所述待处理图像中各个像素位置对应的组合 系数;所述组合系数包括相应像素位置对应的多个滤波参数。
- 根据权利要求1至5任一项所述的方法,其中所述按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器,包括:从预定义的滤波器参数信息表中,确定出多个与所述滤波参数对应的预设滤波器;以及将多个与所述滤波参数对应的预设滤波器进行线性加权,得到所述待处理图像中各个所述像素位置对应的目标滤波器。
- 一种对图像进行去噪和去JPEG压缩的方法,包括:获取待处理图像的初始分辨率图像;提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到与所述待处理图像中各个所述像素位置对应的目标滤波器;以及通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到所述待处理图像的超分辨率图像。
- 根据权利要求8所述的方法,其中所述提取所述待处理图像的图像特征信息,包括:将所述待处理图像分为多个大小相同的图像分块;以及通过残差网络中包含的多个残差分支,分别提取多个所述图像分块的特征信息,作为所述待处理图像的所述图像特征信息。
- 根据权利要求8所述的方法,其中所述根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数,包括:对所述图像特征信息进行卷积处理,得到所述待处理图像中各个像素位置对应的组合系数;所述组合系数包括相应像素位置对应的多个滤波参数。
- 根据权利要求8至10任一项所述的方法,其中所述按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器,包括:从预定义的滤波器参数信息表中,确定出多个与所述滤波参数对应的预设滤波器;以及将多个与所述滤波参数对应的预设滤波器进行线性加权,得到所述待处理图像中各个所述像素位置对应的目标滤波器。
- 一种超分辨率图像重建装置,包括:图像获取模块,用于获取待处理图像的初始分辨率图像;特征提取模块,用于提取所述待处理图像的图像特征信息,根据所述图像特征信息确定出与所述待处理图像中各个像素位置对应的多个滤波参数;滤波器组合模块,用于按照各个所述像素位置对应的多个滤波参数,分别将对应的各个预设滤波器进行组合,得到所述待处理图像中各个所述像素位置对应的目标滤波器;以及图像重建模块,用于通过所述目标滤波器对所述初始分辨率图像中对应的像素位置的像素信息进行滤波处理,得到所述待处理图像的超分辨率图像。
- 一种计算机设备,包括存储器和处理器,所述存储器存储有计算机程序,所述处理器执行所述计算机程序时实现权利要求1至7中任一项所述的方法的步骤。
- 一种计算机可读存储介质,其上存储有计算机程序,所述计算机程序被处理器执行时实现权利要求1至7中任一项所述的方法的步骤。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2022564634A JP7482253B2 (ja) | 2020-08-25 | 2021-07-16 | 画像処理方法、装置、コンピュータ機器及び記憶媒体 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010862957.2A CN111951167B (zh) | 2020-08-25 | 2020-08-25 | 超分辨率图像重建方法、装置、计算机设备和存储介质 |
CN202010862957.2 | 2020-08-25 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022042124A1 true WO2022042124A1 (zh) | 2022-03-03 |
Family
ID=73366367
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/106647 WO2022042124A1 (zh) | 2020-08-25 | 2021-07-16 | 超分辨率图像重建方法、装置、计算机设备和存储介质 |
Country Status (3)
Country | Link |
---|---|
JP (1) | JP7482253B2 (zh) |
CN (1) | CN111951167B (zh) |
WO (1) | WO2022042124A1 (zh) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116503933A (zh) * | 2023-05-24 | 2023-07-28 | 北京万里红科技有限公司 | 一种眼周特征提取方法、装置、电子设备及存储介质 |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111951167B (zh) * | 2020-08-25 | 2021-05-18 | 深圳思谋信息科技有限公司 | 超分辨率图像重建方法、装置、计算机设备和存储介质 |
CN113034368A (zh) * | 2021-04-01 | 2021-06-25 | 深圳思谋信息科技有限公司 | 图像超分辨率模型训练方法、装置、计算机设备和介质 |
CN113674153A (zh) * | 2021-08-10 | 2021-11-19 | Oppo广东移动通信有限公司 | 图像处理芯片、电子设备、图像处理方法和存储介质 |
CN113808020A (zh) * | 2021-09-18 | 2021-12-17 | 北京字节跳动网络技术有限公司 | 图像处理方法及设备 |
CN115564653A (zh) * | 2022-09-30 | 2023-01-03 | 江苏济远医疗科技有限公司 | 一种多因子融合的图像超分辨率方法 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105844590A (zh) * | 2016-03-23 | 2016-08-10 | 武汉理工大学 | 基于稀疏表示的图像超分辨率重建方法及*** |
CN107194879A (zh) * | 2017-06-26 | 2017-09-22 | 司马大大(北京)智能***有限公司 | 超分辨率重建方法、装置及电子设备 |
CN108776954A (zh) * | 2018-06-26 | 2018-11-09 | 北京字节跳动网络技术有限公司 | 用于生成图像的方法和装置 |
CN110838085A (zh) * | 2018-08-15 | 2020-02-25 | Tcl集团股份有限公司 | 一种图像的超分辨率重建方法、装置及电子设备 |
CN110866870A (zh) * | 2019-10-29 | 2020-03-06 | 中山大学 | 一种医学图像任意倍数放大的超分辨处理方法 |
CN111951167A (zh) * | 2020-08-25 | 2020-11-17 | 深圳思谋信息科技有限公司 | 超分辨率图像重建方法、装置、计算机设备和存储介质 |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015166976A (ja) | 2014-03-04 | 2015-09-24 | 学校法人慶應義塾 | 画像処理装置、画像処理方法及びプログラム |
CN104574336B (zh) * | 2015-01-19 | 2017-08-01 | 上海交通大学 | 基于自适应亚模字典选择的超分辨率图像重建*** |
US9633274B2 (en) * | 2015-09-15 | 2017-04-25 | Mitsubishi Electric Research Laboratories, Inc. | Method and system for denoising images using deep Gaussian conditional random field network |
CN105389778B (zh) * | 2015-11-04 | 2018-10-12 | 北京大学深圳研究生院 | 一种基于字典匹配的图像超分辨率重建方法及装置 |
US10339633B2 (en) * | 2015-11-04 | 2019-07-02 | Peking University Shenzhen Graduate School | Method and device for super-resolution image reconstruction based on dictionary matching |
CN107644398B (zh) | 2017-09-25 | 2021-01-26 | 上海兆芯集成电路有限公司 | 图像插补方法及其相关图像插补装置 |
CN110070486A (zh) * | 2018-01-24 | 2019-07-30 | 杭州海康威视数字技术股份有限公司 | 一种图像处理方法、装置及电子设备 |
CN108830791B (zh) * | 2018-05-09 | 2022-05-06 | 浙江师范大学 | 基于自身样例和稀疏表示的图像超分辨率方法 |
-
2020
- 2020-08-25 CN CN202010862957.2A patent/CN111951167B/zh active Active
-
2021
- 2021-07-16 JP JP2022564634A patent/JP7482253B2/ja active Active
- 2021-07-16 WO PCT/CN2021/106647 patent/WO2022042124A1/zh active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105844590A (zh) * | 2016-03-23 | 2016-08-10 | 武汉理工大学 | 基于稀疏表示的图像超分辨率重建方法及*** |
CN107194879A (zh) * | 2017-06-26 | 2017-09-22 | 司马大大(北京)智能***有限公司 | 超分辨率重建方法、装置及电子设备 |
CN108776954A (zh) * | 2018-06-26 | 2018-11-09 | 北京字节跳动网络技术有限公司 | 用于生成图像的方法和装置 |
CN110838085A (zh) * | 2018-08-15 | 2020-02-25 | Tcl集团股份有限公司 | 一种图像的超分辨率重建方法、装置及电子设备 |
CN110866870A (zh) * | 2019-10-29 | 2020-03-06 | 中山大学 | 一种医学图像任意倍数放大的超分辨处理方法 |
CN111951167A (zh) * | 2020-08-25 | 2020-11-17 | 深圳思谋信息科技有限公司 | 超分辨率图像重建方法、装置、计算机设备和存储介质 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116503933A (zh) * | 2023-05-24 | 2023-07-28 | 北京万里红科技有限公司 | 一种眼周特征提取方法、装置、电子设备及存储介质 |
CN116503933B (zh) * | 2023-05-24 | 2023-12-12 | 北京万里红科技有限公司 | 一种眼周特征提取方法、装置、电子设备及存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN111951167B (zh) | 2021-05-18 |
JP2023523047A (ja) | 2023-06-01 |
CN111951167A (zh) | 2020-11-17 |
JP7482253B2 (ja) | 2024-05-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2022042124A1 (zh) | 超分辨率图像重建方法、装置、计算机设备和存储介质 | |
Hui et al. | Lightweight image super-resolution with information multi-distillation network | |
Wang et al. | Real-esrgan: Training real-world blind super-resolution with pure synthetic data | |
Lee et al. | Local texture estimator for implicit representation function | |
Zhang et al. | Learning deep CNN denoiser prior for image restoration | |
Sun et al. | Shufflemixer: An efficient convnet for image super-resolution | |
CN111369440B (zh) | 模型训练、图像超分辨处理方法、装置、终端及存储介质 | |
CN111105352A (zh) | 超分辨率图像重构方法、***、计算机设备及存储介质 | |
WO2020215644A1 (zh) | 视频图像处理方法及装置 | |
CN110533594B (zh) | 模型训练方法、图像重建方法、存储介质及相关设备 | |
CN114820341A (zh) | 一种基于增强Transformer的图像盲去噪方法及*** | |
CN113129212B (zh) | 图像超分辨率重建方法、装置、终端设备及存储介质 | |
CN107680043B (zh) | 基于图模型的单幅图像超分辨率输出方法 | |
RU2697928C1 (ru) | Способ сверхразрешения изображения, имитирующего повышение детализации на основе оптической системы, выполняемый на мобильном устройстве, обладающем ограниченными ресурсами, и мобильное устройство, его реализующее | |
CN115358932B (zh) | 一种多尺度特征融合的人脸超分辨率重构方法及*** | |
CN112907448A (zh) | 一种任意比率图像超分辨率方法、***、设备及存储介质 | |
CN110428382A (zh) | 一种用于移动终端的高效视频增强方法、装置和存储介质 | |
CN114418853A (zh) | 基于相似图像检索的图像超分辨率优化方法、介质及设备 | |
WO2023184913A1 (zh) | 模型训练方法、超分辨率重建方法、装置、设备及介质 | |
Conde et al. | Lens-to-lens bokeh effect transformation. NTIRE 2023 challenge report | |
Li et al. | Lightweight single image super-resolution with dense connection distillation network | |
CN114913094A (zh) | 图像修复方法、装置、计算机设备、存储介质和程序产品 | |
Xie et al. | Adaptive densely connected single image super-resolution | |
CN111951171A (zh) | Hdr图像生成方法、装置、可读存储介质及终端设备 | |
CN114494006A (zh) | 图像重建模型的训练方法、装置、电子设备及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21859964 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2022564634 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21859964 Country of ref document: EP Kind code of ref document: A1 |