CN116385267A

CN116385267A - Image processing method, apparatus, program product, computer device, and storage medium

Info

Publication number: CN116385267A
Application number: CN202310362728.8A
Authority: CN
Inventors: 贺思颖
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-03-29
Filing date: 2023-03-29
Publication date: 2023-07-04

Abstract

The application discloses an image processing method, an apparatus, a program product, a computer device, and a storage medium, the method comprising: performing first feature extraction processing on the image to obtain first image features of the image, wherein the first image features comprise image features for deblocking effects; performing second feature extraction processing on the image to obtain second image features of the image, wherein the second image features comprise image features for strengthening image details; fusing the first image features and the second image features to obtain fused image features; and performing super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than that of the image. By adopting the method and the device, the blocking effect can be removed, super-resolution details can be reserved, and the image quality can be improved.

Description

Image processing method, apparatus, program product, computer device, and storage medium

Technical Field

The present application relates to the field of computer technology, and in particular, to an image processing method, apparatus, program product, computer device, and storage medium.

Background

With the continuous development of computer and artificial intelligence (Artificial Intelligence, AI) technology, image processing has become an important direction in the fields of computer science and artificial intelligence. The deblocking effect and the super-resolution are two most critical technologies in the image processing and coding technology, the main effect of the deblocking effect is to remove the blocking effect brought by the video coding process through the AI, and the super-resolution technology is mainly to upsample the video with low resolution through the AI so as to upsample the video with low resolution transmission, thereby achieving the purposes of saving bandwidth and guaranteeing the subjective quality of the video. However, the deblocking effect removes details of the image, and the super-resolution enlarges the image to increase details and brings additional noise, so that the deblocking effect and the super-resolution technology are generally difficult to reconcile when applied in combination.

Therefore, how to eliminate the blocking effect and retain the super-resolution details, so as to improve the image quality, is a problem to be solved urgently.

Disclosure of Invention

The application provides an image processing method, an image processing device, a program product, computer equipment and a storage medium, which can remove blocking effect, retain super-resolution details and improve image quality.

In one aspect, the present application provides an image processing method, including: performing first feature extraction processing on the image to obtain first image features of the image, wherein the first image features comprise image features for deblocking effects; performing second feature extraction processing on the image to obtain second image features of the image, wherein the second image features comprise image features for strengthening image details; fusing the first image features and the second image features to obtain fused image features; and performing super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than that of the image.

Optionally, fusing the first image feature and the second image feature to obtain a fused image feature, including: performing dimension reduction processing on the first image feature to obtain a first offset feature; correcting the second image characteristic based on the first offset characteristic to obtain a corrected second image characteristic; performing dimension reduction processing on the second image feature to obtain a second offset feature; correcting the first image characteristic based on the second offset characteristic to obtain a corrected first image characteristic; and fusing the corrected first image characteristic with the corrected second image characteristic to obtain a fused image characteristic.

Optionally, the processed image is obtained by calling an image processing model to process, and a training mode of the image processing model includes: acquiring a sample image and a reference image corresponding to the sample image, wherein the resolution of the reference image is higher than that of the sample image; invoking an initial image processing model to extract first sample image features of the sample image, wherein the first sample image features include image features for deblocking effects; invoking the initial image processing model to extract second sample image features of the sample image, wherein the second sample image features include image features for enhancing image details; fusing the first sample image features and the second sample image features to obtain fused sample image features; performing super-resolution image restoration processing on the fused sample image characteristics to obtain an output image, wherein the resolution of the output image is matched with the resolution of a reference image; and training the initial image processing model by taking the difference between the reduced output image and the reference image as a training target to obtain an image processing model.

Optionally, acquiring the sample image includes: and performing first resolution reduction processing on the reference image to obtain a sample image.

Further, the image processing method further includes: performing second resolution reduction processing on the reference image to obtain an intermediate reference image, wherein the intermediate reference image is matched with the resolution of the sample image, and the definition of the intermediate reference image is higher than that of the sample image; performing image restoration processing on the first sample image characteristics to obtain an intermediate image; training the initial image processing model by taking the difference between the reduced output image and the reference image as a training target to obtain an image processing model, wherein the method comprises the following steps: and training the initial image processing model by taking the difference between the reduced output image and the reference image and the difference between the intermediate image and the intermediate reference image as training targets to obtain an image processing model.

Optionally, acquiring the sample image and the reference image includes: performing super-resolution processing on the source reference image to obtain an amplified image; deblocking the amplified image to obtain a deblocked image; and performing resolution reduction processing on the deblocked image to obtain a reference image.

An aspect of the present application provides an image processing apparatus including: the image processing device comprises a feature extraction module, a processing module and a processing module, wherein the feature extraction module is used for carrying out first feature extraction processing on an image to obtain first image features of the image, and the first image features comprise image features for deblocking effects; the feature extraction module is further used for carrying out second feature extraction processing on the image to obtain second image features of the image, wherein the second image features comprise image features for strengthening details of the image; the fusion module is also used for fusing the first image feature and the second image feature to obtain a fused image feature; the processing module is further used for carrying out super-resolution image restoration processing on the fused image characteristics to obtain a processed image, wherein the resolution of the processed image is higher than that of the image

Optionally, the fusing module is configured to fuse the first image feature and the second image feature to obtain a fused image feature, and includes: performing dimension reduction processing on the first image feature to obtain a first offset feature; correcting the second image characteristic based on the first offset characteristic to obtain a corrected second image characteristic; performing dimension reduction processing on the second image feature to obtain a second offset feature; correcting the first image characteristic based on the second offset characteristic to obtain a corrected first image characteristic; and fusing the corrected first image characteristic with the corrected second image characteristic to obtain a fused image characteristic.

Optionally, the processed image is obtained by the processing module calling an image processing model to process, and the training mode of the image processing model includes: acquiring a sample image and a reference image corresponding to the sample image, wherein the resolution of the reference image is higher than that of the sample image; invoking an initial image processing model to extract first sample image features of the sample image, wherein the first sample image features include image features for deblocking effects; invoking the initial image processing model to extract second sample image features of the sample image, wherein the second sample image features include image features for enhancing image details; fusing the first sample image features and the second sample image features to obtain fused sample image features; performing super-resolution image restoration processing on the fused sample image characteristics to obtain an output image, wherein the resolution of the output image is matched with the resolution of a reference image; and training the initial image processing model by taking the difference between the reduced output image and the reference image as a training target to obtain an image processing model.

Optionally, the processing module acquires a sample image, including: and performing first resolution reduction processing on the reference image to obtain a sample image.

Optionally, the processing module is further configured to: performing second resolution reduction processing on the reference image to obtain an intermediate reference image, wherein the intermediate reference image is matched with the resolution of the sample image, and the definition of the intermediate reference image is higher than that of the sample image; performing image restoration processing on the first sample image characteristics to obtain an intermediate image; training the initial image processing model by taking the difference between the reduced output image and the reference image as a training target to obtain an image processing model, wherein the method comprises the following steps: and training the initial image processing model by taking the difference between the reduced output image and the reference image and the difference between the intermediate image and the intermediate reference image as training targets to obtain an image processing model.

Optionally, the processing module acquires a sample image and a reference image, including: performing super-resolution processing on the source reference image to obtain an amplified image; deblocking the amplified image to obtain a deblocked image; and performing resolution reduction processing on the deblocked image to obtain a reference image.

In one aspect, the present application provides a computer device including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform a method in one aspect of the present application.

In one aspect, the present application provides a computer readable storage medium storing a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method of one of the aspects described above.

According to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the methods provided in the various alternatives of the above aspect and the like.

The method comprises the steps of performing first feature extraction processing on an image to obtain first image features of the image, wherein the first image features comprise image features for deblocking effects; performing second feature extraction processing on the image to obtain second image features of the image, wherein the second image features comprise image features for strengthening image details; fusing the first image features and the second image features to obtain fused image features; and performing super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than that of the image. Therefore, in the method provided by the application, the extracted first image feature is beneficial to removing the blocking effect of the image, the extracted second image feature can strengthen the image detail and is beneficial to super-resolution processing, and after the first image feature and the second image feature are fused, super-resolution reduction processing is carried out on the fused image feature, so that the blocking effect can be removed, meanwhile, the super-resolution detail is reserved, and the image quality is improved.

Drawings

In order to more clearly illustrate the technical solutions of the present application or the prior art, the following description will briefly introduce the drawings that are required to be used in the embodiments or the prior art descriptions, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of an architecture of an image processing system provided herein;

FIG. 2 is a flow chart of a training method of an image processing model provided in the present application;

FIG. 3 is a flow chart of a method of training an image processing model provided herein;

FIG. 4 is a block diagram of an example of a method of acquiring a reference image provided herein;

FIG. 5 is a block diagram of an example of acquiring a difference between a sample image and an intermediate reference image provided herein;

FIG. 6 is an example block diagram of training of an image processing model provided herein;

FIG. 7 is a flow chart of an image processing method provided herein;

FIG. 8 is a flowchart of an example of another image processing method provided herein;

Fig. 9 is a schematic structural view of an image processing apparatus provided in the present application;

fig. 10 is a schematic structural diagram of a computer device provided in the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

With research and progress of artificial intelligence technology, research and application of artificial intelligence technology are developed in various fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned, automatic driving, unmanned aerial vehicles, robots, smart medical treatment, smart customer service, etc., and with development of technology, artificial intelligence technology is applied in more fields and is of increasing importance.

The present application relates to artificial intelligence related technology. Among these, artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.

The artificial intelligence technology is a comprehensive subject, and relates to the technology with wide fields, namely the technology with a hardware level and the technology with a software level. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

The present application relates generally to machine learning in artificial intelligence. Machine Learning (ML) is a multi-domain interdisciplinary, and relates to multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc., and it is specially studied how a computer simulates or implements Learning behavior of a human being to obtain new knowledge or skill, and reorganizes the existing knowledge structure to continuously improve its own performance. Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, confidence networks, reinforcement learning, transfer learning, induction learning, teaching learning, and the like.

The machine learning referred in the present application mainly refers to that a neural network for image processing is obtained by machine learning, so that the image processing is performed through the neural network, and the super-resolution details are reserved while the blocking effect is removed, so that the image quality is improved. See for details the description of the embodiments below.

The image processing method provided by the embodiment of the application can be applied to computer equipment such as a terminal or a server. After the computer equipment acquires a certain image, the image can be subjected to deblocking effect and super-resolution processing so as to realize that the blocking effect is removed and the super-resolution details are reserved at the same time, thereby acquiring the image with better image quality.

Specifically, the computer device may perform a first feature extraction process on the image to obtain a first image feature of the image, where the first image feature includes an image feature for deblocking. In addition, the computer device may further perform a second feature extraction process on the image to obtain a second image feature of the image, where the second image feature includes an image feature for enhancing details of the image. Then, the computer device may fuse the first image feature and the second image feature to obtain a fused image feature, and perform super-resolution image restoration processing on the fused image feature to obtain a processed image, where the resolution of the processed image is higher than that of the image.

In other words, after the computer equipment receives the low-resolution image, the computer equipment can perform deblocking effect processing and super-resolution processing on the low-resolution image to obtain the high-resolution image, so that the bandwidth is saved, and the subjective quality of the image is ensured.

Optionally, the image may be obtained from a local memory of the computer device, or may be obtained from the cloud, or may be obtained from another device.

Referring to fig. 1, fig. 1 is a schematic architecture diagram of an image processing system provided in the present application. Taking the architecture diagram of the image processing system shown in fig. 1 as an example, the image processing system may comprise a server 100 and a terminal device cluster, which may comprise one or more terminal devices, the number of terminal devices will not be limited here. As shown in fig. 1, the plurality of terminal devices may specifically include a terminal device 101, a terminal device 102, terminal devices 103, …, and a terminal device 104; as shown in fig. 1, the terminal device 101, the terminal device 102, the terminal devices 103, …, and the terminal device 104 may be connected to the server 100 through a network, so that each terminal device may interact with the server 100 through the network connection.

The server 100 shown in fig. 1 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, basic cloud computing services such as big data and artificial intelligence platforms, and the like. The terminal device may be: intelligent terminals such as smart phones, tablet computers, notebook computers, desktop computers, intelligent televisions and the like.

A specific description of the embodiment of the present application will be made below taking communication between the terminal device 101 and the server 100 as an example. If the computer device is a terminal device 101, the terminal device 101 may obtain an image from the server 100, for example, a user may download an image from the server 100 using the terminal device 101. If the computer device is a server 100, the server 100 may obtain an image from the terminal device 101, e.g., a user may upload an image to the server 100 using the terminal device 101.

In general, super resolution methods are classified into three types: interpolation-based methods, reconstruction-based methods, and learning-based methods (i.e., deep learning methods). Deep learning-based methods can be classified into an SR method based on a convolutional neural network (Convolutional Neural Networks, CNN), an SR method based on a residual network (Residual Networks, resNet), and an SR method based on a generated countermeasure network (Generative Adversarial Networks, GAN).

Alternatively, the computer device may perform the deblocking process and the super-resolution process on the image by the AI deep learning-based method, for example, the computer device may call an image processing model based on a convolutional neural network to perform the deblocking process and the super-resolution process on the image. The computer device may acquire the sample image and the reference image in advance, and then the computer device may train the image processing model based on the sample image and the reference image and process the received low resolution image using the trained image processing model.

Referring to fig. 2, fig. 2 is a flow chart of a training method of an image processing model provided in the present application. As shown in fig. 2, the computer device acquires a sample image and a reference image, the reference image corresponds to the sample image, and the resolution of the reference image is greater than the resolution of the sample image. These sample images and reference images constitute a training set 201 of image processing models. The image processing model may extract first sample image features of the sample image including image features for deblocking effects and extract second sample image features of the sample image including image features for enhancing image details. After the first sample image features and the second sample image features are fused, super-resolution image restoration processing is performed on the fused sample image features to obtain a high-resolution image, and a trained image processing model 202 is obtained based on the obtained high-resolution image and a reference image, wherein the trained image processing model 202 is used for executing an image processing task.

Furthermore, the computer device can process the image by using the trained image processing model 202, and finally obtain a processed image 203, so that the processed image 203 can be ensured to remove the blocking effect, retain the super-resolution details and improve the image quality.

Referring to fig. 3, fig. 3 is a flowchart of a training method of an image processing model provided in the present application. The execution subject in the embodiment of the present application may be the above-described computer device. The computer device may invoke the initial image processing model and train it to obtain the image processing model provided by the present application. As illustrated in fig. 3, the training method of the image processing model may include:

in step S301, a sample image and a reference image corresponding to the sample image are acquired, wherein the resolution of the reference image is higher than the resolution of the sample image.

The computer device may obtain a training set of image processing models, which may include a sample image and a reference image corresponding to the sample image. In particular, the sample image may be a low resolution image and the reference image may be a high resolution image corresponding to the sample image.

In some exemplary scenarios, the image quality of the acquired high resolution images may be less sharp when the computer device trains the image processing model, and the computer device may pre-process these less sharp high resolution images, i.e., the source reference image, to obtain a sharp reference image to facilitate training the initial image processing model.

Referring to fig. 4 together, fig. 4 is a block diagram illustrating an example of a method for acquiring a reference image provided in the present application. As shown in fig. 4, the computer device acquiring the reference image corresponding to the sample image may include: performing super-resolution processing on the source reference image to obtain an amplified image; performing deblocking processing on the amplified image to obtain a deblocking image; and performing resolution reduction processing on the image subjected to the deblocking effect to obtain a reference image.

Alternatively, the resolution reduction algorithm may be a Lanczos resolution reduction algorithm.

Further, the computer device performs a resolution reduction process on the image subjected to the deblocking effect to obtain a reference image, and may further include: and carrying out sharpening treatment and denoising treatment on the image subjected to the deblocking effect, and then carrying out resolution reduction treatment to obtain a reference image.

For example, as shown in fig. 4, a computer device may super-resolution process 402 a less sharp high resolution source reference image 401 to increase image detail; then, performing a deblocking process 403 to remove blocking in the image; optionally, since some image details are removed smoothly by the deblocking algorithm, the computer device may further perform sharpening 404 to enhance image edge details, and then perform noise reduction 405 to remove image noise; finally, lanczos resolution reduction processing 406 is performed using, for example, lanczos resolution reduction algorithm, thereby obtaining a high resolution reference image 407 with clear image quality.

It can be found that, for some images with high resolution but not necessarily clear image quality, the computer equipment directly uses the images as reference images in the training process of the image processing model, and does not necessarily guide the image processing model to learn better parameters well. Further, the computer equipment uses a sharpening algorithm to strengthen edge details of the image subjected to the super-resolution algorithm and the deblocking algorithm, then uses a noise reduction algorithm to remove image noise, and finally processes the image through the resolution algorithm, so that a reference image with better image quality can be obtained, and the image processing model is more beneficial to guiding the image processing model to learn better model parameters.

In some exemplary scenarios, the high resolution image acquired by the computer device may be a clear image when training the image processing model, and the computer device may train the initial image processing model based on the clear high resolution image.

Alternatively, the computer device may take the high-resolution image with the sharp image quality as a reference image, and acquire the sample image based on the reference image. The computer device acquiring the sample image may include: and performing first resolution reduction processing on the reference image to obtain a sample image.

Specifically, the computer device may directly obtain a sample image with a lower resolution corresponding to the reference image by performing the first resolution reduction processing on the high-resolution reference image with clear image quality. In this way, the operation of finding the sample image-reference image pair for training the image processing model can be simplified, and a lower-resolution sample image is obtained from the high-resolution reference image with sharp image quality using the resolution-reduction algorithm, so that the training data for training the image processing model can retain only the high-resolution image with sharp image quality, whereby the storage space requirement for training the training set of the image processing model can be reduced. In general, the algorithm used in the first resolution reduction process may be a low-complexity and high-computational-performance algorithm, for example, as shown in fig. 5, the computer device may use a Bicubic resolution reduction algorithm to perform Bicubic resolution reduction 502 on the reference image 501, so as to obtain a sample image 503. The first resolution reduction algorithm may also be any other resolution reduction algorithm of lower complexity and higher computational performance.

It can be found that the reference image is subjected to the resolution reduction process, so that a matched sample image can be directly obtained, the operation of searching a sample image-reference image pair for training the image processing model can be simplified, the storage space requirement of a training set for training the image processing model can be reduced, and the sample image of the training image processing model can be obtained more efficiently by using a resolution reduction algorithm with lower complexity and higher calculation performance, so that the training efficiency of the image processing model is improved.

Step S302, an initial image processing model is invoked to extract a first sample image feature of a sample image, wherein the first sample image feature comprises an image feature for deblocking.

The initial image processing model may include a first image feature extraction module, and the computer device may invoke the initial image processing model to perform a first feature extraction process on the sample image through the first image feature extraction module to obtain a first sample image feature of the sample image. For example, the first image feature extraction module may include a Convolution (Conv) plus parameter correction linear unit (Parametric Rectified Linear Unit, PReLU), that is, the computer device may extract the first sample image feature of the acquired sample image through a deblocking (AR) link based on the Convolution plus parameter correction linear unit in the initial image processing model.

In some exemplary scenarios, the high resolution image acquired by the computer device in training the image processing model may be a sharp quality image, and the training of the image processing model may further include: performing second resolution reduction processing on the reference image to obtain an intermediate reference image, wherein the intermediate reference image is matched with the resolution of the sample image, and the definition of the intermediate reference image is higher than that of the sample image; and performing image restoration processing on the first sample image characteristic to obtain an intermediate image.

In particular, to prevent the AR link of the image processing model from culling excessive image detail during deblocking, the computer device may provide image quality constraints to the AR link of the image processing model during training. The computer device may perform a second resolution reduction process on the high resolution reference image with the sharp image quality to obtain an intermediate reference image, the intermediate reference image matching the resolution of the sample image and having a sharpness higher than the sharpness of the sample image as a constraint on the extracted first sample image feature in the AR link.

Generally, as an image quality constraint, the intermediate reference image obtained by the second resolution reduction process has relatively high definition and good image quality, so that the trained image processing model can have clear output. Therefore, the algorithm adopted by the second resolution reduction process can be different from the algorithm adopted by the first resolution reduction process, and the second resolution reduction algorithm can obtain the image with clear image quality.

For example, as shown in fig. 5, the computer device may perform Lanczos resolution reduction 504 on the reference image 501 using Lanczos resolution reduction algorithm to obtain an intermediate reference image 505, and the second resolution reduction algorithm may be any other resolution reduction algorithm capable of obtaining an image with clear image quality.

Optionally, matching the resolution of the intermediate reference image with the sample image may include: the intermediate reference image is the same or substantially the same resolution as the sample image. For example, the resolution of the intermediate reference image differs from the resolution of the sample image by within a preset threshold range.

Further, the computer device may perform an image restoration process on the first sample image features extracted from the AR link of the image processing model, obtain an intermediate image, and provide image quality constraints to the AR link of the image processing model based on the intermediate reference image.

Optionally, providing image quality constraints to the AR link of the image processing model is based on controlling differences between the intermediate image and the intermediate reference image.

Alternatively, the difference between the intermediate image and the intermediate reference image may comprise a corresponding AR loss function between the intermediate image and the intermediate reference image. The AR loss function may comprise, for example, an L2 loss function, an entropy function having a defined form, or any other function capable of representing the difference between the intermediate image and the intermediate reference image.

For example, as shown in fig. 5, the AR link of the image processing model may extract a first sample image feature 506 from the sample image 503, the first sample image feature 506 comprising image features for deblocking. The AR link of the image processing model may perform image restoration processing on the first sample image feature 506 using, for example, 1 x 1 convolution to obtain the intermediate image 507. Further, the computer device may derive an AR L2 loss 508 based on the intermediate image 507 and the intermediate reference image 505.

Step S303, calling the initial image processing model to extract second sample image features of the sample image, wherein the second sample image features comprise image features for enhancing image details.

The initial image processing model may include a second image feature extraction module, and the computer device may invoke the initial image processing model to perform a second feature extraction process on the sample image through the second image feature extraction module to obtain a second sample image feature of the sample image. For example, the second image feature extraction module may include a convolution plus parameter correction linear unit, that is, the computer device may extract the second sample image feature of the acquired sample image through a Super Resolution (SR) link composed of the convolution plus parameter correction linear unit in the initial image processing model.

It is to be understood that the sequence of step S302 and step S303 is not limited herein. For example, the computer device may perform first feature extraction processing on the sample image to obtain first sample image features of the sample image, and then perform second feature extraction processing on the sample image to obtain second sample image features of the sample image; for another example, the computer device may perform the second feature extraction process on the sample image to obtain a second sample image feature of the sample image, and then perform the first feature extraction process on the sample image to obtain a first sample image feature of the sample image; for another example, the computer device may perform the first feature extraction process on the sample image to obtain a first sample image feature of the sample image, and perform the second feature extraction process on the sample image to obtain a second sample image feature of the sample image.

Specifically, the computer device may first extract first sample image features of the acquired sample image using the AR link of the initial image processing model, and then extract second sample image features of the acquired sample image using the SR link of the initial image processing model; or firstly, extracting the second sample image characteristics of the obtained sample image by utilizing the SR link of the initial image processing model, and then extracting the first sample image characteristics of the obtained sample image by utilizing the AR link of the initial image processing model; the AR link and the SR link of the initial image processing model can be utilized to simultaneously perform feature extraction processing on the acquired sample image, so as to respectively obtain a first sample image feature and a second sample image feature of the acquired sample image.

Step S304, fusing the first sample image features and the second sample image features to obtain fused sample image features.

The computer device may invoke the initial image processing model to fuse, e.g., splice, the first sample image features with the second sample image features to obtain fused image features.

Stitching may refer to stitching of channels based on image features. For example, the first sample image feature may be an L-channel image feature, the second sample image feature may be an M-channel image feature, and the computer device may stitch the first sample image feature and the second sample image feature based on the channels to obtain a fused image feature of the (l+m) channel.

Alternatively, the computer device may invoke an initial image processing model to connect or stitch the first sample image features obtained at the AR link with the second sample image features obtained at the SR link to obtain fused image features.

Step S305, performing super-resolution image restoration processing on the fused sample image features to obtain an output image, wherein the resolution of the output image is matched with the resolution of the reference image.

The computer equipment can call the initial image processing model, and the fused sample image features are subjected to super-resolution image restoration processing by utilizing a proper super-resolution algorithm to obtain an output image. For example, the computer device may invoke an initial image processing model to perform super-resolution image restoration processing on the fused image using a Pixel Shuffle function or any other suitable super-resolution algorithm function to obtain a high-resolution output image. The resolution of the output image matches the resolution of the reference image.

Optionally, matching the resolution of the output image with the resolution of the reference image may include: the resolution of the output image is the same or substantially the same as the resolution of the reference image. For example, the resolution of the output image differs from the resolution of the reference image by within a preset threshold range.

And step S306, training the initial image processing model by taking the difference between the reduced output image and the reference image as a training target to obtain an image processing model.

The computer device may train the initial image processing model with the reduced difference between the output image and the reference image as a training target, and may obtain a trained image processing model when the difference is reduced to within the target range.

Alternatively, the difference between the output image and the reference image may be a loss value calculated by a loss function. The loss function may comprise, for example, an L2 loss function, an entropy function having a defined form, or any other function capable of representing the difference between the output image and the reference image.

Optionally, the training of the initial image processing model by the computer device with the aim of reducing the difference between the output image and the reference image may include: the initial image processing model parameters are adjusted so that the difference between the output image and the reference image is within a preset threshold range. For example, the computer device may adjust the parameters of the convolution plus parameter correction linear elements in the invoked initial image processing model to reduce the difference between the output image and the reference image to within a preset threshold range.

Optionally, the computer device trains the initial image processing model with the reduced difference between the output image and the reference image as a training target to obtain the image processing model, and trains the initial image processing model with the reduced difference between the output image and the reference image and the reduced difference between the intermediate image and the intermediate reference image as a training target to obtain the image processing model.

Optionally, the computer device trains the initial image processing model with the reduced difference between the output image and the reference image and the intermediate reference image as training targets, resulting in an image processing model, which may include reducing a weighted sum of the reduced difference between the output image and the reference image and the intermediate reference image. For example, the computer device may adjust the parameters of the convolution plus parameter correction linear unit in the invoked image processing model to bring the weighted sum of the differences of the output image and the reference image and the differences of the intermediate image and the intermediate reference image within a preset threshold range.

In the embodiment of the application, the computer equipment calls the first sample image feature extracted by the initial image processing model to help remove the blocking effect of the sample image, the extracted second sample image feature can strengthen the image detail to help carry out super-resolution processing, and after the first sample image feature and the second sample image feature are fused, the super-resolution reduction processing is carried out on the fused image feature, so that the blocking effect can be removed, meanwhile, the super-resolution detail is reserved, and the image quality is improved. And it can be found that the intermediate reference image obtained by using the resolution-reducing algorithm with higher definition and better image quality can provide better constraint of image quality in the training process of the image processing model, and can avoid that the image is excessively smooth due to the image characteristics extracted by the AR link in the training process of the image processing model, and excessive image details are lost; and the difference between the output image and the reference image and the difference between the intermediate image and the intermediate reference image are taken as training targets, the initial image processing model is trained, the training of the image processing model can be more finely adjusted, and the image processing model with better performance can be obtained.

Referring to fig. 6, fig. 6 is an exemplary block diagram of training of an image processing model provided herein.

As shown in fig. 6, this example block diagram illustrates a training method of the image processing model.

Firstly, the computer equipment acquires a low-resolution input image with resolution of (h, w), wherein the low-resolution input image is a single-channel sample image, an AR link and an SR link of an image processing model respectively extract a first sample image characteristic of a 4 channel and a second sample image characteristic of the 4 channel by using a convolution+parameter rectification linear unit (Conv+PReLU), the first sample image characteristic comprises an image characteristic used for deblocking effect, and the second sample image characteristic comprises an image characteristic used for strengthening image details.

On the AR link, the image processing model may perform an image reduction process on the first sample image feature of the 4 channels using, for example, the method described with respect to fig. 5 to obtain an intermediate image, and may solve for the L2 loss along with the intermediate reference image obtained using, for example, the method described with respect to fig. 5 to obtain loss-AR-1.

Meanwhile, the image processing model may dimension down the 4-channel first sample image feature to the 1-channel first sample image offset feature using the G (β) function. For example, the G (β) function may be a 1×1 convolution, directly performing the dimension reduction process on the first sample image feature of the AR link. And correcting the 4-channel second sample image characteristics of the SR link by utilizing the 1-channel first sample image offset characteristics, for example, directly splicing to obtain 5-channel corrected second sample image characteristics. The G (β) function may be implemented in other ways. The method mainly utilizes G (beta) function to integrate the deblocking effect information of the AR link, and the deblocking effect information is fused into the SR link through splicing operation. It can be understood that, by using the G (β) function, the number of channels of the AR feature map to be fused is smaller than the number of channels of the feature map of the SR link to be fused, so that the main effect of the SR link after fusion is still focused on the processing of the super-resolution task, and the features of multiple channels are synthesized into 1 channel by using 1×1 convolution, so as to perform single-pixel level fusion, and thus, the potential noise that may be amplified in the process of adding details of the SR link can be offset.

On the SR link, the image processing model may dimension down the 4-channel second sample image feature to a 1-channel second sample image offset feature using an F (α) function. For example, the F (α) function may be a 3×3 convolution, which performs a dimension reduction process on the second sample image feature of the SR link. And correcting the 4-channel first sample image characteristics of the AR link, for example, directly splicing, by using the second sample image offset characteristics of the 1-channel, so as to obtain corrected first sample image characteristics of 5 channels. The F (α) function may be implemented in other ways. The method mainly utilizes F (alpha) function to integrate the detail information of the second sample image characteristic of the super-resolution task of the SR link, and the detail information is fused into the AR link through splicing operation. It can be understood that, by using the F (α) function, the number of channels of the SR feature map to be fused is smaller than the number of channels of the feature map of the AR link to be fused, so that the main effect of the fused AR link is still focused on the processing of the deblocking task, and the features of the multiple channels are synthesized into 1 channel by using 3×3 convolution, so that the relatively abundant details in the feature map of the SR link can be used to make up for the image details of the gradual loss of the AR link.

Further, the AR link and the SR link of the image processing model may repeat the above operations, except that the input of the above operations is a single channel, and the input of the AR link and the SR link in the subsequent repeated operations is 5 channels, and at the same time, loss-AR-2 to loss-AR-n may be obtained on the AR link, which is not described herein.

And secondly, the image processing model splices and fuses the 5-channel characteristics of the AR link and the 5-channel characteristics of the SR link into 10-channel characteristics, and the 4-channel characteristics are obtained through 3X 3 convolution so as to facilitate super-resolution processing by using a Pixel Shuffle function.

The image processing model then performs super-resolution reduction processing on the 4-channel features using the Pixel Shuffle function to obtain a high-resolution output image with a resolution of (2 h,2 w), and may solve for the L2 penalty along with the reference image obtained using, for example, the method described with respect to fig. 4 to obtain the penalty loss-sr.

Finally, the computer device may get a final L2 penalty. The final loss function is L _total ＝W _ar *(∑loss-ar-n)/n+W _sr * loss-sr, e.g. W _ar ＝0.5,W _sr =0.5 represents the weight of the AR link and the SR link being half of each, and represents the degree of fusion of the AR and the SR, and α and β may be used to adjust the degree of fusion of the AR and the SR, for example, α and β may indicate the number of channels after dimension reduction. The computer device may be based on the final loss L _total For purposes of, e.g., let L _total And adjusting parameters of the image processing model to be smaller than a preset threshold value, so that training of the image processing model is completed.

Based on the trained image processing model, the image processing method of the present application can be applied. Illustratively, the computer device acquires a low resolution input image of resolution (h, w) that can be processed using an image processing model trained by the above method.

On the AR link, the trained image processing model performs first feature extraction processing on the input image to obtain 4-channel first image features of the image, wherein the first image features comprise image features for deblocking effects.

And on the SR link, the trained image processing model carries out second feature extraction processing on the image to obtain 4-channel second image features of the image, wherein the second image features comprise image features for strengthening image details.

Meanwhile, the trained image processing model utilizes G (beta) function to reduce the dimension of the first image features of the 4 channels into first image offset features of 1 channel, and fusion correction processing, such as splicing, is carried out on the second image features based on the first image offset features, so that corrected second image features of 5 channels are obtained. The trained image processing model utilizes F (alpha) function to reduce the dimension of the second image features of the 4 channels into second image offset features of 1 channel, and fusion correction processing, such as splicing, is carried out on the first image features based on the second image offset features, so that corrected first image features of the 5 channels are obtained.

Further, the AR link and the SR link of the image processing model may repeat the above operations, except that the input of the above operations is a single channel, and the inputs of the AR link and the SR link in the subsequent repeated operations are 5 channels, which are not described herein.

And then, the trained image processing model is used for splicing and fusing the 5-channel characteristics of the AR link and the 5-channel characteristics of the SR link into 10-channel characteristics, and the 4-channel characteristics are obtained through 3X 3 convolution so as to facilitate super-resolution processing by using a Pixel Shuffle function.

Finally, the trained image processing model utilizes the Pixel Shuffle function to perform super-resolution reduction processing on the 4-channel features to obtain a high-resolution output image with resolution of (2 h,2 w), namely a processed image.

Referring to fig. 7, fig. 7 is a flowchart of an image processing method provided in the present application. The execution subject in the embodiment of the present application may be a computer device, which may include, for example, a server and a terminal device. As shown in fig. 7, the image processing method may include:

step S701, an image is acquired.

In an exemplary scenario, if a user wants to perform image processing on a certain image by using the image processing manner mentioned in the embodiments of the present application, the user may send an image processing instruction on the image to the computer device, and after detecting the image processing instruction, the computer device may acquire the image to perform subsequent steps S702 to S705.

In another exemplary scenario, the image processing method mentioned in the embodiment of the present application may be applied to a video scenario, for example, after a computer device obtains a video, if the image quality of an image in the video is low, the computer device may obtain each image in the video, for any image, by executing subsequent steps S702 to S705, a processed image corresponding to any image may be obtained, and then the processed images corresponding to each image may be spliced to obtain an encoded video corresponding to the video, so as to implement encoding of the video, and ensure that the image quality of each image in the encoded video is high.

In step S702, a first feature extraction process is performed on the image, so as to obtain a first image feature of the image, where the first image feature includes an image feature for deblocking.

The computer device may invoke the image processing model to perform a first feature extraction process on the image to obtain a first image feature of the image, where the first image feature includes an image feature for deblocking to facilitate removal of blocking of the image.

Specifically, the image processing model may include a first image feature extraction module, and the computer device may call the image processing model, and perform a first feature extraction process on the image through the first image feature extraction module, to obtain a first image feature of the image. For example, the first image feature extraction module may comprise a convolution plus parameter correction linear unit, that is, the computer device may extract the first image feature of the acquired image through an AR link in the image processing model constituted based on the convolution plus parameter correction linear unit.

In step S703, a second feature extraction process is performed on the image, so as to obtain a second image feature of the image, where the second image feature includes an image feature for enhancing details of the image.

The computer device may invoke the image processing model to perform a second feature extraction process on the image to obtain a second image feature of the image, where the second image feature includes an image feature for enhancing details of the image, so as to facilitate super-resolution processing of the image.

Specifically, the image processing model may include a second image feature extraction module, and the computer device may call the image processing model, and perform second feature extraction processing on the image through the second image feature extraction module, to obtain a second image feature of the image. For example, the second image feature extraction module may include a convolution plus parameter correction linear unit, that is, the computer device may extract the second image feature of the acquired image through an SR link in the image processing model that is based on the convolution plus parameter correction linear unit.

It is to be understood that the sequence of step S702 and step S703 is not limited herein. For example, the computer device may perform first feature extraction processing on the image to obtain a first image feature of the image, and then perform second feature extraction processing on the image to obtain a second image feature of the image; for another example, the computer device may perform the second feature extraction process on the image to obtain the second image feature of the image, and then perform the first feature extraction process on the image to obtain the first image feature of the image; for another example, the computer device may perform a first feature extraction process on the image to obtain a first image feature of the image, and perform a second feature extraction process on the image to obtain a second image feature of the image.

Specifically, the computer device may first extract a first image feature of the acquired image using the AR link of the image processing model, and then extract a second image feature of the acquired image using the SR link of the image processing model; or firstly, extracting the second image characteristic of the acquired image by utilizing the SR link of the image processing model, and then extracting the first image characteristic of the acquired image by utilizing the AR link of the image processing model; the AR link and the SR link of the image processing model can be utilized to simultaneously perform feature extraction processing on the acquired image, so as to respectively obtain a first image feature and a second image feature of the acquired image.

Step S704, fusing the first image features and the second image features to obtain fused image features.

The computer device may fuse, e.g., splice, the first image feature with the second image feature to obtain a fused image feature.

Stitching may refer to stitching of channels based on image features. For example, the first image feature may be an image feature of an L channel, the second image feature may be an image feature of an M channel, and the computer device may stitch the first image feature and the second image feature based on the channels to obtain a fused image feature of the (l+m) channel.

Alternatively, the computer device may connect or stitch the first image feature obtained at the AR link with the second image feature obtained at the SR link using the image processing model, thereby obtaining the fused image feature.

Step S705, performing super-resolution image restoration processing on the fused image features to obtain a processed image, where the resolution of the processed image is higher than that of the image.

The computer equipment can call the image processing model, and the fused sample image features are subjected to super-resolution image restoration processing by utilizing a proper super-resolution algorithm to obtain an output image. For example, the computer device may invoke an image processing model to perform super-resolution image restoration processing on the fused image using a Pixel Shuffle function or any other suitable super-resolution algorithm function to obtain a high-resolution SR image.

In the embodiment of the application, the computer equipment invokes the first image feature extracted by the image processing model to help remove the blocking effect of the image, the extracted second image feature can strengthen the image detail to help carry out super-resolution processing, and after the first image feature and the second image feature are fused, the super-resolution reduction processing is carried out on the fused image feature, so that the blocking effect can be removed, meanwhile, the super-resolution detail is reserved, and the image quality is improved.

Referring to fig. 8, fig. 8 is a flowchart of an example of another image processing method provided in the present application. The execution subject in the embodiment of the present application may be a computer device, which may include, for example, a server and a terminal device. As shown in fig. 8, the image processing method may include:

step S801, an image is acquired.

Step S802, performing a first feature extraction process on the image to obtain a first image feature of the image, where the first image feature includes an image feature for deblocking.

Step S803, performing a second feature extraction process on the image to obtain a second image feature of the image, where the second image feature includes an image feature for enhancing details of the image.

The specific description of steps S801 to S803 can be found in the description of steps S701 to S703 in fig. 7, and will not be repeated here.

Step S804, performing dimension reduction processing on the first image feature to obtain a first offset feature.

The computer device may perform a dimension reduction process on the first image feature using the image processing model to obtain a first offset feature.

Specifically, the first image feature acquired by the image processing model AR link may be a multi-channel or multi-dimensional image feature, and the image processing model may perform a dimension reduction process on the multi-channel first image feature based on, for example, a convolution of 1×1, and integrate the first image feature to obtain, for example, a first offset feature of 1 channel.

Step S805, performing correction processing on the second image feature based on the first offset feature, to obtain a corrected second image feature.

The computer device may perform a correction process on the second image feature based on the first offset feature to obtain a corrected second image feature.

Specifically, the image processing model may connect or splice the first offset feature with the second image feature, where the number of channels of the first offset feature is smaller than the number of channels of the second image feature, and correct the second image feature of the SR link by fusing the first offset feature with the second image feature, to obtain a corrected second image feature.

Step S806, performing dimension reduction processing on the second image feature to obtain a second offset feature.

The computer device may perform a dimension reduction process on the second image feature to obtain a second offset feature.

Specifically, the second image feature acquired by the image processing model SR link may be a multi-channel or multi-dimensional image feature, and the image processing model may perform a dimension reduction process on the multi-channel second image feature based on, for example, a convolution of 3×3, and integrate the second offset feature of, for example, 1 channel.

In step S807, the first image feature is corrected based on the second offset feature, and the corrected first image feature is obtained.

The computer device may perform a correction process on the first image feature based on the second offset feature to obtain a corrected first image feature.

Specifically, the image processing model may connect or splice the second offset feature with the first image feature, where the number of channels of the second offset feature is smaller than the number of channels of the first image feature, and correct the first image feature of the AR link by fusing the second offset feature with the first image feature, to obtain a corrected first image feature.

It is easy to understand that the number of channels of the first offset feature is smaller than that of the second image feature, so that the main effect of the second image feature after the SR link is corrected after fusion is still focused on the processing of the super-resolution task. Also, the image features for deblocking included in the first image features in the AR link may have a characteristic of smoothing the image, and thus the first image features may be directly reduced in dimension by convolution of, for example, 1×1 to obtain first offset features without further smoothing, and the obtained first offset features may still have a characteristic of smoothing the image. Thus, the first offset feature is utilized to correct the second image feature, so that potential noise possibly amplified in the second image feature in the process of enhancing the image details of the SR link can be counteracted.

Similarly, it is known that the number of channels of the second offset feature is smaller than that of the first image feature, so that it is ensured that the primary effect of the first image feature after the AR link correction after fusion is still focused on the deblocking task. And, the image features included in the second image features in the SR link for enhancing image details may enrich the details of the image, so that the second image features may be reduced in size and smoothed simultaneously with, for example, a 3×3 convolution to obtain second offset features, which may be relatively smooth and still have the characteristics of enriching image details. Therefore, the first image feature is corrected by using the second offset feature, so that the image details gradually lost in the first image feature in the deblocking process of the AR link can be compensated.

It is to be understood that the order of steps S804 and S805 and steps S806 and S807 is not limited herein. The computer equipment can utilize the image processing model to simultaneously and respectively perform dimension reduction processing on the first image feature of the AR link and the second image feature of the SR link to obtain a first offset feature and a second offset feature, and simultaneously and respectively utilize the first offset feature and the second offset feature to correct the second image feature and the first image feature to obtain a corrected second image feature and a corrected first image feature.

Step S808, fusing the corrected first image feature and the corrected second image feature to obtain a fused image feature.

The computer device may fuse the corrected first image feature with the corrected second image feature to obtain a fused image feature.

Step S809, performing super-resolution image restoration processing on the fused image features to obtain a processed image, where the resolution of the processed image is higher than that of the image.

The specific explanation of step S809 can be found in the description of step S705 in fig. 7, and will not be repeated here.

It can be found that the potential noise possibly amplified in the second image feature in the process of enhancing the image details of the SR link can be counteracted by correcting the second image feature by utilizing the first offset feature; and correcting the first image characteristic by using the second offset characteristic, so that the image details gradually lost in the first image characteristic in the deblocking process of the AR link can be compensated. And the corrected first image features and the corrected second image features are fused to obtain fused image features, so that the subsequent image obtained through super-resolution image reduction processing is facilitated, the blocking effect is removed, and the image quality is improved.

Referring to fig. 9, fig. 9 is a schematic structural diagram of an image processing apparatus 900 provided in the present application. The image processing apparatus may be a computer program (comprising program code) running in a computer device, for example the image processing apparatus is an application software, which may be used to perform the respective steps in the methods provided in the embodiments of the present application. As shown in fig. 9, the image processing apparatus 900 may include: an acquisition module 901, a feature extraction module 902, a fusion module 903 and a processing module 904.

The acquisition module 901 is used for acquiring an image.

The feature extraction module 902 is configured to perform a first feature extraction process on an image to obtain a first image feature of the image, where the first image feature includes an image feature for deblocking.

The feature extraction module 902 is further configured to perform a second feature extraction process on the image to obtain a second image feature of the image, where the second image feature includes an image feature for enhancing details of the image.

The fusion module 903 is configured to fuse the first image feature and the second image feature to obtain a fused image feature;

the processing module 904 is configured to perform super-resolution image restoration processing on the fused image features to obtain a processed image, where the resolution of the processed image is higher than the resolution of the image.

Optionally, the fusing module 903 fuses the first image feature and the second image feature to obtain a fused image feature, including: performing dimension reduction processing on the first image feature to obtain a first offset feature; correcting the second image characteristic based on the first offset characteristic to obtain a corrected second image characteristic; performing dimension reduction processing on the second image feature to obtain a second offset feature; correcting the first image characteristic based on the second offset characteristic to obtain a corrected first image characteristic; and fusing the corrected first image characteristic with the corrected second image characteristic to obtain a fused image characteristic.

Optionally, the processed image is obtained by the processing module 904 calling an image processing model to process, and a training manner of the image processing model includes: acquiring a sample image and a reference image corresponding to the sample image, wherein the resolution of the reference image is higher than that of the sample image; invoking an initial image processing model to extract first sample image features of the sample image, wherein the first sample image features include image features for deblocking effects; invoking the initial image processing model to extract second sample image features of the sample image, wherein the second sample image features include image features for enhancing image details; fusing the first sample image features and the second sample image features to obtain fused sample image features; performing super-resolution image restoration processing on the fused sample image characteristics to obtain an output image, wherein the resolution of the output image is matched with the resolution of a reference image; and training the initial image processing model by taking the difference between the reduced output image and the reference image as a training target to obtain an image processing model.

Optionally, the processing module 904 obtains a sample image, including: and performing first resolution reduction processing on the reference image to obtain a sample image.

Optionally, the processing module 904 is further configured to: performing second resolution reduction processing on the reference image to obtain an intermediate reference image, wherein the intermediate reference image is matched with the resolution of the sample image, and the definition of the intermediate reference image is higher than that of the sample image; performing image restoration processing on the first sample image characteristics to obtain an intermediate image; training the initial image processing model by taking the difference between the reduced output image and the reference image as a training target to obtain an image processing model, wherein the method comprises the following steps: and training the initial image processing model by taking the difference between the reduced output image and the reference image and the difference between the intermediate image and the intermediate reference image as training targets to obtain an image processing model.

Optionally, the processing module 904 acquires a reference image corresponding to the sample image, including: performing super-resolution processing on the source reference image to obtain an amplified image; performing deblocking processing on the amplified image to obtain a deblocking image; and performing resolution reduction processing on the image subjected to the deblocking effect to obtain a reference image.

According to one embodiment of the present application, the steps involved in the image processing method shown in fig. 7 may be performed by respective modules in the image processing apparatus 900 shown in fig. 9. For example, step S701 shown in fig. 7 may be performed by the acquisition module 901 in fig. 9, steps S702-S703 shown in fig. 7 may be performed by the feature extraction module 902 in fig. 9, step S304 shown in fig. 7 may be performed by the fusion module 903 in fig. 9, and step S705 shown in fig. 7 may be performed by the processing module 904 in fig. 9.

The method and the device can conduct first feature extraction processing on the image to obtain first image features of the image, wherein the first image features comprise image features for deblocking effects; performing second feature extraction processing on the image to obtain second image features of the image, wherein the second image features comprise image features for strengthening image details; fusing the first image features and the second image features to obtain fused image features; and performing super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than that of the image. Therefore, in the method provided by the application, the extracted first image feature is beneficial to removing the blocking effect of the image, the extracted second image feature can strengthen the image detail and is beneficial to super-resolution processing, and after the first image feature and the second image feature are fused, super-resolution reduction processing is carried out on the fused image feature, so that the blocking effect can be removed, meanwhile, the super-resolution detail is reserved, and the image quality is improved.

According to one embodiment of the present application, each module in the image processing apparatus 900 shown in fig. 9 may be separately or completely combined into one or several units to form a unit, or some (some) units may be further split into a plurality of sub-units with smaller functions, so that the same operation may be implemented without affecting the implementation of the technical effects of the embodiments of the present application. The above modules are divided based on logic functions, and in practical applications, the functions of one module may be implemented by a plurality of units, or the functions of a plurality of modules may be implemented by one unit. In other embodiments of the present application, the image processing apparatus 900 may also include other units, and in practical applications, these functions may also be implemented with assistance of other units, and may be implemented by cooperation of a plurality of units.

According to one embodiment of the present application, the image processing apparatus 900 shown in fig. 9 may be constructed by running a computer program (including program code) capable of executing the steps involved in the respective methods shown in fig. 7 on a general-purpose computer device such as a computer including a processing element such as a Central Processing Unit (CPU), a random access storage medium (RAM), a read only storage medium (ROM), and the like, and a storage element, and implementing the image processing method of the embodiments of the present application. The computer program may be recorded on, for example, a computer readable storage medium, and loaded into and executed by the computing device.

Referring to fig. 10, fig. 10 is a schematic structural diagram of a computer device 1000 provided in the present application. As shown in fig. 9, the computer device 1000 may include: processor 1001, network interface 1004, and memory 1005, in addition, computer device 1000 may further comprise: a user interface 1003, and at least one communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display (Display), a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface, among others. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1005 may also optionally be at least one storage device located remotely from the processor 1001. As shown in fig. 10, an operating system, a network communication module, a user interface module, and a device control application program may be included in the memory 1005, which is one type of computer storage medium.

In the computer device 1000 shown in FIG. 10, the network interface 1004 may provide network communication functions; while user interface 1003 is primarily used as an interface for providing input to a user; the processor 1001 may be configured to invoke the device control application stored in the memory 1005 to execute the description of the image processing method in the embodiment corresponding to fig. 7, or execute the description of the image processing apparatus 900 in the embodiment corresponding to fig. 9, which is not described herein. In addition, the description of the beneficial effects of the same method is omitted.

Furthermore, it should be noted here that: the present application further provides a computer readable storage medium, in which the aforementioned computer program executed by the image processing apparatus 900 is stored, and the computer program includes program instructions, when executed by a processor, can execute the description of the image processing method in the embodiment corresponding to fig. 7, and therefore, a detailed description will not be given here. In addition, the description of the beneficial effects of the same method is omitted. For technical details not disclosed in the embodiments of the computer storage medium related to the present application, please refer to the description of the method embodiments of the present application.

As an example, the above-described program instructions may be executed on one computer device or on a plurality of computer devices disposed at one site, or alternatively, on a plurality of computer devices distributed at a plurality of sites and interconnected by a communication network, which may constitute a blockchain network.

The computer readable storage medium may be the image processing apparatus provided in any one of the foregoing embodiments or an internal storage unit of the computer device, for example, a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (flash card) or the like, which are provided on the computer device. Further, the computer-readable storage medium may also include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

The present application provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above description of the image processing method in the corresponding embodiment of fig. 7, and therefore, a detailed description will not be given here. In addition, the description of the beneficial effects of the same method is omitted. For technical details not disclosed in the embodiments of the computer-readable storage medium according to the present application, please refer to the description of the method embodiments of the present application.

The terms first, second and the like in the description and in the claims and drawings of the embodiments of the present application are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the term "include" and any variations thereof is intended to cover a non-exclusive inclusion. For example, a process, method, apparatus, article, or device that comprises a list of steps or elements is not limited to the list of steps or modules but may, in the alternative, include other steps or modules not listed or inherent to such process, method, apparatus, article, or device.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The methods and related devices provided in the embodiments of the present application are described with reference to the method flowcharts and/or structure diagrams provided in the embodiments of the present application, and each flowchart and/or block of the method flowcharts and/or structure diagrams may be implemented by computer program instructions, and combinations of flowcharts and/or blocks in the flowchart and/or block diagrams. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or structural diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or structures.

The foregoing disclosure is only illustrative of the preferred embodiments of the present application and is not intended to limit the scope of the claims herein, as the equivalent of the claims herein shall be construed to fall within the scope of the claims herein.

Claims

1. An image processing method, the method comprising:

performing first feature extraction processing on an image to obtain first image features of the image, wherein the first image features comprise image features for deblocking effects;

performing second feature extraction processing on the image to obtain second image features of the image, wherein the second image features comprise image features for strengthening image details;

fusing the first image features and the second image features to obtain fused image features;

and performing super-resolution image restoration processing on the fused image features to obtain a processed image, wherein the resolution of the processed image is higher than that of the image.

2. The image processing method according to claim 1, wherein the fusing the first image feature and the second image feature to obtain a fused image feature includes:

Performing dimension reduction processing on the first image feature to obtain a first offset feature;

correcting the second image feature based on the first offset feature to obtain a corrected second image feature;

performing dimension reduction processing on the second image feature to obtain a second offset feature;

correcting the first image feature based on the second offset feature to obtain a corrected first image feature;

and fusing the corrected first image feature with the corrected second image feature to obtain the fused image feature.

3. The image processing method according to claim 1, wherein the processed image is obtained by calling an image processing model for processing, and the training manner of the image processing model includes:

acquiring a sample image and a reference image corresponding to the sample image, wherein the resolution of the reference image is higher than that of the sample image;

invoking an initial image processing model to extract first sample image features of the sample image, wherein the first sample image features include image features for deblocking effects;

Invoking the initial image processing model to extract second sample image features of the sample image, wherein the second sample image features include image features for enhancing image details;

fusing the first sample image features and the second sample image features to obtain fused sample image features;

performing super-resolution image restoration processing on the fused sample image characteristics to obtain an output image, wherein the resolution of the output image is matched with the resolution of the reference image;

and training the initial image processing model by taking the difference between the reduced output image and the reference image as a training target to obtain the image processing model.

4. The image processing method according to claim 3, wherein the acquiring the sample image includes:

and performing first resolution reduction processing on the reference image to obtain the sample image.

5. The image processing method according to claim 4, characterized in that the method further comprises:

performing second resolution reduction processing on the reference image to obtain an intermediate reference image, wherein the intermediate reference image is matched with the resolution of the sample image, and the definition of the intermediate reference image is higher than that of the sample image;

Performing image restoration processing on the first sample image characteristics to obtain an intermediate image;

the training the initial image processing model by taking the difference between the reduced output image and the reference image as a training target to obtain the image processing model comprises the following steps:

and training the initial image processing model by taking the difference between the reduced output image and the reference image and the difference between the intermediate image and the intermediate reference image as training targets to obtain the image processing model.

6. The method according to any one of claims 3 to 5, wherein the acquiring the reference image corresponding to the sample image includes:

performing super-resolution processing on the source reference image to obtain an amplified image;

performing deblocking processing on the amplified image to obtain a deblocking image;

and performing resolution reduction processing on the image subjected to the deblocking effect to obtain the reference image.

7. An image processing apparatus, characterized in that the apparatus comprises:

the image processing device comprises a feature extraction module, a processing module and a processing module, wherein the feature extraction module is used for carrying out first feature extraction processing on an image to obtain first image features of the image, and the first image features comprise image features for deblocking effects;

The feature extraction module is further configured to perform a second feature extraction process on the image to obtain a second image feature of the image, where the second image feature includes an image feature for enhancing details of the image;

the fusion module is used for fusing the first image features and the second image features to obtain fused image features;

and the processing module is used for carrying out super-resolution image restoration processing on the fused image characteristics to obtain a processed image, wherein the resolution of the processed image is higher than that of the image.

8. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the steps of the method of any of claims 1-6.

9. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method of any of claims 1-6.

10. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program adapted to be loaded by a processor and to perform the method of any of claims 1-6.