CN114638748A

CN114638748A - Image processing method, image restoration method, computer device, and storage medium

Info

Publication number: CN114638748A
Application number: CN202011489206.7A
Authority: CN
Inventors: 杨凌波; 王攀; 高占宁; 任沛然; 谢宣松
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2020-12-16
Filing date: 2020-12-16
Publication date: 2022-06-17

Abstract

The embodiment of the application discloses an image processing method, an image restoration method, computer equipment and a storage medium. The image processing method comprises the following steps: determining a feature space level and an image restoration level applied to an image to be restored based on application environment data of the image to be restored; extracting image features layer by layer until image features of the image to be repaired in the determined feature space hierarchy are obtained; processing the image features extracted from each feature space level into combined image features; and performing image restoration level by level based on the combined image features until the restored image is obtained after restoration of the determined image restoration level is completed. The method has the advantages that the degradation assumption is not needed to be carried out in advance, the prior fitting error caused by artificial introduction is reduced, the method is suitable for image restoration of various scenes, the generalization capability of the general scenes is improved, the law and the characteristic of the images do not need to be calculated in advance, and a good restoration effect can be achieved for all areas except the target object in the images.

Description

Image processing method, image restoration method, computer device, and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to an image processing method and apparatus, an image restoration model processing method and apparatus, a commodity retrieval method and apparatus, a live broadcast method and apparatus, an image recognition method and apparatus, a computer device, and a computer-readable storage medium.

Background

Image degradation is any image transformation which may cause image information loss and degrade the visual impression of human eyes, and is usually irreversible, such as down-sampling, random noise, JPEG (Joint Photographic Experts Group) compression, and the like.

One is degradation prior, that is, a degradation transformation form is deduced according to observation of a degraded image, a specific degradation type needs to be preset, a random simulation method is used for sampling and constructing a data set, and a corresponding degradation model is trained for repairing the degraded image. The scheme can only be applied to scenes which accord with corresponding degradation assumptions, and has poor generalization effect on real scenes.

The other is content prior, namely, according to observation of a high-quality image sample, the probability distribution to which the real image is subjected and local statistical characteristics, such as sparsity and local smoothness of the characteristics, are deduced, and the degraded image is repaired according to the statistical characteristics. The scheme does not need degradation prior, can adapt to various degradation types, and has certain requirements on image content prior. Taking face restoration as an example, such algorithms usually use the characteristics of the face and use indexes such as face key point matching, face recognition similarity and the like to constrain the original restoration problem, thereby achieving a better restoration effect. But correspondingly, the use of face priors also easily affects the repair of non-face areas in the background, so that the repair effect of the whole image is poor.

Disclosure of Invention

In view of the above, the present application is made to provide an image processing method, an image searching method, an object searching method, and a computer device, computer-readable storage medium that overcome or at least partially solve the above problems.

According to an aspect of the present application, there is provided an image processing method including:

determining a feature space level and an image restoration level applied to an image to be restored based on application environment data of the image to be restored; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

extracting image features layer by layer until image features of the image to be repaired in the determined feature space hierarchy are obtained;

processing the image features extracted from each feature space level into combined image features;

and performing image restoration level by level based on the combined image features until the restored image is obtained after restoration of the determined image restoration level is completed.

According to another aspect of the present application, there is provided an image restoration method including:

obtaining a repaired target image under the determined characteristic space level and image repairing level based on the image repairing model; the image restoration model comprises a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit, wherein the multi-level feature analysis unit is used for carrying out image feature extraction layer by layer until the image features of the image to be restored in the determined feature space level are obtained, and the multi-level image restoration unit is used for carrying out image restoration layer by layer based on the extracted image features until the restored image is obtained after the restoration of the determined image restoration level is completed.

According to another aspect of the present application, there is provided a method for processing an image inpainting model, including:

iteratively training an image inpainting model based on a plurality of sample image pairs, the sample image pairs comprising a low-quality image and a corresponding high-quality image; the image restoration model comprises a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit;

after each training, obtaining a repaired image based on the low-quality image and the image repairing model;

determining correction coefficients respectively corresponding to the multi-level feature analysis unit, the feature processing unit and the multi-level image restoration unit according to the difference between the restored image and the high-quality image;

and correcting the multi-level feature analysis unit, the feature processing unit and the multi-level image restoration unit based on the correction coefficient to obtain a corrected image restoration model.

According to another aspect of the present application, there is provided a commodity search method including:

receiving a commodity retrieval request carrying a commodity reference image;

determining a feature space level and an image restoration level applied to the commodity reference image based on the retrieval environment data; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

extracting image features layer by layer until the image features of the commodity reference image in the feature space hierarchy are obtained;

performing image restoration level by level based on the combined image features until restoration at the determined image restoration level is completed;

a product search result is provided based on the repaired product reference image.

In accordance with another aspect of the present application, there is provided a live broadcasting method including:

collecting live broadcast video stream in transmission, and determining a video frame to be repaired;

determining a feature space level and an image restoration level applied to the video frame to be restored based on the application environment data of the video; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

extracting image features layer by layer until obtaining the image features of the video frame to be restored in the feature space level;

processing the image features extracted from each feature space hierarchy into combined image features;

acquiring a repaired live video stream according to the repaired video frame;

and transmitting the repaired live video stream.

In accordance with another aspect of the present application, there is provided an image recognition method including:

acquiring an image to be identified;

determining a feature space level and an image restoration level applied to the image to be recognized based on application environment data of image recognition; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

extracting image features layer by layer until image features of the image to be repaired in the determined feature space level are obtained;

performing image restoration level by level based on the combined image characteristics until restoration at the determined image restoration level is completed;

and identifying the target object of the repaired image to be identified.

According to another aspect of the present application, there is provided an image processing apparatus including:

the system comprises a hierarchy determining module, a feature space hierarchy and an image restoration hierarchy, wherein the hierarchy determining module is used for determining the feature space hierarchy and the image restoration hierarchy applied to an image to be restored based on application environment data of the image to be restored; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

the characteristic extraction module is used for carrying out image characteristic extraction layer by layer until image characteristics of the image to be repaired in the determined characteristic space level are obtained;

the characteristic combination module is used for processing the image characteristics extracted by each characteristic space level into combined image characteristics;

and the image restoration module is used for carrying out image restoration level by level based on the combined image characteristics until the restored image is obtained after the restoration of the determined image restoration level is finished.

According to another aspect of the present application, there is provided an image restoration apparatus including:

the system comprises a hierarchy determining module, a feature space hierarchy determining module and an image restoration module, wherein the hierarchy determining module is used for determining a feature space hierarchy and an image restoration hierarchy applied to an image to be restored based on application environment data of the image to be restored; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

the image restoration module is used for acquiring a restored target image under the determined characteristic space level and the image restoration level based on the image restoration model; the image restoration model comprises a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit, wherein the multi-level feature analysis unit is used for extracting image features layer by layer until the image features of an image to be restored in the determined feature space level are obtained, and the multi-level image restoration unit is used for restoring the image layer by layer based on the extracted image features until the restored image is obtained after the restoration of the determined image restoration level is completed.

According to another aspect of the present application, there is provided an image inpainting model processing apparatus, including:

a model training module to iteratively train an image inpainting model based on a plurality of sample image pairs, the sample image pairs including low-quality images and corresponding high-quality images; the image restoration model comprises a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit;

wherein the model training module comprises:

the image restoration unit is used for obtaining a restoration image based on the low-quality image and the image restoration model after each training;

the correction coefficient determining unit is used for determining correction coefficients corresponding to the multi-level feature analyzing unit, the feature processing unit and the multi-level image repairing unit according to the difference between the repaired image and the high-quality image;

and the correcting unit is used for correcting the multi-level feature analyzing unit, the feature processing unit and the multi-level image repairing unit based on the correction coefficient to obtain a corrected image repairing model.

According to another aspect of the present application, there is provided an article search device including:

the request receiving module is used for receiving a commodity retrieval request carrying a commodity reference image;

the level determining module is used for determining a characteristic space level and an image repairing level applied to the commodity reference image based on the retrieval environment data; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

the characteristic extraction module is used for carrying out image characteristic extraction layer by layer until the image characteristics of the commodity reference image in the characteristic space hierarchy are obtained;

the image restoration module is used for carrying out image restoration level by level based on the combined image characteristics until restoration at the determined image restoration level is completed;

and a result providing module for providing a commodity search result based on the repaired commodity reference image.

In accordance with another aspect of the present application, there is provided a live broadcasting apparatus including:

the video frame determining module is used for acquiring a live video stream in transmission and determining a video frame to be repaired;

the level determining module is used for determining a characteristic space level and an image repairing level applied to the video frame to be repaired based on the application environment data of the video; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

the characteristic extraction module is used for carrying out image characteristic extraction layer by layer until the image characteristics of the video frame to be repaired in the characteristic space level are obtained;

the video stream updating module is used for obtaining a repaired live video stream according to the repaired video frame;

and the video stream transmission module is used for transmitting the repaired live video stream.

According to another aspect of the present application, there is provided an image recognition apparatus including:

the image acquisition module is used for acquiring an image to be identified;

the hierarchy determining module is used for determining a feature space hierarchy and an image restoration hierarchy applied to the image to be identified based on application environment data of image identification; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

and the object identification module is used for identifying the target object of the repaired image to be identified.

In accordance with another aspect of the present application, there is provided an electronic device including: a processor; and

a memory having executable code stored thereon, which when executed, causes the processor to perform a method as in any one of the above.

According to another aspect of the application, there is provided one or more machine-readable media having stored thereon executable code that, when executed, causes a processor to perform a method as any one of the above.

According to the embodiment of the application, a mechanism that multi-feature space levels are used for feature extraction of an image to be repaired and multi-image repairing levels are combined for image repairing is adopted, in the framework, different feature space levels correspondingly extract image features of different feature spaces, and different image repairing levels correspondingly repair the image features of the different feature spaces. When image processing is carried out, firstly, a characteristic space level and an image restoration level applied to an image to be restored are determined based on application environment data of the image to be restored; extracting image features layer by layer until image features of the image to be restored in a feature space level are obtained; processing the image features extracted from each feature space level into combined image features; and performing image restoration level by level based on the combined image features until the restored image is obtained after restoration of the determined image restoration level is completed.

Compared with a scheme of degradation prior, the method does not need to perform degradation hypothesis in advance, reduces prior fitting errors caused by artificial introduction, is suitable for image restoration of various scenes, improves the generalization capability of general scenes, does not need to calculate the rule and the characteristic of the image in advance compared with a scheme of content prior, and can achieve a better restoration effect on all regions except a target object in the image. In addition, the model is not constrained according to prior information, so that model parameters do not need to be adjusted according to the image degradation degree and the application scene, the images with different degradation degrees and different application scenes are processed in a single model in a self-adaptive mode, and the convenience of image restoration model deployment is greatly improved.

In addition, as the plurality of feature space levels and the plurality of image restoration levels are set, different levels are selected to correspond to different processing complexities, and further, in specific application, the levels are selected based on application environment data, so that the method is convenient and flexible to regulate and control, and is applied to scenes with different requirements. For example, for a simple scene or a situation of resource shortage, only the image features of the low feature space level may be extracted, and the image restoration is performed in combination with the low image restoration level to obtain a simple restoration result. Aiming at a complex scene or the condition of sufficient resources, the image features of a high feature space level can be extracted to obtain more comprehensive and fine image features, and the image restoration is carried out by combining a low image restoration level to obtain a high-quality restoration result.

The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 illustrates a particular example of an object recognition method of the present application;

FIG. 2 illustrates another specific example of an object recognition method of the present application;

FIG. 3 is a flow chart of an image processing method according to a first embodiment of the present application;

FIG. 4 is a flowchart of an image restoration method according to the second embodiment of the present application;

FIG. 5 is a flowchart of a processing method of an image restoration model according to the third embodiment of the present application;

fig. 6 is a flowchart illustrating a commodity retrieval method according to a fourth embodiment of the present application;

fig. 7 shows a flow chart of a live method according to a fifth embodiment of the present application;

FIG. 8 is a flow chart of an image recognition method according to the sixth embodiment of the present application;

fig. 9 is a block diagram showing a configuration of an image processing apparatus according to a seventh embodiment of the present application;

fig. 10 is a block diagram showing a configuration of an image restoration apparatus according to an eighth embodiment of the present application;

fig. 11 is a block diagram showing a processing apparatus of an image inpainting model according to a ninth embodiment of the present application;

fig. 12 is a block diagram showing a structure of an article search device according to a tenth embodiment of the present application;

fig. 13 is a block diagram illustrating a configuration of a live device according to an eleventh embodiment of the present application;

fig. 14 is a block diagram showing a configuration of an image recognition apparatus according to a twelfth embodiment of the present application;

fig. 15 illustrates an exemplary system that can be used to implement the various embodiments described in this disclosure.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

The method is characterized in that a completely implicit image restoration model is adopted, after a characteristic space level and an image restoration level applied to an image to be restored are determined based on application environment data of the image to be restored, image feature extraction can be performed layer by layer until image features of the image to be restored in the determined characteristic space level are obtained, the image features extracted from all the characteristic space levels are processed into combined image features, image restoration is performed layer by layer based on the combined image features until restoration of the determined image restoration level is completed, and a restored image is obtained.

The image to be restored can be derived from urban traffic videos, industrial control videos (such as monitoring video images in industrial scenes of intelligent manufacturing, mines and the like), extracted images of live telecast videos and the like, images used in software application and the like, and is used for restoring the images in links of video restoration, commodity retrieval, three-dimensional image construction, image retrieval, image identification, image optimization and the like so as to obtain a better processing result. The image to be restored may be an image with missing part of content, an image with lower resolution that is already blurred or deformed, and may be obtained through an image restoration request submitted by a client, or an image acquired by a corresponding client or device. For example, the target tracking can be performed on the image in the monitoring video acquired by urban traffic, the target in the acquired image is blurred due to the fact that the distance between the monitoring camera and the target is far away, and then the clearer image can be obtained after the image is repaired, and the more accurate identification result can be obtained by further using for object identification and tracking. Or, as a functional component of the photographing software, image restoration is performed on the photographed image or the image selected from the album to obtain a higher quality image.

And performing at least one application of video restoration, image recognition, commodity retrieval, three-dimensional image construction and image retrieval based on the restored image.

Feature space may also be understood as feature dimensions, and image features of the feature space may include at least one of: image object marker information, image edge information, image texture information, image shading, image reflection information, image tone, image shading. The image features may be specifically expressed in the form of a feature vector, or may be directly expressed as the features themselves, which is not limited in this application.

According to the embodiment of the application, a plurality of characteristic space levels and a plurality of image restoration levels are set, the different characteristic space levels correspondingly extract the image characteristics of different characteristic spaces, and in the concrete implementation, the identification mode of which image characteristics or characteristic space levels are extracted at each level can be set according to actual requirements when a model is trained, so that the application is not limited. For example, a first feature space level corresponds to extracting image object labeling information (e.g., location information of facial features) of an image, a second feature space level extracts edge information and texture features, a third feature space level extracts shadow and reflection features, a fourth feature space level extracts hue and illumination features, and a fifth feature space level extracts shading features.

In the embodiment of the present application, when feature extraction is performed correspondingly, image feature extraction is performed layer by layer until image features of an image to be repaired in a determined feature space hierarchy are obtained, that is, features are extracted from a starting-level feature space hierarchy until image features corresponding to the determined feature space hierarchy are extracted, where the starting feature space hierarchy may be any specified hierarchy, for example, a first feature space hierarchy. The extraction of the image features of each layer may be independent from each other, or may also be associated, for example, the image feature extracted according to the previous feature space level and the image to be repaired are taken as the extraction basis of the image feature of the current feature space level together, and the image feature of the next feature space level is extracted according to the image feature extracted according to the previous feature space level and the image to be repaired until the image feature of the image to be repaired in the feature space level is obtained.

In this embodiment, the feature space hierarchy and the image repair hierarchy are determined according to application environment data, where the application environment data may be at least one of processing performance data, storage performance data, application requirement data, and release requirement data of an image to be repaired, where the processing performance data represents a current processing capability, and may include performance of hardware of a device or a processing resource of a cloud server that can be called, such as an operation capability of a cloud server, a CPU, or a GPU graphics processor, the storage performance data may include a size of a storage space, a storage speed, and the like, the release requirement data may be a requirement submitted for image repair, such as required repair efficiency, repair quality, and the like, and the application requirement data may be a requirement submitted for image repair, such as image repair efficiency, repair quality, and the like, of a data system or software currently applied for image repair, Pixel resolution of the repair result, and the like.

The application environment data may also be configured with corresponding weight coefficients, and after the one or more types of application environment data are acquired, the hierarchical evaluation data of the image to be restored may be determined according to the corresponding weight coefficients, for example, a weighted value of the application environment data and the weight coefficients is used as the hierarchical evaluation data, or in other determination manners, and the application environment data that is not a specific numerical value may be mapped to a specific numerical value according to a mapping table. Furthermore, the corresponding relation between the numerical range and the characteristic space level or the image restoration level can be configured, the numerical range corresponding to the level evaluation data is determined, and the characteristic space level and the image restoration level applied to the image to be restored can be determined according to the numerical range.

Or the characteristic space level and the image restoration level can be further determined according to the mapping relation between the pre-configured application environment data and the levels.

The method is used for selecting the levels based on the application environment data, is convenient and flexible to regulate and control, and is applied to scenes with different requirements. As can be understood from the above examples, selecting different levels corresponds to different processing complexities, for example, for a simple scene or a resource shortage situation, only the image features of the low feature space level may be extracted, and the image restoration performed in combination with the low image restoration level may obtain a simple restoration result. Aiming at a complex scene or the condition of sufficient resources, the image features of a high feature space level can be extracted to obtain more comprehensive and fine image features, and the image restoration is carried out by combining a low image restoration level to obtain a high-quality restoration result.

In an optional embodiment, the hierarchy may be selected according to the quality of the image to be repaired, for example, the degradation degree, and the images with different qualities are repaired to different degrees, for example, a feature space hierarchy and an image repair hierarchy with a higher degradation degree are selected, and a feature space hierarchy and an image repair hierarchy with a lower degradation degree are selected, so that a stable image repair capability is maintained, thereby automatically adapting the degradation severity of the image, adjusting a repair policy, and ensuring a highly consistent repair effect.

After the image features extracted step by step are obtained, the features extracted correspondingly in each level of feature space hierarchy are combined to obtain combined image features, for example, the image features extracted in each level of feature space hierarchy may be combined and connected to obtain combined image features, for example, the image features may be sequentially connected in the order of the feature space hierarchy. Alternatively, the extracted image features may be combined by weighting to obtain a combined image feature.

Similarly, different image restoration levels correspondingly restore image features of different feature spaces, and the corresponding relationship between the image restoration level and the image feature of the restored feature space can be set according to actual requirements, which is not limited in the present application. And performing image restoration level by level based on the combined image features until restoration of the determined image restoration level is completed, performing image restoration level by level from the initial image restoration level, performing restoration of the next image restoration level according to the image restored by the previous image restoration level, and obtaining the restored image after restoration of the determined image restoration level is completed.

For example, in correspondence with the above example, the first image restoration level restores facial element features of an image, the second image restoration level adds restoration of edge features and texture features, i.e., restores facial element features, edge features, and texture features, on the basis of the first image restoration level, the third image restoration level adds restoration of shadow and reflection features, i.e., restores facial element features, edge features, texture features, shadows, and reflection features, on the basis of the second image restoration level, the fourth image restoration level adds restoration of feature and illumination features, i.e., restores facial element features, edge features, texture features, shadows, reflection features, tones, and illumination features, on the basis of the third image restoration level, the fifth image restoration level adds restoration of illumination features and shading features on the basis of the fourth image restoration level, namely, repair face element features, edge features, texture features, shadows, reflection features, distinctive tones, lighting features, and shading features.

It should be noted that, the selected feature space level and the image inpainting level may not correspond to the same image feature, for example, the feature space level is a third feature space level, and the image inpainting level is in a first image inpainting level, where a corresponding rule may be configured according to actual needs, which is not limited in this application. It can be understood that the feature space hierarchy is consistent with the image features corresponding to the image restoration hierarchy, and the image restoration hierarchy can restore the combined features obtained after the feature space hierarchy is extracted, so that a better restoration effect can be achieved.

The image restoration process can be completed by an image restoration model, the image restoration model can be deployed at a cloud end or an equipment end, correspondingly, the image restoration model is called to carry out image feature extraction layer by layer until the image features of the image to be restored in the determined feature space hierarchy are obtained, and the image restoration model is called to carry out image feature extraction layer by layer until the image features of the image to be restored in the determined feature space hierarchy are obtained.

In the image restoration model, the input data is the image to be restored and the output data is the restored image, the degradation type of the image to be restored does not need to be determined in the whole process, and the prior statistics on the image content is also not needed, so that the prior fitting error caused by human introduction can be reduced, the image restoration model is suitable for image restoration of various scenes, the generalization capability of the general scenes is improved, compared with the scheme with prior content, the law and the characteristic of the image do not need to be calculated in advance, and a better restoration effect can be achieved for all areas except the target object in the image.

In addition, the model is not constrained according to prior information, so that model parameters do not need to be adjusted according to the image degradation degree and the application scene, the images with different degradation degrees and different application scenes are processed in a single model in a self-adaptive mode, and the convenience of image restoration model deployment is greatly improved.

The image restoration model may be a Neural Network model, such as a Convolutional Neural Network (CNN), RNN, Recurrent Neural Network (current Neural Network), Deep Neural Network (DNN), Generate Adaptive Network (GANs), and the like. The three parts of image feature extraction, image feature combination and image restoration may be trained as separate neural network models, or a plurality of or three parts of the three parts may be trained in one neural network model, for example, the image restoration model may be divided into a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit, which are respectively used for executing each step in the image restoration.

Prior to use, an image inpainting model may be trained based on a plurality of sample image pairs, the sample image pairs including a low-quality image and a high-quality image corresponding to the low-quality image, and the particular training may train the neural network model using a deep learning approach. During specific training, the image restoration model can be iteratively trained based on a plurality of sample image pairs, and prediction of the image restoration model can be more accurate through multiple iterations.

After each training, a restored image is obtained based on the low-quality image and the initial image restoration model, the difference between the restored image and the high-quality image represents the accuracy of the image restoration model, if the difference is too large, for example, a loss function is used for measurement, and if the loss function exceeds a certain value, the training process of the model needs to be continued.

The method comprises the steps of determining correction coefficients corresponding to a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit according to the difference between a restored image and a high-quality image, and correcting the multi-level feature analysis unit, the feature processing unit and the multi-level image restoration unit based on the correction coefficients to obtain a corrected image restoration model. And the gradient of the loss function to the first, second and third network model parameters is realized, so that the training result is more accurately improved.

Compared with the existing image restoration scheme, the scheme of the application obtains better restoration evaluation results in multiple aspects in various degradation scenes such as face restoration, image illusion, denoising, deblurring, JPEG artifact, complete degradation and the like.

A specific example of an image processing method of the present application is given with reference to fig. 1.

Based on the fact that the client submits the image to be identified to the server, the internal structure of the image restoration model of the server is shown in the figure, the left side comprises three feature space levels which are Y2, Y1 and Y0 respectively, the image feature of the next feature space level is extracted based on the previous feature space level and the image to be restored, the right side comprises three image restoration levels which are G0, G1 and G2 respectively, and the middle parts F0, F1 and F2 are used for combining the image features extracted from the left side respectively and restoring the image on the right side.

And sequentially extracting corresponding image features of the image to be restored according to the sequence of Y2, Y1 and Y0, and restoring the image according to the sequence of G0, G1 and G2. The dashed portions identify various mapping possibilities for the feature space level and the image inpainting level.

The levels respectively selected at two sides can be determined according to application environment data, for example, Y0 and G2 are selected, three parts of features extracted at the left side can be merged to obtain a merged feature F0, and sequential repair of G0, G1 and G2 is carried out. If Y0 and G1 are selected, the three-part features extracted from the left side can be merged into sequential repairing of G1 and G2, and in this case, F0 can be used as the merged feature, or F1 for merging the image features of Y1 and Y0 can be used as the merged feature. If Y2 and G0 are selected, sequential repairing of G0, G1 and G2 is carried out, and the feature extracted by Y2 is taken as a combined feature, or F0 obtained by combining three parts of features extracted on the left side is taken as a combined feature.

Another specific example of an image processing method of the present application is given with reference to fig. 2, which shows an image restoration process, and the first six images include seven small images, and respectively show the characteristics of sequential restoration at five image restoration levels. The leftmost canvas is the canvas, the second is the position information of Facial Landmarks (Facial Landmarks) on the canvas, the third is the position information of Facial five-sense organs (Facial Landmarks), the third is the edge information and texture features (Edges and textures) are continuously repaired on the canvas, the fourth is the shadow and reflection features (shadows and Reflections) are continuously repaired on the canvas, the fifth is the hue and Illumination features (Tune and Illumination) are continuously repaired on the canvas, and the sixth is the coloring feature (coloration) is continuously repaired on the canvas, so that the repaired image is obtained. The seventh image is a restored image obtained by the method in the prior art, and as can be seen from the comparison of the images, the quality of the image restored by the scheme of the application is much higher than that of the image restored by the prior art.

It should be noted that the image repairing method according to the present disclosure may be implemented as a functional module loaded with a model application program, a service, an instance, a software form, a Virtual Machine (VM) or a container, or may also be implemented as a hardware device (such as a server or a terminal device) or a hardware chip (such as a CPU, a GPU or an FPGA) having an image processing function. The image restoration service can be provided through the client on the device, the image to be restored is obtained through the client, the image to be restored is further uploaded to the cloud to achieve the image restoration process, the image restoration process can be deployed on the device locally, and the method and the device are not limited to this.

Referring to fig. 3, a flowchart of an image processing method according to a first embodiment of the present application is shown, where the method may specifically include the following steps:

step 101, determining a feature space level and an image restoration level applied to an image to be restored based on application environment data of the image to be restored; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

step 102, extracting image features layer by layer until image features of an image to be repaired in the determined feature space level are obtained;

103, processing the image features extracted from each feature space level into combined image features;

and 104, performing image restoration level by level based on the combined image features until restoration of the determined image restoration level is completed, and obtaining a restored image.

In an optional embodiment of the present application, the determining, based on application environment data of an image to be repaired, a feature space level and an image repair level applied to the image to be repaired includes:

and determining a characteristic space level and an image restoration level applied to the image to be restored based on at least one of processing performance data, storage performance data, application demand data and release demand data of the image to be restored.

determining the hierarchical evaluation data of the image to be restored based on the multiple application environment data and the corresponding weight coefficients;

and determining a characteristic space level and an image repairing level applied to the image to be repaired according to the numerical value range corresponding to the level evaluation data.

In an optional embodiment of the present application, the performing image feature extraction layer by layer until obtaining the image feature of the image to be repaired in the determined feature space hierarchy includes:

and extracting image features from the initial feature space level layer by layer, and extracting the image features of the next feature space level according to the image features extracted from the previous feature space level and the image to be repaired until the image features of the image to be repaired in the feature space level are obtained.

In an optional embodiment of the present application, the image feature of the feature space includes at least one of: image object marker information, image edge information, image texture information, image shading, image reflection information, image tone, image shading.

In an optional embodiment of the present application, the processing the image features extracted by each feature space hierarchy into combined image features includes:

combining and connecting the image features extracted by each feature space level to obtain combined image features;

or weighting and combining the extracted image features to obtain combined image features.

In an optional embodiment of the present application, the performing image restoration level by level based on the combined image feature until restoration at the determined image restoration level is completed, and obtaining a restored image includes:

and starting from the initial image repairing level, performing image repairing layer by layer, and repairing the next image repairing level according to the image repaired by the previous image repairing level until the repaired image of the determined image repairing level is obtained.

and calling an image restoration model to execute image feature extraction layer by layer until the image features of the image to be restored in the determined feature space level are obtained.

In an optional embodiment of the present application, the method further includes:

training an image inpainting model based on the plurality of sample image pairs; the sample image pair includes a low quality image and a corresponding high quality image.

In an optional embodiment of the present application, the image inpainting model includes a multi-level feature analysis unit, a feature processing unit, and a multi-level image inpainting unit, where training the image inpainting model based on a plurality of sample image pairs includes:

iteratively training the image inpainting model based on a plurality of sample image pairs;

after each training, obtaining a repaired image based on the low-quality image and the initial image repairing model;

In this embodiment, the extracted image features may not be combined, that is, the image features extracted step by step are directly used for image restoration, and referring to fig. 4, a flowchart of an image restoration method according to the second embodiment of the present application is shown, and the method specifically may include the following steps:

step 201, determining a feature space level and an image restoration level applied to an image to be restored based on application environment data of the image to be restored; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

step 202, obtaining a repaired target image under the determined characteristic space level and image repairing level based on the image repairing model; the image restoration model comprises a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit, wherein the multi-level feature analysis unit is used for carrying out image feature extraction layer by layer until the image features of the image to be restored in the determined feature space level are obtained, and the multi-level image restoration unit is used for carrying out image restoration layer by layer based on the extracted image features until the restored image is obtained after the restoration of the determined image restoration level is completed.

According to the embodiment of the application, a mechanism for performing feature extraction on an image to be repaired by using multiple feature space levels and combining the multiple image repairing levels to perform image repairing is adopted, in the framework, image features of different feature spaces are correspondingly extracted by different feature space levels, and the image features of the different feature spaces are correspondingly repaired by the different image repairing levels. When image processing is carried out, firstly, a characteristic space level and an image restoration level applied to an image to be restored are determined based on application environment data of the image to be restored; extracting image features layer by layer until image features of the image to be restored in a feature space level are obtained; processing the image features extracted from each feature space level into combined image features; and performing image restoration level by level based on the combined image features until the restored image is obtained after restoration of the determined image restoration level is completed.

The following embodiment provides a training process of an image inpainting model, and referring to fig. 5, a flowchart of a processing method of an image inpainting model according to a third embodiment of the present application is shown, where the method specifically includes the following steps:

step 301, iteratively training an image inpainting model based on a plurality of sample image pairs, wherein the sample image pairs comprise low-quality images and corresponding high-quality images; the image restoration model comprises a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit;

wherein, step 301 may include:

substep 3011, after each training, obtaining a restored image based on the low-quality image and the image restoration model;

substep 3012, determining correction coefficients corresponding to the multi-level feature analysis unit, the feature processing unit and the multi-level image restoration unit according to the difference between the restored image and the high-quality image;

and a substep 3013, modifying the multi-level feature analysis unit, the feature processing unit, and the multi-level image restoration unit based on the modification coefficients, and obtaining a modified image restoration model.

In an optional embodiment of the present application, the multi-level feature analysis unit is configured to perform image feature extraction layer by layer until an image feature of an image to be repaired in a determined feature space level is obtained, the feature processing unit is configured to process the extracted image feature into a combined image feature, and the multi-level image repairing unit is configured to perform image repairing layer by layer based on the combined image feature until the repaired image is obtained after the repair of the determined image repairing layer is completed.

According to the embodiment of the application, a mechanism for performing feature extraction on an image to be repaired by using multiple feature space levels and combining the multiple image repairing levels to perform image repairing is adopted, in the framework, image features of different feature spaces are correspondingly extracted by different feature space levels, and the image features of the different feature spaces are correspondingly repaired by the different image repairing levels. When image processing is carried out, firstly, a characteristic space level and an image restoration level applied to an image to be restored are determined based on application environment data of the image to be restored; extracting image features layer by layer until image features of the image to be restored in a feature space level are obtained; processing the image features extracted from each feature space hierarchy into combined image features; and performing image restoration level by level based on the combined image features until the restored image is obtained after restoration of the determined image restoration level is completed.

In addition, as the plurality of feature space levels and the plurality of image restoration levels are set, different levels are selected to correspond to different processing complexities, and further, in specific application, the levels are selected based on application environment data, so that the method is convenient and flexible to regulate and control, and is applied to scenes with different requirements. For example, for a simple scene or a resource shortage situation, only the image features of a low feature space level may be extracted, and a simple restoration result may be obtained by performing image restoration in combination with the low image restoration level. Aiming at a complex scene or the condition of sufficient resources, the image features of a high feature space level can be extracted to obtain more comprehensive and fine image features, and the image restoration is carried out by combining a low image restoration level to obtain a high-quality restoration result.

The following embodiment provides an application of the image restoration scheme to commodity retrieval, namely, after receiving a commodity reference image submitted based on a commodity retrieval request, image restoration is performed first, and then commodity retrieval is performed. Referring to fig. 6, a flowchart of a commodity retrieval method according to a fourth embodiment of the present application is shown, where the method specifically includes the following steps:

step 401, receiving a commodity retrieval request carrying a commodity reference image;

step 402, determining a characteristic space level and an image restoration level applied to the commodity reference image based on retrieval environment data; extracting image features of different feature spaces correspondingly from different feature space levels, and repairing the image features of different feature spaces correspondingly from different image repairing levels;

step 403, extracting image features layer by layer until image features of the commodity reference image in the feature space hierarchy are obtained;

step 404, processing the image features extracted from each feature space level into combined image features;

step 405, performing image restoration level by level based on the combined image features until restoration at the determined image restoration level is completed;

step 406 is to provide a product search result based on the repaired product reference image.

The following embodiment provides an application of the image restoration scheme to live webcasting, that is, for a collected live webcasting video stream, image restoration is performed on a video frame needing restoration, and the restored video frame is used to replace an original video frame for transmission, so that the quality of a live webcasting video can be improved. Referring to fig. 7, a flowchart of a live broadcasting method according to a fifth embodiment of the present application is shown, where the method specifically includes the following steps:

step 501, collecting live broadcast video stream in transmission, and determining video frames to be repaired;

step 502, determining a feature space level and an image restoration level applied to the video frame to be restored based on the application environment data of the video; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

step 503, extracting image features layer by layer until obtaining image features of the video frame to be restored in the feature space hierarchy;

step 504, processing the image features extracted from each feature space level into combined image features;

step 505, performing image restoration level by level based on the combined image characteristics until restoration at the determined image restoration level is completed;

step 506, acquiring a repaired live video stream according to the repaired video frame;

and step 507, transmitting the repaired live video stream.

The following embodiment provides an application of the image restoration scheme to image recognition, namely, image restoration is performed before recognition is performed based on an image, so that the accuracy of image recognition is improved. Referring to fig. 8, a flowchart of an image recognition method according to a sixth embodiment of the present application is shown, where the method specifically includes the following steps:

601, acquiring an image to be identified;

step 602, determining a feature space level and an image restoration level applied to the image to be recognized based on application environment data of image recognition; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

step 603, extracting image features layer by layer until image features of the image to be repaired in the determined feature space level are obtained;

step 604, processing the image features extracted from each feature space level into combined image features;

step 605, performing image restoration level by level based on the combined image features until restoration at the determined image restoration level is completed;

and 606, identifying the target object of the repaired image to be identified.

Referring to fig. 9, a block diagram of an image processing apparatus according to a seventh embodiment of the present application is shown, which specifically may include:

a hierarchy determining module 701, configured to determine, based on application environment data of an image to be repaired, a feature space hierarchy and an image repair hierarchy applied to the image to be repaired; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

a feature extraction module 702, configured to perform image feature extraction layer by layer until image features of an image to be repaired in the determined feature space hierarchy are obtained;

a feature combination module 703, configured to process the image features extracted by each feature space level into combined image features;

and an image restoration module 704, configured to perform image restoration level by level based on the combined image feature until a restored image is obtained after restoration at the determined image restoration level is completed.

In an optional embodiment of the present application, the level determining module is specifically configured to determine a feature space level and an image restoration level applied to the image to be restored, based on at least one of processing performance data, storage performance data, application requirement data, and release requirement data of the image to be restored.

In an optional embodiment of the present application, the hierarchy determining module comprises:

the evaluation data determining unit is used for determining the hierarchical evaluation data of the image to be repaired based on the multiple application environment data and the corresponding weight coefficients;

and the numerical value determining unit is used for determining the characteristic space level and the image repairing level applied to the image to be repaired according to the numerical value range corresponding to the level evaluation data.

In an optional embodiment of the present application, the feature extraction module is specifically configured to, starting from a starting feature space level, perform image feature extraction layer by layer, and extract an image feature of a next feature space level according to an image feature extracted by a previous feature space level and the image to be repaired, until an image feature of the image to be repaired in the feature space level is obtained.

In an optional embodiment of the present application, the feature combination module is specifically configured to combine and connect image features extracted by each feature space level to obtain a combined image feature; or weighting and combining the extracted image features to obtain combined image features.

In an optional embodiment of the present application, the image repairing module is specifically configured to perform image repairing layer by layer from a starting image repairing level, and perform repairing of a next image repairing level according to an image repaired by a previous image repairing level until a repaired image is obtained after the repairing of the determined image repairing level is completed.

In an optional embodiment of the present application, the feature extraction module is specifically configured to invoke an image restoration model to perform image feature extraction layer by layer until an image feature of an image to be restored in the determined feature space hierarchy is obtained.

In an optional embodiment of the present application, the apparatus further comprises:

the model training module is used for training an image restoration model based on a plurality of sample image pairs; the sample image pair includes a low quality image and a corresponding high quality image.

In an optional embodiment of the present application, the image inpainting model includes a multi-level feature analysis unit, a feature processing unit, and a multi-level image inpainting unit, and the model training module is specifically configured to iteratively train the image inpainting model based on a plurality of sample image pairs; after each training, obtaining a repaired image based on the low-quality image and the initial image repairing model; determining correction coefficients respectively corresponding to the multi-level feature analysis unit, the feature processing unit and the multi-level image restoration unit according to the difference between the restored image and the high-quality image; and correcting the multi-level feature analysis unit, the feature processing unit and the multi-level image restoration unit based on the correction coefficient to obtain a corrected image restoration model.

and the image application module is used for performing at least one application of video restoration, image identification, commodity retrieval, three-dimensional image construction and image retrieval on the basis of the restored image.

In addition, because the plurality of feature space levels and the plurality of image restoration levels are set in the embodiment of the application, different levels are selected to correspond to different processing complexities, and further, in specific application, the levels are selected based on application environment data, so that the method is convenient and flexible to regulate and control, and is applied to scenes with different requirements. For example, for a simple scene or a situation of resource shortage, only the image features of the low feature space level may be extracted, and the image restoration is performed in combination with the low image restoration level to obtain a simple restoration result. Aiming at a complex scene or the condition of sufficient resources, the image features of a high feature space level can be extracted to obtain more comprehensive and fine image features, and the image restoration is carried out by combining a low image restoration level to obtain a high-quality restoration result.

Referring to fig. 10, a block diagram of an image restoration apparatus according to an eighth embodiment of the present application is shown, which may specifically include:

a hierarchy determining module 801, configured to determine, based on application environment data of an image to be repaired, a feature space hierarchy and an image repair hierarchy applied to the image to be repaired; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

an image restoration module 802, configured to obtain a restored target image under the determined feature space level and image restoration level based on an image restoration model; the image restoration model comprises a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit, wherein the multi-level feature analysis unit is used for carrying out image feature extraction layer by layer until the image features of the image to be restored in the determined feature space level are obtained, and the multi-level image restoration unit is used for carrying out image restoration layer by layer based on the extracted image features until the restored image is obtained after the restoration of the determined image restoration level is completed.

According to the embodiment of the application, a mechanism for performing feature extraction on an image to be repaired by using multiple feature space levels and combining the multiple image repairing levels to perform image repairing is adopted, in the framework, image features of different feature spaces are correspondingly extracted by different feature space levels, and the image features of the different feature spaces are correspondingly repaired by the different image repairing levels. When image processing is carried out, firstly, determining a characteristic space level and an image restoration level applied to an image to be restored based on application environment data of the image to be restored; extracting image features layer by layer until image features of the image to be restored in a feature space level are obtained; processing the image features extracted from each feature space level into combined image features; and performing image restoration level by level based on the combined image features until the restored image is obtained after restoration of the determined image restoration level is completed.

Compared with a scheme of degradation prior, the method does not need to perform degradation hypothesis in advance, reduces prior fitting errors caused by artificial introduction, is suitable for image restoration of various scenes, improves the generalization capability of general scenes, does not need to calculate the rule and the characteristic of the image in advance compared with a scheme of content prior, and can achieve a better restoration effect on all regions except a target object in the image. In addition, the model is not constrained according to prior information, so that the model parameters do not need to be adjusted according to the image degradation degree and the application scene, the images with different degradation degrees and different application scenes can be adaptively processed by a single model, and the convenience of deployment of the image restoration model is greatly improved.

Referring to fig. 11, a block diagram of a processing apparatus of an image restoration model according to the ninth embodiment of the present application is shown, and specifically, the processing apparatus may include:

a model training module 901, configured to iteratively train an image inpainting model based on a plurality of sample image pairs, where the sample image pairs include low-quality images and corresponding high-quality images; the image restoration model comprises a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit;

wherein the model training module 901 comprises:

the image restoration unit 9011 is configured to obtain a restored image based on the low-quality image and the image restoration model after each training;

a correction coefficient determining unit 9012, configured to determine, according to a difference between the repaired image and the high-quality image, correction coefficients corresponding to the multi-level feature analyzing unit, the feature processing unit, and the multi-level image repairing unit, respectively;

and the correcting unit 9013 is configured to correct the multi-level feature analyzing unit, the feature processing unit, and the multi-level image repairing unit based on the correction coefficient, so as to obtain a corrected image repairing model.

In an optional embodiment of the present application, the multi-level feature analysis unit is configured to perform image feature extraction layer by layer until an image feature of an image to be repaired in a determined feature space level is obtained, the feature processing unit is configured to process the extracted image feature into a combined image feature, and the multi-level image repairing unit is configured to perform image repairing layer by layer based on the combined image feature until a repaired image is obtained after the repair of the determined image repairing level is completed.

Referring to fig. 12, a block diagram of a product retrieval device according to a tenth embodiment of the present application is shown, which may specifically include:

a request receiving module 1001, configured to receive a product search request carrying a product reference image;

a hierarchy determining module 1002 configured to determine a feature space hierarchy and an image restoration hierarchy applied to the reference image of the commodity based on the search environment data; extracting image features of different feature spaces correspondingly from different feature space levels, and repairing the image features of different feature spaces correspondingly from different image repairing levels;

the feature extraction module 1003 is configured to perform image feature extraction layer by layer until image features of the commodity reference image in the feature space hierarchy are obtained;

a feature combination module 1004 for processing the image features extracted by each feature space level into combined image features;

an image restoration module 1005, configured to perform image restoration level by level based on the combined image features until restoration at the determined image restoration level is completed;

a result providing module 1006, configured to provide a product search result based on the repaired product reference image.

Referring to fig. 13, a block diagram of a live broadcast apparatus according to an eleventh embodiment of the present application is shown, which may specifically include:

a video frame determining module 1101, configured to collect a live video stream in transmission and determine a video frame to be repaired;

a hierarchy determining module 1102, configured to determine, based on application environment data of the video, a feature space hierarchy and an image restoration hierarchy applied to the video frame to be restored; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

the feature extraction module 1103 is configured to perform image feature extraction layer by layer until image features of the video frame to be restored in the feature space hierarchy are obtained;

a feature combination module 1104, configured to process the image features extracted by each feature space hierarchy into combined image features;

an image restoration module 1105, configured to perform image restoration level by level based on the combined image features until restoration at the determined image restoration level is completed;

a video stream updating module 1106, configured to obtain a repaired live video stream according to the repaired video frame;

a video streaming module 1107, configured to transmit the repaired live video stream.

According to the embodiment of the application, a mechanism for performing feature extraction on an image to be repaired by using multiple feature space levels and combining the multiple image repairing levels to perform image repairing is adopted, in the framework, image features of different feature spaces are correspondingly extracted by different feature space levels, and the image features of the different feature spaces are correspondingly repaired by the different image repairing levels. When image processing is carried out, firstly, a characteristic space level and an image restoration level applied to an image to be restored are determined based on application environment data of the image to be restored; extracting image features layer by layer until image features of the image to be restored in a feature space level are obtained; processing the image features extracted from each feature space level into combined image features; and carrying out image restoration level by level based on the combined image characteristics until the restoration of the determined image restoration level is completed, and obtaining a restored image.

Referring to fig. 14, a block diagram of an image recognition apparatus according to a twelfth embodiment of the present application is shown, which may specifically include:

an image obtaining module 1201, configured to obtain an image to be identified;

a hierarchy determining module 1202, configured to determine, based on application environment data of image recognition, a feature space hierarchy and an image restoration hierarchy applied to the image to be recognized; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

a feature extraction module 1203, configured to perform image feature extraction layer by layer until image features of the image to be repaired in the determined feature space hierarchy are obtained;

a feature combination module 1204, configured to process the image features extracted by each feature space hierarchy into combined image features;

an image restoration module 1205 for performing image restoration level by level based on the combined image features until restoration at the determined image restoration level is completed;

and the object identification module 1206 is used for identifying the target object of the repaired image to be identified.

For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.

Embodiments of the disclosure may be implemented as a system using any suitable hardware, firmware, software, or any combination thereof, in a desired configuration. Fig. 15 schematically illustrates an example system (or apparatus) 1300 that can be used to implement various embodiments described in this disclosure.

For one embodiment, fig. 11 illustrates an exemplary system 1300 having one or more processors 1302, a system control module (chipset) 1304 coupled to at least one of the processor(s) 1302, system memory 1306 coupled to the system control module 1304, non-volatile memory (NVM)/storage 1308 coupled to the system control module 1304, one or more input/output devices 1310 coupled to the system control module 1304, and a network interface 1312 coupled to the system control module 1306.

Processor 1302 may include one or more single-core or multi-core processors, and processor 1302 may include any combination of general-purpose or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the system 1300 can function as a browser as described in embodiments herein.

In some embodiments, system 1300 may include one or more computer-readable media (e.g., system memory 1306 or NVM/storage 1308) having instructions and one or more processors 1302, which in conjunction with the one or more computer-readable media, are configured to execute the instructions to implement modules to perform the actions described in this disclosure.

For one embodiment, the system control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and/or any suitable device or component in communication with the system control module 1304.

The system control module 1304 may include a memory controller module to provide an interface to the system memory 1306. The memory controller module may be a hardware module, a software module, and/or a firmware module.

System memory 1306 may be used, for example, to load and store data and/or instructions for system 1300. For one embodiment, system memory 1306 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, the system memory 1306 may include a double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

For one embodiment, system control module 1304 may include one or more input/output controllers to provide an interface to NVM/storage 1308 and input/output device(s) 1310.

For example, NVM/storage 1308 may be used to store data and/or instructions. NVM/storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and/or may include any suitable non-volatile storage device(s) (e.g., one or more Hard Disk Drives (HDDs), one or more Compact Disc (CD) drives, and/or one or more Digital Versatile Disc (DVD) drives).

NVM/storage 1308 may include storage resources that are physically part of the device on which system 1300 is installed or may be accessed by the device and not necessarily part of the device. For example, NVM/storage 1308 may be accessible over a network via input/output device(s) 1310.

Input/output device(s) 1310 may provide an interface for system 1300 to communicate with any other suitable device, input/output device(s) 1310 may include communication components, audio components, sensor components, and so forth. Network interface 1312 may provide an interface for system 1300 to communicate over one or more networks, and system 1300 may communicate wirelessly with one or more components of a wireless network according to any of one or more wireless network standards and/or protocols, such as access to a communication standard-based wireless network, such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof.

For one embodiment, at least one of the processor(s) 1302 may be packaged together with logic for one or more controllers (e.g., memory controller modules) of the system control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with logic for one or more controllers of the system control module 1304 to form a System In Package (SiP). For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with logic for one or more controller(s) of the system control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with logic of one or more controllers of the system control module 1304 to form a system on chip (SoC).

In various embodiments, system 1300 may be, but is not limited to being: a browser, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, etc.). In various embodiments, system 1300 may have more or fewer components and/or different architectures. For example, in some embodiments, system 1300 includes one or more cameras, a keyboard, a Liquid Crystal Display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an Application Specific Integrated Circuit (ASIC), and speakers.

Wherein, if the display includes a touch panel, the display screen may be implemented as a touch screen display to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also identify the duration and pressure associated with the touch or slide operation.

The present application further provides a non-volatile readable storage medium, where one or more modules (programs) are stored in the storage medium, and when the one or more modules are applied to a terminal device, the one or more modules may cause the terminal device to execute instructions (instructions) of method steps in the present application.

In one example, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method according to the embodiments of the present application when executing the computer program.

There is also provided in one example a computer readable storage medium having stored thereon a computer program, characterized in that the program, when executed by a processor, implements a method as one or more of the embodiments of the application.

The application discloses an image processing method, example 1 includes:

Example 2 includes the method of example 1, wherein determining, based on the application environment data of the image to be repaired, a feature space level and an image repair level to apply to the image to be repaired includes:

and determining a feature space level and an image restoration level applied to the image to be restored based on at least one of processing performance data, storage performance data, application demand data and release demand data of the image to be restored.

Example 3 includes the method of example 1, wherein determining, based on application environment data of an image to be inpainted, a feature space level and an image inpainting level to apply to the image to be inpainted comprises:

and determining a characteristic space level and an image repairing level applied to the image to be repaired according to the numerical range corresponding to the level evaluation data.

Example 4 includes the method of example 1, wherein performing image feature extraction level-by-level until obtaining image features of the image to be repaired in the determined feature space hierarchy comprises:

Example 5 includes the method of example 1, wherein the image features of the feature space include at least one of: image object marker information, image edge information, image texture information, image shading, image reflection information, image tone, image shading.

Example 6 includes the method of example 1, wherein processing the image features extracted by each feature space level into combined image features comprises:

Example 7 includes the method of example 1, wherein performing image inpainting level by level based on the combined image feature until inpainting at the determined image inpainting level is completed, obtaining an inpainted image comprises:

Example 8 includes the method of example 1, wherein performing image feature extraction level-by-level until obtaining image features of the image to be repaired in the determined feature space hierarchy comprises:

Example 9 includes the method of example 8, further comprising:

Example 10 includes the method of example 9, the image inpainting model comprising a multi-level feature parsing unit, a feature processing unit, a multi-level image inpainting unit, the training the image inpainting model based on the plurality of sample image pairs comprising:

determining correction coefficients corresponding to the multi-level feature analysis unit, the feature processing unit and the multi-level image restoration unit according to the difference between the restored image and the high-quality image;

Example 11 includes the method of example 1, further comprising:

The present application also discloses an image inpainting method, example 12 includes:

The present application further discloses a method for processing an image restoration model, example 13 includes:

after each training, obtaining a repairing image based on the low-quality image and the image repairing model;

Example 14 includes the method of example 13, the multi-level feature analysis unit to perform image feature extraction level by level until an image feature of an image to be repaired in the determined feature space level is obtained, the feature processing unit to process the extracted image feature into a combined image feature, and the multi-level image repair unit to perform image repair level by level based on the combined image feature until a repaired image is obtained after completing repair at the determined image repair level.

The present application also discloses a commodity retrieval method, example 15 includes:

receiving a commodity retrieval request carrying a commodity reference image;

The present application further discloses a live broadcasting method, example 16 includes:

acquiring a repaired live broadcast video stream according to the repaired video frame;

and transmitting the repaired live video stream.

The present application also discloses an image recognition method, example 17 includes:

acquiring an image to be identified;

and identifying the target object of the repaired image to be identified.

The present application also discloses an image processing apparatus, example 18 including:

The present application also discloses an image restoration apparatus, example 19 including:

the image restoration module is used for acquiring a restored target image under the determined characteristic space level and the image restoration level based on the image restoration model; the image restoration model comprises a multi-level feature analysis unit, a feature processing unit and a multi-level image restoration unit, wherein the multi-level feature analysis unit is used for carrying out image feature extraction layer by layer until the image features of the image to be restored in the determined feature space level are obtained, and the multi-level image restoration unit is used for carrying out image restoration layer by layer based on the extracted image features until the restored image is obtained after the restoration of the determined image restoration level is completed.

The present application also discloses an image inpainting model processing apparatus, example 20 includes:

wherein the model training module comprises:

The present application also discloses a commodity retrieval apparatus, example 21 including:

The present application also discloses a live device, example 22 comprising:

The present application also discloses an image recognition apparatus, example 23 including:

the image acquisition module is used for acquiring an image to be identified;

the level determining module is used for determining a characteristic space level and an image restoration level which are applied to the image to be recognized based on application environment data of image recognition; correspondingly extracting image features of different feature spaces by different feature space levels, and correspondingly repairing the image features of the different feature spaces by different image repairing levels;

The present application further discloses an electronic device, example 24 comprising: a processor; and

a memory having executable code stored thereon that, when executed, causes the processor to perform the method of any of examples 1-17.

The application also discloses one or more machine-readable media having executable code stored thereon, which when executed, causes a processor to perform the method of any of examples 1-17

Although certain examples have been illustrated and described for purposes of description, a wide variety of alternate and/or equivalent implementations, or calculations, may be made to achieve the same objectives without departing from the scope of practice of the present application. This application is intended to cover any adaptations or variations of the embodiments discussed herein. Therefore, it is manifestly intended that the embodiments described herein be limited only by the claims and the equivalents thereof.

Claims

1. An image processing method, comprising:

2. The method according to claim 1, wherein the determining a feature space level and an image inpainting level applied to the image to be inpainted based on application environment data of the image to be inpainted comprises:

3. The method according to claim 1, wherein the determining, based on the application environment data of the image to be repaired, a feature space level and an image repairing level applied to the image to be repaired comprises:

4. The method according to claim 1, wherein the performing image feature extraction level by level until obtaining image features of the image to be repaired in the determined feature space level comprises:

5. The method of claim 1, wherein the image features of the feature space comprise at least one of: image object marker information, image edge information, image texture information, image shading, image reflection information, image tone, image shading.

6. The method according to claim 1, wherein the processing the image features extracted at each feature space level into combined image features comprises:

7. The method of claim 1, wherein performing image inpainting level by level based on the combined image features until obtaining an inpainted image after inpainting at the determined image inpainting level is completed comprises:

8. The method according to claim 1, wherein the performing image feature extraction level by level until obtaining image features of the image to be repaired in the determined feature space level comprises:

9. The method of claim 8, further comprising:

10. The method of claim 9, wherein the image inpainting model comprises a multi-level feature parsing unit, a feature processing unit, and a multi-level image inpainting unit, and wherein training the image inpainting model based on the plurality of sample image pairs comprises:

after each training, obtaining a repairing image based on the low-quality image and the initial image repairing model;

11. The method of claim 1, further comprising:

12. An image restoration method, comprising:

13. A processing method of an image restoration model is characterized by comprising the following steps:

14. The method according to claim 13, wherein the multi-level feature analysis unit is configured to perform image feature extraction layer by layer until obtaining image features of an image to be repaired in the determined feature space level, the feature processing unit is configured to process the extracted image features into combined image features, and the multi-level image repairing unit is configured to perform image repairing layer by layer based on the combined image features until obtaining a repaired image after completing repairing of the determined image repairing layer.

15. A method for retrieving a commodity, comprising:

receiving a commodity retrieval request carrying a commodity reference image;

a product search result is provided based on the restored product reference image.

16. A live broadcast method, comprising:

acquiring a repaired live video stream according to the repaired video frame;

and transmitting the repaired live video stream.

17. An image recognition method, comprising:

acquiring an image to be identified;

determining a feature space level and an image restoration level applied to the image to be recognized based on application environment data of image recognition; extracting image features of different feature spaces correspondingly from different feature space levels, and repairing the image features of different feature spaces correspondingly from different image repairing levels;

and identifying the target object of the repaired image to be identified.

18. An image processing apparatus characterized by comprising:

19. An image restoration apparatus, comprising:

20. An apparatus for processing an image restoration model, comprising:

wherein the model training module comprises:

21. An article search device, comprising:

the characteristic combination module is used for processing the image characteristics extracted by each characteristic space hierarchy into combined image characteristics;

22. A live broadcast apparatus, comprising:

23. An image recognition apparatus, characterized by comprising:

the image acquisition module is used for acquiring an image to be identified;

24. An electronic device, comprising: a processor; and

a memory having executable code stored thereon that, when executed, causes the processor to perform the method of any of claims 1-17.

25. One or more machine-readable media having executable code stored thereon that, when executed, causes a processor to perform the method of any of claims 1-17.