CN113781309A - Image processing method and device and electronic equipment - Google Patents

Image processing method and device and electronic equipment Download PDF

Info

Publication number
CN113781309A
CN113781309A CN202111093837.1A CN202111093837A CN113781309A CN 113781309 A CN113781309 A CN 113781309A CN 202111093837 A CN202111093837 A CN 202111093837A CN 113781309 A CN113781309 A CN 113781309A
Authority
CN
China
Prior art keywords
image
mask
network model
edge
processed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111093837.1A
Other languages
Chinese (zh)
Other versions
CN113781309B (en
Inventor
贺沁雯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Kingsoft Cloud Network Technology Co Ltd
Original Assignee
Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Kingsoft Cloud Network Technology Co Ltd filed Critical Beijing Kingsoft Cloud Network Technology Co Ltd
Priority to CN202111093837.1A priority Critical patent/CN113781309B/en
Publication of CN113781309A publication Critical patent/CN113781309A/en
Application granted granted Critical
Publication of CN113781309B publication Critical patent/CN113781309B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/403—Edge-driven scaling; Edge-based scaling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/13—Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The application provides an image processing method, an image processing device and electronic equipment, wherein the method comprises the following steps: acquiring an image to be processed; carrying out edge detection on an image to be processed to obtain an edge image; respectively inputting the image to be processed into a CNN network model and a GAN network model for image processing to obtain a first image output by the CNN network model and a second image output by the GAN network model; the CNN network model and the GAN network model are both pre-trained image super-resolution network models or both image enhancement network models; and according to the edge image, carrying out image fusion on the first image and the second image to obtain a target image corresponding to the image to be processed. According to the method and the device, the output results of the CNN and GAN-based image super-resolution network model or image enhancement network model are fused by utilizing the image edge information to optimize the results, and the visual effect of super-resolution or image enhancement is improved.

Description

Image processing method and device and electronic equipment
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image processing method and apparatus, and an electronic device.
Background
The image super-resolution technology is used for reconstructing a corresponding high-resolution image according to a low-resolution image so as to improve the resolution and definition of the image; image enhancement techniques refer to enhancing useful information in an image to improve the visual appearance of the image. In the conventional image processing method, network models of the above two technologies are used, for example, a CNN (Convolutional Neural network) based image super-resolution network model, a GAN (generic adaptive network) based image super-resolution network model, a CNN based image enhancement network model, or a GAN based image enhancement network model. However, the CNN-based network model often cannot generate sufficiently realistic details when outputting a high-resolution image or a high-definition image, and the GAN-based network model often generates distortion in an image edge region when outputting a high-resolution image or a high-definition image.
Disclosure of Invention
The application aims to provide an image processing method, an image processing device and an electronic device, which can optimize the result by fusing the output results of a CNN and GAN-based image super-resolution network model or an image enhancement network model through an edge image, and improve the visual effect of super-resolution or image enhancement.
In a first aspect, an embodiment of the present application provides an image processing method, including: acquiring an image to be processed; carrying out edge detection on an image to be processed to obtain an edge image; respectively inputting the image to be processed into a CNN network model and a GAN network model for image processing to obtain a first image output by the CNN network model and a second image output by the GAN network model; the CNN network model and the GAN network model are both pre-trained image super-resolution network models or both image enhancement network models; and according to the edge image, carrying out image fusion on the first image and the second image to obtain a target image corresponding to the image to be processed.
In an optional embodiment, the step of performing edge detection on the image to be processed to obtain an edge image includes: carrying out edge detection on the image to be processed by utilizing a preset edge detection operator to obtain an edge image; the preset edge detection operator includes one of: a canny operator, a sobel operator, or a Laplace operator.
In an optional embodiment, the step of performing image fusion on the first image and the second image according to the edge image to obtain a target image corresponding to the image to be processed includes: determining a mask according to the edge image; and carrying out image fusion on the first image and the second image by using the mask to obtain a target image corresponding to the image to be processed.
In an alternative embodiment, the step of determining the mask according to the edge image includes: taking the edge image as a mask; or, expanding the adjacent pixel points by taking the target pixel points in the edge image as the reference; taking the expanded edge image as a mask; the target pixel point is a pixel point with a pixel value of 1 or 0.
In an optional embodiment, the step of expanding the adjacent pixel points by using the target pixel point in the edge image as a reference includes: determining a preset number of pixel points adjacent to the target pixel point in the edge image; and setting the pixel values of the preset number of pixel points to be the same as the pixel values of the target pixel points.
In an optional embodiment, the step of performing image fusion on the first image and the second image by using a mask to obtain a target image corresponding to the image to be processed includes: performing image fusion calculation according to the following formula to obtain a target image corresponding to the image to be processed:
result＝mask*A+(1-mask)*B；
wherein result represents the target image; mask represents a mask; A. b represents a first image and a second image, respectively; mask A represents that pixel points with pixel values of 1 in the mask take corresponding pixel values in A; and (1-mask) B represents that pixel points with pixel values of 0 in the mask take corresponding pixel values in B.
In an optional embodiment, the step of performing image fusion on the first image and the second image according to the mask to obtain a target image corresponding to the image to be processed further includes: performing image fusion calculation according to the following formula to obtain a target image corresponding to the image to be processed:
result＝mask*(k*A+(1-k)*B)+(1–mask)*(k*B+(1-k)*A)；
wherein result represents the target image; mask represents a mask; A. b represents the first image and the second image, respectively; k represents a fusion weight; mask (k A + (1-k) B) represents that the pixel value of the pixel point with the pixel value of 1 in the mask is k A + (1-k) B; (1-mask) × (k × B + (1-k) × a) indicates that the pixel value of the pixel having a pixel value of 0 in the mask is (k × B + (1-k) × a).
In a second aspect, an embodiment of the present application provides an image processing apparatus, including: the image acquisition module is used for acquiring an image to be processed; the edge detection module is used for carrying out edge detection on the image to be processed to obtain an edge image; the image processing module is used for respectively inputting the image to be processed into the CNN network model and the GAN network model for image processing to obtain a first image output by the CNN network model and a second image output by the GAN network model; the CNN network model and the GAN network model are both pre-trained image super-resolution network models or both image enhancement network models; and the image fusion module is used for carrying out image fusion on the first image and the second image according to the edge image to obtain a target image corresponding to the image to be processed.
In a third aspect, an embodiment of the present application further provides an electronic device, which includes a processor and a memory, where the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method in the first aspect.
In a fourth aspect, embodiments of the present application further provide a computer-readable storage medium storing computer-executable instructions that, when invoked and executed by a processor, cause the processor to implement the method of the first aspect.
The embodiment of the application brings the following beneficial effects:
in the image processing method, the image processing device and the electronic equipment, an edge image of an image to be processed is obtained firstly; then, image processing is carried out on the image to be processed through the CNN network model and the GAN network model respectively to obtain a first image output by the CNN network model and a second image output by the GAN network model; the CNN network model and the GAN network model are both pre-trained image super-resolution network models or both image enhancement network models; and finally, carrying out image fusion on the first image and the second image according to the edge image to obtain a target image corresponding to the image to be processed. According to the embodiment of the application, the output results of the CNN and GAN-based image super-resolution network model or image enhancement network model can be fused by utilizing the image edge information to optimize the results, and the visual effect of super-resolution or image enhancement is improved.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the detailed description of the present application or the technical solutions in the prior art, the drawings needed to be used in the detailed description of the present application or the prior art description will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of an image processing method according to an embodiment of the present application;
fig. 2 is an original image to be processed according to an embodiment of the present disclosure;
fig. 3 is an edge image obtained after edge extraction is performed by using a canny operator according to an embodiment of the present disclosure;
FIG. 4 is a flowchart of another image processing method provided in the embodiments of the present application;
fig. 5 is a block diagram of an image processing apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions of the present application will be described clearly and completely with reference to the following embodiments, and it should be understood that the described embodiments are some, but not all embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the existing image processing method, when a high-resolution image or a high-definition image is output, the CNN-based image super-resolution network model or the image enhancement network model often cannot generate details with sufficient fidelity, and when the high-resolution image or the high-definition image is output, the GAN-based image super-resolution network model or the image enhancement network model often generates distortion in an image edge area.
Based on this, embodiments of the present application provide an image processing method, an image processing apparatus, and an electronic device, which are capable of optimizing a result by fusing output results of a CNN and GAN-based image super-resolution network model or image enhancement network model by using image edge information, so as to improve a visual effect of super-resolution or image enhancement. To facilitate understanding of the present embodiment, a detailed description will be given first of all of an image processing method disclosed in the embodiments of the present application.
An embodiment of the present application provides an image processing method, referring to a flowchart of the image processing method shown in fig. 1, the method specifically includes the following steps:
step S102, acquiring an image to be processed. The image to be processed may be a photographed image or a certain frame image in a photographed video.
And step S104, carrying out edge detection on the image to be processed to obtain an edge image.
The edge image is an image obtained by extracting the edge of an original image, the edge is the junction of an image region and another attribute region, the region attribute is suddenly changed, the image is the place with the largest uncertainty and the place with the most concentrated image information, the edge of the image contains rich information, and the image quality can be effectively improved by fully utilizing the edge image.
In the embodiment of the application, edge detection can be performed on the image to be processed by using a preset edge detection operator, such as a canny operator, a sobel operator or a Laplace operator, so as to obtain an edge image. Taking canny detection operator as an example, fig. 2 is an image to be processed, and fig. 3 is an edge image obtained by performing edge extraction on the image to be processed by using canny operator. The edge image is a black-and-white image, the white pixel points are edge pixel points, and the black pixel points are non-edge pixel points.
And step S106, respectively inputting the image to be processed into the CNN network model and the GAN network model for image processing to obtain a first image output by the CNN network model and a second image output by the GAN network model.
The CNN network model and the GAN network model are both pre-trained image super-resolution network models or image enhancement network models, that is, if the CNN network model is the CNN image super-resolution network model, the corresponding GAN network model is the GAN image super-resolution network model; if the CNN network model is the CNN image enhancement network model, the corresponding GAN network model is the GAN image enhancement network model; the specific training process of the above model will be described in detail later, and will not be described in detail herein.
The image processing process comprises the following steps: respectively processing the images to be processed through the CNN image super-resolution network model and the GAN image super-resolution network model to obtain a first image and a second image which are respectively output by the CNN image super-resolution network model and the GAN image super-resolution network model; or, the images to be processed are processed through the CNN image enhancement network model and the GAN image enhancement network model respectively, so that a first image and a second image which are output by the CNN image enhancement network model and the GAN image enhancement network model respectively are obtained.
And S108, carrying out image fusion on the first image and the second image according to the edge image to obtain a target image corresponding to the image to be processed.
Specifically, the first image and the second image may be fused using the edge image as a mask, or the edge image may be pixel-expanded and the expanded image may be used as a mask for image fusion. The principle of image fusion by using masks is as follows: and taking the pixel point with the pixel value of 1 in the mask as the corresponding pixel value in the first image, and taking the pixel point with the pixel value of 0 in the mask as the corresponding pixel value in the second image.
Therefore, in the embodiment of the application, in the mask determined by the edge image, the pixel value corresponding to the edge pixel point is 1, and the first image and the second image are fused by using the mask determined by the edge image, so that in the obtained target image, the edge pixel point adopts the pixel value corresponding to the pixel point in the first image as much as possible, and the non-edge pixel point adopts the pixel value corresponding to the pixel point in the second image as much as possible, so that the advantage of better image edge performance of the CNN image super-resolution method and the advantage of better image edge performance of the GAN image super-resolution method in the texture-rich area can be synthesized, or the advantage of better image edge performance of the CNN image enhancement method and the advantage of better image edge performance of the GAN image enhancement method in the texture-rich area are synthesized, and the target image with better overall image effect can be obtained.
According to the image processing method provided by the embodiment of the application, firstly, an edge image of an image to be processed is obtained; then, image processing is carried out on the image to be processed through the CNN network model and the GAN network model respectively to obtain a first image output by the CNN network model and a second image output by the GAN network model; the CNN network model and the GAN network model are both pre-trained image super-resolution network models or both image enhancement network models; and finally, carrying out image fusion on the first image and the second image according to the edge image to obtain a target image corresponding to the image to be processed. According to the embodiment of the application, the output results of the CNN and GAN-based image super-resolution network model or image enhancement network model can be fused by utilizing the image edge information to optimize the results, and the visual effect of super-resolution or image enhancement is improved.
The embodiment of the application also provides an image processing method, which is realized on the basis of the method of the embodiment; the method mainly describes a step of performing image fusion on a first image and a second image according to an edge image and a training process of a network model, as shown in fig. 4, the step of image fusion includes:
step S402, determining a mask according to the edge image.
The mask is a binary image containing values of 1 or 0, the visualization result being an image containing only black and white, 0 being black and 1 being white, distinguishing different areas in the image.
The step of determining the mask according to the edge image includes the following two ways:
(1) taking the edge image as a mask; the edge image is detected by an edge detection operator, and the edge image only contains 0 and 1 pixel points, so that the mask requirement is met. As shown in fig. 3, an edge image obtained by performing edge extraction on an image to be processed by a canny operator is a black-and-white image, and edge pixel points in the image are white, that is, the values of the edge pixel points are 1, and the values of other pixel points are 0.
(2) Expanding adjacent pixel points by taking a target pixel point in the edge image as a reference; taking the expanded edge image as a mask; the target pixel point is a pixel point with a pixel value of 1 or 0.
The specific expansion mode is as follows: determining a preset number of pixel points adjacent to the target pixel point in the edge image; and setting the pixel values of the preset number of pixel points to be the same as the pixel values of the target pixel points. For example, the preset number is 2, and the target pixel point is a pixel point with a pixel value of 1, so that the extension process is to set the values of 2 consecutive adjacent pixel points of the pixel point with a pixel value of 1 in the edge image to 1. In this way, when the images are fused, the first image, namely the image output by the CNN model network can contribute more pixel values at the edge image of the target image, namely, the occupation ratio of the output image of the CNN model in the finally output target image is enlarged; relatively speaking, the effect at the edge image in the target image is better.
Otherwise, pixel points with the value of 0 in the edge image can be expanded, adjacent pixel points of the pixel points with the value of 0 are also set to be 0, and the proportion of the image output by the GAN network model in the finally output target image is enlarged; or other adjustment processes.
And S404, performing image fusion on the first image and the second image by using the mask to obtain a target image corresponding to the image to be processed.
Fusion can be performed in two ways:
the first method comprises the following steps: performing image fusion calculation according to the following formula to obtain a target image corresponding to the image to be processed:
result＝mask*A+(1-mask)*B；
wherein result represents the target image; mask represents a mask; A. b represents a first image and a second image, respectively; mask A represents that pixel points with pixel values of 1 in the mask take corresponding pixel values in A; and (1-mask) B represents that pixel points with pixel values of 0 in the mask take corresponding pixel values in B.
And the second method comprises the following steps: performing image fusion calculation according to the following formula to obtain a target image corresponding to the image to be processed:
result＝mask*(k*A+(1-k)*B)+(1–mask)*(k*B+(1-k)*A)；
wherein result represents the target image; mask represents a mask; A. b represents the first image and the second image, respectively; k represents a fusion weight; mask (k A + (1-k) B) represents that the pixel value of the pixel point with the pixel value of 1 in the mask is k A + (1-k) B; (1-mask) × (k × B + (1-k) × a) indicates that the pixel value of the pixel having a pixel value of 0 in the mask is (k × B + (1-k) × a).
For example, a pixel value of a certain pixel point a is 200, a pixel value of B is 100, and a mask is 1 at the point, and if the weight is 0.8, the corresponding pixel value is 0.8 × 200+ (1-0.8) × 100 ═ 180.
By setting the weight, the image fusion is carried out, and different fusion weights can be set according to actual conditions so as to obtain better image effect. This way the fusion of the first image and the second image can be made softer.
The CNN network model and the GAN network model are both pre-trained image super-resolution network models or both image enhancement network models; the specific training process is as follows:
(1) obtaining training samples
For the condition of training an image super-resolution network model based on CNN or GAN, a large number of low-resolution-high-resolution image pairs are required to be obtained as training samples; in the case of training the image enhancement network model based on CNN or GAN, a large number of low-definition-high-definition image pairs need to be acquired as training samples.
In particular implementations, a large number of high-definition images may be collected, down-sampled by interpolation, and JPEG-compressed or otherwise quantized with respect to the low-resolution images, introducing blocking, ringing or other distortions, to obtain a low-resolution-high-resolution image pair or a low-definition-high-definition image pair.
The blocking effect is mainly a phenomenon that a video image adopts a block-based coding mode and quantization to cause obvious difference between adjacent blocks, and the human eyes perceive discontinuity at a small block boundary in video coding; the ringing effect refers to that an image is filtered in image processing, and if a selected frequency domain filter has steep change, the filtered image generates ringing, and the ringing refers to oscillation generated at the position where the gray scale of the output image changes sharply, and is like air oscillation generated after a clock is knocked.
The difference method can select a nearest neighbor interpolation method, a bilinear interpolation method, a bicubic interpolation method and the like, or different interpolation methods are mixed, namely different interpolation methods are used for different high-definition samples.
(2) Construction of image super-resolution network model or image enhancement network model
And training the CNN and the GAN by using the prepared training samples. The CNN-based training image super-resolution network model is taken as an example for explanation, and the specific training process is as follows: and in the training stage, putting paired low-resolution-high-resolution images into the network, using the low-resolution images as CNN network input, utilizing the corresponding high-resolution images to constrain the output result of the CNN network, and learning parameters of each layer of the CNN network according to the mapping relation between the low-resolution images and the high-resolution images so that the CNN network learns to reconstruct the low-resolution images into the high-resolution images. The process of the super-resolution network model based on the GAN training image is similar, and only the learning network needs to be changed into the GAN, which is not described herein again. The process of training the image enhancement network model based on the CNN or GAN is similar to that, and only the training sample needs to be converted into a low-definition-high-definition image pair, which is not described herein again.
The image processing method provided by the embodiment of the application can perform image fusion on the first image output by the CNN network model and the second image output by the GAN network model according to the edge image, integrates the advantage of better image edge performance of the CNN image super-resolution method, integrates the advantage of better image edge performance of the GAN image super-resolution method in a texture-rich area, or integrates the advantage of better image edge performance of the CNN image enhancement method and the advantage of better image edge performance of the GAN image enhancement method in the texture-rich area, obtains a target image with better overall image effect, and can effectively improve the visual effect of super-resolution or image enhancement.
Based on the above method embodiment, an embodiment of the present application further provides an apparatus for image processing, and referring to fig. 5, the apparatus includes:
an image obtaining module 51, configured to obtain an image to be processed; the edge detection module 52 is configured to perform edge detection on the image to be processed to obtain an edge image; the image processing module 53 is configured to input the image to be processed into the CNN network model and the GAN network model respectively for image processing, so as to obtain a first image output by the CNN network model and a second image output by the GAN network model; the CNN network model and the GAN network model are both pre-trained image super-resolution network models or both image enhancement network models; and the image fusion module 54 is configured to perform image fusion on the first image and the second image according to the edge image to obtain a target image corresponding to the image to be processed.
The image processing device provided by the embodiment of the application can optimize the result by fusing the output results of the image super-resolution network model or the image enhancement network model based on the CNN and the GAN by using the image edge information, and improve the visual effect of super-resolution or image enhancement.
The edge detection module 52 is further configured to: carrying out edge detection on the image to be processed by utilizing a preset edge detection operator to obtain an edge image; the preset edge detection operator comprises one of the following operators: canny operator, sobel operator, Laplace operator.
The image fusion module 54 is further configured to: determining a mask according to the edge image; and carrying out image fusion on the first image and the second image by using the mask to obtain a target image corresponding to the image to be processed.
The image fusion module 54 is further configured to: taking the edge image as a mask; or, expanding the adjacent pixel points by taking the target pixel points in the edge image as the reference; taking the expanded edge image as a mask; the target pixel point is a pixel point with a pixel value of 1 or 0.
The image fusion module 54 is further configured to: determining a preset number of pixel points adjacent to the target pixel point in the edge image; and setting the pixel values of the preset number of pixel points to be the same as the pixel values of the target pixel points.
The image fusion module 54 is further configured to: performing image fusion calculation according to the following formula to obtain a target image corresponding to the image to be processed: result ═ mask a + (1-mask) B; wherein result represents the target image; mask represents a mask; A. b denote the first image and the second image, respectively.
The image fusion module 54 is further configured to: performing image fusion calculation according to the following formula to obtain a target image corresponding to the image to be processed: result ═ mask (k × a + (1-k) × B) + (1-mask) (k × B + (1-k) × a); wherein result represents the target image; mask represents a mask; A. b represents the first image and the second image, respectively; k represents a fusion weight; mask (k A + (1-k) B) represents that the pixel value of the pixel point with the pixel value of 1 in the mask is k A + (1-k) B; (1-mask) × (k × B + (1-k) × a) indicates that the pixel value of the pixel having a pixel value of 0 in the mask is (k × B + (1-k) × a).
The device provided by the embodiment of the present application has the same implementation principle and technical effect as those of the foregoing method embodiments, and for the sake of brief description, no mention is made in the embodiment of the device, and reference may be made to the corresponding contents in the foregoing method embodiments.
An electronic device is further provided in the embodiment of the present application, as shown in fig. 6, which is a schematic structural diagram of the electronic device, where the electronic device includes a processor 61 and a memory 60, the memory 60 stores computer-executable instructions that can be executed by the processor 61, and the processor 61 executes the computer-executable instructions to implement the method.
In the embodiment shown in fig. 6, the electronic device further comprises a bus 62 and a communication interface 63, wherein the processor 61, the communication interface 63 and the memory 60 are connected by the bus 62.
The Memory 60 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus 62 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 62 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 6, but that does not indicate only one bus or one type of bus.
The processor 61 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 61. The Processor 61 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory, and the processor 61 reads information in the memory and performs the steps of the method of the previous embodiment in combination with its hardware.
Embodiments of the present application further provide a computer-readable storage medium, where computer-executable instructions are stored, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to implement the method, and specific implementation may refer to the foregoing method embodiments, and is not described herein again.
The method, the apparatus, and the computer program product of the electronic device provided in the embodiments of the present application include a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the method described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
Unless specifically stated otherwise, the relative steps, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present application.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, and do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present application, and are used for illustrating the technical solutions of the present application, but not limiting the same, and the scope of the present application is not limited thereto, and although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope disclosed in the present application; such modifications, changes or substitutions do not depart from the spirit and scope of the exemplary embodiments of the present application, and are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. An image processing method, characterized in that the method comprises:
acquiring an image to be processed;
carrying out edge detection on the image to be processed to obtain an edge image;
respectively inputting the image to be processed into a CNN network model and a GAN network model for image processing to obtain a first image output by the CNN network model and a second image output by the GAN network model; the CNN network model and the GAN network model are both pre-trained image super-resolution network models or both image enhancement network models;
and according to the edge image, carrying out image fusion on the first image and the second image to obtain a target image corresponding to the image to be processed.
2. The method according to claim 1, wherein the step of performing edge detection on the image to be processed to obtain an edge image comprises:
carrying out edge detection on the image to be processed by utilizing a preset edge detection operator to obtain an edge image; the preset edge detection operator comprises one of the following: a canny operator, a sobel operator, or a Laplace operator.
3. The method according to claim 1, wherein the step of performing image fusion on the first image and the second image according to the edge image to obtain a target image corresponding to the image to be processed comprises:
determining a mask according to the edge image;
and carrying out image fusion on the first image and the second image by applying the mask to obtain a target image corresponding to the image to be processed.
4. The method of claim 3, wherein the step of determining a mask from the edge image comprises:
taking the edge image as a mask; alternatively, the first and second electrodes may be,
expanding adjacent pixel points by taking a target pixel point in the edge image as a reference; taking the expanded edge image as a mask; the target pixel point is a pixel point with a pixel value of 1 or 0.
5. The method of claim 4, wherein the step of expanding the neighboring pixels based on the target pixel in the edge image comprises:
determining a preset number of pixel points adjacent to the target pixel point in the edge image;
and setting the pixel values of the preset number of pixel points to be the same as the pixel values of the target pixel points.
6. The method according to claim 3, wherein the step of performing image fusion on the first image and the second image by using the mask to obtain the target image corresponding to the image to be processed comprises:
performing image fusion calculation according to the following formula to obtain a target image corresponding to the image to be processed:
result＝mask*A+(1-mask)*B；
wherein result represents the target image; mask represents a mask; A. b represents the first image and the second image, respectively; mask A represents that pixel points with pixel values of 1 in the mask take corresponding pixel values in A; and (1-mask) B represents that pixel points with pixel values of 0 in the mask take corresponding pixel values in B.
7. The method according to claim 3, wherein the step of performing image fusion on the first image and the second image according to the mask to obtain a target image corresponding to the image to be processed further comprises:
performing image fusion calculation according to the following formula to obtain a target image corresponding to the image to be processed:
result＝mask*(k*A+(1-k)*B)+(1–mask)*(k*B+(1-k)*A)；
wherein result represents the target image; mask represents a mask; A. b represents the first image and the second image, respectively; k represents a fusion weight; mask (k A + (1-k) B) represents that the pixel value of the pixel point with the pixel value of 1 in the mask is k A + (1-k) B; (1-mask) × (k × B + (1-k) × a) indicates that the pixel value of the pixel having a pixel value of 0 in the mask is (k × B + (1-k) × a).
8. An image processing apparatus, characterized in that the apparatus comprises:
the image acquisition module is used for acquiring an image to be processed;
the edge detection module is used for carrying out edge detection on the image to be processed to obtain an edge image;
the image processing module is used for respectively inputting the image to be processed into a CNN network model and a GAN network model for image processing to obtain a first image output by the CNN network model and a second image output by the GAN network model; the CNN network model and the GAN network model are both pre-trained image super-resolution network models or both image enhancement network models;
and the image fusion module is used for carrying out image fusion on the first image and the second image according to the edge image to obtain a target image corresponding to the image to be processed.
9. An electronic device comprising a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any of claims 1 to 7.
10. A computer-readable storage medium having computer-executable instructions stored thereon which, when invoked and executed by a processor, cause the processor to implement the method of any of claims 1 to 7.
CN202111093837.1A 2021-09-17 2021-09-17 Image processing method and device and electronic equipment Active CN113781309B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111093837.1A CN113781309B (en) 2021-09-17 2021-09-17 Image processing method and device and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111093837.1A CN113781309B (en) 2021-09-17 2021-09-17 Image processing method and device and electronic equipment

Publications (2)

Publication Number Publication Date
CN113781309A true CN113781309A (en) 2021-12-10
CN113781309B CN113781309B (en) 2024-06-28

Family

ID=78851893

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111093837.1A Active CN113781309B (en) 2021-09-17 2021-09-17 Image processing method and device and electronic equipment

Country Status (1)

Country Link
CN (1) CN113781309B (en)

Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104700371A (en) * 2015-03-18 2015-06-10 厦门美图之家科技有限公司 Generation method and system of masking
EP2919190A1 (en) * 2014-03-14 2015-09-16 Thomson Licensing Method and apparatus for alpha matting
CN109712095A (en) * 2018-12-26 2019-05-03 西安工程大学 A kind of method for beautifying faces that rapid edge retains
CN110288530A (en) * 2019-06-28 2019-09-27 北京金山云网络技术有限公司 A kind of pair of image carries out the processing method and processing device of super-resolution rebuilding
CN110310229A (en) * 2019-06-28 2019-10-08 Oppo广东移动通信有限公司 Image processing method, image processing apparatus, terminal device and readable storage medium storing program for executing
CN111080527A (en) * 2019-12-20 2020-04-28 北京金山云网络技术有限公司 Image super-resolution method and device, electronic equipment and storage medium
CN111080528A (en) * 2019-12-20 2020-04-28 北京金山云网络技术有限公司 Image super-resolution and model training method, device, electronic equipment and medium
CN111583116A (en) * 2020-05-06 2020-08-25 上海瀚正信息科技股份有限公司 Video panorama stitching and fusing method and system based on multi-camera cross photography
CN111612683A (en) * 2020-04-08 2020-09-01 西安万像电子科技有限公司 Data processing method and system
CN112184585A (en) * 2020-09-29 2021-01-05 中科方寸知微(南京)科技有限公司 Image completion method and system based on semantic edge fusion
CN112308770A (en) * 2020-12-29 2021-02-02 北京世纪好未来教育科技有限公司 Portrait conversion model generation method and portrait conversion method
CN112733681A (en) * 2020-12-31 2021-04-30 上海眼控科技股份有限公司 Image processing method, model training method, device, equipment and storage medium
WO2021121108A1 (en) * 2019-12-20 2021-06-24 北京金山云网络技术有限公司 Image super-resolution and model training method and apparatus, electronic device, and medium
WO2021139062A1 (en) * 2020-01-12 2021-07-15 大连理工大学 Full-automatic natural image matting method
KR20210096926A (en) * 2020-01-29 2021-08-06 안수환 Color Correction Method Using Low Resolution Color Image Of Large-capacity Aerial Orthoimage
WO2021169307A1 (en) * 2020-02-28 2021-09-02 北京旷视科技有限公司 Makeup try-on processing method and apparatus for face image, computer device, and storage medium
CN113344836A (en) * 2021-06-28 2021-09-03 展讯通信（上海）有限公司 Face image processing method and device, computer readable storage medium and terminal

Patent Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2919190A1 (en) * 2014-03-14 2015-09-16 Thomson Licensing Method and apparatus for alpha matting
CN104700371A (en) * 2015-03-18 2015-06-10 厦门美图之家科技有限公司 Generation method and system of masking
CN109712095A (en) * 2018-12-26 2019-05-03 西安工程大学 A kind of method for beautifying faces that rapid edge retains
CN110288530A (en) * 2019-06-28 2019-09-27 北京金山云网络技术有限公司 A kind of pair of image carries out the processing method and processing device of super-resolution rebuilding
CN110310229A (en) * 2019-06-28 2019-10-08 Oppo广东移动通信有限公司 Image processing method, image processing apparatus, terminal device and readable storage medium storing program for executing
WO2021121108A1 (en) * 2019-12-20 2021-06-24 北京金山云网络技术有限公司 Image super-resolution and model training method and apparatus, electronic device, and medium
CN111080528A (en) * 2019-12-20 2020-04-28 北京金山云网络技术有限公司 Image super-resolution and model training method, device, electronic equipment and medium
CN111080527A (en) * 2019-12-20 2020-04-28 北京金山云网络技术有限公司 Image super-resolution method and device, electronic equipment and storage medium
WO2021139062A1 (en) * 2020-01-12 2021-07-15 大连理工大学 Full-automatic natural image matting method
KR20210096926A (en) * 2020-01-29 2021-08-06 안수환 Color Correction Method Using Low Resolution Color Image Of Large-capacity Aerial Orthoimage
WO2021169307A1 (en) * 2020-02-28 2021-09-02 北京旷视科技有限公司 Makeup try-on processing method and apparatus for face image, computer device, and storage medium
CN111612683A (en) * 2020-04-08 2020-09-01 西安万像电子科技有限公司 Data processing method and system
CN111583116A (en) * 2020-05-06 2020-08-25 上海瀚正信息科技股份有限公司 Video panorama stitching and fusing method and system based on multi-camera cross photography
CN112184585A (en) * 2020-09-29 2021-01-05 中科方寸知微(南京)科技有限公司 Image completion method and system based on semantic edge fusion
CN112308770A (en) * 2020-12-29 2021-02-02 北京世纪好未来教育科技有限公司 Portrait conversion model generation method and portrait conversion method
CN112733681A (en) * 2020-12-31 2021-04-30 上海眼控科技股份有限公司 Image processing method, model training method, device, equipment and storage medium
CN113344836A (en) * 2021-06-28 2021-09-03 展讯通信（上海）有限公司 Face image processing method and device, computer readable storage medium and terminal

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
付耀衡;彭泳卿;刘鹏;冯红亮;: "基于图像处理的液位测量技术研究现状与发展趋势", 遥测遥控, no. 04, 15 July 2020 (2020-07-15), pages 22 - 31 *
毕玉萍;李劲华;赵俊莉;: "基于SRCGAN的超分辨率图像重建算法研究", 计算机应用与软件, no. 07, 12 July 2020 (2020-07-12), pages 154 - 158 *
邱佳梁;戴声奎;: "结合肤色分割与平滑的人脸图像快速美化", 中国图象图形学报, no. 07, 16 July 2016 (2016-07-16), pages 35 - 44 *

Also Published As

Publication number Publication date
CN113781309B (en) 2024-06-28

Similar Documents

Publication Publication Date Title
CN110324664B (en) Video frame supplementing method based on neural network and training method of model thereof
US11127117B2 (en) Information processing method, information processing apparatus, and recording medium
CN112102204B (en) Image enhancement method and device and electronic equipment
KR100912335B1 (en) Image processing apparatus, image processing method, and recording medium storing image processing program
CN111445424B (en) Image processing method, device, equipment and medium for processing mobile terminal video
JP2006050494A (en) Image photographing apparatus
Harb et al. Improved image magnification algorithm based on Otsu thresholding
CN110378893B (en) Image quality evaluation method and device and electronic equipment
CN113469897A (en) Training method and device of image enhancement model, image enhancement method and device and electronic equipment
CN111696064A (en) Image processing method, image processing device, electronic equipment and computer readable medium
CN112489063A (en) Image segmentation method, and training method and device of image segmentation model
Chang et al. Beyond camera motion blur removing: How to handle outliers in deblurring
CN110223277A (en) Method, apparatus, terminal device and the storage medium that image generates
CN113808021B (en) Image processing and model training method and device and electronic equipment
CN113781309B (en) Image processing method and device and electronic equipment
CN113781310A (en) Image processing method, and training method and device of image processing model
CN113298829B (en) Image processing method, device, electronic equipment and computer readable storage medium
CN112669204B (en) Image processing method, training method and device of image processing model
CN109727193B (en) Image blurring method and device and electronic equipment
CN114119376A (en) Image processing method and device, electronic equipment and storage medium
CN111125413B (en) Text picture filtering method and device and electronic equipment
Peng et al. A FPGA Verification of Improvement Edge Detection using Separation and Buffer Line
CN113674298A (en) Image segmentation method and device and server
KR100595693B1 (en) Method for converting image size in mobile terminal
JPH08265572A (en) Picture processor

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant