WO2021212810A1 - 图像处理方法、装置、电子设备及存储介质 - Google Patents
图像处理方法、装置、电子设备及存储介质 Download PDFInfo
- Publication number
- WO2021212810A1 WO2021212810A1 PCT/CN2020/127564 CN2020127564W WO2021212810A1 WO 2021212810 A1 WO2021212810 A1 WO 2021212810A1 CN 2020127564 W CN2020127564 W CN 2020127564W WO 2021212810 A1 WO2021212810 A1 WO 2021212810A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- sky
- image
- area
- target
- processed
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 34
- 238000003860 storage Methods 0.000 title claims abstract description 16
- 238000012545 processing Methods 0.000 claims abstract description 106
- 238000000034 method Methods 0.000 claims abstract description 48
- 238000001914 filtration Methods 0.000 claims abstract description 35
- 238000003709 image segmentation Methods 0.000 claims abstract description 34
- 239000000463 material Substances 0.000 claims abstract description 28
- 230000008569 process Effects 0.000 claims description 33
- 238000011156 evaluation Methods 0.000 claims description 23
- 238000009826 distribution Methods 0.000 claims description 15
- 230000004927 fusion Effects 0.000 claims description 12
- 238000007499 fusion processing Methods 0.000 claims description 5
- 230000003247 decreasing effect Effects 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 12
- 238000005516 engineering process Methods 0.000 description 13
- 238000003062 neural network model Methods 0.000 description 12
- 230000011218 segmentation Effects 0.000 description 12
- 238000004891 communication Methods 0.000 description 10
- 238000013473 artificial intelligence Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 8
- 238000012805 post-processing Methods 0.000 description 7
- 230000007704 transition Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000011946 reduction process Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 230000001133 acceleration Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000005520 cutting process Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 238000013467 fragmentation Methods 0.000 description 2
- 238000006062 fragmentation reaction Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000002829 reductive effect Effects 0.000 description 2
- 230000000717 retained effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000003796 beauty Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/143—Segmentation; Edge detection involving probabilistic approaches, e.g. Markov random field [MRF] modelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/35—Categorising the entire scene, e.g. birthday party or wedding scene
- G06V20/38—Outdoor scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Definitions
- This application relates to the field of image processing technology, in particular to image processing methods, devices, electronic equipment, and storage media.
- AI Artificial Intelligence matting technology
- AI portrait blurring a typical scene where AI matting technology is applied, and it is also a more challenging task.
- the outdoor photos taken by the user are all-encompassing, there will be branches, birds, buildings, wires, flags, windmills, glass, etc. in the sky. It is not easy to accurately separate the sky area from the non-sky area, especially on the branches. In a scene where there are subtle matters in the sky between the sky, the details of electric wires, etc., how to segment the sky area becomes extremely complicated.
- the current "change of sky” function is mainly to first determine the sky area through AI matting and then perform post-processing.
- segmentation effect of the traditional technology is poor, which leads to inaccurate determination of the sky area, and imperfect post-processing.
- There are defects such as unnatural sky edge transition, resulting in a low filming rate.
- the present application provides an image processing method, device, electronic device, and storage medium, so as to at least solve the problem that the sky area replacement rate is not high in the related art.
- the technical solution of this application is as follows:
- an image processing method which includes: performing image segmentation processing on a picture to be processed, and obtaining an initial mask image according to the result of the image segmentation processing; the initial mask image includes the to-be-processed image Each pixel in the picture belongs to the probability value of a pixel in the sky area; according to the initial mask image, it is determined whether the picture to be processed meets the preset sky area replacement condition, and if the sky area replacement condition is satisfied, the The grayscale image of the picture to be processed is a oriented image.
- an image processing device including: a mask map determining unit configured to perform image segmentation processing on a picture to be processed, and obtain an initial mask map according to a result of the image segmentation processing;
- the initial mask map contains the probability value that each pixel in the picture to be processed belongs to a pixel in the sky area;
- the guiding filter unit is configured to perform a determination according to the initial mask map whether the picture to be processed satisfies a preset If the sky area replacement condition is satisfied, use the grayscale image of the picture to be processed as the guiding image to perform guided filtering processing on the initial mask image to obtain a target mask image;
- sky scene acquisition unit Is configured to execute the acquisition of the target sky scene;
- the target sky scene is selected from preset sky scene materials;
- the sky area replacement unit is configured to execute according to the target mask map and the target sky The scene performs replacement processing on the sky area in the to-be-processed picture to obtain the first processed picture.
- an electronic device including a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to execute the instructions to achieve the above The described image processing method.
- a storage medium when instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the image processing method described above.
- a computer program product includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of the device reads from the readable storage medium The computer program is read and executed, so that the device executes the image processing method in the above embodiment.
- Fig. 1 is a diagram showing an application environment of an image processing method according to an exemplary embodiment
- Fig. 2 is a flowchart showing an image processing method according to an exemplary embodiment
- Fig. 3 is a schematic diagram showing a probability map according to an exemplary embodiment
- Fig. 4 is a picture to be processed according to an exemplary embodiment
- Fig. 5 is a processed picture according to an exemplary embodiment
- Fig. 6 is another processed picture according to an exemplary embodiment
- Fig. 7 is a flowchart showing an image processing method according to another exemplary embodiment
- Fig. 8 is a flowchart showing an image processing method according to another exemplary embodiment
- Fig. 9 is a block diagram showing an image processing device according to an exemplary embodiment.
- the image processing method provided by the embodiment of the present application can be applied to the electronic device as shown in FIG. 1.
- the electronic device can be various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices.
- the electronic device 100 may include one or more of the following components: a processing component 101, a memory 102, a power supply component 103, a multimedia component 104, an audio component 105, an input/output (I/O) interface 106, a sensor component 107, and Communication component 108.
- the processing component 101 generally controls overall operations of the electronic device 100, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations.
- the processing component 101 may include one or more processors 109 to execute instructions to complete all or part of the steps of the foregoing method.
- the processing component 101 may include one or more modules to facilitate the interaction between the processing component 101 and other components.
- the processing component 101 may include a multimedia module to facilitate the interaction between the multimedia component 104 and the processing component 101.
- the memory 102 is configured to store various types of data to support operations in the electronic device 100. Examples of these data include instructions for any application or method operating on the electronic device 100, contact data, phone book data, messages, pictures, videos, and the like.
- the memory 102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
- SRAM static random access memory
- EEPROM electrically erasable programmable read-only memory
- EPROM erasable Programmable read only memory
- PROM programmable read only memory
- ROM read only memory
- magnetic memory flash memory
- flash memory magnetic or optical disk.
- the power supply component 103 provides power for various components of the electronic device 100.
- the power supply component 103 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 100.
- the multimedia component 104 includes a screen that provides an output interface between the electronic device 100 and the user.
- the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user.
- the touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure related to the touch or slide operation.
- the multimedia component 104 includes a front camera and/or a rear camera. When the electronic device 100 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.
- the audio component 105 is configured to output and/or input audio signals.
- the audio component 105 includes a microphone (MIC).
- the microphone is configured to receive external audio signals.
- the received audio signal can be further stored in the memory 102 or sent via the communication component 108.
- the audio component 105 further includes a speaker for outputting audio signals.
- the I/O interface 106 provides an interface between the processing component 101 and a peripheral interface module.
- the above-mentioned peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: home button, volume button, start button, and lock button.
- the sensor component 107 includes one or more sensors for providing the electronic device 100 with various aspects of state evaluation.
- the sensor component 107 can detect the on/off status of the electronic device 100 and the relative positioning of the components.
- the component is the display and the keypad of the electronic device 100.
- the sensor component 107 can also detect the electronic device 100 or the electronic device 100.
- the position of the component changes, the presence or absence of contact between the user and the electronic device 100, the orientation or acceleration/deceleration of the electronic device 100, and the temperature change of the electronic device 100.
- the sensor assembly 107 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact.
- the sensor component 107 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications.
- the sensor component 107 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
- the communication component 108 is configured to facilitate wired or wireless communication between the electronic device 100 and other devices.
- the electronic device 100 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G, or 5G), or a combination thereof.
- the communication component 108 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel.
- the communication component 108 further includes a near field communication (NFC) module to facilitate short-range communication.
- the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
- RFID radio frequency identification
- IrDA infrared data association
- UWB ultra-wideband
- Bluetooth Bluetooth
- the electronic device 100 may be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field-available A programmable gate array (FPGA), controller, microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.
- ASIC application specific integrated circuits
- DSP digital signal processors
- DSPD digital signal processing devices
- PLD programmable logic devices
- FPGA field-available A programmable gate array
- controller microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.
- Fig. 2 is a flowchart showing an image processing method according to an exemplary embodiment. As shown in Fig. 2, the image processing method is used in the electronic device of Fig. 1 and includes the following steps.
- step S201 perform image segmentation processing on the picture to be processed, and obtain an initial mask map according to the result of the image segmentation processing; the initial mask map contains the probability value of each pixel in the picture to be processed belonging to a pixel in the sky area .
- the picture to be processed refers to the picture that needs to be replaced by the sky area, which can be a picture input by the user through the client (for example, a picture downloaded through the client, a picture taken by a camera on the client, etc.), which can be The picture obtained in advance can also be a picture obtained in real time, so it can also be called an input picture.
- the pictures to be processed can be pictures in various formats and scenes containing sky regions (the sky regions can be large or small).
- the sky area refers to an area belonging to the sky, and the sky can be a sky under a scene such as a sunny day, a cloudy day, after rain, evening, night, rainbow, and so on. In addition to the sky area, there can be other areas such as buildings and hills in the image to be processed.
- the size and shape of the picture to be processed can be of multiple types.
- the image segmentation processing of the picture to be processed may refer to the classification of the regions to which pixels in the picture belong, that is, the division of the image into disjoint regions.
- image segmentation technology has developed by leaps and bounds.
- the technology-related scene object segmentation, human body front background segmentation and other technologies have been obtained in unmanned driving, augmented reality, security monitoring and other industries. Wide range of applications.
- the image to be processed may be segmented by means of deep learning. For example, a neural network model is used to determine the probability value of each pixel in the picture to be processed belonging to the sky area, thereby realizing the sky area segmentation of the picture to be processed.
- the probability value can be selected according to the actual situation. The higher the value, the more likely the pixel is to belong to the sky area (that is, the pixel within the sky area). For example, a value of 1 indicates that the pixel definitely belongs to the sky area, a value of 0 indicates that the pixel definitely does not belong to the sky area, and a value of 0.9 indicates that the pixel is 90% likely to belong to the sky area.
- the probability value may be understood as the mask information of the corresponding pixel. Therefore, the probability value may be referred to as a mask value, and correspondingly, the initial mask image may also be referred to as a probability mask image.
- the probability value can be arranged according to the position of the pixel to obtain the initial mask map; in addition, the probability value of each pixel can also be converted into the corresponding gray scale For example, a probability value of 1 is converted into a gray value of 255, a probability value of 0.5 is converted to a gray value of 127, and a probability value of 0 is converted to a gray value of 0. In this way, the probability value can be output
- the electronic device can obtain the corresponding probability value after obtaining the gray value of the initial mask image.
- the initial mask image displayed in the interface can be as shown in Figure 3.
- step S202 it is determined whether the picture to be processed meets a preset sky area replacement condition according to the initial mask image, and if the sky area replacement condition is satisfied, the grayscale image of the picture to be processed is used as a guide map A guided filtering process is performed on the initial mask image to obtain a target mask image.
- the sky area replacement condition can be determined according to the size of the sky area, whether it is believed, the scene of the picture to be processed, and so on. In an optional implementation manner, it may be considered that the sky area replacement condition is not satisfied when the picture to be processed meets at least one of the sky area is too small, the picture to be processed is overwhelming, the night scene, and the foggy scene.
- the quantization process may make the difference of the probability value between adjacent pixels inaccurate, which will easily cause a sense of separation between the various areas of the picture, and the initial mask image and grayscale image Input to the guided filtering algorithm, which can fuse the information of the initial mask image and grayscale image, for example, if there is a large color gradient between the building and the sky in the input image, and the building in the initial mask image The probability value difference between the object and the sky is small, and the problem in the initial mask image can be corrected after the guided filtering.
- the guided filtering can also correct the color gradient in the initial mask image.
- This kind of problem Therefore, the effect of feathering the initial mask image can be achieved through the guided filtering process, so that the output processed image is closer to the input image, and a more realistic and natural changing sky effect is achieved.
- the initial mask image after the guided filtering process can be shown in the partial enlarged image on the right side of Figure 3.
- the algorithm can fuse the information of the two images to achieve the effect of feathering and gradual transition (gradual transition from black area to white Area), to prevent the appearance of a sense of fragmentation, so that the resulting picture will not be so blunt.
- step S203 a target sky scene is acquired; the target sky scene is selected from preset sky scene materials.
- the target sky scene refers to the type of sky that needs to be replaced
- the sky scene material can refer to scenes such as sunny, cloudy, after rain, rainbow, sunset, evening, night, etc.
- the target sky scene is selected from the sky scene materials, so that the subsequent pictures to be processed can be processed in a targeted manner.
- the target sky scene can be obtained according to scene selection information input by the user. For example, if the scene selection information is "sunset", the target sky scene can be a sunset scene.
- step S204 the sky area in the to-be-processed picture is replaced according to the target mask image and the target sky scene to obtain a first processed picture.
- the process of performing the replacement processing on the sky area can be as follows: determine the sky area in the target mask map, obtain the target sky scene, obtain the target sky map (that is, the sky picture used for replacement) according to the target sky scene, and pass the target sky
- the picture replaces the sky area, and the replaced picture can be used as the picture that has undergone the sky area replacement process, that is, the first processed picture.
- the target sky map may be a sky picture obtained from a sky material library according to the target sky scene. For example, if the target sky scene is sunset, the target sky map may be a sky picture in the sunset scene.
- the shape and size of the target sky map can be determined according to the shape and size of the sky area, and the shapes and sizes of the two can be the same or inconsistent (for example, the distance between the target sky map and the sides of the sky area is 50 pixels, that is, the target sky map is larger than the sky area).
- image segmentation is performed on the image to be processed, and an initial mask image containing the probability value of each pixel in the image to be processed belonging to the pixel point in the sky area is obtained according to the image segmentation processing result, and the initial mask image can be accurate Determine the sky area; if it is determined according to the initial mask image that the image to be processed satisfies the preset sky area replacement condition, use the gray image of the image to be processed as the guiding image to perform guided filtering on the initial mask image to obtain the target mask image,
- the guided filtering achieves the feathering effect on the initial mask image, and can effectively correct the color gradient between adjacent image regions in the initial mask image; making the processed image change effect obtained from the target mask image real and natural.
- the film rate is high.
- the step of using the grayscale image of the picture to be processed as a oriented image to perform oriented filtering processing on the initial mask image to obtain a target mask image includes: acquiring the to-be-processed image The blue channel pixel value of each pixel in the picture; the pixel with the blue channel pixel value in the initial mask in the target distribution interval and the probability value greater than the first threshold is determined as the first pixel; wherein
- the target distribution interval is the interval in which the number of blue channel pixel values of each pixel in the first evaluation area is the largest among a plurality of preset intervals, and the first evaluation area is that the probability value in the initial mask is greater than the first
- the target blue channel pixel value is the minimum value of the blue channel pixel value of each pixel in the second evaluation area, and the
- the first evaluation area can correspond to an area that is most likely to be a sky area in the initial mask. Therefore, the size of the second threshold can be a larger value to ensure the accuracy of the selected first evaluation area as much as possible. sex. In an optional implementation manner, the second threshold may be 0.9, 0.92, and so on. Further, after the first evaluation area is determined, the blue channel pixel value of each pixel in the first evaluation area can be classified into a set interval (the interval can be divided from 0 to 255 according to the actual situation), of which the number is the largest The interval of is determined as the target distribution interval.
- the first pixel point can be determined in combination with the target distribution interval and the probability value.
- the probability value in the initial mask image is greater than a preset threshold (which can be determined according to the actual situation, such as 0.5) and the blue channel pixel value is in the target
- the area where the pixels in the distribution interval are located is determined as the first pixel.
- a more accurate first pixel can be determined by combining the blue channel pixel value and the probability value, thereby improving the accuracy of the sky area replacement.
- the blue channel pixel value is the value corresponding to the B (Blue) channel in the RGB value.
- the probability value reduction processing may also be performed on the area that is highly likely to be not the sky area.
- the second evaluation area can correspond to the area in the initial mask image that may be the sky area.
- the third threshold can be set to a higher value (or even It can be greater than the second threshold.
- the third threshold can be 0.93, 0.95, etc.), so as to take the area into consideration as much as possible to determine the target blue channel pixel value from it, and use the target blue channel The upper limit of the pixel value determines the second pixel point in the initial mask image.
- the target blue channel pixel value corresponding to the second evaluation area can be understood as the lowest critical point of the blue channel pixel value in the sky area. If the blue channel pixel value of a certain pixel is less than the critical point, it is considered that the pixel does not belong to the sky area. Then the probability value reduction process is performed.
- the second pixel point determined according to the target blue channel pixel value can effectively prevent the non-sky area from being missed and classified into the sky area.
- the second threshold and the third threshold may also be the same size, and even the third threshold is smaller than the second threshold.
- the processing of increasing the probability value can refer to assigning the probability value to a larger value, such as: 1, 0.99, 0.98, etc.
- the processing of reducing the probability value can refer to assigning the probability value to a smaller value, for example: 0.5 , 0.6, etc.
- a pixel with a probability value of 0.9 if it is determined that it is most likely to belong to the sky area according to the blue channel pixel value (for example, the value corresponding to the blue channel B is higher), then its probability value can be assigned to 1; and
- the probability value can be halved or reduced to 0.
- the process of halving the probability value of the first pixel can also be replaced by a uniform subtraction of a certain probability value (for example, uniform subtraction of 0.1, 0.2, etc.).
- the probability value increase processing is performed on the pixels that are most likely to be the sky area
- the probability value reduction processing is performed on the pixels that are most likely not the sky area, which can highlight the probability value of the sky area for subsequent replacement of the sky area. This part of the area can be accurately identified at the time, and then the sky can be changed to improve the accuracy of the sky.
- the step of increasing the probability value of the first pixel and reducing the probability value of the second pixel includes: setting the probability value of the first pixel Is 1; the probability value of the second pixel is halved.
- the implementation process of adding and reducing pixels can be: calculating the histogram of the blue channel pixel value of each pixel in the area of the probability value> 0.9 of the picture to be processed, and calculating according to the histogram
- the blue channel pixel value is in which of the following intervals (Q1:0-63, Q2:64-127, Q3:128-191, Q4:192-255), the interval containing the most pixels is regarded as the target distribution interval, which is recorded as Qi, the probability value in the target distribution interval that the probability value is greater than 0.5 and the blue channel value belongs to the Qi interval is set to 1.0.
- the way of dividing the interval can be adjusted according to the actual situation, for example, it can be divided into larger or smaller intervals.
- the second pixel that is most likely not the sky area is halved, which can highlight the probability value of the sky area for subsequent sky areas. This part of the area can be accurately identified when replacing, and then the sky is changed to improve the accuracy of the sky.
- the red channel pixel value can be used to implement it.
- Such a processing manner can determine a more accurate scene for the scene in an optional implementation manner. Accurate sky area, and then achieve accurate sky area replacement operation, improve the filming rate.
- the probability value reduction process may be performed on the second pixel first, and then the probability value increase process of the first pixel point is performed on the probability map after the probability value reduction process to obtain the candidate mask map; It is also possible to perform the process of increasing the probability value of the first pixel on the initial mask image, perform the process of reducing the probability value of the second pixel on the initial mask image, and integrate the probability maps obtained by the two processing methods (for example: For the pixels that have not changed in the initial mask image, the original probability value is retained; for those that have only been processed for increasing the probability value or processed with only a small probability value, the processed probability value is retained; for both the probability value If the value increase processing is performed with the probability value reduction processing, the average value of the probability values obtained by the two processing methods can be taken to obtain the candidate mask map. Afterwards, the candidate mask image is subjected to guided filtering processing using the gray image of the picture to be processed as the guided image to obtain the target mask image.
- the step of performing replacement processing on the sky area in the picture to be processed according to the target mask map and the target sky scene to obtain the first processed picture includes: determining all The non-sky area in the picture to be processed is used as the foreground image; the sky material image is cropped according to the size of the target sky scene and the sky area, and the scene corresponds to the target sky scene and the size is the same as the sky area
- the target sky map corresponding to the size of the target sky map; the foreground map and the target sky map are combined according to the target mask map to obtain the first processed picture; wherein the sky in the first processed picture The area is replaced by the target sky map.
- determining the non-sky area in the picture to be processed can be to determine the pixels with a probability value lower than 0.85 (or other values or combined with the blue color channel value) as non-sky area pixels, and for these non-sky areas Integrating regional pixels (for example, integrating scattered points into a complete area) can obtain a non-sky area.
- the process of obtaining the target sky map may be: determining the smallest rectangular bounding box of the sky area, obtaining candidate sky material images corresponding to the target sky scene from the sky material images, and maintaining the aspect ratio In the case of, crop out the target sky image in the candidate sky material image.
- a center cutting method can be used (of course, other methods can also be used), for example: determining the center point of the candidate sky material map, and cutting the center point as the center to a rectangle surrounded by The box size gets the target sky map.
- the candidate sky material image can be scaled and then cropped; of course, it can also be cropped in the candidate sky material image first, and then the cropped image can be scaled to obtain the target sky. picture.
- the realization process of combining the foreground image and the target sky image according to the target mask image to obtain the processed image can be: in a blank area, determine an area consistent with the size of the image to be processed as the area to be filled; according to the target mask The map determines the sky area in the area to be filled, fills the target sky map into the sky area, and fills the foreground image into the remaining area to obtain the processed image.
- the sky area is determined in the picture to be processed according to the target mask map, the target sky map is filled in the sky area, and the foreground image is filled in the remaining area to obtain the processed picture.
- the probability values in the overlapping area can be integrated to obtain a final probability value, and then the area filling can be completed according to this final probability value, for example , If it is determined to be a sky area (for example, the probability value is greater than 0.5), the target sky map is filled, and if it is determined to be a non-sky area, the foreground map is filled. Assuming that the picture to be processed is shown in FIG. 4, the area formed by two rectangular frames represents the sky area 401, and the sky area 401 is currently a rainbow scene. After replacing it with a sunny scene, it can be shown in Figure 5.
- the sky area has been replaced with the target sky map, that is, the sky area replacement process is realized, and the buildings (foreground map) in it are not replaced.
- the above process combines the sky area and the foreground map. Differentiating processing can effectively prevent the aliasing of the picture content, and ensure the clarity of the processed pictures obtained while meeting the needs of users.
- the step of performing replacement processing on the sky area in the to-be-processed picture according to the target mask map and the target sky scene to obtain the first processed picture includes : According to the target mask map, determine the first area, the second area, and the remaining area from the to-be-processed picture; wherein the probability value of the pixels contained in the first area is 1, and the first area The probability value of the pixel points contained in the second area is 0, and the remaining area is the area in the picture to be processed excluding the first area and the second area; replacing the first area with the Target sky map; replace the second area with the foreground image; compare the foreground image and the target according to the probability value, red channel pixel value, green channel pixel value, and blue channel pixel value corresponding to the remaining area
- the sky map performs color channel information fusion of pixels; and obtains the first processed picture according to the target sky map after the color channel information fusion processing.
- the first area may refer to an area that is most likely or definitely a sky area
- the second area may refer to an area that is most likely or definitely not a sky area.
- the probability value can also be replaced with other values, such as 0.98, 0.99, etc.; in the case of the above probability value of 0, the probability value can also be replaced with other values, for example: 0.02 , 0.01 and so on.
- the color channel information can refer to the values corresponding to the three RGB channels of the picture, and the color of the corresponding pixel can be obtained through the color channel information. Further, the color channel information may be a value corresponding to a certain channel or multiple channels. In an optional implementation manner, it may include a red channel pixel value, a green channel pixel value, and a blue channel pixel value.
- the fusion of the foreground image and the target sky image can be to perform arithmetic processing on the color channel information corresponding to the two parts, such as: multiply processing, gamma transformation processing, etc., and use the processed color channel information as the corresponding pixel
- the color channel information can be all or part of the RGB value.
- multiplying can be multiplying the RGB values of the corresponding pixels of the foreground image and the target sky image
- the gamma transformation processing can be the exponentiation processing of the RGB values of the pixels corresponding to the foreground image and the target sky image.
- the step of combining the foreground image and the target sky image according to the target mask image to obtain the first processed image includes: comparing all the images according to the target sky scene At least one of the brightness, contrast, and saturation of the foreground image is adjusted to obtain a target foreground image whose brightness, contrast, and saturation match the target sky scene; The foreground image and the target sky image are combined to obtain the first processed image.
- the first processed picture obtained through such processing can adapt to the style of the target sky scene.
- the foreground image can be filtered and beautified to obtain the target foreground image.
- the implementation process of this embodiment may be:
- the target sky map and the foreground map are fused according to the probability value, and the mixRGB of a certain pixel A obtained by fusion is as follows:
- mixRGB src*(1.0-mask)+sky*mask; where src represents the RGB value of pixel A in the foreground image, mask represents the probability value of pixel A in the reference probability map, and sky represents the pixel in the target sky image.
- the RGB value of point A is the RGB value of point A.
- tempRGB sqrt(mixRGB*sky,0.5); among them, 0.5 is a preset parameter, which can also be set to other values as needed.
- the other pixels are merged in the same way as above.
- the area that is definitely the sky area is replaced with the target sky map, and the area that is definitely not the sky area is replaced with the foreground map.
- the foreground map and the target sky map are If the area in between is not processed, there will be a sense of fragmentation that the foreground suddenly changes to the sky area.
- the area in the middle part (the area that is neither the foreground image nor the target sky image) is merged, and the merged area is integrated
- the color channel information of the sky part and the non-sky part can be ensured that the foreground transitions to the sky naturally, and a more realistic and natural picture can be obtained.
- the step of correcting the processed image may be included, for example, comparing the processed image with the image to be processed, if the foreground image or the target sky image is If there is a position or angle deviation, you can adjust it. If there is an unnatural edge transition, you can adjust it. In this way, the final output processed image can have higher accuracy and sky area replacement effect, and the filming rate can be improved.
- the step of performing image segmentation processing on the picture to be processed, and obtaining an initial mask image according to the result of the image segmentation processing includes: performing image segmentation processing on the picture to be processed through a pre-trained neural network model , Obtain the image segmentation processing result; determine the probability value of each pixel in the to-be-processed picture belonging to a pixel in the sky area according to the image segmentation processing result; obtain the probability value of each pixel in the to-be-processed picture The initial mask map.
- the neural network model may be a CNN (Convolutional Neural Network) model or the like.
- the neural network may be u-net, u-net variant, ic-net, deeplab series, and so on.
- the step of performing image segmentation processing on the picture to be processed through a pre-trained neural network model includes: scaling the size of the picture to be processed to a preset size; The picture to be processed after the size scaling is subjected to normalization processing; the picture to be processed after the normalization processing is subjected to image segmentation processing through a pre-trained neural network model.
- the preset size can be set according to the actual situation, for example: 512*512, 768*768, etc.
- the normalization process can be: obtain picture samples, determine the mean and variance of the RGB value of each pixel of these picture samples, and subtract the mean and divide by the variance of the RGB value of each pixel of the picture to be processed for better performance
- Machine learning and feature learning are used to classify whether each pixel in the image to be processed belongs to the sky area pixel to obtain a probability map.
- the neural network model can be obtained through pre-determined training images.
- the trained neural network model can be down-sampled to process the image to be processed, and then extract the feature information, analyze the feature information, and output the image segmentation processing result.
- the computer can determine the probability value that each pixel belongs to the sky area according to the image segmentation processing result, and then obtain the initial mask image.
- the neural network model fully analyzes the feature information in the image to be processed. Compared with the traditional graph cut image segmentation method, this machine learning method can more accurately segment the image to be processed, and then obtain an accurate probability map.
- the picture to be processed when the picture to be processed is obtained, it is preprocessed first, and then the neural network model is used to segment the network to obtain the initial mask image, and then the sky is accurately distinguished from the image to be processed according to the initial mask image.
- Region can be understood as a post-processing process
- the segmentation network and post-processing are combined to retain the accurate segmentation effect of the neural network model.
- accurate sky and non-sky areas are post-processed to make the final The sky area replacement processing is more accurate.
- the step of determining whether the picture to be processed meets a preset sky area replacement condition according to the initial mask image includes: determining the picture to be processed according to the initial mask image Whether it meets at least one of the following, if it meets at least one of the following, it is determined that the to-be-processed picture does not meet the sky area replacement condition; if it does not meet any of the following, it is determined that the to-be-processed picture meets the sky area replacement condition: determine the to-be-processed picture The first proportion of the mid-sky area, if the first proportion is less than the preset fourth threshold, it is determined that the sky area is too small; the second proportion of the enormous area in the image to be processed is determined, if the If the second proportion is greater than the preset fifth threshold, it is determined to be enormous; wherein, the enormous area is the area where the probability value of each pixel is in the middle interval, and the middle interval is determined by the middle of the probability value.
- the average brightness of the sky area in the picture to be processed determines the average brightness of the sky area in the picture to be processed, and if the average brightness is less than the preset sixth threshold, determine it as a night scene; determine the picture to be processed
- the third proportion of the target dark channel area if the third proportion is greater than the preset seventh threshold, it is determined to be a foggy scene; wherein, the target dark channel area is the sky area where the dark channel value is less than eighth The area where the threshold pixel is located.
- the size of the fourth, fifth, sixth, seventh, and eightth thresholds can be determined according to actual conditions.
- the fourth threshold can be 0.9
- the fifth threshold can be 0.1
- the sixth threshold can be 0.3cd/m 2
- the seventh The threshold may be 0.4
- the eighth threshold may be 0.8.
- a pixel with a probability value of 1 can be understood as definitely belonging to the sky area
- a pixel with a probability value of 0 can be understood as definitely not belonging to the sky area
- a pixel with an intermediate probability value (for example: 0.5) may belong to
- the sky area may not belong to the sky area, and this part of the pixels belongs to the uncertain area, that is, the overwhelming area.
- the disbelief area may be an area where the probability value falls between 0.3 and 0.7 (the boundary value of the interval may also be other values).
- the area with a probability value greater than 0.9 can be extracted from the probability map obtained in the current step, as the sky area, different implementations
- the sky area in the example can be the same or different.
- the realization process of determining the average brightness of the sky area in the picture to be processed may be: calculating the average gray value of the area of the original image (picture to be processed) corresponding to the area with the probability value greater than 0.9 in the initial mask image.
- the dark channel value of the sky area (for RGB images, the dark channel value is the minimum of the three values of RGB, for example, if R/G/B is 10/20/30 respectively, the dark channel value is 10)
- a dark channel map can be obtained, and then based on the dark channel map, it is determined whether the picture to be processed is a foggy scene (if the proportion of the area with a dark channel value less than a certain threshold is greater than the seventh threshold, it is considered to be a foggy day).
- the third proportion of the target dark channel area can be understood as the statistical information of the dark channel value.
- the implementation process of the foregoing embodiment may be:
- any one of the above conditions is met, it can be determined that the image to be processed is not suitable for the replacement of the sky area, that is, it is not suitable for changing the sky, otherwise, if the above conditions are not met (or most of them are not met), it can be considered suitable for changing Day, then the subsequent day change processing will be performed.
- the method further includes: if the sky area replacement condition is not satisfied , Acquiring a target filter tool according to the target sky scene, and performing filter processing on the picture to be processed through the target filter tool to obtain a second processed picture.
- the following processing can be performed: obtain a predetermined target sky scene; select a comprehensive filter corresponding to the target sky scene; The integrated filter processes the picture to be processed to obtain a second processed picture.
- an image processing method including the following steps:
- S703 Obtain the blue channel pixel value of each pixel in the picture to be processed.
- S704 Determine a pixel with a blue channel pixel value in the initial mask image in the target distribution interval and with the probability value greater than a first threshold as the first pixel.
- S705 Determine a pixel with a blue channel pixel value less than a target blue channel pixel value in the initial mask image as a second pixel.
- Channel pixel values perform pixel color channel information fusion on the foreground image and the target sky map; obtain the first processed picture according to the target sky map after color channel information fusion processing.
- S712 Obtain a target filter tool according to the target sky scene, and perform filter processing on the to-be-processed picture through the target filter tool to obtain a second processed picture.
- image segmentation is performed on the picture to be processed to accurately obtain an initial mask map containing the probability value of each pixel in the picture to be processed belonging to the sky area; according to the blue channel pixel value of each pixel in the picture to be processed, The pixel points corresponding to the sky area in the initial mask image are increased by the probability value process to obtain a reference probability map.
- the resulting reference probability map further expands the difference in probability values between the sky area and the non-sky area; according to the reference probability map Accurately identify the sky area in the picture to be processed. Therefore, the processed picture obtained from the replacement process of the sky area has high accuracy, and the effect of changing the sky is natural, and the rate of changing the sky is high.
- an image processing method is provided.
- the method is applied to a mobile terminal and includes the following steps:
- Send to the segmentation network for processing send the preprocessed picture to be processed into the segmentation network to obtain a probability map.
- Crop the sky material calculate the minimum rectangular bounding box of the sky area in the picture to be processed according to the feathering probability map, and cut and scale the sky material to the size of the rectangular bounding box at the center while maintaining the aspect ratio;
- (4)Adaptively adjust the foreground adjust the brightness, contrast, and saturation of the foreground image in the image to be processed according to the target sky scene to match the style of the material. Then use the foreground filter corresponding to the target sky scene to beautify to obtain the final used foreground image;
- Segmentation fusion fusion adjusted foreground image and cropped sky material according to the feathering probability map
- the new sky area is merged with the original image background, using the method of segmented layer aliasing and segmented linear fusion, so that the transition of the synthesized image between the sky and the non-sky is real and natural;
- the non-sky area namely the foreground image
- the final overall color is adjusted to ensure the uniformity and beauty of the final rendering
- Fig. 9 is a block diagram showing an image processing device 900 according to an exemplary embodiment.
- the device includes a mask map determining unit 901, a guidance filtering unit 902, a sky scene acquisition unit 903, and a sky area replacement unit 904.
- the mask map determining unit 901 is configured to perform image segmentation processing on the picture to be processed, and obtain an initial mask map according to the result of the image segmentation processing; the initial mask map includes that each pixel in the picture to be processed belongs to the sky area The probability value of the inner pixel.
- the guiding filtering unit 902 is configured to determine whether the picture to be processed meets a preset sky area replacement condition according to the initial mask map, and if the sky area replacement condition is satisfied, use the gray scale of the picture to be processed
- the figure is a directional graph that performs directional filtering processing on the initial mask graph to obtain a target mask graph.
- the sky scene acquisition unit 903 is configured to perform acquisition of a target sky scene; the target sky scene is selected from preset sky scene materials.
- the sky area replacement unit 904 is configured to perform replacement processing on the sky area in the to-be-processed picture according to the target mask map and the target sky scene, to obtain a first processed picture.
- the guiding filtering unit includes: a pixel value obtaining subunit configured to perform obtaining the blue channel pixel value of each pixel in the picture to be processed; a first pixel determining subunit, Is configured to execute the determination of the pixel points with the blue channel pixel value in the initial mask image in the target distribution interval and the probability value greater than the first threshold as the first pixel point; wherein, the target distribution interval is a preset Among the multiple intervals, the interval with the largest number of blue channel pixel values of each pixel in the first evaluation area, where the first evaluation area is the area where the pixels with the probability value greater than the second threshold in the initial mask map are located The second threshold is greater than the first threshold; a second pixel point determining sub-unit is configured to perform the determination of a pixel with a blue channel pixel value less than the target blue channel pixel value in the initial mask as the second Pixel; the target blue channel pixel value is the minimum value of the blue channel pixel value of each pixel in the second evaluation area, and
- the guided filtering subunit includes: a first probability value setting module configured to perform setting the probability value of the first pixel point to 1; and a second probability value setting module configured to It is configured to perform the halving of the probability value of the second pixel point.
- the sky area replacement unit includes: a foreground image determining subunit, configured to perform determining a non-sky area in the picture to be processed, as a foreground image; and a sky material cropping subunit, which is It is configured to perform cropping of the sky material image according to the size of the target sky scene and the sky area to obtain a target sky image with a scene corresponding to the target sky scene and a size corresponding to the size of the sky area; a first foreground The combination subunit is configured to perform the combination of the foreground image and the target sky image according to the target mask image to obtain the first processed image; wherein the sky in the first processed image The area is replaced by the target sky map.
- the sky area replacement unit further includes: an area determining subunit, configured to perform determining the first area and the second area from the to-be-processed picture according to the target mask map And the remaining area; wherein the probability value of the pixels contained in the first area is 1, the probability value of the pixels contained in the second area is 0, and the remaining area is the picture to be processed Excluding the first area and the second area; the sky map replacement sub-unit is configured to perform the replacement of the first area with the target sky map; the foreground image replacement sub-unit is configured to execute The second area is replaced with the foreground image; the channel information fusion subunit is configured to perform the matching of the probability value, the red channel pixel value, the green channel pixel value, and the blue channel pixel value corresponding to the remaining area.
- the foreground image and the target sky map perform color channel information fusion of pixels; the processed picture acquiring subunit is configured to execute the target sky map processed according to the color channel information fusion process to obtain the first processed picture.
- the sky area replacement unit further includes: a foreground image adjustment sub-unit configured to perform an adjustment of at least one of the brightness, contrast, and saturation of the foreground image according to the target sky scene. Adjustment to obtain a target foreground image with brightness, contrast, and saturation matching the target sky scene; a second foreground combination subunit configured to perform a comparison between the target foreground image and the target foreground image according to the target mask image The target sky images are combined to obtain the first processed image.
- a foreground image adjustment sub-unit configured to perform an adjustment of at least one of the brightness, contrast, and saturation of the foreground image according to the target sky scene. Adjustment to obtain a target foreground image with brightness, contrast, and saturation matching the target sky scene
- a second foreground combination subunit configured to perform a comparison between the target foreground image and the target foreground image according to the target mask image The target sky images are combined to obtain the first processed image.
- the guided filtering unit is further configured to perform determining whether the picture to be processed meets at least one of the following preset conditions according to the initial mask image, and if it meets at least one of the following preset conditions, It is determined that the image to be processed does not meet the sky area replacement condition, and if it does not meet any of the following preset conditions, it is determined that the image to be processed meets the sky area replacement condition:
- Preset condition 1 The first proportion of the sky area in the picture to be processed is less than a preset fourth threshold
- Preset condition 2 The second proportion of the enormous area in the picture to be processed is greater than the preset fifth threshold; wherein, the overwhelming area is the area where the probability value of each pixel is in the middle interval, and the middle The interval is composed of the median value of the probability value and the adjacent value of the median value;
- Preset condition 3 The average brightness of the sky area in the picture to be processed is less than a preset sixth threshold
- Preset condition 4 The third proportion of the target dark channel area in the image to be processed is greater than the preset seventh threshold; wherein, the target dark channel area is the pixel point in the sky area whose dark channel value is less than the eighth threshold your region.
- the image processing device further includes: a filter processing unit configured to perform, if the sky area replacement condition is not met, obtain a target filter tool according to the target sky scene, and pass the target The filter tool performs filter processing on the to-be-processed picture to obtain a second processed picture.
- a filter processing unit configured to perform, if the sky area replacement condition is not met, obtain a target filter tool according to the target sky scene, and pass the target The filter tool performs filter processing on the to-be-processed picture to obtain a second processed picture.
- non-transitory computer-readable storage medium including instructions, such as the memory 102 including instructions, and the foregoing instructions may be executed by the processor 109 of the device 100 to complete the foregoing method.
- the non-transitory computer-readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
- a computer program product includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of the device reads from the readable storage medium The computer program is read and executed, so that the device executes the image processing method described in the above embodiment.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Probability & Statistics with Applications (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
- Image Analysis (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
Claims (18)
- 一种图像处理方法,其特征在于,包括:对待处理图片进行图像分割处理,根据图像分割处理结果得到初始掩码图;所述初始掩码图包含有所述待处理图片中各个像素点属于天空区域内像素点的概率值;根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件,若满足所述天空区域替换条件,以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理,得到目标掩码图;获取目标天空场景;所述目标天空场景为从预设的天空场景素材中选择得到;根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理,得到第一已处理图片。
- 根据权利要求1所述的图像处理方法,其特征在于,所述以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理,得到目标掩码图,包括:获取所述待处理图片中各个像素点的蓝通道像素值;将所述初始掩码图中蓝通道像素值处于目标分布区间且所述概率值大于第一阈值的像素点确定为第一像素点;其中,所述目标分布区间为预设的多个区间中,第一评价区域中各个像素点的蓝通道像素值数量最多的区间,所述第一评价区域为所述初始掩码图中所述概率值大于第二阈值的像素点所在区域;所述第二阈值大于所述第一阈值;将所述初始掩码图中蓝通道像素值小于目标蓝通道像素值的像素点确定为第二像素点;所述目标蓝通道像素值为第二评价区域中各个像素点的蓝通道像素值的最小值,所述第二评价区域为所述初始掩码图中所述概率值大于第三阈值的像素点所在区域;所述第三阈值大于所述第二阈值;将所述第一像素点的概率值增加,并将所述第二像素点的概率值减少,得到参考掩码图;以所述待处理图片的灰度图为导向图对所述参考掩码图进行导向滤波处理,得到所述目标掩码图。
- 根据权利要求2所述的图像处理方法,其特征在于,所述将所述第一像素点的概率值增加,并将所述第二像素点的概率值减少,包括:将所述第一像素点的概率值设置为1;将所述第二像素点的概率值减半。
- 根据权利要求3所述的图像处理方法,其特征在于,所述根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理,得到第一已处理图片,包括:确定所述待处理图片中的非天空区域,作为前景图;根据所述目标天空场景和所述天空区域的尺寸对天空素材图进行裁剪,得到场景与所述目标天空场景对应且尺寸与所述天空区域的尺寸对应的目标 天空图;根据所述目标掩码图对所述前景图和所述目标天空图进行组合,得到所述第一已处理图片;其中,所述第一已处理图片中的天空区域被所述目标天空图替换。
- 根据权利要求4所述的图像处理方法,其特征在于,所述根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理,得到第一已处理图片,包括:根据所述目标掩码图,从所述待处理图片中确定第一区域、第二区域以及剩余区域;其中,所述第一区域内部所包含的像素点的概率值为1,所述第二区域内部所包含的像素点的概率值为0,所述剩余区域为所述待处理图片中除去所述第一区域和所述第二区域的区域;将所述第一区域替换为所述目标天空图;将所述第二区域替换为所述前景图;根据所述剩余区域对应的概率值、红通道像素值、绿通道像素值以及蓝通道像素值对所述前景图和所述目标天空图进行像素点的颜色通道信息融合;根据颜色通道信息融合处理后的所述目标天空图得到所述第一已处理图片。
- 根据权利要求4所述的图像处理方法,其特征在于,所述根据所述目标掩码图对所述前景图和所述目标天空图进行组合,得到所述第一已处理图片,包括:根据目标天空场景对所述前景图的亮度、对比度和饱和度中的至少一项进行调整,以得到亮度、对比度和饱和度与所述目标天空场景相匹配的目标前景图;根据所述目标掩码图对所述目标前景图和所述目标天空图进行组合,得到所述第一已处理图片。
- 根据权利要求1所述的图像处理方法,其特征在于,所述根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件,包括:根据所述初始掩码图确定所述待处理图片是否符合以下至少一个预设条件,若符合以下至少一个预设条件,判定所述待处理图片不满足天空区域替换条件,若不符合以下任意一个预设条件,判定所述待处理图片满足天空区域替换条件:预设条件1:所述待处理图片中天空区域的第一占比小于预设的第四阈值;预设条件2:所述待处理图片中不置信区域的第二占比大于预设的第五阈值;其中,所述不置信区域为各个像素点的概率值处于中部区间的区域,所述中部区间由所述概率值的中值以及所述中值的临近值构成;预设条件3:所述待处理图片中天空区域的平均亮度小于预设的第六阈值;预设条件4:所述待处理图片中目标暗通道区域的第三占比大于预设的第 七阈值;其中,所述目标暗通道区域为天空区域中暗通道值小于第八阈值的像素点所在区域。
- 根据权利要求7所述的图像处理方法,其特征在于,在所述根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件之后,还包括:若不满足天空区域替换条件,根据所述目标天空场景获取目标滤镜工具,通过所述目标滤镜工具对所述待处理图片进行滤镜处理,得到第二已处理图片。
- 一种图像处理装置,其特征在于,包括:掩码图确定单元,被配置为执行对待处理图片进行图像分割处理,根据图像分割处理结果得到初始掩码图;所述初始掩码图包含有所述待处理图片中各个像素点属于天空区域内像素点的概率值;导向滤波单元,被配置为执行根据所述初始掩码图确定所述待处理图片是否满足预设的天空区域替换条件,若满足所述天空区域替换条件,以所述待处理图片的灰度图为导向图对所述初始掩码图进行导向滤波处理,得到目标掩码图;天空场景获取单元,被配置为执行获取目标天空场景;所述目标天空场景为从预设的天空场景素材中选择得到;天空区域替换单元,被配置为执行根据所述目标掩码图和所述目标天空场景对所述待处理图片中的天空区域进行替换处理,得到第一已处理图片。
- 根据权利要求9所述的图像处理装置,其特征在于,所述导向滤波单元,包括:像素值获取子单元,被配置为执行获取所述待处理图片中各个像素点的蓝通道像素值;第一像素点确定子单元,被配置为执行将所述初始掩码图中蓝通道像素值处于目标分布区间且所述概率值大于第一阈值的像素点确定为第一像素点;其中,所述目标分布区间为预设的多个区间中,第一评价区域中各个像素点的蓝通道像素值数量最多的区间,所述第一评价区域为所述初始掩码图中所述概率值大于第二阈值的像素点所在区域;所述第二阈值大于所述第一阈值;第二像素点确定子单元,被配置为执行将所述初始掩码图中蓝通道像素值小于目标蓝通道像素值的像素点确定为第二像素点;所述目标蓝通道像素值为第二评价区域中各个像素点的蓝通道像素值的最小值,所述第二评价区域为所述初始掩码图中所述概率值大于第三阈值的像素点所在区域;所述第三阈值大于所述第二阈值;概率值处理子单元,被配置为执行将所述第一像素点的概率值增加,并将所述第二像素点的概率值减少,得到参考掩码图;导向滤波子单元,被配置为执行以所述待处理图片的灰度图为导向图对所述参考掩码图进行导向滤波处理,得到所述目标掩码图。
- 根据权利要求10所述的图像处理装置,其特征在于,所述导向滤波 子单元,包括:第一概率值设置模块,被配置为执行将所述第一像素点的概率值设置为1;第二概率值设置模块,被配置为执行将所述第二像素点的概率值减半。
- 根据权利要求11所述的图像处理装置,其特征在于,所述天空区域替换单元,包括:前景图确定子单元,被配置为执行确定所述待处理图片中的非天空区域,作为前景图;天空素材裁剪子单元,被配置为执行根据所述目标天空场景和所述天空区域的尺寸对天空素材图进行裁剪,得到场景与所述目标天空场景对应且尺寸与所述天空区域的尺寸对应的目标天空图;第一前景组合子单元,被配置为执行根据所述目标掩码图对所述前景图和所述目标天空图进行组合,得到所述第一已处理图片;其中,所述第一已处理图片中的天空区域被所述目标天空图替换。
- 根据权利要求12所述的图像处理装置,其特征在于,所述天空区域替换单元,还包括:区域确定子单元,被配置为执行根据所述目标掩码图,从所述待处理图片中确定第一区域、第二区域以及剩余区域;其中,所述第一区域内部所包含的像素点的概率值为1,所述第二区域内部所包含的像素点的概率值为0,所述剩余区域为所述待处理图片中除去所述第一区域和所述第二区域的区域;天空图替换子单元,被配置为执行将所述第一区域替换为所述目标天空图;前景图替换子单元,被配置为执行将所述第二区域替换为所述前景图;通道信息融合子单元,被配置为执行根据所述剩余区域对应的概率值、红通道像素值、绿通道像素值以及蓝通道像素值对所述前景图和所述目标天空图进行像素点的颜色通道信息融合;已处理图片获取子单元,被配置为执行根据颜色通道信息融合处理后的所述目标天空图得到所述第一已处理图片。
- 根据权利要求12所述的图像处理装置,其特征在于,所述天空区域替换单元,还包括:前景图调整子单元,被配置为执行根据目标天空场景对所述前景图的亮度、对比度和饱和度中的至少一项进行调整,以得到亮度、对比度和饱和度与所述目标天空场景相匹配的目标前景图;第二前景组合子单元,被配置为执行根据所述目标掩码图对所述目标前景图和所述目标天空图进行组合,得到所述第一已处理图片。
- 根据权利要求9所述的图像处理装置,其特征在于,所述导向滤波单元,还被配置为执行根据所述初始掩码图确定所述待处理图片是否符合以下至少一个预设条件,若符合以下至少一个预设条件,判定所述待处理图片不满足天空区域替换条件,若不符合以下任意一个预设条件,判定所述待处 理图片满足天空区域替换条件:预设条件1:所述待处理图片中天空区域的第一占比小于预设的第四阈值;预设条件2:所述待处理图片中不置信区域的第二占比大于预设的第五阈值;其中,所述不置信区域为各个像素点的概率值处于中部区间的区域,所述中部区间由所述概率值的中值以及所述中值的临近值构成;预设条件3:所述待处理图片中天空区域的平均亮度小于预设的第六阈值;预设条件4:所述待处理图片中目标暗通道区域的第三占比大于预设的第七阈值;其中,所述目标暗通道区域为天空区域中暗通道值小于第八阈值的像素点所在区域。
- 根据权利要求15所述的图像处理装置,其特征在于,所述图像处理装置,还包括:滤镜处理单元,被配置为执行若不满足天空区域替换条件,根据所述目标天空场景获取目标滤镜工具,通过所述目标滤镜工具对所述待处理图片进行滤镜处理,得到第二已处理图片。
- 一种电子设备,其特征在于,包括:处理器;用于存储所述处理器可执行指令的存储器;其中,所述处理器被配置为执行所述指令,以实现如权利要求1至8中任一项所述的图像处理方法。
- 一种存储介质,当所述存储介质中的指令由电子设备的处理器执行时,使得所述电子设备能够执行如权利要求1至8中任一项所述的图像处理方法。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2022548804A JP2023513726A (ja) | 2020-04-23 | 2020-11-09 | 画像処理方法、装置、電子機器及び記録媒体 |
US17/883,165 US20220383508A1 (en) | 2020-04-23 | 2022-08-08 | Image processing method and device, electronic device, and storage medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010327149.6 | 2020-04-23 | ||
CN202010327149.6A CN113554658B (zh) | 2020-04-23 | 2020-04-23 | 图像处理方法、装置、电子设备及存储介质 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/883,165 Continuation US20220383508A1 (en) | 2020-04-23 | 2022-08-08 | Image processing method and device, electronic device, and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021212810A1 true WO2021212810A1 (zh) | 2021-10-28 |
Family
ID=78129377
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/127564 WO2021212810A1 (zh) | 2020-04-23 | 2020-11-09 | 图像处理方法、装置、电子设备及存储介质 |
Country Status (4)
Country | Link |
---|---|
US (1) | US20220383508A1 (zh) |
JP (1) | JP2023513726A (zh) |
CN (1) | CN113554658B (zh) |
WO (1) | WO2021212810A1 (zh) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116363148A (zh) * | 2022-06-21 | 2023-06-30 | 上海玄戒技术有限公司 | 图像处理方法、装置、芯片及存储介质 |
CN116600210A (zh) * | 2023-07-18 | 2023-08-15 | 长春工业大学 | 基于机器人视觉的图像采集优化*** |
GB2619381A (en) * | 2022-05-13 | 2023-12-06 | Adobe Inc | Object class inpainting in digital images utilizing class-specific inpainting neural networks |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114004834B (zh) * | 2021-12-31 | 2022-04-19 | 山东信通电子股份有限公司 | 一种图像处理中的雾天情况分析方法、设备及装置 |
CN114494298A (zh) * | 2022-01-28 | 2022-05-13 | 北京字跳网络技术有限公司 | 对象分割方法、装置、设备及存储介质 |
CN115150390B (zh) * | 2022-06-27 | 2024-04-09 | 山东信通电子股份有限公司 | 一种图像显示方法、装置、设备及介质 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020085243A1 (en) * | 2000-12-21 | 2002-07-04 | Tomotoshi Kanatsu | Document processing apparatus and method |
CN105761230A (zh) * | 2016-03-16 | 2016-07-13 | 西安电子科技大学 | 基于天空区域分割处理的单幅图像去雾方法 |
CN110533616A (zh) * | 2019-08-30 | 2019-12-03 | 福建省德腾智能科技有限公司 | 一种图像天空区域分割的方法 |
CN111047540A (zh) * | 2019-12-27 | 2020-04-21 | 嘉应学院 | 一种基于天空分割的图像去雾方法及其应用*** |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10074161B2 (en) * | 2016-04-08 | 2018-09-11 | Adobe Systems Incorporated | Sky editing based on image composition |
CN106447638A (zh) * | 2016-09-30 | 2017-02-22 | 北京奇虎科技有限公司 | 一种美颜处理方法及装置 |
CN107025457B (zh) * | 2017-03-29 | 2022-03-08 | 腾讯科技(深圳)有限公司 | 一种图像处理方法和装置 |
CN108280809B (zh) * | 2017-12-26 | 2021-07-30 | 浙江工商大学 | 一种基于大气散射物理模型的有雾图像天空区域估计方法 |
CN109255759B (zh) * | 2018-08-02 | 2021-06-15 | 辽宁师范大学 | 基于天空分割和透射率自适应修正的图像去雾方法 |
CN110782407B (zh) * | 2019-10-15 | 2021-10-19 | 北京理工大学 | 一种基于天空区域概率分割的单幅图像去雾方法 |
-
2020
- 2020-04-23 CN CN202010327149.6A patent/CN113554658B/zh active Active
- 2020-11-09 WO PCT/CN2020/127564 patent/WO2021212810A1/zh active Application Filing
- 2020-11-09 JP JP2022548804A patent/JP2023513726A/ja active Pending
-
2022
- 2022-08-08 US US17/883,165 patent/US20220383508A1/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020085243A1 (en) * | 2000-12-21 | 2002-07-04 | Tomotoshi Kanatsu | Document processing apparatus and method |
CN105761230A (zh) * | 2016-03-16 | 2016-07-13 | 西安电子科技大学 | 基于天空区域分割处理的单幅图像去雾方法 |
CN110533616A (zh) * | 2019-08-30 | 2019-12-03 | 福建省德腾智能科技有限公司 | 一种图像天空区域分割的方法 |
CN111047540A (zh) * | 2019-12-27 | 2020-04-21 | 嘉应学院 | 一种基于天空分割的图像去雾方法及其应用*** |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2619381A (en) * | 2022-05-13 | 2023-12-06 | Adobe Inc | Object class inpainting in digital images utilizing class-specific inpainting neural networks |
CN116363148A (zh) * | 2022-06-21 | 2023-06-30 | 上海玄戒技术有限公司 | 图像处理方法、装置、芯片及存储介质 |
CN116363148B (zh) * | 2022-06-21 | 2024-04-02 | 上海玄戒技术有限公司 | 图像处理方法、装置、芯片及存储介质 |
CN116600210A (zh) * | 2023-07-18 | 2023-08-15 | 长春工业大学 | 基于机器人视觉的图像采集优化*** |
CN116600210B (zh) * | 2023-07-18 | 2023-10-10 | 长春工业大学 | 基于机器人视觉的图像采集优化*** |
Also Published As
Publication number | Publication date |
---|---|
CN113554658A (zh) | 2021-10-26 |
US20220383508A1 (en) | 2022-12-01 |
CN113554658B (zh) | 2024-06-14 |
JP2023513726A (ja) | 2023-04-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2021212810A1 (zh) | 图像处理方法、装置、电子设备及存储介质 | |
CN108764091B (zh) | 活体检测方法及装置、电子设备和存储介质 | |
EP3125158B1 (en) | Method and device for displaying images | |
US10091409B2 (en) | Improving focus in image and video capture using depth maps | |
US10007841B2 (en) | Human face recognition method, apparatus and terminal | |
CN108701439B (zh) | 一种图像显示优化方法及装置 | |
CN111402135A (zh) | 图像处理方法、装置、电子设备及计算机可读存储介质 | |
EP3057304B1 (en) | Method and apparatus for generating image filter | |
CN109712177B (zh) | 图像处理方法、装置、电子设备和计算机可读存储介质 | |
US9195880B1 (en) | Interactive viewer for image stacks | |
KR102525293B1 (ko) | 촬영 방법 및 장치, 단말기, 저장 매체 | |
EP3933675B1 (en) | Method and apparatus for detecting finger occlusion image, and storage medium | |
CN109784327B (zh) | 边界框确定方法、装置、电子设备及存储介质 | |
CN112017137A (zh) | 图像处理方法、装置、电子设备及计算机可读存储介质 | |
CN113610865B (zh) | 图像处理方法、装置、电子设备及计算机可读存储介质 | |
CN113610884A (zh) | 图像处理方法、装置、电子设备及计算机可读存储介质 | |
CN113658197A (zh) | 图像处理方法、装置、电子设备及计算机可读存储介质 | |
CN106469446B (zh) | 深度图像的分割方法和分割装置 | |
CN114697530B (zh) | 一种智能取景推荐的拍照方法及装置 | |
WO2023230927A1 (zh) | 图像处理方法、装置及可读存储介质 | |
CN113256503B (zh) | 图像优化方法及装置、移动终端及存储介质 | |
CN115409916A (zh) | 三分图制作方法、三分图制作装置及存储介质 | |
JP5962491B2 (ja) | 画角調整装置、方法、およびプログラム | |
CN112465721A (zh) | 图像校正方法及装置、移动终端及存储介质 | |
CN115346233A (zh) | 图像处理方法、装置、电子设备及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20931726 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2022548804 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07.02.2023) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20931726 Country of ref document: EP Kind code of ref document: A1 |