WO2023124201A1 - 图像处理方法与电子设备 - Google Patents

图像处理方法与电子设备 Download PDF

Info

Publication number
WO2023124201A1
WO2023124201A1 PCT/CN2022/117324 CN2022117324W WO2023124201A1 WO 2023124201 A1 WO2023124201 A1 WO 2023124201A1 CN 2022117324 W CN2022117324 W CN 2022117324W WO 2023124201 A1 WO2023124201 A1 WO 2023124201A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
camera module
mask
electronic device
processing
Prior art date
Application number
PCT/CN2022/117324
Other languages
English (en)
French (fr)
Inventor
肖斌
乔晓磊
朱聪超
王宇
邵涛
Original Assignee
荣耀终端有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202210108237.6A external-priority patent/CN116437198B/zh
Application filed by 荣耀终端有限公司 filed Critical 荣耀终端有限公司
Priority to EP22857103.0A priority Critical patent/EP4231621A4/en
Priority to US18/024,365 priority patent/US20240185389A1/en
Publication of WO2023124201A1 publication Critical patent/WO2023124201A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/11—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present application relates to the field of image processing, and in particular, relates to an image processing method and electronic equipment.
  • image enhancement processing is a method for enhancing useful information in the image and improving the visual effect of the image.
  • the present application provides an image processing method and electronic equipment, which can perform image enhancement on images acquired by a camera module of a main camera to improve image quality.
  • an image processing method which is applied to an electronic device, and the electronic device includes a first camera module and a second camera module, and the first camera module is a near-infrared camera module or an infrared camera module, the image processing method includes:
  • the first interface includes a first control
  • the first image is an image collected by the first camera module
  • the second image is an image collected by the second camera module
  • the first image and the second image are images in a first color space
  • the at least two masks include a first mask, a second mask or a third mask
  • the first mask is used to mark the image area in the third image whose resolution is better than that of the fourth image
  • the second mask is used to mark the image area in the third image
  • the third mask is used to mark the image area where the object of the target category in the third image is located, and the detail information of the fused image is better than the detail information of the second image.
  • the second camera module may be a visible light camera module (for example, the acquired spectral range is 400nm-700nm), or the second camera module may be other camera modules capable of acquiring visible light.
  • the electronic device may include a first camera module and a second camera module, wherein the first camera module is a near-infrared camera module or an infrared camera module; through the first camera module The first image can be collected, and the second image can be collected through the second camera module; the first image and the second image can be image-processed respectively, and the third image and the fourth image can be obtained; based on at least two masks, the second image can be processed
  • the three images are fused with the fourth image to obtain a fused image; since the third image is a near-infrared image or an infrared image, the third image may include information that cannot be obtained in the fourth image, and by combining the third image with the fourth Image fusion processing can realize multi-spectral information fusion of near-infrared image information and visible light image information, so that the fused image includes more detailed information; in addition, in the embodiments of the present application, based on at least The two masks perform fusion processing on the third image and the fourth
  • the third mask for example, Semantic segmentation mask
  • the image area where the subject of the target category for example, green plants, distant mountains, etc.
  • the first color space may refer to a Raw color space.
  • the second color space may refer to a YUV color space.
  • the third image and the fourth image may refer to images in the YUV color space, and fusion processing may be performed in the YUV color space; since the fusion processing in the YUV color space requires less calculation examples, therefore , performing fusion processing on the third image and the fourth image based on at least two masks in the YUV color space, which can improve the efficiency of the fusion processing.
  • the fusion processing of the third image and the fourth image based on at least two masks to obtain a fusion image includes:
  • the target mask can be obtained according to the intersection of at least two masks; the local image area in the third image can be obtained according to the target mask; the local image area in the third image and the fourth The images are fused to obtain a fused image.
  • performing first image processing on the first image to obtain a third image includes:
  • the sixth image can be obtained by performing global registration processing on the fifth image; further, local registration processing can be performed on the sixth image region, so that the registration of the third image and the fourth image is achieved; thus It can avoid the occurrence of ghost area in the fused image due to incomplete registration during the fused processing of the third image and the fourth image.
  • global registration and local registration may be performed on the fourth image with the third image as a reference.
  • performing global registration processing on the fifth image based on the fourth image to obtain a sixth image includes:
  • screening target feature points in the fourth image that meet preset conditions Based on the depth information, screening target feature points in the fourth image that meet preset conditions
  • a global registration process is performed on the fifth image with the target feature point as a reference to obtain the sixth image.
  • the global registration process can be performed based on the depth information; since the third image (for example, NIR image) is compared with the fourth image (for example, RGB image), not all pixel positions have better effect; for example, for the close-up of the subject, the detail information in the third image is lower than that in the fourth image; if more feature points corresponding to the close-up are extracted from the third image, it may cause the detail information in the third image
  • the distant scene cannot be registered with the fourth image, which makes the ghosting problem prone to appear in the fused image; therefore, when performing global registration on the third image, the target feature points of the distant scene can be selected from the fourth image based on the depth information;
  • the target feature point in the image is used as a reference to perform global registration processing on the third image; thereby improving the accuracy of the global registration.
  • the preset condition is that the distance between the photographed object and the electronic device is greater than a first preset threshold.
  • performing local registration processing on the sixth image based on the fourth image to obtain the third image includes:
  • the brightness of the sixth image for example, the NIR image after global registration
  • the fourth image for example, the RGB image
  • performing local registration processing on the sixth image based on the fourth image to obtain the third image includes:
  • the local registration process is performed on the second image area with the first image area as a reference to obtain the third image.
  • the first color space may refer to a Raw color space.
  • the second color space may refer to a YUV color space.
  • the third image and the fourth image may refer to images in the YUV color space, and fusion processing may be performed in the YUV color space; since the fusion processing in the YUV color space requires less calculation examples, therefore , performing fusion processing on the third image and the fourth image based on at least two masks in the YUV color space, which can improve the efficiency of the fusion processing.
  • the at least two masks are the first mask and the third mask.
  • the at least two masks are the first mask, the second mask, and the third mask.
  • the electronic device further includes an infrared flash lamp
  • the image processing method further includes:
  • the dark light scene means that the ambient brightness of the shooting environment where the electronic device is located is less than a second preset threshold
  • the acquiring the first image and the second image in response to the first operation includes:
  • the infrared flash lamp When the infrared flash lamp is turned on, the first image and the second image are acquired.
  • the infrared flashlight in the electronic device can be turned on; since the electronic device can include the first camera module and the second module, when the infrared flashlight is turned on, the reflected light of the object increases, so that The amount of light entering the first camera module increases; due to the increase in the amount of light entering the first camera module, the detail information included in the first image collected by the first camera module increases; through the image processing method of the embodiment of the application
  • the images collected by the first camera module and the second camera module are fused together to enhance the image obtained by the camera module of the main camera and improve the detail information in the image.
  • the infrared flash is imperceptible to the user, and improves the detail information in the image without the user's perception.
  • the first interface includes a second control; the turning on the infrared flash in a dark scene includes:
  • the infrared flashlight is turned on in response to the second operation.
  • the first interface refers to a photographing interface
  • the first control refers to a control for instructing photographing
  • the first operation may refer to a click operation on a control indicating to take a photo in the photo taking interface.
  • the first interface refers to a video recording interface
  • the first control refers to a control for instructing video recording.
  • the first operation may refer to a click operation on a control indicating to record a video in the video recording interface.
  • the first interface refers to a video call interface
  • the first control refers to a control for instructing a video call.
  • the first operation may refer to a click operation on a control indicating a video call in the video call interface.
  • the first operation may also include a voice instruction operation, or other operations instructing the electronic device to take a photo or make a video call; make any restrictions.
  • an electronic device in a second aspect, includes one or more processors, memory, a first camera module and a second camera module; the first camera module is a near-infrared camera module Or an infrared camera module, the memory is coupled with the one or more processors, the memory is used to store computer program codes, the computer program codes include computer instructions, and the one or more processors call the computer instructions to cause the electronic device to:
  • the first interface includes a first control
  • the first image is an image collected by the first camera module
  • the second image is an image collected by the second camera module
  • the at least two masks include a first mask, a second mask or a third mask
  • the first mask is used to mark the image area in the third image whose resolution is better than that of the fourth image
  • the second mask is used to mark the image area in the third image
  • the third mask is used to mark the image area where the object of the target category in the third image is located, and the detail information of the fused image is better than the detail information of the second image.
  • the one or more processors call the computer instructions so that the electronic device executes:
  • the one or more processors call the computer instructions so that the electronic device executes:
  • the one or more processors call the computer instructions so that the electronic device executes:
  • screening target feature points in the fourth image that meet preset conditions Based on the depth information, screening target feature points in the fourth image that meet preset conditions
  • a global registration process is performed on the fifth image with the target feature point as a reference to obtain the sixth image.
  • the preset condition is that the distance between the subject and the electronic device is greater than a first preset threshold.
  • the one or more processors call the computer instructions so that the electronic device executes:
  • the one or more processors call the computer instructions so that the electronic device executes:
  • the local registration process is performed on the second image area with the first image area as a reference to obtain the third image.
  • the at least two masks are the first mask and the third mask.
  • the at least two masks are the first mask, the second mask, and the third mask.
  • the electronic device further includes an infrared flashlight, and the one or more processors invoke the computer instructions so that the electronic device executes:
  • the dark light scene means that the ambient brightness of the shooting environment where the electronic device is located is less than a second preset threshold
  • the infrared flash lamp When the infrared flash lamp is turned on, the first image and the second image are acquired.
  • the first interface includes a second control; the one or more processors call the computer instructions to make the electronic device execute:
  • the infrared flashlight is turned on in response to the second operation.
  • the first interface refers to a photographing interface
  • the first control refers to a control for instructing photographing
  • the first interface refers to a video recording interface
  • the first control refers to a control for instructing video recording.
  • the first interface refers to a video call interface
  • the first control refers to a control for instructing a video call.
  • an electronic device including a module/unit for executing the first aspect or any image processing method in the first aspect.
  • an electronic device in a fourth aspect, includes one or more processors, a memory, a first camera module, and a second camera module; the memory is coupled to the one or more processors, The memory is used to store computer program codes, the computer program codes include computer instructions, and the one or more processors call the computer instructions to make the electronic device execute the first aspect or any one of the first aspects. way.
  • a chip system is provided, the chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call a computer instruction so that the electronic device executes the first aspect or any of the methods in the first aspect.
  • a computer-readable storage medium stores computer program code, and when the computer program code is run by an electronic device, the electronic device executes the first aspect or the first Either method in the aspect.
  • a computer program product comprising: computer program code, when the computer program code is run by an electronic device, the electronic device is made to execute the first aspect or any one of the first aspects. a way.
  • the electronic device may include a first camera module and a second camera module, wherein the first camera module is a near-infrared camera module or an infrared camera module; through the first camera module The first image can be collected, and the second image can be collected through the second camera module; the first image and the second image can be image-processed respectively, and the third image and the fourth image can be obtained; based on at least two masks, the second image can be processed
  • the three images are fused with the fourth image to obtain a fused image; since the third image is a near-infrared image or an infrared image, the third image may include information that cannot be obtained in the fourth image, and by combining the third image with the fourth Image fusion processing can realize multi-spectral information fusion of near-infrared image information and visible light image information, so that the fused image includes more detailed information; in addition, in the embodiments of the present application, based on at least The two masks perform fusion processing on the third image and the fourth
  • FIG. 1 is a schematic diagram of a hardware system applicable to an electronic device of the present application
  • FIG. 2 is a schematic diagram of a software system of an electronic device suitable for the electronic device of the present application
  • FIG. 3 is a schematic diagram of an application scenario applicable to an embodiment of the present application.
  • FIG. 4 is a schematic diagram of an application scenario applicable to an embodiment of the present application.
  • Fig. 5 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • Fig. 6 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • Fig. 7 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • FIG. 8 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • FIG. 9 is a schematic diagram of a target mask provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of a global registration method provided by an embodiment of the present application.
  • Fig. 11 is a schematic flowchart of a local registration method provided by an embodiment of the present application.
  • Fig. 12 is a schematic flowchart of a local registration method provided by an embodiment of the present application.
  • Fig. 13 is a schematic diagram showing the effect of the image processing method provided by the embodiment of the present application.
  • Fig. 14 is a schematic diagram of a graphical user interface applicable to the embodiment of the present application.
  • Fig. 15 is a schematic diagram of an optical path of a shooting scene applicable to an embodiment of the present application.
  • Fig. 16 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 17 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • NIR near infrared light
  • Near-infrared light refers to electromagnetic waves between visible light and mid-infrared light; the near-infrared light region can be divided into two regions: near-infrared short-wave (780nm-1100nm) and near-infrared long-wave (1100nm-2526nm).
  • the main camera module refers to a camera module that receives visible light in a spectral range; for example, the sensor included in the main camera module receives a spectral range of 400nm to 700nm.
  • a near-infrared camera module refers to a camera module that receives near-infrared light in a spectral range; for example, a sensor included in a near-infrared camera module receives a spectral range of 700 nm to 1100 nm.
  • the high-frequency information of an image refers to the region where the gray value changes drastically in the image; for example, the high-frequency information in the image includes edge information, texture information, etc. of an object.
  • the low-frequency information of the image refers to the area where the gray value changes slowly in the image; for an image, the part except the high-frequency information is low-frequency information; for example, the low-frequency information of the image can include the content information within the edge of the object.
  • Image registration refers to the process of matching and superimposing two or more images acquired at different times, different sensors (imaging devices) or under different conditions (weather, illumination, camera position and angle, etc.).
  • the brightness value is used to estimate the ambient brightness, and its specific calculation formula is as follows:
  • Exposure is the exposure time; Aperture is the aperture size; Iso is the sensitivity; Luma is the average value of Y in the XYZ space of the image.
  • feature points are also called key points (key points), interest points (interest points); it is widely used to solve object recognition, image recognition, image matching, visual tracking , 3D reconstruction and a series of problems (for example, the intersection of two edges in an image can be called a feature point).
  • Color correction matrix color correction matrix
  • a color correction matrix is used to calibrate the accuracy of colors other than white.
  • Three-dimensional look up table (Threedimension look up table, 3DLUT)
  • lookup tables can be used for image color correction, image enhancement, or image gamma correction; for example, LUTs can be loaded in image signal processors, and the original image can be processed according to the LUT table Processing, realize the pixel value mapping of the original image frame and change the color style of the image, so as to achieve different image effects.
  • GTM Global tone mapping
  • Global tone mapping is used to solve the problem of uneven distribution of gray values in high dynamic images.
  • Gamma processing is used to adjust the brightness, contrast and dynamic range of an image by adjusting the gamma curve.
  • a neural network refers to a network formed by connecting multiple single neural units together, that is, the output of one neural unit can be the input of another neural unit; the input of each neural unit can be connected to the local receptive field of the previous layer, To extract the features of the local receptive field, the local receptive field can be an area composed of several neural units.
  • the neural network can use the error back propagation (back propagation, BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, passing the input signal forward until the output will generate an error loss, and updating the parameters in the initial neural network model by backpropagating the error loss information, so that the error loss converges.
  • the backpropagation algorithm is a backpropagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.
  • Fig. 1 shows a hardware system applicable to the electronic equipment of this application.
  • the electronic device 100 may be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, a vehicle electronic device, an augmented reality (augmented reality, AR) device, a virtual reality (virtual reality, VR) device, a notebook computer, a super mobile personal computer ( ultra-mobile personal computer (UMPC), netbook, personal digital assistant (personal digital assistant, PDA), projector, etc.
  • augmented reality augmented reality
  • VR virtual reality
  • a notebook computer a super mobile personal computer ( ultra-mobile personal computer (UMPC), netbook, personal digital assistant (personal digital assistant, PDA), projector, etc.
  • UMPC ultra-mobile personal computer
  • PDA personal digital assistant
  • the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like.
  • SIM subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.
  • the structure shown in FIG. 1 does not constitute a specific limitation on the electronic device 100 .
  • the electronic device 100 may include more or fewer components than those shown in FIG. 1 , or the electronic device 100 may include a combination of some of the components shown in FIG. 1 , or , the electronic device 100 may include subcomponents of some of the components shown in FIG. 1 .
  • the components shown in FIG. 1 can be realized in hardware, software, or a combination of software and hardware.
  • Processor 110 may include one or more processing units.
  • the processor 110 may include at least one of the following processing units: an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor) , ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, neural network processor (neural-network processing unit, NPU).
  • an application processor application processor, AP
  • modem processor graphics processing unit
  • graphics processing unit graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit
  • a memory may also be provided in the processor 110 for storing instructions and data.
  • the memory in processor 110 is a cache memory.
  • the memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.
  • the processor 110 may be configured to execute the image processing method of the embodiment of the present application; for example, display a first interface, the first interface includes a first control; detect a first operation on the first control; respond to the first Operation, acquire the first image and the second image, the first image is the image collected by the first camera module, the second image is the image collected by the second camera module, the first image and the second image are the first color space image; performing first image processing on the first image to obtain a third image, and the third image is an image in the second color space; performing second image processing on the second image to obtain a fourth image in the second color space; based on at least Two masks perform fusion processing on the third image and the fourth image to obtain a fusion image, wherein at least two masks include at least two items of the first mask, the second mask or the third mask, and the first The mask is used to mark the image area in the third image whose sharpness is better than that of the fourth image, the second mask is used to mark the ghost area in the third image, and the third
  • connection relationship between the modules shown in FIG. 1 is only a schematic illustration, and does not constitute a limitation on the connection relationship between the modules of the electronic device 100 .
  • each module of the electronic device 100 may also adopt a combination of various connection modes in the foregoing embodiments.
  • the wireless communication function of the electronic device 100 may be realized by components such as the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, and a baseband processor.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas.
  • Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna may be used in conjunction with a tuning switch.
  • the electronic device 100 can realize the display function through the GPU, the display screen 194 and the application processor.
  • the GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • Display 194 may be used to display images or video.
  • the electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 , and the application processor.
  • the ISP is used for processing the data fed back by the camera 193 .
  • the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye.
  • ISP can optimize the algorithm of image noise, brightness and color, and ISP can also optimize parameters such as exposure and color temperature of the shooting scene.
  • the ISP may be located in the camera 193 .
  • Camera 193 is used to capture still images or video.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
  • the ISP outputs the digital image signal to the DSP for processing.
  • DSP converts digital image signals into standard red green blue (red green blue, RGB), YUV and other image signals.
  • the electronic device 100 may include 1 or N cameras 193 , where N is a positive integer greater than 1.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the electronic device 100 may support one or more video codecs.
  • the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3 and MPEG4.
  • the gyro sensor 180B can be used to determine the motion posture of the electronic device 100 .
  • the angular velocity of the electronic device 100 around three axes may be determined by the gyro sensor 180B.
  • the gyro sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 100 through reverse movement to achieve anti-shake.
  • the gyro sensor 180B can also be used in scenarios such as navigation and somatosensory games.
  • the acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally x-axis, y-axis and z-axis). The magnitude and direction of gravity can be detected when the electronic device 100 is stationary. The acceleration sensor 180E can also be used to identify the posture of the electronic device 100 as an input parameter for application programs such as horizontal and vertical screen switching and pedometer.
  • the distance sensor 180F is used to measure distance.
  • the electronic device 100 may measure the distance by infrared or laser. In some embodiments, for example, in a shooting scene, the electronic device 100 can use the distance sensor 180F for distance measurement to achieve fast focusing.
  • the ambient light sensor 180L is used for sensing ambient light brightness.
  • the electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness.
  • the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket, so as to prevent accidental touch.
  • the fingerprint sensor 180H is used to collect fingerprints.
  • the electronic device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing the application lock, taking pictures, and answering incoming calls.
  • the touch sensor 180K is also referred to as a touch device.
  • the touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a touch screen.
  • the touch sensor 180K is used to detect a touch operation on or near it.
  • the touch sensor 180K may transmit the detected touch operation to the application processor to determine the touch event type.
  • Visual output related to the touch operation can be provided through the display screen 194 .
  • the touch sensor 180K may also be disposed on the surface of the electronic device 100 and disposed at a different position from the display screen 194 .
  • the hardware system of the electronic device 100 has been described in detail above, and the software system of the image electronic device 100 will be introduced below.
  • Fig. 2 is a schematic diagram of the software system of the device provided by the embodiment of the present application.
  • the system architecture may include an application layer 210 , an application framework layer 220 , a hardware abstraction layer 230 , a driver layer 240 and a hardware layer 250 .
  • the application layer 210 may include applications such as camera application, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.
  • the application framework layer 220 provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer; the application framework layer may include some predefined functions.
  • API application programming interface
  • the application framework layer 220 may include a camera access interface; the camera access interface may include camera management and camera equipment.
  • the camera management can be used to provide an access interface for managing the camera; the camera device can be used to provide an interface for accessing the camera.
  • the hardware abstraction layer 230 is used to abstract hardware.
  • the hardware abstraction layer can include the camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call the algorithms in the camera algorithm library.
  • a library of camera algorithms may include software algorithms for image processing.
  • the driver layer 240 is used to provide drivers for different hardware devices.
  • the driver layer may include a camera device driver; a digital signal processor driver, a graphics processor driver, or a central processing unit driver.
  • the hardware layer 250 may include camera devices as well as other hardware devices.
  • the hardware layer 250 includes a camera device, a digital signal processor, a graphics processor or a central processing unit; for example, the camera device may include an image signal processor, and the image signal processor may be used for image processing.
  • the spectral range obtained by the main camera module on the terminal device is visible light (400nm ⁇ 700nm); in some shooting scenes, for example, shooting scenes with poor lighting conditions; Due to the poor light conditions of the shooting scene and the small amount of light entering the electronic device, there is a problem that some image detail information is lost in the image obtained by the camera module of the main camera.
  • an embodiment of the present application provides an image processing method, which can be applied to an electronic device;
  • the electronic device can include a first camera module and a second camera module, wherein the first camera module The group is a near-infrared camera module or an infrared camera module;
  • the first image can be collected by the first camera module, and the second image can be collected by the second camera module;
  • image processing is performed on the first image and the second image respectively,
  • a third image and a fourth image can be obtained; based on at least two masks, the third image and the fourth image can be fused to obtain a fused image; since the third image is a near-infrared image or an infrared image, the third image Information that cannot be obtained in the fourth image may be included.
  • the third image and the fourth image By performing fusion processing on the third image and the fourth image, multi-spectral information fusion of image information of near-infrared light and image information of visible light can be realized, so that the fused image includes more detailed information; in addition, in the embodiment of the present application, the third image and the fourth image are fused based on at least two masks, and at least two
  • image enhancement is performed; thus, image enhancement is performed on the second image acquired by the second camera module (for example, the main camera module), the detailed information in the image is enhanced, and the image quality is improved.
  • the image processing method in the embodiment of the present application can be applied to the field of photography (for example, single-view photography, dual-view photography, etc.), recording video field, video call field or other image processing fields;
  • a dual-camera module is used, and the dual-camera module includes a first camera module that can obtain near-infrared light (for example, a near-infrared camera module, or an infrared camera module) and a second camera module that can obtain visible light ; Visible light images and near-infrared light images can be image processed and fused based on at least two masks (for example, sharpness masks, ghost masks or voice segmentation masks), to obtain images with enhanced image quality;
  • the image processing method in the embodiment of the present application processes the image, which can enhance the detailed information in the image and improve the image quality.
  • the spectral range that the near-infrared camera module can obtain is near-infrared light, which is similar to the spectral range of visible light
  • the wavelength of the spectrum that can be obtained by the near-infrared camera module is longer, so the diffraction ability is stronger.
  • the camera module and the near-infrared camera module collect images, and the images collected by the two camera modules are fused through the image processing method provided in the embodiment of the present application, and the obtained fused image is as shown in Figure 3; as shown in Figure 3
  • the detailed information of the fused image is relatively rich, and the detailed information of the mountains can be clearly displayed; the image obtained by the main camera module can be image enhanced through the image processing method provided by the embodiment of the application, and the detailed information in the image can be enhanced.
  • the terminal device shown in FIG. 3 may include a first camera module, a second camera module, and an infrared flashlight; wherein, the spectral range that the first camera module can obtain is near-infrared light (700nm-1100nm); The spectral range that the second camera module can obtain includes but is not limited to visible light (400nm-700nm).
  • the camera module of the main camera and the The green scene captured by the near-infrared camera module has more detail information, which can enhance the detail information of the green scene in the dark area of the image.
  • the image processing method provided by the embodiment of the present application can be applied to night scene portrait shooting; when night scene portrait shooting, the infrared flashlight in the electronic device can be turned on; for example, the portrait can include the face, eyes, and nose of the subject's face , mouth, ears, eyebrows, etc.; since the electronic device includes the main camera module and the near-infrared camera module, when the infrared flash is turned on, the reflected light of the subject increases, which increases the amount of light entering the near-infrared camera module ; so that the detail information of the portrait taken by the near-infrared camera module is increased, and the images collected by the main camera module and the near-infrared camera module are fused through the image processing method of the embodiment of the application, and the main camera camera can be fused.
  • the image acquired by the module is enhanced to improve the details in the image.
  • the infrared flash is imperceptible to the user, and improves the detail information in the image without the user's perception.
  • the electronic device when it detects food or a portrait, it can turn off the near-infrared camera module.
  • a food shooting scene may include multiple foods, and the near-infrared camera module may collect images of some of the foods in the multiple foods; for example, the multiple foods may be peaches, apples, or watermelons, etc., and the near-infrared camera module may collect Images of peaches and apples are collected, and images of watermelons are not collected.
  • the near-infrared camera module can display prompt information, prompting whether to enable the near-infrared camera module; the near-infrared camera module can only be enabled to collect images after the user authorizes the near-infrared camera module to be activated.
  • the image processing method in the embodiment of the present application can be applied to a folding screen terminal device;
  • the folding screen terminal device can include an outer screen and an inner screen; between the outer screen and the inner screen of the folding screen terminal device
  • the preview image can be displayed on the outer screen, as shown in (a) in Figure 4; Display a preview image, as shown in (b) in Figure 4; when the angle between the outer screen and the inner screen of the folding screen terminal device is an obtuse angle, the preview image can be displayed on one side of the inner screen, and the other side Display controls for instructing shooting, as shown in (c) in Figure 4; when the angle between the outer screen and the inner screen of the folding screen terminal device is 180 degrees, a preview image can be displayed on the inner screen, as shown in As shown in (d) of FIG.
  • the folding screen terminal device shown in FIG. 4 may include a first camera module, a second camera module, and an infrared flash lamp; wherein, the spectral range that the first camera module can acquire is visible light (400nm-700nm);
  • the first camera module is a near-infrared camera module or an infrared camera module;
  • the second camera module may be a visible light camera module, or other camera modules that can obtain but are not limited to visible light.
  • FIG. 5 is a schematic diagram of an image processing method provided by an embodiment of the present application.
  • the image processing method can be executed by the electronic device shown in FIG. 1 ; the method 200 includes steps S201 to S206 , and the steps S201 to S206 will be described in detail below.
  • the image processing method shown in FIG. 5 is applied to an electronic device, and the electronic device includes a first camera module and a second camera module, and the spectrum acquired by the first camera module is a near infrared camera module or an infrared camera module. group (for example, the acquired spectral range is 700nm ⁇ 1100nm).
  • the second camera module can be a visible light camera module (for example, the acquired spectral range is 400nm-700nm), or the second camera module can be other camera modules that can acquire visible light (for example, the acquired spectral range Including 400nm ⁇ 700nm).
  • Step S201 displaying a first interface, where the first interface includes a first control.
  • the first interface may refer to the photographing interface of the electronic device
  • the first control may refer to a control in the photographing interface for instructing photographing, as shown in FIG. 3 or FIG. 4 .
  • the first interface may refer to a video recording interface of the electronic device
  • the first control may refer to a control in the video recording interface for instructing to record a video
  • the first interface may refer to a video call interface of the electronic device
  • the first control may refer to a control on the video call interface used to indicate a video call.
  • Step S202 detecting a first operation on the first control.
  • the first operation may refer to a click operation on a control indicating to take a photo in the photo taking interface.
  • the first operation may refer to a click operation on a control indicating to record a video in the video recording interface.
  • the first operation may refer to a click operation on a control indicating a video call in the video call interface.
  • the first operation may also include a voice instruction operation, or other operations instructing the electronic device to take a photo or make a video call; make any restrictions.
  • Step S203 in response to the first operation, acquiring the first image and the second image.
  • the first image is an image collected by the first camera module;
  • the second image is an image collected by the second camera module;
  • the first image and the second image are images in the first color space.
  • the first camera module may capture a first image
  • the second camera module may capture a second image; for example, the first camera module and the second camera module may capture images simultaneously.
  • the first color space may refer to a Raw color space
  • the first image and the second image may refer to images in a Raw color space.
  • the first image may refer to an NIR image in a Raw color space
  • the second image may refer to an RGB image in a Raw color space.
  • the NIR image of the Raw color space may refer to NIRRaw; NIRRaw may refer to a single-channel image; the NIR Raw image is used to represent the intensity information of photons superimposed together; for example, the NIR Raw image Can be a grayscale image in single channel.
  • Step S204 performing first image processing on the first image to obtain a third image.
  • the third image is an image in the second color space.
  • the second color space may refer to a YUV color space, or other color spaces.
  • the first image processing is performed on the first image to obtain the third image, including:
  • the first camera module and the second camera module are respectively arranged in different positions in the electronic device, there is a certain baseline distance between the first camera module and the second camera module, That is, there is a certain parallax between the image collected by the first camera module and the image collected by the second camera module, and the parallax between the third image and the fourth image can be eliminated by global registration processing and local registration processing ; Therefore, when performing fusion processing on the third image and the fourth image, the ghost image in the fusion image can be reduced.
  • the above-mentioned global registration process is performed on the fifth image based on the fourth image to obtain the sixth image, including:
  • Depth estimation is performed on the fourth image to obtain depth information, the depth information is used to represent the distance information between the subject in the fourth image and the electronic device; based on the depth information, target feature points in the fourth image that meet the preset conditions are screened ; Perform global registration processing on the fifth image based on the target feature point to obtain the sixth image.
  • the feature point refers to a point where the gray value of the image changes drastically, or a point with a large curvature on the edge of the image; the target feature point may refer to a feature point in the fourth image that satisfies a preset condition.
  • the preset condition can be that the distance between the subject and the electronic device is greater than the first preset threshold; that is, the feature points corresponding to the distant view of the subject in the fourth image are selected as the target feature points; Perform global registration processing.
  • the purpose of the global registration process is to avoid ghost areas in the fused image when the fusion process is performed on the third image and the fourth image;
  • the image details of the foreground area of the subject in the middle image, the image details of the foreground area of the object in the fourth image are better than the image details of the foreground area of the object in the third image;
  • the image area is fused with the fourth image; then when performing global registration on the third image, the foreground area of the object in the third image and the object in the fourth image may be disregarded
  • the distant view area (for example, the local image area corresponding to the target feature point) performs registration processing, so as to realize the global registration of the third image and the fourth image.
  • the global registration process can be performed based on the depth information; since the third image (for example, NIR image) is compared with the fourth image (for example, RGB image), not all pixel positions have better effect; for example, for the close-up of the subject, the detail information in the third image is lower than that in the fourth image; if more feature points corresponding to the close-up are extracted from the third image, it may cause the detail information in the third image
  • the remote view and the fourth image cannot be registered, which makes the fusion image prone to ghosting; therefore, when performing global registration on the third image, the target feature points in the fourth image can be selected based on depth information;
  • the target feature points are used as the reference to perform registration processing on the third image.
  • the sixth image is subjected to local registration processing based on the fourth image to obtain the third image, including:
  • the sixth image is subjected to local registration processing to obtain the third image.
  • the brightness of the sixth image for example, the NIR image after global registration
  • the fourth image for example, the RGB image
  • the sixth image is subjected to local registration processing based on the fourth image to obtain the third image, including:
  • the seventh image is obtained; the first image area in the fourth image is determined based on the seventh image; the area in the sixth image is determined based on the seventh image
  • the second image area performing local registration processing on the second image area based on the first image area to obtain a third image.
  • Step S205 performing second image processing on the second image to obtain a fourth image in the second color space.
  • the second image may be converted to a second color space to obtain a fourth image.
  • the second color space may refer to a YUV color space, or other color spaces.
  • the second image processing may include ISP image processing; the ISP processing may include Raw algorithm processing, RGB algorithm processing, or YUV algorithm processing.
  • Raw algorithm processing can include but not limited to:
  • Black level correction blacklevelcorrection, BLC
  • lens shading correction lens shading correction
  • LSC lens shading correction
  • autowhitebalance autowhitebalance
  • demosaic demosaic
  • the black level correction is used to correct the black level
  • the lens shading correction is used to eliminate the problem that the color and brightness of the image around the image are inconsistent with the image center caused by the lens optical system
  • the automatic white balance is used to make white in any The camera can render white at any color temperature.
  • black level correction lens shading correction
  • automatic white balance automatic white balance
  • demosaicing are used as examples for illustration; this application does not make any limitation on the Raw algorithm processing.
  • RGB algorithm processing includes but is not limited to:
  • Color correction matrix processing or three-dimensional lookup table processing, etc.
  • the color correction matrix (color correction matrix, CCM) is used to calibrate the accuracy of colors other than white.
  • the three-dimensional look-up table (Look Up Table, LUT) is widely used in image processing; for example, the look-up table can be used for image color correction, image enhancement or image gamma correction, etc.; for example, the LUT can be loaded in the image signal processor, according to the LUT
  • the table can perform image processing on the original image, and realize the color style of the original image mapped to other images, so as to achieve different image effects.
  • color correction matrix processing and three-dimensional lookup table processing are used as examples for illustration; this application does not make any limitation on RGB image processing.
  • YUV algorithm processing includes but is not limited to:
  • Global tone mapping (Global tone mapping, GTM) is used to solve the problem of uneven gray value distribution of high dynamic images.
  • Gamma processing is used to adjust the brightness, contrast and dynamic range of an image by adjusting the gamma curve.
  • Step S206 performing fusion processing on the third image and the fourth image based on at least two masks to obtain a fusion image.
  • the at least two masks include at least two items of the first mask, the second mask or the third mask, and the first mask is used to mark image regions in the third image whose definition is better than that of the fourth image,
  • the second mask is used to mark the ghost area in the third image,
  • the third mask is used to mark the image area where the object of the target category is located in the third image, and the detail information of the fused image is better than that of the second image .
  • the detail information of the fused image is better than the detail information of the second image may mean that the detail information in the fused image is more than the detail information in the second image; or, the detail information of the fused image is better than the detail information of the second image It may mean that the definition of the fused image is better than that of the second image. It can also be other situations, which are not limited in this application.
  • the detail information may include edge information and texture information of the subject (for example, hair edges, face details, clothes folds, edges of each tree of a large number of trees, branches and leaves of green plants, etc.).
  • the third mask for example, Semantic segmentation mask
  • the image area where the subject of the target category for example, green plants, distant mountains, etc.
  • the at least two masks are the first mask and the third mask; for example, a sharpness mask and a semantic segmentation mask can be obtained; based on the sharpness mask and the semantic segmentation mask, the third image and the first The four images are fused to obtain a fused image.
  • the at least two masks are the first mask, the second mask and the third mask; a sharpness mask, a ghost mask and a semantic segmentation mask can be obtained; based on the sharpness mask, the ghost The mask and the semantic segmentation mask perform fusion processing on the third image and the fourth image to obtain a fusion image.
  • the third image and the fourth image may refer to images in the YUV color space, and fusion processing may be performed in the YUV color space; since the fusion processing in the YUV color space requires less calculation example, therefore, in the YUV color space
  • the fusion processing of the third image and the fourth image is performed spatially based on at least two masks, which can improve the efficiency of the fusion processing.
  • the third image and the fourth image are fused based on at least two masks to obtain a fused image, including:
  • the target mask is obtained according to the intersection of at least two masks; the third image and the fourth image are fused according to the target mask to obtain a fused image.
  • the sharpness mask, ghost mask and semantic segmentation mask can be obtained; according to the intersection of the sharpness mask, ghost mask and semantic segmentation mask, the target mask is obtained; as shown in Figure 9, the sharpness The position where the pixel in the mask is 1 can represent the confidence that the sharpness of the third image in this area is greater than the sharpness of the fourth image is 1; the position where the pixel is 0 in the sharpness mask can represent the sharpness of the third image in this area The degree of confidence is 0 if the sharpness is greater than the sharpness of the fourth image; the position where the pixel is 1 in the ghost mask can indicate that the region is relative to the fourth image, and the confidence degree of the ghost region is 1; the pixel in the ghost mask The position of 0 can indicate that the confidence of this area is 0 relative to the ghost area in the fourth image; the position of a pixel in the semantic segmentation mask can indicate that the area is the object of the target category in the third image.
  • the position where the pixel in the semantic segmentation mask is 0 can indicate that the confidence level of the object in this area in the third image is 0; based on the sharpness mask, ghost mask and semantic segmentation mask Intersect to get the target mask; for example, for the sharpness mask, the ghost mask and the semantic segmentation mask are all 1 areas, the pixel value of the target mask corresponding to this area is 1; for the sharpness mask, In the area where both the ghost mask and the semantic segmentation mask are 0, the pixel value of the area corresponding to the target mask is 0; , the pixel value of the area corresponding to the target mask is 0.
  • the partial image area in the third image may be obtained according to the target mask; the partial image area in the third image is fused with the fourth image to obtain a fused image.
  • the target mask and the third image are multiplied pixel by pixel, and based on the pixel values of different regions in the target mask, the local image area in the third image can be determined; for example, it can be obtained that the pixel of the target mask is 1
  • the partial image area of the third image corresponding to the image area of the image area; the partial image area of the third image is fused with the fourth image to obtain a fused image.
  • the electronic device may include a first camera module and a second camera module, wherein the first camera module is a near-infrared camera module or an infrared camera module; through the first camera module The first image can be collected, and the second image can be collected through the second camera module; the first image and the second image can be image-processed respectively, and the third image and the fourth image can be obtained; based on at least two masks, the second image can be processed
  • the three images are fused with the fourth image to obtain a fused image; since the third image is a near-infrared image or an infrared image, the third image may include information that cannot be obtained in the fourth image, and by combining the third image with the fourth Image fusion processing can realize multi-spectral information fusion of near-infrared image information and visible light image information, so that the fused image includes more detailed information; in addition, in the embodiments of the present application, based on at least The two masks perform fusion processing on the third image and the fourth
  • FIG. 6 is a schematic diagram of an image processing method provided by an embodiment of the present application.
  • the image processing method can be executed by the electronic device shown in FIG. 1 ; the method 300 includes step S301 to step S306 , and step S301 to step S306 will be described in detail below.
  • the image processing method shown in FIG. 6 can be applied to the electronic device shown in FIG. 1 , the electronic device includes a first camera module and a second camera module; wherein the first camera module is a near-infrared camera module, or an infrared camera module (for example, the acquired spectral range is 700nm to 1100nm); the second camera module can be a visible light camera module, or other camera modules that can acquire visible light (for example, the acquired spectral range includes 400nm ⁇ 700nm).
  • the first camera module is a near-infrared camera module, or an infrared camera module (for example, the acquired spectral range is 700nm to 1100nm)
  • the second camera module can be a visible light camera module, or other camera modules that can acquire visible light (for example, the acquired spectral range includes 400nm ⁇ 700nm).
  • Step S301 the first camera module collects NIRRaw images (an example of the first image).
  • the NIRRaw image may refer to an NIR image in a Raw color space.
  • the first camera module may be a near-infrared camera module or an infrared camera module; the first camera module may include a first lens, a first lens and an image sensor, and the spectral range that the first lens can pass is Near-infrared light (700nm ⁇ 1100nm).
  • the first lens may refer to a filter lens; the first lens may be used to absorb light of certain specific wavelength bands and allow light of near-infrared wavelength bands to pass through.
  • the NIRRaw image collected by the first camera module may refer to a single-channel image; the NIRRaw image is used to represent the intensity information of photons superimposed together; for example, the NIRRaw image may be a single-channel image grayscale image.
  • Step S302 the second camera module captures an RGBRaw image (an example of a second image).
  • the RGBRaw image may refer to an RGB image in a Raw color space.
  • the second camera module may be a visible light camera module, or other camera modules capable of capturing visible light (for example, the acquired spectral range includes 400nm to 700nm); the second camera module may include a second lens, a second The second lens and the image sensor, the spectral range that the second lens can pass includes visible light (400nm-700nm).
  • step S301 and step S302 may be executed synchronously; that is, the first camera module and the second camera module may output frames synchronously, and acquire NIRRaw images and RGBRaw images respectively.
  • Step S303 performing ISP processing on the image output in step S301.
  • Step S304 performing ISP processing on the image output in step S302.
  • step S303 and step S304 may not have timing requirements; for example, step S303 and step S304 may also be executed simultaneously.
  • step S303 and step S304 may be partly or completely the same.
  • the ISP processing may include Raw algorithm processing, RGB algorithm processing or YUV algorithm processing.
  • Raw algorithm processing can include but not limited to:
  • Black level correction blacklevelcorrection, BLC
  • lens shading correction lens shading correction
  • LSC lens shading correction
  • AWB automatic white balance
  • demosaic demosaic
  • the black level correction is used to correct the black level
  • the lens shading correction is used to eliminate the problem that the color and brightness of the image around the image are inconsistent with the image center caused by the lens optical system
  • the automatic white balance is used to make white in any The camera can render white at any color temperature.
  • black level correction lens shading correction
  • automatic white balance automatic white balance
  • demosaicing are used as examples for illustration; this application does not make any limitation on the Raw algorithm processing.
  • RGB algorithm processing includes but is not limited to:
  • Color correction matrix processing or three-dimensional lookup table processing, etc.
  • the color correction matrix (color correction matrix, CCM) is used to calibrate the accuracy of colors other than white.
  • the three-dimensional look-up table (Look Up Table, LUT) is widely used in image processing; for example, the look-up table can be used for image color correction, image enhancement or image gamma correction, etc.; for example, the LUT can be loaded in the image signal processor, according to the LUT
  • the table can perform image processing on the original image, and realize the color style of the original image mapped to other images, so as to achieve different image effects.
  • color correction matrix processing and three-dimensional lookup table processing are used as examples for illustration; this application does not make any limitation on RGB image processing.
  • YUV algorithm processing includes but is not limited to:
  • Global tone mapping (Global tone mapping, GTM) is used to solve the problem of uneven gray value distribution of high dynamic images.
  • Gamma processing is used to adjust the brightness, contrast and dynamic range of an image by adjusting the gamma curve.
  • Step S305 acquiring at least two masks.
  • the at least two masks may be a sharpness mask and a semantic segmentation mask; for example, the third image and the fourth image are fused based on the sharpness mask and the semantic segmentation mask to obtain a fused image.
  • the at least two masks may be a sharpness mask, a ghost mask and a semantic segmentation mask.
  • the sharpness mask can be used to mark the image area in the NIR image whose sharpness is better than that of the RGB image
  • the ghost mask can be used to mark the ghost area in the NIR image
  • the semantic segmentation mask can be used to mark the NIR image The image area where the subject of the medium target category is located.
  • the NIR image can be divided into blocks to obtain N NIR image blocks; the variance A of each NIR image block in the N NIR image blocks can be calculated; the RGB image can be divided into blocks to obtain N RGB image blocks ; Calculate the variance B of each RGB image block in N RGB image blocks; when the variance A of an NIR image block is greater than the variance B of the corresponding RGB image block, the sharpness mask corresponding to the image block position is 1, 1 It can be used to indicate that the confidence level of the NIR image definition is greater than that of the RGB image is 1; when the variance A of a NIR image block is less than or equal to the variance B of the corresponding RGB image block, the sharpness mask corresponding to the image block position is 0 , 0 can be used to indicate that the confidence of the NIR image definition is greater than the RGB image is 0.
  • the image area in the NIR image whose definition is better than the RGB image can be determined through the definition mask; ) is fused with the RGB image, so that the definition of the fused image can be improved, and the image quality of the fused image can be improved.
  • ghost detection may be performed on the NIR image and the RGB image to obtain a ghost mask.
  • the NIR image and the RGB image can be filtered through the Sobel operator to obtain the gradient image of the NIR image and the gradient image of the RGB image; wherein, the gradient image is used to represent the change speed of the texture information; By subtracting the gradient image, a ghost mask can be obtained.
  • Sobel operator is mainly used for edge detection; it is a discrete difference operator used to calculate the approximate value of the gradient of the image brightness function; using this operator at any point in the image, will generate the corresponding gradient vector or its normal vector.
  • a local area (for example, a non-ghost image area) in the NIR image can be selected for fusion processing with the RGB image through the ghost mask, thereby avoiding the ghost area in the fused image , to improve the image quality of the fused image.
  • the semantic segmentation image corresponding to the fourth image can be obtained, and some labels are selected from the semantic segmentation image to obtain the semantic segmentation mask;
  • the semantic segmentation image includes 5 labels, namely label 0 to label 4; wherein, Label 0 is used to label portraits; label 1 is used to label animals; label 2 is used to label plants (for example, green plants); label 3 is used to label distant scenery (for example, distant mountains); label 4 is used to label sky; Select some labels from multiple labels in the semantic segmentation image, for example, select the image area of label 2 and label 3 as 1, and mark the rest of the image area as 0 to obtain the semantic segmentation mask.
  • the NIR image includes more detailed information (for example, the texture information of the distant mountain) for the distant scene (for example, the texture information of the distant mountain); by Selecting a local area in the NIR image (for example, an image area including more detailed information) and RGB image for fusion processing can enhance the local detail information of the fused image and improve the image quality of the fused image.
  • Step S306 fusion processing.
  • the NIR image and the RGB image are fused based on at least two masks to obtain a fused image.
  • the NIR image and the RGB image are fused, and image enhancement can be performed from at least two aspects such as clarity, ghosting or local details; thereby realizing image enhancement to the RGB image and enhancing the image details in the image, improving image quality.
  • the above steps S301 to S306 may be used to perform the steps provided by this embodiment of the present application. image processing method.
  • the electronic device can also include an infrared flashlight; when the electronic device is in a dark scene, that is, when the ambient brightness of the shooting environment where the electronic device is located is less than a second preset threshold (for example, it can be determined according to the brightness value). Judgment), the electronic device can execute the method 400 shown in FIG. 7; for example, step S400 can be executed to turn on the infrared flashlight; after the infrared flashlight is turned on, the NIRRaw image can be obtained through the first camera module, and the RGBRaw image can be obtained through the second camera module. image, execute steps S401 to S406 as shown in FIG. 7; it should be understood that steps S401 to S406 are applicable to relevant descriptions of steps S301 to S306, and will not be repeated here.
  • a second preset threshold for example, it can be determined according to the brightness value.
  • the electronic device can execute the method 400 shown in FIG. 7; for example, step S400 can be executed to turn on the infrared flashlight; after the inf
  • the ambient brightness of the shooting environment where the electronic device is located can be determined through the brightness value of the electronic device, and the brightness value of the electronic device is less than the second When the threshold is preset, the electronic device can turn on the infrared flashlight.
  • the brightness value is used to estimate the ambient brightness, and its specific calculation formula is as follows:
  • Exposure is the exposure time; Aperture is the aperture size; Iso is the sensitivity; Luma is the average value of Y in the XYZ space of the image.
  • the electronic device can focus first after detecting the shooting instruction, and perform scene detection synchronously; after recognizing the dark scene and completing the focus, the infrared flashlight can be turned on. After the infrared flashlight is turned on, the NIRRaw image and RGBRaw images can be out-of-frame synchronously.
  • the ambient brightness of the shooting environment where the electronic device is located is relatively low; after the electronic device turns on the infrared flashlight, it can increase the reflected light acquired by the first camera module, thereby increasing the amount of incoming light of the first camera module ; increase the definition of the NIRRaw image collected by the first camera module; increase the definition of the NIR image due to the increase in the definition of the NIRRaw image; increase the definition of the fused image due to the increase in the definition of the NIR image.
  • the electronic device may include a first camera module and a second camera module, wherein the first camera module is a near-infrared camera module or an infrared camera module; through the first camera module NIR Raw images can be collected, and RGB Raw images can be collected through the second camera module; NIR images and RGB Raw images can be processed separately to obtain NIR images and RGB images; NIR images and RGB images can be obtained based on at least two masks The RGB image is fused to obtain a fused image; since the NIR image is a near-infrared image or an infrared image, the NIR image can include information that cannot be obtained in the RGB image. By fusing the NIR image and the RGB image, the near-infrared image can be realized.
  • the multi-spectral information of the image information of light and the image information of visible light is fused, so that the fused image includes more detailed information; in addition, in the embodiment of the application, it is based on at least two masks for NIR image and RGB
  • the image is fused, and the image can be enhanced from at least two aspects such as sharpness, ghosting or local details; thereby realizing the RGB image acquired by the second camera module (for example, the main camera camera module), the enhanced The detailed information in the image improves the image quality.
  • FIG. 8 is a schematic diagram of an image processing method provided by an embodiment of the present application.
  • the image processing method can be executed by the electronic device shown in FIG. 1 ; the method 500 includes step S501 to step S512 , and step S501 to step S512 will be described in detail below.
  • the image processing method shown in FIG. 8 can be applied to the electronic device shown in FIG. 1 , the electronic device includes a first camera module and a second camera module; wherein the first camera module is a near-infrared camera module, or an infrared camera module (for example, the acquired spectral range is 700nm to 1100nm); the second camera module can be a visible light camera module, or other camera modules that can acquire visible light (for example, the acquired spectral range includes But not limited to 400nm-700nm).
  • the first camera module is a near-infrared camera module, or an infrared camera module (for example, the acquired spectral range is 700nm to 1100nm)
  • the second camera module can be a visible light camera module, or other camera modules that can acquire visible light (for example, the acquired spectral range includes But not limited to 400nm-700nm).
  • Step S501 the first camera module captures a NIRRaw image (an example of a first image).
  • the NIRRaw image may refer to an NIR image in a Raw color space.
  • the first camera module may be a near-infrared camera module or an infrared camera module; the first camera module may include a first lens, a first lens and an image sensor, and the spectral range that the first lens can pass is Near-infrared light (700nm ⁇ 1100nm).
  • the first lens may refer to a filter lens; the first lens may be used to absorb light of certain specific wavelength bands and allow light of near-infrared wavelength bands to pass through.
  • the NIRRaw image collected by the first camera module may refer to a single-channel image; the NIRRaw image is used to represent the intensity information of photons superimposed together; for example, the NIRRaw image may be a single-channel image grayscale image.
  • Step S502 the second camera module captures an RGBRaw image (an example of a second image).
  • the RGBRaw image may refer to an RGB image in a Raw color space.
  • the second camera module may be a visible light camera module, or other camera modules capable of capturing visible light (for example, the acquired spectral range includes 400nm to 700nm); the second camera module may include a second lens, a second For the second lens and the image sensor, the spectral range that the second lens can pass includes but is not limited to visible light (400nm-700nm).
  • step S501 and step S502 may be executed synchronously; that is, the first camera module and the second camera module may output frames synchronously to acquire NIRRaw images and RGBRaw images respectively.
  • Step S503 performing ISP processing on the image output in step S501.
  • Step S504 performing ISP processing on the image output in step S502.
  • step S503 and step S504 may not have timing requirements; for example, step S503 and step S504 may also be executed simultaneously.
  • step S503 and step S504 may be partially or completely identical.
  • the ISP processing may include Raw algorithm processing, RGB algorithm processing or YUV algorithm processing.
  • Raw algorithm processing can include but not limited to:
  • Black level correction blacklevelcorrection, BLC
  • lens shading correction lens shading correction
  • LSC lens shading correction
  • autowhitebalance autowhitebalance
  • demosaic demosaic
  • the black level correction is used to correct the black level
  • the lens shading correction is used to eliminate the problem that the color and brightness of the image around the image are inconsistent with the image center caused by the lens optical system
  • the automatic white balance is used to make white in any The camera can render white at any color temperature.
  • black level correction lens shading correction
  • automatic white balance automatic white balance
  • demosaicing are used as examples for illustration; this application does not make any limitation on the Raw algorithm processing.
  • RGB algorithm processing includes but is not limited to:
  • Color correction matrix processing or three-dimensional lookup table processing, etc.
  • the color correction matrix (color correction matrix, CCM) is used to calibrate the accuracy of colors other than white.
  • the three-dimensional look-up table (Look Up Table, LUT) is widely used in image processing; for example, the look-up table can be used for image color correction, image enhancement or image gamma correction, etc.; for example, the LUT can be loaded in the image signal processor, according to the LUT
  • the table can perform image processing on the original image, and realize the color style of the original image mapped to other images, so as to achieve different image effects.
  • color correction matrix processing and three-dimensional lookup table processing are used as examples for illustration; this application does not make any limitation on RGB image processing.
  • YUV algorithm processing includes but is not limited to:
  • Global tone mapping (Global tone mapping, GTM) is used to solve the problem of uneven gray value distribution of high dynamic images.
  • Gamma processing is used to adjust the brightness, contrast and dynamic range of an image by adjusting the gamma curve.
  • the above step S503 is an example; the NIR image can also be obtained by other methods; the above step S504 is an example; the RGB image can also be obtained by other methods; this application does not make any limitation thereto.
  • Step S505 obtaining an NIR image (an example of a fifth image).
  • the NIR image may refer to an NIR image in a YUV color space.
  • Step S506 obtaining an RGB image (an example of a fourth image).
  • the RGB image may refer to an RGB image in a YUV color space.
  • Step S507 global registration processing.
  • first camera module and the second camera module are respectively arranged at different positions in the electronic device, there is a certain baseline distance between the first camera module and the second camera module, that is, through the first camera module. There is a certain parallax between the image collected by the module and the image collected by the second camera module, and the parallax between the NIR image and the RGB image can be eliminated through global registration processing.
  • the global registration process is based on the RGB image, and the overall mapping of the NIR image to the coordinate system of the RGB image is taken as an example for illustration; or, the global registration process can also be based on the NIR image, and the The ensemble of the RGB image is mapped to the coordinate system of the NIR image.
  • the target pixel position in the RGB image can be selected based on the depth information; based on the target pixel position, the target feature point in the RGB image is obtained; the NIR image is globally registered with the target feature point in the RGB image as a benchmark;
  • the specific flow of the global registration processing can be referred to as shown in FIG. 10 .
  • the feature point refers to the point where the gray value of the image changes drastically, or the point with a large curvature on the edge of the image
  • the target feature point refers to the feature point in RGB that meets the preset conditions; for example, the target feature point can be Refers to the feature points in the RGB image whose depth information is greater than the first preset threshold.
  • the global registration process can be performed based on the depth information; because not all pixel positions of the NIR image have a better effect than the RGB image; for example, for the close-up in the object, the NIR image
  • the detailed information of the RGB image is lower than that of the RGB image; if more feature points corresponding to the near view are extracted from the NIR image, it may cause the distant view in the NIR image to be unable to be registered with the RGB image, making the fusion image prone to ghosting problems; Therefore, when performing global registration on the NIR image, the target pixel position in the RGB image can be selected based on the depth information; the target feature point in the RGB image can be obtained based on the target pixel position in the RGB image, and the target feature point in the RGB image is The benchmark performs a global registration process on NIR images.
  • Step S508 obtaining a globally registered NIR image (an example of a sixth image).
  • Step S509 local registration processing.
  • a local registration process is performed on the globally registered NIR image.
  • local registration processing can be further performed on the basis of global registration processing, so that the local details in the globally registered NIR image are subjected to image registration processing again; thus the local details of the fused image can be improved information.
  • Step S510 obtaining a locally registered NIR image (an example of a third image).
  • a local registration process may be performed on the globally registered NIR image through the local registration method shown in FIG. 11 or FIG. 12 to obtain a locally registered NIR image.
  • Step S511 acquiring at least two masks.
  • the information of at least two masks can be obtained; when the fusion processing is performed on the NIR image and the RGB image, the fusion processing can be performed based on the information of at least two masks, and the definition, removal Image enhancement is performed in at least two aspects, such as ghosting or local details; the image quality of the fused image can be improved.
  • the at least two masks may include but not limited to: at least two items of a sharpness mask, a ghost mask, or a semantic segmentation mask.
  • the image area whose sharpness is better than the RGB image can be obtained from the NIR image through the sharpness mask;
  • the image area is fused with the RGB image, so that the definition of the fused image can be improved.
  • the NIR image and the RGB image may not be registered; the NIR image can be eliminated through the ghost mask In the area that cannot be registered with the RGB image; further, based on the ghost mask, the NIR image and the RGB image are fused, so that the ghost area in the fused image can be effectively reduced.
  • the semantic segmentation mask can be obtained from The image area of the target category object is obtained in the NIR image; the local detail information in the fused image can be improved by fusing the image area with the RGB image.
  • a sharpness mask and a semantic segmentation mask may be obtained; based on the sharpness mask and the semantic segmentation mask, the locally registered NIR image and the RGB image are fused.
  • a sharpness mask, a ghost mask, and a semantic segmentation mask can be obtained; based on the sharpness mask, the ghost mask, and the semantic segmentation mask, the locally registered NIR image and the RGB image are fused .
  • the sharpness mask, ghost mask and semantic segmentation mask can be obtained; according to the intersection of the sharpness mask, ghost mask and semantic segmentation mask, the target mask is obtained, as shown in Figure 9. To repeat it again; based on the target mask, fusion processing is performed on the locally registered NIR image (an example of the third image) and the RGB image (an example of the fourth image).
  • the local image area where the NIR image is better than the RGB image can be marked in the NIR image through the sharpness mask; the non-ghosted local area in the NIR image can be marked in the NIR image through the ghost mask region or ghost region; the local image region with more detailed information in the NIR image can be marked in the NIR image by semantically segmenting the image.
  • the NIR image can be divided into blocks to obtain N NIR image blocks; the variance A of each NIR image block in the N NIR image blocks can be calculated; the RGB image can be divided into blocks to obtain N RGB image blocks ; Calculate the variance B of each RGB image block in N RGB image blocks; when the variance A of an NIR image block is greater than the variance B of the corresponding RGB image block, the sharpness mask corresponding to the image block position is 1, 1 It can be used to represent the confidence that the sharpness of the NIR image is greater than that of the RGB image is 1; when the variance A of a NIR image block is less than or equal to the variance B of the corresponding RGB image block, the sharpness mask corresponding to the position of the image block Membrane is 0, and 0 can be used to indicate that the sharpness of the NIR image is greater than that of the RGB image, and the confidence level is 0.
  • the image area in the NIR image whose definition is better than the RGB image can be determined through the definition mask; ) is fused with the RGB image, so that the definition of the fused image can be improved, and the image quality of the fused image can be improved.
  • ghost detection may be performed on the NIR image and the RGB image to obtain a ghost mask.
  • the NIR image and the RGB image can be filtered through the Sobel operator to obtain the gradient image of the NIR image and the gradient image of the RGB image; wherein, the gradient image is used to represent the change speed of the texture information; By subtracting the gradient image, a ghost mask can be obtained.
  • Sobel operator is mainly used for edge detection. It is a discrete difference operator, which is used to calculate the approximate value of the gradient of the image brightness function; using this operator at any point in the image will generate the corresponding gradient vector or its normal vector.
  • a local image area (for example, a non-ghost image area) in the NIR image can be selected for fusion processing with the RGB image through the ghost mask, thereby avoiding ghost images in the fused image area, which provides the image quality of the fused image.
  • some labels can be selected from the semantic segmentation image to obtain the semantic segmentation mask; for example, the semantic segmentation image includes 5 labels, label 0 to label 4; label 0 is used to label portraits; label 1 is used to Label 2 is used to label plants (for example, green plants); label 3 is used to label distant scenes (for example, distant mountains); label 4 is used to label the sky; can be selected from multiple labels in semantically segmented images Part of the label, for example, select the image area of label 2 and label 3 as 1, and mark the rest of the image area as 0 to obtain the semantic segmentation mask.
  • the semantic segmentation image includes 5 labels, label 0 to label 4; label 0 is used to label portraits; label 1 is used to Label 2 is used to label plants (for example, green plants); label 3 is used to label distant scenes (for example, distant mountains); label 4 is used to label the sky; can be selected from multiple labels in semantically segmented images Part of the label, for example, select the image area of label 2 and label 3 as 1, and mark the rest of the image
  • the NIR image includes more detailed information (for example, the texture information of the distant mountain) for the distant scene (for example, the texture information of the distant mountain); by Selecting a local area in the NIR image (for example, an image area including more detailed information) and RGB image for fusion processing can enhance the local detail information of the fused image and improve the image quality of the fused image.
  • Step S512 fusion processing.
  • the locally registered NIR image (an example of the third image) and the RGB image (an example of the fourth image) are fused based on at least two masks to obtain a fused image.
  • the electronic device may include a first camera module and a second camera module, wherein the first camera module is a near-infrared camera module or an infrared camera module; through the first camera module NIR Raw images can be collected, and RGB Raw images can be collected through the second camera module; NIR images and RGB Raw images can be processed separately to obtain NIR images and RGB images; NIR images and RGB images can be obtained based on at least two masks The RGB image is fused to obtain a fused image; since the NIR image is a near-infrared image or an infrared image, the NIR image can include information that cannot be obtained in the RGB image. By fusing the NIR image and the RGB image, the near-infrared image can be realized.
  • the multi-spectral information of the image information of light and the image information of visible light is fused, so that the fused image includes more detailed information; in addition, in the embodiment of the application, it is based on at least two masks for NIR image and RGB
  • the image is fused, and the image can be enhanced from at least two aspects such as sharpness, deghosting or local details; thereby realizing the image enhancement of the RGB image acquired by the second camera module (for example, the main camera camera module), Enhance the detail information in the image and improve the image quality.
  • step S507 shown in FIG. 8 with an example.
  • Fig. 10 is a schematic diagram of a global registration method provided by an embodiment of the present application.
  • the method 600 includes steps S601 to S604, which will be described in detail below respectively.
  • Step S601 performing depth estimation on the RGB image.
  • depth estimation may refer to estimating distance information of each pixel in the image relative to the camera module.
  • the depth information of the RGB image can be obtained through a depth estimation algorithm; the near view area and the far view area in the RGB image can be distinguished through the depth information of the RGB image.
  • the depth estimation algorithm refers to an algorithm for obtaining distance information from each point in the scene in the image to the camera; the depth estimation algorithm may include a monocular depth estimation algorithm, a binocular depth estimation algorithm, and the like.
  • Step S602 acquiring position information of some pixels in the RGB image.
  • position information of some pixels in the RGB image is acquired.
  • the depth information may be compared with a first preset threshold, and position information of some pixels in the RGB image whose depth information is greater than the first preset threshold may be determined.
  • Step S603 extracting feature points in the RGB image according to the location information.
  • some feature points in the NIR image can also be extracted according to the position information; or, depth estimation is performed on the NIR image to obtain depth information of the NIR image; based on the depth information of the NIR image, it is determined that the depth information in the NIR image is greater than the first Part of the pixel positions of the preset threshold; this application does not make any limitation on this.
  • Step S604 obtaining a globally registered NIR image.
  • registration processing is performed on some feature points in the NIR image to obtain a globally registered NIR image.
  • the global registration process is to avoid ghost areas in the fused image when the NIR image and the RGB image are fused;
  • the image details of the distant view area, the image details of the near view area of the subject in the RGB image are better than the image details of the near view area of the subject in the NIR image; therefore, the image area corresponding to the distant view area can be selected from the NIR image for fusion with the RGB image ; That is, when performing global registration for the NIR image, the foreground area of the object in the NIR image can be registered with the foreground area of the object in the RGB image regardless of the close-range area of the object in the NIR image, so as to realize the alignment between the NIR image and the object in the RGB image.
  • Global registration of RGB images are to avoid ghost areas in the fused image when the NIR image and the RGB image are fused;
  • some feature points in the NIR image and some feature points in the RGB image are extracted, and some feature points in the NIR image are mapped to some feature points in the RGB image through a homography matrix.
  • the local registration processing method in step S509 shown in FIG. 8 may be as shown in FIG. 11 .
  • the fusion image after global registration and the brightness of the RGB image may be quite different, if the fusion process is directly performed on the NIR image and the RGB image after the global registration, the fusion image will appear The problem of brightness distortion; for example, the color of the fusion image may appear gray, or the color of the fusion image is unnatural. Therefore, local registration processing can be performed on the NIR image after global registration, that is, brightness processing can be performed on the NIR image after global registration, so that the brightness of the NIR image after global registration is close to the brightness of the RGB image; The brightness of the image is close to that of the RGB image, so that the color accuracy of the fusion image after the fusion processing is higher, and the image quality is improved.
  • Fig. 11 is a schematic diagram of a local registration method provided by an embodiment of the present application.
  • the method 700 includes steps S701 to S706, which will be described in detail below respectively.
  • Step S701 acquiring an NIR image (an example of a fifth image).
  • the first camera module can collect NIRRaw images; perform image processing on the NIRRaw images to obtain NIR images.
  • the first camera module may be a near-infrared camera module, or the first camera module may be an infrared camera module.
  • the first camera module may include a first lens, a first lens and an image sensor, and the spectral range that the first lens can pass is near-infrared light (700nm ⁇ 1100nm).
  • the first lens may refer to a filter lens; the first lens may be used to absorb light of certain specific wavelength bands and allow light of near-infrared wavelength bands to pass through.
  • Step S702 acquiring an RGB image and performing color space conversion.
  • the local registration can be performed based on the brightness value in the image; therefore, it is necessary to convert the RGB image to another color space capable of extracting the brightness channel image, so as to obtain the brightness channel image corresponding to the RGB image.
  • the second camera module can collect RGBRaw images; perform image processing on the RGBRaw images to obtain RGB images; convert the RGB images to YUV color space to obtain YUV images.
  • the second camera module may be a visible light camera module; or the second camera module may be other camera modules capable of obtaining visible light; this application does not make any limitation thereto.
  • the second camera module may include a second lens, a second lens and an image sensor.
  • the spectral range that the second lens can pass is visible light (400nm-700nm), or the spectral range that the second lens can pass includes but is not limited to Visible light (400nm ⁇ 700nm).
  • the second lens may refer to a filter lens; the first lens may be used to absorb light of certain specific wavelength bands and allow light of visible light bands to pass through.
  • Step S703 extracting the Y channel.
  • Step S704 neural network model processing.
  • brightness processing is performed on the NIR image through a neural network model.
  • the local registration process of the image can usually be performed through the optical flow map, and the accuracy of the optical flow map will be affected by the gray scale; for the same shooting scene (for example, the scene of green plants), the gray scale of the RGB image may be below 150 , while the grayscale of the NIR image may be above 200; for the same shooting scene, the grayscale difference between the RGB image and the NIR image is large; the brightness of the NIR image can be processed through the neural network model, so that the brightness grayscale in the NIR image The brightness gray scale of the area with too high brightness decreases, and the brightness gray scale of the area with too low brightness gray scale increases.
  • the neural network model may be a neural network pre-trained through a backpropagation algorithm.
  • Step S705 obtaining the NIR image after brightness mapping.
  • the image content of the NIR image after brightness mapping is consistent with that of the NIR image, and the image brightness of the NIR image after brightness mapping is consistent with that of the RGB image.
  • Step S706 registration processing.
  • the brightness-mapped NIR image is registered to obtain a locally registered NIR image (an example of the third image).
  • the brightness of the NIR image can be Mapping processing to obtain the NIR image after brightness mapping; obtain the brightness channel corresponding to the RGB image, and perform registration processing on the brightness channel of the NIR image after brightness mapping and the RGB image; because the brightness of the NIR image is close to the brightness of the RGB image, making The color accuracy of the fusion image after fusion processing is higher, and the image quality is improved.
  • the local registration processing method in step S509 shown in FIG. 8 may be shown in FIG. 12 .
  • the edge region of the fused image may be ghosting will appear; therefore, local registration can be performed on the edge area of the NIR image after global registration, thereby reducing ghosting in the edge area of the fused image.
  • Fig. 12 is a schematic diagram of a local registration processing method provided by an embodiment of the present application.
  • the method 800 includes steps S801 to S808; the steps S801 to S808 will be described in detail below.
  • Step S801 acquiring an NIR image (an example of a fifth image).
  • the first camera module can collect NIRRaw images; perform image processing on the NIRRaw images to obtain NIR images.
  • the first camera module may be a near-infrared camera module, or the first camera module may be an infrared camera module.
  • the first camera module may include a first lens, a first lens and an image sensor, and the spectral range that the first lens can pass is near-infrared light (700nm ⁇ 1100nm).
  • the first lens may refer to a filter lens; the first lens may be used to absorb light of certain specific wavelength bands and allow light of near-infrared wavelength bands to pass through.
  • Step S802 extracting high-frequency information of the NIR image.
  • the high-frequency information of an image refers to the region where the gray value changes drastically in the image; for example, the high-frequency information in the image includes edge information, texture information, etc. of an object.
  • Step S803 acquiring an RGB image (an example of a fourth image).
  • RGBRaw images can be collected by the second camera module; image processing is performed on the RGBRaw images to obtain RGB images.
  • the second camera module is a visible light camera module; or the second camera module may be other camera modules; this application does not make any limitation on this.
  • the second camera module may include a second lens, a second lens and an image sensor.
  • the spectral range that the second lens can pass is visible light (400nm-700nm), or the spectral range that the second lens can pass includes but is not limited to Visible light (400nm ⁇ 700nm) and other light.
  • Step S804 extracting high-frequency information of the RGB image.
  • the high-frequency information of the image refers to the region in the image where the gray value changes rapidly; for example, the high-frequency information in the image includes edge information, texture information, etc. of the object.
  • Step S805 obtaining an edge difference image.
  • an edge difference image (an example of a seventh image) is obtained according to the high-frequency information of the NIR image and the high-frequency information of the RGB image.
  • the edge difference image is used to represent all the high-frequency information in the NIR image and the RGB image.
  • the edge difference image can be obtained according to the union of the high frequency information of the NIR image and the high frequency information of the RGB image.
  • Step S806 acquiring a partial image area in the NIR image.
  • a local image area (an example of the first image area) in the NIR image is acquired based on the edge difference image.
  • the edge difference image can be multiplied by the NIR image to obtain the local area in the NIR image.
  • multiplying the edge difference image by the NIR image may refer to multiplying the edge difference image by pixel values of corresponding pixel points in the NIR image.
  • Step S807 acquiring a partial image area in the RGB image.
  • a local image area (an example of the second image area) in the RGB image is acquired based on the edge difference image.
  • the edge difference image can be multiplied by the RGB image to obtain the local area in the RGB image.
  • multiplying the edge difference image by the RGB image may refer to multiplying the edge difference image by pixel values of corresponding pixel points in the RGB image.
  • Step S808 registration processing.
  • registration processing is performed on the local image area in the NIR image to obtain a locally registered NIR image (an example of the third image).
  • the edge difference image by obtaining the high-frequency information in the NIR image and the RGB image, the edge difference image can be obtained; the edge difference image includes all the high-frequency information in the NIR image and the RGB image, and the edge difference image can be respectively Acquire the local image area from the NIR image and the RGB image; by registering the two local areas, for example, using the local image area in the RGB image as a reference, the local area in the NIR image is registered, so that it can be The high-frequency information in the NIR image is registered with the high-frequency information in the RGB image, so that ghost images in the fused image can be avoided to a certain extent.
  • steps S801 to S808 shown in FIG. 12 can also be further processed after step S706 shown in FIG. 11; that is, the brightness processing can be performed on the NIR image of the global registration process through FIG. 11; further, it can be used
  • the method shown in FIG. 12 performs edge area registration on the NIR image after brightness processing, so as to obtain the NIR image after local registration.
  • FIG. 13 is a schematic diagram of the effect of the image processing method provided by the embodiment of the present application.
  • (a) in Figure 13 is the output image obtained by the existing main camera camera module;
  • (b) in Figure 13 is the output image obtained by the image processing method provided by the embodiment of the present application ;
  • the image shown in (a) in Figure 13 shows that the detailed information in the mountains is severely distorted; compared with the output image shown in (a) in Figure 13, the image shown in (b) in Figure 13
  • the detailed information of the displayed output image is relatively rich, and can clearly display the detailed information of mountains; through the image processing method provided in the embodiment of the present application, the image obtained by the main camera camera module can be image enhanced to improve the detailed information in the image.
  • the user in a dark scene, can turn on the infrared flashlight in the electronic device; collect images through the main camera module and the near-infrared camera module, and use the image processing method provided by the embodiment of the application to process the collected images.
  • the image is processed to output the processed image or video.
  • FIG. 14 shows a graphical user interface (graphical user interface, GUI) of an electronic device.
  • the GUI shown in (a) in FIG. 14 is the desktop 910 of the electronic device; when the electronic device detects that the user clicks the operation of the camera application (application, APP) icon 920 on the desktop 910, the camera application can be started, and the display is as follows: Another GUI shown in (b) in Figure 14; the GUI shown in (b) in Figure 14 can be the display interface of the camera APP in the camera mode, and the GUI can include a shooting interface 930; Including a viewfinder frame 931 and controls; for example, the shooting interface 930 may include a control 932 for instructing to take pictures and a control 933 for instructing to turn on an infrared flash; in the preview state, a preview image can be displayed in real time in the viewfinder 931; wherein , the preview state can mean that before the user turns on the camera and does not press the photo/record button, the preview image can be displayed in the viewfinder in real time.
  • the shooting interface shown in (c) in Figure 14 is displayed; when the infrared flashlight is turned on, the main camera module and the near The infrared camera module collects images, performs fusion processing on the collected images through the image processing method provided in the embodiment of the present application, and outputs the processed fusion image.
  • Fig. 15 is a schematic diagram of an optical path of a shooting scene applicable to an embodiment of the present application.
  • the electronic device also includes an infrared flashlight; in a dark scene, the electronic device can turn on the infrared flashlight; when the infrared flashlight is turned on, the lighting in the shooting environment can include street lights and infrared flashlights; The light in the shooting environment is reflected, so that the electronic device obtains an image of the shooting object.
  • the infrared flashlight when the infrared flashlight is turned on, the reflected light of the shooting object increases, so that the amount of incoming light of the near-infrared camera module in the electronic device increases; thereby making the details of the image captured by the near-infrared camera module
  • the information is increased, and the images collected by the main camera module and the near-infrared camera module are fused through the image processing method of the embodiment of the present application, so that the image acquired by the main camera module can be image enhanced, and the details in the image can be improved.
  • the infrared flash is imperceptible to the user, and improves the detail information in the image without the user's perception.
  • FIG. 16 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the electronic device 1000 includes a display module 1010 and a processing module 1020 .
  • the electronic device includes a first camera module and a second camera module, and the first camera module is a near-infrared camera module or an infrared camera module.
  • the display module 1010 is used to display a first interface, and the first interface includes a first control; the processing module 1020 is used to detect a first operation on the first control; in response to the first operation, Acquiring a first image and a second image, the first image is an image collected by the first camera module, the second image is an image collected by the second camera module, and the first image and the The second image is an image in the first color space; the first image is processed on the first image to obtain a third image, and the third image is an image in the second color space; the second image is processed on the second image Two image processing to obtain a fourth image in the second color space; performing fusion processing on the third image and the fourth image based on at least two masks to obtain a fusion image, wherein the at least two masks are the film includes at least two of a first mask, a second mask, or a third mask, the first mask being used to mark image regions in the third image that have better clarity than the fourth image, The second mask is used to mark the ghost image area
  • processing module 1020 is specifically configured to:
  • processing module 1020 is specifically configured to:
  • processing module 1020 is specifically configured to:
  • screening target feature points in the fourth image that meet preset conditions Based on the depth information, screening target feature points in the fourth image that meet preset conditions
  • a global registration process is performed on the fifth image with the target feature point as a reference to obtain the sixth image.
  • the preset condition is that the distance between the subject and the electronic device is greater than a first preset threshold.
  • processing module 1020 is specifically configured to:
  • processing module 1020 is specifically configured to:
  • the local registration process is performed on the second image area with the first image area as a reference to obtain the third image.
  • the at least two masks are the first mask and the third mask.
  • the at least two masks are the first mask, the second mask and the third mask.
  • the electronic device further includes an infrared flash lamp, and the processing module 1020 is specifically configured to:
  • the dark light scene means that the ambient brightness of the shooting environment where the electronic device is located is less than a second preset threshold
  • the infrared flash lamp When the infrared flash lamp is turned on, the first image and the second image are acquired.
  • the first interface includes a second control; the processing module 1020 is specifically configured to:
  • the infrared flashlight is turned on in response to the second operation.
  • the first interface refers to a photographing interface
  • the first control refers to a control for instructing photographing
  • the first interface refers to a video recording interface
  • the first control refers to a control for instructing video recording.
  • the first interface refers to a video call interface
  • the first control refers to a control for instructing a video call.
  • module here may be implemented in the form of software and/or hardware, which is not specifically limited.
  • a “module” may be a software program, a hardware circuit or a combination of both to realize the above functions.
  • the hardware circuitry may include application specific integrated circuits (ASICs), electronic circuits, processors (such as shared processors, dedicated processors, or group processors) for executing one or more software or firmware programs. etc.) and memory, incorporating logic, and/or other suitable components to support the described functionality.
  • ASICs application specific integrated circuits
  • processors such as shared processors, dedicated processors, or group processors for executing one or more software or firmware programs. etc.
  • memory incorporating logic, and/or other suitable components to support the described functionality.
  • the units of each example described in the embodiments of the present application can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.
  • FIG. 17 shows a schematic structural diagram of an electronic device provided by the present application.
  • the dotted line in FIG. 17 indicates that this unit or this module is optional; the electronic device 1100 can be used to implement the methods described in the foregoing method embodiments.
  • the electronic device 1100 includes one or more processors 1101, and the one or more processors 1101 can support the electronic device 1100 to implement the image processing method in the method embodiment.
  • the processor 1101 may be a general purpose processor or a special purpose processor.
  • the processor 1101 may be a central processing unit (central processing unit, CPU), a digital signal processor (digital signal processor, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices such as discrete gates, transistor logic devices, or discrete hardware components.
  • the processor 1101 may be used to control the electronic device 1100, execute software programs, and process data of the software programs.
  • the electronic device 1100 may also include a communication unit 1105, configured to implement signal input (reception) and output (transmission).
  • the electronic device 1100 can be a chip, and the communication unit 1105 can be an input and/or output circuit of the chip, or the communication unit 1105 can be a communication interface of the chip, and the chip can be used as a component of a terminal device or other electronic devices .
  • the electronic device 1100 may be a terminal device, and the communication unit 1105 may be a transceiver of the terminal device, or the communication unit 1105 may be a transceiver circuit of the terminal device.
  • the electronic device 1100 may include one or more memories 1102, on which a program 1104 is stored, and the program 1104 may be run by the processor 1101 to generate an instruction 1103, so that the processor 1101 executes the image processing described in the above method embodiment according to the instruction 1103 method.
  • data may also be stored in the memory 11002 .
  • the processor 1101 may also read data stored in the memory 1102, the data may be stored in the same storage address as the program 1104, and the data may also be stored in a different storage address from the program 1104.
  • the processor 1101 and the memory 1102 may be set separately, or may be integrated together, for example, integrated on a system-on-chip (system on chip, SOC) of the terminal device.
  • SOC system on chip
  • the memory 1102 can be used to store the related program 1104 of the image processing method provided in the embodiment of the present application, and the processor 1101 can be used to call the related program 1104 of the image processing method stored in the memory 1102 when performing image processing, Executing the image processing method of the embodiment of the present application; for example, displaying a first interface, the first interface includes a first control; detecting a first operation on the first control; in response to the first operation, acquiring the first image and the second image , the first image is the image collected by the first camera module, the second image is the image collected by the second camera module, the first image and the second image are images in the first color space; the first image is performed on the first image processing to obtain a third image, the third image is an image in the second color space; performing second image processing on the second image to obtain a fourth image in the second color space; based on at least two masks, the third image and the first image The four images are fused to obtain a fused image, wherein at least two masks include at least two
  • the present application also provides a computer program product, which implements the image processing method in any method embodiment in the present application when the computer program product is executed by the processor 1101 .
  • the computer program product may be stored in the memory 1102 , such as a program 1104 , and the program 1104 is finally converted into an executable object file executable by the processor 1101 through processes such as preprocessing, compiling, assembling and linking.
  • the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a computer, the image processing method described in any method embodiment in the present application is implemented.
  • the computer program may be a high-level language program or an executable object program.
  • the computer-readable storage medium is, for example, the memory 1102 .
  • the memory 1102 may be a volatile memory or a nonvolatile memory, or, the memory 1102 may include both a volatile memory and a nonvolatile memory.
  • the non-volatile memory can be read-only memory (read-only memory, ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically programmable Erases programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • Volatile memory can be random access memory (RAM), which acts as external cache memory.
  • RAM random access memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • DRAM synchronous dynamic random access memory
  • SDRAM double data rate synchronous dynamic random access memory
  • double data rate SDRAM double data rate SDRAM
  • DDR SDRAM enhanced synchronous dynamic random access memory
  • ESDRAM enhanced synchronous dynamic random access memory
  • serial link DRAM SLDRAM
  • direct memory bus random access memory direct rambus RAM, DR RAM
  • the disclosed systems, devices and methods may be implemented in other ways.
  • the embodiments of the electronic equipment described above are only illustrative.
  • the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
  • multiple units or components can be Incorporation may either be integrated into another system, or some features may be omitted, or not implemented.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • sequence numbers of the processes do not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, rather than by the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium.
  • the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Studio Devices (AREA)

Abstract

本申请涉及图像处理领域，提供了一种图像处理方法与电子设备，该方法应用于电子设备，电子设备包括第一相机模组与第二相机模组，第一相机模组为近红外相机模组或者红外相机模组，该方法包括：显示第一界面，第一界面包括第一控件；检测到对第一控件的第一操作；响应于第一操作，获取第一图像与第二图像；对第一图像进行第一图像处理，得到第三图像；对第二图像进行第二图像处理，得到第四图像；基于至少两个掩膜对第三图像与第四图像进行融合处理，得到融合图像，至少两个掩膜包括第一掩膜、第二掩膜或者第三掩膜中的至少两项。基于本申请的方案，能够避免主摄像头相机模组获取的图像中存在部分图像细节信息丢失的问题，提高图像质量。

Description

图像处理方法与电子设备
本申请要求于2021年12月29日提交国家知识产权局、申请号为202111638949.0、申请名称为“图像处理方法与电子设备”的中国专利申请的优先权，以及要求于2022年01月28日提交国家知识产权局、申请号为202210108237.6、申请名称为“图像处理方法与电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理领域，具体地，涉及一种图像处理方法与电子设备。
背景技术
随着多媒体技术和网络技术的飞速发展和广泛应用，人们在日常生活和生产活动中大量的使用图像信息。在一些拍照场景中，例如，光照条件较差的拍摄场景中，比如，夜晚场景或者浓雾场景中，由于拍摄场景的光线条件较差，电子设备的进光量较少，导致主摄像头相机模组获取的图像中存在部分图像细节信息丢失的问题；为了提高图像质量，通常可以采用图像增强处理；图像增强处理是一种用于增强图像中的有用信息，改善图像的视觉效果的方法。
因此，如何对主摄像头相机模组获取的图像进行图像增强，提高图像质量成为一个亟需解决的问题。
发明内容
本申请提供了一种图像处理方法与电子设备，能够对主摄像头相机模组获取的图像进行图像增强，提高图像质量。
第一方面，提供了一种图像处理方法，应用于电子设备，所述电子设备包括第一相机模组与第二相机模组，所述第一相机模组为近红外相机模组或者红外相机模组，所述图像处理方法包括：
显示第一界面，所述第一界面包括第一控件；
检测到对所述第一控件的第一操作；
响应于所述第一操作，获取第一图像与第二图像，所述第一图像为所述第一相机模组采集的图像，所述第二图像为所述第二相机模组采集的图像，所述第一图像与所述第二图像为第一颜色空间的图像；
对所述第一图像进行第一图像处理，得到第三图像，所述第三图像为第二颜色空间的图像；
对所述第二图像进行第二图像处理，得到所述第二颜色空间的第四图像；
基于至少两个掩膜对所述第三图像与所述第四图像进行融合处理，得到融合图像，其中，所述至少两个掩膜包括第一掩膜、第二掩膜或者第三掩膜中的至少两项，所述 第一掩膜用于标记所述第三图像中清晰度优于所述第四图像的图像区域，所述第二掩膜用于标记所述第三图像中的鬼影区域，所述第三掩膜用于标记所述第三图像中目标类别的拍摄对象所在的图像区域，所述融合图像的细节信息优于所述第二图像的细节信息。
在一种可能的实现方式中，第二相机模组可以为可见光相机模组(例如，获取的光谱范围为400nm～700nm)，或者第二相机模组为其他可以获取可见光的相机模组。
在本申请的实施例中，电子设备中可以包括第一相机模组与第二相机模组，其中，第一相机模组为近红外相机模组或者红外相机模组；通过第一相机模组可以采集第一图像，通过第二相机模组可以采集第二图像；对第一图像与第二图像分别进行图像处理，可以得到第三图像与第四图像；基于至少两个掩膜可以对第三图像与第四图像进行融合处理，得到融合图像；由于第三图像为近红外图像或者红外图像，因此第三图像中可以包括第四图像中无法获取的信息，通过对第三图像与第四图像进行融合处理，可以实现近红外光的图像信息与可见光的图像信息的多光谱信息融合，使得融合后的图像中包括更多的细节信息；此外，在本申请的实施例中，是基于至少两个掩膜对第三图像与第四图像进行融合处理，可以从清晰度、去鬼影或者局部细节等至少两个方面进行图像增强；从而实现对第二相机模组获取(例如，主摄像头相机模组)的第二图像进行图像增强，增强图像中的细节信息，提高图像质量。
应理解，在本申请的实施例中，由于第三图像中并非所有像素点的清晰度均优于第四图像；通过第一掩膜(例如，清晰度掩膜)可以从第三图像中获取清晰度优于第四图像的图像区域；将该图像区域与第四图像进行融合，从而能够提高融合图像的清晰度。
应理解，在本申请的实施例中，由于第三图像与第四图像中可能存在部分区域可能无法配准；通过第二掩膜(例如，鬼影掩膜)可以剔除第三图像中无法与第四图像配准的区域；从而能够有效地避免融合图像中出现鬼影。
应理解，在本申请的实施例中，由于近红外光对不同类别的物体反射率不同，从而导致对于不同的物体第三图像中包括的细节信息不同；因此，根据第三掩膜(例如，语义分割掩膜)可以从第三图像中获取目标类别(例如，绿色植物、远山等)的拍摄对象所在的图像区域；通过该图像区域与第四图像进行融合处理，能够提高融合图像中的局部细节信息。
在一种可能的实现方式中，第一颜色空间可以是指Raw颜色空间。
在一种可能的实现方式中，第二颜色空间可以是指YUV颜色空间。
在本申请的实施例中，第三图像与第四图像可以是指YUV颜色空间的图像，可以在YUV颜色空间进行融合处理；由于YUV颜色空间进行融合处理时对算例的需求较小，因此，在YUV颜色空间基于至少两个掩膜对第三图像与第四图像进行融合处理，能够提高融合处理的效率。
结合第一方面，在第一方面的某些实现方式中，所述基于至少两个掩膜对所述第三图像与所述第四图像进行融合处理，得到融合图像，包括：
根据所述至少两个掩膜的交集得到目标掩膜；
根据所述目标掩膜对所述第三图像与所述第四图像进行融合处理，得到所述融合图像。
在一种可能的实现方式中，可以根据至少两个掩膜的交集得到目标掩膜；根据目标掩膜可以获取第三图像中的局部图像区域；将第三图像中的局部图像区域与第四图像进行融合处理，得到融合图像。
结合第一方面，在第一方面的某些实现方式中，所述对所述第一图像进行第一图像处理，得到第三图像，包括：
将所述第一图像转换至所述第二颜色空间，得到第五图像；
以所述第四图像为基准对所述第五图像进行全局配准处理，得到第六图像；
以所述第四图像为基准对所述第六图像进行局部配准处理，得到所述第三图像。
在本申请的实施例中，可以对第五图像进行全局配准处理得到第六图像；进一步，可以对第六图像区域进行局部配准处理，使得第三图像与第四图像实现配准；从而能够避免第三图像与第四图像融合处理时，由于未完全配准导致融合图像中出现鬼影区域。
在一种可能的实现方式中，可以以第三图像为基准，对第四图像进行全局配准与局部配准处理。
结合第一方面，在第一方面的某些实现方式中，所述以所述第四图像为基准对所述第五图像进行全局配准处理，得到第六图像，包括：
对所述第四图像进行深度估计，得到深度信息，所述深度信息用于表示所述第四图像中的拍摄对象与所述电子设备的距离；
基于所述深度信息，筛选所述第四图像中满足预设条件的目标特征点；
以所述目标特征点为基准对所述第五图像进行全局配准处理，得到所述第六图像。
在本申请的实施例中，可以基于深度信息进行全局配准处理；由于第三图像(例如，NIR图像)与第四图像(例如，RGB图像)相比并非所有的像素位置均具有较好的效果；例如，对于拍摄对象中的近景，第三图像中的细节信息低于第四图像中的细节信息；若从第三图像中提取较多近景对应的特征点，可能导致第三图像中的远景与第四图像无法配准，使得融合图像中容易出现鬼影问题；因此，在对第三图像进行全局配准时，可以基于深度信息从第四图中选取远景的目标特征点；以第四图像中的目标特征点为基准对第三图像进行全局配准处理；从而提高全局配准的准确性。
结合第一方面，在第一方面的某些实现方式中，所述预设条件为所述拍摄对象与所述电子设备的距离大于第一预设阈值。
结合第一方面，在第一方面的某些实现方式中，所述以所述第四图像为基准对所述第六图像进行局部配准处理，得到所述第三图像，包括：
以所述第四图像对应的亮度通道图像为基准，对所述第六图像进行局部配准处理，得到所述第三图像。
在本申请的实施例中，由于第六图像(例如，全局配准后的NIR图像)与第四图像(例如，RGB图像)的亮度可能差别较大；通过对第六图像进行局部配准处理，即可以对第六图像进行亮度处理，使得第三图像与第四图像的亮度接近；由于第三图像与第四图像的亮度接近，因此在对第三图像与第四图像进行融合处理时，能够有效地降低融合图像出现颜色失真的问题。
结合第一方面，在第一方面的某些实现方式中，所述以所述第四图像为基准对所 述第六图像进行局部配准处理，得到所述第三图像，包括：
根据所述第四图像的高频信息与所述第六图像的高频信息的并集，得到第七图像；
基于所述第七图像确定所述第四图像中的第一图像区域；
基于所述第七图像确定所述第六图像中的第二图像区域；
以所述第一图像区域为基准对所述第二图像区域进行所述局部配准处理，得到所述第三图像。
在本申请的实施例中，由于第六图像的高频信息与第四图像的高频可能存在差异，即第六图像中拍摄对象的边缘区域与第四图像中拍摄对象的边缘区域可能存在差异；因此，可以对第六图像的边缘区域进行局部配准处理，从而能够有效地减少融合图像中的边缘区域出现鬼影。
在一种可能的实现方式中，第一颜色空间可以是指Raw颜色空间。
在一种可能的实现方式中，第二颜色空间可以是指YUV颜色空间。
在本申请的实施例中，第三图像与第四图像可以是指YUV颜色空间的图像，可以在YUV颜色空间进行融合处理；由于YUV颜色空间进行融合处理时对算例的需求较小，因此，在YUV颜色空间基于至少两个掩膜对第三图像与第四图像进行融合处理，能够提高融合处理的效率。
结合第一方面，在第一方面的某些实现方式中，所述至少两个掩膜为所述第一掩膜与所述第三掩膜。
结合第一方面，在第一方面的某些实现方式中，所述至少两个掩膜为所述第一掩膜、所述第二掩膜与所述第三掩膜。
结合第一方面，在第一方面的某些实现方式中，所述电子设备还包括红外闪光灯，所述图像处理方法还包括：
在暗光场景下，开启所述红外闪光灯，所述暗光场景是指所述电子设备所处的拍摄环境的环境亮度小于第二预设阈值；
所述响应于所述第一操作，获取第一图像与第二图像，包括：
在开启所述红外闪光灯的情况下，获取所述第一图像与所述第二图像。
在本申请的实施例中，可以开启电子设备中的红外闪光灯；由于电子设备中可以包括第一相机模组与第二模组，在红外闪光灯开启的情况下，拍摄对象的反射光增加，使得第一相机模组的进光量增加；由于第一相机模组的进光量增加，通过第一相机模组采集的第一图像中包括的细节信息增加；通过本申请实施例的图像处理方法对第一相机模组与第二相机模组采集的图像进行融合处理，能够对主摄像头相机模组获取的图像进行图像增强，提高图像中的细节信息。此外，红外闪光灯是用户无法感知的，在用户无感知的情况下，提高图像中的细节信息。
结合第一方面，在第一方面的某些实现方式中，所述第一界面包括第二控件；所述在暗光场景下，开启所述红外闪光灯，包括：
检测到对所述第二控件的第二操作；
响应于所述第二操作开启所述红外闪光灯。
结合第一方面，在第一方面的某些实现方式中，所述第一界面是指拍照界面，所述第一控件是指用于指示拍照的控件。
可选地，第一操作可以是指对拍照界面中指示拍照的控件的点击操作。
结合第一方面，在第一方面的某些实现方式中，所述第一界面是指视频录制界面，所述第一控件是指用于指示录制视频的控件。
可选地，第一操作可以是指对视频录制界面中指示录制视频的控件的点击操作。
结合第一方面，在第一方面的某些实现方式中，所述第一界面是指视频通话界面，所述第一控件是指用于指示视频通话的控件。
可选地，第一操作可以是指对视频通话界面中指示视频通话的控件的点击操作。
应理解，上述以第一操作为点击操作为例进行举例说明；第一操作还可以包括语音指示操作，或者其它的指示电子设备进行拍照或者视频通话的操作；上述为举例说明，并不对本申请作任何限定。
第二方面，提供了一种电子设备，所述电子设备包括一个或多个处理器、存储器、第一相机模组与第二相机模组；所述第一相机模组为近红外相机模组或者红外相机模组，所述存储器与所述一个或多个处理器耦合，所述存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行：
显示第一界面，所述第一界面包括第一控件；
检测到对所述第一控件的第一操作；
响应于所述第一操作，获取第一图像与第二图像，所述第一图像为所述第一相机模组采集的图像，所述第二图像为所述第二相机模组采集的图像；
对所述第一图像进行第一图像处理，得到第三图像，所述第三图像为第一颜色空间的图像；
对所述第二图像进行第二图像处理，得到所述第二颜色空间的第四图像；
基于至少两个掩膜对所述第三图像与所述第四图像进行融合处理，得到融合图像，其中，所述至少两个掩膜包括第一掩膜、第二掩膜或者第三掩膜中的至少两项，所述第一掩膜用于标记所述第三图像中清晰度优于所述第四图像的图像区域，所述第二掩膜用于标记所述第三图像中的鬼影区域，所述第三掩膜用于标记所述第三图像中目标类别的拍摄对象所在的图像区域，所述融合图像的细节信息优于所述第二图像的细节信息。
结合第二方面，在第二方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行：
根据所述至少两个掩膜的交集得到目标掩膜；
根据所述目标掩膜对所述第三图像与所述第四图像进行融合处理，得到所述融合图像。
结合第二方面，在第二方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行：
将所述第一图像转换至第二颜色空间，得到第五图像；
以所述第四图像为基准对所述第五图像进行全局配准处理，得到第六图像；
以所述第四图像为基准对所述第六图像进行局部配准处理，得到所述第三图像。
结合第二方面，在第二方面的某些实现方式中，所述一个或多个处理器调用所述 计算机指令以使得所述电子设备执行：
对所述第四图像进行深度估计，得到深度信息，所述深度信息用于表示所述第四图像中的拍摄对象与所述电子设备的距离信息；
基于所述深度信息，筛选所述第四图像中满足预设条件的目标特征点；
以所述目标特征点为基准对所述第五图像进行全局配准处理，得到所述第六图像。
结合第二方面，在第二方面的某些实现方式中，所述预设条件为所述拍摄对象与所述电子设备的距离大于第一预设阈值。
结合第二方面，在第二方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行：
以所述第四图像对应的亮度通道图像为基准，对所述第六图像进行局部配准处理，得到所述第三图像。
结合第二方面，在第二方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行：
根据所述第四图像的高频信息与所述第六图像的高频信息的并集，得到第七图像；
基于所述第七图像确定所述第四图像中的第一图像区域；
基于所述第七图像确定所述第六图像中的第二图像区域；
以所述第一图像区域为基准对所述第二图像区域进行所述局部配准处理，得到所述第三图像。
结合第二方面，在第二方面的某些实现方式中，所述至少两个掩膜为所述第一掩膜与所述第三掩膜。
结合第二方面，在第二方面的某些实现方式中，所述至少两个掩膜为所述第一掩膜、所述第二掩膜与所述第三掩膜。
结合第二方面，在第二方面的某些实现方式中，所述电子设备还包括红外闪光灯，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行：
在暗光场景下，开启所述红外闪光灯，所述暗光场景是指所述电子设备所处的拍摄环境的环境亮度小于第二预设阈值；
在开启所述红外闪光灯的情况下，获取所述第一图像与所述第二图像。
结合第二方面，在第二方面的某些实现方式中，所述第一界面包括第二控件；所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行：
检测到对所述第二控件的第二操作；
响应于所述第二操作开启所述红外闪光灯。
结合第二方面，在第二方面的某些实现方式中，所述第一界面是指拍照界面，所述第一控件是指用于指示拍照的控件。
结合第二方面，在第二方面的某些实现方式中，所述第一界面是指视频录制界面，所述第一控件是指用于指示录制视频的控件。
结合第二方面，在第二方面的某些实现方式中，所述第一界面是指视频通话界面，所述第一控件是指用于指示视频通话的控件。
第三方面，提供了一种电子设备，包括用于执行第一方面或者第一方面中任一种图像处理方法的模块/单元。
第四方面，提供一种电子设备，所述电子设备包括一个或多个处理器、存储器、第一相机模组与第二相机模组；所述存储器与所述一个或多个处理器耦合，所述存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行第一方面或者第一方面中的任一种方法。
第五方面，提供了一种芯片***，所述芯片***应用于电子设备，所述芯片***包括一个或多个处理器，所述处理器用于调用计算机指令以使得所述电子设备执行第一方面或第一方面中的任一种方法。
第六方面，提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序代码，当所述计算机程序代码被电子设备运行时，使得该电子设备执行第一方面或第一方面中的任一种方法。
第七方面，提供了一种计算机程序产品，所述计算机程序产品包括：计算机程序代码，当所述计算机程序代码被电子设备运行时，使得该电子设备执行第一方面或第一面中的任一种方法。
在本申请的实施例中，电子设备中可以包括第一相机模组与第二相机模组，其中，第一相机模组为近红外相机模组或者红外相机模组；通过第一相机模组可以采集第一图像，通过第二相机模组可以采集第二图像；对第一图像与第二图像分别进行图像处理，可以得到第三图像与第四图像；基于至少两个掩膜可以对第三图像与第四图像进行融合处理，得到融合图像；由于第三图像为近红外图像或者红外图像，因此第三图像中可以包括第四图像中无法获取的信息，通过对第三图像与第四图像进行融合处理，可以实现近红外光的图像信息与可见光的图像信息的多光谱信息融合，使得融合后的图像中包括更多的细节信息；此外，在本申请的实施例中，是基于至少两个掩膜对第三图像与第四图像进行融合处理，可以从清晰度、去鬼影或者局部细节等至少两个方面进行图像增强；从而实现对第二相机模组获取(例如，主摄像头相机模组)的第二图像进行图像增强，增强图像中的细节信息，提高图像质量。
附图说明
图1是一种适用于本申请的电子设备的硬件***的示意图；
图2是一种适用于本申请的电子设备的电子设备的软件***的示意图；
图3是一种适用于本申请实施例的应用场景的示意图；
图4是一种适用于本申请实施例的应用场景的示意图；
图5是本申请实施例提供的图像处理方法的示意性流程图；
图6是本申请实施例提供的图像处理方法的示意性流程图；
图7是本申请实施例提供的图像处理方法的示意性流程图；
图8是本申请实施例提供的图像处理方法的示意性流程图；
图9是本申请实施例提供的目标掩膜的示意图；
图10是本申请实施例提供的一种全局配准方法的示意性流程图；
图11是本申请实施例提供的一种局部配准方法的示意性流程图；
图12是本申请实施例提供的一种局部配准方法的示意性流程图；
图13是根据是本申请实施例提供的图像处理方法的效果示意图；
图14是一种适用于本申请实施例的图形用户界面的示意图；
图15是一种适用于本申请实施例的拍摄场景的光路示意图；
图16是本申请实施例提供的一种电子设备的结构示意图；
图17是本申请实施例提供的一种电子设备的结构示意图。
具体实施方式
在本申请的实施例中，以下术语“第一”、“第二”、“第三”、“第四”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。
为了便于对本申请实施例的理解，首先对本申请实施例中涉及的相关概念进行简要说明。
1、近红外光(near infrared，NIR)
近红外光是指介于可见光与中红外光之间的电磁波；可以将近红外光区划分为近红外短波(780nm～1100nm)和近红外长波(1100nm～2526nm)两个区域。
2、主摄相机模组
主摄相机模组是指接收光谱范围为可见光的相机模组；例如，主摄相机模组中包括的传感器接收的光谱范围为400nm～700nm。
3、近红外相机模组
近红外相机模组是指接收光谱范围为近红外光的相机模组；例如，近红外相机模组中包括的传感器接收的光谱范围为700nm～1100nm。
4、图像的高频信息
图像的高频信息是指图像中灰度值变化剧烈的区域；例如，图像中的高频信息包括物体的边缘信息、纹理信息等。
5、图像的低频信息
图像的低频信息是指图像中灰度值变化缓慢的区域；对于一幅图像而言，除去高频信息外的部分为低频信息；例如，图像的低频信息可以包括物体边缘以内的内容信息。
6、图像配准(Image registration)
图像配准是指就是将不同时间、不同传感器(成像设备)或者不同条件下(天候、照度、摄像位置和角度等)获取的两幅或多幅图像进行匹配、叠加的过程。
7、亮度值(Lighting Value，LV)
亮度值用于估计环境亮度，其具体计算公式如下：
Figure PCTCN2022117324-appb-000001
其中，Exposure为曝光时间；Aperture为光圈大小；Iso为感光度；Luma为图像在XYZ空间中，Y的平均值。
8、特征点
在图像处理和与计算机视觉领域，特征点(feature points)也被称作关键点(key points)、兴趣点(interest points)；它被大量用于解决物体识别、图像识别、图像匹配、视觉跟踪、三维重建等一系列的问题(例如，图像中两个边缘的交点可以称为特征点)。
9、颜色校正矩阵(color correctionmatrix，CCM)
颜色校正矩阵用于校准除白色以外其他颜色的准确度。
10、三维查找表(Threedimension look up table，3DLUT)
三维查找表广泛应用于图像处理；例如，查找表可以用于图像颜色校正、图像增强或者图像伽马校正等；例如，可以在图像信号处理器中加载LUT，根据LUT表可以对原始图像进行图像处理，实现原始图像帧的像素值映射改变图像的颜色风格，从而实现不同的图像效果。
11、全局色调映射(Global tone Mapping，GTM)
全局色调映射用于解决高动态图像的灰度值分布不均匀的问题。
12、伽马处理
伽马处理用于通过调整伽马曲线来调整图像的亮度、对比度与动态范围等。
13、神经网络
神经网络是指将多个单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入；每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
14、反向传播算法
神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的神经网络模型中参数的大小，使得神经网络模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的神经网络模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的神经网络模型的参数，例如权重矩阵。
下面将结合附图，对本申请实施例中图像处理方法与电子设备进行描述。
图1示出了一种适用于本申请的电子设备的硬件***。
电子设备100可以是手机、智慧屏、平板电脑、可穿戴电子设备、车载电子设备、增强现实(augmented reality，AR)设备、虚拟现实(virtual reality，VR)设备、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、个人数字助理(personal digital assistant，PDA)、投影仪等等，本申请实施例对电子设备100的具体类型不作任何限制。
电子设备100可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E， 距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L，骨传导传感器180M等。
需要说明的是，图1所示的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中，电子设备100可以包括比图1所示的部件更多或更少的部件，或者，电子设备100可以包括图1所示的部件中某些部件的组合，或者，电子设备100可以包括图1所示的部件中某些部件的子部件。图1示的部件可以以硬件、软件、或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元。例如，处理器110可以包括以下处理单元中的至少一个：应用处理器(application processor，AP)、调制解调处理器、图形处理器(graphics processing unit，GPU)、图像信号处理器(image signal processor，ISP)、控制器、视频编解码器、数字信号处理器(digital signal processor，DSP)、基带处理器、神经网络处理器(neural-network processing unit，NPU)。其中，不同的处理单元可以是独立的器件，也可以是集成的器件。控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了***的效率。
示例性地，处理器110可以用于执行本申请实施例的图像处理方法；例如，显示第一界面，第一界面包括第一控件；检测到对第一控件的第一操作；响应于第一操作，获取第一图像与第二图像，第一图像为第一相机模组采集的图像，第二图像为第二相机模组采集的图像，第一图像与第二图像为第一颜色空间的图像；对第一图像进行第一图像处理，得到第三图像，第三图像为第二颜色空间的图像；对第二图像进行第二图像处理，得到第二颜色空间的第四图像；基于至少两个掩膜对第三图像与第四图像进行融合处理，得到融合图像，其中，至少两个掩膜包括第一掩膜、第二掩膜或者第三掩膜中的至少两项，第一掩膜用于标记第三图像中清晰度优于第四图像的图像区域，第二掩膜用于标记第三图像中的鬼影区域，第三掩膜用于标记第三图像中目标类别的拍摄对象所在的图像区域，融合图像的细节信息优于第二图像的细节信息。
图1所示的各模块间的连接关系只是示意性说明，并不构成对电子设备100的各模块间的连接关系的限定。可选地，电子设备100的各模块也可以采用上述实施例中多种连接方式的组合。
电子设备100的无线通信功能可以通过天线1、天线2、移动通信模块150、无线通信模块160、调制解调处理器以及基带处理器等器件实现。
天线1和天线2用于发射和接收电磁波信号。电子设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
电子设备100可以通过GPU、显示屏194以及应用处理器实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算， 用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏194可以用于显示图像或视频。
电子设备100可以通过ISP、摄像头193、视频编解码器、GPU、显示屏194以及应用处理器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP可以对图像的噪点、亮度和色彩进行算法优化，ISP还可以优化拍摄场景的曝光和色温等参数。在一些实施例中，ISP可以设置在摄像头193中。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的红绿蓝(red green blue，RGB)，YUV等格式的图像信号。在一些实施例中，电子设备100可以包括1个或N个摄像头193，N为大于1的正整数。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当电子设备100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。电子设备100可以支持一种或多种视频编解码器。这样，电子设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1、MPEG2、MPEG3和MPEG4。
陀螺仪传感器180B可以用于确定电子设备100的运动姿态。在一些实施例中，可以通过陀螺仪传感器180B确定电子设备100围绕三个轴(即，x轴、y轴和z轴)的角速度。陀螺仪传感器180B可以用于拍摄防抖。例如，当快门被按下时，陀螺仪传感器180B检测电子设备100抖动的角度，根据角度计算出镜头模组需要补偿的距离，让镜头通过反向运动抵消电子设备100的抖动，实现防抖。陀螺仪传感器180B还可以用于导航和体感游戏等场景。
加速度传感器180E可检测电子设备100在各个方向上(一般为x轴、y轴和z轴)加速度的大小。当电子设备100静止时可检测出重力的大小及方向。加速度传感器180E还可以用于识别电子设备100的姿态，作为横竖屏切换和计步器等应用程序的输入参数。
距离传感器180F用于测量距离。电子设备100可以通过红外或激光测量距离。在一些实施例中，例如在拍摄场景中，电子设备100可以利用距离传感器180F测距以实现快速对焦。
环境光传感器180L用于感知环境光亮度。电子设备100可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180L也可用于拍照时自动调节白平衡。环境光传感器180L还可以与接近光传感器180G配合，检测电子设备100是否在口袋 里，以防误触。
指纹传感器180H用于采集指纹。电子设备100可以利用采集的指纹特性实现解锁、访问应用锁、拍照和接听来电等功能。
触摸传感器180K，也称为触控器件。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，触摸屏也称为触控屏。触摸传感器180K用于检测作用于其上或其附近的触摸操作。触摸传感器180K可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于电子设备100的表面，并且与显示屏194设置于不同的位置。
上文详细描述了电子设备100的硬件***，下面介绍图像电子设备100的软件***。
图2是本申请实施例提供的装置的软件***的示意图。
如图2所示，***架构中可以包括应用层210、应用框架层220、硬件抽象层230、驱动层240以及硬件层250。
应用层210可以包括相机应用程序、图库、日历、通话、地图、导航、WLAN、蓝牙、音乐、视频、短信息等应用程序。
应用框架层220为应用层的应用程序提供应用程序编程接口(application programming interface，API)和编程框架；应用框架层可以包括一些预定义的函数。
例如，应用框架层220可以包括相机访问接口；相机访问接口中可以包括相机管理与相机设备。其中，相机管理可以用于提供管理相机的访问接口；相机设备可以用于提供访问相机的接口。
硬件抽象层230用于将硬件抽象化。比如，硬件抽象层可以包相机抽象层以及其他硬件设备抽象层；相机硬件抽象层可以调用相机算法库中的算法。
例如，相机算法库中可以包括用于图像处理的软件算法。
驱动层240用于为不同硬件设备提供驱动。例如，驱动层可以包括相机设备驱动；数字信号处理器驱动、图形处理器驱动或者中央处理器驱动。
硬件层250可以包括相机设备以及其他硬件设备。
例如，硬件层250包括相机设备、数字信号处理器、图形处理器或者中央处理器；示例性地，相机设备中可以包括图像信号处理器，图像信号处理器可以用于图像处理。
目前，终端设备上的主摄像头相机模组获取的光谱范围为可见光(400nm～700nm)；在一些拍照场景中，例如，光照条件较差的拍摄场景中；比如，夜晚场景或者浓雾场景中，由于拍摄场景的光线条件较差，电子设备的进光量较少，导致主摄像头相机模组获取的图像中存在部分图像细节信息丢失的问题。
有鉴于此，本申请的实施例提供了一种图像处理方法，该图像处理方法可以应用于电子设备；电子设备中可以包括第一相机模组与第二相机模组，其中，第一相机模组为近红外相机模组或者红外相机模组；通过第一相机模组可以采集第一图像，通过第二相机模组可以采集第二图像；对第一图像与第二图像分别进行图像处理，可以得到第三图像与第四图像；基于至少两个掩膜可以对第三图像与第四图像进行融合处理，得到融合图像；由于第三图像为近红外图像或者红外图像，因此第三图像中可以包括第 四图像中无法获取的信息，通过对第三图像与第四图像进行融合处理，可以实现近红外光的图像信息与可见光的图像信息的多光谱信息融合，使得融合后的图像中包括更多的细节信息；此外，在本申请的实施例中，是基于至少两个掩膜对第三图像与第四图像进行融合处理，可以从清晰度、去鬼影或者局部细节等至少两个方面进行图像增强；从而实现对第二相机模组获取(例如，主摄像头相机模组)的第二图像进行图像增强，增强图像中的细节信息，提高图像质量。
下面结合图3对本申请实施例提供的图像处理方法的应用场景进行举例说明。
示例性地，本申请实施例中的图像处理方法可以应用于拍照领域(例如，单景拍照、双景拍照等)、录制视频领域、视频通话领域或者其他图像处理领域；由于本申请实施例中采用的是双相机模组，双相机模组包括可以获取近红外光的第一相机模组(例如，近红外相机模组，或者，红外相机模组)与可以获取可见光的第二相机模组；通过基于至少两个掩膜(例如，清晰度掩膜、鬼影掩膜或者语音分割掩膜)可以对可见光图像与近红外光图像进行图像处理与融合处理，得到画质增强的图像；通过本申请实施例中的图像处理方法对图像进行处理，能够增强图像中的细节信息，提高图像质量。
在一个示例中，如图3所示，本申请实施例应用于阳光下拍摄风景(例如，云雾场景)时，由于近红外相机模组可以获取的光谱范围为近红外光，与可见光光谱范围相比近红外相机模组可以获取的光谱的波长较长，因此绕射能力较强，例如，波长较长的光谱的穿透性更强，采集的图像的画面通透感更强；通过主摄像头相机模组与近红外相机模组采集图像，通过本申请实施例提供的图像处理方法对两个相机模组采集的图像进行融合处理，得到的融合图像，如图3所示；图3所示的融合图像的细节信息较丰富，可以清晰的显示山脉的细节信息；通过本申请实施例提供的图像处理方法可以对主摄像头模组获取的图像进行图像增强，增强图像中的细节信息。
示例性地，图3所示的终端设备可以包括第一相机模组、第二相机模组以及红外闪光灯；其中，第一相机模组可以获取的光谱范围为近红外光(700nm～1100nm)；第二相机模组可以获取的光谱范围包括但不限于可见光(400nm～700nm)。
在一个示例中，本申请实施例应用于包括绿色景物的场景拍照时，对于进光量较少的暗光区域，由于近红外光对绿色景物的反射率较高，因此通过主摄像头相机模组与近红外相机模组拍摄得到的绿色景物的细节信息更多，能够增强图像中暗光区域中绿色景物的细节信息。
在一个示例中，本申请实施例提供的图像处理方法可以应用于夜景人像拍摄；在夜景人像拍摄时，可以开启电子设备中的红外闪光灯；例如，人像可以包括拍摄对象面部的脸、眼睛、鼻子、嘴巴、耳朵、眉毛等；由于电子设备中包括主摄像头相机模组与近红外相机模组，在红外闪光灯开启的情况下，拍摄对象的反射光增加，使得近红外相机模组的进光量增加；从而使得通过近红外相机模组拍摄的人像的细节信息增加，通过本申请实施例的图像处理方法对主摄像头相机模组与近红外相机模组采集的图像进行融合处理，能够对主摄像头相机模组获取的图像进行图像增强，提高图像中的细节信息。此外，红外闪光灯是用户无法感知的，在用户无感知的情况下，提高图像中的细节信息。
可选地，在本申请的实施例中，电子设备在检测到食物或者人像时可以关闭近红外相机模组。
例如，在食物拍摄场景中可以包括多个食物，近红外相机模组可以采集多个食物中部分食物的图像；例如，多个食物可以为桃子、苹果或者西瓜等，近红外相机模组可以采集桃子和苹果的图像，且不采集西瓜的图像。
可选地，近红外相机模组可以显示提示信息，提示用于是否开启近红外相机模组；在用户授权开启近红外相机模组后，近红外相机模组才能够开启采集图像。
在一个示例中，本申请实施例中的图像处理方法可以应用于折叠屏终端设备中；例如，折叠屏终端设备可以包括外屏与内屏；在折叠屏终端设备的外屏与内屏之间的夹角为零度时，可以在外屏上显示预览图像，如图4中的(a)所示；在折叠屏终端设备的外屏与内屏之间的夹角为锐角时，可以在外屏上显示预览图像，如图4中的(b)所示；在折叠屏终端设备的外屏与内屏之间的夹角为钝角时，可以在内屏上的一侧显示预览图像，另一侧显示用于指示拍摄的控件，如图4中的(c)所示；在折叠屏终端设备的外屏与内屏之间的夹角为180度时，可以在内屏上显示预览图像，如图4中的(d)所示；上述预览图像可以是通过本申请实施例提供的图像处理方法对采集的图像进行处理得到的。示例性地，图4所示的折叠屏终端设备可以包括第一相机模组、第二相机模组以及红外闪光灯；其中，第一相机模组可以获取的光谱范围为可见光(400nm～700nm)；第一相机模组为近红外相机模组或者红外相机模组；第二相机模组可以为可见光相机模组，或者，其他可以获取但不限于可见光的相机模组。
应理解，上述为对本申请实施例的应用场景的举例说明，并不对本申请的应用场景作任何限定。
下面结合图5至图15对本申请实施例提供的图像处理方法进行详细描述。
图5是本申请实施例提供的图像处理方法的示意图。该图像处理方法可以由图1所示的电子设备执行；该方法200包括步骤S201至步骤S206，下面分别对步骤S201至步骤S206进行详细的描述。
应理解，图5所示的图像处理方法应用于电子设备，电子设备中包括第一相机模组与第二相机模组，第一相机模组获取的光谱为近红外相机模组或者红外相机模组(例如，获取的光谱范围为700nm～1100nm)。
可选地，第二相机模组可以为可见光相机模组(例如，获取的光谱范围为400nm～700nm)，或者第二相机模组为其他可以获取可见光的相机模组(例如，获取的光谱范围包括400nm～700nm)。
步骤S201、显示第一界面，第一界面包括第一控件。
可选地，第一界面可以是指电子设备的拍照界面，第一控件可以是指拍照界面中用于指示拍照的控件，如图3或者图4所示。
可选地，第一界面可以是指电子设备的视频录制界面，第一控件可以是指视频录制界面中用于指示录制视频的控件。
可选地，第一界面可以是指电子设备的视频通话界面，第一控件可以是指视频通话界面用于指示视频通话的控件。
步骤S202、检测到对第一控件的第一操作。
可选地，第一操作可以是指对拍照界面中指示拍照的控件的点击操作。
可选地，第一操作可以是指对视频录制界面中指示录制视频的控件的点击操作。
可选地，第一操作可以是指对视频通话界面中指示视频通话的控件的点击操作。
应理解，上述以第一操作为点击操作为例进行举例说明；第一操作还可以包括语音指示操作，或者其它的指示电子设备进行拍照或者视频通话的操作；上述为举例说明，并不对本申请作任何限定。
步骤S203、响应于第一操作，获取第一图像与第二图像。
其中，第一图像为第一相机模组采集的图像；第二图像为第二相机模组采集的图像；第一图像与第二图像为第一颜色空间的图像。
示例性地，响应于第一操作，第一相机模组可以采集第一图像，第二相机模组可以采集第二图像；例如，第一相机模组与第二相机模组可以同时采集图像。
可选地，第一颜色空间可以是指Raw颜色空间；第一图像与第二图像可以是指Raw颜色空间的图像。
例如，第一图像可以是指Raw颜色空间的NIR图像；第二图像可以是指Raw颜色空间的RGB图像。
应理解，在本申请的实施例中，Raw颜色空间的NIR图像可以是指NIRRaw；NIRRaw可以是指单通道的图像；NIR Raw图像用于表示光子叠加在一起的强度信息；例如，NIR Raw图像可以是在单通道的灰度图像。
步骤S204、对第一图像进行第一图像处理，得到第三图像。
其中，第三图像为第二颜色空间的图像。
可选地，第二颜色空间可以是指YUV颜色空间，或者其他颜色空间。
可选地，对第一图像进行第一图像处理，得到第三图像，包括：
将第一图像转换至第二颜色空间，得到第五图像；以第四图像为基准对第五图像进行全局配准处理，得到第六图像；以第四图像为基准对第六图像进行局部配准处理，得到第三图像。
在本申请的实施例中，由于第一相机模组与第二相机模组分别设置在电子设备中的不同位置，因此第一相机模组与第二相机模组之间存在一定的基线距离，即通过第一相机模组采集的图像与通过第二相机模组采集的图像之间存在一定的视差，通过全局配准处理与局部配准处理可以消除第三图像与第四图像之间的视差；从而在对第三图像与第四图像进行融合处理时，能够降低融合图像中的鬼影。
可选地，上述以第四图像为基准对第五图像进行全局配准处理，得到第六图像，包括：
对第四图像进行深度估计，得到深度信息，深度信息用于表示第四图像中的拍摄对象与所述电子设备的距离信息；基于深度信息，筛选第四图像中满足预设条件的目标特征点；以目标特征点为基准对第五图像进行全局配准处理，得到第六图像。
应理解，特征点是指图像灰度值发生剧烈变化的点，或者在图像边缘上曲率较大的点；目标特征点可以是指第四图像中满足预设条件的特征点。
例如，预设条件可以为拍摄对象与电子设备的距离大于第一预设阈值；即筛选第四图像中拍摄对象中远景对应的特征点作为目标特征点；以目标特征点为基准对第五图像进行全局配准处理。
应理解，全局配准处理是为了使得对第三图像与第四图像进行融合处理时，避免 融合图像中出现鬼影区域；由于第三图像中拍摄对象的远景区域的图像细节优于第四图像中拍摄对象的远景区域的图像细节，第四图像中拍摄对象的近景区域的图像细节优于第三图像中拍摄对象的近景区域的图像细节；因此，可以从第三图像中选取远景区域对应的图像区域与第四图像进行融合；则对于第三图像进行全局配准时，可以不考虑第三图像中拍摄对象的近景区域，将第三图像中拍摄对象的远景区域与第四图像中拍摄对象的远景区域(例如，目标特征点对应的局部图像区域)进行配准处理，从而实现第三图像与第四图像的全局配准。
在本申请的实施例中，可以基于深度信息进行全局配准处理；由于第三图像(例如，NIR图像)与第四图像(例如，RGB图像)相比并非所有的像素位置均具有较好的效果；例如，对于拍摄对象中的近景，第三图像中的细节信息低于第四图像中的细节信息；若从第三图像中提取较多近景对应的特征点，可能导致第三图像中的远景与第四图像无法配准，使得融合图像中容易出现鬼影问题；因此，在对第三图像进行全局配准时，可以基于深度信息选取第四图像中的目标特征点；以第四图像中的目标特征点为基准对第三图像进行配准处理。
可选地，以第四图像为基准对第六图像进行局部配准处理，得到第三图像，包括：
以第四图像对应的亮度通道图像为基准，对第六图像进行局部配准处理，得到第三图像。
在本申请的实施例中，由于第六图像(例如，全局配准后的NIR图像)与第四图像(例如，RGB图像)的亮度可能差别较大；通过对第六图像进行局部配准处理，即可以对第六图像进行亮度处理，使得第三图像与第四图像的亮度接近；由于第三图像与第四图像的亮度接近，因此在对第三图像与第四图像进行融合处理时，能够有效地降低融合图像出现颜色失真的问题。
可选地，以第四图像为基准对第六图像进行局部配准处理，得到第三图像，包括：
根据第四图像的高频信息与第六图像的高频信息的并集，得到第七图像；基于第七图像确定第四图像中的第一图像区域；基于第七图像确定第六图像中的第二图像区域；以第一图像区域为基准对第二图像区域进行局部配准处理，得到第三图像。
在本申请的实施例中，由于第六图像的高频信息与第四图像的高频可能存在差异，即第六图像中拍摄对象的边缘区域与第四图像中拍摄对象的边缘区域可能存在差异；因此，可以对第六图像的边缘区域进行局部配准处理，从而能够有效地减少融合图像中的边缘区域出现鬼影。
步骤S205、对第二图像进行第二图像处理，得到第二颜色空间的第四图像。
可选地，可以将第二图像转换至第二颜色空间，得到第四图像。
示例性地，第二颜色空间可以是指YUV颜色空间，或者其他颜色空间。
可选地，第二图像处理可以包括ISP图像处理；ISP处理可以包括Raw算法处理、RGB算法处理或者YUV算法处理。
例如，Raw算法处理可以包括但不限于：
黑电平校正(blacklevelcorrection，BLC)、镜头阴影校正(lensshadingcorrection，LSC)、自动白平衡(autowhitebalance，AWB)或者去马赛克等。
其中，黑电平校正用于对黑电平进行校正处理；镜头阴影校正用于消除由于镜头 光学***原因造成的图像四周颜色以及亮度与图像中心不一致的问题；自动白平衡用于使得白色在任何色温下相机均能呈现出白色。
需要说明的是，上述以黑电平校正、镜头阴影校正、自动白平衡、去马赛克为例进行举例说明；本申请对Raw算法处理并不作任何限定。
例如，RGB算法处理包括但不限于：
颜色校正矩阵处理，或者三维查找表处理等。
其中，颜色校正矩阵(color correctionmatrix，CCM)，用于校准除白色以外其他颜色的准确度。三维查找表(Look Up Table，LUT)广泛应用于图像处理；例如，查找表可以用于图像颜色校正、图像增强或者图像伽马校正等；例如，可以在图像信号处理器中加载LUT，根据LUT表可以对原始图像进行图像处理，实现原始图像映射到其他图像的颜色风格，从而实现不同的图像效果。
需要说明的是，上述以颜色校正矩阵处理与三维查找表处理为例进行举例说明；本申请对RGB图像处理并不作任何限定。
例如，YUV算法处理包括但不限于：
全局色调映射处理或者伽马处理等。
其中，全局色调映射(Global tone Mapping，GTM)用于解决高动态图像的灰度值分布不均匀的问题。伽马处理用于通过调整伽马曲线来调整图像的亮度、对比度与动态范围等。
需要说明的是，上述以全局色调映射处理与伽马处理为例进行举例说明；本申请对YUV图像处理并不作任何限定。
步骤S206、基于至少两个掩膜对第三图像与第四图像进行融合处理，得到融合图像。
其中，至少两个掩膜包括第一掩膜、第二掩膜或者第三掩膜中的至少两项，第一掩膜用于标记第三图像中清晰度优于第四图像的图像区域，第二掩膜用于标记第三图像中的鬼影区域，第三掩膜用于标第三图像中目标类别的拍摄对象所在的图像区域，融合图像的细节信息优于第二图像的细节信息。
应理解，融合图像的细节信息优于第二图像的细节信息可以是指融合图像中的细节信息多于第二图像中的细节信息；或者，融合图像的细节信息优于第二图像的细节信息可以是指融合图像的清晰度优于第二图像的清晰度。也可以是其他情况，本申请不进行限定。例如，细节信息可以包括拍摄对象的边缘信息、纹理信息等(例如，发丝边缘，人脸细节，衣服褶皱、大量树木的每颗树木边缘，绿植的枝叶脉络等)。
应理解，在本申请的实施例中，由于第三图像中并非所有像素点的清晰度均优于第四图像；通过第一掩膜(例如，清晰度掩膜)可以从第三图像中获取清晰度优于第四图像的图像区域；将该图像区域与第四图像进行融合，从而能够提高融合图像的清晰度。
应理解，在本申请的实施例中，由于第三图像与第四图像中可能存在部分区域可能无法配准；通过第二掩膜(例如，鬼影掩膜)可以剔除第三图像中无法与第四图像配准的区域；从而能够有效地避免融合图像中出现鬼影。
应理解，在本申请的实施例中，由于近红外光对不同类别的物体反射率不同，从而导致对于不同的物体第三图像中包括的细节信息不同；因此，根据第三掩膜(例如，语义分割掩膜)可以从第三图像中获取目标类别(例如，绿色植物、远山等)的拍摄对象所在的 图像区域；通过该图像区域与第四图像进行融合处理，能够提高融合图像中的局部细节信息。
可选地，至少两个掩膜为第一掩膜与第三掩膜；例如，可以获取清晰度掩膜与语义分割掩膜；基于清晰度掩膜与语义分割掩膜对第三图像与第四图像进行融合处理，得到融合图像。
可选地，至少两个掩膜为第一掩膜、第二掩膜与第三掩膜；可以获取清晰度掩膜、鬼影掩膜与语义分割掩膜；基于清晰度掩膜、鬼影掩膜与语义分割掩膜对第三图像与第四图像进行融合处理，得到融合图像。
可选地，第三图像与第四图像可以是指YUV颜色空间的图像，可以在YUV颜色空间进行融合处理；由于YUV颜色空间进行融合处理时对算例的需求较小，因此，在YUV颜色空间基于至少两个掩膜对第三图像与第四图像进行融合处理，能够提高融合处理的效率。
可选地，基于至少两个掩膜对第三图像与第四图像进行融合处理，得到融合图像，包括：
根据至少两个掩膜的交集得到目标掩膜；根据目标掩膜对第三图像与第四图像进行融合处理，得到融合图像。
例如，可以获取清晰度掩膜、鬼影掩膜与语义分割掩膜；根据清晰度掩膜、鬼影掩膜与语义分割掩膜的交集，得到目标掩膜；如图9所示，清晰度掩膜中像素为1的位置可以表示该区域第三图像的清晰度大于第四图像的清晰度的置信度为1；清晰度掩膜中像素为0的位置可以表示该区域第三图像的清晰度大于第四图像的清晰度的置信度为0；鬼影掩膜中像素为1的位置可以表示该区域为相对于第四图像为鬼影区域的置信度为1；鬼影掩膜中像素为0的位置可以表示该区域为相对于第四图像为鬼影区域的置信度为0；语义分割掩膜中像素为1的位置可以表示第三图像中该区域为目标类别的拍摄对象的置信度为1；语义分割掩膜中像素为0的位置可以表示第三图像中该区域为目标类别的拍摄对象的置信度为0；基于清晰度掩膜、鬼影掩膜与语义分割掩膜的交集，得到目标掩膜；例如，对于清晰度掩膜、鬼影掩膜与语义分割掩膜中均为1的区域，目标掩膜对应的该区域的像素值为1；对于清晰度掩膜、鬼影掩膜与语义分割掩膜中均为0的区域，目标掩膜对应的该区域的像素值为0；对于清晰度掩膜、鬼影掩膜与语义分割掩膜中部分为1的区域，目标掩膜对应的该区域的像素值为0。
可选地，可以根据目标掩膜获取第三图像中的局部图像区域；将第三图像的局部图像区域与第四图像进行融合处理，得到融合图像。
例如，将目标掩膜与第三图像进行逐像素相乘，基于目标掩膜中不同区域的像素值，可以确定第三图像中的局部图像区域；比如，可以得到目标掩膜的像素为1所在的图像区域对应的第三图像的局部图像区域；将第三图像的局部图像区域与第四图像进行融合处理，可以得到融合图像。
在本申请的实施例中，电子设备中可以包括第一相机模组与第二相机模组，其中，第一相机模组为近红外相机模组或者红外相机模组；通过第一相机模组可以采集第一图像，通过第二相机模组可以采集第二图像；对第一图像与第二图像分别进行图像处 理，可以得到第三图像与第四图像；基于至少两个掩膜可以对第三图像与第四图像进行融合处理，得到融合图像；由于第三图像为近红外图像或者红外图像，因此第三图像中可以包括第四图像中无法获取的信息，通过对第三图像与第四图像进行融合处理，可以实现近红外光的图像信息与可见光的图像信息的多光谱信息融合，使得融合后的图像中包括更多的细节信息；此外，在本申请的实施例中，是基于至少两个掩膜对第三图像与第四图像进行融合处理，可以从清晰度、去鬼影或者局部细节等至少两个方面进行图像增强；从而实现对第二相机模组获取(例如，主摄像头相机模组)的第二图像进行图像增强，增强图像中的细节信息，提高图像质量。
图6是本申请实施例提供的图像处理方法的示意图。该图像处理方法可以由图1所示的电子设备执行；该方法300包括步骤S301至步骤S306，下面分别对步骤S301至步骤S306进行详细的描述。
应理解，图6所示的图像处理方法可以应用于如图1所示的电子设备，该电子设备包括第一相机模组与第二相机模组；其中，第一相机模组为近红外相机模组，或者红外相机模组(例如，获取的光谱范围为700nm～1100nm)；第二相机模组可以为可见光相机模组，或者其他可以获取可见光的相机模组(例如，获取的光谱范围包括400nm～700nm)。
步骤S301、第一相机模组采集NIRRaw图像(第一图像的一个示例)。
其中，NIRRaw图像可以是指Raw颜色空间的NIR图像。
示例性地，第一相机模组可以是近红外相机模组，或者红外相机模组；第一相机模组可以包括第一镜片、第一镜头与图像传感器，第一镜片可以通过的光谱范围为近红外光(700nm～1100nm)。
应理解，第一镜片可以是指滤光镜片；第一镜片可以用于吸收某些特定波段的光，让近红外光波段的光通过。
还应理解，在本申请的实施例中第一相机模组采集的NIRRaw图像可以是指单通道的图像；NIRRaw图像用于表示光子叠加在一起的强度信息；例如，NIRRaw图像可以是在单通道的灰度图像。
步骤S302、第二相机模组采集RGBRaw图像(第二图像的一个示例)。
其中，RGBRaw图像可以是指Raw颜色空间的RGB图像。
示例性地，第二相机模组可以为可见光相机模组，或者其他可以获取可见光的相机模组(例如，获取的光谱范围包括400nm～700nm)；第二相机模组可以包括第二镜片、第二镜头与图像传感器，第二镜片可以通过的光谱范围包括可见光(400nm～700nm)。
可选地，上述步骤S301与步骤S302可以是同步执行的；即第一相机模组与第二相机模组可以同步出帧，分别获取NIRRaw图像与RGBRaw图像。
步骤S303、对步骤S301输出的图像进行ISP处理。
可选地，对NIRRaw图像进行ISP处理，得到NIR图像(第三图像的一个示例)。
步骤S304、对步骤S302输出的图像进行ISP处理。
可选地，对RGBRaw图像进行ISP处理，得到RGB图像(第四图像的一个示例)。
可选地，步骤S303与步骤S304可以没有时序要求；例如，步骤S303与步骤S304也可以是同时执行的。
可选地，步骤S303与步骤S304可以部分相同，或者完全相同。
可选地，ISP处理可以包括Raw算法处理、RGB算法处理或者YUV算法处理。
例如，Raw算法处理可以包括但不限于：
黑电平校正(blacklevelcorrection，BLC)、镜头阴影校正(lensshadingcorrection，LSC)、自动白平衡(autowhitebalance，AWB)、去马赛克等。
其中，黑电平校正用于对黑电平进行校正处理；镜头阴影校正用于消除由于镜头光学***原因造成的图像四周颜色以及亮度与图像中心不一致的问题；自动白平衡用于使得白色在任何色温下相机均能呈现出白色。
需要说明的是，上述以黑电平校正、镜头阴影校正、自动白平衡、去马赛克为例进行举例说明；本申请对Raw算法处理并不作任何限定。
例如，RGB算法处理包括但不限于：
颜色校正矩阵处理，或者三维查找表处理等。
其中，颜色校正矩阵(color correctionmatrix，CCM)，用于校准除白色以外其他颜色的准确度。三维查找表(Look Up Table，LUT)广泛应用于图像处理；例如，查找表可以用于图像颜色校正、图像增强或者图像伽马校正等；例如，可以在图像信号处理器中加载LUT，根据LUT表可以对原始图像进行图像处理，实现原始图像映射到其他图像的颜色风格，从而实现不同的图像效果。
需要说明的是，上述以颜色校正矩阵处理与三维查找表处理为例进行举例说明；本申请对RGB图像处理并不作任何限定。
例如，YUV算法处理包括但不限于：
全局色调映射处理或者伽马处理等。
其中，全局色调映射(Global tone Mapping，GTM)用于解决高动态图像的灰度值分布不均匀的问题。伽马处理用于通过调整伽马曲线来调整图像的亮度、对比度与动态范围等。
需要说明的是，上述以全局色调映射处理与伽马处理为例进行举例说明；本申请对YUV图像处理并不作任何限定。
步骤S305、获取至少两个掩膜。
可选地，至少两个掩膜可以为清晰度掩膜与语义分割掩膜；例如，基于清晰度掩膜与语义分割掩膜对第三图像与第四图像进行融合处理，得到融合图像。
可选地，至少两个掩膜可以为清晰度掩膜、鬼影掩膜与语义分割掩膜。
示例性地，清晰度掩膜可以用于标记NIR图像中清晰度优于RGB图像的图像区域，鬼影掩膜可以用于标记NIR图像中的鬼影区域，语义分割掩膜用于标NIR图像中目标类别的拍摄对象所在的图像区域。
示例性地，可以对NIR图像进行分块处理，得到N个NIR图像块；计算N个NIR图像块中每一个NIR图像块的方差A；对RGB图像进行分块处理，得到N个RGB图像块；计算N个RGB图像块中每一个RGB图像块的方差B；在一个NIR图像块的方差A大于对应的RGB图像块的方差B时，该图像块位置对应的清晰度掩膜为1，1可以用于表示NIR图像清晰度大于RGB图像的置信度为1；在一个NIR图像块的方差A小于或者等于对应的RGB图像块的方差B时，该图像块位置对应的清晰度掩膜为0， 0可以用于表示NIR图像清晰度大于RGB图像的置信度为0。
应理解，在本申请的实施例中，通过清晰度掩膜可以确定NIR图像中清晰度优于RGB图像的图像区域；将NIR图像中的局部区域(例如，清晰度优于RGB图像的图像区域)与RGB图像进行融合处理，从而能够提高融合图像的清晰度，提高融合图像的图像质量。
示例性地，可以对NIR图像与RGB图像进行鬼影检测，得到鬼影掩膜。
示例性地，可以通过索贝尔算子对NIR图像与RGB图像进行滤波处理，得到NIR图像的梯度图像与RGB图像的梯度图像；其中，梯度图像用于表示纹理信息的变化快慢；通过对两个梯度图像作差，可以得到鬼影掩膜。
需要说明的是，索贝尔算子(Sobel operator)主要用作边缘检测；它是一种离散性差分算子，用来运算图像亮度函数的梯度之近似值；在图像的任何一点使用此算子，将会产生对应的梯度矢量或是其法矢量。
应理解，在本申请的实施例中，通过鬼影掩膜可以在NIR图像中选取局部区域(例如，非鬼影的图像区域)与RGB图像进行融合处理，从而避免融合图像中出现鬼影区域，提高融合图像的图像质量。
示例性地，可以获取第四图像对应的语义分割图像，从语义分割图像中选取部分标签，得到语义分割掩膜；例如，语义分割图像中包括5个标签分别为标签0～标签4；其中，标签0用于标记人像；标签1用于标记动物；标签2用于标记植物(例如，绿色植物)；标签3用于标签远处景物(例如，远山)；标签4用于标记天空；可以从语义分割图像中多个标签中选取部分标签，比如，选取标签2与标签3的图像区域标记为1，其余图像区域标记为0，得到语义分割掩膜。
应理解，对于部分景物(例如，绿色植物或者远山)NIR图像中具有更多的细节信息；例如，由于近红外光对绿色景物的反射率较高，因此NIR图像与RGB图像相比，具有更多绿色景物的细节信息更多；例如，由于近红外光与可见光相比波长较长，因此近红外光的绕射能力较强；此外，波长较长的光的穿透性更强，采集的图像的画面通透感更强；因此，NIR图像与RGB图像相比，对于远处景物(例如，远山)NIR图像中包括更多的细节信息(例如，远山的纹理信息)；通过选取NIR图像中的局部区域(例如，包括更多细节信息的图像区域)与RGB图像进行融合处理，能够增强融合图像的局部细节信息，提高融合图像的图像质量。
步骤S306、融合处理。
可选地，基于至少两个掩膜对NIR图像与RGB图像进行融合处理，得到融合图像。
应理解，基于至少两个掩膜对NIR图像与RGB图像进行融合处理，可以从清晰度、去鬼影或者局部细节等至少两个方面进行图像增强；从而实现对RGB图像进行图像增强，增强图像中的细节信息，提高图像质量。
可选地，在电子设备处于非暗光场景(例如，电子设备所处的拍摄环境的环境亮度大于或者等于第二预设阈值)时，可以通过上述步骤S301至步骤S306执行本申请实施例提供的图像处理方法。
可选地，电子设备中还可以包括红外闪光灯；在电子设备处于暗光场景时，即电 子设备所处的拍摄环境的环境亮度小于第二预设阈值的情况下(例如，可以根据亮度值进行判断)，电子设备可以执行图7所示的方法400；例如，可以执行步骤S400开启红外闪光灯；在红外闪光灯开启之后，可以通过第一相机模组获取NIRRaw图像，通过第二相机模组获取RGBRaw图像，执行如图7所示的步骤S401至步骤S406；应理解，步骤S401至步骤S406适用于步骤S301至步骤S306的相关描述，此处不再赘述。
可选地，由于电子设备的亮度值越大，表示电子设备的环境亮度越高；可以通过电子设备的亮度值确定电子设备所处的拍摄环境的环境亮度，在电子设备的亮度值小于第二预设阈值时，则可以电子设备开启红外闪光灯。
其中，亮度值用于估计环境亮度，其具体计算公式如下：
Figure PCTCN2022117324-appb-000002
其中，Exposure为曝光时间；Aperture为光圈大小；Iso为感光度；Luma为图像在XYZ空间中，Y的平均值。
示例性地，在暗光场景下，电子设备在检测到拍摄指示后可以先进行对焦，同步进行场景检测；识别到暗光场景并完成对焦后可以开启红外闪光灯，红外闪光灯开启后，NIRRaw图像与RGBRaw图像可以同步出帧。
应理解，对于暗光场景电子设备所处的拍摄环境的环境亮度较低；电子设备开启红外闪光灯后，可以使得第一相机模组获取的反射光增加，从而增加第一相机模组的进光量；使得第一相机模组采集的NIRRaw图像的清晰度增加；由于NIRRaw图像的清晰度增加，使得NIR图像的清晰度增加；由于NIR图像的清晰度增加，使得融合图像的清晰度增加。
在本申请的实施例中，电子设备中可以包括第一相机模组与第二相机模组，其中，第一相机模组为近红外相机模组或者红外相机模组；通过第一相机模组可以采集NIR Raw图像，通过第二相机模组可以采集RGB Raw图像；对NIR Raw图像与RGB Raw图像分别进行图像处理，可以得到NIR图像与RGB图像；基于至少两个掩膜可以对NIR图像与RGB图像进行融合处理，得到融合图像；由于NIR图像为近红外图像或者红外图像，因此NIR图像中可以包括RGB图像中无法获取的信息，通过对NIR图像与RGB图像进行融合处理，可以实现近红外光的图像信息与可见光的图像信息的多光谱信息融合，使得融合后的图像中包括更多的细节信息；此外，在本申请的实施例中，是基于至少两个掩膜对NIR图像与RGB图像进行融合处理，可以从清晰度、去鬼影或者局部细节等至少两个方面进行图像增强；从而实现对第二相机模组获取(例如，主摄像头相机模组)的RGB进行图像增强，增强图像中的细节信息，提高图像质量。
图8是本申请实施例提供的图像处理方法的示意图。该图像处理方法可以由图1所示的电子设备执行；该方法500包括步骤S501至步骤S512，下面分别对步骤S501至步骤S512进行详细的描述。
应理解，图8所示的图像处理方法可以应用于如图1所示的电子设备，该电子设备包括第一相机模组与第二相机模组；其中，第一相机模组为近红外相机模组，或者 红外相机模组(例如，获取的光谱范围为700nm～1100nm)；第二相机模组可以为可见光相机模组，或者其他可以获取可见光的相机模组(例如，获取的光谱范围包括但不限于400nm～700nm)。
步骤S501、第一相机模组采集NIRRaw图像(第一图像的一个示例)。
其中，NIRRaw图像可以是指Raw颜色空间的NIR图像。
示例性地，第一相机模组可以是近红外相机模组，或者红外相机模组；第一相机模组可以包括第一镜片、第一镜头与图像传感器，第一镜片可以通过的光谱范围为近红外光(700nm～1100nm)。
应理解，第一镜片可以是指滤光镜片；第一镜片可以用于吸收某些特定波段的光，让近红外光波段的光通过。
还应理解，在本申请的实施例中第一相机模组采集的NIRRaw图像可以是指单通道的图像；NIRRaw图像用于表示光子叠加在一起的强度信息；例如，NIRRaw图像可以是在单通道的灰度图像。
步骤S502、第二相机模组采集RGBRaw图像(第二图像的一个示例)。
其中，RGBRaw图像可以是指Raw颜色空间的RGB图像。
示例性地，第二相机模组可以为可见光相机模组，或者其他可以获取可见光的相机模组(例如，获取的光谱范围包括400nm～700nm)；第二相机模组可以包括第二镜片、第二镜头与图像传感器，第二镜片可以通过的光谱范围包括但不限于可见光(400nm～700nm)。
可选地，上述步骤S501与步骤S502可以是同步执行的；即第一相机模组与第二相机模组可以同步出帧，分别获取NIRRaw图像与RGBRaw图像。
步骤S503、对步骤S501输出的图像进行ISP处理。
可选地，对NIRRaw图像进行ISP处理。
步骤S504、对步骤S502输出的图像进行ISP处理。
可选地，对RGBRaw图像进行ISP处理。
可选地，步骤S503与步骤S504可以没有时序要求；例如，步骤S503与步骤S504也可以是同时执行的。
可选地，步骤S503与步骤S504可以部分相同或者全部相同。
可选地，ISP处理可以包括Raw算法处理、RGB算法处理或者YUV算法处理。
例如，Raw算法处理可以包括但不限于：
黑电平校正(blacklevelcorrection，BLC)、镜头阴影校正(lensshadingcorrection，LSC)、自动白平衡(autowhitebalance，AWB)或者去马赛克等。
其中，黑电平校正用于对黑电平进行校正处理；镜头阴影校正用于消除由于镜头光学***原因造成的图像四周颜色以及亮度与图像中心不一致的问题；自动白平衡用于使得白色在任何色温下相机均能呈现出白色。
需要说明的是，上述以黑电平校正、镜头阴影校正、自动白平衡、去马赛克为例进行举例说明；本申请对Raw算法处理并不作任何限定。
例如，RGB算法处理包括但不限于：
颜色校正矩阵处理，或者三维查找表处理等。
其中，颜色校正矩阵(color correctionmatrix，CCM)，用于校准除白色以外其他颜色的准确度。三维查找表(Look Up Table，LUT)广泛应用于图像处理；例如，查找表可以用于图像颜色校正、图像增强或者图像伽马校正等；例如，可以在图像信号处理器中加载LUT，根据LUT表可以对原始图像进行图像处理，实现原始图像映射到其他图像的颜色风格，从而实现不同的图像效果。
需要说明的是，上述以颜色校正矩阵处理与三维查找表处理为例进行举例说明；本申请对RGB图像处理并不作任何限定。
例如，YUV算法处理包括但不限于：
全局色调映射处理或者伽马处理等。
其中，全局色调映射(Global tone Mapping，GTM)用于解决高动态图像的灰度值分布不均匀的问题。伽马处理用于通过调整伽马曲线来调整图像的亮度、对比度与动态范围等。
需要说明的是，上述以全局色调映射处理与伽马处理为例进行举例说明；本申请对YUV图像处理并不作任何限定。
可选地，上述步骤S503为举例说明；也可以通过其他方法得到NIR图像；上述步骤S504为举例说明；也可以通过其他方法得到RGB图像；本申请对此不作任何限定。
步骤S505、得到NIR图像(第五图像的一个示例)。
可选地，NIR图像可以是指YUV颜色空间的NIR图像。
步骤S506、得到RGB图像(第四图像的一个示例)。
可选地，RGB图像可以是指YUV颜色空间的RGB图像。
步骤S507、全局配准处理。
应理解，由于第一相机模组与第二相机模组分别设置在电子设备中的不同位置，因此第一相机模组与第二相机模组之间存在一定的基线距离，即通过第一相机模组采集的图像与通过第二相机模组采集的图像之间存在一定的视差，通过全局配准处理可以消除NIR图像与RGB图像之间的视差。
可选地，通过全局配准处理是以RGB图像为基准，将NIR图像的整体映射到RGB图像的坐标系为例进行举例说明；或者，全局配准处理也可以是以NIR图像为基准，将RGB图像的整体映射到NIR图像的坐标系。
可选地，可以基于深度信息选取RGB图像中的目标像素位置；基于目标像素位置，得到RGB图像中的目标特征点；以RGB图像中的目标特征点为基准对NIR图像进行全局配准处理；可选地，全局配准处理的具体流程可以参见图10所示。
应理解，特征点是指图像灰度值发生剧烈变化的点，或者在图像边缘上曲率较大的点；目标特征点是指RGB中满足预设条件的特征点；例如，目标特征点可以是指RGB图像中深度信息大于第一预设阈值的特征点。
在本申请的实施例中，可以基于深度信息进行全局配准处理；由于NIR图像与RGB图像相比并非所有的像素位置均具有较好的效果；例如，对于拍摄对象中的近景，NIR图像中的细节信息低于RGB图像中的细节信息；若从NIR图像中提取较多近景对应的特征点，可能导致NIR图像中的远景与RGB图像无法配准，使得融合图像中容易 出现鬼影问题；因此，在对NIR图像进行全局配准时，可以基于深度信息选取RGB图像中的目标像素位置；基于RGB图像中的目标像素位置得到RGB图像中的目标特征点，以RGB图像中的目标特征点为基准对NIR图像进行全局配准处理。
步骤S508、得到全局配准的NIR图像(第六图像的一个示例)。
步骤S509、局部配准处理。
可选地，对全局配准的NIR图像进行局部配准处理。
在本申请的实施例中，在全局配准处理的基础上可以进一步执行局部配准处理，使得全局配准的NIR图像中的局部细节进行再次图像配准处理；从而能够提高融合图像的局部细节信息。
步骤S510、得到局部配准的NIR图像(第三图像的一个示例)。
可选地，可以通过图11或者图12所示的局部配准方法对全局配准后的NIR图像进行局部配准处理，得到局部配准后的NIR图像。
步骤S511、获取至少两个掩膜。
应理解，在本申请的实施例可以获取至少两个掩膜的信息；在对NIR图像与RGB图像进行融合处理时，可以基于至少两个掩膜的信息进行融合处理，可以从清晰度、去鬼影或者局部细节等至少两个方面进行图像增强；能够提高融合图像的图像质量。
可选地，在本申请的实施例中，至少两个掩膜可以包括但不限于：清晰度掩膜、鬼影掩膜，或者语义分割掩膜中的至少两项。
应理解，在本申请的实施例中，由于NIR图像中并非所有像素点的清晰度均优于RGB图像；通过清晰度掩膜可以从NIR图像中获取清晰度优于RGB图像的图像区域；将该图像区域与RGB图像进行融合，从而能够提高融合图像的清晰度。
还应理解，在本申请的实施例中，由于在全局配准处理与局部配准处理后，NIR图像与RGB图像中可能依然存在部分区域可能无法配准；通过鬼影掩膜可以剔除NIR图像中无法与RGB图像配准的区域；进一步，基于鬼影掩膜对NIR图像与RGB图像进行融合处理，从而能够有效减少融合图像中出现鬼影区域。
还应理解，在本申请的实施例中，由于近红外光对不同类别的物体，反射率不同，从而导致对于不同的物体NIR图像中包括的细节信息不同；因此，基于语义分割掩膜可以从NIR图像中获取目标类别物体的图像区域；通过该图像区域与RGB图像进行融合处理，能够提高融合图像中的局部细节信息。
可选地，可以获取清晰度掩膜与语义分割掩膜；基于清晰度掩膜与语义分割掩膜对局部配准后的NIR图像与RGB图像进行融合处理。
可选地，可以获取清晰度掩膜、鬼影掩膜与语义分割掩膜；基于清晰度掩膜、鬼影掩膜与语义分割掩膜对局部配准后的NIR图像与RGB图像进行融合处理。
例如，可以获取清晰度掩膜、鬼影掩膜与语义分割掩膜；根据清晰度掩膜、鬼影掩膜与语义分割掩膜的交集，得到目标掩膜，如图9所示此处不再赘述；基于目标掩膜对局部配准后的NIR图像(第三图像的一个示例)与RGB图像(第四图像的一个示例)进行融合处理。
在本申请的实施例中，通过清晰度掩膜可以在NIR图像中标记NIR图像优于RGB图像的局部图像区域；通过鬼影掩膜可以在NIR图像中标记NIR图像中的非鬼影的局 部区域或者鬼影区域；通过语义分割图像可以在NIR图像中标记NIR图像中具有更多细节信息的局部图像区域。
示例性地，可以对NIR图像进行分块处理，得到N个NIR图像块；计算N个NIR图像块中每一个NIR图像块的方差A；对RGB图像进行分块处理，得到N个RGB图像块；计算N个RGB图像块中每一个RGB图像块的方差B；在一个NIR图像块的方差A大于对应的RGB图像块的方差B时，该图像块位置对应的清晰度掩膜为1，1可以用于表示NIR图像的清晰度大于RGB图像的清晰度的置信度为1；在一个NIR图像块的方差A小于或者等于对应RGB图像块的方差B时，该图像块位置对应的清晰度掩膜为0，0可以用于表示NIR图像的清晰度大于RGB图像的清晰度的置信度为0。
应理解，在本申请的实施例中，通过清晰度掩膜可以确定NIR图像中清晰度优于RGB图像的图像区域；将NIR图像中的局部区域(例如，清晰度优于RGB图像的图像区域)与RGB图像进行融合处理，从而能够提高融合图像的清晰度，提高融合图像的图像质量。
示例性地，可以对NIR图像与RGB图像进行鬼影检测，得到鬼影掩膜。
示例性地，可以通过索贝尔算子对NIR图像与RGB图像进行滤波处理，得到NIR图像的梯度图像与RGB图像的梯度图像；其中，梯度图像用于表示纹理信息的变化快慢；通过对两个梯度图像作差，可以得到鬼影掩膜。
需要说明的是，索贝尔算子(Sobel operator)主要用作边缘检测。它是一离散性差分算子，用来运算图像亮度函数的梯度之近似值；在图像的任何一点使用此算子，将会产生对应的梯度矢量或是其法矢量。
应理解，在本申请的实施例中，通过鬼影掩膜可以在NIR图像中选取局部图像区域(例如，非鬼影的图像区域)与RGB图像进行融合处理，从而避免融合图像中出现鬼影区域，提供融合图像的图像质量。
示例性地，可以从语义分割图像中选取部分标签，得到语义分割掩膜；例如，语义分割图像中包括5个标签分别为标签0～标签4；其中，标签0用于标记人像；标签1用于标记动物；标签2用于标记植物(例如，绿色植物)；标签3用于标签远处景物(例如，远山)；标签4用于标记天空；可以从语义分割图像中多个标签中选取部分标签，比如，选取标签2与标签3的图像区域标记为1，其余图像区域标记为0，得到语义分割掩膜。
应理解，对于部分景物(例如，绿色植物或者远山)NIR图像中具有更多的细节信息；例如，由于近红外光对绿色景物的反射率较高，因此NIR图像与RGB图像相比，具有更多绿色景物的细节信息更多；例如，由于近红外光与可见光相比波长较长，因此近红外光的绕射能力较强；此外，波长较长的光的穿透性更强，采集的图像的画面通透感更强；因此，NIR图像与RGB图像相比，对于远处景物(例如，远山)NIR图像中包括更多的细节信息(例如，远山的纹理信息)；通过选取NIR图像中的局部区域(例如，包括更多细节信息的图像区域)与RGB图像进行融合处理，能够增强融合图像的局部细节信息，提高融合图像的图像质量。
步骤S512、融合处理。
可选地，基于至少两个掩膜对局部配准后的NIR图像(第三图像的一个示例)与 RGB图像(第四图像的一个示例)进行融合处理，得到融合图像。
在本申请的实施例中，电子设备中可以包括第一相机模组与第二相机模组，其中，第一相机模组为近红外相机模组或者红外相机模组；通过第一相机模组可以采集NIR Raw图像，通过第二相机模组可以采集RGB Raw图像；对NIR Raw图像与RGB Raw图像分别进行图像处理，可以得到NIR图像与RGB图像；基于至少两个掩膜可以对NIR图像与RGB图像进行融合处理，得到融合图像；由于NIR图像为近红外图像或者红外图像，因此NIR图像中可以包括RGB图像中无法获取的信息，通过对NIR图像与RGB图像进行融合处理，可以实现近红外光的图像信息与可见光的图像信息的多光谱信息融合，使得融合后的图像中包括更多的细节信息；此外，在本申请的实施例中，是基于至少两个掩膜对NIR图像与RGB图像进行融合处理，可以从清晰度、去鬼影或者局部细节等至少两个方面进行图像增强；从而实现对第二相机模组获取(例如，主摄像头相机模组)的RGB图像进行图像增强，增强图像中的细节信息，提高图像质量。
下面对上述图8所示的步骤S507全局配准处理进行举例说明。
图10是本申请实施例提供的一种全局配准方法的示意图。该方法600包括步骤S601至步骤S604，下面分别对步骤S601至步骤S604进行详细的描述。
步骤S601、对RGB图像进行深度估计。
可选地，对RGB图像进行深度估计，得到深度信息。
应理解，深度估计可以是指估计图像中每个像素相对于相机模组的距离信息。
可选地，可以通过深度估计算法得到RGB图像的深度信息；通过RGB图像的深度信息可以区分RGB图像中的近景区域与远景区域。
示例性地，深度估计算法是指获取图像中场景里的每个点到相机的距离信息的算法；深度估计算法可以包括单目深度估计算法、双目深度估计算法等。
步骤S602、获取RGB图像中部分像素的位置信息。
可选地，根据深度信息，获取RGB图像中部分像素的位置信息。
可选地，可以将深度信息与第一预设阈值进行比较，可以确定RGB图像中深度信息大于第一预设阈值的部分像素的位置信息。
步骤S603、根据位置信息提取RGB图像中的特征点。
可选地，根据位置信息还可以提取NIR图像中的部分特征点；或者，对NIR图像进行深度估计，得到NIR图像的深度信息；基于NIR图像的深度信息，确定NIR图像中深度信息大于第一预设阈值的部分像素位置；本申请对此不作任何限定。
步骤S604、得到全局配准的NIR图像。
可选地，以RGB图像中的部分特征点为基准，对NIR图像中的部分特征点进行配准处理得到全局配准的NIR图像。
应理解，全局配准处理是为了使得对NIR图像与RGB图像进行融合处理时，避免融合图像中出现鬼影区域；由于NIR图像中拍摄对象的远景区域的图像细节优于RGB图像中拍摄对象的远景区域的图像细节，RGB图像中拍摄对象的近景区域的图像细节优于NIR图像中拍摄对象的近景区域的图像细节；因此，可以从NIR图像中选取远景区域对应的图像区域与RGB图像进行融合；即对于NIR图像进行全局配准时， 可以不考虑NIR图像中拍摄对象的近景区域，将NIR图像中拍摄对象的远景区域与RGB图像中拍摄对象的远景区域进行配准处理，从而实现NIR图像与RGB图像的全局配准。
可选地，提取NIR图像中的部分特征点与RGB图像中的部分特征点，通过单应性矩阵将NIR图像中的部分特征点映射到RGB图像的部分特征点。
可选地，图8所示的步骤S509局部配准处理的方法可以如图11所示。
在本申请的实施例中，由于全局配准后的NIR图像的亮度与RGB图像的亮度可能差别较大，若直接对全局配准后的NIR图像与RGB图像进行融合处理，则融合图像会出现亮度失真的问题；比如，融合图像可能会出现颜色发灰，或者融合图像颜色不自然等问题。因此，可以对全局配准后的NIR图像进行局部配准处理，即可以对全局配准后的NIR图像进行亮度处理，使得全局配准后的NIR图像的亮度与RGB图像的亮度接近；由于NIR图像的亮度与RGB图像的亮度接近，使得融合处理后的融合图像的颜色准确性更高，提高图像质量。
图11是本申请实施例提供的一种局部配准方法的示意图。该方法700包括步骤S701至步骤S706，下面分别对步骤S701至步骤S706进行详细的描述。
步骤S701、获取NIR图像(第五图像的一个示例)。
可选地，可以通过第一相机模组采集NIRRaw图像；对NIRRaw图像进行图像处理，得到NIR图像。
示例性地，第一相机模组可以为近红外相机模组，或者第一相机模组为红外光相机模组。
例如，第一相机模组可以包括第一镜片、第一镜头与图像传感器，第一镜片可以通过的光谱范围为近红外光(700nm～1100nm)。
应理解，第一镜片可以是指滤光镜片；第一镜片可以用于吸收某些特定波段的光，让近红外光波段的光通过。
步骤S702、获取RGB图像并进行颜色空间转换。
可选地，获取RGB图像并进行颜色空间转换，得到YUV图像。
应理解，局部配准可以基于图像中的亮度值进行的配准处理；因此，需要将RGB图像转换至其他能够抽取亮度通道图像的颜色空间，从而获取RGB图像对应的亮度通道图像。
可选地，可以通过第二相机模组采集RGBRaw图像；对RGBRaw图像进行图像处理，得到RGB图像；将RGB图像转换至YUV颜色空间，得到YUV图像。
示例性地，第二相机模组可以为可见光相机模组；或者第二相机模组可以为其他可以获取可见光的相机模组；本申请对此不作任何限定。
例如，第二相机模组可以包括第二镜片、第二镜头与图像传感器，第二镜片可以通过的光谱范围为可见光(400nm～700nm)，或者，第二镜片可以通过的光谱范围包括但不限于可见光(400nm～700nm)。
应理解，第二镜片可以是指滤光镜片；第一镜片可以用于吸收某些特定波段的光，让可见光波段的光通过。
步骤S703、提取Y通道。
可选地，提取YUV图像的Y通道。
步骤S704、神经网络模型处理。
可选地，通过神经网络模型对NIR图像进行亮度处理。
应理解，通常可以通过光流图对图像进行局部配准处理，光流图的精度会受到灰阶的影响；对于同一拍摄场景(例如，绿色植物的场景)RGB图像的灰阶可能在150以下，而NIR图像的灰阶可能在200以上；对于相同的拍摄场景，RGB图像与NIR图像的灰阶差异较大；通过神经网络模型可以对NIR图像的亮度进行处理，使得NIR图像中亮度灰阶过高的区域亮度灰阶降低，且亮度灰阶过低的区域亮度灰阶增加。
可选地，神经网络模型可以是通过反向传播算法预先训练得到的神经网络。
步骤S705、得到亮度映射后的NIR图像。
应理解，亮度映射后的NIR图像的图像内容与NIR图像保持一致，亮度映射后的NIR图像的图像亮度与RGB图像保持一致。
步骤S706、配准处理。
可选地，以Y通道图像为基准，对亮度映射后的NIR图像进行配准处理，得到局部配准后的NIR图像(第三图像的一个示例)。
在本申请的实施例中，通过对NIR图像的亮度与RGB图像的亮度进行配准处理，使得对于同一拍摄对象，NIR图像的亮度与RGB图像的亮度接近；例如，可以对NIR图像的亮度进行映射处理，得到亮度映射后的NIR图像；获取RGB图像对应的亮度通道，对亮度映射后的NIR图像与RGB图像的亮度通道进行配准处理；由于NIR图像的亮度与RGB图像的亮度接近，使得融合处理后的融合图像的颜色准确性更高，提高图像质量。
可选地，在本申请的实施例中，图8所示的步骤S509局部配准处理的方法可以如图12所示。
在本申请的实施例中，由于全局配准后的NIR图像与RGB图像的边缘区域可能存在差异，若直接对全局配准后的NIR图像与RGB图像进行融合处理，则融合图像的边缘区域可能会出现鬼影；因此，可以对全局配准后的NIR图像的边缘区域进行局部配准处理，从而减少融合图像边缘区域出现鬼影。
图12是本申请实施例提供的一种局部配准处理方法的示意图。该方法800包括步骤S801至步骤S808；下面分别对步骤S801至步骤S808进行详细的描述。
步骤S801、获取NIR图像(第五图像的一个示例)。
示例性地，可以通过第一相机模组采集NIRRaw图像；对NIRRaw图像进行图像处理，得到NIR图像。其中，第一相机模组可以为近红外相机模组，或者第一相机模组为红外光相机模组。
例如，第一相机模组可以包括第一镜片、第一镜头与图像传感器，第一镜片可以通过的光谱范围为近红外光(700nm～1100nm)。
应理解，第一镜片可以是指滤光镜片；第一镜片可以用于吸收某些特定波段的光，让近红外光波段的光通过。
步骤S802、提取NIR图像的高频信息。
应理解，图像的高频信息是指图像中灰度值变化剧烈的区域；例如，图像中的高 频信息包括物体的边缘信息、纹理信息等。
步骤S803、获取RGB图像(第四图像的一个示例)。
示例性地，可以通过第二相机模组采集RGBRaw图像；对RGBRaw图像进行图像处理，得到RGB图像。其中，第二相机模组为可见光相机模组；或者第二相机模组可以为其他相机模组；本申请对此不作任何限定。
例如，第二相机模组可以包括第二镜片、第二镜头与图像传感器，第二镜片可以通过的光谱范围为可见光(400nm～700nm)，或者，第二镜片可以通过的光谱范围包括但不限于可见光(400nm～700nm)以及其他光。
步骤S804、提取RGB图像的高频信息。
应理解，图像的高频信息是指图像中灰度值变化剧烈的区域；例如，图像中的高频信息包括物体的边缘信息、纹理信息等。
步骤S805、得到边缘差异图像。
可选地，根据NIR图像的高频信息与RGB图像的高频信息，得到边缘差异图像(第七图像的一个示例)。
其中，边缘差异图像用于表示NIR图像与RGB图像中的所有高频信息。
可选地，可以根据NIR图像的高频信息与RGB图像的高频信息的并集，得到边缘差异图像。
步骤S806、获取NIR图像中的局部图像区域。
可选地，基于边缘差异图像获取NIR图像中的局部图像区域(第一图像区域的一个示例)。
可选地，可以将边缘差异图像与NIR图像相乘，得到NIR图像中的局部区域。
例如，将边缘差异图像与NIR图像相乘可以是指将边缘差异图像与NIR图像中对应的像素点的像素值进行相乘。
步骤S807、获取RGB图像中的局部图像区域。
可选地，基于边缘差异图像获取RGB图像中的局部图像区域(第二图像区域的一个示例)。
可选地，可以将边缘差异图像与RGB图像相乘，得到RGB图像中的局部区域。
例如，将边缘差异图像与RGB图像相乘可以是指将边缘差异图像与RGB图像中对应的像素点的像素值进行相乘。
步骤S808、配准处理。
可选地，以RGB图像中的局部图像区域为基准，对NIR图像中的局部图像区域进行配准处理，得到局部配准后的NIR图像(第三图像的一个示例)。
在本申请的实施例中，通过获取NIR图像与RGB图像中的高频信息，可以得到边缘差异图像；边缘差异图像中包括NIR图像与RGB图像中的全部高频信息，通过边缘差异图像可以分别从NIR图像与RGB图像中获取局部图像区域；通过对两个局部区域进行配准处理，例如，以RGB图像中的局部图像区域为基准，对NIR图像中的局部区域进行配准处理，从而能够使得NIR图像中的高频信息与RGB图像中的高频信息配准，从而在一定程度上能够避免融合图像中出现鬼影。
可选地，图12所示的步骤S801至步骤S808也可以在图11所示的步骤S706之 后的进一步处理；即可以通过图11对全局配准处理的NIR图像进行亮度处理；进一步，可以采用图12所示的方法对亮度处理后的NIR图像进行边缘区域配准，从而得到局部配准后的NIR图像。
图13根据是本申请实施例提供的图像处理方法的效果示意图。
如图13所示，图13中的(a)是通过现有的主摄像头相机模组得到的输出图像；图13中的(b)是通过本申请实施例提供的图像处理方法得到的输出图像；如图13中的(a)所示的图像可以看出山脉中的细节信息出现了严重失真；与图13中的(a)所示的输出图像相比，图13中的(b)所示的输出图像的细节信息较丰富，可以清晰的显示山脉的细节信息；通过本申请实施例提供的图像处理方法可以对主摄像头相机模组获取的图像进行图像增强，提高图像中的细节信息。
在一个示例中，在暗光场景下，用户可以开启电子设备中的红外闪光灯；通过主摄像头相机模组与近红外相机模组采集图像，并通过本申请实施例提供的图像处理方法对采集的图像进行处理，从而输出处理后的图像或者视频。
图14示出了电子设备的一种图形用户界面(graphical user interface，GUI)。
图14中的(a)所示的GUI为电子设备的桌面910；当电子设备检测到用户点击桌面910上的相机应用(application，APP)的图标920的操作后，可以启动相机应用，显示如图14中的(b)所示的另一GUI；图14中的(b)所示的GUI可以是相机APP在拍照模式下的显示界面，在GUI可以包括拍摄界面930；拍摄界面930中可以包括取景框931与控件；比如，拍摄界面930中可以包括用于指示拍照的控件932与用于指示开启红外闪光灯的控件933；在预览状态下，该取景框931内可以实时显示预览图像；其中，预览状态下可以是指用户打开相机且未按下拍照/录像按钮之前，此时取景框内可以实时显示预览图。
在电子设备检测到用户点击指示开启红外闪光灯的控件933的操作后，显示如图14中的(c)所示的拍摄界面；在红外闪光灯开启的情况下，可以通过主摄像头相机模组与近红外相机模组采集图像，并通过本申请实施例提供的图像处理方法对采集的图像进行融合处理，输出处理后的融合图像。
图15是一种适用于本申请实施例的拍摄场景的光路示意图。
如图15所示，电子设备还包括红外闪光灯；在暗光场景下，电子设备可以开启红外闪光灯；在开启红外闪光灯的情况下，拍摄环境中的光照可以包括路灯与红外闪光灯；拍摄对象可以对拍摄环境中的光照进行反射，使得电子设备得到拍摄对象的图像。
在本申请的实施例中，在红外闪光灯开启的情况下，拍摄对象的反射光增加，使得电子设备中近红外相机模组的进光量增加；从而使得通过近红外相机模组拍摄的图像的细节信息增加，通过本申请实施例的图像处理方法对主摄像头相机模组与近红外相机模组采集的图像进行融合处理，能够对主摄像头相机模组获取的图像进行图像增强，提高图像中的细节信息。此外，红外闪光灯是用户无法感知的，在用户无感知的情况下，提高图像中的细节信息。
应理解，上述举例说明是为了帮助本领域技术人员理解本申请实施例，而非要将本申请实施例限于所例示的具体数值或具体场景。本领域技术人员根据所给出的上述举例说明，显然可以进行各种等价的修改或变化，这样的修改或变化也落入本申请实 施例的范围内。
上文结合图1至图15详细描述了本申请实施例提供的图像处理方法；下面将结合图16与图17详细描述本申请的装置实施例。应理解，本申请实施例中的装置可以执行前述本申请实施例的各种方法，即以下各种产品的具体工作过程，可以参考前述方法实施例中的对应过程。
图16是本申请实施例提供的一种电子设备的结构示意图。该电子设备1000包括显示模块1010与处理模块1020。该电子设备中包括第一相机模组与第二相机模组，第一相机模组为近红外相机模组或者红外相机模组。
其中，所述显示模块1010用于显示第一界面，所述第一界面包括第一控件；处理模块1020用于检测到对所述第一控件的第一操作；响应于所述第一操作，获取第一图像与第二图像，所述第一图像为所述第一相机模组采集的图像，所述第二图像为所述第二相机模组采集的图像，所述第一图像与所述第二图像为第一颜色空间的图像；对所述第一图像进行第一图像处理，得到第三图像，所述第三图像为第二颜色空间的图像；对所述第二图像进行第二图像处理，得到所述第二颜色空间的第四图像；基于至少两个掩膜对所述第三图像与所述第四图像进行融合处理，得到融合图像，其中，所述至少两个掩膜包括第一掩膜、第二掩膜或者第三掩膜中的至少两项，所述第一掩膜用于标记所述第三图像中清晰度优于所述第四图像的图像区域，所述第二掩膜用于标记所述第三图像中的鬼影区域，所述第三掩膜用于标记所述第三图像中目标类别的拍摄对象所在的图像区域，所述融合图像的细节信息优于所述第二图像的细节信息。
可选地，作为一个实施例，所述处理模块1020具体用于：
根据所述至少两个掩膜的交集得到目标掩膜；
根据所述目标掩膜对所述第三图像与所述第四图像进行融合处理，得到所述融合图像。
可选地，作为一个实施例，所述处理模块1020具体用于：
将所述第一图像转换至第二颜色空间，得到第五图像；
以所述第四图像为基准对所述第五图像进行全局配准处理，得到第六图像；
以所述第四图像为基准对所述第六图像进行局部配准处理，得到所述第三图像。
可选地，作为一个实施例，所述处理模块1020具体用于：
对所述第四图像进行深度估计，得到深度信息，所述深度信息用于表示所述第四图像中的拍摄对象与所述电子设备的距离信息；
基于所述深度信息，筛选所述第四图像中满足预设条件的目标特征点；
以所述目标特征点为基准对所述第五图像进行全局配准处理，得到所述第六图像。
可选地，作为一个实施例，所述预设条件为所述拍摄对象与所述电子设备的距离大于第一预设阈值。
可选地，作为一个实施例，所述处理模块1020具体用于：
以所述第四图像对应的亮度通道图像为基准，对所述第六图像进行局部配准处理，得到所述第三图像。
可选地，作为一个实施例，所述处理模块1020具体用于：
根据所述第四图像的高频信息与所述第六图像的高频信息的并集，得到第七图像；
基于所述第七图像确定所述第四图像中的第一图像区域；
基于所述第七图像确定所述第六图像中的第二图像区域；
以所述第一图像区域为基准对所述第二图像区域进行所述局部配准处理，得到所述第三图像。
可选地，作为一个实施例，所述至少两个掩膜为所述第一掩膜与所述第三掩膜。
可选地，作为一个实施例，所述至少两个掩膜为所述第一掩膜、所述第二掩膜与所述第三掩膜。
可选地，作为一个实施例，所述电子设备还包括红外闪光灯，所述处理模块1020具体用于：
在暗光场景下，开启所述红外闪光灯，所述暗光场景是指所述电子设备所处的拍摄环境的环境亮度小于第二预设阈值；
在开启所述红外闪光灯的情况下，获取所述第一图像与所述第二图像。
可选地，作为一个实施例，所述第一界面包括第二控件；所述处理模块1020具体用于：
检测到对所述第二控件的第二操作；
响应于所述第二操作开启所述红外闪光灯。
可选地，作为一个实施例，所述第一界面是指拍照界面，所述第一控件是指用于指示拍照的控件。
可选地，作为一个实施例，所述第一界面是指视频录制界面，所述第一控件是指用于指示录制视频的控件。
可选地，作为一个实施例，所述第一界面是指视频通话界面，所述第一控件是指用于指示视频通话的控件。
需要说明的是，上述电子设备1000以功能模块的形式体现。这里的术语“模块”可以通过软件和/或硬件形式实现，对此不作具体限定。
例如，“模块”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit，ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。
因此，在本申请的实施例中描述的各示例的单元，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
图17示出了本申请提供的一种电子设备的结构示意图。图17中的虚线表示该单元或该模块为可选的；电子设备1100可以用于实现上述方法实施例中描述的方法。
电子设备1100包括一个或多个处理器1101，该一个或多个处理器1101可支持电子设备1100实现方法实施例中的图像处理方法。处理器1101可以是通用处理器或者专用处理器。例如，处理器1101可以是中央处理器(central processing unit，CPU)、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array，FPGA) 或者其它可编程逻辑器件，如分立门、晶体管逻辑器件或分立硬件组件。
处理器1101可以用于对电子设备1100进行控制，执行软件程序，处理软件程序的数据。电子设备1100还可以包括通信单元1105，用以实现信号的输入(接收)和输出(发送)。
例如，电子设备1100可以是芯片，通信单元1105可以是该芯片的输入和/或输出电路，或者，通信单元1105可以是该芯片的通信接口，该芯片可以作为终端设备或其它电子设备的组成部分。
又例如，电子设备1100可以是终端设备，通信单元1105可以是该终端设备的收发器，或者，通信单元1105可以是该终端设备的收发电路。
电子设备1100中可以包括一个或多个存储器1102，其上存有程序1104，程序1104可被处理器1101运行，生成指令1103，使得处理器1101根据指令1103执行上述方法实施例中描述的图像处理方法。
可选地，存储器11002中还可以存储有数据。
可选地，处理器1101还可以读取存储器1102中存储的数据，该数据可以与程序1104存储在相同的存储地址，该数据也可以与程序1104存储在不同的存储地址。
处理器1101和存储器1102可以单独设置，也可以集成在一起，例如，集成在终端设备的***级芯片(system on chip，SOC)上。
示例性地，存储器1102可以用于存储本申请实施例中提供的图像处理方法的相关程序1104，处理器1101可以用于在执行图像处理时调用存储器1102中存储的图像处理方法的相关程序1104，执行本申请实施例的图像处理方法；例如，显示第一界面，第一界面包括第一控件；检测到对第一控件的第一操作；响应于第一操作，获取第一图像与第二图像，第一图像为第一相机模组采集的图像，第二图像为第二相机模组采集的图像，第一图像与第二图像为第一颜色空间的图像；对第一图像进行第一图像处理，得到第三图像，第三图像为第二颜色空间的图像；对第二图像进行第二图像处理，得到第二颜色空间的第四图像；基于至少两个掩膜对第三图像与第四图像进行融合处理，得到融合图像，其中，至少两个掩膜包括第一掩膜、第二掩膜或者第三掩膜中的至少两项，第一掩膜用于标记第三图像中清晰度优于第四图像的图像区域，第二掩膜用于标记第三图像中的鬼影区域，第三掩膜用于标记第三图像中目标类别的拍摄对象所在的图像区域，融合图像的细节信息优于第二图像的细节信息。
本申请还提供了一种计算机程序产品，该计算机程序产品被处理器1101执行时实现本申请中任一方法实施例的图像处理方法。
该计算机程序产品可以存储在存储器1102中，例如是程序1104，程序1104经过预处理、编译、汇编和链接等处理过程最终被转换为能够被处理器1101执行的可执行目标文件。
本申请还提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被计算机执行时实现本申请中任一方法实施例所述的图像处理方法。该计算机程序可以是高级语言程序，也可以是可执行目标程序。
该计算机可读存储介质例如是存储器1102。存储器1102可以是易失性存储器或非易失性存储器，或者，存储器1102可以同时包括易失性存储器和非易失性存储器。 其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(dynamic RAM，DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的***、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的***、装置和方法，可以通过其它的方式实现。例如，以上所描述的电子设备的实施例仅仅是示意性的，例如，所述模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
应理解，在本申请的各种实施例中，各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请的实施例的实施过程构成任何限定。
另外，本文中的术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而 前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准总之，以上所述仅为本申请技术方案的较佳实施例而已，并非用于限定本申请的保护范围。凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (18)

  1. 一种图像处理方法，其特征在于，应用于电子设备，所述电子设备包括第一相机模组与第二相机模组，所述第一相机模组为近红外相机模组或者红外相机模组，所述图像处理方法包括：
    显示第一界面，所述第一界面包括第一控件；
    检测到对所述第一控件的第一操作；
    响应于所述第一操作，获取第一图像与第二图像，所述第一图像为所述第一相机模组采集的图像，所述第二图像为所述第二相机模组采集的图像，所述第一图像与所述第二图像为第一颜色空间的图像；
    对所述第一图像进行第一图像处理，得到第三图像，所述第三图像为第二颜色空间的图像；
    对所述第二图像进行第二图像处理，得到所述第二颜色空间的第四图像；
    基于至少两个掩膜对所述第三图像与所述第四图像进行融合处理，得到融合图像，其中，所述至少两个掩膜包括第一掩膜、第二掩膜或者第三掩膜中的至少两项，所述第一掩膜用于标记所述第三图像中清晰度优于所述第四图像的图像区域，所述第二掩膜用于标记所述第三图像中的鬼影区域，所述第三掩膜用于标记所述第三图像中目标类别的拍摄对象所在的图像区域，所述融合图像的细节信息优于所述第二图像的细节信息。
  2. 如权利要求1所述的图像处理方法，其特征在于，所述基于至少两个掩膜对所述第三图像与所述第四图像进行融合处理，得到融合图像，包括：
    根据所述至少两个掩膜的交集得到目标掩膜；
    根据所述目标掩膜对所述第三图像与所述第四图像进行融合处理，得到所述融合图像。
  3. 如权利要求1或2所述的图像处理方法，其特征在于，所述对所述第一图像进行第一图像处理，得到第三图像，包括：
    将所述第一图像转换至所述第二颜色空间，得到第五图像；
    以所述第四图像为基准对所述第五图像进行全局配准处理，得到第六图像；
    以所述第四图像为基准对所述第六图像进行局部配准处理，得到所述第三图像。
  4. 如权利要求3所述的图像处理方法，其特征在于，所述以所述第四图像为基准对所述第五图像进行全局配准处理，得到第六图像，包括：
    对所述第四图像进行深度估计，得到深度信息，所述深度信息用于表示所述第四图像中的拍摄对象与所述电子设备的距离；
    基于所述深度信息，筛选所述第四图像中满足预设条件的目标特征点；
    以所述目标特征点为基准对所述第五图像进行所述全局配准处理，得到所述第六图像。
  5. 如权利要求4所述的图像处理方法，其特征在于，所述预设条件为所述拍摄对象与所述电子设备的距离大于第一预设阈值。
  6. 如权利要求3至5中任一项所述的图像处理方法，其特征在于，所述以所述第四图像为基准对所述第六图像进行所述局部配准处理，得到所述第三图像，包括：
    以所述第四图像对应的亮度通道图像为基准，对所述第六图像进行局部配准处理，得到所述第三图像。
  7. 如权利要求3至5中任一项所述的图像处理方法，其特征在于，所述以所述第四图像为基准对所述第六图像进行局部配准处理，得到所述第三图像，包括：
    根据所述第四图像的高频信息与所述第六图像的高频信息的并集，得到第七图像；
    基于所述第七图像确定所述第四图像中的第一图像区域；
    基于所述第七图像确定所述第六图像中的第二图像区域；
    以所述第一图像区域为基准对所述第二图像区域进行所述局部配准处理，得到所述第三图像。
  8. 如权利要求1至7中任一项所述的图像处理方法，其特征在于，所述至少两个掩膜为所述第一掩膜与所述第三掩膜。
  9. 如权利要求1至7中任一项所述的图像处理方法，其特征在于，所述至少两个掩膜为所述第一掩膜、所述第二掩膜与所述第三掩膜。
  10. 如权利要求1至9中任一项所述的图像处理方法，其特征在于，所述电子设备还包括红外闪光灯，所述图像处理方法还包括：
    在暗光场景下，开启所述红外闪光灯，所述暗光场景是指所述电子设备所处的拍摄环境的环境亮度小于第二预设阈值；
    所述响应于所述第一操作，获取第一图像与第二图像，包括：
    在开启所述红外闪光灯的情况下，获取所述第一图像与所述第二图像。
  11. 如权利要求10所述的图像处理方法，其特征在于，所述第一界面包括第二控件；所述在暗光场景下，开启所述红外闪光灯，包括：
    检测到对所述第二控件的第二操作；
    响应于所述第二操作开启所述红外闪光灯。
  12. 如权利要求1至11中任一项所述的图像处理方法，其特征在于，所述第一界面是指拍照界面，所述第一控件是指用于指示拍照的控件。
  13. 如权利要求1至11中任一项所述的图像处理方法，其特征在于，所述第一界面是指视频录制界面，所述第一控件是指用于指示录制视频的控件。
  14. 如权利要求1至11中任一项所述的图像处理方法，其特征在于，所述第一界面是指视频通话界面，所述第一控件是指用于指示视频通话的控件。
  15. 一种电子设备，其特征在于，包括：
    一个或多个处理器和存储器；
    所述存储器与所述一个或多个处理器耦合，所述存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行如权利要求1至14中任一项所述的图像处理方法。
  16. 一种芯片***，其特征在于，所述芯片***应用于电子设备，所述芯片***包括一个或多个处理器，所述处理器用于调用计算机指令以使得所述电子设备执行如权利要求1至14中任一项所述的图像处理方法。
  17. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储了计算机程序，当所述计算机程序被处理器执行时，使得处理器执行权利要求1至14中任 一项所述的图像处理方法。
  18. 一种计算机程序产品，其特征在于，所述计算机程序产品包括计算机程序代码，当所述计算机程序代码被处理器执行时，使得处理器执行权利要求1至14中任一项所述的图像处理方法。
PCT/CN2022/117324 2021-12-29 2022-09-06 图像处理方法与电子设备 WO2023124201A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP22857103.0A EP4231621A4 (en) 2021-12-29 2022-09-06 IMAGE PROCESSING METHOD AND ELECTRONIC DEVICE
US18/024,365 US20240185389A1 (en) 2021-12-29 2022-09-06 Image processing method and electronic device

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202111638949 2021-12-29
CN202111638949.0 2021-12-29
CN202210108237.6A CN116437198B (zh) 2021-12-29 2022-01-28 图像处理方法与电子设备
CN202210108237.6 2022-01-28

Publications (1)

Publication Number Publication Date
WO2023124201A1 true WO2023124201A1 (zh) 2023-07-06

Family

ID=85800400

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/117324 WO2023124201A1 (zh) 2021-12-29 2022-09-06 图像处理方法与电子设备

Country Status (3)

Country Link
US (1) US20240185389A1 (zh)
EP (1) EP4231621A4 (zh)
WO (1) WO2023124201A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117729445A (zh) * 2024-02-07 2024-03-19 荣耀终端有限公司 一种图像处理方法、电子设备以及计算机可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110122251A1 (en) * 2009-11-20 2011-05-26 Fluke Corporation Comparison of Infrared Images
CN110971841A (zh) * 2019-12-09 2020-04-07 Oppo广东移动通信有限公司 图像处理方法、装置、存储介质及电子设备
CN111970432A (zh) * 2019-05-20 2020-11-20 华为技术有限公司 一种图像处理方法及图像处理装置
CN113379615A (zh) * 2021-04-25 2021-09-10 北京迈格威科技有限公司 图像处理方法及装置、存储介质和电子设备

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7435962B2 (en) * 2005-05-18 2008-10-14 Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. Imaging device and method for producing an infrared filtered digital image
US10382674B2 (en) * 2013-04-15 2019-08-13 Qualcomm Incorporated Reference image selection for motion ghost filtering
CN108605099B (zh) * 2016-10-17 2020-10-09 华为技术有限公司 用于终端拍照的方法及终端
US11496694B2 (en) * 2020-09-04 2022-11-08 Altek Semiconductor Corp. Dual sensor imaging system and imaging method thereof
WO2021184029A1 (en) * 2020-11-12 2021-09-16 Innopeak Technology, Inc. Systems and methods for fusing color image and near-infrared image

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110122251A1 (en) * 2009-11-20 2011-05-26 Fluke Corporation Comparison of Infrared Images
CN111970432A (zh) * 2019-05-20 2020-11-20 华为技术有限公司 一种图像处理方法及图像处理装置
CN110971841A (zh) * 2019-12-09 2020-04-07 Oppo广东移动通信有限公司 图像处理方法、装置、存储介质及电子设备
CN113379615A (zh) * 2021-04-25 2021-09-10 北京迈格威科技有限公司 图像处理方法及装置、存储介质和电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4231621A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117729445A (zh) * 2024-02-07 2024-03-19 荣耀终端有限公司 一种图像处理方法、电子设备以及计算机可读存储介质

Also Published As

Publication number Publication date
US20240185389A1 (en) 2024-06-06
EP4231621A1 (en) 2023-08-23
EP4231621A4 (en) 2024-03-06

Similar Documents

Publication Publication Date Title
WO2023130922A1 (zh) 图像处理方法与电子设备
US20240119566A1 (en) Image processing method and apparatus, and electronic device
CN116416122B (zh) 图像处理方法及其相关设备
WO2023040725A1 (zh) 白平衡处理方法与电子设备
WO2023124201A1 (zh) 图像处理方法与电子设备
CN116437198B (zh) 图像处理方法与电子设备
CN115767290B (zh) 图像处理方法和电子设备
WO2023060921A1 (zh) 图像处理方法与电子设备
CN116668862B (zh) 图像处理方法与电子设备
WO2023124202A1 (zh) 图像处理方法与电子设备
CN117135293A (zh) 图像处理方法和电子设备
WO2023015985A1 (zh) 图像处理方法与电子设备
US20230058472A1 (en) Sensor prioritization for composite image capture
CN116258633A (zh) 图像去反光的方法、图像去反光模型的训练方法与装置
CN109447925B (zh) 图像处理方法和装置、存储介质、电子设备
CN116668838B (zh) 图像处理方法与电子设备
CN115526786B (zh) 图像处理方法及其相关设备
JP2020191546A (ja) 画像処理装置、画像処理方法、およびプログラム
CN115426458B (zh) 光源检测方法及其相关设备
CN115955611B (zh) 图像处理方法与电子设备
WO2023160221A1 (zh) 一种图像处理方法和电子设备
WO2023160220A1 (zh) 一种图像处理方法和电子设备
CN116029951B (zh) 图像处理方法与电子设备
CN116029914B (zh) 图像处理方法与电子设备
CN116051368B (zh) 图像处理方法及其相关设备

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2022857103

Country of ref document: EP

Effective date: 20230224

WWE Wipo information: entry into national phase

Ref document number: 18024365

Country of ref document: US