CN115908221A

CN115908221A - Image processing method, electronic device, and storage medium

Info

Publication number: CN115908221A
Application number: CN202310217839.XA
Authority: CN
Inventors: 钱彦霖; 邹卓良
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2023-03-08
Filing date: 2023-03-08
Publication date: 2023-04-04
Anticipated expiration: 2043-03-08
Also published as: CN115908221B

Abstract

The embodiment of the application provides an image processing method, electronic equipment and a storage medium. In the method, electronic equipment acquires a first RGB image and a Near Infrared (NIR) image which are acquired in the same scene; respectively calculating pixel gradients on each color channel of the first RGB image to obtain a first gradient corresponding to the first RGB image; calculating the gradient of the NIR image to obtain a second gradient corresponding to the NIR image; fusing the first gradient and the second gradient to obtain a target gradient matched with the structure tensor of the second gradient; and the electronic equipment determines the approximate pixel value of each pixel point in the second RGB image according to the target gradient so as to generate the second RGB image. Therefore, the electronic equipment realizes the fusion of the RGB image and the NIR image based on the image gradient fusion and the RGB image reconstruction, not only can the fusion effect be ensured, but also the time consumption for the RGB image reconstruction is shortened, so that the application of the RGB image in intelligent terminals such as mobile phones becomes possible.

Description

Image processing method, electronic device, and storage medium

Technical Field

The present application relates to the field of intelligent terminal technologies, and in particular, to an image processing method, an electronic device, and a storage medium.

Background

Near-infrared light is an electromagnetic wave between visible and mid-infrared light and is imperceptible to the human eye. Near infrared light images are imaged on a near infrared light band imperceptible to the human eye, which has more detailed information than RGB images imaged on a visible light band. Therefore, the RGB image obtained by fusing the RGB image and the near-infrared image in the same scene not only can keep the color stability of the image, but also can have the detail information of the near-infrared image.

In view of the limitations of the fusion method of the RGB image and the near-infrared image, such as too long time consumption and high calculation requirement, the current scheme that the RGB image and the near-infrared image are fused to improve the picture quality cannot be applied to an intelligent terminal (such as a mobile phone).

Disclosure of Invention

In order to solve the above technical problem, embodiments of the present application provide an image processing method, an electronic device, and a storage medium. In the method, when the electronic equipment performs gradient fusion on the RGB image and the NIR image, image reconstruction is performed according to the image subjected to gradient fusion to obtain the fused RGB image, so that not only can the fusion effect be ensured, but also the time consumption for reconstructing the RGB image is shortened, and the method can be applied to intelligent terminals such as mobile phones.

In a first aspect, an embodiment of the present application provides an image processing method. The method is applied to the electronic equipment and comprises the following steps: the method comprises the steps that electronic equipment obtains a first RGB image and a Near Infrared (NIR) image which are collected in the same scene; the electronic equipment respectively calculates pixel gradients on each color channel of the first RGB image to obtain a first gradient corresponding to the first RGB image; the electronic equipment calculates the gradient of the NIR image to obtain a second gradient corresponding to the NIR image; the electronic equipment fuses the first gradient and the second gradient to obtain a target gradient; wherein the target gradient matches the structure tensor of the second gradient; and the electronic equipment determines the approximate pixel value of each pixel point in the second RGB image according to the target gradient so as to generate a second RGB image.

Therefore, the electronic equipment acquires an RGB image and an NIR image corresponding to the same scene, performs gradient calculation on the RGB image and the NIR image respectively to obtain a gradient of the RGB image and a gradient of the NIR image, fuses the gradient of the RGB image and the gradient of the NIR image to obtain a fused gradient, and performs RGB image reconstruction based on the fused gradient to obtain a fused RGB image.

When the electronic equipment performs gradient fusion, the fusion target is that the structure tensor of the gradient of the second RGB image is matched with the structure tensor of the gradient of the NIR image, so that the gradient of the second RGB image is closer to the gradient of the NIR image relative to the gradient of the RGB image, and the detail information of the NIR image is fully fused. Moreover, when the electronic device reconstructs the RGB image based on the fused gradient, an approximate solution of the pixel value is obtained to replace an accurate solution, so that the time of the poisson fusion process is shortened, and the fusion scheme of the RGB image and the NIR image can be applied to intelligent terminals such as mobile phones.

According to a first aspect, the method further comprises: the target gradient is linear with the first gradient.

In this way, the gradient after the fusion and the gradient of the RGB image are in a linear relation, so that the overlarge color deviation between the RGB image obtained after the image fusion and the original RGB image can be avoided.

According to the first aspect or any one implementation manner of the first aspect, the determining, by the electronic device, an approximate pixel value of each pixel point in the second RGB image according to the target gradient includes: the electronic equipment constructs a Poisson equation according to the target gradient, and the unknown quantity of the Poisson equation is the pixel value of each pixel point in the second RGB image; the electronic device solves an approximate solution of the Poisson equation based on discrete cosine transform, and the approximate solution is used as an approximate pixel value of each pixel point in the second RGB image.

Therefore, the RGB image reconstruction of the gradient domain is carried out by using the idea of solving the Poisson equation through discrete cosine transform, the reconstruction efficiency of the RGB image can be obviously improved, and the method for improving the RGB image quality by fusing the RGB image and the NIR image can be applied to intelligent terminals such as mobile phones.

According to the first aspect or any one implementation manner of the first aspect, the determining, by the electronic device, an approximate pixel value of each pixel point in the second RGB image according to the target gradient includes: the electronic equipment constructs a first Poisson equation according to the R channel component in the target gradient, and the unknown quantity of the first Poisson equation is the R channel pixel value of each pixel point in the second RGB image; the electronic equipment constructs a second Poisson equation according to the G channel component in the target gradient, and the unknown quantity of the second Poisson equation is the G channel pixel value of each pixel point in the second RGB image; the electronic equipment constructs a third Poisson equation according to the B channel component in the target gradient, and the unknown quantity of the third Poisson equation is the B channel pixel value of each pixel point in the second RGB image; the electronic equipment solves an approximate solution of the first Poisson equation based on discrete cosine transform, and the approximate solution is used as an R channel approximate pixel value of each pixel point in the second RGB image; the electronic equipment solves an approximate solution of a second Poisson equation based on discrete cosine transform, and the approximate solution is used as a G channel approximate pixel value of each pixel point in the second RGB image; and the electronic equipment solves the approximate solution of the third Poisson equation based on discrete cosine transform, and the approximate solution is used as the B-channel approximate pixel value of each pixel point in the second RGB image.

According to a first aspect or any one of the above implementation manners of the first aspect, an electronic device includes a visible light camera module and a near infrared light camera module. Before the electronic device acquires the first RGB image and the NIR image acquired under the same scene, the method further includes: the electronic equipment responds to the first operation, and acquires RGB image data through the visible light camera module and NIR image data through the near infrared light camera module; the electronic equipment processes the RGB image data to generate a first RGB image, and processes the NIR image data to obtain an NIR image; after generating the second RGB image, the method further comprises: the electronic device displays and saves the second RGB image.

According to the first aspect, or any one of the above implementation manners of the first aspect, the method further includes:

and responding to the second operation, and starting the near-infrared camera module.

According to the first aspect, or any one of the foregoing implementation manners of the first aspect, the first RGB image is fused based on N RGB image frames, and the NIR image is fused based on N NIR image frames; n is an integer greater than 1.

According to the first aspect, or any implementation manner of the first aspect, the calculating, by the electronic device, a pixel gradient on each channel of the first RGB image to obtain a first gradient corresponding to the first RGB image includes:

on each color channel of the first RGB image, aiming at each first target pixel point, the electronic equipment calculates a gradient value corresponding to the first target pixel point according to pixel values of four pixel points which are adjacent to the first target pixel point up, down, left and right so as to obtain a first gradient corresponding to the first RGB image;

the electronic device calculates a gradient of the NIR image, resulting in a second gradient corresponding to the NIR image, comprising:

and aiming at each second target pixel point of the NIR image, the electronic equipment calculates the gradient value corresponding to the second target pixel point according to the pixel values of four pixel points which are adjacent to the second target pixel point in the vertical, horizontal and left directions so as to obtain a second gradient corresponding to the NIR image.

According to a first aspect, or any one of the above implementation manners of the first aspect, the electronic device is a mobile phone.

In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored on the memory, and when executed by the one or more processors, cause the electronic device to perform the image processing method of any one of the first aspect and the first aspect.

Any one implementation manner of the second aspect and the second aspect corresponds to any one implementation manner of the first aspect and the first aspect, respectively. For technical effects corresponding to any one implementation manner of the second aspect and the second aspect, reference may be made to the technical effects corresponding to any one implementation manner of the first aspect and the first aspect, and details are not repeated here.

In a third aspect, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the image processing method of any one of the first aspect and the first aspect.

Any one implementation manner of the third aspect corresponds to any one implementation manner of the first aspect. For technical effects corresponding to any one implementation manner of the third aspect and the third aspect, reference may be made to the technical effects corresponding to any one implementation manner of the first aspect and the first aspect, and details are not repeated here.

In a fourth aspect, the present application provides a computer program product, which includes a computer program and when the computer program is executed, causes a computer to execute the image processing method according to the first aspect or any one of the first aspects.

Any one implementation manner of the fourth aspect and the fourth aspect corresponds to any one implementation manner of the first aspect and the first aspect, respectively. For technical effects corresponding to any one implementation manner of the fourth aspect and the fourth aspect, reference may be made to the technical effects corresponding to any one implementation manner of the first aspect and the first aspect, and details are not described here again.

In a fifth aspect, the present application provides a chip comprising a processing circuit, a transceiver pin. Wherein the transceiving pin and the processing circuit communicate with each other through an internal connection path, the processing circuit performing the image processing method as in any one of the first aspect or the first aspect to control the receiving pin to receive a signal and to control the transmitting pin to transmit a signal.

Any one implementation manner of the fifth aspect and the fifth aspect corresponds to any one implementation manner of the first aspect and the first aspect, respectively. For technical effects corresponding to any one of the implementation manners of the fifth aspect and the fifth aspect, reference may be made to the technical effects corresponding to any one of the implementation manners of the first aspect and the first aspect, and details are not repeated here.

Drawings

Fig. 1 is a schematic diagram of a hardware configuration of an exemplary electronic device;

fig. 2 is a schematic diagram of a software structure of an exemplary electronic device;

FIG. 3 is a schematic diagram of an exemplary application scenario;

FIG. 4 is an exemplary module interaction diagram;

fig. 5 is an exemplary illustration of a fusion effect of an RGB image and a near-infrared light image;

FIG. 6 is a schematic diagram illustrating an exemplary fusion process of an RGB image and a near-infrared image;

FIG. 7 is an exemplary illustration of gradient fusion of an RGB image and a near-infrared image;

fig. 8 is an exemplary diagram illustrating a reconstruction result of an RGB image.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone.

The terms "first" and "second," and the like, in the description and in the claims of the embodiments of the present application are used for distinguishing between different objects and not for describing a particular order of the objects. For example, the first target object and the second target object, etc. are specific sequences for distinguishing different target objects, rather than describing target objects.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion.

In the description of the embodiments of the present application, the meaning of "a plurality" means two or more unless otherwise specified. For example, a plurality of processing units refers to two or more processing units; the plurality of systems refers to two or more systems.

Near Infrared (NIR) refers to electromagnetic waves between visible and mid-infrared light; the near-infrared region can be divided into two regions of near-infrared short wave (wavelength of 780 nm-1100 nm) and near-infrared long wave (wavelength of 1100 nm-2526 nm). An image imaged on a near infrared light band is referred to as a near infrared light image (NIR image). The near-infrared light can well penetrate through haze and has good brightness response to vegetation and cloud layers, so that the NIR image can contain more detailed information than an RGB image imaged on a visible light waveband. The detail information is edge information, texture information, and the like of the shooting object, such as hair edges, clothes folds, edges of each tree in a large number of trees, branches and leaves of green plants, and the like.

In the embodiment of the application, the RGB image and the NIR image in the same scene are fused, so that multispectral information fusion of the NIR image and the RGB image can be realized, the RGB image obtained after fusion can include more detailed information, and the detailed information in the image can be enhanced. Therefore, the RGB image obtained by fusing the RGB image and the NIR image not only can keep the color stability of the RGB image, but also can have the detail information of the NIR image.

The following describes an image processing method provided in an embodiment of the present application with reference to the drawings.

Fig. 1 is a schematic structural diagram of an electronic device 100. Alternatively, the electronic device 100 may be a terminal, and may also be referred to as a terminal device, a smart terminal, or the like.

The electronic device 100 may be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, an in-vehicle electronic device, an Augmented Reality (AR) device, a Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), a projector, and the like, and the embodiment of the present application does not limit the specific type of the electronic device 100.

It should be understood that the electronic device 100 shown in fig. 1 is only one example of an electronic device, and that the electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. The various components shown in fig. 1 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

The electronic device 100 may include: the mobile terminal includes a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a Subscriber Identity Module (SIM) card interface 195, and the like. Wherein the sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

The controller may be, among other things, a neural center and a command center of the electronic device 100. The controller can generate an operation control signal according to the instruction operation code and the time sequence signal to finish the control of instruction fetching and instruction execution.

A memory may also be provided in processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transmit data between the electronic device 100 and a peripheral device. And the earphone can also be used for connecting an earphone and playing audio through the earphone. The interface may also be used to connect other electronic devices, such as AR devices and the like.

The charging management module 140 is configured to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100. The charging management module 140 may also supply power to the electronic device through the power management module 141 while charging the battery 142.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The

antennas

1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device 100 may be used to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution including 2G/3G/4G/5G wireless communication applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a Low Noise Amplifier (LNA), and the like.

The wireless communication module 160 may provide a solution for wireless communication applied to the electronic device 100, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (bluetooth, BT), global Navigation Satellite System (GNSS), frequency Modulation (FM), near Field Communication (NFC), infrared (IR), and the like.

In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 and antenna 2 is coupled to wireless communication module 160 so that electronic device 100 can communicate with networks and other devices through wireless communication techniques.

The electronic device 100 implements display functions via the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, video, and the like. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, N being a positive integer greater than 1.

The electronic device 100 may implement a shooting function through the ISP, the camera 193, the video codec, the GPU, the display 194, the application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when a photo is taken, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converting into an image visible to naked eyes. The ISP can also carry out algorithm optimization on the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV and other formats. In some embodiments, electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

In the embodiment of the present application, the camera module of the camera includes a visible light camera module and a near infrared light camera module. The spectral range which can be obtained by the visible light camera module comprises visible light (the wavelength ranges from 400nm to 700nm), and the spectral range which can be obtained by the near infrared light camera module is near infrared light (the wavelength ranges from 700nm to 1100 nm). Therefore, for the same scene, the electronic equipment can acquire visible light images through the visible light camera module, and the near infrared light camera module can acquire near infrared light images.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to extend the memory capability of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

The internal memory 121 may be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121, for example, so that the electronic device 100 implements the image processing method in the embodiment of the present application. The internal memory 121 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The storage data area may store data (such as audio data, phone book, etc.) created during use of the electronic device 100, and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like.

The electronic device 100 may implement audio functions via the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playing, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be disposed in the processor 110, or some functional modules of the audio module 170 may be disposed in the processor 110.

The speaker 170A, also called a "horn", is used to convert the audio electrical signal into a sound signal. The electronic apparatus 100 can listen to music through the speaker 170A or listen to a hands-free call. In some embodiments, the electronic device 100 may be provided with a plurality of speakers 170A.

The receiver 170B, also called "earpiece", is used to convert the electrical audio signal into a sound signal. When the electronic apparatus 100 receives a call or voice information, it can receive voice by placing the receiver 170B close to the ear of the person.

The microphone 170C, also referred to as a "microphone," is used to convert sound signals into electrical signals. When making a call or transmitting voice information, the user can input a voice signal to the microphone 170C by speaking near the microphone 170C through the mouth. The electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C to achieve a noise reduction function in addition to collecting sound signals. In other embodiments, the electronic device 100 may further include three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, perform directional recording, and so on.

The earphone interface 170D is used to connect a wired earphone. The headset interface 170D may be the USB interface 130, or may be a 3.5mm open mobile electronic device platform (OMTP) standard interface, a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The pressure sensor is used for sensing a pressure signal and converting the pressure signal into an electric signal. In some embodiments, the pressure sensor may be disposed on the display screen 194. The electronic apparatus 100 may also calculate the touched position based on the detection signal of the pressure sensor.

The gyro sensor may be used to determine the motion pose of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) may be determined by a gyroscope sensor. The gyroscope sensor can also be used for shooting anti-shake. For example, when the shutter is pressed, the gyro sensor detects the shake angle of the electronic device 100, calculates the distance to be compensated for by the lens module according to the shake angle, and enables the lens to counteract the shake of the electronic device 100 through reverse movement, thereby achieving anti-shake.

The acceleration sensor may detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes). The acceleration sensor can detect the magnitude and direction of gravity when the electronic device 100 is stationary. The acceleration sensor can also be used for recognizing the posture of the electronic equipment, and is applied to horizontal and vertical screen switching, pedometers and other applications.

Touch sensors, also known as "touch panels". The touch sensor may be disposed on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor is used to detect a touch operation applied thereto or nearby. The touch sensor can communicate the detected touch operation to the application processor to determine the touch event type.

The buttons 190 include a power-on key (or power key), a volume key, and the like. The keys 190 may be mechanical keys. Or may be touch keys. The electronic apparatus 100 may receive a key input, and generate a key signal input related to user setting and function control of the electronic apparatus 100.

The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration cues, as well as for touch vibration feedback. For example, touch operations applied to different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects.

Indicator 192 may be an indicator light that may be used to indicate a state of charge, a change in charge, or a message, missed call, notification, etc.

The software system of the electronic device 100 may employ a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes an Android system with a layered architecture as an example, and exemplarily illustrates a software structure of the electronic device 100.

Fig. 2 is a block diagram of a software structure of the electronic device 100 according to the embodiment of the present application.

The layered architecture of the electronic device 100 divides the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, an application layer, an application framework layer, a Hardware Abstraction Layer (HAL), and a kernel layer from top to bottom.

The application layer may include a series of application packages.

As shown in fig. 2, the application packages may include cameras, galleries, third party applications with camera functionality, and the like. The application packages may also include applications such as telephony, calendar, maps, navigation, music, video, short messages, etc.

The application framework layer provides an Application Programming Interface (API) and a programming framework for the application program of the application layer. The application framework layer includes a number of predefined functions.

As shown in FIG. 2, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, a Camera Service (Camera Service), and the like.

The window manager is used for managing window programs. The window manager can obtain the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

Content providers are used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, calls made and answered, browsing history and bookmarks, phone books, etc.

The view system includes visual controls such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like.

The notification manager enables the application to display notification information in the status bar, can be used to convey notification-type messages, can disappear automatically after a brief dwell, and does not require user interaction. Such as notification messages used to inform download completion, message alerts, etc. The notification information may also be in the form of a chart or scroll bar text that appears in the top status bar of the system, such as a notification of a background running application, or in the form of a dialog window that appears on the screen. The notification information includes, for example, a text message presented in the status bar, a sound presented, vibration of the electronic device, and flashing of an indicator light.

The camera service may provide an interface to access the camera and may also provide an interface to manage the camera. For example, the camera service may invoke a camera (including a front camera and/or a rear camera) to capture image frames in response to a request by an application. In the embodiment of the application, the camera service can call the visible light camera module to acquire visible light images and can also call the near infrared light camera module to acquire near infrared light images.

The HAL is an interface layer between the operating system kernel and the hardware circuitry to abstract the hardware. The HAL layer may include a camera HAL and other hardware device abstraction layers. Wherein the camera HAL may invoke algorithms in a camera algorithm library. For example, software algorithms for image processing may be included in the camera algorithm library.

In the embodiment of the present application, the camera HAL may invoke an image fusion algorithm in the camera algorithm library to fuse the RGB image and the NIR image in the same scene. Illustratively, the image fusion algorithm may include a gradient calculation module, a gradient fusion module, and an image reconstruction module. The gradient calculation module is used for calculating gradients of the RGB image and the NIR image respectively, the gradient fusion module is used for fusing the gradients of the RGB image and the gradients of the NIR image, and the image reconstruction module is used for reconstructing the RGB image according to the fused image gradients.

The kernel layer is a layer between hardware and software. The kernel layer at least comprises a display driver, a camera driver, an image signal processing module and the like. The image signal processing module is used for processing the image preview request and the photographing request according to the instruction of the camera HAL module so as to enable the camera driving module to acquire images according to the corresponding request. The image signal processing module can also be used for processing the raw image data acquired by the camera, such as image data denoising, image data optimization, and the like.

It is to be understood that the layers in the software structure shown in fig. 2 and the components included in each layer do not constitute a specific limitation of the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer layers than those shown, and may include more or fewer components in each layer, which is not limited in this application.

It is understood that, in order to implement the image processing method in the embodiment of the present application, the electronic device includes hardware and/or software modules for performing respective functions. The present application is capable of being implemented in hardware or a combination of hardware and computer software in conjunction with the exemplary algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed in hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, with the embodiment described in connection with the particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In some shooting scenes, for example, shooting scenes with poor lighting conditions (such as night scenes or dense fog scenes), due to poor light conditions of the shooting scenes, the light incoming amount of the electronic device is small, so that the problem that the image detail information of the visible light image collected by the visible light camera module is lost is caused.

In the embodiment of the application, for the same shooting scene, when the electronic equipment collects the visible light image through the visible light camera module, the electronic equipment can also collect the near infrared light image through the near infrared light camera module, so that the electronic equipment can fuse the visible light image and the near infrared light image and display the fused visible light image, the image quality of the visible light image is improved, and more image detail information is reserved.

Fig. 3 shows an exemplary application scenario. In an example, a user can turn on an infrared flash of an electronic device (e.g., a mobile phone) in a shooting scene with poor lighting conditions, so as to simultaneously acquire images through a visible light camera module and a near-infrared light camera module of the mobile phone. Fig. 3 (1) shows a display interface of the camera application in the photographing mode, which may include a photographing interface 310; the shooting interface 310 may include a view box 311 and a control; for example, the shooting interface 310 may include a control 312 for instructing shooting and a control 313 for instructing turning on of an infrared flash; in the preview state, a preview image can be displayed in real time in the viewing frame 311; in the preview state, before the user turns on the camera and does not press the photo/video button, the preview image may be displayed in the view finder in real time.

After the electronic device detects that a user clicks the control 313 for instructing to turn on the infrared flash, turning on a near-infrared camera module in the electronic device, and displaying a shooting interface shown in (2) in fig. 3; under the condition that the near-infrared light camera module is turned on, after the electronic device detects that a user clicks the control 312 for instructing shooting, the electronic device respectively acquires an RGB image and an NIR image corresponding to the same scene through the visible light camera module and the near-infrared light camera module, and processes the acquired visible light image and the near-infrared light image through the image processing method provided by the embodiment of the application to generate an RGB image with enhanced image quality.

Fig. 4 is a schematic diagram illustrating module interaction. As shown in fig. 4, the image processing method provided in the embodiment of the present application may specifically include the following steps:

s401, in response to an operation of the user clicking the shutter control, the camera application sends an image capturing request to the camera service.

The camera application is in different shooting modes, such as a high-brightness forward light mode, a high-brightness backward light mode, a large aperture mode, a portrait mode, a night view mode, and the like, the number of the shooting image frames required by the multi-frame synthesis algorithm corresponding to the shooting modes to change the exposure parameters is not necessarily the same, and further, the number of the image shooting requests issued by the camera application is not necessarily the same. The present embodiment does not limit the number of image capturing requests, and the following explanation takes one image capturing request as an example.

Illustratively, the multi-frame composition algorithm requires 2 captured image frames, and the camera application issues two image capture requests in sequence. Also exemplarily, the multi-frame composition algorithm requires 5 image preview frames, and the camera application issues four image capturing requests in sequence.

The exposure parameters corresponding to different image capturing requests are not necessarily the same, and this embodiment is not limited.

In this embodiment, the mode setting state of the camera application at least includes that the near infrared light mode is on.

S402, the camera service sends an image capture request to the camera HAL module.

The camera service receives the image capturing request, performs relevant processing corresponding to the image capturing request, such as creating a corresponding service instance or the like, and sends the image capturing request to the camera HAL module. For the related processing of the camera service, reference may be made to the prior art, and details thereof are not repeated herein.

And S403, the camera HAL module receives the image shooting request, determines that the near infrared light mode is started, generates a visible light frame request and a near infrared light frame request according to the image shooting request, and sends the visible light frame request and the near infrared light frame request to the ISP module.

S404, the ISP module sends the visible light frame request and the near infrared light frame request to the camera driving module.

S405, the camera driving module drives the visible light camera module to acquire visible light image data according to the visible light frame request.

Here, the visible light image data may also be referred to as RGB image data.

S406, the camera driving module drives the near infrared camera module to acquire near infrared image data according to the near infrared light frame request.

The present embodiment does not limit the execution sequence of S405 and S406.

S407, the visible light camera module collects the visible light image data and sends the visible light image data to the camera HAL through the camera driver and the ISP mode.

S408, the near infrared light camera module collects the near infrared light image data and sends the near infrared light image data to the camera HAL through the camera drive and ISP mode.

The present embodiment does not limit the execution sequence of S407 and S408.

S409, the camera HAL generates a visible light image frame from the visible light image data.

S410, the camera HAL generates a near-infrared light image frame from the near-infrared light image data.

S411, the camera HAL fuses the visible light image and the near infrared light image to obtain a target visible light image.

Alternatively, the visible light image may be generated from a visible light image frame and, correspondingly, the near infrared light image may be generated from a near infrared light image frame.

Optionally, if the image synthesis algorithm corresponding to the current shooting mode needs N shooting image frames, the visible light image is formed by fusing the N visible light image frames, and correspondingly, the near-infrared light image is also formed by fusing the N near-infrared light image frames. For the fusion algorithm of the images of the same type, reference may be made to the prior art, and details thereof are not described herein.

In this embodiment, after the camera HAL obtains the visible light image (i.e., RGB image) and the near infrared light image, the RGB image and the NIR image are first aligned, and then the target RGB image is obtained by performing fusion processing based on the aligned RGB image and NIR image. For the method of image alignment, reference may be made to the prior art, and details are not repeated here.

In the example shown in fig. 5, for the same shooting scene, an RGB image imaged on a visible light band is shown as an image 501, an NIR image imaged on a near infrared light band is shown as an image 502, and an RGB image obtained by fusing the RGB image and the NIR image is shown as an image 503. The

images

501, 502, 503 are shown in the form of gray-scale images. Comparing image 501 with image 502 shows that image 502 has more image detail information than image 501, such as the edge of each of a large number of trees on a mountain. The image 503 obtained by fusing the image 501 and the image 502 can not only keep the color stability of the RGB image, but also have the detail information of the NIR image, thereby significantly improving the image quality.

Fig. 6 shows a schematic flow chart of the image fusion performed by the camera HAL for the RGB image and the NIR image corresponding to the same scene. As shown in fig. 6, in the image processing method provided in the embodiment of the present application: firstly, a camera HAL acquires an RGB image and an NIR image corresponding to the same scene, and the RGB image and the NIR image are respectively input to a gradient calculation module to obtain the gradient of the RGB image and the gradient of the NIR image; then, the camera HAL inputs the gradient of the RGB image and the gradient of the NIR image into a gradient fusion module to obtain a fused gradient; and finally, inputting the fused gradient into an image reconstruction module by the camera HAL to obtain a fused RGB image.

The gradient calculation module, the gradient fusion module and the image reconstruction module are respectively used for realizing different functions. As the name suggests, the gradient calculation module is used for calculating the gradient of the image, the gradient fusion module is used for fusing the gradient of the image, and the image reconstruction module is used for reconstructing the RGB image based on the image gradient. The gradient calculation module, the gradient fusion module, and the image reconstruction module may be separately arranged, or may be integrated together, for example, implemented together in an image processing module of the camera HAL, which is not limited in this embodiment.

It should be noted that the visible light camera module of the electronic device collects YUV images, and before the electronic device performs the image processing method provided by the embodiment of the present application, the YUV images need to be converted into RGB images. Wherein, the camera HAL can convert the format of the visible light image from YUV format to RGB format after obtaining the visible light image. Regarding the processing flow of the YUV image by the camera HAL, including but not limited to image enhancement processing (such as denoising, distortion correction, etc.), image cropping processing, image format conversion processing, etc., reference may be made to the prior art, and details are not repeated here.

The image gradient calculation, image gradient fusion, and image reconstruction processes will be explained below.

(I) image gradient calculation

Considering an image as a two-dimensional discrete function, the image gradient is the derivative of this two-dimensional function. In other words, it is the magnitude of the image change that is calculated for the image gradient.

The gradient calculation of the image can be realized by different gradient operators, and the specific realization process is to carry out convolution operation on the image by using the gradient operators. The neighborhood contrast contained in different gradient operators (e.g., roberts operator, prewitt operator, sobel operator, etc.) is different. The Roberts operator is commonly used for the edge detection task of the image with obvious vertical edges or steep low noise, the Prewitt operator is commonly used for the edge detection task of the image with more noise and gradually changed gray level, and the sobel operator is commonly used for the edge detection task of the image with more noise and gradually changed gray level.

Considering that the stronger the gradient in the NIR image, the closer the fused gradient is to the gradient of the NIR image, making RGB image color reconstruction based on the fused gradient more difficult. In order to optimize the comprehensive benefits of the reconstructed RGB image in terms of fusion effect and color preservation, in the embodiment of the present application, the gradient calculation module performs gradient calculation only according to four pixel points adjacent to the gradient calculation module in the vertical and horizontal directions when calculating the gradient of each pixel point of the image.

When the gradient calculation module performs image gradient calculation on the RGB image, the gradient calculation module performs image gradient calculation on each color channel of the RGB image. In this way, the gradient image of the RGB image can better retain the color information of the RGB image.

On the R channel, the image gradient GR (x, y) = dx1 (i, j) + dy1 (i, j) of the RGB image;

dx1(i,j) = R(i+1,j) - R(i,j);

dy1(i,j) = R(i,j+1) - R(i,j);

where R is the R value of the RGB values of a pixel on the RGB image, and (i, j) is the coordinates of the pixel.

On the G channel, an image gradient GG (x, y) = dx2 (i, j) + dy2 (i, j) of the RGB image;

dx2(i,j) = G(i+1,j) - G(i,j);

dy2(i,j) = G(i,j+1) - G(i,j);

where G is the G value of the RGB values of the pixel on the RGB image, and (i, j) is the coordinates of the pixel.

On the B channel, the image gradient GB (x, y) = dx3 (i, j) + dy3 (i, j) of the RGB image;

dx3(i,j) = B(i+1,j) - B(i,j);

dy3(i,j) = B(i,j+1) - B(i,j);

where B is a B value among RGB values of pixels on the RGB image, and (i, j) is a coordinate of the pixel.

Thus, the gradient of an RGB image may be expressed as

。

Similarly, the image gradient H (x, y) = dx (i, j) + dy (i, j) of the NIR image;

dx(i,j) = I(i+1,j) - I(i,j);

dy(i,j) = I(i,j+1) - I(i,j);

where I is the gray value of the pixel on the NIR image and (I, j) is the coordinate of the pixel.

(II) image gradient fusion

Assume that the gradient of the RGB image (i.e., the RGB image to be fused) is

Image gradient of NIR image is->

Fused gradients in->

。

For the three gradient fields mentioned above

、/>

And &>

The following three matrices are defined:

；

；

。

in the implementation of the application, since the significance of fusing the RGB image and the NIR image is to add the detail information in the NIR image to the RGB image, so that the fused RGB image is not only color-stable but also rich in detail information, when the gradient fusion module fuses the gradient of the RGB image and the gradient of the NIR image, the gradient weight of the NIR image is set to be greater, instead of the gradient of the RGB image being equal to the gradient weight of the NIR image, so that the gradient image obtained after fusion is closer to the gradient of the NIR image than the gradient of the RGB image.

Thus, the gradient fusion module fuses the gradients of the RGB images

And a gradient of the NIR image>

To get->

The goal may be to match the structure tensor of the RGB image gradients (structure tensor) obtained after the fusion with the structure tensor of the NIR image gradients. In the present embodiment, is->

。

Gradient to RGB image

And gradient of NIR image>

Singular Value Decomposition (SVD): />

；

。

In order to match the structure tensor of the RGB image and the NIR image obtained after the fusion, the matrix is exchanged

And matrix->

Give to>

Obtaining:

。

further, it is possible to prevent the occurrence of,

。

meanwhile, in order to avoid excessive color deviation between the RGB image obtained after image fusion and the original RGB image, the gradient after fusion

Should also be as close as possible to the gradient of the RGB image->

. The present exemplary embodiment thus defines @bymeans of a linear matrix a>

And/or>

The relationship of (a) is as follows: />

。

In this way,

。

due to the fact that

Therefore is based on>

。

In view of the matrix A and

、/>

the above relationship of (2) can be solvedThe matrix A is obtained as follows:

(ii) a Wherein it is present>

Represents a real symmetric half positive decision matrix>

、/>

The + represents the mole-penrose pseudoinverse.

Alternatively, the first and second electrodes may be,

，/>

，/>

is an identity matrix.

According to

Singular value decomposition of the deducing->

And its inverse or pseudo-inverse decomposition, as follows:

and then derive,>

。

thus, the gradient after fusion

The calculation formula of (a) is as follows:

。（1）

in the embodiment of the application, the gradient fusion module is based on the gradient of the RGB image

And gradient of NIR image

When a gradient fusion is carried out, the resulting gradient after fusion is->

The calculation can be made according to equation (1).

FIG. 7 schematically illustrates an example of gradient fusion. As shown in fig. 7, the gradient image 601 is a gradient map of an RGB image, the gradient image 602 is a gradient map of an NIR image, and the gradient image 601 and the gradient image 602 are fused to obtain a gradient image 603. Comparing the gradient image 601 and the gradient image 602, it can be seen that the gradient image 602 has more detail than the gradient image 601. According to the gradient fusion method provided by the embodiment of the present application, the structure tensor of the fused gradient image 603 matches the structure tensor of the gradient image 602, and is also close to the gradient image 601.

Therefore, the gradient fusion method provided by the embodiment of the application performs gradient fusion on the RGB image and the NIR image, not only can the detail information included in the NIR image be fused into the RGB image, but also the fused RGB image can be ensured not to generate larger color difference with the initial RGB image.

The color difference of the fused image is small, for example, blue is slightly changed into dark blue, the saturation of red is slightly reduced, and the like, and the fused image belongs to the normal acceptable range of the mobile phone photographing color difference. Meanwhile, the detail information included in the NIR image obviously enhances the details of the final RGB image, and the local area originally having few details can have characteristics of abundant textures, outlines and the like, so that the photographing experience is improved.

(III) image reconstruction

After the gradient of the RGB image and the gradient of the NIR image are fused to obtain a fused image gradient, the image reconstruction module can reconstruct the RGB image according to the fused image gradient. The image reconstruction module can realize the reconstruction processing of the RGB image based on the Poisson fusion process.

In the embodiment of the application, the image reconstruction module may calculate the divergence value of each pixel point according to the fused image gradient, and establish a multivariate one-time equation relationship between the pixel value of each pixel point in the reconstructed RGB image and the corresponding divergence value. Thus, the poisson fusion process may be described as solving the system of equations AX = b, a being the coefficient matrix, determined based on the laplace transform, X being the pixel vector (or pixel matrix), b being the divergence vector. And the image reconstruction module solves the equation set to obtain the pixel value of each pixel point in the reconstructed RGB image, and the reconstruction operation of the RGB image is completed.

To illustrate with a simple example, assume that there is a 4 × 4 image X:

，/>

representing the image pixel values at each location, there are 16 unknown parameters that need to be solved for. At this time, 16 equations (i.e., equation set AX = b) need to be constructed to solve the 16 unknown parameters.

Currently, the equation set AX = b can be solved by using a jacobian iteration method or a gaussian seidel iteration method. However, for images shot by smart terminals such as mobile phones, the dimension of X can reach tens of millions, and it is impractical to directly calculate, solve and apply the X to the smart terminals such as mobile phones, and the time cost is too high.

Therefore, in the embodiment of the application, when the image reconstruction module solves the equation set AX = b to determine the pixel value of each pixel point in the reconstructed RGB image, the accurate solution of the pixel value is not solved, but the approximate solution of the pixel value is solved instead of the accurate solution, so as to shorten the time of the poisson fusion process. When the image reconstruction module reconstructs an RGB image, the pixel value of each pixel point in the reconstructed RGB image is solved based on DCT (Discrete Cosine Transform).

In the embodiment of the present application, when the image reconstruction module reconstructs the target RGB image T, the image reconstruction module reconstructs the image of the target RGB image T on the R channel

Image on G channel>

And an image on the G channel->

To obtain the target RGB image T.

Wherein, the gradient of the target RGB image T is:

。

thus, R channel image

The gradient of (d) is: />

(ii) a G channel image>

The gradient of (a) is:

(ii) a B channel image->

The gradient of (d) is: />

。

The image reconstruction module reconstructs an R channel image

Then, the poisson equation on the R channel is constructed as follows:

(ii) a Wherein it is present>

For the Laplace operator, based on the sum of the values of the coefficients>

For a divergence vector of the target RGB image on the R channel, <' >>

。

Discretizing the Poisson equation on the R channel to obtain

. Wherein it is present>

Refers to an image->

A laplace convolution is performed.

Under the Noumann boundary condition (Neumann boundary), or under the second kind of boundary condition, the following solutions are solved:

r channel image

。

Wherein the content of the first and second substances,

for the currently calculated coordinates of the pixel, < > or>

For discrete cosine transform in two-dimensional space, is selected>

Is the inverse of the discrete cosine transform in two-dimensional space.

Wherein, the first and the second end of the pipe are connected with each other,

，/>

for the size of the target RGB image, K is a position-dependent quantity, which is calculated as: />

。

Thus, the image reconstruction module is based on the R-channel image based on the way to solve the solution of the discrete two-dimensional Poisson equation in the form of M × N grid under the Noiman boundary condition

Is solved to the R channel image->

I.e., solving for the R value in each pixel RGB value of the target RGB image.

Similarly, the image reconstruction module reconstructs G channel image

Then, the poisson equation on the G channel is constructed as follows:

(ii) a Wherein it is present>

Is Laplace operator, is->

For the divergence vector of the target RGB image on the G channel,

。

discretizing the Poisson equation on the G channel to obtain the P channel

. Wherein it is present>

Refers to an image->

A laplace convolution is performed.

Under the Noumann boundary condition (Neumann boundary), or under the second kind of boundary condition, the following solutions are obtained:

g-channel images

。

for the coordinates of the currently calculated pixel, in conjunction with a coordinate selection unit>

For discrete cosine transform in two-dimensional space, is selected>

Is the inverse of the discrete cosine transform in two-dimensional space.

Wherein the content of the first and second substances,

，/>

。

Thus, the image reconstruction module is based on the G-channel image based on the way to solve the solution of the discrete two-dimensional Poisson equation in the form of M × N grid under the Noiman boundary condition

Is solved to a G-channel image->

I.e., solving for the G value in each pixel RGB value of the target RGB image.

Similarly, the image reconstruction module reconstructs the B channel image

Then, the poisson equation on the B channel is constructed as follows: />

(ii) a Wherein it is present>

Is Laplace operator, is->

For the divergence vector of the target RGB image on the B channel,

。

discretizing the Poisson equation on the channel B to obtain

. Wherein +>

Refers to a picture->

A laplace convolution is performed.

b channel image

。

Wherein the content of the first and second substances,

for the currently calculated coordinates of the pixel, < > or>

For discrete cosine transform in two-dimensional space, is selected>

Is the inverse of the discrete cosine transform in two-dimensional space.

Wherein the content of the first and second substances,

，/>

。

Thus, the image reconstruction module is based on the B-channel image based on the way to solve the solution of the discrete two-dimensional Poisson equation in the form of M × N grid under the Noiman boundary condition

Is solved to the B-channel image->

I.e., solve to the B value in each pixel RGB value of the target RGB image.

Therefore, the image reconstruction module respectively reconstructs R channel images

And G channel image->

B channel image>

And then combining the RGB values of each pixel to obtain the target RGB image.

When the image reconstruction method provided by the embodiment is adopted, some ultrahigh frequency information is lost due to the fact that an accurate solution is not calculated, but most of high frequency information in the image, such as edge information and texture information of an object, is reserved, and the image quality of the target RGB image cannot be affected. Compared with the RGB image before fusion, the image quality of the target RGB image is obviously improved. Moreover, when the image reconstruction method provided by the embodiment is adopted, the accurate solution is replaced by a mode of solving the approximate solution of the pixel value, so that the time length of the image reconstruction process is greatly shortened, and the photographing experience of a user on the mobile phone and other electronic equipment is improved.

Fig. 8 schematically shows an image reconstruction example. In the RGB image reconstruction example shown in (1) in fig. 8, the gradients of the RGB image and the near-infrared light image are calculated based on the Roberts operator; in the image reconstruction example shown in (2) in fig. 8, the gradients of the RGB image and the near-infrared light image are calculated based on the Prewitt operator; in the image reconstruction example shown in (3) in fig. 8, the gradients of the RGB image and the near-infrared light image are obtained by the gradient calculation method adopted by the gradient calculation section of the image of the present embodiment; in the image reconstruction example shown in (4) in fig. 8, the gradients of the RGB image and the near-infrared light image are calculated based on the sobel operator.

Comparing the reconstructed RGB images shown in (1), (2), (3) and (4) in fig. 8, the RGB image shown in (3) in 8 not only retains the color information of the RGB image, but also has the detail information of the NIR image, and meanwhile, since the gradients of the RGB image and the near-infrared light image are obtained by the gradient calculation method adopted by the image gradient calculation portion in this embodiment, the difficulty level of the image reconstruction module in performing RGB image color reconstruction based on the fused gradient is not high, so that the reconstructed RGB image has the best comprehensive benefit in fusion effect and color preservation.

Therefore, in the embodiment of the application, the RGB image reconstruction of the gradient domain is performed by using the idea of solving the Poisson equation through discrete cosine transform, the reconstruction efficiency of the RGB image can be remarkably improved, and the method for improving the RGB image quality by fusing the RGB image and the NIR image can be applied to the intelligent terminal. In addition, the gradient image reconstructed by the RGB image not only retains the color information of the RGB image, but also has the detail information of the NIR image, so that the fusion degree of the RGB image and the NIR image is better and more stable, and the effect is better.

S412, the camera HAL module sends the target visible light image to the camera service.

S413, the camera HAL module generates a thumbnail of the target visible light image and sends the thumbnail of the target visible light image to the camera application via the camera service.

The present embodiment does not limit the execution order of S412 and S413.

S414, the camera service sends the target visible light image to the gallery application.

S415, the gallery application stores the target visible light image.

S417, the camera application refreshes the display of the thumbnail.

After generating the target RGB image resulting from fusing the RGB image and the NIR image, the electronic device may display the target RGB image on the screen and save the target RGB image in the gallery. When the near-infrared camera module of the electronic device is not turned on, the process of displaying and storing the target RGB image is similar to the process of displaying and storing the RGB image acquired by the electronic device only through the visible light camera module, and is not described herein again.

Other parts of the process that are not explained in detail can be referred to the prior art and are not described in detail herein. It should be noted that, in the above embodiments, the camera application in the mobile phone is taken as an example for explanation, and the same is true for other third party applications with camera functions, which are not described herein again.

The present embodiment also provides a computer storage medium, in which computer instructions are stored, and when the computer instructions are run on an electronic device, the computer instructions cause the electronic device to execute the above related method steps to implement the image processing method in the above embodiment.

The present embodiment also provides a computer program product, which when run on a computer causes the computer to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component or a module, and may include a processor and a memory connected to each other; the memory is used for storing computer execution instructions, and when the apparatus runs, the processor can execute the computer execution instructions stored by the memory, so that the chip executes the image processing method in the above method embodiments.

In addition, the electronic device (such as a mobile phone, etc.), the computer storage medium, the computer program product, or the chip provided in this embodiment are all configured to execute the corresponding method provided above, so that the beneficial effects achieved by the electronic device can refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Through the description of the foregoing embodiments, those skilled in the art will understand that, for convenience and simplicity of description, only the division of the functional modules is used for illustration, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the device may be divided into different functional modules, so as to complete all or part of the functions described above.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, a module or a unit may be divided into only one logic function, and may be implemented in other ways, for example, a plurality of units or components may be combined or integrated into another apparatus, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and these modifications or substitutions do not depart from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method applied to an electronic device includes:

acquiring a first RGB image and a Near Infrared (NIR) image which are acquired in the same scene;

respectively calculating pixel gradients on each color channel of a first RGB image to obtain a first gradient corresponding to the first RGB image;

calculating the gradient of the NIR image to obtain a second gradient corresponding to the NIR image;

fusing the first gradient and the second gradient to obtain a target gradient; wherein the target gradient matches a structure tensor of the second gradient;

and determining the approximate pixel value of each pixel point in the second RGB image according to the target gradient so as to generate the second RGB image.

2. The method of claim 1, further comprising:

the target gradient is in a linear relationship with the first gradient.

3. The method of claim 1, wherein determining an approximate pixel value for each pixel in the second RGB image based on the target gradient comprises:

constructing a poisson equation according to the target gradient, wherein the unknown quantity of the poisson equation is the pixel value of each pixel point in the second RGB image;

and solving an approximate solution of the Poisson equation based on discrete cosine transform, wherein the approximate solution is used as an approximate pixel value of each pixel point in the second RGB image.

4. The method of claim 3, wherein determining an approximate pixel value for each pixel in the second RGB image based on the target gradient comprises:

constructing a first Poisson equation according to the R channel component in the target gradient, wherein the unknown quantity of the first Poisson equation is the R channel pixel value of each pixel point in the second RGB image;

constructing a second Poisson equation according to the G channel component in the target gradient, wherein the unknown quantity of the second Poisson equation is the G channel pixel value of each pixel point in the second RGB image;

constructing a third Poisson equation according to the B channel component in the target gradient, wherein the unknown quantity of the third Poisson equation is the B channel pixel value of each pixel point in the second RGB image;

solving an approximate solution of the first Poisson equation based on discrete cosine transform, wherein the approximate solution is used as an R channel approximate pixel value of each pixel point in the second RGB image;

solving an approximate solution of the second Poisson equation based on discrete cosine transform, wherein the approximate solution is used as a G channel approximate pixel value of each pixel point in the second RGB image;

and solving an approximate solution of the third Poisson equation based on discrete cosine transform, wherein the approximate solution is used as a B-channel approximate pixel value of each pixel point in the second RGB image.

5. The method of claim 1, wherein the electronic device comprises a visible light camera module and a near infrared light camera module;

before the acquiring the first RGB image and the NIR image acquired in the same scene, further comprising:

in response to a first operation, collecting RGB image data through the visible light camera module and NIR image data through the near infrared light camera module;

processing the RGB image data to generate the first RGB image, and processing the NIR image data to obtain the NIR image;

after generating the second RGB image, further comprising:

and displaying and saving the second RGB image.

6. The method of claim 5, further comprising:

and responding to a second operation, and starting the near infrared camera module.

7. The method of claim 5, wherein the first RGB image is fused based on N RGB image frames, and the NIR image is fused based on N NIR image frames; and N is an integer greater than 1.

8. The method of claim 1, wherein calculating pixel gradients separately on each channel of a first RGB image, resulting in a first gradient corresponding to the first RGB image, comprises:

on each color channel of a first RGB image, aiming at each first target pixel point, calculating a gradient value corresponding to the first target pixel point according to pixel values of four pixel points which are adjacent to the first target pixel point in the vertical, horizontal and left directions so as to obtain a first gradient corresponding to the first RGB image;

calculating a gradient of a NIR image to obtain a second gradient corresponding to the NIR image, comprising:

and calculating a gradient value corresponding to each second target pixel point of the NIR image according to pixel values of four pixel points which are adjacent to the second target pixel point up, down, left and right so as to obtain a second gradient corresponding to the NIR image.

9. The method of claim 1, wherein the electronic device is a mobile phone.

10. An electronic device, comprising:

one or more processors;

a memory;

and one or more computer programs, wherein the one or more computer programs are stored on the memory, and when executed by the one or more processors, cause the electronic device to perform the image processing method of any of claims 1-9.

11. A computer-readable storage medium comprising a computer program, which, when run on an electronic device, causes the electronic device to perform the image processing method of any one of claims 1-9.