CN114331915A

CN114331915A - Image processing method and electronic device

Info

Publication number: CN114331915A
Application number: CN202210217112.7A
Authority: CN
Inventors: 周俊伟; 刘小伟; 王国毅
Original assignee: Honor Device Co Ltd
Current assignee: Shanghai Glory Smart Technology Development Co ltd
Priority date: 2022-03-07
Filing date: 2022-03-07
Publication date: 2022-04-12
Anticipated expiration: 2042-03-07
Also published as: CN114331915B

Abstract

The embodiment of the application provides an image processing method and electronic equipment. In the image processing method, aiming at a depth image with invalid depth values, the electronic equipment firstly converts the depth image into a point cloud image, repairs the point cloud image, and then converts the repaired point cloud image back into the depth image, so that the correction of the depth image is realized, and the accuracy of the depth image is improved.

Description

Image processing method and electronic device

Technical Field

The present application relates to the field of intelligent terminal technologies, and in particular, to an image processing method and an electronic device.

Background

Depth image (depth image) is an image in which the vertical distance (depth) from an image pickup to each point in a scene is a pixel value. The depth image directly reflects the geometry of the visible surface of the object and is a three-dimensional representation of the object. Wherein, the depth image can be acquired by a Time of Flight (TOF) camera. However, the depth map acquired by the TOF camera may have errors.

Disclosure of Invention

In order to solve the above technical problem, an embodiment of the present application provides an image processing method and an electronic device. In the method, the depth image is corrected based on a point cloud image repairing mode, so that the accuracy of the depth image is improved.

In a first aspect, an embodiment of the present application provides an image processing method. Wherein, the method comprises the following steps: the electronic equipment acquires a first depth image; the depth value of a target pixel point in the first depth image is an invalid value; the electronic equipment converts the first depth image into a first point cloud image; the electronic equipment repairs the first point cloud image to obtain a second point cloud image; the electronic device converts the second point cloud image to a second depth image. Therefore, the effect of correcting the depth image based on the point cloud image restoration is achieved, and the accuracy of the depth image is improved.

The first depth image may be understood as a depth image to be corrected, and a pixel point with an invalid depth value exists in the depth image. Accordingly, the second depth image may be understood as a corrected depth image corresponding to the first depth image.

The first point cloud image can be understood as a point cloud image to be repaired. For example, the first cloud image can be repaired by hole repairing. Accordingly, the second point cloud image can be understood as a repaired point cloud image corresponding to the first point cloud image.

According to the first aspect, the electronic device further comprises, before acquiring the first depth image: the electronic equipment acquires a third depth image and an infrared image corresponding to the third depth image; the third depth image and the infrared image are collected through a first TOF camera, and the first TOF camera is arranged below a display screen of the electronic equipment; the electronic equipment uses the diffraction light spot template set to perform multi-target template matching on the infrared image, and determines a successfully matched target diffraction light spot template; the electronic side sets the depth value of the pixel point in the target area of the third depth image as an invalid value to obtain a first depth image; and the target area is an area successfully matched with the target diffraction spot template.

Therefore, under the condition that the depth image is not accurate due to diffraction, the depth values of the pixels in the diffraction area can be corrected based on a point cloud image repairing mode, and the accuracy of the depth image is improved.

The third depth image may be understood as a depth image directly acquired by the under-screen TOF camera, and the first depth image is an image obtained by processing the third depth image. In the first depth image, the depth values of the pixels of the diffraction area are set to invalid values.

The diffraction light spot template set comprises a plurality of diffraction light spot templates, and each diffraction light spot template is used for describing one diffraction situation. And the target area which can be successfully matched with the diffraction light spot template in the infrared image is the diffraction area needing depth value correction.

According to the first aspect, or any implementation manner of the first aspect above, after determining that the matching of the target diffraction spot template is successful, the electronic device further includes: and the electronic equipment counts the number of diffraction pixel points in the target diffraction light spot template. Correspondingly, the electronic device sets the depth value of the pixel point in the target region of the third depth image to an invalid value, which may include: and if the quantity of diffraction pixel points in the target diffraction spot template does not exceed the preset quantity threshold value, the electronic equipment sets the depth value of the pixel points in the target area of the third depth image as an invalid value.

Because diffraction pixel number in the diffraction facula template can be used for instructing the degree of diffraction, diffraction pixel number is big more, and the degree of diffraction is more serious. Therefore, under the condition that the infrared image can be successfully matched with the diffraction spot template, only when the number of diffraction pixel points in the successfully matched diffraction spot template does not exceed a preset number threshold value, namely the diffraction condition related in the infrared image is not too serious, the depth value of the diffraction area pixel is corrected based on the point cloud image restoration mode, so that the correction effect of the depth image is ensured, and the problem that the depth image correction is invalid is avoided.

According to the first aspect, or any implementation manner of the first aspect above, the image processing method further includes: if the number of diffraction pixel points in the target diffraction spot template exceeds a preset number threshold, the electronic equipment displays prompt information; the prompt information is used for instructing a user to adjust the shooting angle to shoot again.

Therefore, under the condition that the infrared image can be successfully matched with the diffraction spot template, if the number of diffraction pixel points in the successfully matched diffraction spot template exceeds a preset number threshold value, namely the diffraction condition related to the infrared image is serious, the depth value of pixels in a diffraction area is not corrected by a point cloud image restoration-based mode, but a user is prompted to adjust the shooting angle to shoot again, and therefore the diffraction degree is reduced.

According to the first aspect, or any implementation manner of the first aspect above, the electronic device may set the depth value of the pixel point in the third depth image target region to an invalid value, and may include: setting the depth value of the diffraction pixel point in the target area of the third depth image as an invalid value; and the gray value of the diffraction pixel point in the infrared image is higher than a preset gray threshold value.

Therefore, only the depth values of the pixel points with higher diffraction degree in the diffraction area are set to be invalid values, the depth values of the pixel points are corrected in a targeted mode, the reliability of depth image correction can be improved, and the correction effect of the depth image is guaranteed.

According to the first aspect, or any one of the above implementation manners of the first aspect, the method for generating the diffraction spot template includes: the electronic equipment acquires a diffraction simulation point spread function; the electronic equipment creates a blank image matrix mask; the electronic equipment creates polygonal light spots in the center of the blank image matrix mask to obtain a light spot mask; the electronic equipment redistributes the central energy of the polygonal light spots in the light spot mask to obtain a light spot adjusting mask; and the electronic equipment convolutes the diffraction simulation point spread function with the light spot adjusting mask to obtain a diffraction light spot template. Therefore, the diffraction region in the infrared image can be effectively identified based on the pre-generated diffraction spot template.

The shape of the polygonal light spot can be determined according to the shape of the reflected light spot in an actual imaging system so as to simulate an actual diffraction light spot. The polygonal light spots created by the electronic equipment in the center of the blank image matrix mask are different in size and number, the positions of the created polygonal light spots are arranged differently, the obtained light spot masks are different, and therefore diffraction light spots in different situations can be simulated. Aiming at each light spot mask, the electronic equipment can amplify the central energy of the polygonal light spot to different degrees and randomly distribute the amplified energy to the surrounding pixel points of the center of the light spot again, so that various different light spot adjusting masks corresponding to the light spot mask can be obtained. In this way, the electronics can generate a series of spot adjustment masks.

According to the first aspect, or any implementation manner of the first aspect above, after converting the second point cloud image into the second depth image, the electronic device further includes: the electrons obtain a brightness difference template caused by diffraction according to a brightness mask of the gray value of the target diffraction spot template; the electronic equipment calculates the depth value difference of the diffraction region pixel points in the second depth image and the third depth image to obtain a depth difference template; if the change rules of the brightness difference template and the depth difference template are consistent, and the depth difference value of each pixel point in the depth difference template is within a preset depth difference range, the electronic equipment calculates the structural similarity of the second depth image and the third depth image; the electronic device determines that the second depth image correction is correct if the structural similarity exceeds a preset similarity threshold. Therefore, the electronic equipment carries out correctness verification on the corrected depth image, further ensures the correction effect of the depth image and avoids the problem of invalid or wrong correction of the depth image as much as possible.

According to the first aspect, or any implementation manner of the first aspect above, the image processing method further includes: if the electronic device determines that the second depth image correction is incorrect, the electronic device displays a prompt message; the prompt information is used for instructing a user to adjust the shooting angle to shoot again.

Therefore, when the corrected depth image fails to pass the correctness verification, the corrected depth image cannot be used by the application program, and at the moment, the electronic equipment prompts a user to adjust the shooting angle for shooting again, so that the diffraction degree is reduced, and the application program can perform related operations based on the depth image with higher precision.

According to the first aspect, or any implementation manner of the first aspect above, before the electronic device acquires the first depth image, the method further includes: the electronic equipment acquires a fourth depth image and overexposure pixel points in the fourth depth image; wherein the fourth depth image is acquired by a TOF camera; in the fourth depth image, the electronic device sets the depth value of the overexposure pixel point to an invalid value to obtain the first depth image.

Therefore, under the condition that the depth image is not accurate due to overexposure, the depth value of the overexposed pixel can be corrected based on a point cloud image repairing mode, and the accuracy of the depth image is improved.

The fourth depth image may be understood as a depth image directly acquired by the TOF camera, and the first depth image is an image obtained by processing the fourth depth image. In the first depth image, the depth value of the overexposed pixel point is set to be an invalid value.

In a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored on the memory, and when executed by the one or more processors, cause the electronic device to perform the image processing method as in any one of the first aspect and the first aspect.

Any one implementation manner of the second aspect and the second aspect corresponds to any one implementation manner of the first aspect and the first aspect, respectively. For technical effects corresponding to any one implementation manner of the second aspect and the second aspect, reference may be made to the technical effects corresponding to any one implementation manner of the first aspect and the first aspect, and details are not repeated here.

In a third aspect, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the image processing method of any one of the first aspect and the first aspect.

Any one implementation manner of the third aspect corresponds to any one implementation manner of the first aspect. For technical effects corresponding to any one implementation manner of the third aspect and the third aspect, reference may be made to the technical effects corresponding to any one implementation manner of the first aspect and the first aspect, and details are not repeated here.

In a fourth aspect, the present application provides a computer program product, which includes a computer program and when the computer program is executed, causes a computer to execute the image processing method according to the first aspect or any one of the first aspects.

Any one implementation manner of the fourth aspect and the fourth aspect corresponds to any one implementation manner of the first aspect and the first aspect, respectively. For technical effects corresponding to any one implementation manner of the fourth aspect and the fourth aspect, reference may be made to the technical effects corresponding to any one implementation manner of the first aspect and the first aspect, and details are not repeated here.

Drawings

FIG. 1a is an exemplary illustration of one of the depth maps acquired by a TOF camera;

FIG. 1b is one of the exemplary illustrated depth maps acquired by an off-screen TOF camera;

FIG. 1c is one of the exemplary illustrated depth maps acquired by an off-screen TOF camera;

fig. 2 is a schematic diagram of a hardware structure of an exemplary electronic device;

fig. 3 is a schematic diagram of a software structure of an exemplary electronic device;

FIG. 4 is a schematic diagram of module interaction provided by an embodiment of the present application;

fig. 5 is a schematic view of a depth image correction process provided in the embodiment of the present application;

fig. 6 is a schematic diagram of a process of generating a diffraction spot template according to an embodiment of the present disclosure;

FIG. 7a is an exemplary illustration of a series of spot masks;

FIG. 7b is an exemplary illustration of a series of diffraction spot templates;

FIG. 7c is another exemplary series of diffraction spot templates;

FIG. 8 is one of exemplary illustrative application scenario diagrams;

FIG. 9 is an exemplary illustration of the change in diffraction spot template for successful infrared image matching;

FIG. 10 is an exemplary illustration of a depth image to be corrected and a corresponding point cloud image to be repaired;

FIG. 11 is an exemplary illustration of a point cloud image to be repaired and a repaired point cloud image corresponding thereto;

FIG. 12 is a depth image obtained at various stages of an exemplary depth image correction procedure;

fig. 13 is a schematic view illustrating a process of verifying the correctness of the depth image correction according to the embodiment of the present application;

fig. 14 is a schematic view of a depth image correction process provided in an embodiment of the present application;

fig. 15 is an exemplary depth image to be corrected and a point cloud image to be repaired corresponding thereto;

fig. 16 is a schematic view of a verification process of the correctness of the depth image correction according to the embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone.

The terms "first" and "second," and the like, in the description and in the claims of the embodiments of the present application are used for distinguishing between different objects and not for describing a particular order of the objects. For example, the first target object and the second target object, etc. are specific sequences for distinguishing different target objects, rather than describing target objects.

In the embodiments of the present application, words such as "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion.

In the description of the embodiments of the present application, the meaning of "a plurality" means two or more unless otherwise specified. For example, a plurality of processing units refers to two or more processing units; the plurality of systems refers to two or more systems.

In an application scenario, when the energy emitted by the TOF camera encounters a high reflection region (e.g., a spectacle lens, etc.) or the exposure time is too long, an overexposure phenomenon occurs, and the confidence of the depth value calculated by the TOF camera collecting data is lower. In general, the depth value of the overexposed area is determined as an invalid depth, and the depth information of the area is lost. Therefore, the depth map acquired by the TOF camera is erroneous and needs to be corrected to improve the accuracy of the depth map.

Fig. 1a shows exemplarily a depth map acquired by a TOF camera (the depth map is shown in fig. 1a in the form of a grey scale map). When the TOF camera captures an image, the overexposure phenomenon is caused by the emitted energy encountering the spectacle lens, the overexposure areas being shown as area 1 and area 2 in fig. 1 a. In the area 1 and the area 2, the confidence of the depth values of the overexposed pixel points is relatively low, and the overexposed pixel points are generally determined as invalid depths. Thus, the depth map shown in fig. 1a is erroneous and needs to be corrected.

In another application scenario, the TOF camera is disposed under a screen of an electronic device (e.g., a mobile phone, etc.). In this case, when the TOF camera acquires data, a diffraction phenomenon occurs due to the structure of the screen. At this time, in the depth map acquired by the TOF camera, the depth value of the pixel point in the diffraction region may deviate from the actual depth value. Therefore, the depth map acquired by the electronic device under-screen TOF camera has errors, and needs to be corrected to improve the accuracy of the depth map.

Fig. 1b and 1c exemplarily show depth maps acquired by an off-screen TOF camera, respectively (the depth maps are shown in fig. 1b and 1c in the form of grey-scale maps). Diffraction phenomena occur when the image is acquired by the under-screen TOF camera, and the diffraction regions can be seen as region 3 in fig. 1b, and as

regions

4 and 5 in fig. 1 c. In the area 3, the area 4, and the area 5, there is a deviation between the depth values of a large number of pixel points and the actual depth values. Thus, the depth maps shown in fig. 1b and 1c are also erroneous and need to be corrected.

The depth map with errors is used in applications such as 3D anti-counterfeiting, depth measurement, face payment and the like, and the corresponding recognition result or measurement result and the like are undoubtedly influenced. Therefore, it is an urgent technical problem to correct a depth map acquired by a TOF camera and having an error to improve the accuracy of the depth map.

Fig. 2 is a schematic structural diagram of the electronic device 100. Optionally, the electronic device 100 may be a terminal, which may also be referred to as a terminal device, and the terminal may be a cellular phone (cellular phone), a tablet computer (pad), or the like, which is not limited in this application. It should be noted that the schematic structural diagram of the electronic device 100 may be applied to the electronic device with the TOF camera in fig. 1a to 1c, such as a mobile phone. It should be understood that the electronic device 100 shown in fig. 2 is only one example of an electronic device, and that the electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. The various components shown in fig. 2 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

The electronic device 100 may include: the mobile terminal includes a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a Subscriber Identity Module (SIM) card interface 195, and the like. Wherein the sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors. A memory may also be provided in processor 110 for storing instructions and data.

The charging management module 140 is configured to receive charging input from a charger. The charging management module 140 may also supply power to the electronic device through the power management module 141 while charging the battery 142.

The wireless communication function of the electronic device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like. The mobile communication module 150 may provide a solution including 2G/3G/4G/5G wireless communication applied to the electronic device 100. The wireless communication module 160 may provide a solution for wireless communication applied to the electronic device 100, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (bluetooth, BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), and the like.

In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 and antenna 2 is coupled to wireless communication module 160 so that electronic device 100 can communicate with networks and other devices through wireless communication techniques.

The electronic device 100 implements display functions via the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, video, and the like. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 194, with N being a positive integer greater than 1.

The electronic device 100 may implement a shooting function through the ISP, the camera 193, the video codec, the GPU, the display 194, the application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when a photo is taken, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converting into an image visible to naked eyes. The ISP can also carry out algorithm optimization on the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, the electronic device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

In embodiments of the present application, camera 193 may include a TOF camera (alternatively referred to as a TOF camera). TOF cameras are used to acquire TOF data. In certain implementations, the TOF camera is provided as a front-facing camera of the electronic device for acquiring TOF data in front of a display screen of the electronic device. For example, TOF data of a human face located in front of a display screen of an electronic device is acquired.

In certain implementations, the TOF camera includes a TOF sensor, a TOF sensor controller, a TOF light source, and a TOF light source controller.

In certain embodiments, the TOF light source controller is controlled by the TOF sensor controller to effect control of the TOF light source. The TOF light source emits Infrared (IR) light under control of the TOF light source controller. TOF sensors are used to sense light reflected from infrared light off an object (e.g., a human face, etc.) to acquire TOF data. The TOF sensor controller and TOF light source controller may be in communication with the processor 110. The processor 110 may also be configured to generate TOF images, including infrared images and depth images, from TOF data acquired by the TOF camera.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to extend the memory capability of the electronic device 100. The internal memory 121 may be used to store computer-executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121, so that the electronic device 100 implements the image processing method in the embodiment of the present application.

The electronic device 100 may implement audio functions via the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playing, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be disposed in the processor 110, or some functional modules of the audio module 170 may be disposed in the processor 110.

The pressure sensor is used for sensing a pressure signal and converting the pressure signal into an electric signal. In some embodiments, the pressure sensor may be disposed on the display screen 194. The electronic apparatus 100 may also calculate the touched position based on the detection signal of the pressure sensor.

Touch sensors, also known as "touch panels". The touch sensor may be disposed on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor is used to detect a touch operation applied thereto or nearby. The touch sensor can communicate the detected touch operation to the application processor to determine the touch event type.

The keys 190 include a power-on key, a volume key, and the like. The keys 190 may be mechanical keys. Or may be touch keys. The electronic apparatus 100 may receive a key input, and generate a key signal input related to user setting and function control of the electronic apparatus 100.

The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration cues, as well as for touch vibration feedback.

Indicator 192 may be an indicator light that may be used to indicate a charge status, a charge change, or a message.

The software system of the electronic device 100 may employ a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes an operating system with a layered architecture as an example, and exemplifies a software structure of the electronic device 100.

Fig. 3 is a block diagram of a software structure of the electronic device 100 according to the embodiment of the present application.

The layered architecture of the electronic device 100 divides the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, an application layer, an application framework layer, a Hardware Abstraction Layer (HAL), and a kernel layer from top to bottom.

The application layer may include a series of application packages.

As shown in fig. 3, the application package may include applications such as a payment application, a 3D anti-counterfeiting application, a depth measurement application, a depth map correction application, and the like.

Wherein the depth map correction application may be used to perform pixel depth correction on the diffraction regions and the overexposed regions in the depth map.

In some embodiments, the depth map correction application may be a system level application.

In some embodiments, the depth map correction application may be adapted as a depth map correction service in an application framework layer, which is not limited in this application.

The Application framework layer provides an Application Programming Interface (API) and a Programming framework for the Application of the Application layer, including various components and services to support android development by developers. The application framework layer includes a number of predefined functions. As shown in FIG. 3, the application framework layer may include a view system, a window manager, a camera service, and the like. Wherein the content of the first and second substances,

the window manager is used for managing window programs. The window manager can obtain the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

The view system includes visual controls such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The camera service is used for responding to the request of the application and calling the camera (comprising a front camera and a rear camera).

The HAL layer is an interface layer between the operating system kernel and the hardware circuitry. As shown in fig. 3, the HAL layer includes, but is not limited to: camera hardware abstraction layer (Camera HAL). Wherein Camera HAL is used to process the image stream.

The kernel layer is a layer between the hardware and the software layers described above. The inner core layer at least comprises a display driver, a TOF camera driver and a sensor driver. The hardware may include TOF cameras, display screens, processors, and memory, among other devices.

It is to be understood that the layers in the software structure shown in fig. 3 and the components included in each layer do not constitute a specific limitation of the electronic device 100. In other embodiments of the present application, electronic device 100 may include more or fewer layers than those shown, and may include more or fewer components in each layer, which is not limited in this application.

It is understood that, in order to implement the image processing method in the embodiment of the present application, the electronic device includes hardware and/or software modules for performing respective functions. The present application is capable of being implemented in hardware or a combination of hardware and computer software in conjunction with the exemplary algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, with the embodiment described in connection with the particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The embodiment of the application provides an image processing method, and particularly relates to an image processing method, aiming at a depth image to be corrected, after an electronic device sets the depth values of diffraction region pixel points and/or overexposure pixel points of the electronic device to be invalid values, the depth image to be corrected is firstly converted into a point cloud image to be restored, then the point cloud image to be restored is restored to obtain a restored point cloud image, the restored point cloud image is converted back into the depth image to be used as a corrected depth image corresponding to the depth image to be corrected, therefore, correction of the depth image obtained through a TOF camera is achieved, and the accuracy of the depth image is improved.

In one possible scenario, some applications are based on depth images to perform related functions, such as 3D anti-counterfeiting, depth measurement, face recognition, etc. If there is an error in the depth image, the processing result of the application program is inevitably affected, and therefore, it is necessary to improve the accuracy of the depth image. The following explains the relevant application scenario in detail by taking the payment application as an example. Similar to the depth measurement application and the 3D anti-counterfeiting application, the embodiments of the present application are not described in detail.

With reference to the schematic interaction flow diagram of each module shown in fig. 4, a process of performing face recognition based on a depth image for a payment application is described in detail below, and with reference to fig. 4, the process specifically includes:

s401, the payment application calls the camera service, and the camera service carries out corresponding processing.

For example, after the payment application is started, the payment application starts the camera service in response to a relevant operation of the user, for example, clicking a certain option or the like. Optionally, the payment application sends a request message to the camera service, and the request hour may include, but is not limited to, an application ID (which may be an application package name, for example) of the payment application.

For the processing flow of the camera service, reference may be made to the prior art, and details thereof are not described herein.

S402, calling Camera Hal by the Camera service.

And S403, calling the Camera drive in the kernel layer by Camera Hal.

And S404, driving and calling the TOF camera by the camera.

Illustratively, the camera service calls the Carema Hal, and the Carema Hal performs corresponding processing, for example, establishes a corresponding instance. Illustratively, the Carema Hal calls a camera driver, and the camera driver performs corresponding processing, for example, establishes a corresponding instance.

Illustratively, the TOF camera begins acquiring TOF data, which may generate infrared images and depth images, in response to a camera-driven invocation. The TOF camera collects the infrared image of the image and the pixel points of the depth image in a one-to-one correspondence mode. Regarding the generation process of the infrared image and the depth image, reference may be made to the technical solutions in the embodiments of the prior art, which is not described herein again.

And S405, outputting the acquired image to a camera for driving by the TOF camera.

And S406, the Camera drives the image to be output to Camera Hal.

S407, Camera Hal outputs the image to the Camera service.

Illustratively, the camera driver acquires images acquired by the TOF camera and outputs the images acquired by the TOF camera to Carema Hal. And the card Hal outputs the images collected by the TOF camera to a camera for service.

The images acquired by the TOF camera may include, among other things, infrared images and depth images.

S408, the camera service outputs the image to the depth image correction application.

The camera service outputs images (including infrared images and depth images) collected by the TOF camera to the depth image correction application, so that the depth images are corrected through the depth image correction application, and the accuracy of the depth images is improved.

In another possible implementation, the camera service outputs the image to a payment application, which sends the image to a depth image correction application to correct the depth image by the depth image correction application.

And S409, the depth image correction application corrects the depth image and sends the corrected depth image to the payment application.

In another possible embodiment, after the depth image correction application corrects the depth image, the corrected depth image may be sent to the payment application along with the infrared image. This is not limited in this application.

Further, the payment application may perform face recognition and other related operations based on the corrected depth image. The related operations of the payment application may refer to the technical solutions in the prior art embodiments, and are not described herein again.

One possible scenario is: the TOF camera is disposed under the screen of the electronic device (the TOF camera may be referred to as an off-screen TOF camera). At the moment, images are collected through a TOF camera, the depth value of a pixel point in an image diffraction area is deviated from the actual depth, and the depth value needs to be corrected. Another possible scenario is: the TOF camera is disposed in a screenless environment (i.e., the TOF camera is a non-screenless TOF camera, or the TOF camera may be referred to as a screenless TOF camera). At the moment, images are collected through a TOF camera, and the confidence coefficient of the depth values of the overexposed pixel points in the images is low.

Thus, the depth image acquired by the TOF camera needs to be corrected, which may be referred to as a depth image to be corrected. After receiving the depth image to be corrected, the depth image correction application corrects the depth image to be corrected and sends the corrected depth image to the payment application, so that the payment application can perform face recognition and other related operations according to the accurate depth image.

Scene one

In the scene, the TOF camera is an off-screen TOF camera, and depth image correction application needs to correct depth values of pixels in a diffraction area. The following explains the depth image correction flow in detail.

Fig. 5 is a schematic diagram illustrating a depth image correction process. Referring to fig. 5, the process of correcting a depth image by the depth image correction application provided in this embodiment specifically includes:

s501, the depth image correction application acquires an infrared image and a diffraction light spot template set, and multi-target template matching is carried out on the infrared image.

Because, the TOF camera collects the depth image of the image and the pixel point of the infrared image in one-to-one correspondence. In the present embodiment, the identification of the diffraction region is performed based on the infrared image. Specifically, the depth image correction application performs multi-target template matching on the infrared image based on the diffraction spot template set to identify diffraction regions in the infrared image.

Wherein, the diffraction light spot template set comprises a series of diffraction light spot templates. The diffraction spot template may also be referred to as a diffraction spot template image. Different diffraction spot templates are used to describe different diffraction scenarios.

For diffraction spots of the same shape, a plurality of corresponding diffraction spot templates can be preset. The diffraction spot template set may include a plurality of diffraction spot templates respectively corresponding to the diffraction spots of the plurality of shapes. For example, the image size of the diffraction spot template may be 137 × 137 pixels.

Fig. 6 is a schematic diagram of a process for generating a diffraction spot template. Referring to fig. 6, the generation process of the diffraction spot template provided in this embodiment specifically includes:

s601, the electronic equipment acquires a diffraction simulation PSF.

PSF (Point Spread Function) is used to describe the response of the imaging system to a Point source or Point object, and may also be referred to as an impulse response used to describe the focusing optics. In this embodiment, the PSF acquired by the electronic device is diffraction-dependent and used for performing diffraction simulation.

S602, the electronic equipment creates a blank image matrix mask.

The blank image matrix mask refers to an image matrix mask that does not contain any image content, and may be, for example, an image matrix mask in which the dot pixel values are all 0. Illustratively, the image size of the blank image matrix mask is 137 × 137 pixels.

S603, the electronic equipment creates polygonal light spots in the center of the blank image matrix mask to obtain a light spot mask.

Illustratively, in the spot mask, the pixel value of the polygonal spot is 1.

The shape of the polygonal light spot can be determined according to the shape of the reflected light spot in an actual imaging system so as to simulate the actual situation. The shape of the polygonal spot is not limited in this embodiment.

The polygonal light spots created by the electronic equipment in the center of the blank image matrix mask are different in size and number, the positions of the created polygonal light spots are arranged differently, the obtained light spot masks are different, and therefore diffraction light spots in different situations can be simulated.

The electronic equipment can obtain a series of light spot masks based on the blank image matrix mask and a plurality of different polygonal light spots. Fig. 7a exemplarily shows a series of spot masks, and (1) - (6) in fig. 7a are examples of the spot masks.

S604, the electronic equipment redistributes the central energy of the polygonal light spots in the light spot mask to obtain the light spot adjusting mask.

Illustratively, the electronic device performs energy recovery of a spot center point in a spot mask according to cross talk of a sensor (sensor), and randomly reassigns energy to pixel points around the spot center to obtain a spot adjustment mask. The light spot adjusting mask is an image matrix mask obtained by simulating light spots on the basis of a blank image matrix mask.

Aiming at each light spot mask, the electronic equipment can amplify the central energy of the polygonal light spot to different degrees and randomly distribute the amplified energy to the surrounding pixel points of the center of the light spot again, so that various different light spot adjusting masks corresponding to the light spot mask can be obtained. In this way, the electronics can generate a series of spot adjustment masks.

And S605, the electronic equipment convolutes the diffraction simulation PSF with the light spot adjusting mask to obtain a diffraction light spot template.

Wherein, the light spot adjusting masks are different, and the obtained diffraction light spot templates are different.

Aiming at a series of light spot adjusting masks, the electronic equipment convolutes the diffraction simulation PSF with each light spot adjusting mask respectively to obtain a series of diffraction light spot templates.

Fig. 7b schematically shows a series of diffraction spot templates corresponding to the series of spot masks of fig. 7a, and fig. 7c schematically shows another series of diffraction spot templates corresponding to the series of spot masks of fig. 7 a. Taking the spot mask shown in (1) in fig. 7a as an example, the diffraction spot templates shown in (1) in fig. 7b and (1) in fig. 7c are both corresponding to the spot mask, and the spot energies corresponding to the diffraction spot template shown in (1) in fig. 7b and the diffraction spot template shown in (1) in fig. 7c are different. The spot masks shown in (2) to (6) in fig. 7a are also similar, and are not described again here.

It should be noted that the electronic device executing the generation process of the diffraction spot template may be an electronic device executing a depth image correction process, or may be another electronic device, which is not limited in this embodiment. For example, if the electronic device performing the generation process of the diffraction spot template is an electronic device performing a depth image correction process, the generation process of the diffraction spot template may be performed by a template generation module in an application layer of the electronic device.

And S502, judging whether the diffraction spot template is matched or not by the depth image correction application, if so, executing S503, and if not, executing S511.

If the infrared image cannot be matched with any diffraction light spot template, the depth image corresponding to the infrared image is not required to be corrected. At this time, the depth image correction application may send the received depth image to the payment application as a corrected depth image.

If the infrared image is matched with any diffraction spot template, the depth image corresponding to the infrared image is required to be corrected. At this time, the depth image correction application continues to perform an operation of correcting the received depth image.

S503, counting the number of diffraction pixel points in a target diffraction light spot template by the depth image correction application, wherein the target diffraction light spot template is a successfully matched diffraction light spot template.

The depth image correction application carries out multi-target template matching on an infrared image based on a diffraction light spot template set, if a certain area in the infrared image is successfully matched with a certain diffraction light spot template, the diffraction light spot template is used as a target diffraction light spot template, the area is used as an area needing depth correction, and the area can be called as a target area or a diffraction area.

In each diffraction light spot template, the diffraction intensity of each pixel point is different. The diffraction intensity can be determined according to the gray value of the pixel point, and the larger the gray value is, the higher the diffraction intensity is.

The depth image correction application can divide diffraction pixel points and non-diffraction pixel points in the diffraction light spot template according to the diffraction intensity of the pixel points. Illustratively, if the gray value of a pixel point is greater than the first gray threshold, the pixel point is divided into diffraction pixel points, otherwise, the pixel point is divided into non-diffraction pixel points.

And S504, the depth image correction application judges whether a target diffraction light spot template with the diffraction pixel number exceeding a preset number threshold exists, if so, S505 is executed, and if not, S506 is executed.

And counting the number of diffraction pixel points in each target diffraction spot template, and comparing the number with a preset number threshold. The number of diffraction pixel points in the diffraction spot template can be used for representing the size of a diffraction area in the diffraction spot template, and the preset number threshold can be used for evaluating the size of the diffraction area in the diffraction spot template.

For example, if the number of diffraction pixel points in the target diffraction spot template exceeds a preset number threshold, the depth image correction application may consider that the correction difficulty of the target region matched with the target diffraction spot template is high, or that the target region cannot be corrected. If the number of diffraction pixel points in the target diffraction spot template does not exceed the preset number threshold, the depth image correction application may consider that the correction difficulty of the target region matched with the target diffraction spot template is low, or call that the target region may be corrected.

S505, the depth image correction application sends instruction information to the payment application.

When the infrared image can be successfully matched with the diffraction light spot template with the number of any diffraction pixel points exceeding the preset number threshold, the depth image correction application can send indication information to the payment application to indicate that the depth image cannot be corrected, and the TOF image and the like are acquired again after the shooting direction needs to be adjusted.

Illustratively, the indication information may include, but is not limited to, indication content, TOF image acquisition adjustment mode, and the like. Wherein, the indication content may be, for example, an identifier for indicating that the TOF image needs to be reacquired; the TOF image acquisition adjustment mode may be, for example, a shooting angle adjustment direction and/or an adjustment amplitude.

For example, the depth image correction application may determine the shooting angle adjustment direction and/or the adjustment amplitude, and the like, according to the size of the diffraction region in the target diffraction spot template in which the number of diffraction pixel points exceeds the preset number threshold, and the position of the target region matched with the target diffraction spot template in the infrared image.

And then, after the payment application receives the indication information, the payment application prompts the user through the information prompt box so as to prompt the user to adjust the angle and enable the TOF camera to acquire the image again.

Generally, the size and number of diffraction spots in an image are related to the shooting angle. When the diffraction spot is too large, the overall quality of the image is biased, and the accuracy of the increase in the difficulty of correcting the depth map is also reduced. Therefore, in order to improve the quality of the shot image, the electronic equipment can judge according to the currently shot image and carry out corresponding shooting guidance.

Fig. 8 (1) and fig. 8 (2) each schematically show an application scenario. As shown in (1) in fig. 8, the payment application prompts the user to adjust the shooting angle for image acquisition again through a prompt box 801; as shown in (2) in fig. 8, the payment application prompts the user to adjust the shooting angle for image capture again through a prompt box 802, and prompts the angle adjustment direction.

After the user adjusts the shooting angle, the electronic device re-executes the steps S401-S408 shown above, and after the depth map correction application receives the infrared image re-collected by the TOF camera, the depth image correction flow shown in FIG. 5 is repeatedly executed. If the infrared image collected by the TOF camera again can be matched with the diffraction spot template with the number of any diffraction pixel points exceeding the preset number threshold value successfully, the depth image correction application continues to send indication information to the payment application to indicate a user to adjust the shooting angle to collect the image again until the infrared image cannot be matched with any diffraction spot template successfully or the number of the diffraction pixel points in any diffraction spot template successfully matched does not exceed the preset number threshold value.

Fig. 9 shows an exemplary application scenario. With the adjustment of the shooting angle of the user, the infrared image collected by the TOF camera can be matched with the change of the successful diffraction light spot template, which can be referred to fig. 9. As shown in fig. 9, when the infrared image collected by the TOF camera can be successfully matched with the diffraction spot template 901, if the number of diffraction pixel points in the diffraction spot template 901 exceeds a preset number threshold, the electronic device prompts a user to adjust a shooting angle. After the user adjusts the shooting angle, it is assumed that the infrared image re-collected by the TOF camera can be successfully matched with the diffraction spot template 902, and if the number of diffraction pixel points in the diffraction spot template 902 still exceeds the preset number threshold, the electronic device prompts the user to adjust the shooting angle again. After the user adjusts the shooting angle again, it is assumed that the infrared image re-collected by the TOF camera can be successfully matched with the diffraction spot template 903, and if the number of diffraction pixel points in the diffraction spot template 903 still exceeds the preset number threshold, the electronic device prompts the user to adjust the shooting angle again. After the user adjusts the shooting angle again, assuming that the infrared image re-collected by the TOF camera can be successfully matched with the diffraction spot template 904, if the number of diffraction pixel points in the diffraction spot template 904 does not exceed the preset number threshold, the electronic device corrects the depth image through depth image correction application.

S506, the depth image correction application sets the depth values of the pixel points in the target area in the depth image to be invalid values; and the target area is an area successfully matched with the target diffraction spot template.

In any diffraction spot template in which the infrared image can be successfully matched, if the number of diffraction pixel points does not exceed a preset number threshold, the depth image correction application corrects the corresponding depth image.

First, the depth image correction application sets the depth value of each pixel point in the target area (i.e., diffraction area) successfully matched with each target diffraction spot template in the depth image to an invalid value, for example, to 0.

As the actual analysis can know, the depth value difference of each pixel point in the diffraction area is related to the diffraction intensity. Therefore, as an alternative embodiment, the depth image correction application sets the depth value of the pixel point in the diffraction area whose gray value is higher than the second gray threshold value as an invalid value.

S507, the depth image correction application converts the depth image into a point cloud image.

And (4) resetting the depth image with invalid depth values for the pixels in the diffraction area, and converting the depth image into a point cloud image by the depth image correction application.

Illustratively, the depth image correction application converts the depth image into a point cloud image by using the internal parameters of the TOF camera, and the point cloud image obtained at this time may be referred to as a point cloud image to be repaired.

Wherein, the internal parameters of the TOF camera mainly include an optical center (or projection center) C (C: (C)cx,cy) And a focal length F: (fx,fy）。

The calculation formula for transforming the depth image into the point cloud image is as follows:

。

wherein the content of the first and second substances,x、y、zis the coordinates of the point cloud,x’、y' is the coordinates of the image pixel,Drepresenting depth values of pixels of the image.

Fig. 10 shows an exemplary application scenario. Fig. 10 (1) shows an exemplary depth image (shown in the form of a gray scale map) obtained by resetting invalid depth values for the pixel points in the diffraction region, and fig. 10 (2) shows an exemplary point cloud image (shown in the form of a gray scale map) obtained by converting the depth map shown in fig. 10 (1). Referring to (1) in fig. 10, the region 1001 and the region 1002 are diffraction regions, and the depth values of some pixel points in the region are invalid values. Referring to (2) in fig. 10, a region 1003 and a region 1004 are point cloud regions to be repaired.

And S508, repairing the point cloud image by the depth image correction application.

After the depth image correction application maps the depth image to be corrected into a point cloud image, point cloud information restoration can be performed based on a point cloud hole filling algorithm.

Illustratively, the depth image correction application first performs preprocessing and denoising on an input point cloud image, then extracts boundary points of a hole, and finally reduces a hole area in an iterative propagation manner until no new filling point is generated, thereby completing the restoration of the point cloud image.

When the depth image correction is applied to the point cloud image restoration, the correction can be performed only on the diffraction area and the overexposure area. Fig. 11 shows an exemplary application scenario. Fig. 11 (1) schematically shows the point cloud image before hole repairing (shown in the form of a gray scale), and fig. 11 (2) schematically shows the point cloud image after hole repairing (shown in the form of a gray scale). Referring to fig. 11 (1),

areas

1101 and 1102 show the case where the diffraction area point cloud is not repaired. Referring to (2) in fig. 11, an area 1103 and an area 1104 show the case after the point cloud restoration of the diffraction area.

In addition, referring to (1) in fig. 11, an area 1105 exemplarily shows a point cloud hole situation in a non-diffraction area. The method for repairing the point cloud holes in the diffraction area and the overexposure area can also be used for repairing the point cloud holes in the non-diffraction area.

And S509, converting the repaired point cloud image into a depth image by the depth image correction application, wherein the depth image is used as a corrected depth image.

For example, the depth image correction application converts the repaired point cloud image into a depth image by using internal and external parameters of the TOF camera, and the depth image obtained at this time may be referred to as a corrected depth image.

The calculation formula for converting the point cloud image into the depth image is as follows:

。

wherein (A), (B), (C), (D), (C), (B), (C)cx,cy) Is the optical center (or projection center) ((ii))fx,fy) Is a focal length of (x,y,z) As point cloud coordinates (a)u,v) Are the coordinates of the pixels of the image,zfor the depth value of the corresponding pixel coordinate, can be at pixel point (u,v) Corresponding position filling depthzThe value is obtained.

It should be noted that when the depth image correction is applied to transform the point cloud image into the depth image, the occlusion effect and the projection problem need to be considered to improve the accuracy of the depth image.

In this embodiment, when the depth image correction is applied to transform the point cloud image into the depth image, in order to eliminate the occlusion effect, when there are a plurality of objects mapped to the same pixel coordinate, only the object with the smallest depth value is reserved in the depth image, that is, only the smallest one of the depth values corresponding to the pixel coordinate is selected as the target depth value corresponding to the pixel coordinate.

In the embodiment, when the depth image correction is applied to transform the point cloud image into the depth image, in order to solve the transmission problem, a method of morphological filtering of a gray scale image is adopted. Wherein, the conventional morphological filtering generally acts on the binary image, and the morphological filtering of the gray-scale map comprises morphological expansion and corrosion based on the gray-scale map. Here, the depth map is used as a gray scale map, and the depth value of the pixel at the center point of the window is adjusted by erosion operation or dilation operation. Specifically, if the depth value of a certain window center point is greater than the depth values of other pixel points around the window center point, the depth value of the window center point is determined again according to the depth values of other pixel points around the center point. Wherein the content of the first and second substances,

the calculation formula of the depth map morphological dilation operation is as follows:

。

wherein the content of the first and second substances,

is a depth map after expansion.

The calculation formula of the depth map morphological corrosion operation is as follows:

。

wherein the content of the first and second substances,

is a depth map after corrosion.

At this point, the depth image correction application completes the correction of the depth image to be corrected (the diffraction region exists), and obtains the corrected depth image.

Fig. 12 shows an exemplary application scenario. Fig. 12 (1) schematically shows a diffracted depth map image (shown in the form of a gray scale map), i.e., a depth image to be corrected, in which a diffraction region can be shown with reference to a region 1201. The depth image correction application performs multi-target template matching on the infrared image corresponding to the depth image shown in (1) in fig. 12, and if the area 1201 is successfully matched with a certain area and the number of diffraction pixel points in the diffraction spot template does not exceed the preset number threshold, the depth value of the pixel point with stronger diffraction (for example, the middle gray scale value is higher than the second gray scale threshold) in the area 1201 may be set as an invalid value, at this time, the depth image may refer to (2) in fig. 12 (shown in the form of a gray scale), and the diffraction area may refer to the area 1202. The depth image correction application converts the depth image shown in (2) in fig. 12 into a point cloud image, performs hole repairing on the point cloud image, and converts the point cloud image into a depth image to obtain a corrected depth image, which may be referred to as (3) in fig. 12 (shown in a form of a gray scale), and the corrected diffraction region may be referred to as a region 1203.

S510, the depth image correction application verifies whether the corrected depth image is correct, if yes, then S511 is executed, and if not, then S505 is executed.

The depth image correction application performs correctness verification on the corrected depth image. If the verification result indicates correct, the depth image correction application sends the corrected depth image to the payment application; if the verification result indicates that the image is incorrect, the depth image correction application may send indication information to the payment application to indicate that the depth image cannot be corrected, that the TOF image needs to be reacquired after the shooting direction is adjusted, and the like.

Fig. 13 is a flowchart illustrating a method for verifying correctness of depth image correction. Referring to fig. 15, a process of performing depth image correction correctness verification by a depth image correction application for a depth image obtained by correcting a depth value of a pixel point in a diffraction region specifically includes:

and S1301, correcting the depth image by using a brightness mask according to the gray value of the target diffraction spot template to obtain a brightness difference template caused by diffraction.

And the target diffraction light spot template is a diffraction light spot template successfully matched with the infrared image.

Illustratively, the brightness difference value of each pixel point in the brightness difference template is: the brightness difference between the pixel point and the central pixel point.

S1302, the depth image correction application calculates the depth value difference of the pixel points in the diffraction areas before and after correction to obtain a depth difference template.

Wherein, the depth difference value of each pixel point in the depth difference template is: and the difference value between the depth value of the pixel point after the diffraction area correction and the depth value of the pixel point before the diffraction area correction.

S1303, the depth image correction application determines whether the change rules of the brightness difference template and the depth difference template are consistent, if yes, then S1304 is executed, and if not, then S1308 is executed.

S1304, the depth image correction application determines whether the depth difference value of each pixel point in the depth difference template is within a preset depth difference range, if so, then S1305 is executed, and if not, then S1308 is executed.

Wherein, assuming that the maximum depth deviation of the diffraction region except the central overexposure region is d _ max and the minimum depth deviation is d _ min, the difference range at the preset depth is [ d _ min, dmax ].

S1305, the depth image correction application calculates structural similarity of the depth images before and after correction.

On the premise that the above-mentioned determination conditions are all satisfied, the depth image correction application calculates SSIM (Structural Similarity) values of the depth images before and after correction in a manner similar to the calculation of the Similarity of the grayscale map, so as to finally confirm the Similarity values of the depth images before and after correction.

For the calculation method of the SSIM value of the image, reference may be made to the prior art, and details thereof are not repeated here.

S1306, the depth image correction application determines whether the structural similarity exceeds a preset similarity threshold, if so, then S1307 is executed, and if not, then S1308 is executed.

If the depth image correction application determines that the structural similarity exceeds a preset similarity threshold, the corrected depth image may be deemed correct, otherwise the corrected depth image may be deemed incorrect.

S1307, the depth image correction application verifies that the corrected depth image is correct.

S1308, the depth image correction application verifies that the corrected depth image is incorrect.

So far, the depth image correction application is used to verify the correctness of the depth image correction for the depth image obtained by correcting the depth value of the pixel point in the diffraction region.

As can be seen from the actual analysis, there is usually a certain range of variation in the depth values of the diffraction regions. Therefore, an upper and lower depth deviation threshold can be preset, and the depth image correction application verifies the corrected depth image according to the upper and lower depth deviation threshold and the depth value of the adjacent pixel point which is not diffracted, so as to obtain the correctness verification result of the corrected depth image.

S511, the depth image correction application sends the depth image to the payment application.

In one case, if the infrared image does not match any of the diffraction spot templates, it indicates that the depth image corresponding to the infrared image is not to be corrected. At this time, the depth image correction application may send the received depth image to the payment application as a corrected depth image.

In another case, the depth image correction application corrects the received depth image and sends the corrected depth image with the correctness verified to the payment application.

Furthermore, the payment application can perform face recognition and other related operations according to the depth image with higher accuracy.

Scene two

In the scene, the TOF camera is a screenless TOF camera, and depth image correction application needs to correct the depth value of an overexposure pixel point. The following explains the depth image correction flow in detail.

Fig. 14 is a schematic diagram illustrating a depth image correction process. Referring to fig. 14, the process of correcting a depth image by the depth image correction application provided in this embodiment specifically includes:

s1401, the depth image correction application sets the depth value of the overexposed pixel point in the depth image to an invalid value.

Illustratively, the invalid value may be 0.

S1402, the depth image correction application converts the depth image into a point cloud image.

And aiming at the depth image obtained by resetting the invalid depth value of the overexposed pixel point in the depth image, converting the depth image into a point cloud image by the depth image correction application.

WhereinThe internal reference of the TOF camera mainly includes an optical center (or projection center) C: (cx,cy) And a focal length F: (fx,fy）。

。

Fig. 15 shows an exemplary application scenario. Fig. 15 (1) schematically shows a depth image (shown in the form of a gray scale map) in which an overexposed pixel point exists, and fig. 15 (2) schematically shows a point cloud image (shown in the form of a gray scale map) obtained by converting the depth map shown in fig. 15 (1). Referring to fig. 15 (1), the region 1501 includes overexposed pixels. Referring to fig. 15 (2), a region 1502 is a point cloud region to be repaired.

And S1403, the point cloud image is repaired by the depth image correction application.

S1404, the depth image correction application converts the repaired point cloud image into a depth image as a corrected depth image.

。

。

wherein the content of the first and second substances,

is a depth map after expansion.

。

wherein the content of the first and second substances,

is a depth map after corrosion.

At this point, the depth image correction application finishes correcting the depth image to be corrected (including the overexposed pixel points), and a corrected depth image is obtained.

Further, the depth image correction application may perform correctness verification on the corrected depth image. If the verification result indicates correct, the depth image correction application sends the corrected depth image to the payment application; if the verification result indicates that the image is incorrect, the depth image correction application may send indication information to the payment application to indicate that the depth image cannot be corrected, that the TOF image needs to be reacquired after the shooting direction is adjusted, and the like.

Fig. 16 is a flowchart illustrating an exemplary method for verifying correctness of depth image correction. Referring to fig. 16, a process of performing, by a depth image correction application, verification of correctness of correction of a depth image for a depth image obtained by correcting a depth value of an overexposed pixel includes:

s1601, the depth image correction application calculates structural similarities of the depth images before and after correction.

The depth image correction application calculates the SSIM value of the depth image before and after correction based on a manner similar to the grayscale map similarity calculation to confirm the similarity value of the depth image before and after correction.

S1602, the depth image correction application determines whether the structural similarity exceeds a preset similarity threshold, if so, then S1603 is executed, and if not, then S1604 is executed.

S1603, the depth image correction application verifies that the corrected depth image is correct.

S1604, the depth image correction application verifies that the corrected depth image is incorrect.

At this point, for the depth image obtained by correcting the depth value of the overexposed pixel point, the depth image correction application is used to verify the correctness of the depth image correction.

It should be noted that, for different scenes, for example, the depth image correction application corrects the depth values of the pixels in the diffraction region, and for example, the depth image correction application corrects the depth values of the overexposed pixels, and the method for verifying the correctness of the depth image correction by the depth image correction application may be different, which is not limited in this embodiment of the present application.

The present embodiment also provides a computer storage medium, in which computer instructions are stored, and when the computer instructions are run on an electronic device, the electronic device executes the above related method steps to implement the image processing method in the above embodiment.

The present embodiment also provides a computer program product, which when run on a computer causes the computer to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

In addition, embodiments of the present application also provide an apparatus, which may be specifically a chip, a component or a module, and may include a processor and a memory connected to each other; the memory is used for storing computer execution instructions, and when the device runs, the processor can execute the computer execution instructions stored in the memory, so that the chip can execute the image processing method in the above-mentioned method embodiments.

In addition, the electronic device (such as a mobile phone, etc.), the computer storage medium, the computer program product, or the chip provided in this embodiment are all configured to execute the corresponding method provided above, so that the beneficial effects achieved by the electronic device can refer to the beneficial effects in the corresponding method provided above, and are not described herein again.

Through the description of the above embodiments, those skilled in the art will understand that, for convenience and simplicity of description, only the division of the above functional modules is used as an example, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described embodiments of the apparatus are merely illustrative, and for example, a module or a unit may be divided into only one logic function, and may be implemented in other ways, for example, a plurality of units or components may be combined or integrated into another apparatus, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, comprising:

acquiring a first depth image; the depth value of a target pixel point in the first depth image is an invalid value;

converting the first depth image into a first point cloud image;

repairing the first point cloud image to obtain a second point cloud image;

and converting the second point cloud image into a second depth image.

2. The method of claim 1, further comprising, prior to said acquiring the first depth image:

acquiring a third depth image and an infrared image corresponding to the third depth image; wherein the third depth image and the infrared image are acquired by a first TOF camera; the first TOF camera is arranged below a display screen of the electronic equipment;

using a diffraction light spot template set to perform multi-target template matching on the infrared image, and determining a successfully matched target diffraction light spot template;

setting the depth value of the pixel point in the target area of the third depth image as an invalid value to obtain the first depth image; and the target area is an area successfully matched with the target diffraction spot template.

3. The method of claim 2, further comprising, after determining the successfully matched target diffraction spot template:

counting the number of diffraction pixel points in the target diffraction spot template;

setting the depth value of the pixel point in the third depth image target area to be an invalid value, including:

and if the number of diffraction pixel points in the target diffraction spot template does not exceed a preset number threshold, setting the depth value of the pixel points in the target area of the third depth image as an invalid value.

4. The method of claim 3, further comprising:

if the number of diffraction pixel points in the target diffraction spot template exceeds a preset number threshold, displaying prompt information; and the prompt information is used for instructing the user to adjust the shooting angle to shoot again.

5. The method according to claim 2 or 3, wherein setting the depth value of the pixel point in the target region of the third depth image to an invalid value comprises:

setting the depth value of the diffraction pixel point in the target area of the third depth image as an invalid value; and the gray value of the diffraction pixel point in the infrared image is higher than a preset gray threshold value.

6. The method of claim 2, wherein the method of generating the diffraction spot template comprises:

acquiring a diffraction simulation point spread function;

creating a blank image matrix mask;

creating polygonal light spots in the center of the blank image matrix mask to obtain a light spot mask;

redistributing the central energy of the polygonal light spots in the light spot mask to obtain a light spot adjusting mask;

and convolving the diffraction simulation point spread function with the light spot adjusting mask to obtain the diffraction light spot template.

7. The method of claim 2, after converting the second point cloud image to the second depth image, further comprising:

obtaining a brightness difference template caused by diffraction according to the brightness mask of the gray value of the target diffraction spot template;

calculating the depth value difference of the diffraction region pixel points in the second depth image and the third depth image to obtain a depth difference template;

if the change rules of the brightness difference template and the depth difference template are consistent, and the depth difference value of each pixel point in the depth difference template is within a preset depth difference range, calculating the structural similarity of the second depth image and the third depth image;

and if the structural similarity exceeds a preset similarity threshold, determining that the second depth image is corrected correctly.

8. The method of claim 7, further comprising:

if the second depth image is determined to be incorrectly corrected, displaying a prompt message; and the prompt information is used for instructing the user to adjust the shooting angle to shoot again.

9. The method of claim 1, further comprising, prior to said acquiring the first depth image:

acquiring a fourth depth image and overexposed pixel points in the fourth depth image; wherein the fourth depth image is acquired by a second TOF camera;

and in the fourth depth image, setting the depth value of the overexposure pixel point as an invalid value to obtain the first depth image.

10. An electronic device, comprising:

one or more processors;

a memory;

and one or more computer programs, wherein the one or more computer programs are stored on the memory, and when executed by the one or more processors, cause the electronic device to perform the image processing method of any of claims 1 to 9.

11. A computer-readable storage medium comprising a computer program, which, when run on an electronic device, causes the electronic device to perform an image processing method according to any one of claims 1-9.