CN116520348A - Depth imaging system, method, equipment and medium based on modulated light field - Google Patents
Depth imaging system, method, equipment and medium based on modulated light field Download PDFInfo
- Publication number
- CN116520348A CN116520348A CN202310043248.5A CN202310043248A CN116520348A CN 116520348 A CN116520348 A CN 116520348A CN 202310043248 A CN202310043248 A CN 202310043248A CN 116520348 A CN116520348 A CN 116520348A
- Authority
- CN
- China
- Prior art keywords
- multipath interference
- light field
- image
- depth
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 38
- 238000000034 method Methods 0.000 title abstract description 19
- 238000005259 measurement Methods 0.000 claims abstract description 11
- 230000008030 elimination Effects 0.000 claims abstract description 6
- 238000003379 elimination reaction Methods 0.000 claims abstract description 6
- 230000004927 fusion Effects 0.000 claims description 27
- 238000012545 processing Methods 0.000 claims description 10
- 238000005286 illumination Methods 0.000 claims description 5
- 230000004907 flux Effects 0.000 claims description 3
- 230000005855 radiation Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 17
- 238000012216 screening Methods 0.000 description 9
- 238000012986 modification Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 4
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/481—Constructional features, e.g. arrangements of optical elements
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Electromagnetism (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
A modulated light field based depth imaging system comprising the following modules: the light projection module is used for projecting discrete light beams to a target object and switching a modulated light field mode and a uniform light field mode through modulation; the light source control module is used for emitting light rays and controlling switching light field modes; the TOF sensing module is used for receiving the back measuring beam reflected by the measured target object; and the multipath interference elimination module is used for acquiring the modulated light field mode and the image data under the uniform light field, estimating multipath interference components from the modulated light field mode and the image data, removing measurement errors caused by multipath interference and outputting a high-precision depth image. The invention can determine the multipath interference component through the background area data in the first image data acquired when the discrete light beams are projected to the target object, and process the light spot area data to remove the multipath interference component, thereby eliminating the measurement error caused by multipath interference and realizing the output of the high-precision depth image.
Description
Technical Field
The present invention relates to TOF depth cameras, and in particular to a depth imaging system, method, apparatus and medium based on modulated light fields.
Background
A Time of flight (TOF) depth camera obtains a depth image of a measured space by emitting a flood beam of a specific wavelength band, receiving a reflected beam of an object in the measured space with a sensor, and measuring the Time of flight of the beam in space to calculate a distance. The TOF depth camera can obtain gray level images and depth images at the same time, and is widely applied to the technical fields of gesture recognition, face recognition, 3D modeling, somatosensory games, machine vision, auxiliary focusing, security protection, automatic driving and the like related to 3D depth vision.
Conventional TOF depth cameras assume that the received light beam is reflected only once in the target scene, while there is always a specular or diffuse surface of material in the actual scene that reflects the incident light in all directions, so that the TOF sensor may receive a superposition of the once reflected light beam and the multiple reflected light beams, thereby interfering with the accuracy of the distance measurement by the TOF depth camera, an effect known as multipath interference.
The prior art mainly utilizes a multi-frequency multi-frame fusion mode to estimate the original depth of multi-path interference reconstruction. The method has the problems of high computational complexity, poor robustness and poor reconstruction accuracy due to the limitation of frame rate and frequency quantity, and has higher practical application difficulty. Therefore, how to suppress multipath interference and improve depth measurement accuracy is a problem to be solved in the aspect of practical application of TOF depth cameras.
Disclosure of Invention
In view of the drawbacks of the prior art, an object of the present invention is to provide a depth imaging system, method, apparatus and medium based on a modulated light field.
The depth imaging system based on the modulated light field provided by the invention comprises the following modules:
a light projection module for projecting floodlight or a plurality of discrete light beams toward a target object;
the light source control module is used for controlling the floodlight and the plurality of discrete light beams to be switched and projected according to a preset time period;
a TOF sensing module for receiving the plurality of discrete light beams reflected by the target object and the flood light, respectively, and generating first image data and second image data;
and the multipath interference elimination module is used for determining multipath interference components according to the first image data, and further generating a target depth image according to the first image data and the second image data after the multipath interference components are removed.
Preferably, the multipath interference cancellation module comprises the following modules:
the data acquisition module is used for acquiring first image data, wherein the first image data comprises a plurality of light spot area data and background area data;
the multipath interference determining module is used for determining the multipath interference basic quantity corresponding to each facula area data according to the background area data;
the data generation module is used for processing each light spot area data based on the multipath interference basic quantity so as to remove multipath interference components corresponding to each light spot area data and further generate target light spot area data;
and the depth image generation module is used for fusing the first depth image generated according to the target light spot area data and the second depth image generated according to the second image data to generate a target depth image.
Preferably, the first image data comprises a plurality of infrared images acquired by a TOF sensor;
each infrared image comprises a plurality of light spot areas and a background area;
the background area is a multipath interference area adjacent to the light spot area;
the multipath interference basic quantity and the multipath interference component are expressed by any physical quantity of amplitude, gray value, pixel value, illumination, luminous flux and radiation power.
Preferably, the diameter of each light spot area is two pixels;
the distance between any two adjacent spot areas is four pixels.
Preferably, the data generating module includes the following modules:
the interference area determining module is used for dividing the light spot areas into a plurality of groups of light spot areas, and each group of light spot areas corresponds to or is adjacent to at least one background area;
the interference component generating module is used for acquiring the multipath interference basic quantity of each background area;
and the interference component removing module is used for determining the multipath interference component corresponding to each light spot area according to the multipath interference basic quantity of the background area corresponding to each light spot area, and further removing the multipath interference component corresponding to each light spot area data so as to generate target light spot area data.
Preferably, the data generating module includes the following modules:
the area association module is used for determining at least one background area corresponding to or adjacent to each light spot area;
the interference acquisition module is used for acquiring the multipath interference basic quantity of each background area;
and the interference component removing module is used for determining the multipath interference component corresponding to each light spot area according to the multipath interference basic quantity of the background area corresponding to the light spot area, and further removing the multipath interference component corresponding to each light spot area data so as to generate target light spot area data.
Preferably, the depth image generating module includes the following modules:
the image acquisition module is used for acquiring a first infrared image corresponding to the first depth image and a second infrared image corresponding to the second depth image;
the confidence coefficient generation module is used for determining a first confidence coefficient based on the depth information of each pixel point in the first depth image of the first infrared image and determining a second confidence coefficient based on the depth information of each pixel point in the second depth image of the second infrared image;
the fusion coefficient generation module is used for extracting the depth information of the edge contour area of the target object from the second depth image, determining a first fusion coefficient for the depth information of the edge contour area and determining a second fusion coefficient for the depth information of the area where the target object is located;
and the image fusion module is used for fusing the pixel points screened out from the first depth image based on the first confidence coefficient, the first fusion coefficient and the second fusion coefficient and the pixel points screened out from the second depth image based on the second confidence coefficient to generate a target depth image.
The depth imaging method based on the modulated light field provided by the invention comprises the following steps:
step S1: projecting floodlight or multiple discrete beams toward a target object;
step S2: controlling the floodlight and the plurality of discrete beams to perform switching projection according to a preset time period;
step S3: receiving a plurality of discrete light beams reflected by the target object and the floodlight, thereby generating first image data and second image data;
step S4: and determining a multipath interference component according to the first image data, and further generating a target depth image according to the first image data and the second image data after the multipath interference component is removed.
The depth imaging device based on the modulated light field provided by the invention comprises:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the modulated light field based depth imaging method via execution of the executable instructions.
According to the present invention, a computer readable storage medium is provided for storing a program which, when executed, implements the steps of the modulated light field based depth imaging method.
Compared with the prior art, the invention has the following beneficial effects:
the invention can determine the multipath interference basic quantity through the background area data in the first image data acquired when the discrete light beams are projected to the target object, and further can process the light spot area data to remove multipath interference components, thereby eliminating measurement errors caused by multipath interference and realizing the output of high-precision depth images.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art. Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
FIG. 1 is a block diagram of a modulated light field based depth imaging system in accordance with an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating a use state of a depth imaging system based on a modulated light field according to an embodiment of the present invention;
fig. 3 is a schematic block diagram of a multipath interference cancellation module according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a data generation module according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a data generation module according to a modification of the present invention;
FIG. 6 is a schematic block diagram of a depth image generating module according to an embodiment of the present invention;
FIG. 7 (a) is a schematic diagram of an infrared image according to an embodiment of the present invention;
FIG. 7 (b) is a schematic diagram of another infrared image according to an embodiment of the present invention;
FIG. 8 is a flow chart of the steps of a modulated light field based depth imaging method in an embodiment of the present invention;
FIG. 9 is a schematic diagram of a depth imaging apparatus based on a modulated light field in an embodiment of the present invention; and
fig. 10 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.
In the figure:
1 is a facula area;
2 is the background area.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present invention.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
The invention provides a depth imaging method based on a modulated light field, which aims to solve the problems in the prior art.
The following describes the technical scheme of the present invention and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.
Fig. 1 is a schematic block diagram of a depth imaging system based on a modulated light field according to an embodiment of the present invention, and as shown in fig. 1, a depth imaging system 100 based on a modulated light field provided by the present invention includes the following modules:
a light projection module 101 for projecting floodlight or a plurality of discrete light beams toward a target object;
the light source control module 102 is used for controlling the floodlight and the plurality of discrete light beams to be switched and projected according to a preset time period;
a TOF sensing module 103 for receiving the plurality of discrete light beams reflected by the target object and the flood light, respectively, and generating first image data and second image data;
the multipath interference elimination module 104 is configured to determine a multipath interference component according to the first image data, and further generate a target depth image according to the first image data and the second image data after removing the multipath interference component.
FIG. 2 is a schematic diagram illustrating a use state of a depth imaging system based on a modulated light field according to an embodiment of the present invention; when the depth imaging system based on the modulated light field is used, the modulated light projection module projects discrete light beams to a target object, and the space distribution of the light field intensity and the phase of the projected discrete light beams is modulated, so that the modulated light field mode and the uniform light field mode can be switched according to the requirement, and the light source control module drives the light source to emit light rays and control the switching light field mode; the TOF sensing module receives the back measuring beam reflected by the measured target object and outputs first image data to the multipath interference elimination module; and the multipath interference elimination module is connected with the light source control module and the TOF sensing module, acquires image data in a modulated light field mode and a uniform light field, estimates multipath interference components from the image data, removes measurement errors caused by multipath interference and outputs a high-precision depth image.
In the embodiment of the invention, the modulated light field projection module can be used for carrying out switching projection of speckle light and uniform light through the cooperation of a diffuser made of laser and a nano photon chip, and can also be used for carrying out projection of the speckle light and the uniform light respectively through a speckle projector and a uniform light projector.
Fig. 3 is a schematic block diagram of a multipath interference cancellation module according to an embodiment of the present invention, as shown in fig. 3, the multipath interference cancellation module 104 includes the following modules:
a data acquisition module 1041, configured to acquire first image data, where the first image data includes a plurality of spot area data and background area data;
in an embodiment of the present invention, the first image data includes a plurality of infrared images acquired by a TOF sensor;
each of the infrared images includes a plurality of spot areas 1 and a background area 2, as shown in fig. 7 (a), 7 (b);
the background area is a multipath interference area adjacent to the light spot area. The infrared image also comprises a region except a facula region and a background region, which is a signal-free region.
The diameter of each light spot area is two pixels; the distance between any two adjacent spot areas is four pixels.
More specifically, the number of the infrared images is four infrared images acquired in four image acquisition periods, so that a first depth image is generated by calculating a time difference between optical signals according to amplitude changes among the four infrared images.
The multipath interference determining module 1042 is configured to determine a multipath interference base corresponding to each spot area data according to the background area data.
In the embodiment of the invention, the influence of each local area of the infrared image is the same or approximate, so that the determination of the multipath interference basic quantity can be performed through a background area. The multipath interference basic quantity is the multipath interference quantity of a single pixel point in a background area, and can be expressed by amplitude.
In the embodiment of the invention, the multipath interference basic quantity is expressed by amplitude, photoelectric conversion is carried out firstly for convenience in processing, the unit is changed into voltage, and the voltage is quantized by an ADC to generate a corresponding number, and the unit is LSB.
In the embodiment of the present invention, the multipath interference basic quantity may also be represented by any one of gray value, pixel value, illuminance, luminous flux and radiation power.
The data generating module 1043 is configured to process each piece of light spot area data based on the multipath interference basic quantity to remove a multipath interference component corresponding to each piece of light spot area data, thereby generating target light spot area data;
a depth image generating module 1044, configured to fuse the first depth image generated according to the target spot area data and the second depth image generated according to the second image data to generate a target depth image.
Fig. 4 is a schematic block diagram of a data generating module according to an embodiment of the present invention, as shown in fig. 4, the data generating module 1043 includes the following modules:
the interference area determining module 10431 is configured to divide the plurality of light spot areas into a plurality of groups of light spot areas, where each group of light spot areas corresponds to or is adjacent to at least one background area;
an interference component generating module 10432, configured to obtain a multipath interference base for each background area;
the interference component removing module 10433 is configured to determine a multipath interference component corresponding to each light spot area according to the multipath interference basic quantity of the background area corresponding to each light spot area, and further remove the multipath interference component corresponding to each light spot area data, so as to generate target light spot area data.
In the embodiment of the invention, the multipath interference component of each light spot area can be calculated according to the number of pixels of the light spot area and the multipath interference basic quantity, for example, when the number of pixels of the light spot area is 2, the multipath interference component of the light spot area is twice the multipath interference basic quantity.
In the embodiment of the invention, the target light spot area data can be generated according to the difference between the amplitude of the light spot area and the multipath interference component, the target light spot area data can be generated according to the difference between the illumination of the light spot area and the illumination of the multipath interference area, and the target light spot area data can be generated according to the gray value of the light spot area and the gray value of the illumination of the multipath interference area.
Fig. 5 is a schematic block diagram of a data generating module according to a modification of the present invention, as shown in fig. 5, the data generating module 1043 includes the following modules:
the area association module 10434 is configured to determine at least one background area corresponding to or adjacent to each of the spot areas;
an interference acquisition module 10435, configured to acquire a multipath interference base for each background area;
the interference component removing module 10436 is configured to determine a multipath interference component corresponding to each light spot area according to the multipath interference basic quantity of the background area corresponding to the light spot area, and further remove the multipath interference component corresponding to each light spot area data, so as to generate target light spot area data.
In a modification of the present invention, the working process of the area association module 10434 specifically includes determining each light spot area, determining each background area, and associating the light spot area with the background area.
In a modification of the present invention, the multipath interference basic quantity is determined according to a background area adjacent to each light spot area, so that the multipath interference component corresponding to each light spot area can be determined more accurately.
Fig. 6 is a schematic block diagram of a depth image generating module according to an embodiment of the present invention, as shown in fig. 6, the depth image generating module 1044 includes the following modules:
an image obtaining module 10441, configured to obtain a first infrared image corresponding to the first depth image and a second infrared image corresponding to the second depth image;
a confidence coefficient generating module 10442, configured to determine a first confidence coefficient based on the depth information of the first infrared image for each pixel point in the first depth image, and determine a second confidence coefficient based on the depth information of the second infrared image for each pixel point in the second depth image;
in the embodiment of the invention, a first confidence coefficient is determined for the depth information of each pixel point in the first depth image based on the amplitude of each region in the first infrared image, when the amplitude of one region in the first infrared image is higher, a higher confidence coefficient is given to the pixel point in the same region corresponding to the first depth image, and when the amplitude of one region in the first infrared image is lower, a lower confidence coefficient is given to the pixel point in the same region corresponding to the first depth image. Similarly, for the second depth image, when the amplitude of a region in the second infrared image is higher, a higher confidence is given to the pixel point in the same region corresponding to the second depth image, and when the amplitude of a region in the second infrared image is lower, a lower confidence is given to the pixel point in the same region corresponding to the second depth image.
More specifically, if the amplitude of a region in the first infrared image is 300, a confidence level of 3 is assigned to the pixel points in the same region corresponding to the first depth image; when the amplitude of a region in the first infrared image is 100, confidence is given to the pixel point in the same region corresponding to the first depth image, wherein the confidence is 1.
The fusion coefficient generation module 10443 is configured to extract edge contour region depth information of the target object from the second depth image, determine a first fusion coefficient for the edge contour region depth information, and determine a second fusion coefficient for the depth information of the region where the target object is located.
In the embodiment of the invention, the first fusion coefficient is greater than the second fusion coefficient, for example, the first fusion coefficient can be set to be 0.8, so that more pixels are selected for the edge contour area, and the second fusion coefficient is 0.2, so that smaller pixels are selected for the area where the target object is located, and the edge of the generated target depth map can be finer.
The image fusion module 10444 is configured to fuse the pixel point screened out in the second depth image based on the second confidence coefficient and the pixel point screened out in the first depth image based on the first confidence coefficient, the first fusion coefficient and the second fusion coefficient to generate a target depth image.
More specifically, a first screening coefficient is generated according to the second confidence coefficient and the first fusion coefficient, and a second screening coefficient is generated according to the second confidence coefficient and the second fusion coefficient; selecting the target object and a plurality of pixel points of the scene edge contour area in the second depth image according to the first screening coefficient, and selecting the area where the target object is located and a plurality of pixel points of the scene area in the second depth image according to the second screening coefficient; generating a third screening coefficient according to the first confidence coefficient, and selecting a plurality of pixel points in the first depth image according to the third screening coefficient; and selecting a plurality of pixel points to be fused according to the plurality of pixel points selected from the first depth image and the second depth image to generate the target depth image.
In the embodiment of the present invention, the first screening coefficient, the second screening coefficient and the third screening coefficient are all greater than 0 and less than 1; if the first fusion coefficient is 0.7 when the second confidence coefficient of the first region of the second depth image is 3, if the first fusion coefficient is 0.2 when the second confidence coefficient of the first region of the second depth image is 1, and the confidence coefficient range is [0,5], the first filtering coefficient can take the value of 3/5×0.7=0.42; the second filter coefficient may take on a value of 1/5 x 0.2=0.04. When the first confidence is 4, then the third filter coefficient is 4/5=0.8.
Fig. 8 is a step flowchart of a depth imaging method based on a modulated light field according to an embodiment of the present invention, as shown in fig. 8, the depth imaging method based on a modulated light field provided by the present invention includes the following steps:
step S1: projecting floodlight or multiple discrete beams toward a target object;
step S2: controlling the floodlight and the plurality of discrete beams to perform switching projection according to a preset time period;
step S3: receiving a plurality of discrete light beams reflected by the target object and the floodlight, thereby generating first image data and second image data;
step S4: and determining a multipath interference component according to the first image data, and further generating a target depth image according to the first image data and the second image data after the multipath interference component is removed.
The embodiment of the invention also provides depth imaging equipment based on the modulated light field, which comprises a processor. A memory having stored therein executable instructions of a processor. Wherein the processor is configured to perform the steps of a modulated light field based depth imaging method via execution of the executable instructions.
As described above, the embodiment can determine the multipath interference basic quantity by the background area data in the first image data acquired when the discrete light beam is projected to the target object, and further can process the light spot area data to remove the multipath interference component, thereby eliminating the measurement error caused by the multipath interference and realizing the output of the high-precision depth image.
Those skilled in the art will appreciate that the various aspects of the invention may be implemented as a system, method, or program product. Accordingly, aspects of the invention may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" platform.
Fig. 9 is a schematic structural diagram of a depth imaging apparatus based on a modulated light field in an embodiment of the present invention. An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 9. The electronic device 600 shown in fig. 9 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 9, the electronic device 600 is in the form of a general purpose computing device. Components of electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including memory unit 620 and processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code that is executable by the processing unit 610 such that the processing unit 610 performs the steps according to various exemplary embodiments of the invention described in the above-described modulated light field based depth imaging method section of this specification. For example, the processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 6201 and/or cache memory unit 6202, and may further include Read Only Memory (ROM) 6203.
The storage unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 630 may be a local bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or using any of a variety of bus architectures.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and/or any device (e.g., router, modem, etc.) that enables the electronic device 600 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 650. Also, electronic device 600 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown in fig. 9, other hardware and/or software modules may be used in connection with electronic device 600, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage platforms, and the like.
The embodiment of the invention also provides a computer readable storage medium for storing a program, and the program is executed to realize the steps of the depth imaging method based on the modulated light field. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention as described in the modulated light field based depth imaging method section of this specification when the program product is run on the terminal device.
As described above, the program of the computer-readable storage medium of this embodiment, when executed, can determine the multipath interference basic quantity by the background area data in the first image data acquired when projecting the discrete light beam to the target object, and further can process the spot area data to remove the multipath interference component, thereby being capable of eliminating the measurement error caused by the multipath interference and realizing the output of the depth image with high precision.
Fig. 10 is a schematic structural view of a computer-readable storage medium in an embodiment of the present invention. Referring to fig. 10, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In the embodiment of the invention, the multipath interference basic quantity can be determined through the background area data in the first image data acquired when the discrete light beams are projected to the target object, and then the light spot area data can be processed to remove multipath interference components, so that measurement errors caused by multipath interference can be eliminated, and the output of a high-precision depth image can be realized.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing describes specific embodiments of the present invention. It is to be understood that the invention is not limited to the particular embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the claims without affecting the spirit of the invention.
Claims (10)
1. A modulated light field based depth imaging system comprising the following modules:
the light projection module is used for projecting discrete light beams to a target object and switching a modulated light field mode and a uniform light field mode through modulation;
the light source control module is used for emitting light rays and controlling switching light field modes;
the TOF sensing module is used for receiving the back measuring beam reflected by the measured target object;
and the multipath interference elimination module is used for acquiring the modulated light field mode and the image data under the uniform light field, estimating multipath interference components from the modulated light field mode and the image data, removing measurement errors caused by multipath interference and outputting a high-precision depth image.
2. The modulated light field based depth imaging system of claim 1, wherein the multipath interference cancellation module comprises the following modules:
the data acquisition module is used for acquiring first image data, wherein the first image data comprises a plurality of light spot area data and background area data; wherein the first image data is generated from a reflected signal in the modulated light field mode;
the multipath interference determining module is used for determining the multipath interference basic quantity corresponding to each facula area data according to the background area data;
the data generation module is used for processing each light spot area data based on the multipath interference basic quantity so as to remove multipath interference components corresponding to each light spot area data and further generate target light spot area data;
the depth image generation module is used for fusing a first depth image generated according to the target light spot area data and a second depth image generated according to the second image data to generate a target depth image; wherein the second image data is generated from the reflected signal in the uniform light field mode.
3. The modulated light field based depth imaging system of claim 2, wherein the first image data comprises a plurality of infrared images acquired by a TOF sensor;
each infrared image comprises a plurality of light spot areas and a background area;
the background area is a multipath interference area adjacent to the light spot area;
the multipath interference basic quantity and the multipath interference component are expressed by any physical quantity of amplitude, gray value, pixel value, illumination, luminous flux and radiation power.
4. A modulated light field based depth imaging system according to claim 3, wherein each of said spot areas has a diameter of two pixels;
the distance between any two adjacent spot areas is four pixels.
5. A modulated light field based depth imaging system according to claim 3, wherein the data generation module comprises the following modules:
the interference area determining module is used for dividing the light spot areas into a plurality of groups of light spot areas, and each group of light spot areas corresponds to or is adjacent to at least one background area;
the interference component generating module is used for acquiring the multipath interference basic quantity of each background area;
and the interference component removing module is used for determining the multipath interference component corresponding to each light spot area according to the multipath interference basic quantity of the background area corresponding to each light spot area, and further removing the multipath interference component corresponding to each light spot area data so as to generate target light spot area data.
6. A modulated light field based depth imaging system according to claim 3, wherein the data generation module comprises the following modules:
the area association module is used for determining at least one background area corresponding to or adjacent to each light spot area;
the interference acquisition module is used for acquiring the multipath interference basic quantity of each background area;
and the interference component removing module is used for determining the multipath interference component corresponding to each light spot area according to the multipath interference basic quantity of the background area corresponding to the light spot area, and further removing the multipath interference component corresponding to each light spot area data so as to generate target light spot area data.
7. The modulated light field based depth imaging system of claim 2, wherein the depth image generation module comprises the following modules:
the image acquisition module is used for acquiring a first infrared image corresponding to the first depth image and a second infrared image corresponding to the second depth image;
the confidence coefficient generation module is used for determining a first confidence coefficient based on the depth information of each pixel point in the first depth image of the first infrared image and determining a second confidence coefficient based on the depth information of each pixel point in the second depth image of the second infrared image;
the fusion coefficient generation module is used for extracting the depth information of the edge contour area of the target object from the second depth image, determining a first fusion coefficient for the depth information of the edge contour area and determining a second fusion coefficient for the depth information of the area where the target object is located;
and the image fusion module is used for fusing the pixel points screened out from the first depth image based on the first confidence coefficient, the first fusion coefficient and the second fusion coefficient and the pixel points screened out from the second depth image based on the second confidence coefficient to generate a target depth image.
8. A depth imaging method based on a modulated light field, comprising the steps of:
step S1: projecting discrete light beams to a target object, and switching a modulated light field mode and a uniform light field mode through modulation;
step S2: controlling to switch the light field mode;
step S3: receiving a reflected measuring beam reflected by a measured target object;
step S4: and acquiring the modulated light field mode and the image data under the uniform light field, estimating multipath interference components from the modulated light field mode and the image data, removing measurement errors caused by multipath interference, and outputting a high-precision depth image.
9. A modulated light field based depth imaging apparatus comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the modulated light field based depth imaging method of claim 8 via execution of the executable instructions.
10. A computer readable storage medium storing a program, characterized in that the program when executed implements the steps of the modulated light field based depth imaging method of claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310043248.5A CN116520348A (en) | 2019-11-20 | 2019-11-21 | Depth imaging system, method, equipment and medium based on modulated light field |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911142394.3A CN112824935B (en) | 2019-11-20 | 2019-11-20 | Depth imaging system, method, device and medium based on modulated light field |
| CN202310043248.5A CN116520348A (en) | 2019-11-20 | 2019-11-21 | Depth imaging system, method, equipment and medium based on modulated light field |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911142394.3A Division CN112824935B (en) | 2019-11-20 | 2019-11-20 | Depth imaging system, method, device and medium based on modulated light field |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116520348A true CN116520348A (en) | 2023-08-01 |
Family
ID=75906995
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911142394.3A Active CN112824935B (en) | 2019-11-20 | 2019-11-20 | Depth imaging system, method, device and medium based on modulated light field |
| CN202310043248.5A Pending CN116520348A (en) | 2019-11-20 | 2019-11-21 | Depth imaging system, method, equipment and medium based on modulated light field |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911142394.3A Active CN112824935B (en) | 2019-11-20 | 2019-11-20 | Depth imaging system, method, device and medium based on modulated light field |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN112824935B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113542534A (en) * | 2021-09-17 | 2021-10-22 | 珠海视熙科技有限公司 | TOF camera control method and device and storage medium |
| CN113945951B (en) * | 2021-10-21 | 2022-07-08 | 浙江大学 | Multipath interference suppression method in TOF (time of flight) depth calculation, TOF depth calculation method and device |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9041914B2 (en) * | 2013-03-15 | 2015-05-26 | Faro Technologies, Inc. | Three-dimensional coordinate scanner and method of operation |
| US9778363B2 (en) * | 2013-10-24 | 2017-10-03 | Massachusetts Institute Of Technology | Methods and apparatus for coded time-of-flight camera |
| US9874638B2 (en) * | 2014-03-06 | 2018-01-23 | University Of Waikato | Time of flight camera system which resolves direct and multi-path radiation components |
| EP2955544B1 (en) * | 2014-06-11 | 2020-06-17 | Sony Depthsensing Solutions N.V. | A TOF camera system and a method for measuring a distance with the system |
| US20180227566A1 (en) * | 2017-02-06 | 2018-08-09 | Microsoft Technology Licensing, Llc | Variable field of view and directional sensors for mobile machine vision applications |
| US10628950B2 (en) * | 2017-03-01 | 2020-04-21 | Microsoft Technology Licensing, Llc | Multi-spectrum illumination-and-sensor module for head tracking, gesture recognition and spatial mapping |
| US10445893B2 (en) * | 2017-03-10 | 2019-10-15 | Microsoft Technology Licensing, Llc | Dot-based time of flight |
| CN109831660B (en) * | 2019-02-18 | 2021-04-23 | Oppo广东移动通信有限公司 | Depth image acquisition method, depth image acquisition module and electronic equipment |
| CN110244318B (en) * | 2019-04-30 | 2021-08-17 | 深圳市光鉴科技有限公司 | 3D imaging method based on asynchronous ToF discrete point cloud |
| CN110441784A (en) * | 2019-08-27 | 2019-11-12 | 浙江舜宇光学有限公司 | Depth image imaging system and method |
-
2019
- 2019-11-20 CN CN201911142394.3A patent/CN112824935B/en active Active
- 2019-11-21 CN CN202310043248.5A patent/CN116520348A/en active Pending
Also Published As
| Publication number | Publication date |
|---|---|
| CN112824935A (en) | 2021-05-21 |
| CN112824935B (en) | 2023-02-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11625845B2 (en) | Depth measurement assembly with a structured light source and a time of flight camera | |
| EP2976878B1 (en) | Method and apparatus for superpixel modulation | |
| CN112824934B (en) | TOF multipath interference removal method, system, equipment and medium based on modulated light field | |
| US20170059305A1 (en) | Active illumination for enhanced depth map generation | |
| US10935371B2 (en) | Three-dimensional triangulational scanner with background light cancellation | |
| US10255682B2 (en) | Image detection system using differences in illumination conditions | |
| KR20120071970A (en) | 3d image acquisition apparatus and method of extractig depth information in the 3d image acquisition apparatus | |
| KR20190053799A (en) | Method for processing a raw image of a time-of-flight camera, image processing apparatus and computer program | |
| KR20130091194A (en) | 3d image acquisition apparatus and method of extractig depth information in the 3d image acquisition apparatus | |
| US9258548B2 (en) | Apparatus and method for generating depth image | |
| JP6104662B2 (en) | Measuring device, method and program | |
| CN112824935B (en) | Depth imaging system, method, device and medium based on modulated light field | |
| JP2015184056A (en) | Measurement device, method, and program | |
| CN112446843A (en) | Image reconstruction method, system, device and medium based on multiple depth maps | |
| CN112513670B (en) | Distance meter, distance measuring system, distance measuring method and program | |
| CN109100740B (en) | Three-dimensional image imaging device, imaging method and system | |
| US20220329716A1 (en) | Method for determining one or more groups of exposure settings to use in a 3d image acquisition process | |
| US11037316B2 (en) | Parallax calculation apparatus, parallax calculation method, and control program of parallax calculation apparatus | |
| CN116670713A (en) | Method for depth sensing using candidate images selected based on epipolar lines | |
| CN113327317A (en) | Three-dimensional point cloud picture acquisition method and device, electronic equipment and storage medium | |
| CN114697521A (en) | TOF camera motion blur detection method, system, equipment and storage medium | |
| US20200320725A1 (en) | Light projection systems | |
| CN113447954B (en) | Scene depth measurement method, system, equipment and storage medium | |
| CN115250316A (en) | TOF mirror surface multipath removal method, system, equipment and medium based on modulated light field | |
| US20230258810A1 (en) | Enhancing depth estimation with brightness image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |