CN112509023A - Multi-source camera system and RGBD registration method - Google Patents
Multi-source camera system and RGBD registration method Download PDFInfo
- Publication number
- CN112509023A CN112509023A CN202011461732.2A CN202011461732A CN112509023A CN 112509023 A CN112509023 A CN 112509023A CN 202011461732 A CN202011461732 A CN 202011461732A CN 112509023 A CN112509023 A CN 112509023A
- Authority
- CN
- China
- Prior art keywords
- depth image
- rgb
- acquisition module
- depth
- image acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20104—Interactive definition of region of interest [ROI]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Measurement Of Optical Distance (AREA)
- Studio Devices (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention relates to a multi-source camera system and an RGBD (red, green and blue) registration method, wherein the system comprises an RGB (red, green and blue) image acquisition module, a depth image acquisition module and an upper computer, wherein the depth image acquisition module is connected with the upper computer; the method comprises the steps of collecting RGB image data and depth image data of a plurality of groups of calibration plates; obtaining internal and external parameters of the RGB camera; initializing the internal and external parameters of the obtained RGB image as input; converting the depth image into a 2D plane, selecting an ROI (region of interest) to obtain a first external reference matrix; optimizing the first external parameter matrix to obtain an external parameter matrix of a second stage; and inputting an RGB image and a depth image which need to be registered, and obtaining a rendered RGBD image by using a second external parameter matrix. The invention has the advantages that: the use is flexible; the depth image acquisition module has a simple structure and low cost, and can quickly acquire a depth image; the registration speed of the RGB image and the depth image is high, and the accuracy is high.
Description
Technical Field
The invention relates to the field of image acquisition, in particular to a multi-source camera system and an RGBD registration method.
Background
With the wide application of emerging technologies such as robot navigation, automobile automatic driving, virtual reality and the like, the advantage of depth information imaging is increasingly prominent. Conventional two-dimensional imaging maps a stereoscopic scene to a two-dimensional plane through perspective projection transformation, and such low-dimensional mapping causes depth information of the scene to be lost. The traditional infrared depth image can cause color information of a scene to be lost, and no matter the single traditional two-dimensional imaging image or the single infrared depth image cannot meet the requirements of the fields of three-dimensional reconstruction, augmented reality, virtual reality, human-computer interaction and the like on RGB information and depth information, so that RGBD data combining depth information and RGB image information are indispensable, and the problem of accurate calibration between a depth sensor and an RGB sensor needs to be solved firstly when the RGBD data is acquired.
Disclosure of Invention
The invention mainly solves the problems of complex structure and low registration precision of depth information and RGB information of the existing RGBD imaging device, and provides a multisource camera system and an RGBD registration method which are flexible to use, simple in structure and high in registration precision of depth information and RGB information.
The technical scheme adopted by the invention for solving the technical problem is that the multi-source camera system comprises an RGB image acquisition module, a depth image acquisition module and an upper computer, wherein the RGB image acquisition module comprises an RGB sensor unit and a first data transmission unit which are connected, the depth image acquisition module comprises a TOF sensor unit, an active illumination unit, a processor and a second data sensor unit, the processor is respectively connected with the TOF sensor unit, the active illumination unit and the second data sensor unit, the depth image acquisition module is connected with the upper computer through the second data transmission unit, and the RGB image acquisition module is connected with the upper computer through the first data transmission unit.
The RGB image acquisition module is used for acquiring RGB images; the depth image acquisition module adopts an active depth measurement method and is used for acquiring depth phase data; the upper computer is used for sending control instructions to each acquisition module and finishing the calculation of depth images, the calculation of image registration and the data storage.
As a preferable scheme of the above scheme, the active illumination unit includes a TOF lens, an illumination board, a sensor board and a main control board, the illumination board, the sensor board and the main control board are in communication with each other, the TOF lens and the active illumination unit are disposed on the illumination board, the TOF sensor unit is disposed on the sensor board, and the processor is disposed on the main control board.
As a preferable scheme of the above scheme, the TOF lens is provided with an infrared narrow band-pass filtering optical sheet. The active illumination unit emits a beam of modulated infrared light, after diffuse reflection is carried out on the surface of a target, reflected light passes through a lens of an integrated infrared narrow band-pass filter with a fixed focal length and is finally captured by the TOF sensor unit, and the distance between the target and the depth camera is calculated by obtaining the time difference or the phase difference between the emitted light and the reflected light.
As a preferable scheme of the above scheme, the active illumination unit includes a plurality of infrared light sources, the infrared light sources are disposed on the illumination plate, and the light emitting direction of the infrared light sources is the same as the direction of the TOF lens.
Correspondingly, the invention also provides a multi-source camera RGBD registration method, which is used for the multi-source camera system and comprises the following steps:
s1: simultaneously collecting RGB image data and depth image data of a plurality of groups of calibration plates;
s2: calibrating by Zhangyingyou to obtain internal and external parameters of the RGB camera;
s3: initializing the internal and external parameters of the obtained RGB image as input;
s4: converting the depth image into a 2D plane, selecting an ROI (region of interest) to obtain a first external reference matrix;
s5: optimizing the first external parameter matrix to obtain an external parameter matrix of a second stage;
s6: and inputting an RGB image and a depth image which need to be registered, and obtaining a rendered RGBD image by using a second external parameter matrix.
As a preferable mode of the above, the step S4 includes the steps of:
s41: converting the depth value of the input depth image to a given set of colors, and displaying a two-dimensional plane map with colors;
s42: selecting a depth image distance range needing to be registered, and filtering out pixel points beyond the range;
s43: selecting a region of interest by using a marking tool, and calculating a selected error value;
s44: and calculating all the calibrated interested areas to obtain an internal and external parameter matrix of the depth image, namely a first external parameter matrix.
As a preferable solution of the foregoing solution, in step S5, the first external reference matrix is optimized by using robust total least squares to obtain the second external reference matrix.
As a preferable scheme of the above scheme, the depth image is acquired by the following steps:
s01: the active illumination unit emits continuous wave modulated infrared light;
s02: the TOF sensor unit receives light beams diffusely reflected by the surface of a target through a TOF lens and an infrared narrow band-pass filter;
s03: the TOF sensor unit obtains raw depth phase data of the scene by calculating the time or phase difference of the emitted and reflected light.
The invention has the advantages that: the RGB image acquisition module and the depth image acquisition module are not limited, and a user can select RGB cameras with different resolutions and depth cameras with different detection distances according to a specific application scene, so that the use is flexible; the depth image acquisition module has a simple structure and low cost, and can quickly acquire a depth image; the registration speed of the RGB image and the depth image is high, and the accuracy is high.
Drawings
Fig. 1 is a block diagram of a multi-source camera system in an embodiment.
Fig. 2 is a block diagram of a depth image acquisition module in the embodiment.
Fig. 3 is a schematic flow chart of an RGBD registration method of the multi-source camera in the embodiment.
Fig. 4 is a schematic flowchart of acquiring a depth image according to an embodiment.
FIG. 5 is a schematic flow chart of obtaining the first external reference matrix in the embodiment.
1-RGB image acquisition module 2-depth image acquisition module 3-upper computer 11-RGB sensor unit 12-first data transmission unit 21-TOF sensor unit 22-active illumination unit 23-processor 24-second data sensor unit 601-illumination board 602-sensor board 603-main control board 701-infrared VSCEL light source 606-TOF lens 607-infrared narrow bandpass filter.
Detailed Description
The technical solution of the present invention is further described below by way of examples with reference to the accompanying drawings.
Example (b):
the embodiment provides a multisource camera system, as shown in fig. 1, including RGB image acquisition module 1, depth image acquisition module 2 and host computer 3, RGB image acquisition module is including continuous RGB sensor unit 11 and first data transmission unit 12, depth image acquisition module includes TOF sensor unit 21, initiative lighting unit 22, treater 23 and second data sensor unit 24, the treater respectively with TOF sensor unit, initiative lighting unit and second data sensor unit link to each other, depth image acquisition module passes through the second data transmission unit and links to each other with the host computer, RGB image acquisition module passes through first data transmission unit and links to each other with the host computer.
The RGB image acquisition module adopts an industrial RGB image sensor of Basler company, the model is AcA1300, the pixel size is 4.8 mu m multiplied by 4.8 mu m, the size of a photosensitive chip is 6.1mm multiplied by 4.9mm, the RGB image acquisition module is matched with a 5mm lens 605 of computer company, data transmission passes through a USB3.0 interface, and the frame rate is 200 fps.
The depth image acquisition module is as shown in fig. 2, adopt three-layer PCB panel design, lighting panel 601 respectively, sensor board 602, main control board 603, TOF camera lens and initiative lighting unit set up on lighting panel, TOF sensor unit sets up on the sensor board, the processor sets up on main control board, transmit through MIPI data format between each panel, the initiative lighting unit includes the infrared VSCEL light source 701 of 4 setting 850nm wave bands on lighting panel 601, the light-emitting direction of infrared VSCEL light source is the same with TOF camera lens direction, TOF camera lens 606 of fixed focus has integrateed the infrared narrow bandpass filter 607 of 850nm wave band, TOF camera lens 606 covers TOF sensor unit's photosurface completely simultaneously.
Correspondingly, the embodiment also provides a multisource camera RGBD registration method, as shown in fig. 3, including the following steps:
s1: simultaneously collecting RGB image data and depth image data of a plurality of groups of calibration plates, wherein the calibration plates are standard checkerboard plates, and when collecting depth images, as shown in figure 4, a depth image collecting module executes the following steps:
s01: the active illumination unit emits continuous wave modulated infrared light;
s02: the TOF sensor unit receives the light beam diffusely reflected by the surface of the target through the TOF lens and the infrared narrow band-pass filter;
s03: the TOF sensor unit obtains raw depth phase data of the scene by calculating the time or phase difference of the emitted and reflected light.
S2: calibrating by Zhangyingyou to obtain internal and external parameters of the RGB camera;
s3: initializing the internal and external parameters of the obtained RGB image as input;
s4: converting the depth image into a 2D plane, selecting an ROI (region of interest) to obtain a first external reference matrix; the specific steps are shown in fig. 5, and include:
s41: converting the depth value of the input depth image to a given set of colors, and displaying a two-dimensional plane map with colors;
s42: selecting a depth image distance range needing to be registered, and filtering out pixel points beyond the range;
s43: selecting a region of interest by using a marking tool, and calculating a selected error value;
s44: and calculating all the calibrated interested areas to obtain an internal and external parameter matrix of the depth image, namely a first external parameter matrix.
S5: optimizing the first external parameter matrix to obtain a second external parameter matrix, wherein the first external parameter matrix mainly solves basic rotation and translation matrixes according to internal and external parameters of RGB and internal and external parameters of the depth image, and for each selected area in the depth image, the first external parameter matrix is optimized by utilizing a robust total least square estimation to obtain the second external parameter matrix;
s6: inputting an RGB image and a depth image which need to be registered, and obtaining a rendered RGBD image by using a second external parameter matrix, wherein the method specifically comprises the following steps:
s61: the method comprises the steps of keeping the space relation between an RGB image acquisition module and a depth image acquisition module stable, and acquiring an RGB image and a depth image of a scene;
s62: inputting the collected data into a registration code, loading a second external parameter matrix, and performing registration;
s63: the rendered depth image, i.e., RGBD image, is output in the wrl file format.
The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.
Claims (8)
1. A multi-source camera system is characterized in that: including RGB image acquisition module, degree of depth image acquisition module and host computer, RGB image acquisition module is including the RGB sensor unit and the first data transmission unit that link to each other, degree of depth image acquisition module includes TOF sensor unit, initiative lighting unit, treater and second data sensor unit, the treater links to each other with TOF sensor unit, initiative lighting unit and second data sensor unit respectively, degree of depth image acquisition module passes through the second data transmission unit and links to each other with the host computer, RGB image acquisition module passes through first data transmission unit and links to each other with the host computer.
2. The multi-source camera system of claim 1, wherein: the active lighting unit comprises a TOF lens, a lighting plate, a sensor plate and a main control plate, wherein the lighting plate, the sensor plate and the main control plate are communicated with each other, the TOF lens and the active lighting unit are arranged on the lighting plate, the TOF sensor unit is arranged on the sensor plate, and the processor is arranged on the main control plate.
3. The multi-source camera system of claim 2, wherein: and an infrared narrow band-pass filtering optical sheet is arranged on the TOF lens.
4. The multi-source camera system of claim 2, wherein: the active lighting unit comprises a plurality of infrared light sources, the infrared light sources are arranged on the lighting plate, and the light emitting directions of the infrared light sources are the same as the direction of the TOF lens.
5. A multi-source camera RGBD registration method for the multi-source camera system of any one of claims 1-4, characterized by: the method comprises the following steps:
s1: simultaneously collecting RGB image data and depth image data of a plurality of groups of calibration plates;
s2: calibrating by Zhangyingyou to obtain internal and external parameters of the RGB camera;
s3: initializing the internal and external parameters of the obtained RGB image as input;
s4: converting the depth image into a 2D plane, selecting an ROI (region of interest) to obtain a first external reference matrix;
s5: optimizing the first external parameter matrix to obtain an external parameter matrix of a second stage;
s6: and inputting an RGB image and a depth image which need to be registered, and obtaining a rendered RGBD image by using a second external parameter matrix.
6. The multi-source camera RGBD registration method of claim 5, wherein: the step S4 includes the steps of:
s41: converting the depth value of the input depth image to a given set of colors, and displaying a two-dimensional plane map with colors;
s42: selecting a depth image distance range needing to be registered, and filtering out pixel points beyond the range;
s43: selecting a region of interest by using a marking tool, and calculating a selected error value;
s44: and calculating all the calibrated interested areas to obtain an internal and external parameter matrix of the depth image, namely a first external parameter matrix.
7. The method of claim 6, wherein the method comprises the following steps: in step S5, the first external parameter matrix is optimized by using robust total least squares to obtain a second external parameter matrix.
8. The multi-source camera RGBD registration method of claim 5, wherein: the depth image is acquired by the following steps:
s01: the active illumination unit emits continuous wave modulated infrared light;
s02: the TOF sensor unit receives light beams diffusely reflected by the surface of a target through a TOF lens and an infrared narrow band-pass filter;
s03: the TOF sensor unit obtains raw depth phase data of the scene by calculating the time or phase difference of the emitted and reflected light.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011461732.2A CN112509023B (en) | 2020-12-11 | 2020-12-11 | Multi-source camera system and RGBD registration method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011461732.2A CN112509023B (en) | 2020-12-11 | 2020-12-11 | Multi-source camera system and RGBD registration method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112509023A true CN112509023A (en) | 2021-03-16 |
CN112509023B CN112509023B (en) | 2022-11-22 |
Family
ID=74972543
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202011461732.2A Active CN112509023B (en) | 2020-12-11 | 2020-12-11 | Multi-source camera system and RGBD registration method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112509023B (en) |
Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060221250A1 (en) * | 2004-01-28 | 2006-10-05 | Canesta, Inc. | Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing |
CN106682663A (en) * | 2015-11-11 | 2017-05-17 | 天津中科智能识别产业技术研究院有限公司 | Multi-modal biological feature identifying system and method for mobile platform |
CN107507235A (en) * | 2017-08-31 | 2017-12-22 | 山东大学 | A kind of method for registering of coloured image and depth image based on the collection of RGB D equipment |
CN207396939U (en) * | 2017-11-20 | 2018-05-22 | 江西服装学院 | A kind of animation filming apparatus |
CN109360230A (en) * | 2018-11-08 | 2019-02-19 | 武汉库柏特科技有限公司 | A kind of method for registering images and system based on 2D camera Yu 3D camera |
CN109741405A (en) * | 2019-01-21 | 2019-05-10 | 同济大学 | A kind of depth information acquisition system based on dual structure light RGB-D camera |
CN109816731A (en) * | 2017-11-21 | 2019-05-28 | 西安交通大学 | A kind of method of RGB and depth information accuracy registration |
CN110012280A (en) * | 2019-03-22 | 2019-07-12 | 盎锐(上海)信息科技有限公司 | TOF mould group and VSLAM calculation method for VSLAM system |
CN110619617A (en) * | 2019-09-27 | 2019-12-27 | 中国科学院长春光学精密机械与物理研究所 | Three-dimensional imaging method, device, equipment and computer readable storage medium |
CN111427048A (en) * | 2020-02-25 | 2020-07-17 | 深圳奥比中光科技有限公司 | ToF depth measuring device, method for controlling ToF depth measuring device and electronic equipment |
-
2020
- 2020-12-11 CN CN202011461732.2A patent/CN112509023B/en active Active
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060221250A1 (en) * | 2004-01-28 | 2006-10-05 | Canesta, Inc. | Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing |
CN106682663A (en) * | 2015-11-11 | 2017-05-17 | 天津中科智能识别产业技术研究院有限公司 | Multi-modal biological feature identifying system and method for mobile platform |
CN107507235A (en) * | 2017-08-31 | 2017-12-22 | 山东大学 | A kind of method for registering of coloured image and depth image based on the collection of RGB D equipment |
CN207396939U (en) * | 2017-11-20 | 2018-05-22 | 江西服装学院 | A kind of animation filming apparatus |
CN109816731A (en) * | 2017-11-21 | 2019-05-28 | 西安交通大学 | A kind of method of RGB and depth information accuracy registration |
CN109360230A (en) * | 2018-11-08 | 2019-02-19 | 武汉库柏特科技有限公司 | A kind of method for registering images and system based on 2D camera Yu 3D camera |
CN109741405A (en) * | 2019-01-21 | 2019-05-10 | 同济大学 | A kind of depth information acquisition system based on dual structure light RGB-D camera |
CN110012280A (en) * | 2019-03-22 | 2019-07-12 | 盎锐(上海)信息科技有限公司 | TOF mould group and VSLAM calculation method for VSLAM system |
CN110619617A (en) * | 2019-09-27 | 2019-12-27 | 中国科学院长春光学精密机械与物理研究所 | Three-dimensional imaging method, device, equipment and computer readable storage medium |
CN111427048A (en) * | 2020-02-25 | 2020-07-17 | 深圳奥比中光科技有限公司 | ToF depth measuring device, method for controlling ToF depth measuring device and electronic equipment |
Non-Patent Citations (2)
Title |
---|
FILIPPO BASSO, ET.AL: "Robust intrinsic and extrinsic calibration of RGB-D cameras", 《IEEE TRANSACTIONS ON ROBOTICS》 * |
徐光禹等: "多源数据融合技术在古建筑三维重建中的应用", 《测绘通报》 * |
Also Published As
Publication number | Publication date |
---|---|
CN112509023B (en) | 2022-11-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109615652B (en) | Depth information acquisition method and device | |
US12010431B2 (en) | Systems and methods for multi-camera placement | |
US10582121B2 (en) | System and method for fusing outputs of sensors having different resolutions | |
EP3364643B1 (en) | Method for obtaining combined depth image, and depth camera | |
EP3531066B1 (en) | Three-dimensional scanning method including a plurality of lasers with different wavelengths, and scanner | |
CN1238691C (en) | Combined stereovision, color 3D digitizing and motion capture system | |
KR102015606B1 (en) | Multi-line laser array three-dimensional scanning system and multi-line laser array three-dimensional scanning method | |
US11397475B2 (en) | Display device, electronic device and method for driving display device | |
CN102438111A (en) | Three-dimensional measurement chip and system based on double-array image sensor | |
CN105004324A (en) | Monocular vision sensor with triangulation ranging function | |
CN113034567A (en) | Depth truth value acquisition method, device and system and depth camera | |
CN102316355A (en) | Generation method of 3D machine vision signal and 3D machine vision sensor | |
DE102017128369A1 (en) | DEVICE AND METHOD FOR LOCATING A FIRST COMPONENT, LOCALIZATION DEVICE AND METHOD OF LOCATING | |
EP4314707A2 (en) | Configuration method for the display of a building information model | |
CN215810705U (en) | Three-dimensional scanning system | |
US20220180541A1 (en) | Three-dimensional coordinate scanner | |
CN112509023B (en) | Multi-source camera system and RGBD registration method | |
CN111654626B (en) | High-resolution camera containing depth information | |
CN113406604A (en) | Device and method for calibrating positions of laser radar and camera | |
CN110012280B (en) | TOF module for VSLAM system and VSLAM calculation method | |
CN111866490A (en) | Depth image imaging system and method | |
JP7414090B2 (en) | Imaging device and method of controlling the imaging device | |
CN115601275A (en) | Point cloud augmentation method and device, computer readable storage medium and terminal equipment | |
CN112257535B (en) | Three-dimensional matching equipment and method for avoiding object | |
CN113052884A (en) | Information processing method, information processing apparatus, storage medium, and electronic device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |