CN108267097B - Three-dimensional reconstruction method and device based on binocular three-dimensional scanning system - Google Patents
Three-dimensional reconstruction method and device based on binocular three-dimensional scanning system Download PDFInfo
- Publication number
- CN108267097B CN108267097B CN201710581732.8A CN201710581732A CN108267097B CN 108267097 B CN108267097 B CN 108267097B CN 201710581732 A CN201710581732 A CN 201710581732A CN 108267097 B CN108267097 B CN 108267097B
- Authority
- CN
- China
- Prior art keywords
- image
- camera
- dimensional
- stripe
- stripes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
- G01B11/254—Projection of a pattern, viewing through a pattern, e.g. moiré
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a binocular three-dimensional scanning system-based three-dimensional reconstruction method and device. The method comprises the following steps: projecting the fringe image onto a measured object by using projection equipment to generate a modulated fringe image; acquiring the modulated stripe image by using a first camera to obtain a first image, acquiring the modulated stripe image by using a second camera to obtain a second image, and acquiring a depth map of the measured object by using an invisible structured light scanning module; matching the stripes in the first image with the stripes in the second image according to the depth map; searching the corresponding relation of single points in the matched corresponding stripe center line segment to obtain a corresponding point pair; and reconstructing the corresponding point pairs into three-dimensional point cloud data. The invention solves the technical problem that the matching error rate of corresponding points is increased along with the increase of the number of stripes in the conventional handheld stripe binocular three-dimensional scanning technology.
Description
Technical Field
The invention relates to the field of three-dimensional scanning, in particular to a three-dimensional reconstruction method and a three-dimensional reconstruction device based on a binocular three-dimensional scanning system.
Background
The principle of the existing hand-held three-dimensional scanner is mainly based on an active stereoscopic vision mode of structured light, and the mode of the structured light can be various, such as infrared laser speckle, D L P projection speckle, simulated laser stripe projected by D L P, laser stripe and the like, the simulated laser stripe projected by D L P in the structured light modes has the highest precision and the best scanning details, and the basic working flow taking the simulated laser stripe projected by D L P and the laser stripe as the structured light is as follows:
(1) performing plane fitting on the projected stripes;
(2) extracting a mark point and a fringe center according to the acquired fringe pattern;
(3) carrying out connected domain segmentation on the centers of the stripes, and carrying out corresponding point matching on the stripes on the left camera image and the right camera image according to a plane equation;
(4) searching the centers of the corresponding mark points on the images of the left camera and the right camera by utilizing the polar constraint relation of the two cameras;
(5) and performing three-dimensional reconstruction on the matched corresponding stripes and the centers of the corresponding mark points by adopting a three-dimensional reconstruction algorithm according to the calibration parameters of the scanning system.
In the process, the matching of the corresponding stripes on the left camera image and the right camera image is mainly guided based on a stripe plane equation, when the number of the stripes is more than 15, the matching error rate of the corresponding stripes on the left camera image and the right camera image is obviously improved, and further, the noise is increased and the accuracy of scanning data is reduced. When the number of stripes is less than 15, the scanning efficiency is not effectively improved. An effective way to increase the scanning efficiency under the inherent limitations of the scanning frame rate is to increase the number of fringes while increasing the accuracy of the fringe matching.
However, in the existing handheld multi-stripe binocular three-dimensional scanning technology, the matching error rate of corresponding points is increased along with the increase of the number of stripes in the scanning process, so that the number of scanned data miscellaneous points is increased; before scanning, the light plane needs to be calibrated, so that the requirements on the equipment installation precision and stability of the system are more strict; in addition, the searching complexity of the stripes corresponding to the left image and the right image is increased rapidly along with the increase of the number of the stripes; and, the number of stripes is limited, and the full range of the camera field of view cannot be fully utilized, so that the scanning efficiency cannot be improved.
Aiming at the problem that the matching error rate of corresponding points is increased along with the increase of the number of stripes in the existing handheld stripe binocular three-dimensional scanning technology, an effective solution is not provided at present.
Disclosure of Invention
The embodiment of the invention provides a binocular three-dimensional scanning system-based three-dimensional reconstruction method and device, which at least solve the technical problem that the matching error rate of corresponding points is increased along with the increase of the number of stripes in the conventional handheld stripe binocular three-dimensional scanning technology.
According to an aspect of an embodiment of the present invention, there is provided a three-dimensional reconstruction method based on a binocular three-dimensional scanning system, the binocular three-dimensional scanning system including: invisible structured light scanning module, first camera, second camera, projection equipment, wherein, the method includes: projecting the fringe image onto a measured object by using the projection equipment to generate a modulated fringe image; acquiring the modulated stripe image by using the first camera to obtain a first image, acquiring the modulated stripe image by using the second camera to obtain a second image, and acquiring the depth map of the measured object by using the invisible structured light scanning module; matching the stripes in the first image with the stripes in the second image according to the depth map; searching the corresponding relation of single points in the matched corresponding stripe center line segment to obtain a corresponding point pair; and reconstructing the corresponding point pairs into three-dimensional point cloud data.
Further, matching the stripes in the first image with the stripes in the second image according to the depth map comprises: converting the depth map into a first three-dimensional point cloud coordinate under the invisible structured light scanning module coordinate system; converting the first three-dimensional point cloud coordinate into a second three-dimensional point cloud coordinate under the first camera coordinate system according to a rotation translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera; projecting the second three-dimensional point cloud coordinates into the first image and the second image respectively; and determining stripes in the second image corresponding to the stripes in the first image according to the preset corresponding relation of point coordinates in the first image and the second image.
Further, determining the stripe in the second image corresponding to the stripe in the first image according to the predetermined correspondence of the point coordinates in the first image and the second image comprises: determining a target serial number corresponding to a point in a target stripe in the first image; searching a point corresponding to the target sequence number in the second image according to a preset sequence number lookup table, wherein the preset sequence number lookup table is used for indicating the corresponding relation of point coordinates in the first image and the second image; and determining the stripe where the point corresponding to the target sequence number in the second image is located as the stripe matched with the target stripe.
Further, before projecting the fringe image onto the object to be measured by using the projection device to generate the modulated fringe image, the method further comprises: and calibrating the binocular three-dimensional scanning system to obtain the structural parameters of the binocular three-dimensional scanning system.
Further, calibrating the binocular three-dimensional scanning system, and acquiring the structural parameters of the binocular three-dimensional scanning system comprises: calibrating the first camera and the second camera to obtain internal and external parameters of the first camera and the second camera; acquiring a rotation and translation matrix corresponding to the relative position relationship between the first camera and the second camera; and acquiring a rotation and translation matrix corresponding to the relative position relation between the invisible structured light scanning module and the first camera.
According to yet another embodiment of the present invention, there is also provided a storage medium including a stored program, wherein the program performs any one of the above methods when executed.
According to yet another embodiment of the present invention, there is also provided a processor for executing a program, wherein the program executes to perform the method of any one of the above.
According to another aspect of the embodiments of the present invention, there is also provided a three-dimensional reconstruction apparatus based on a binocular three-dimensional scanning system, the binocular three-dimensional scanning system including: invisible structure light scanning module, first camera, second camera, projection equipment, wherein, the device includes: the projection unit is used for projecting the fringe image onto a measured object by using the projection equipment to generate a modulated fringe image; the acquisition unit is used for acquiring the modulated stripe image by using the first camera to obtain a first image, acquiring the modulated stripe image by using the second camera to obtain a second image, and acquiring the depth map of the measured object by using the invisible structured light scanning module; a matching unit, configured to match the stripes in the first image with the stripes in the second image according to the depth map; the searching unit is used for searching the corresponding relation of a single point in the matched corresponding stripe center line segment to obtain a corresponding point pair; and the reconstruction unit is used for reconstructing the corresponding point pairs into three-dimensional point cloud data.
Further, the matching unit includes: the conversion module is used for converting the depth map into a first three-dimensional point cloud coordinate under the invisible structured light scanning module coordinate system; the conversion module is used for converting the first three-dimensional point cloud coordinate into a second three-dimensional point cloud coordinate under a first camera coordinate system according to a rotation and translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera; the projection module is used for projecting the second three-dimensional point cloud coordinate into the first image and the second image respectively; and the determining module is used for determining the stripes in the second image corresponding to the stripes in the first image according to the preset corresponding relation of the point coordinates in the first image and the second image.
Further, the determining module includes: the first determining submodule is used for determining a target serial number corresponding to a point in a target stripe in the first image; the searching module is used for searching the points corresponding to the target sequence numbers in the second image according to a preset sequence number searching table, wherein the preset sequence number searching table is used for indicating the corresponding relation of point coordinates in the first image and the second image; and the second determining submodule is used for determining the stripe where the point corresponding to the target serial number in the second image is positioned as the stripe matched with the target stripe.
In the embodiment of the invention, in a binocular three-dimensional scanning system, an invisible structured light scanning module, a first camera, a second camera and a projection device are provided, a fringe image is projected on a measured object through the projection device, the modulated fringe image is collected, then a first image of the modulated fringe image is collected through the first camera, a second image of the modulated fringe image is collected through the second camera, a depth map of the measured object is collected through the invisible structured light scanning module, then the fringe in the first image is matched with the fringe in the second image through the depth map, the corresponding relation of single points in the central line segment of the corresponding fringe matched in the first image and the second image is searched, the matched corresponding point pair is reconstructed into three-dimensional point cloud data, and therefore, the mode of combining the collected fringe in the first image and the second image with the depth image collected by the invisible structured light wave group is utilized, the matching accuracy of the stripes in the first image and the stripes in the second image in the three-dimensional scanning system is improved, the number of the stripes can be increased on the basis of ensuring the matching accuracy of the corresponding points, the scanning efficiency of the three-dimensional scanning system is improved, and the technical problem that the matching error rate of the corresponding points is increased along with the increase of the number of the stripes in the existing handheld stripe binocular three-dimensional scanning technology is solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a flow chart of an alternative binocular three-dimensional scanning system based three-dimensional reconstruction method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the architecture of a three-dimensional handheld multi-stripe binocular three-dimensional scanning system according to an embodiment of the present invention as actually designed;
FIG. 3a is a schematic diagram of fringes captured by an alternative left camera according to the above embodiment of the present invention;
FIG. 3b is a schematic diagram of the fringes captured by an alternative left camera according to the above embodiment of the present invention;
FIG. 4a is a schematic diagram of an alternative left camera image stripe line segment segmentation and IR structured light three-dimensional scanning module back projection according to the above embodiment of the present invention;
FIG. 4b is a schematic diagram of an alternative stripe line segment segmentation and IR structured light three-dimensional scanning module back projection for the right camera image according to the above embodiment of the present invention;
FIG. 5 is a schematic view of an alternative epipolar geometry constraint according to the above-described embodiment of the present invention;
fig. 6 is a schematic diagram of an alternative binocular three-dimensional scanning system-based three-dimensional reconstruction apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In accordance with an embodiment of the present invention, there is provided a binocular three-dimensional scanning system based three-dimensional reconstruction method embodiment, it is noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions and that, although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
The binocular three-dimensional scanning system in the binocular three-dimensional scanning system-based three-dimensional reconstruction method of the embodiment of the invention may include: fig. 1 is a flowchart of an optional binocular three-dimensional scanning system-based three-dimensional reconstruction method according to an embodiment of the present invention, and as shown in fig. 1, the method includes the following steps:
step S102, projecting the light stripe image onto a measured object by using projection equipment to generate a modulated stripe image;
step S104, acquiring the modulated stripe image by using a first camera to obtain a first image, acquiring the modulated stripe image by using a second camera to obtain a second image, and acquiring a depth map of the measured object by using an invisible structured light scanning module;
step S106, matching the stripes in the first image with the stripes in the second image according to the depth map;
s108, searching the corresponding relation of single points in the matched corresponding stripe center line segment to obtain corresponding point pairs;
and step S110, reconstructing the corresponding point pairs into three-dimensional point cloud data.
Through the steps, in the binocular three-dimensional scanning system, an invisible structured light scanning module, a first camera, a second camera and a projection device are provided, a stripe image is projected on a measured object through the projection device, then a first image of the modulated stripe image is collected through the first camera, a second image of the modulated stripe image is collected through the second camera, a depth map of the measured object is collected through the invisible structured light scanning module, then the stripe in the first image is matched with the stripe in the second image through the depth map, the corresponding relation of a single point in a central line segment of the corresponding stripe matched in the first image and the second image is searched, the matched corresponding point pair is found, the corresponding point pair is reconstructed into three-dimensional point cloud data, and therefore, the mode that the collected stripe in the first image and the second image is combined with the depth image collected by the invisible structured light wave group is utilized, the matching accuracy of the stripes in the first image and the stripes in the second image in the three-dimensional scanning system is improved, the number of the stripes can be increased on the basis of ensuring the matching accuracy of the corresponding points, the scanning efficiency of the three-dimensional scanning system is improved, and the technical problem that the matching error rate of the corresponding points is increased along with the increase of the number of the stripes in the existing handheld stripe binocular three-dimensional scanning technology is solved.
Alternatively, the invisible structured light scanning module may be an infrared structured light scanning module.
Alternatively, the first camera may be a left camera.
Alternatively, the second camera may be a right camera.
Alternatively, the first image may be a left image captured by a left camera.
Alternatively, the second image may be a right image captured by a right camera.
Alternatively, the projection device may be a digital projector and the corresponding projected fringe image may be a digital analog laser fringe image, wherein the digital analog laser fringe image may be generated by a computer in the binocular three-dimensional scanning system and projected onto the object to be measured by the digital projector. Alternatively, the projection device may also be a laser projection device, and the projected fringe image may be a laser fringe image that can be projected directly onto the object to be measured by the laser projection device. It should be noted that, in the embodiment of the present invention, the projection apparatus is a digital projector, and the projected fringe image is a digital analog laser fringe image, but the projection apparatus is not limited to be only a digital projector, and the projected fringe image is only a digital analog laser fringe image.
As an alternative embodiment, matching the stripes in the first image with the stripes in the second image according to the depth map may comprise: converting the depth map into a first three-dimensional point cloud coordinate under an invisible structured light scanning module coordinate system; converting the first three-dimensional point cloud coordinate into a second three-dimensional point cloud coordinate under a first camera coordinate system according to a rotation translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera; projecting the second three-dimensional point cloud coordinate into the first image and the second image respectively; and determining the stripes in the second image corresponding to the stripes in the first image according to the preset corresponding relation of the point coordinates in the first image and the second image.
By adopting the embodiment of the invention, the depth map is converted into the first three-dimensional point cloud coordinate under the invisible structured light scanning module coordinate system, the corresponding rotation and translation matrix is determined according to the relative position relation between the invisible structured light scanning module and the first camera, and the first three-dimensional point cloud coordinate is converted into the second three-dimensional point cloud coordinate under the first camera coordinate system through the rotation and translation matrix; and then projecting the second three-dimensional point cloud coordinates into the first image and the second image respectively, and determining the stripes in the second image corresponding to the stripes in the first image according to the preset corresponding relationship of the point coordinates in the first image and the second image, so that the corresponding relationship of the stripes in the first image and the second image can be accurately determined.
As an alternative embodiment, determining the stripes in the second image corresponding to the stripes in the first image according to the predetermined correspondence of the point coordinates in the first image and the second image may include: determining a target serial number corresponding to a point in a target stripe in a first image; searching points corresponding to the target sequence number in the second image according to a preset sequence number lookup table, wherein the preset sequence number lookup table is used for indicating the corresponding relation of point coordinates in the first image and the second image; and determining the stripe where the point corresponding to the target serial number in the second image is located as the stripe matched with the target stripe.
By adopting the embodiment of the invention, the corresponding relation between each stripe in the first image and each stripe in the second image is accurately determined by determining the corresponding target serial number of the target stripe in the first image, searching the point corresponding to the target serial number in the second image according to the preset serial number lookup table for indicating the corresponding relation of point coordinates in the first image and the second image, and determining the corresponding relation between each stripe in the first image and each stripe in the second image according to the fact that the stripe where the point corresponding to the target serial number in the second image is located is the stripe mutually matched with the target stripe in the first image.
Alternatively, the target stripe in the first image may be any one stripe in the left image.
As an alternative embodiment, before the fringe image is projected onto the measured object by the projection device to generate the modulated fringe image, the embodiment may further include: and calibrating the binocular three-dimensional scanning system to obtain the structural parameters of the binocular three-dimensional scanning system.
By adopting the embodiment of the invention, before the fringe image is projected to the measured object, the binocular three-dimensional scanning system is calibrated to obtain the structural parameters of the binocular three-dimensional scanning system, so that the corresponding relation between the fringes in the first image and the fringes in the second image can be accurately determined according to the structural parameters obtained after calibration.
As an optional embodiment, calibrating the binocular three-dimensional scanning system, and acquiring the structural parameters of the binocular three-dimensional scanning system may include: calibrating the first camera and the second camera to obtain internal and external parameters of the first camera and the second camera; acquiring a rotation and translation matrix corresponding to the relative position relationship between the first camera and the second camera; and acquiring a rotation and translation matrix corresponding to the relative position relation between the invisible structured light scanning module and the first camera.
By adopting the embodiment of the invention, in the structural parameters obtained by calibration, the internal and external parameters of the first camera and the second camera can be obtained by calibrating the first camera and the second camera; the rotation and translation matrix corresponding to the position relation between the first camera and the second camera can be obtained by calibrating the relative position relation between the first camera and the second camera; the relative position relationship between the invisible structured light scanning module and the first camera can be calibrated to obtain a rotation and translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera, so that the corresponding relationship between the stripes in the first image and the stripes in the second image can be accurately determined according to the internal and external parameters of the first camera and the second camera, the rotation and translation matrix corresponding to the relative position relationship between the first camera and the second camera, and the rotation and translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera.
The invention also provides a preferred embodiment, which provides a three-dimensional reconstruction method for guiding the visible structured light multi-line stripes by the invisible structured light.
The invention mainly takes the technical improvement of the three-dimensional reconstruction of the multi-line stripe of the combination of the invisible light wave band (infrared structural light) and the visible light wave band (white light multi-line stripe) as an example. The method realizes the accurate matching of the white light multi-line stripes by the three-dimensional reconstruction data of the infrared structure light in a mode of combining the infrared structure light and the white light multi-line stripes, improves the matching accuracy of the multi-stripes, and increases the number of the matched stripes, thereby improving the scanning efficiency of the handheld three-dimensional scanning system. The camera with the resolution of 130 ten thousand pixels can reach 100 stripes, and the scanning efficiency can be improved by more than 10 times compared with the scanning efficiency in the prior art under the same frame rate and the same camera resolution.
The technical scheme provided by the invention comprises the following parts: the method comprises the steps of equipment construction, system calibration, digital projection and image acquisition, matching of corresponding stripes in left and right camera images and three-dimensional reconstruction.
Optionally, the device construction part is configured to construct a three-dimensional digital imaging sensor composed of the infrared structured light three-dimensional scanning module, the two cameras and the digital projector, and the relative positions of the device components are fixed, so that a measured object is placed in a measurement range of the sensor, and the object needs to be placed in a position where both the two cameras and the low-resolution three-dimensional scanning module can see.
Optionally, the system calibration part may calibrate the left and right cameras to obtain a rotation and translation matrix Mc corresponding to the relative position between the inside and outside reference cameras of the cameras. And simultaneously calibrating a rotation and translation matrix Ms corresponding to the relative position relation between the infrared structured light three-dimensional scanning module and the left camera.
Optionally, the digital projection and image acquisition part may generate a digital analog laser stripe pattern with a stripe number greater than 15 (the maximum stripe number may be 100 or more) by a computer, project the digital laser stripe pattern to the object to be measured with a digital projector, the digital laser stripe pattern is modulated by the height of the object and deformed, the modulated digital analog laser stripe pattern is generated, an external trigger signal is sent, and the two cameras and the low-resolution three-dimensional scanning module synchronously acquire the digital analog laser stripe pattern. The two cameras synchronously acquire modulated digital simulation laser stripe patterns, and the infrared structured light three-dimensional scanning module acquires a depth map of a measured object.
Alternatively, the matching portions of the corresponding stripes in the left and right camera images may perform centerline extraction on the stripes in the left and right camera images, and then segment each centerline connected domain to form a plurality of independent line segments. And converting the depth map acquired by the infrared structured light three-dimensional scanning module into a three-dimensional point cloud coordinate pi (a first three-dimensional point cloud coordinate) under a self coordinate system according to the corresponding calibration internal reference. And (3) converting pi into a three-dimensional point cloud qi (a second three-dimensional point cloud coordinate) under a left camera coordinate system according to a rotation and translation matrix Ms between the calibrated infrared structured light three-dimensional scanning module and the left camera. And then, back projecting the three-dimensional point cloud qi to the left image and the right image according to respective internal references of the left camera and the right camera in sequence, wherein each corresponding point has a corresponding serial number, and forming a lookup table corresponding to the coordinates of the left image and the right image. And traversing the corresponding serial number of each point of each stripe line segment on the left image, and directly searching the stripe line segment matched with the right image according to the lookup table. Therefore, accurate matching of the left image line segment and the right image line segment can be realized.
Optionally, the three-dimensional reconstruction part searches the corresponding relation of single points in the central line segment of the corresponding stripe by matching the central line segment of the corresponding stripe with the left and right images and using the epipolar geometric constraint relation of the left and right cameras. And then, reconstructing the corresponding point pairs into three-dimensional point cloud data according to the calibration parameters of the system.
Fig. 2 is a schematic diagram of a structure of a three-dimensional handheld multi-stripe binocular three-dimensional scanning system designed in practice according to an embodiment of the present invention, as shown in fig. 2, including: the system comprises a digital projector 101, an infrared structured light three-dimensional scanning module 102, a left camera 103, a right camera 104, a computer 105 and a tested sample 106.
Optionally, the internal parameters of the left camera 103:
optionally, the internal parameters of the right camera 104:
optionally, the system configuration parameters between the cameras:
T＝[-1.778995e+002,-4.162821e-001,5.074737e+001]。
optionally, internal parameters of the infrared structured light three-dimensional scanning module are as follows:
optionally, the system structure parameters between the infrared structured light three-dimensional scanning module and the left camera are as follows:
Ts＝[9.13387457e+001,2.81182536e+001,1.79046857e+000]。
according to the parts, a digital simulation laser stripe pattern is projected to the sample and is simultaneously collected by the camera and the infrared structured light three-dimensional scanning module. According to the collected fringe pattern, there is also a depth map of the infrared structured light three-dimensional scanning module (as shown in fig. 3a and fig. 3b, where fig. 3a is a schematic diagram of fringes collected by an optional left camera according to the above-mentioned embodiment of the present invention, fig. 3b is a schematic diagram of fringes collected by an optional left camera according to the above-mentioned embodiment of the present invention), the depth map is converted into three-dimensional coordinates by using the internal reference of the infrared structured light three-dimensional scanning module, and the three-dimensional coordinates are sequentially projected onto the images of the left and right cameras according to the calibration parameters (as shown in fig. 4a and fig. 4b, fig. 4a is a schematic diagram of optional left camera image fringe line segment segmentation and infrared structured light three-dimensional scanning module back projection according to the above-mentioned embodiment of the present invention, fig. 4b is a schematic diagram of optional right camera image fringe segment segmentation and infrared structured light three-dimensional scanning module back projection according to the above-mentioned embodiment of the present invention, and the left and right corresponding points are endowed with serial numbers to form a serial number lookup table. And extracting the centers of the stripes on the left camera image and the right camera image, carrying out connected domain segmentation, and matching the corresponding line segments of the stripes according to a sequence number lookup table. The matched line segments are subjected to corresponding point search according to epipolar geometric constraint relation of the two cameras (as shown in fig. 5, fig. 5 is a schematic diagram of optional epipolar geometric constraint according to the above embodiment of the invention), and then three-dimensional reconstruction is performed according to calibration parameters to generate point cloud data.
According to the technical scheme provided by the invention, the infrared structured light three-dimensional scanning module is used for guiding the accurate matching of the visible light multi-stripe center line segment in the binocular three-dimensional imaging system.
The technical scheme provided by the invention utilizes a multi-stripe three-dimensional reconstruction technology combining structured light with two different wave bands which do not interfere with each other.
According to the technical scheme provided by the invention, the infrared structured light three-dimensional scanning module is used for guiding the matching algorithm of the visible light multi-line stripe binocular stereo vision, and the cooperative work of light scanning of different wave band structured light is realized. The infrared structured light three-dimensional scanning module can be a Realsense, Kinect, mantis or other invisible structured light three-dimensional imaging module on the market and the like.
The technical scheme provided by the invention can simplify the difficulty of binocular stereo matching and increase the matching accuracy. The method simplifies the step of calibrating the light plane in the prior art, simultaneously removes the limitation on the number of the projected stripes in the prior art, and can improve the scanning rate by more than ten times under the same condition.
According to another aspect of the present invention, there is further provided a storage medium including a stored program, wherein when the program runs, an apparatus on which the storage medium is located is controlled to execute the binocular three-dimensional scanning system-based three-dimensional reconstruction method.
According to another aspect of the present invention, an embodiment of the present invention further provides a processor, where the processor is configured to execute a program, where the program executes the binocular three-dimensional scanning system-based three-dimensional reconstruction method described above.
According to the embodiment of the present invention, an embodiment of a three-dimensional reconstruction device based on a binocular three-dimensional scanning system is further provided, and it should be noted that the three-dimensional reconstruction device based on the binocular three-dimensional scanning system may be used to execute the three-dimensional reconstruction method based on the binocular three-dimensional scanning system in the embodiment of the present invention, and the three-dimensional reconstruction method based on the binocular three-dimensional scanning system in the embodiment of the present invention may be executed in the three-dimensional reconstruction device based on the binocular three-dimensional scanning system.
Fig. 6 is a schematic diagram of an alternative binocular three-dimensional scanning system-based three-dimensional reconstruction apparatus according to an embodiment of the present invention, as shown in fig. 6, the binocular three-dimensional scanning system includes: invisible structure light scanning module, first camera, second camera, projection equipment, wherein, the device can include: a projection unit 61 for projecting the fringe image onto the object to be measured by using a projection device to generate a modulated fringe image; the acquisition unit 63 is used for acquiring the modulated fringe image by using a first camera to obtain a first image, acquiring the modulated fringe image by using a second camera to obtain a second image, and acquiring the depth map of the measured object by using the invisible structured light scanning module; a matching unit 65 for matching the stripes in the first image with the stripes in the second image according to the depth map; the searching unit 67 is configured to search a correspondence relationship between single points in the matched corresponding stripe center line segments to obtain corresponding point pairs; a reconstruction unit 69 for reconstructing the corresponding point pairs into three-dimensional point cloud data.
It should be noted that the projection unit 61 in this embodiment may be configured to perform step S102 in this embodiment, the acquisition unit 63 in this embodiment may be configured to perform step S104 in this embodiment, the matching unit 65 in this embodiment may be configured to perform step S106 in this embodiment, the search unit 67 in this embodiment may be configured to perform step S108 in this embodiment, and the reconstruction unit 69 in this embodiment may be configured to perform step S110 in this embodiment. The modules are the same as the corresponding steps in the realized examples and application scenarios, but are not limited to the disclosure of the above embodiments.
According to the above embodiment of the present invention, in the binocular three-dimensional scanning system, there are an invisible structured light scanning module, a first camera, a second camera, and a projection device, the projection device projects a fringe image onto a measured object, the first camera collects a first image of the modulated fringe image, the second camera collects a second image of the modulated fringe image, the invisible structured light scanning module collects a depth map of the measured object, the depth map is used to match the fringes in the first image with the fringes in the second image, the corresponding relationship of a single point in the central line segment of the corresponding fringe matched in the first image and the second image is searched, the corresponding point pair is matched, the corresponding point pair is reconstructed into three-dimensional data, so that the manner of combining the collected fringes in the first image and the second image with the depth image collected by the incorporable structured light wave group is used, the matching accuracy of the stripes in the first image and the stripes in the second image in the three-dimensional scanning system is improved, the number of the stripes can be increased on the basis of ensuring the matching accuracy of the corresponding points, the scanning efficiency of the three-dimensional scanning system is improved, and the technical problem that the matching error rate of the corresponding points is increased along with the increase of the number of the stripes in the existing handheld stripe binocular three-dimensional scanning technology is solved.
As an alternative embodiment, the matching unit comprises: the conversion module is used for converting the depth map into a first three-dimensional point cloud coordinate under an invisible structured light scanning module coordinate system; the conversion module is used for converting the first three-dimensional point cloud coordinate into a second three-dimensional point cloud coordinate under a first camera coordinate system according to a rotation translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera; the projection module is used for projecting the second three-dimensional point cloud coordinate into the first image and the second image respectively; and the determining module is used for determining the stripes in the second image corresponding to the stripes in the first image according to the preset corresponding relation between the point coordinates in the first image and the second image.
As an alternative embodiment, the determining module includes: the first determining submodule is used for determining a target serial number corresponding to a point in a target stripe in the first image; the searching module is used for searching points corresponding to the target sequence number in the second image according to a preset sequence number searching table, wherein the preset sequence number searching table is used for indicating the corresponding relation of point coordinates in the first image and the second image; and the second determining submodule is used for determining the stripe where the point corresponding to the target serial number in the second image is positioned as the stripe matched with the target stripe.
As an alternative embodiment, the embodiment may further include: and the calibration unit is used for calibrating the binocular three-dimensional scanning system before the fringe image is projected onto the measured object by using the projection equipment and the modulated fringe image is generated, so as to obtain the structural parameters of the binocular three-dimensional scanning system.
As an alternative embodiment, the calibration unit comprises: the calibration module is used for calibrating the first camera and the second camera to acquire internal and external parameters of the first camera and the second camera; the first acquisition module is used for acquiring a rotation and translation matrix corresponding to the relative position relationship between the first camera and the second camera; and the second acquisition module is used for acquiring a rotation and translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (6)
1. A binocular three-dimensional scanning system-based three-dimensional reconstruction method is characterized in that the binocular three-dimensional scanning system comprises: invisible structured light scanning module, first camera, second camera, projection equipment, wherein, the method includes:
projecting the fringe image onto a measured object by using the projection equipment to generate a modulated fringe image;
acquiring the modulated stripe image by using the first camera to obtain a first image, acquiring the modulated stripe image by using the second camera to obtain a second image, and acquiring the depth map of the measured object by using the invisible structured light scanning module;
matching the stripes in the first image with the stripes in the second image according to the depth map;
searching the corresponding relation of single points in the matched corresponding stripe center line segment to obtain a corresponding point pair;
reconstructing the corresponding point pairs into three-dimensional point cloud data;
wherein before projecting the fringe image onto the object to be measured using the projection device to produce a modulated fringe image, the method further comprises:
calibrating the binocular three-dimensional scanning system to obtain structural parameters of the binocular three-dimensional scanning system;
the calibration of the binocular three-dimensional scanning system, and the acquisition of the structural parameters of the binocular three-dimensional scanning system comprise:
calibrating the first camera and the second camera to obtain internal and external parameters of the first camera and the second camera;
acquiring a rotation and translation matrix corresponding to the relative position relationship between the first camera and the second camera;
acquiring a rotation and translation matrix corresponding to the relative position relation between the invisible structured light scanning module and the first camera;
wherein matching the stripes in the first image with the stripes in the second image according to the depth map comprises:
converting the depth map into a first three-dimensional point cloud coordinate under the invisible structured light scanning module coordinate system;
converting the first three-dimensional point cloud coordinate into a second three-dimensional point cloud coordinate under the first camera coordinate system according to a rotation translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera;
projecting the second three-dimensional point cloud coordinates into the first image and the second image respectively;
and determining stripes in the second image corresponding to the stripes in the first image according to the preset corresponding relation of point coordinates in the first image and the second image.
2. The method of claim 1, wherein determining the stripes in the second image corresponding to the stripes in the first image according to a predetermined correspondence of point coordinates in the first image and the second image comprises:
determining a target serial number corresponding to a point in a target stripe in the first image;
searching a point corresponding to the target sequence number in the second image according to a preset sequence number lookup table, wherein the preset sequence number lookup table is used for indicating the corresponding relation of point coordinates in the first image and the second image;
and determining the stripe where the point corresponding to the target sequence number in the second image is located as the stripe matched with the target stripe.
3. A binocular three-dimensional scanning system-based three-dimensional reconstruction device is characterized in that the binocular three-dimensional scanning system comprises: invisible structure light scanning module, first camera, second camera, projection equipment, wherein, the device includes:
the projection unit is used for projecting the fringe image onto a measured object by using the projection equipment to generate a modulated fringe image;
the acquisition unit is used for acquiring the modulated stripe image by using the first camera to obtain a first image, acquiring the modulated stripe image by using the second camera to obtain a second image, and acquiring the depth map of the measured object by using the invisible structured light scanning module;
a matching unit, configured to match the stripes in the first image with the stripes in the second image according to the depth map;
the searching unit is used for searching the corresponding relation of a single point in the matched corresponding stripe center line segment to obtain a corresponding point pair;
the reconstruction unit is used for reconstructing the corresponding point pairs into three-dimensional point cloud data;
wherein the apparatus further comprises:
the calibration unit is used for calibrating the binocular three-dimensional scanning system before the fringe image is projected onto a measured object by the projection equipment and the modulated fringe image is generated, so as to obtain the structural parameters of the binocular three-dimensional scanning system;
wherein, the calibration unit includes:
the calibration module is used for calibrating the first camera and the second camera to acquire internal and external parameters of the first camera and the second camera;
the first acquisition module is used for acquiring a rotation and translation matrix corresponding to the relative position relationship between the first camera and the second camera;
the second acquisition module is used for acquiring a rotation and translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera;
wherein the matching unit includes:
the conversion module is used for converting the depth map into a first three-dimensional point cloud coordinate under the invisible structured light scanning module coordinate system;
the conversion module is used for converting the first three-dimensional point cloud coordinate into a second three-dimensional point cloud coordinate under a first camera coordinate system according to a rotation and translation matrix corresponding to the relative position relationship between the invisible structured light scanning module and the first camera;
the projection module is used for projecting the second three-dimensional point cloud coordinate into the first image and the second image respectively;
and the determining module is used for determining the stripes in the second image corresponding to the stripes in the first image according to the preset corresponding relation of the point coordinates in the first image and the second image.
4. The apparatus of claim 3, wherein the determining module comprises:
the first determining submodule is used for determining a target serial number corresponding to a point in a target stripe in the first image;
the searching module is used for searching the points corresponding to the target sequence numbers in the second image according to a preset sequence number searching table, wherein the preset sequence number searching table is used for indicating the corresponding relation of point coordinates in the first image and the second image;
and the second determining submodule is used for determining the stripe where the point corresponding to the target serial number in the second image is positioned as the stripe matched with the target stripe.
5. A storage medium, characterized in that the storage medium comprises a stored program, wherein a device on which the storage medium is located is controlled to perform the method of claim 1 or 2 when the program is run.
6. A processor, characterized in that the processor is configured to run a program, wherein the program when running performs the method of claim 1 or 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710581732.8A CN108267097B (en) | 2017-07-17 | 2017-07-17 | Three-dimensional reconstruction method and device based on binocular three-dimensional scanning system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710581732.8A CN108267097B (en) | 2017-07-17 | 2017-07-17 | Three-dimensional reconstruction method and device based on binocular three-dimensional scanning system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108267097A CN108267097A (en) | 2018-07-10 |
| CN108267097B true CN108267097B (en) | 2020-07-10 |
Family
ID=62771705
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710581732.8A Active CN108267097B (en) | 2017-07-17 | 2017-07-17 | Three-dimensional reconstruction method and device based on binocular three-dimensional scanning system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108267097B (en) |
Families Citing this family (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109357633B (en) * | 2018-09-30 | 2022-09-30 | 先临三维科技股份有限公司 | Three-dimensional scanning method, device, storage medium and processor |
| CN109472741B (en) * | 2018-09-30 | 2023-05-30 | 先临三维科技股份有限公司 | Three-dimensional splicing method and device |
| CN109470140A (en) * | 2018-11-12 | 2019-03-15 | 珠海格力智能装备有限公司 | Screw detection method and device and screw detection system |
| CN109712230B (en) * | 2018-11-27 | 2023-02-28 | 先临三维科技股份有限公司 | Three-dimensional model supplementing method and device, storage medium and processor |
| CN109499010B (en) * | 2018-12-21 | 2021-06-08 | 苏州雷泰医疗科技有限公司 | Radiotherapy auxiliary system based on infrared and visible light three-dimensional reconstruction and method thereof |
| CN111508012B (en) * | 2019-01-31 | 2024-04-19 | 先临三维科技股份有限公司 | Method and device for line stripe mismatching detection and three-dimensional reconstruction |
| CN110363806B (en) * | 2019-05-29 | 2021-12-31 | 中德（珠海）人工智能研究院有限公司 | Method for three-dimensional space modeling by using invisible light projection characteristics |
| CN110702025B (en) * | 2019-05-30 | 2021-03-19 | 北京航空航天大学 | Grating type binocular stereoscopic vision three-dimensional measurement system and method |
| WO2020259625A1 (en) * | 2019-06-28 | 2020-12-30 | 先临三维科技股份有限公司 | Three-dimensional scanning method, scanner, three-dimensional scanning system, computer device, and computer-readable storage medium |
| EP3761220A1 (en) * | 2019-07-05 | 2021-01-06 | Everdrone AB | Method for improving the interpretation of the surroundings of a vehicle |
| CN112595262B (en) * | 2020-12-08 | 2022-12-16 | 广东省科学院智能制造研究所 | Binocular structured light-based high-light-reflection surface workpiece depth image acquisition method |
| CN115200510A (en) * | 2021-04-09 | 2022-10-18 | 圣邦微电子（北京）股份有限公司 | Device and method for acquiring depth information of object surface |
| CN114898038A (en) * | 2022-04-28 | 2022-08-12 | 杭州海康机器人技术有限公司 | Image reconstruction method, device and equipment |
| CN115345993A (en) * | 2022-08-10 | 2022-11-15 | 先临三维科技股份有限公司 | Three-dimensional reconstruction method, device and system |
| CN115345995A (en) * | 2022-08-10 | 2022-11-15 | 先临三维科技股份有限公司 | Three-dimensional reconstruction method, device and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0875462A (en) * | 1994-09-08 | 1996-03-22 | Fujitsu Ltd | Binocular stereoscopic viewing device |
| DE102004026090A1 (en) * | 2004-05-25 | 2005-12-22 | Daimlerchrysler Ag | Three dimensional object measurement system has video sensor and laser used for distance measurement in alternate clock intervals |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101577795A (en) * | 2009-06-17 | 2009-11-11 | 深圳华为通信技术有限公司 | Method and device for realizing real-time viewing of panoramic picture |
| CN102884397B (en) * | 2010-05-07 | 2015-07-15 | 深圳泰山在线科技有限公司 | Structured-light measuring method and system |
| CN102564347B (en) * | 2011-12-30 | 2014-11-26 | 中国科学院上海光学精密机械研究所 | Object three-dimensional outline measuring device and method based on Dammann grating |
| CN102609941A (en) * | 2012-01-31 | 2012-07-25 | 北京航空航天大学 | Three-dimensional registering method based on ToF (Time-of-Flight) depth camera |
| JP6100635B2 (en) * | 2012-12-13 | 2017-03-22 | 株式会社デンソー | Moving body detection device |
| CN105737762B (en) * | 2016-05-09 | 2018-03-16 | 中国民用航空飞行学院 | A kind of blade of aviation engine profile measurement method |
-
2017
- 2017-07-17 CN CN201710581732.8A patent/CN108267097B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0875462A (en) * | 1994-09-08 | 1996-03-22 | Fujitsu Ltd | Binocular stereoscopic viewing device |
| DE102004026090A1 (en) * | 2004-05-25 | 2005-12-22 | Daimlerchrysler Ag | Three dimensional object measurement system has video sensor and laser used for distance measurement in alternate clock intervals |
Non-Patent Citations (1)
| Title |
|---|
| 一种基于双目视觉的三维重建方法;陈大平等;《现代测绘》;20160115;第39卷(第1期);第14-16页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108267097A (en) | 2018-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108267097B (en) | Three-dimensional reconstruction method and device based on binocular three-dimensional scanning system | |
| AU2017400983B2 (en) | Three-dimensional scanning system and scanning method thereof | |
| JP6564537B1 (en) | 3D reconstruction method and apparatus using monocular 3D scanning system | |
| CA3022442C (en) | Three-dimensional reconstruction method and device based on monocular three-dimensional scanning system | |
| KR102073205B1 (en) | 3D scanning method and scanner including multiple different wavelength lasers | |
| CN108151671B (en) | A kind of 3 D digital imaging sensor, 3 D scanning system and its scan method | |
| CA2875820C (en) | 3-d scanning and positioning system | |
| CN109357633B (en) | Three-dimensional scanning method, device, storage medium and processor | |
| CN106548489B (en) | A kind of method for registering, the three-dimensional image acquisition apparatus of depth image and color image | |
| CN112330732A (en) | Three-dimensional data splicing method, three-dimensional scanning system and handheld scanner | |
| CN109443245B (en) | Multi-line structured light vision measurement method based on homography matrix | |
| CN107860337B (en) | Structured light three-dimensional reconstruction method and device based on array camera | |
| EP2751521A1 (en) | Method and system for alignment of a pattern on a spatial coded slide image | |
| Zhou et al. | Three-dimensional shape measurement using color random binary encoding pattern projection | |
| CN109712230B (en) | Three-dimensional model supplementing method and device, storage medium and processor | |
| CN107493429B (en) | Selfie stick shielding method and device for taking selfie pictures | |
| CN112325799A (en) | High-precision three-dimensional face measurement method based on near-infrared light projection | |
| Bonacina et al. | A low-cost solution to 3D pinna modeling for HRTF prediction | |
| US11989917B2 (en) | Line stripe mismatch detection and three-dimensional reconstruction method and device | |
| CN106815864B (en) | Depth information measurement method based on single frames modulation template | |
| IL224345A (en) | System and method for spatial channel separation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 311258 No. 1398 Xiangbin Road, Wenyan Street, Xiaoshan District, Hangzhou City, Zhejiang Province Applicant after: SHINING 3D TECH Co.,Ltd. Address before: 311258 No. 1398 Xiangbin Road, Wenyan Street, Xiaoshan District, Hangzhou City, Zhejiang Province Applicant before: SHINING 3D TECH Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |