The application be that January 27, application number in 2011 are 201110029945.2 the applying date, name is called the dividing an application of application for a patent for invention of " a kind of system and method for reconstructing surface of object ".
Embodiment
Further specify technical scheme of the present invention below in conjunction with accompanying drawing and by embodiment.Be understandable that, specific embodiment described herein only is used for explaining the present invention, but not limitation of the invention.Also need to prove in addition, for convenience of description, only show step related to the present invention in the accompanying drawing but not all processes.
See also Fig. 1, be the index path of the system of reconstructing surface of object according to the preferred embodiment of the invention.As shown in Figure 1, the system of the reconstructing surface of object that provides of this embodiment mainly comprises light fixture 10, imageing sensor 20, data processing unit 30 and controller (not shown).
Light fixture 10 comprises light source 11 and specular-reflection unit 12.The light source of this programme preferably adopts the semiconductor laser diode.
Specular-reflection unit 12 can be controlled its minute surface attitude by controller, and the light reflection that described light source 11 is sent forms default structured light patterns to the target area.Preferably, specular-reflection unit can adopt one dimension galvanometer or 2-D vibration mirror, and the corresponding structured light patterns that generates is respectively linear array shape pattern or lattice-like pattern.When light source adopted pointolite, specular-reflection unit can adopt 2-D vibration mirror, and the structured light patterns of generation is linear array shape pattern or lattice-like pattern.When light source adopted line source, specular-reflection unit can adopt the one dimension galvanometer, and the pattern of generation is linear array shape pattern.In addition, when light source adopts pointolite, can also by optical element pointolite be converted to linear light first, generate linear array shape pattern by the one dimension galvanometer again.One dimension galvanometer or 2-D vibration mirror also can adopt MEMS(Micro-Electro-Mechanical Systems, MEMS (micro electro mechanical system)) galvanometer, rotary shaft galvanometer etc. can realize controlling the device of minute surface motion.MEMS is a kind of brand-new research and development field that must consider simultaneously multiple physical field immixture, and with respect to traditional machinery, their size is less, and maximum is no more than one centimetre, even only is several microns, and its thickness is just more small.Employing is take silicon as main material, excellent electrical properties, the intensity of silicon materials, hardness and Young modulus and iron phase ought, density and aluminium are similar, pyroconductivity is near molybdenum and tungsten.Adopt and the similar generation technique of integrated circuit, can utilize in a large number mature technology, technique in the IC production, carry out in enormous quantities, low-cost production, cost performance is increased substantially with respect to tradition " machinery " manufacturing technology.The attitude of the minute surface 121 in the specular-reflection unit that one dimension galvanometer or 2-D vibration mirror consist of can be controlled by controller, by adjusting the attitude of minute surface 121, light source 11 is shone light reflection on the minute surface 121 to the target area, and form the pattern of project organization light in the target area projection. for example 2-D vibration mirror projects the lattice-like pattern, and the one dimension galvanometer is linear array shape pattern.
Imageing sensor 20 is used for progressively exposing with the projected image of the structured light patterns at least part of zone in the photographic subjects zone.Because system of the present invention needs to gather the benchmark image of primary structure light before using first.In native system the relative position and attitude of imageing sensor 20, specular-reflection unit 12 and light source 11 need to when taking benchmark image, remain unchanged, comprise skew and rotation here.If variation has appearred in relative position, just need the Resurvey benchmark image.Unique distinction of the present invention is by the view field of the structured light of the exposure area of controller control chart image-position sensor 20 and specular-reflection unit 12 reflections synchronous.That is to say, when linear structured light progressively is projected on the target area, also expose the simultaneously zone of this linear smooth projection of imageing sensor 12, therefore imageing sensor 12 at least part of zone in the target area only, the specific region acquired projections image at the target object place of for example pre-estimating, and do not need to gather image in all target areas.And the pattern that can the relatively low imageing sensor of option and installment gathers linear light.
Data processing unit 30 is connected with imageing sensor 20, receive in real time the projected image of the structured light patterns of imageing sensor 20 shootings, and store the benchmark image of taking in advance in the data processing unit 30, utilize image algorithm to obtain the shift value of the structured light patterns in described projected image and the benchmark image, calculate the three dimensional point cloud of object in the described target area.Therefore, data processing unit 30 may further include storer 31 and processor 32, and data processing unit 30 also has input/output device 33.Wherein storer 31 can be used for being stored in the benchmark image information of taking when not having target object.32 of processors can receive the projected image that imageing sensor is taken by input/output device 33, and obtain the said reference image information from storer 31, carry out above-mentioned calculating and processing, obtain three dimensional point cloud, realize the reconstruction of body surface cloud data.
The below describes the positional structure of all parts in the system of reconstructing surface of object of the present invention.Set up three-dimensional system of coordinate as shown in Figure 1, in this coordinate system, each device is installed.Coordinate system satisfies the right-hand rule among Fig. 1.
A. the X-direction of coordinate system is the line of the camera lens photocentre C of the galvanometer minute surface center P of as shown in the figure original state and imageing sensor, and positive dirction is CP, and namely as upwards being the X-axis positive dirction among Fig. 1, wherein B is the distance of PC.
The b.Z direction of principal axis is to pass the minute surface center P, and perpendicular to minute surface, and positive dirction is to point to the target area by the minute surface center P, as being to the right the Z axis positive dirction among Fig. 1.
The c.Y direction of principal axis is to determine according to the right-hand rule, in Fig. 1 be vertical paper inwards.
D. light source 11 is launched collimation laser, and the laser axis and passes the center of reflecting surface on plane X PZ.The angle of laser axis and Z axis is relevant with the target area, and native system is set to 45 degree.
E. the photocentre axis of imageing sensor 20 is set to parallelly with Z axis, and guarantees that there are common factor in the visual field of imageing sensor and projection target zone, and the lap of the angular field of view of emergent ray slewing area and imageing sensor camera lens is measured zone.And the equipment lens focus of imageing sensor is f.
If f. the specular-reflection unit of this system adopts the one dimension galvanometer, minute surface 121 can be around the XP axle or around the rotation of YP axle or vibration; If the employing 2-D vibration mirror, then minute surface can rotate or simultaneously vibration simultaneously around diaxon.In order to obtain above-mentioned vibration characteristics, can adopt existing MEMS one dimension or 2-D vibration mirror to realize, its emergent ray slewing area covers whole testee.
G. the data communication device of imageing sensor 20 is crossed existing common data transfer mode, is transferred to data processing unit 30.
The below describes realization flow and the principle of the system of reconstructing surface of object of the present invention.
1) according to above-mentioned positional structure each parts is assembled.Can carry out simultaneously integration packaging to a certain degree, for example light source 11 and specular-reflection unit 12 be integrated formation light fixture 10.
2) design of projection pattern: because the galvanometer system that the present invention adopts can utilize galvanometer posture changing characteristic at a high speed, in the target area, project lattice-like, linear array shape etc. image.Here mainly adopt the one dimension galvanometer to irradiate the linear array shape or adopt 2-D vibration mirror to project dot matrix.
3) gather the image information step:
A. take benchmark image.For example, at system of distance L=2000mm place, the 3*3m standard flat T vertical with Z axis is set, the image of choosing this plane T photographs is benchmark image.Although provided the distance of concrete benchmark image among this embodiment, the present invention is not limited to this, this area basic technology personnel can choose suitable plane according to actual conditions and take benchmark image.Utilize imageing sensor 20 gather linear array shape that light fixtures 10 project or dot pattern in the imaging of standard flat as benchmark image, and be kept in the storer 31 of data processing unit 30.
B. the projected image of photographic subjects.Behind object target approach zone, projection pattern shines on the body surface, utilizes this moment imageing sensor 20 to gather the projection pattern of target area, and for example linear array shape pattern sends to data processing unit 30;
4) surface point cloud data reconstruction step:
See also Fig. 2 and Fig. 3, be respectively the benchmark image of according to the preferred embodiment of the invention system's shooting of reconstructing surface of object and the schematic diagram of projected image.As shown in Figure 2, this structured light patterns is the linear array shape, and the height of line is H, and width is W.Then correspondingly can calculate that the line-spacing between the n bar line is W in the linear array
nCan see among Fig. 3 that the lines in the linear array in the target object region will be offset when target object enters picture, side-play amount is S.All projective patterns are along with the increase of distance L, and being offset towards fixed-direction can appear in pattern, and side-play amount S diminishes gradually.And the maximal value of skew S must be less than the minimum period W of periodic pattern
nSide-play amount S can calculate acquisition for projected image and benchmark image analysis by data processing unit.
A. establish the image that the projected image that collects and benchmark image are m*n pixel.The below analyzes the projected image that obtains take the structured light patterns of projection as linear array light as example, adopts the analytical approach of structured light patterns of lattice-like identical with it.
At first, utilize image algorithm to calculate respectively in projected image and the benchmark image coordinate array P of every line in the linear array
n(i, j) and Q
n(i, j), wherein n represents the numbering of linear array center line, and (i, j) is the coordinate of every bit on the line.At this and since in the image grey scale pixel value of structured light lines greater than the surrounding pixel gray-scale value, so can in background, tell the structured light lines.The present invention can adopt the way of Threshold segmentation that the structured light lines are split.Because Threshold, the structured light line thickness may be comprised of one or more than one pixel, so can adopt in the center of gravity of live width or the live width bright pixel coordinate to come the coordinate of representative structure light line.Wherein, in first method, in structured light line thickness direction, calculate the barycentric coordinates of maximum gradation value pixel, as the lines coordinate.In the second method, in structured light line thickness direction, directly calculate the centre coordinate of live width, as the coordinate of lines.By column the coordinate that calculates the structured light lines respectively after the alignment of projected image and benchmark image.If be the lattice-like pattern, then surpass the situation of a pixel for every profile in the dot matrix, calculate the barycentric coordinates of doing weight by gray threshold, as the coordinate of point.
B. utilize the lines array of pixels to calculate the displacement △ X of lines
n(i, j)=P
n(i, j)-Q
n(i, j).Can be with the shift value of two structured light lines alignment coordinates computed by column.The side-play amount S of lines and △ X in the target object zone among Fig. 3
n(i, j) is corresponding.
C. utilize the ultimate principle of structural light measurement, data processing unit is △ X by following formula computation structure light pattern shift value
nThe distance z of impact point;
Wherein, B is the distance at the center of imageing sensor and galvanometer device, and f is the equipment lens focus of described imageing sensor, and L is the distance of the standard flat at the benchmark image place chosen.The distance z of difference is the object dimensional cloud data that calculates in the projection pattern.
Because B, f can pre-set for target object and the distance of different sizes with L in the above-mentioned formula, therefore the particular value of X ' for setting in the above-mentioned formula.When needs during for different target object project organization light pattern, only need to adjust the mirror status of one dimension or 2-D vibration mirror, just can obtain different structured lights, for example linear array light or dot matrix light, can also adjust the line-spacing of linear array light, thereby be convenient in the projected image that gathers, identify the zone at target object place.
The present invention also correspondingly provides a kind of method of reconstructing surface of object.See also Fig. 4, be the method flow diagram according to reconstructing surface of object in the preferred embodiment of the present invention.As shown in Figure 4, the method S400 starts from step S402:
Subsequently, in next step S404, by the control specular-reflection unit, the light reflection that light source is sent also forms default structured light patterns to the target area.This step mainly adopts light fixture to realize, as shown in Figure 1, light fixture 10 comprises light source 11 and specular-reflection unit 12.The light source of this programme preferably adopts the semiconductor laser diode.
Specular-reflection unit 12 can be controlled its minute surface attitude by controller, and the light reflection that described light source 11 is sent forms default structured light patterns to the target area.Preferably, specular-reflection unit can adopt one dimension galvanometer or 2-D vibration mirror, and the corresponding structured light patterns that generates is respectively linear array shape pattern or lattice-like pattern.When light source adopted pointolite, specular-reflection unit can adopt 2-D vibration mirror, and the structured light patterns of generation is linear array shape pattern or lattice-like pattern.When light source adopted line source, specular-reflection unit can adopt the one dimension galvanometer, and the pattern of generation is linear array shape pattern.In addition, when light source adopts pointolite, can also by optical element pointolite be converted to linear light first, generate linear array shape pattern by the one dimension galvanometer again.One dimension galvanometer or 2-D vibration mirror also can adopt MEMS(Micro-Electro-Mechanical Systems, MEMS (micro electro mechanical system)) galvanometer, rotary shaft galvanometer etc. can realize controlling the device of minute surface motion.The attitude of the minute surface 121 in the specular-reflection unit that one dimension galvanometer or 2-D vibration mirror consist of can be controlled by controller, by adjusting the attitude of minute surface 121, light source 11 is shone light reflection on the minute surface 121 to the target area, and form the pattern of project organization light in the target area projection.For example 2-D vibration mirror projects the lattice-like pattern, and the one dimension galvanometer is linear array shape pattern.
Set up three-dimensional system of coordinate as shown in Figure 1, in this coordinate system, each device is installed.Coordinate system satisfies the right-hand rule among Fig. 1.The X-direction of coordinate system is the line of the camera lens photocentre C of the galvanometer minute surface center P of as shown in the figure original state and imageing sensor, and positive dirction is CP, and namely as upwards being the X-axis positive dirction among Fig. 1, wherein B is the distance of PC.Z-direction is to pass the minute surface center P, and perpendicular to minute surface, and positive dirction is to point to the target area by the minute surface center P, as being to the right the Z axis positive dirction among Fig. 1.Y direction is to determine according to the right-hand rule, in Fig. 1 be vertical paper inwards.Light source 11 is launched collimation laser, and the laser axis and passes the center of reflecting surface on plane X PZ.The angle of laser axis and Z axis is relevant with the target area, is set to 45 degree in this method.If the specular-reflection unit of the method adopts the one dimension galvanometer, minute surface 121 can be around the XP axle or around the rotation of YP axle or vibration; If the employing 2-D vibration mirror, then minute surface can rotate or simultaneously vibration simultaneously around diaxon.In order to obtain above-mentioned vibration characteristics, can adopt existing MEMS one dimension or 2-D vibration mirror to realize, its emergent ray slewing area covers whole testee.
Subsequently, in next step S406, utilize imageing sensor progressively to expose to take the projected image of the structured light patterns at least part of zone in the described target area, and it is synchronous to control the view field of structured light of the exposure area of described imageing sensor and the reflection of described specular-reflection unit.Unique distinction of the present invention is by the view field of the structured light of the exposure area of controller control chart image-position sensor 20 and specular-reflection unit 12 reflections synchronous.That is to say, when linear structured light progressively is projected on the target area, also expose the simultaneously zone of this linear light projection of imageing sensor 12, therefore imageing sensor 12 at least part of zone in the target area only, the specific region acquired projections image at the target object place of for example pre-estimating, and do not need to gather image in all target areas.And the pattern that can the relatively low imageing sensor of option and installment gathers linear light.
As shown in Figure 1, because method of the present invention needs to gather the benchmark image of primary structure light before using first, therefore in this step the relative position and attitude of imageing sensor 20, specular-reflection unit 12 and light source 11 need to when taking benchmark image, remain unchanged, comprise skew and rotation here.If variation has appearred in relative position, just need the Resurvey benchmark image.The photocentre axis of imageing sensor 20 is set to parallel with Z axis, and guarantees that there are common factor in the visual field of imageing sensor and projection target zone, and the lap of the angular field of view of emergent ray slewing area and imageing sensor camera lens is measured zone.And the equipment lens focus of imageing sensor is f.The data communication device of imageing sensor 20 is crossed existing common data transfer mode, is transferred to data processing unit 30.
Subsequently, in next step S408, receive in real time the projected image of the structured light patterns that imageing sensor 20 takes, and utilize image algorithm to obtain shift value in the benchmark image of described projected image and in advance shooting, calculate the three dimensional point cloud of object in the described target area.Can adopt data processing unit as shown in Figure 1 to realize this step.Data processing unit 30 is connected with imageing sensor 20, receive the projected image of the structured light patterns of imageing sensor 20 shootings, and store the benchmark image of taking in advance in the data processing unit 30, utilize image algorithm to obtain the shift value of the structured light patterns in described projected image and the benchmark image, calculate the three dimensional point cloud of object in the described target area.Therefore, data processing unit 30 may further include storer 31 and processor 32, and data processing unit 30 also has input/output device 33.Wherein storer 31 can be used for being stored in the benchmark image information of taking when not having target object.32 of processors can receive the projected image that imageing sensor is taken by input/output device 33, and obtain the said reference image information from storer 31, carry out above-mentioned calculating and processing, obtain three dimensional point cloud, realize the reconstruction of body surface cloud data.
At last, method S400 ends at step S410.
The below describes the specific operation process of the method for reconstructing surface of object of the present invention.
1) according to above-mentioned positional structure each parts is assembled.Can carry out simultaneously integration packaging to a certain degree, for example light source 11 and specular-reflection unit 12 be integrated formation light fixture 10.
2) design of projection pattern: because the galvanometer system that the inventive method adopts can utilize galvanometer posture changing characteristic at a high speed, in the target area, project lattice-like, linear array shape etc. image.Here mainly adopt the one dimension galvanometer to irradiate linear array or adopt 2-D vibration mirror to project dot matrix.
3) gather the image information step:
A. take benchmark image.For example, at system of distance L=2000mm place, the 3*3m standard flat T vertical with Z axis is set, the image of choosing this plane T photographs is benchmark image.Although provided the distance of concrete benchmark image among this embodiment, the present invention is not limited to this, this area basic technology personnel can choose suitable plane according to actual conditions and take benchmark image.Utilize imageing sensor 20 gather linear array shape that light fixtures 10 project or lattice-like pattern in the imaging of standard flat as benchmark image, and be kept in the storer 31 of data processing unit 30.
B. the projected image of photographic subjects.Behind object target approach zone, projection pattern shines on the body surface, utilizes this moment imageing sensor 20 to gather the projection pattern of target area, and for example linear array shape pattern sends to data processing unit 30;
4) surface point cloud data reconstruction step:
See also Fig. 2 and Fig. 3, be respectively the benchmark image of according to the preferred embodiment of the invention system's shooting of reconstructing surface of object and the schematic diagram of projected image.As shown in Figure 2, this structured light patterns is linear array, and the height of line is H, and width is W.Then correspondingly can calculate that the line-spacing between the n bar line is W in the linear array
nCan see among Fig. 3 that the lines in the linear array in the target object region will be offset when target object enters picture, side-play amount is S.All projective patterns are along with the increase of distance L, and being offset towards fixed-direction can appear in pattern, and side-play amount S diminishes gradually.And the maximal value of skew S must be less than the minimum period W of periodic pattern
nSide-play amount S can calculate acquisition for projected image and benchmark image analysis by data processing unit.
A. establish the image that the projected image that collects and benchmark image are m*n pixel.The below analyzes the projected image that obtains take the structured light patterns of projection as linear array light as example, adopts the analytical approach of structured light patterns of lattice-like identical with it.
At first, utilize image algorithm to calculate respectively in projected image and the benchmark image coordinate array P of every line in the linear array
n(i, j) and Q
n(i, j), wherein n represents the numbering of linear array center line, and (i, j) is the coordinate of every bit on the line.At this and since in the image grey scale pixel value of structured light lines greater than the surrounding pixel gray-scale value, so can in background, tell the structured light lines.The present invention can adopt the way of Threshold segmentation that the structured light lines are split.Because Threshold, the structured light line thickness may be comprised of one or more than one pixel, so can adopt in the center of gravity of live width or the live width bright pixel coordinate to come the coordinate of representative structure light line.Wherein, in first method, in structured light line thickness direction, calculate the barycentric coordinates of maximum gradation value pixel, as the lines coordinate.In the second method, in structured light line thickness direction, directly calculate the centre coordinate of live width, as the coordinate of lines.By column the coordinate that calculates the structured light lines respectively after the alignment of projected image and benchmark image.If be the lattice-like pattern, then surpass the situation of a pixel for every profile in the dot matrix, calculate the barycentric coordinates of doing weight by gray threshold, as the coordinate of point.
B. utilize the lines array of pixels to calculate the displacement △ X of lines
n(i, j)=P
n(i, j)-Q
n(i, j).Can be with the shift value of two structured light lines alignment coordinates computed by column.The side-play amount S of lines and △ X in the target object zone among Fig. 3
n(i, j) is corresponding.
C. utilize the ultimate principle of structural light measurement, data processing unit is △ X by following formula computation structure light pattern shift value
nThe distance z of impact point;
Wherein, B is the distance at the center of imageing sensor and galvanometer device, and f is the equipment lens focus of described imageing sensor, and L is the distance of the standard flat at the benchmark image place chosen.The distance z of difference is the object dimensional cloud data that calculates in the projection pattern.
Because B, f can pre-set for target object and the distance of different sizes with L in the above-mentioned formula, therefore the particular value of X ' for setting in the above-mentioned formula.When needs during for different target object project organization light pattern, only need to adjust the mirror status of one dimension or 2-D vibration mirror, just can obtain different structured lights, for example linear array light or dot matrix light, can also adjust the line-spacing of linear array light, thereby be convenient in the projected image that gathers, identify the zone at target object place.
In sum, the system and method for reconstructing surface of object of the present invention, the some light or the linear light that at first light source irradiation are gone out shine on the specular-reflection unit.By controlling the specular-reflection unit rotation and being offset light reflection to the target area, form structured light patterns.When utilizing imageing sensor to obtain without object on the internal standard plane, target area the pattern of structured light as reference pattern.Behind object target approach zone, the view field of the exposure area of recycling imageing sensor control chart image-position sensor and the structured light of described specular-reflection unit reflection is synchronous, obtains the structured light projection pattern at least part of zone in the target area.Store data into data processing unit, calculate the displacement information of two kinds of patterns, then utilize principle of triangulation to calculate the range data of object.This range data can real-time update, thereby obtains the real time kinematics data of moving object.And adopt galvanometer mode of the present invention, just can be to same set of equipment, adjustment projects the pattern of structured light according to diverse ways.The method and system can be applied in the real-time reconstruction of object, can use in the field of man-machine interaction, such as virtual mouse, the man-machine interaction of body-building, the man-machine interface of game, the detection of mobile device objects in front, human motion analysis, distance estimations and warning, intrusion alarm etc. aspect.
The present invention is described according to specific embodiment, but it will be understood by those skilled in the art that when not breaking away from the scope of the invention, can carry out various variations and be equal to replacement.In addition, for adapting to specific occasion or the material of the technology of the present invention, can carry out many modifications and not break away from its protection domain the present invention.Therefore, the present invention is not limited to specific embodiment disclosed herein, and comprises that all drop into the embodiment of claim protection domain.