CN114383543A - Single CCD double prism binocular stereo vision sensor - Google Patents

Single CCD double prism binocular stereo vision sensor Download PDF

Info

Publication number
CN114383543A
CN114383543A CN202111525330.9A CN202111525330A CN114383543A CN 114383543 A CN114383543 A CN 114383543A CN 202111525330 A CN202111525330 A CN 202111525330A CN 114383543 A CN114383543 A CN 114383543A
Authority
CN
China
Prior art keywords
prism
vision sensor
binocular
single ccd
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111525330.9A
Other languages
Chinese (zh)
Other versions
CN114383543B (en
Inventor
许燕玲
郑洵智
陈善本
张华军
闫俊达
侯震
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jiaotong University
Original Assignee
Shanghai Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Jiaotong University filed Critical Shanghai Jiaotong University
Priority to CN202111525330.9A priority Critical patent/CN114383543B/en
Publication of CN114383543A publication Critical patent/CN114383543A/en
Application granted granted Critical
Publication of CN114383543B publication Critical patent/CN114383543B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/30—Measuring arrangements characterised by the use of optical techniques for measuring roughness or irregularity of surfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/97—Determining parameters from multiple pictures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P10/00—Technologies related to metal processing
    • Y02P10/25—Process efficiency

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

The invention provides a WAAM molten pool three-dimensional reconstruction method, which designs a binocular stereoscopic vision sensor capable of adjusting the length of a base line based on prism beam splitting, wherein the sensor utilizes a single video camera and a prism to achieve the purpose of changing the length of the base line by adjusting the distance between the prism and the optical center of the camera, and the flexibility of a shooting angle and a shooting distance is increased. Compared with a traditional binocular stereoscopic vision sensor formed by two left and right parallel cameras, the binocular stereoscopic vision sensor has the advantages of short base length, good image acquisition synchronism and small occupied space, and is particularly suitable for monitoring the process of robot electric arc additive manufacturing. In addition, the designed sensor has compact structure and beautiful appearance.

Description

Single CCD double prism binocular stereo vision sensor
Technical Field
The invention relates to a single CCD double-prism binocular stereoscopic vision sensor.
Background
The key to GMA-AM formation quality control is to keep the inner surface flat and to make the surface height of each layer consistent with the height pre-planned by the slicing model. In the process of stacking the cladding road layer, if constant standard process parameters are adopted, the heat dissipation condition is gradually worsened along with the continuous increase of the stacking height, and the wettability between the liquid metal and the solidified welding bead of the previous layer is improved, so that the height of the cladding road is reduced, the width is increased, and the cladding road becomes flatter. This not only causes the inner surface of the workpiece to be rough and uneven, and the overall forming quality to be degraded, but also may cause the risk of 'gun collision' with the stacking of the height errors. Therefore, the key to controlling the quality of GMA-AM formation is the monitoring of the width, height and surface flatness of the single cladding pass per layer during the build-up process. Most of the existing methods monitor a molten pool and extract the width of the molten pool based on a monocular camera, but height information cannot be obtained through a single visual angle image. In other methods, a linear laser structured light system is used for three-dimensional reconstruction of a welded solidified cladding channel, and the width and the height of the cladding channel are extracted through a point cloud processing algorithm.
Disclosure of Invention
The invention aims to provide a single CCD double-prism binocular stereoscopic vision sensor.
In order to solve the above problems, the present invention provides a single CCD dual-prism binocular stereoscopic vision sensor, comprising:
arranging a single CCD double-prism binocular stereoscopic vision sensor;
calibrating and correcting polar lines of the set single CCD double-prism binocular stereoscopic vision sensor;
based on the calibrated single CCD double-prism binocular stereoscopic vision sensor with epipolar line correction, the molten pool image acquisition and the stereoscopic matching are carried out, so that the WAAM molten pool three-dimensional reconstruction is realized.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
adopt the nylon materials, 3D prints and makes the main part that holds the CCD camera.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
the device is characterized in that an aluminum alloy sheet is adopted, a U-shaped groove structure used for sliding and adjusting the distance between the double prisms is formed at the front end of the single CCD double-prism binocular stereoscopic vision sensor in an assembling mode, the loading platform is fixed through screwing and unscrewing hexagon screws on two sides of the U-shaped groove structure, and the loading platform horizontally moves in a guide rail, so that the distance between the prism and a CCD camera can be adjusted.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
the single CCD double-prism binocular stereoscopic vision sensor is connected with the universal joint through a dovetail groove, and the dovetail groove is made of high-strength aluminum alloy.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
and the single CCD double-prism binocular stereoscopic vision sensor is connected with the robot flange plate through a universal joint.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
the optical filter is additionally arranged by adopting a drawer structure and is connected to the light hole of the CCD camera through the boss.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
and protective glass is arranged at the foremost end of the single CCD double-prism binocular stereoscopic vision sensor through a clamping groove.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
determining a deflection angle delta according to the size of a molten pool and the measurement distance;
based on the determined deflection angle δ, a corresponding prism base angle magnitude α is determined.
Further, in the above method, calibrating and correcting the polar line of the set single CCD double prism binocular stereo vision sensor, includes:
firstly, dividing an acquired complete binocular image from the middle into a left image and a right image with the same resolution; secondly, adjusting a corner response threshold value, and accurately extracting corner coordinates in the checkerboard; then, system calibration is carried out, and calibration of the binocular system is divided into two steps: the first step is to utilize Zhangyingyou calibration method to calibrate the internal parameters and external parameters of the left camera and the right camera respectively, and the second step is to utilize the calibration result in the first step to optimize and solve the rotation and translation of the left camera and the right camera;
epipolar rectification for a calibrated binocular system.
Further, in the above method, based on the single CCD biprism binocular stereo vision sensor after calibration and epipolar line correction, molten pool image acquisition and stereo matching are performed to realize WAAM molten pool three-dimensional reconstruction, including:
collecting a molten pool image by using a binocular stereoscopic vision sensor after calibration and polar line correction, wherein a 660nm filter is adopted to filter out redundant arc light, and a complete and clear binocular molten pool image pair is collected under the condition of only utilizing arc light irradiation by adjusting the size of an aperture and the exposure time;
and carrying out dense stereo matching on the acquired binocular molten pool image pair.
Compared with the prior art, the binocular stereoscopic vision sensor capable of adjusting the length of the base line based on prism beam splitting is designed, the sensor utilizes a single video camera and one prism, the purpose of changing the length of the base line is achieved by adjusting the distance between the prism and the optical center of the camera, and the flexibility of the shooting angle and the shooting distance is improved. Compared with a traditional binocular stereoscopic vision sensor formed by two left and right parallel cameras, the binocular stereoscopic vision sensor has the advantages of short base length, good image acquisition synchronism and small occupied space, and is particularly suitable for monitoring the process of robot electric arc additive manufacturing. In addition, the designed sensor has compact structure and beautiful appearance.
Secondly, the binocular stereo vision sensor is calibrated by the camera, the relative poses of the left camera and the right camera are solved, Bonguet polar line correction is carried out by utilizing the calibration result, and the subsequent stereo matching parallax search is simplified from two dimensions to one dimension.
Thirdly, the invention provides a two-stage stereo matching algorithm, firstly, a semi-global matching algorithm (SGM) is used for matching, and the error matching of the weak texture region is eliminated; and then, completing the incomplete depth map by using a depth map completing method based on depth learning so as to obtain a complete three-dimensional point cloud model of the molten pool. The algorithm provided is fast and can meet the real-time requirement. The complete technical flow proposed is shown in fig. 1.
Drawings
FIG. 1 is a complete three-dimensional reconstruction technique flow of a binocular stereo WAAM molten pool according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an arc additive manufacturing hardware system based on binocular stereo vision sensing according to an embodiment of the invention;
FIG. 3 is a schematic diagram of a binocular stereo vision sensor solid works according to an embodiment of the present invention;
FIG. 4 is an optical schematic of a prism geometry according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of two refractions of light rays passing through a prism according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of an equivalent binocular stereo vision sensor in accordance with an embodiment of the present invention;
FIG. 7 is a schematic diagram of a portion of a calibration image in accordance with an embodiment of the present invention;
FIG. 8 is a schematic diagram of the corner detection and re-projection results of the left and right cameras according to an embodiment of the present invention;
FIG. 9 is a schematic representation of a stereo image pair before and after epipolar rectification in accordance with an embodiment of the present invention;
FIG. 10 is a schematic representation of a single-pass multi-layer differential layer binocular fused pool image pair in accordance with an embodiment of the present invention;
FIG. 11 is a flow chart of SGM algorithm stereo matching and post-processing according to an embodiment of the present invention;
FIG. 12a is a molten pool disparity map obtained by SGM algorithm according to an embodiment of the present invention;
fig. 12b is an embodiment of the present invention. Schematic diagram of 3D point cloud model of the surface of the molten pool.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
The invention firstly designs a binocular stereoscopic vision sensor which is based on prism beam splitting and can adjust the length of a base line, the sensor utilizes a single video camera and a prism, the purpose of changing the length of the base line is achieved by adjusting the distance between the prism and the optical center of the camera, and the flexibility of shooting angle and shooting distance is increased. Compared with a traditional binocular stereoscopic vision sensor formed by two left and right parallel cameras, the binocular stereoscopic vision sensor has the advantages of short base length, good image acquisition synchronism and small occupied space, and is particularly suitable for monitoring the process of robot electric arc additive manufacturing. In addition, the designed sensor has compact structure and beautiful appearance.
Secondly, the binocular stereo vision sensor is calibrated by the camera, the relative poses of the left camera and the right camera are solved, Bonguet polar line correction is carried out by utilizing the calibration result, and the subsequent stereo matching parallax search is simplified from two dimensions to one dimension.
Thirdly, the invention provides a two-stage stereo matching algorithm, firstly, a semi-global matching algorithm (SGM) is used for matching, and the error matching of the weak texture region is eliminated; and then, completing the incomplete depth map by using a depth map completing method based on depth learning so as to obtain a complete three-dimensional point cloud model of the molten pool. The algorithm provided is fast and can meet the real-time requirement. The complete technical flow proposed is shown in fig. 1.
Electric arc additive manufacturing system based on binocular stereoscopic vision sensing
1. Arc additive manufacturing hardware system
Fig. 2 is a diagram showing parts of an arc additive manufacturing hardware system based on binocular stereo vision sensing. The system consists of an industrial robot, a robot control cabinet, an industrial personal computer, a welding machine, a protective gas and a binocular stereo vision sensor. The industrial personal computer is an information acquisition and processing core of the WAAM system, and can simultaneously send coordinates of each point on a robot moving path which is planned in advance to the robot control cabinet and receive the pose state of the robot in real time. The industrial personal computer and the control cabinet establish communication through a network, and the control cabinet and the welding machine also establish communication relation. Actually, when the electric arc additive manufacturing is executed, the welding machine adaptively adjusts process parameters according to the position and the real-time state of a welding gun so as to achieve a better welding effect. The binocular stereoscopic vision sensor is fixed on a flange plate of a sixth shaft of the robot through a universal joint and moves synchronously along with the robot. Image signals acquired by the vision sensor are transmitted to image acquisition software of the industrial personal computer through a network, and key parameters such as image acquisition frame rate, exposure time and the like can be dynamically adjusted through the image acquisition software so as to improve image acquisition quality. Because the image acquisition software provides a large number of API interfaces for secondary development, the acquired image can be conveniently processed in real time.
2. Binocular stereo vision sensor
The binocular stereo vision sensor is the 'eye' of the robot and has significance in the whole system, and a design drawing and a physical drawing are shown in figure 3. During actual image acquisition, the position of the sensor CCD is kept still, and the base length is adjusted by moving the prism loading platform in parallel along the optical axis direction of the camera, so that the binocular molten pool under various working conditions such as different molten pool sizes, different image acquisition distances and the like can be imaged completely.
Besides the advantages of the variable base length, the sensor also has the following characteristics:
(1) compact structure, small occupied space and low production cost. The main body part of the sensor for containing the CCD camera is formed by 3D printing of a nylon material, and is light in weight and good in heat dissipation; the U-shaped groove structure for slidably adjusting the distance between the double prisms is formed by assembling aluminum alloy sheets at the front end of the binocular stereoscopic vision sensor, the loading platform is fixed by screwing and unscrewing the hexagon screws on the two sides, and the loading platform horizontally moves in the guide rail so as to adjust the distance between the prisms and the CCD camera, and the rigidity required by fixing of an optical system is ensured by the aluminum alloy sheets.
(2) The sensor is connected with the universal joint through the dovetail groove, and the dovetail groove is made of high-strength aluminum alloy, so that the enough rigidity of the connection part is ensured. The sensor is connected with the robot flange plate through the universal joint, and the image acquisition angle and the measurement distance can be conveniently adjusted so as to find out the acquisition angle most suitable for binocular stereo vision imaging.
(3) The original installation scheme of the optical filter before the optical filter is embedded into the lens is changed, the optical filter is additionally installed by adopting a drawer structure and is connected to a light hole of the CCD camera through a boss, the optical filter is taken out during focusing and calibration, and the optical filter is installed during collecting images of a molten pool, so that the optical filter can be installed without disassembling the CCD camera after a system is calibrated, and the change of the relative position relationship of the left camera and the right camera caused by the displacement of the CCD camera is prevented.
(4) And protective glass is arranged at the foremost end of the sensor through a clamping groove so as to prevent a large amount of splash generated in the welding process from damaging the surface of the biprism. During actual work, the part of the prism arranged at the front end of the sensor is covered by the aluminum alloy sheet, so that light can be collected by the camera only after passing through the double prisms, and the light path completely accords with the pre-designed light path.
Second, concrete technical implementation
1. Binocular stereoscopic vision sensor design based on single CCD prism light splitting
By using the principle of secondary refraction of the prism, a single image can be manually divided into a left virtual image and a right virtual image under the condition of only using a single CCD to acquire images, so that a binocular image pair is formed. Ideally, when the bottom edge of the prism is aligned with the optical center of the camera and the bottom surface of the prism is parallel to the imaging plane, the system is self-correcting and becomes a standard binocular stereo vision sensor. The scheme utilizes the secondary refraction of the prism to the light, and the presented binocular image is a virtual image of the actual object. Because the presented stereo image pair is actually shot by the same CCD camera, the problem of inconsistent illumination caused by adopting the left CCD camera and the right CCD camera can be overcome to a certain extent, and the stereo matching algorithm robustness can be improved beneficially.
Fig. 4 is a geometrical optical schematic diagram of a binocular stereo vision sensor based on prism beam splitting. The prism coordinate system is defined as (O)p,Xp,Yp,Zp),OpIs the origin coordinate of the prism; the camera coordinate system is defined as (O, X, Y, Z), O being the camera optical center. Guarantee Z when placingpThe prism is coaxial with Z, namely the coordinate of the origin of the prism is ensured to pass through the optical axis of the camera; also ensure XpParallel to X, YpParallel to Y, the imaging plane of the camera is kept parallel to the bottom surface pi of the prism.
As shown in fig. 4, when an arbitrary three-dimensional point X is present in spacepThe emitted light will be deflected by an angle delta when passing through the prism, and the light respectively passes through two side surfaces pi after passing through the bottom surface of the prismrAnd pilWhen the prism deflects the light, the image formed on the imaging plane is equivalent to two virtual object points X on two sides of the original object pointrAnd XlAnd imaging at an imaging plane. The prism's deflecting effect on the light ray is the superposition of the two refracting effects of the prism's bottom and side surfaces on the light ray, and as shown in fig. 5, the magnitude of the deflecting angle δ is determined by the prism's geometric optics and the refractive index n and the base angle α of the prism.
From the law of refraction and the geometric relationship between the incident angle and the refraction angle shown in fig. 5, the basic relationship between the deflection angle δ of the prism for the light ray, the refractive index n of the prism, and the base angle of the prism can be derived as shown in formula (1).
Figure BDA0003410109830000071
The formula (1) gives the basis of the prism model selection: firstly, the deflection angle delta is determined according to the size of a molten pool and the measurement distance, and as the refractive index of the used glass is basically fixed, only the proper size alpha of the prism base angle is needed to be selected to meet the requirement on the deflection angle delta.
Recording arbitrary three-dimensional points XpCoordinate X ofp＝[Xp,Yp,Zp]TRight virtual object point coordinate Xr＝[Xpr,Ypr,Zpr]TLeft virtual object point coordinate Xl＝[Xpl,Ypl,Zpl]TFrom the above geometric relationship analysis, Xr、XlAnd three-dimensional point XpThe Y, Z direction coordinates are the same, and the X direction coordinates have the relation of the formula (2).
Xpr＝Xp+Zp tanδ
Xpl＝Xp-Zp tanδ
Ypr＝Ypl＝Yp
Zpr＝Zpl＝Zp (2)
Writing equation (2) into a matrix transformation form is shown in equation (3).
Xr＝TrXp Xl＝TlXp (3)
In the formula (I), the compound is shown in the specification,
Figure BDA0003410109830000081
the formula (3) shows that the coordinates of the left and right virtual object points formed by the two refractions of the prism can be obtained by transforming the original space points through a simple matrix.
Fixing the world coordinate system on the prism coordinate system without loss of generality, and deducing a space point XpLeft and right virtual points X formed after twice refraction by the prisml、XrTo the left and right image points m of the imaging planel，mrThe transmission projective transformation of (1). Substituting the formula (3) into the pinhole camera projection transformation equation to write two secondary XpRespectively directly converted to mlAnd mrIs shown in equation (4).
Figure BDA0003410109830000082
In the formula (I), the compound is shown in the specification,
Figure BDA0003410109830000083
Figure BDA0003410109830000084
and (4) constructing a projection matrix from any space point of the prism binocular stereo vision sensor to the left and right image points on the imaging plane. Wherein alpha isu，αv，u0，v0For camera reference, R and t are the rotational matrix and translation vector of the camera coordinate system relative to the prism coordinate system, respectively. Ideally, the bottom surface of the prism is parallel to the imaging plane of the camera, the center of the double prisms passes through the optical axis of the camera, and the rotation matrix R is I3×3Is a 3 × 3 unit array, t ═ 0,0, tz]T，tzThe distance from the prism center to the camera optical center.
According to the formula (4), the three-dimensional point X is expressed by homogeneous coordinatesp＝[Xp,Yp,Zp,1]TLeft and right image points m generated by projectionl＝[ul,vl,1]T,mr＝[ur,vr,1]TAnd (3) substituting the expanded results of the terms into a formula (4) to obtain a projection relation equation of the left and right image points and the three-dimensional points of the prism binocular stereo vision sensor as shown in a formula (5).
Figure BDA0003410109830000091
Figure BDA0003410109830000092
V is easily shown by the formulae (5) and (6)l＝vrThat is, in an ideal state, the homonymous point pairs of the left and right images of the binocular stereo vision sensor, which are equivalently formed by prism beam splitting, are positioned on the same horizontal line, no parallax in the vertical direction exists, and the binocular stereo vision sensor is obtained by adopting the horizontal arrangement of the double cameras and polar line correctionThe standard binocular stereo vision sensor. The horizontal parallax can be calculated by the formula (7).
Figure BDA0003410109830000093
Wherein Z ═ Zp+tzAnd represents the depth of a three-dimensional point in the camera coordinate system. The relation between the parallax and the three-dimensional point depth of the standard binocular stereo vision sensor constructed based on the prism is given by the formula (7), and the following conclusion can be drawn.
(1) When the three-dimensional point depth Z tends to be infinite, the parallax tends to be a fixed value d-2 alphautan δ, which is different from the conclusion of a stereo vision system consisting of two parallel placed cameras, which has a disparity that tends to 0 as the depth Z tends to infinity.
(2) When the depth Z of the three-dimensional point is fixed, the parallax d and the x-axis equivalent focal length alpha of the camerauThe size delta of the deflection angle of the prism to the light ray and the distance t between the center of the prism and the optical center of the camerazIt is related. When a camera (fixed internal reference) and a double-prism material (fixed refractive index n) are selected, increasing the base angle alpha is equivalent to increasing the deflection angle delta according to the relation between the prism deflection angle delta and the base angle alpha in the formula (1), and the parallax d is increased under the same depth; and the distance t from the prism to the optical center of the camera is increasedzThe parallax d is reduced.
Fig. 6 is an equivalent schematic diagram of the prism-split stereo vision system and the left and right camera stereo vision systems. Wherein O isLAnd ORThe two areas are respectively equivalent left and right camera optical centers, the area I and the area II are left camera view field ranges, the area I and the area III are right camera view field ranges, and the area I is a view field range in which the left camera and the right camera are overlapped, namely an effective area of the actual binocular stereo vision sensor. The base line length of the equivalent stereo vision system can be expressed as equation (8).
B＝2tz tanδ (8)
The formula (8) shows that when the prism parameters (refractive index n and base angle alpha) are determined, the distance t from the optical center of the camera is adjusted by adjusting the prismzIt is possible to construct a difference in the length of the outgoing base lineThe binocular stereo vision sensor. In the actual image acquisition process, the size of the molten pool can be changed according to different process conditions, and the measurement distance can be adjusted due to the limitation of physical space, so that the adoption of the stereoscopic vision system with variable base length increases the flexibility of image acquisition, and complete binocular molten pool images can be acquired in the visual field range when the distances of various molten pools and measurement distances are changed by adjusting the distance between the prism and the optical center of the camera. The method is equivalent to the theoretical basis of a binocular stereo vision sensor through the method of splitting light by the single-camera prism.
The experiment uses a large constant MER-200-14GM gray camera to shoot a molten pool, the resolution is 1628 multiplied by 1236, the size of a CCD sensor array is 1/1.8 inch (7.1mm multiplied by 5.3mm), and a lens uses a computer series f-35 mm fixed focus lens. From the above data, the half angle θ of the field angle of the camera in the horizontal direction can be calculatedw。
Figure BDA0003410109830000101
Theta was determined by substituting w of 5.3mm and f of 35mm for formula (9)w＝4.3°。
Firstly, the proper size of the bottom surface of the prism is selected, considering that the field of view of the camera is not large, the whole space of the sensor is limited, the physical size of the CCD camera is 29mm multiplied by 29mm, and the size of the double prisms with the bottom surface of 30mm multiplied by 30mm is selected to be close to the external size of the CCD, so that the object point refracted by the prism can fall on the CCD, the space can be saved, and the installation is convenient.
And secondly, selecting a proper base angle size alpha of the prism. Since the width of the molten pool of the collected CMT welding is between 5mm and 10mm, and the width of the molten pool is set to be l, the whole molten pool is positioned in the overlapping area of the left camera and the right camera, and the formula (10) is satisfied according to the similar triangular relation of the equivalent stereoscopic vision system in the figure 6.
Figure BDA0003410109830000102
By the formula (10),
Figure BDA0003410109830000103
in the formula ZpThe distance between the prism and the shot molten pool is mainly considered that the whole sensor is not too close to the molten pool, otherwise, the risk of collision between the sensor and the weld bead is increased along with the increase of the height of the additive layer. Here, Z is first assumed to bepThe material K9 for the prism, which has a refractive index of 1.52, is 20mm, and the prism base angle α is 15 ° in combination with the prism geometry optical expressions (1) and (10).
When the base angle alpha of the double prism is selected, the refraction deflection angle delta of the double prism is also uniquely determined, and according to the formula (8) and the figure 6, the base length of the equivalent binocular stereo vision sensor can only be changed by changing the distance t from the double prism to the optical center of the camerazBut is changed. If the width of the shot molten pool is small and the camera is close to the molten pool, t can be properly increasedzThe length of the base line is increased, the proportion of the molten pool in the visual field is improved, and meanwhile, the base line is increased according to the relation between the depth and the parallax, so that the measurement precision in the depth direction is improved. If the width of the shot molten pool is larger or the distance between the camera and the molten pool is farther, t is properly reducedzThe distance between the base lines is reduced, the overlapping area of the two equivalent cameras is increased, and the fact that a large molten pool can be completely displayed in a visual field after being split is guaranteed. The structural diagram and the physical diagram of the designed binocular stereo vision sensor are shown in figure 3.
2. Calibration and polar line correction of binocular stereoscopic vision sensor
The calibration meaning of the binocular stereo vision sensor is to respectively determine the internal parameters of the left camera and the right camera and the relative pose (rotation and translation) between the two cameras. The calibration result can be used for epipolar line correction, can also be used for obtaining a depth map from the parallax map through conversion, and finally obtains a three-dimensional point cloud model under a world coordinate system through triangulation. Because the actual length of the shot molten pool is within 10mm, in order to obtain higher calibration precision, a calibration plate with the side length of 10mm x 10mm and the checkerboard number of 6mm x 7mm is selected, and the actual measured molten pool width is equivalent. During calibration, the distance from the camera to the calibration plate is equivalent to the distance during actual shooting of the molten pool, the shooting angle from the camera to the calibration plate is changed by adjusting the pose of the robot, 12 binocular images are collected, and partial images are shown in fig. 5.
Calibration was performed using the stereoCameraCalibrator kit of MATLAB. First, the acquired complete binocular image is divided from the middle into left and right images of the same resolution. Then, adjusting the corner response threshold value, and accurately extracting the corner coordinates in the checkerboard, as shown in fig. 7, which are the detection results of the left and right camera corner, and the extraction precision of the corner coordinates determines the calibration precision. Then, system calibration is carried out, and calibration of the binocular system is divided into two steps: the first step is to calibrate the internal parameters and the external parameters of the left camera and the right camera respectively by using a Zhangyingyou calibration method, and the second step is to optimize and solve the rotation and the translation of the left camera and the right camera by using the calibration result in the first step. After the internal reference and pose of the two cameras are uniquely determined, the basic matrix F is also determined, and only when the relative position relation of the biprism distance cameras is adjusted, the calibration needs to be carried out again. The calibration results are shown in Table 1. The calibrated average re-projection error is 0.64 pixel, the re-projection result is shown in figure 8, the precision reaches the sub-pixel level, and the actual engineering requirement is met. The calibration results of the binocular stereo vision sensor are shown in table 1.
Figure BDA0003410109830000121
TABLE 1
For epipolar rectification of a calibrated binocular system, the simplest method is Bonguet algorithm, and the Bonguet algorithm minimizes the number of times of each re-projection in the left image and the right image by using a rotation matrix and a translation vector between binocular images and simultaneously maximizes an observation area. And the left and right homonymous points of the two image pairs after epipolar line correction are on the same horizontal line, so that parallax value search is only needed along the horizontal scanning line during stereo matching. The corrected molten pool image results are shown in fig. 9, and it can be seen that the homologous points of the left and right images are located on the same horizontal scanning line.
3. Weld pool image acquisition and stereo matching
And acquiring a molten pool image by using a calibrated binocular stereoscopic vision sensor after epipolar line correction. In order to prevent the influence of strong arc light in the welding process, a 660nm filter is adopted to filter out redundant arc light, and a complete and clear binocular molten pool image pair is acquired under the condition of only utilizing the arc light irradiation by adjusting the size of an aperture and the exposure time. For multi-layer single pass build-up, the binocular weld pool images of the different layers are shown in fig. 10. Welding current: 200A, welding speed: 24 cm/min. The fourth layer of the molten pool image is the clearest and rich in texture, and the molten pool image of the fourth layer is taken as an example for dense stereo matching.
The purpose of stereo matching is to find homonymous points on the left and right images, and then three-dimensional points can be recovered by a triangulation method. The most difficult point of analyzing the above molten pool images is that the matching is weak in texture, so that more mismatching is easy to generate. The semi-global matching method (SGM) is adopted for matching, the algorithm is proposed in 2005, has matching precision and efficiency, and is the most widely applied algorithm in the current engineering. The specific steps are shown in fig. 11.
As shown in fig. 12a, the disparity map of the left image obtained after matching shows that the disparity values of the weak texture regions are missing, but the overall trend is correct. We further use the neural network to complement the depth map to obtain a more complete depth map. After the completion of the depth map, the depth map is back projected to a three-dimensional space according to the internal parameters of the left camera, and a complete three-dimensional point cloud model of the molten pool can be obtained as shown in fig. 12 b.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims (10)

1. A WAAM molten pool three-dimensional reconstruction method is characterized by comprising the following steps:
arranging a single CCD double-prism binocular stereoscopic vision sensor;
calibrating and correcting polar lines of the set single CCD double-prism binocular stereoscopic vision sensor;
based on the calibrated single CCD double-prism binocular stereoscopic vision sensor with epipolar line correction, the molten pool image acquisition and the stereoscopic matching are carried out, so that the WAAM molten pool three-dimensional reconstruction is realized.
2. The WAAM weld pool three-dimensional reconstruction method according to claim 1, wherein the setting of the single CCD double prism binocular stereo vision sensor comprises:
adopt the nylon materials, 3D prints and makes the main part that holds the CCD camera.
3. The WAAM weld pool three-dimensional reconstruction method according to claim 1, wherein the setting of the single CCD double prism binocular stereo vision sensor comprises:
the device is characterized in that an aluminum alloy sheet is adopted, a U-shaped groove structure used for sliding and adjusting the distance between the double prisms is formed at the front end of the single CCD double-prism binocular stereoscopic vision sensor in an assembling mode, the loading platform is fixed through screwing and unscrewing hexagon screws on two sides of the U-shaped groove structure, and the loading platform horizontally moves in a guide rail, so that the distance between the prism and a CCD camera can be adjusted.
4. The WAAM weld pool three-dimensional reconstruction method according to claim 1, wherein the setting of the single CCD double prism binocular stereo vision sensor comprises:
the single CCD double-prism binocular stereoscopic vision sensor is connected with the universal joint through a dovetail groove, and the dovetail groove is made of high-strength aluminum alloy.
5. The WAAM weld pool three-dimensional reconstruction method according to claim 1, wherein the setting of the single CCD double prism binocular stereo vision sensor comprises:
and the single CCD double-prism binocular stereoscopic vision sensor is connected with the robot flange plate through a universal joint.
6. The WAAM weld pool three-dimensional reconstruction method according to claim 1, wherein the setting of the single CCD double prism binocular stereo vision sensor comprises:
the optical filter is additionally arranged by adopting a drawer structure and is connected to the light hole of the CCD camera through the boss.
7. The WAAM weld pool three-dimensional reconstruction method according to claim 1, wherein the setting of the single CCD double prism binocular stereo vision sensor comprises:
and protective glass is arranged at the foremost end of the single CCD double-prism binocular stereoscopic vision sensor through a clamping groove.
8. The WAAM weld pool three-dimensional reconstruction method according to claim 1, wherein the setting of the single CCD double prism binocular stereo vision sensor comprises:
determining a deflection angle delta according to the size of a molten pool and the measurement distance;
based on the determined deflection angle δ, a corresponding prism base angle magnitude α is determined.
9. The WAAM molten pool three-dimensional reconstruction method according to claim 1, wherein the calibration and epipolar line correction of the set single CCD double prism binocular stereo vision sensor comprises the following steps:
firstly, dividing an acquired complete binocular image from the middle into a left image and a right image with the same resolution; secondly, adjusting a corner response threshold value, and accurately extracting corner coordinates in the checkerboard; then, system calibration is carried out, and calibration of the binocular system is divided into two steps: the first step is to utilize Zhangyingyou calibration method to calibrate the internal parameters and external parameters of the left camera and the right camera respectively, and the second step is to utilize the calibration result in the first step to optimize and solve the rotation and translation of the left camera and the right camera;
epipolar rectification for a calibrated binocular system.
10. The WAAM molten pool three-dimensional reconstruction method according to claim 1, wherein the molten pool image acquisition and the stereo matching are performed based on a single CCD double-prism binocular stereo vision sensor after calibration and epipolar line correction to realize the WAAM molten pool three-dimensional reconstruction, which comprises the following steps:
collecting a molten pool image by using a binocular stereoscopic vision sensor after calibration and polar line correction, wherein a 660nm filter is adopted to filter out redundant arc light, and a complete and clear binocular molten pool image pair is collected under the condition of only utilizing arc light irradiation by adjusting the size of an aperture and the exposure time;
and carrying out dense stereo matching on the acquired binocular molten pool image pair.
CN202111525330.9A 2021-12-14 2021-12-14 WAAM molten pool three-dimensional reconstruction method Active CN114383543B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111525330.9A CN114383543B (en) 2021-12-14 2021-12-14 WAAM molten pool three-dimensional reconstruction method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111525330.9A CN114383543B (en) 2021-12-14 2021-12-14 WAAM molten pool three-dimensional reconstruction method

Publications (2)

Publication Number Publication Date
CN114383543A true CN114383543A (en) 2022-04-22
CN114383543B CN114383543B (en) 2022-12-27

Family

ID=81196466

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111525330.9A Active CN114383543B (en) 2021-12-14 2021-12-14 WAAM molten pool three-dimensional reconstruction method

Country Status (1)

Country Link
CN (1) CN114383543B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115063567A (en) * 2022-08-19 2022-09-16 中国石油大学(华东) Three-dimensional light path analysis method of double-prism monocular stereoscopic vision system
CN117664862A (en) * 2024-01-31 2024-03-08 宁德时代新能源科技股份有限公司 Tab detection system and tab detection method

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103528540A (en) * 2013-10-11 2014-01-22 河北科技大学 Prism-based single-camera stereoscopic vision imaging device applied to sensing of welding pool
CN107796328A (en) * 2017-09-21 2018-03-13 西南交通大学 Metal increasing material manufacturing Pool three-dimensional visual sensor and detection method
CN108015455A (en) * 2017-11-01 2018-05-11 清华大学 Multifunctional vision sensor device based on recombination laser structure light
CN109259717A (en) * 2018-08-27 2019-01-25 彭波 A kind of stereo endoscope and endoscope measurement method
CN109668509A (en) * 2019-01-18 2019-04-23 南京理工大学 Based on biprism single camera three-dimensional measurement industrial endoscope system and measurement method
US20200189114A1 (en) * 2018-12-17 2020-06-18 Divergent Technologies, Inc. System and method for positioning one or more robotic apparatuses

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103528540A (en) * 2013-10-11 2014-01-22 河北科技大学 Prism-based single-camera stereoscopic vision imaging device applied to sensing of welding pool
CN107796328A (en) * 2017-09-21 2018-03-13 西南交通大学 Metal increasing material manufacturing Pool three-dimensional visual sensor and detection method
CN108015455A (en) * 2017-11-01 2018-05-11 清华大学 Multifunctional vision sensor device based on recombination laser structure light
CN109259717A (en) * 2018-08-27 2019-01-25 彭波 A kind of stereo endoscope and endoscope measurement method
US20200189114A1 (en) * 2018-12-17 2020-06-18 Divergent Technologies, Inc. System and method for positioning one or more robotic apparatuses
CN109668509A (en) * 2019-01-18 2019-04-23 南京理工大学 Based on biprism single camera three-dimensional measurement industrial endoscope system and measurement method

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115063567A (en) * 2022-08-19 2022-09-16 中国石油大学(华东) Three-dimensional light path analysis method of double-prism monocular stereoscopic vision system
CN115063567B (en) * 2022-08-19 2022-10-28 中国石油大学(华东) Three-dimensional light path analysis method of double-prism monocular stereoscopic vision system
CN117664862A (en) * 2024-01-31 2024-03-08 宁德时代新能源科技股份有限公司 Tab detection system and tab detection method

Also Published As

Publication number Publication date
CN114383543B (en) 2022-12-27

Similar Documents

Publication Publication Date Title
CN107121109B (en) structural optical parameter calibration device and method based on front coated plane mirror
CN114383543B (en) WAAM molten pool three-dimensional reconstruction method
CN109029299B (en) Dual-camera measuring device and method for butt joint corner of cabin pin hole
CA3068659A1 (en) Augmented reality displays with active alignment and corresponding methods
CN103776419B (en) A kind of binocular distance measurement method improving measurement range
CN111080705B (en) Calibration method and device for automatic focusing binocular camera
Strelow et al. Precise omnidirectional camera calibration
CN111009030A (en) Multi-view high-resolution texture image and binocular three-dimensional point cloud mapping method
JP2008096162A (en) Three-dimensional distance measuring sensor and three-dimensional distance measuring method
CN102155920B (en) Welding seam measuring method based on micro field depth
CN107478203A (en) A kind of 3D imaging devices and imaging method based on laser scanning
CN103528540A (en) Prism-based single-camera stereoscopic vision imaging device applied to sensing of welding pool
CN109636859A (en) A kind of scaling method of the 3D vision detection based on one camera
CN110657750B (en) Detection system and method for passivation of cutting edge of cutter
CN112381888A (en) Dynamic compensation method for H-shaped steel cutting path
CN116804537A (en) Binocular range finding system and method
Pachidis et al. Pseudo-stereo vision system: a detailed study
CN103317241A (en) Laser welding abutted seam measuring system and method based on plano-convex cylindrical lens
CN112419418A (en) Positioning method based on camera mechanical aiming
CN115713545A (en) Bionic binocular vision tracking device and method driven by optical wedge set
CN110533702B (en) Underwater binocular vision stereo matching method based on optical field multilayer refraction model
Zeng et al. A 3D passive optical localization system based on binocular infrared cameras
JP5610579B2 (en) 3D dimension measuring device
CN115077468B (en) Zoom ranging method and device
CN111553955B (en) Multi-camera three-dimensional system and calibration method thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant