CN114383543B - WAAM molten pool three-dimensional reconstruction method - Google Patents
WAAM molten pool three-dimensional reconstruction method Download PDFInfo
- Publication number
- CN114383543B CN114383543B CN202111525330.9A CN202111525330A CN114383543B CN 114383543 B CN114383543 B CN 114383543B CN 202111525330 A CN202111525330 A CN 202111525330A CN 114383543 B CN114383543 B CN 114383543B
- Authority
- CN
- China
- Prior art keywords
- vision sensor
- prism
- stereoscopic vision
- binocular
- molten pool
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/30—Measuring arrangements characterised by the use of optical techniques for measuring roughness or irregularity of surfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/97—Determining parameters from multiple pictures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P10/00—Technologies related to metal processing
- Y02P10/25—Process efficiency
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention provides a WAAM molten pool three-dimensional reconstruction method, which designs a binocular stereoscopic vision sensor capable of adjusting the length of a base line based on prism beam splitting, wherein the sensor utilizes a single video camera and a prism to achieve the purpose of changing the length of the base line by adjusting the distance between the prism and the optical center of the camera, and the flexibility of a shooting angle and a shooting distance is increased. Compared with a traditional binocular stereoscopic vision sensor formed by two left and right parallel cameras, the binocular stereoscopic vision sensor has the advantages of short base length, good image acquisition synchronism and small occupied space, and is particularly suitable for monitoring the process of robot electric arc additive manufacturing. In addition, the designed sensor has compact structure and beautiful appearance.
Description
Technical Field
The invention relates to a WAAM molten pool three-dimensional reconstruction method.
Background
The key to GMA-AM formation quality control is maintaining the flatness of the inner surface and making the surface height of each layer consistent with the height pre-planned by the slicing model. In the process of stacking the cladding road layer, if constant standard process parameters are adopted, the heat dissipation condition is gradually worsened along with the continuous increase of the stacking height, and the wettability between the liquid metal and the solidified welding bead of the previous layer is improved, so that the height of the cladding road is reduced, the width is increased, and the cladding road becomes flatter. This not only causes the inner surface of the workpiece to be rugged and the overall forming quality to be degraded, but also may create a "gun strike" risk with the build-up of height tolerances. Therefore, the key to controlling the quality of GMA-AM formation is the monitoring of the width, height and surface flatness of the single cladding pass per layer during the build-up process. Most of the existing methods monitor a molten pool and extract the width of the molten pool based on a monocular camera, but height information cannot be obtained through a single visual angle image. In other methods, a linear laser structured light system is used for three-dimensional reconstruction of a welded solidified cladding channel, and the width and the height of the cladding channel are extracted through a point cloud processing algorithm.
Disclosure of Invention
The invention aims to provide a WAAM molten pool three-dimensional reconstruction method.
In order to solve the above problems, the present invention provides a WAAM molten pool three-dimensional reconstruction method, which comprises:
arranging a single CCD double-prism binocular stereoscopic vision sensor;
calibrating and correcting polar lines of the set single CCD double-prism binocular stereoscopic vision sensor;
based on the calibrated single CCD double-prism binocular stereoscopic vision sensor after epipolar line correction, molten pool image acquisition and stereoscopic matching are carried out so as to realize WAAM molten pool three-dimensional reconstruction.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
adopt the nylon materials, 3D prints and makes the main part that holds the CCD camera.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
the device is characterized in that an aluminum alloy sheet is adopted, a U-shaped groove structure used for sliding and adjusting the distance between the double prisms is formed at the front end of the single CCD double-prism binocular stereoscopic vision sensor in an assembling mode, the loading platform is fixed through screwing and unscrewing hexagon screws on two sides of the U-shaped groove structure, and the loading platform horizontally moves in a guide rail, so that the distance between the prism and a CCD camera can be adjusted.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
the single CCD double-prism binocular stereoscopic vision sensor is connected with the universal joint through a dovetail groove, and the dovetail groove is made of high-strength aluminum alloy.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
and the single CCD double-prism binocular stereoscopic vision sensor is connected with the robot flange plate through a universal joint.
Further, in the above method, the setting of the single CCD double prism binocular stereoscopic vision sensor includes:
the optical filter is additionally arranged by adopting a drawer structure and is connected to the light hole of the CCD camera through the boss.
Further, in the above method, a single CCD double prism binocular stereo vision sensor is provided, which includes:
and protective glass is arranged at the foremost end of the single CCD double-prism binocular stereoscopic vision sensor through a clamping groove.
Further, in the above method, a single CCD double prism binocular stereo vision sensor is provided, which includes:
determining a deflection angle delta according to the size of a molten pool and the measurement distance;
based on the determined deflection angle δ, a corresponding prism base angle magnitude α is determined.
Further, in the above method, calibrating and epipolar line correcting the set single CCD double prism binocular stereo vision sensor, includes:
firstly, dividing an acquired complete binocular image from the middle into a left image and a right image with the same resolution; secondly, adjusting a corner response threshold value, and accurately extracting corner coordinates in the checkerboard; then, system calibration is carried out, and calibration of the binocular system is divided into two steps: the method comprises the following steps that firstly, an internal parameter and an external parameter of a left camera and a right camera are respectively calibrated by using a Zhang Zhengyou calibration method, and secondly, the rotation and the translation of the left camera and the right camera are optimally solved by using a calibration result in the first step;
and correcting epipolar lines of the calibrated binocular system.
Further, in the above method, based on the calibrated and epipolar-corrected single CCD double prism binocular stereo vision sensor, molten pool image acquisition and stereo matching are performed to realize WAAM molten pool three-dimensional reconstruction, including:
collecting a molten pool image by using a calibrated and epipolar-corrected binocular stereo vision sensor, wherein a 660nm optical filter is adopted to filter out redundant arc light, and a complete and clear binocular molten pool image pair is collected under the condition of only utilizing arc light irradiation by adjusting the size of an aperture and the exposure time;
and carrying out dense stereo matching on the acquired binocular molten pool image pair.
Compared with the prior art, the binocular stereoscopic vision sensor capable of adjusting the length of the base line based on prism beam splitting is designed, the sensor utilizes a single video camera and one prism, the purpose of changing the length of the base line is achieved by adjusting the distance between the prism and the optical center of the camera, and the flexibility of the shooting angle and the shooting distance is improved. Compared with a traditional binocular stereoscopic vision sensor formed by two left and right parallel cameras, the binocular stereoscopic vision sensor has the advantages of short base length, good image acquisition synchronism and small occupied space, and is particularly suitable for monitoring the process of robot electric arc additive manufacturing. In addition, the designed sensor has compact structure and beautiful appearance.
Secondly, the binocular stereo vision sensor is calibrated by the camera, the relative poses of the left camera and the right camera are solved, bonguet polar line correction is carried out by utilizing the calibration result, and the subsequent stereo matching parallax search is simplified from two dimensions to one dimension.
Thirdly, the invention provides a two-stage stereo matching algorithm, firstly, a semi-global matching algorithm (SGM) is used for matching, and the error matching of the weak texture region is eliminated; and then, completing the incomplete depth map by using a depth map completing method based on deep learning so as to obtain a complete three-dimensional point cloud model of the molten pool. The algorithm provided is fast and can meet the real-time requirement. The complete technical flow proposed is shown in fig. 1.
Drawings
FIG. 1 is a complete three-dimensional reconstruction technique flow of a binocular stereo WAAM molten pool according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an arc additive manufacturing hardware system based on binocular stereo vision sensing according to an embodiment of the invention;
FIG. 3 is a schematic diagram of a binocular stereo vision sensor solid works according to an embodiment of the present invention;
FIG. 4 is an optical schematic of a prism geometry according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of two refractions of light rays passing through a prism according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of an equivalent binocular stereo vision sensor of one embodiment of the present invention;
FIG. 7 is a schematic illustration of a portion of a calibration image in accordance with an embodiment of the present invention;
FIG. 8 is a schematic diagram of the corner detection and re-projection results of the left and right cameras according to an embodiment of the present invention;
FIG. 9 is a schematic representation of a stereo image pair before and after epipolar rectification in accordance with an embodiment of the present invention;
FIG. 10 is a schematic representation of a single-pass multi-layer differential layer binocular fused pool image pair in accordance with an embodiment of the present invention;
FIG. 11 is a flow chart of SGM algorithm stereo matching and post-processing according to an embodiment of the present invention;
FIG. 12a is a molten pool disparity map obtained by SGM algorithm according to an embodiment of the present invention;
fig. 12b is an embodiment of the present invention. Schematic diagram of 3D point cloud model of the surface of the molten pool.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
The invention firstly designs a binocular stereo vision sensor which is based on prism beam splitting and can adjust the length of a base line, the sensor utilizes a single camera and a prism to achieve the purpose of changing the length of the base line by adjusting the distance between the prism and the optical center of the camera, and the flexibility of the shooting angle and the shooting distance is increased. Compared with a traditional binocular stereoscopic vision sensor formed by two left and right parallel cameras, the binocular stereoscopic vision sensor has the advantages of short base length, good image acquisition synchronism and small occupied space, and is particularly suitable for monitoring the process of robot electric arc additive manufacturing. In addition, the designed sensor has compact structure and beautiful appearance.
Secondly, the binocular stereo vision sensor is calibrated by the method, the relative poses of the left camera and the right camera are solved, bonguet epipolar line correction is carried out by utilizing the calibration result, and the subsequent stereo matching parallax search is simplified from two dimensions to one dimension.
Thirdly, the invention provides a two-stage stereo matching algorithm, firstly, a semi-global matching algorithm (SGM) is used for matching, and the error matching of the weak texture region is eliminated; and then, completing the incomplete depth map by using a depth map completing method based on deep learning so as to obtain a complete three-dimensional point cloud model of the molten pool. The algorithm provided is fast and can meet the real-time requirement. The complete technical flow proposed is shown in fig. 1. 1. Electric arc additive manufacturing system based on binocular stereoscopic vision sensing
1. Arc additive manufacturing hardware system
Fig. 2 is a diagram showing parts of an arc additive manufacturing hardware system based on binocular stereo vision sensing. The system consists of an industrial robot, a robot control cabinet, an industrial personal computer, a welding machine, a protective gas and a binocular stereo vision sensor. The industrial personal computer is an information acquisition and processing core of the WAAM system, and can simultaneously send coordinates of each point on a robot moving path which is planned in advance to the robot control cabinet and receive the pose state of the robot in real time. The industrial personal computer and the control cabinet establish communication through a network, and the control cabinet and the welding machine also establish communication relation. Actually, when the electric arc additive manufacturing is executed, the welding machine adaptively adjusts process parameters according to the position and the real-time state of a welding gun so as to achieve a better welding effect. The binocular stereoscopic vision sensor is fixed on a flange plate of a sixth shaft of the robot through a universal joint and moves synchronously along with the robot. Image signals acquired by the vision sensor are transmitted to image acquisition software of the industrial personal computer through a network, and key parameters such as image acquisition frame rate, exposure time and the like can be dynamically adjusted through the image acquisition software so as to improve image acquisition quality. Because the image acquisition software provides a large number of API interfaces for secondary development, the acquired image can be conveniently processed in real time.
2. Binocular stereo vision sensor
The binocular stereo vision sensor is the 'eye' of the robot and has significance in the whole system, and a design drawing and a physical drawing are shown in figure 3. During actual image acquisition, the position of the sensor CCD is kept still, and the base length is adjusted by moving the prism loading platform in parallel along the optical axis direction of the camera, so that the binocular molten pool under various working conditions such as different molten pool sizes, different image acquisition distances and the like can be imaged completely.
Besides the advantages of the variable base length, the sensor also has the following characteristics:
(1) Compact structure, small occupied space and low production cost. The main body part of the sensor for containing the CCD camera is formed by 3D printing of a nylon material, and is light in weight and good in heat dissipation; the U-shaped groove structure used for slidably adjusting the distance between the double prisms is formed by assembling aluminum alloy sheets at the front end of the binocular stereoscopic vision sensor, the loading platform is fixed by screwing and unscrewing the hexagon screws on the two sides, and the loading platform is horizontally moved in the guide rail so as to adjust the distance between the prisms and the CCD camera, and the rigidity required by fixing of the optical system is ensured by the aluminum alloy sheets.
(2) The sensor is connected with the universal joint through the dovetail groove, and the dovetail groove is made of high-strength aluminum alloy, so that the enough rigidity of the joint is ensured. The sensor is connected with the robot flange plate through the universal joint, and the image acquisition angle and the measurement distance can be conveniently adjusted so as to find out the acquisition angle most suitable for binocular stereoscopic vision imaging.
(3) The original installation scheme of the optical filter before the optical filter is embedded into the lens is changed, the optical filter is additionally installed by adopting a drawer structure and is connected to a light hole of the CCD camera through a boss, the optical filter is taken out during focusing and calibration, and the optical filter is installed during collecting molten pool images, so that the optical filter can be installed without disassembling the CCD camera after a system is calibrated, and the change of the relative position relationship of the left camera and the right camera caused by the displacement of the CCD camera is prevented.
(4) And protective glass is arranged at the foremost end of the sensor through a clamping groove so as to prevent a large amount of splash generated in the welding process from damaging the surface of the biprism. During actual work, the part of the prism arranged at the front end of the sensor is covered by the aluminum alloy sheet, so that light can be collected by the camera only after passing through the double prisms, and the light path completely accords with the pre-designed light path.
2. Detailed description of the invention
1. Binocular stereoscopic vision sensor design based on single CCD prism light splitting
By using the secondary refraction principle of the prism, a single image can be artificially divided into a left virtual image and a right virtual image under the condition of only using a single CCD to collect images, so that a binocular image pair is formed. Ideally, when the bottom edge of the prism is aligned with the optical center of the camera and the bottom surface of the prism is parallel to the imaging plane, the system is self-correcting and becomes a standard binocular stereo vision sensor. The scheme utilizes the secondary refraction of the prism to the light rays, and the presented binocular images are virtual images of the actual object. Because the presented stereo image pair is actually shot by the same CCD camera, the problem of illumination inconsistency caused by the adoption of the left CCD camera and the right CCD camera can be overcome to a certain extent, and the stereo matching algorithm robustness is improved beneficially.
Fig. 4 is a geometrical optical schematic diagram of a binocular stereo vision sensor based on prism beam splitting. The prism coordinate system is defined as (O) p ,X p ,Y p ,Z p ),O p Is the origin coordinate of the prism; the camera coordinate system is defined as (O, X, Y, Z), O being the camera optical center. Guarantee Z when placing p The prism is coaxial with the Z, namely the coordinate of the origin of the prism is ensured to pass through the optical axis of the camera; also ensure X p Parallel to X, U p And the imaging plane of the camera is parallel to the bottom surface pi of the prism.
As shown in fig. 4, when an arbitrary three-dimensional point X is present in space p The emitted light will be deflected by an angle delta when passing through the prism, and the light respectively passes through two side surfaces pi after passing through the bottom surface of the prism r And pi l When the prism deflects the light, the image formed on the imaging plane is equivalent to two virtual object points X on two sides of the original object point r And X l And imaging at an imaging plane. The deflection action of the prism on the light is the double refraction action of the bottom surface and the side surface of the prism on the lightThe size of the deflection angle δ is determined by the prism geometry and the refractive index n and base angle α of the prism, as shown in fig. 5.
From the law of refraction and the geometric relationship between the incident angle and the refraction angle shown in fig. 5, the basic relationship between the deflection angle δ of the prism for the light ray, the refractive index n of the prism, and the base angle of the prism can be derived as shown in formula (1).
The formula (1) gives the basis of the prism model selection: firstly, the deflection angle delta is determined according to the size of a molten pool and the measurement distance, and as the refractive index of the used glass is basically fixed, only the proper size alpha of the prism base angle is needed to be selected to meet the requirement on the deflection angle delta.
Remember arbitrary three-dimensional points X p Coordinate X of p ＝[X p ,Y p ,Z p ] T Right virtual object point coordinate X r ＝[X pr ,Y pr ,Z pr ] T Left virtual object point coordinate X l ＝[X pl ,Y pl ,Z pl ] T From the above geometric relationship analysis, X r 、X l And three-dimensional point X p The coordinates in the Y and Z directions are the same, and the coordinates in the X direction have the relation of the formula (2).
X pr ＝X p +Z p tanδ
X pl ＝X p -Z p tanδ
Y p r＝Y p l＝Y p
Z p r＝Z p l＝Z p (2)
Writing equation (2) into a matrix transformation form is shown in equation (3).
X r ＝T r X p X l ＝T l X p (3)
In the formula (I), the compound is shown in the specification,
the formula (3) shows that the coordinates of the left and right virtual object points formed by the two refractions of the prism can be obtained by transforming the original space points through a simple matrix.
Fixing the world coordinate system on the prism coordinate system without loss of generality, and deducing a space point X p Left and right virtual points X formed after twice refraction by the prism l 、X r To the left and right image points m of the imaging plane l ，m r The transmission projective transformation of (1). Substituting the formula (3) into the pinhole camera projection transformation equation to write two secondary X p Respectively directly converted to m l And m r Is shown in equation (4).
In the formula (I), the compound is shown in the specification,
and (4) constructing a projection matrix from any space point of the prism binocular stereo vision sensor to the left and right image points on the imaging plane. Wherein alpha is u ，α v ，u 0 ，v 0 For camera reference, R and t are the rotational matrix and translation vector of the camera coordinate system relative to the prism coordinate system, respectively. Ideally, the bottom surface of the prism is parallel to the imaging plane of the camera, the centers of the double prisms pass through the optical axis of the camera, and the rotation matrix R = I 3×3 Is a 3X 3 unit array, t = [0, t = z ] T ，t z The distance from the prism center to the camera optical center.
According to the formula (4), the three-dimensional point X is expressed by homogeneous coordinates p ＝[X p ,Y p ,Z p ,1] T Left and right image points m generated by projection l ＝[u l ,v l ,1] T ,m r ＝[u r ,v r ,1] T The expanded results are substituted by the formula (A)4) The projection relation equation of the left and right image points and the three-dimensional point of the prism binocular stereo vision sensor can be obtained as shown in formula (5).
V is easily shown by the formulae (5) and (6) l ＝v r Namely, in an ideal state, the homonymous point pairs of the left and right images of the binocular stereo vision sensor equivalent to prism beam splitting are located on the same horizontal line, no parallax in the vertical direction exists, and the method is equivalent to a standard binocular stereo vision sensor obtained by adopting a double-camera horizontal placement and polar line correction. The horizontal parallax can be calculated by the formula (7).
Wherein Z = Z p +t z And represents the depth of a three-dimensional point in the camera coordinate system. The relation between the parallax and the three-dimensional point depth of the standard binocular stereo vision sensor constructed based on the prism is given by the formula (7), and the following conclusion can be obtained.
(1) When the three-dimensional point depth Z tends to infinity, the parallax tends to be a constant value d =2 α u tan δ, which is different from the conclusion of a stereo vision system consisting of two parallel placed cameras, which has a disparity that tends to 0 as the depth Z tends to infinity.
(2) When the depth Z of the three-dimensional point is fixed, the parallax d and the x-axis equivalent focal length alpha of the camera u The size delta of the deflection angle of the prism to the light ray and the distance t between the center of the prism and the optical center of the camera z It is relevant. When a camera (fixed internal reference) and a biprism material (fixed refractive index n) are selected, increasing the base angle alpha is equivalent to increasing the deflection angle delta according to the relation between the prism deflection angle delta and the base angle alpha in the formula (1), and the parallax d is increased under the same depth; to enlarge the prismDistance t from optical center of camera z The parallax d is reduced.
Fig. 6 is an equivalent schematic diagram of the prism-split stereo vision system and the left and right camera stereo vision systems. Wherein O is L And O R The two areas are respectively equivalent left and right camera optical centers, the area (1) and the area (2) are left camera view field ranges, the area (1) and the area (3) are right camera view field ranges, and the area (1) is a view field range overlapped by the left and right cameras, namely an effective area of the actual binocular stereo vision sensor. The base length of the equivalent stereo vision system can be expressed as equation (8).
B＝2t z tanδ (8)
The formula (8) shows that when the prism parameters (refractive index n and base angle size alpha) are determined, the distance t from the prism to the optical center of the camera is adjusted z And binocular stereoscopic vision sensors with different base line lengths can be constructed. In the actual image acquisition process, the size of the molten pool can be changed according to different process conditions, and the measurement distance can be adjusted due to the limitation of physical space, so that the adoption of the stereoscopic vision system with variable base length increases the flexibility of image acquisition, and complete binocular molten pool images can be acquired in the visual field range when the distances of various molten pools and measurement distances are changed by adjusting the distance between the prism and the optical center of the camera. The method is equivalent to the theoretical basis of a binocular stereo vision sensor through the method of splitting light by the single-camera prism.
The experiment used a large constant MER-200-14GM grayscale camera to photograph the molten pool with a resolution of 1628 × 1236, a CCD sensor array size of 1/1.8 inch (7.1 mm × 5.3 mm), and a computer series f =35mm fixed focus lens. From the above data, the half angle θ of the field angle of the camera in the horizontal direction can be calculated w 。
Theta was determined by substituting w =5.3mm, f =35mm into the formula (9) w ＝4.3°。
Firstly, the proper size of the bottom surface of the prism is selected, considering that the field of view of the camera is not large, the whole space of the sensor is limited, the physical size of the CCD camera is 29mm multiplied by 29mm, and the size of the double prisms with the bottom surface of 30mm multiplied by 30mm is selected to be close to the external size of the CCD, so that the object point refracted by the prism can fall on the CCD, the space can be saved, and the installation is convenient.
And secondly, selecting a proper base angle size alpha of the prism. Since the width of the molten pool of the collected CMT welding is between 5mm and 10mm, and the width of the molten pool is set to be l, the whole molten pool is positioned in the overlapping area of the left camera and the right camera, and the formula (10) is satisfied according to the similar triangular relation of the equivalent stereoscopic vision system in the figure 6.
By the formula (10),
in the formula Z p For the distance between the prism and the shot weld pool, the sensor should not be too close to the weld pool as a whole, otherwise the risk of collision between the sensor and the weld bead is increased along with the increase of the additive layer. Here, Z is first assumed to be p =20mm, the refractive index of K9 glass used for the prism is 1.52, and the base angle α =15 ° can be selected in combination with the geometric optical equation (1) and equation (10) for the prism.
When the base angle alpha of the double prisms is selected, the refraction deflection angle delta of the double prisms is also uniquely determined, and according to the formula (8) and the figure 6, the length of the base line of the equivalent binocular stereo vision sensor can only be changed by changing the distance t from the double prisms to the optical center of the camera z But is changed. If the width of the shot molten pool is small and the distance between the camera and the molten pool is short, t can be increased properly z The length of the base line is increased, the proportion of a molten pool in a visual field is improved, and meanwhile, the base line is increased according to the relation between the depth and the parallax, so that the measurement precision in the depth direction is improved. If the width of the shot molten pool is larger or the distance between the camera and the molten pool is farther, t is properly reduced z To reduce the baseline distance and increase the overlap of two equivalent camerasThe area ensures that a larger molten pool can be completely displayed in a visual field after being subjected to light splitting. The structural diagram and the real object diagram of the designed binocular stereo vision sensor are shown in figure 3.
2. Calibration and polar line correction of binocular stereo vision sensor
The calibration meaning of the binocular stereo vision sensor is to respectively determine the internal parameters of the left camera and the right camera and the relative pose (rotation and translation) between the two cameras. The calibration result can be used for epipolar line correction, can also be used for obtaining a depth map from the parallax map through conversion, and finally obtains a three-dimensional point cloud model under a world coordinate system through triangulation. Because the actual length of the shot molten pool is within 10mm, in order to obtain higher calibration accuracy, calibration plates with the side length of 10mm/10mm and the checkerboard number of 6 mm/7mm are selected and are equivalent to the actually measured molten pool width. During calibration, the distance from the camera to the calibration plate is equivalent to the distance during actual shooting of the molten pool, the shooting angle from the camera to the calibration plate is changed by adjusting the pose of the robot, 12 binocular images are collected, and partial images are shown in fig. 5.
Calibration was performed using the stereoCameraCalibrator kit of MATLAB. First, the acquired complete binocular image is divided from the middle into left and right images of the same resolution. Then, adjusting the angular point response threshold value, and accurately extracting angular point coordinates in the checkerboard, as shown in fig. 7, which are the detection results of the angular point of the left camera and the angular point of the right camera, wherein the extraction precision of the angular point coordinates determines the calibration precision. Then, system calibration is carried out, and calibration of the binocular system is divided into two steps: the first step is to calibrate the internal parameters and the external parameters of the left camera and the right camera respectively by using a Zhangyingyou calibration method, and the second step is to optimize and solve the rotation and the translation of the left camera and the right camera by using the calibration result in the first step. After the internal reference and pose of the two cameras are uniquely determined, the basic matrix F is also determined, and only when the relative position relation of the biprism distance cameras is adjusted, the calibration needs to be carried out again. The calibration results are shown in Table 1. The calibrated average re-projection error is 0.64 pixel, the re-projection result is shown in figure 8, the precision reaches the sub-pixel level, and the actual engineering requirement is met. The calibration results of the binocular stereo vision sensor are shown in table 1.
TABLE 1
For epipolar rectification of a calibrated binocular system, the simplest method is Bonguet algorithm, and the Bonguet algorithm minimizes the number of times of each re-projection in the left image and the right image by using a rotation matrix and a translation vector between binocular images and simultaneously maximizes an observation area. The left and right homonymy points of the two image pairs after epipolar line correction are on the same horizontal line, so that parallax value search is only needed along the horizontal scanning line during stereo matching. The result of the corrected molten pool image is shown in fig. 9, and it can be seen that the homologous points of the left and right images are located on the same horizontal scanning line.
3. Weld pool image acquisition and stereo matching
And acquiring a molten pool image by using a calibrated binocular stereoscopic vision sensor after epipolar line correction. In order to prevent the influence of strong arc light in the welding process, a 660nm filter is adopted to filter out redundant arc light, and a complete and clear binocular molten pool image pair is acquired under the condition of only utilizing the arc light irradiation by adjusting the size of an aperture and the exposure time. For multi-layer single pass build-up, the binocular weld pool images of the different layers are shown in fig. 10. Welding current: 200A, welding speed: 24cm/min. The molten pool image of the fourth layer is the clearest and rich in texture, and dense stereo matching is performed by taking the molten pool image of the fourth layer as an example.
The purpose of stereo matching is to find homonymous points on the left and right images, and then three-dimensional points can be recovered by a triangulation method. The most difficult point of analyzing the above molten pool images is that the matching is weak in texture, so that more mismatching is easy to generate. The semi-global matching method (SGM) is adopted for matching, the algorithm is proposed in 2005, has matching precision and efficiency, and is the most widely applied algorithm in the current engineering. The specific steps are shown in fig. 11.
As shown in fig. 12a, the disparity map of the left image obtained after matching shows that the disparity values of the weak texture regions are missing, but the overall trend is correct. We further use the neural network to complement the depth map to obtain a more complete depth map. After the completion of the depth map, the depth map is back projected to a three-dimensional space according to the internal parameters of the left camera, and a complete three-dimensional point cloud model of the molten pool can be obtained as shown in fig. 12 b.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (1)
1. A WAAM molten pool three-dimensional reconstruction method is characterized by comprising the following steps:
arranging a single CCD double-prism binocular stereoscopic vision sensor;
calibrating and correcting polar lines of the set single CCD double-prism binocular stereoscopic vision sensor;
based on the calibrated single CCD double-prism binocular stereoscopic vision sensor subjected to polar line correction, carrying out molten pool image acquisition and stereoscopic matching so as to realize WAAM molten pool three-dimensional reconstruction;
set up single CCD biprism binocular stereoscopic vision sensor, include:
3D printing nylon material to form a main body part for containing the CCD camera;
set up single CCD biprism binocular stereoscopic vision sensor, include:
adopting aluminum alloy sheets to assemble a U-shaped groove structure used for sliding and adjusting the distance of the double prisms at the front end of the single CCD double-prism binocular stereoscopic vision sensor, fixing a loading platform by screwing and unscrewing hexagon screws at two sides of the U-shaped groove structure and horizontally moving the loading platform in a guide rail so as to adjust the distance from the prisms to a CCD camera;
set up single CCD biprism binocular stereoscopic vision sensor, include:
the single CCD double-prism binocular stereoscopic vision sensor is connected with the universal joint through a dovetail groove, and the dovetail groove is made of high-strength aluminum alloy;
set up single CCD biprism binocular stereoscopic vision sensor, include:
the single CCD double prism binocular stereoscopic vision sensor is connected with the robot flange plate through a universal joint;
set up single CCD biprism binocular stereoscopic vision sensor, include:
the optical filter is additionally arranged by adopting a drawer structure and is connected to a light hole of the CCD camera through a boss;
set up single CCD biprism binocular stereoscopic vision sensor, include:
mounting protective glass at the foremost end of the single CCD double-prism binocular stereoscopic vision sensor through a clamping groove;
set up single CCD biprism binocular stereoscopic vision sensor, include:
determining a deflection angle delta according to the size of a molten pool and the measurement distance;
determining the size alpha of the corresponding prism base angle based on the determined deflection angle delta;
demarcating the set single CCD double-prism binocular stereoscopic vision sensor and correcting the polar line, including:
firstly, dividing an acquired complete binocular image from the middle into a left image and a right image with the same resolution; secondly, adjusting a corner response threshold value, and accurately extracting corner coordinates in the checkerboard; then, system calibration is carried out, and the calibration of the binocular system comprises two steps: the method comprises the following steps that firstly, an internal parameter and an external parameter of a left camera and a right camera are respectively calibrated by using a Zhang Zhengyou calibration method, and secondly, the rotation and the translation of the left camera and the right camera are optimally solved by using a calibration result in the first step;
correcting the polar line of the calibrated binocular system;
based on single CCD biprism binocular stereoscopic vision sensor after standardization and polar line are rectified, carry out molten pool image acquisition and stereo matching to realize WAAM molten pool three-dimensional reconstruction, include:
collecting a molten pool image by using a calibrated and epipolar-corrected binocular stereo vision sensor, wherein a 660nm optical filter is adopted to filter out redundant arc light, and a complete and clear binocular molten pool image pair is collected under the condition of only utilizing arc light irradiation by adjusting the size of an aperture and the exposure time;
and carrying out dense stereo matching on the acquired binocular molten pool image pair.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111525330.9A CN114383543B (en) | 2021-12-14 | 2021-12-14 | WAAM molten pool three-dimensional reconstruction method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111525330.9A CN114383543B (en) | 2021-12-14 | 2021-12-14 | WAAM molten pool three-dimensional reconstruction method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114383543A CN114383543A (en) | 2022-04-22 |
| CN114383543B true CN114383543B (en) | 2022-12-27 |
Family
ID=81196466
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111525330.9A Active CN114383543B (en) | 2021-12-14 | 2021-12-14 | WAAM molten pool three-dimensional reconstruction method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114383543B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115063567B (en) * | 2022-08-19 | 2022-10-28 | 中国石油大学(华东) | Three-dimensional light path analysis method of double-prism monocular stereoscopic vision system |
| CN117664862B (en) * | 2024-01-31 | 2024-06-18 | 宁德时代新能源科技股份有限公司 | Tab detection system and tab detection method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103528540A (en) * | 2013-10-11 | 2014-01-22 | 河北科技大学 | Prism-based single-camera stereoscopic vision imaging device applied to sensing of welding pool |
| CN107796328A (en) * | 2017-09-21 | 2018-03-13 | 西南交通大学 | Metal increasing material manufacturing Pool three-dimensional visual sensor and detection method |
| CN108015455A (en) * | 2017-11-01 | 2018-05-11 | 清华大学 | Multifunctional vision sensor device based on recombination laser structure light |
| CN109259717A (en) * | 2018-08-27 | 2019-01-25 | 彭波 | A kind of stereo endoscope and endoscope measurement method |
| CN109668509A (en) * | 2019-01-18 | 2019-04-23 | 南京理工大学 | Based on biprism single camera three-dimensional measurement industrial endoscope system and measurement method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11529741B2 (en) * | 2018-12-17 | 2022-12-20 | Divergent Technologies, Inc. | System and method for positioning one or more robotic apparatuses |
-
2021
- 2021-12-14 CN CN202111525330.9A patent/CN114383543B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103528540A (en) * | 2013-10-11 | 2014-01-22 | 河北科技大学 | Prism-based single-camera stereoscopic vision imaging device applied to sensing of welding pool |
| CN107796328A (en) * | 2017-09-21 | 2018-03-13 | 西南交通大学 | Metal increasing material manufacturing Pool three-dimensional visual sensor and detection method |
| CN108015455A (en) * | 2017-11-01 | 2018-05-11 | 清华大学 | Multifunctional vision sensor device based on recombination laser structure light |
| CN109259717A (en) * | 2018-08-27 | 2019-01-25 | 彭波 | A kind of stereo endoscope and endoscope measurement method |
| CN109668509A (en) * | 2019-01-18 | 2019-04-23 | 南京理工大学 | Based on biprism single camera three-dimensional measurement industrial endoscope system and measurement method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114383543A (en) | 2022-04-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114383543B (en) | WAAM molten pool three-dimensional reconstruction method | |
| CN110524580B (en) | Welding robot vision assembly and measuring method thereof | |
| CN109035200B (en) | Bolt positioning and pose detection method based on single-eye and double-eye vision cooperation | |
| CN111243002A (en) | Monocular laser speckle projection system calibration and depth estimation method applied to high-precision three-dimensional measurement | |
| CA3068659A1 (en) | Augmented reality displays with active alignment and corresponding methods | |
| CN101539422B (en) | Monocular vision real time distance measuring method | |
| JP5011528B2 (en) | 3D distance measurement sensor and 3D distance measurement method | |
| CN104052986B (en) | Detection apparatus, stereographic machine testing device and detection method | |
| TWI521471B (en) | 3 - dimensional distance measuring device and method thereof | |
| Strelow et al. | Precise omnidirectional camera calibration | |
| CN105423913A (en) | Three-dimensional coordinate measurement method based on line structure light scanning | |
| CN103727927A (en) | High-velocity motion object pose vision measurement method based on structured light | |
| WO2011125937A1 (en) | Calibration data selection device, method of selection, selection program, and three dimensional position measuring device | |
| CN111009030A (en) | Multi-view high-resolution texture image and binocular three-dimensional point cloud mapping method | |
| CN103134444B (en) | The varifocal three-dimension measuring system of double-view field | |
| CN113155084A (en) | Binocular vision distance measuring device and method based on laser cross standard line assistance | |
| CN109798877B (en) | Bionic underwater robotic fish binocular stereo distance measurement method | |
| CN103528540A (en) | Prism-based single-camera stereoscopic vision imaging device applied to sensing of welding pool | |
| CN109636859A (en) | A kind of scaling method of the 3D vision detection based on one camera | |
| CN111998834B (en) | Crack monitoring method and system | |
| CN113781576A (en) | Binocular vision detection system, method and device for multi-degree-of-freedom pose real-time adjustment | |
| CN110657750B (en) | Detection system and method for passivation of cutting edge of cutter | |
| CN116804537A (en) | Binocular range finding system and method | |
| CN112419418A (en) | Positioning method based on camera mechanical aiming | |
| Pachidis et al. | Pseudo-stereo vision system: a detailed study |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |