CN112958973A - Welding vision locating device of medium plate robot based on structured light three-dimensional vision - Google Patents
Welding vision locating device of medium plate robot based on structured light three-dimensional vision Download PDFInfo
- Publication number
- CN112958973A CN112958973A CN202110171580.0A CN202110171580A CN112958973A CN 112958973 A CN112958973 A CN 112958973A CN 202110171580 A CN202110171580 A CN 202110171580A CN 112958973 A CN112958973 A CN 112958973A
- Authority
- CN
- China
- Prior art keywords
- welding
- robot
- point cloud
- scanning
- track
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003466 welding Methods 0.000 title claims abstract description 268
- 238000000034 method Methods 0.000 claims abstract description 70
- 230000011218 segmentation Effects 0.000 claims abstract description 21
- 229910000831 Steel Inorganic materials 0.000 claims abstract description 18
- 230000008569 process Effects 0.000 claims abstract description 18
- 239000010959 steel Substances 0.000 claims abstract description 18
- 230000007123 defense Effects 0.000 claims abstract description 11
- 238000004364 calculation method Methods 0.000 claims abstract description 5
- 238000005070 sampling Methods 0.000 claims description 18
- 238000007789 sealing Methods 0.000 claims description 18
- 238000007493 shaping process Methods 0.000 claims description 10
- 239000004065 semiconductor Substances 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 3
- 238000003384 imaging method Methods 0.000 claims description 3
- 230000000737 periodic effect Effects 0.000 claims description 3
- 230000010363 phase shift Effects 0.000 claims description 3
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 238000005485 electric heating Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 5
- 239000000284 extract Substances 0.000 abstract description 3
- 238000000605 extraction Methods 0.000 description 8
- 239000007769 metal material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 239000003245 coal Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B23—MACHINE TOOLS; METAL-WORKING NOT OTHERWISE PROVIDED FOR
- B23K—SOLDERING OR UNSOLDERING; WELDING; CLADDING OR PLATING BY SOLDERING OR WELDING; CUTTING BY APPLYING HEAT LOCALLY, e.g. FLAME CUTTING; WORKING BY LASER BEAM
- B23K37/00—Auxiliary devices or processes, not specially adapted to a procedure covered by only one of the preceding main groups
- B23K37/02—Carriages for supporting the welding or cutting element
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B23—MACHINE TOOLS; METAL-WORKING NOT OTHERWISE PROVIDED FOR
- B23K—SOLDERING OR UNSOLDERING; WELDING; CLADDING OR PLATING BY SOLDERING OR WELDING; CUTTING BY APPLYING HEAT LOCALLY, e.g. FLAME CUTTING; WORKING BY LASER BEAM
- B23K37/00—Auxiliary devices or processes, not specially adapted to a procedure covered by only one of the preceding main groups
Landscapes
- Physics & Mathematics (AREA)
- Optics & Photonics (AREA)
- Engineering & Computer Science (AREA)
- Mechanical Engineering (AREA)
- Laser Beam Processing (AREA)
- Manipulator (AREA)
- Butt Welding And Welding Of Specific Article (AREA)
Abstract
The invention discloses a structured light three-dimensional vision-based welding vision locating device for a medium plate robot, which consists of a robot arm and a scanning measuring head; the robot arm is used for driving the scanning measuring head; the scanning measuring head is used for carrying out point cloud collection on the workpiece, and further identifies a welding seam track by a method of fitting a plane and a curved surface to solve an intersection line, so that a robot position finding coordinate is provided. The invention adopts a structured light three-dimensional surface scanning technology for scanning the complex welding seam of the medium plate, extracts the welding seam track and width through the segmentation and fitting calculation of three-dimensional point cloud, provides the welding robot with fast and accurate welding seam locating and welding process parameters, and effectively helps the robot to realize the automatic welding of the medium plate such as carriage plates, steel structures, civil air defense doors, containers, engineering machinery, pipe plate parts and the like.
Description
Technical Field
The invention relates to the field of robot vision, in particular to a robot welding vision position-finding device and method, and particularly relates to a structured light three-dimensional vision-based medium plate robot welding position-finding device and method.
Background
The medium plate generally refers to a steel plate or other metal material plate with the thickness of more than 4mm, and the robot medium plate welding is a common welding application and generally has the following characteristics: before welding, performing group butt welding on a tool fixture; the functions of weld searching, weld tracking and multilayer multi-pass welding are required; most of welding needs to be provided with a large positioner, so that the welding is carried out at the optimal position of the ship-shaped welding or fillet welding. The application fields include engineering machinery, building steel structures, coal machinery, special vehicle carriage plates, heat exchanger tube plates, container plates, bridges, boilers and ships and the like. The robot for welding the medium plate mainly has the following advantages:
the welding quality is stabilized and improved;
the labor productivity is improved;
the labor intensity of workers is improved, and the workers can work in a harmful environment;
the requirements on the operation technology of workers are reduced;
the track programming teaching is a necessary link for guiding the medium plate welding robot, but because the medium plate is large in size, welding seams are often not regular enough, the blanking and group pairing precision of a previous process is difficult to control, and errors are large, so that the robot is difficult to complete batch welding through traditional teaching programming and fixture positioning when welding the medium plate, and the application and popularization of the robot in the field are greatly hindered.
In order to reduce the programming time and to enable the identification of irregular welds, the use of visual measurement methods is gaining more and more attention to the guidance of welding robots. Common visual guidance methods include passive vision methods and laser seam tracking methods. The passive vision uses a camera to shoot a welding seam picture, and the two-dimensional picture is processed to extract the welding seam plane track. The laser welding seam tracking method is based on the laser triangle three-dimensional distance measuring principle, a line laser is projected to the surface of a welding seam, a camera shoots a laser pattern from another angle, a computer extracts a section profile in the pattern, a profile characteristic point is solved, and characteristic points at a plurality of positions are connected to obtain a welding seam track. The method has the advantages of high precision and good robustness, and has the following disadvantages:
the robot teaching programming is needed to realize rough path planning in advance, the alignment of the welding seam position is well done,
is only suitable for extracting straight-line welding seams and circular welding seams
Only one section profile of the workpiece can be acquired each time, the whole shape of the workpiece cannot be scanned and analyzed, and the method is not suitable for extracting complex welding seam tracks such as a space free curve, a cylinder-plane intersecting line and a cylinder-cylinder intersecting line.
On the other hand, the optical surface scanning three-dimensional measurement technology is becoming mature day by day, and the structured light projection method is based on the principle that speckles or grating patterns with codes are projected on the surface of a measured object, a camera shoots deformation patterns modulated by the surface of the measured object, then three-dimensional information is obtained through further demodulation, and finally the three-dimensional appearance of the surface of the object is obtained through calibration. The technique has the advantages that the point cloud of the whole area of the object can be scanned at one time, and the function of three-dimensional reconstruction of the shape is realized. However, for metal materials such as welding workpieces, the device is also large in size and inconvenient to install, and therefore, the device is rarely applied to extraction of the welding track of a welding robot.
In conclusion, these problems hinder further application of the vision technology in the field of welding robots, and indirectly influence further popularization of the welding robots in the field of medium plate application. Therefore, it is desirable to invent a device and method that can use structured light scanning to break through the existing bottleneck.
Disclosure of Invention
The invention aims to overcome the difficulty in locating the welding track of the robot for the medium plate and provides several optical welding seam locating methods with surface scanning structures aiming at typical workpieces of the medium plate. The miniature structured light three-dimensional scanning device is particularly realized by adopting an MEMS micro mirror scanning method. The robot arm is adopted to drive the scanning device to carry out point cloud collection on the workpiece, and further, the welding seam track is identified by a method of fitting a plane and a curved surface to solve an intersection line, so that the robot locating coordinate is provided. Meanwhile, the specific method and steps for automatic scanning and track extraction of complex welding seam tracks of different types of workpieces are provided, programming teaching time is effectively saved, and track extraction accuracy is guaranteed.
The technical scheme adopted by the invention is as follows:
a scanning device:
a welding vision locating device of a medium plate robot based on structured light three-dimensional vision is arranged at the tail end of a robot arm and comprises 1 structured light projection unit and 1 image receiving unit, wherein the image receiving unit consists of 1 camera or 2 cameras at a certain distance; the medium plate is placed in the motion range of the robot arm, and the welded area faces the scanning device.
The types of structured light projection units include digital micromirror projectors, speckle projectors, MEMS projectors.
The structured light projection unit comprises a group of semiconductor lasers, a word line beam shaping focusing lens and a single-axis MEMS micro-mirror; the MEMS micro-mirror driving mode comprises an electrostatic driving mode, an electromagnetic driving mode and an electric heating driving mode. A beam shaping focusing lens is a Powell prism. The wavelength range of the semiconductor laser is between 400-1000 nm.
The MEMS micro-mirror and the semiconductor laser are driven and controlled by an external circuit board, and the device executes the following scanning mode when in work:
the method comprises the following steps: the circuit board outputs periodic sine wave signals to the MEMS micro-mirror to drive the MEMS micro-mirror to scan at a high speed; the circuit board outputs another light intensity modulation signal to the laser to form brightness change;
step two: point laser emitted by a laser device is incident to the surface of an MEMS (micro-electromechanical system) micromirror through a word line beam shaping and focusing lens and is reflected to the surface of an imaged object through the micromirror to form a grating pattern;
step three: the circuit board outputs a synchronous trigger signal to the camera, and the camera receives the trigger signal to complete shooting of a plurality of grating patterns;
step five: the shot pattern is sent to a computer for calculation, and the three-dimensional information of the object is obtained by utilizing the imaging principle of a phase shift method.
The welding seam locating method comprises the following steps:
the medium plate comprises a carriage plate, a steel structure, a civil air defense door, a container, engineering machinery, a pipe plate and the like.
The following welding seam track scanning and extracting steps are adopted for a common medium plate:
the tail end of the robot arm is moved to be close to a workpiece to be welded,
scanning the three-dimensional shape of the surface of the workpiece in a surface scanning mode to obtain three-dimensional point cloud of the surface of the workpiece,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of plane and curved surface point clouds,
fitting the point cloud by using a plane and curved surface equation, extracting intersection characteristic points by using a method of solving intersection lines between planes, between planes and curved surfaces and between curved surfaces and curved surfaces,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding seam, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
and (5) giving the angle posture of the welding gun through manual teaching, and starting welding.
When the welding workpiece is a carriage plate, the following welding seam track scanning and extracting steps are adopted:
the tail end of the robot arm is moved to the central position above the square frame of the carriage plate, so that the measuring range of the scanning device covers four corners of the square frame,
the rotary robot arm scans the three-dimensional shape of the surface of the workpiece at four corners of the square frame in a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of plane and curved surface point clouds,
fitting the point cloud by using a plane and curved surface equation, extracting intersection characteristic points by using a method of solving intersection lines between planes, between planes and curved surfaces and between curved surfaces and curved surfaces,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding seam, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
the angle posture of the welding gun is given through manual teaching, the welding is started,
and after one four-square frame is welded, moving the robot arm to the adjacent four square frames, repeating the process, and finally completing the welding of the four square frames of all the carriage plates.
When the welded workpiece is an H-shaped steel structure rib plate, the following welding seam track scanning and extracting steps are adopted:
the tail end of the robot arm is moved to the position above the steel structure web plate, the scanning device is opposite to the ribbed slab for shooting,
the rotary robot arm scans the surfaces near the intersection line of the ribbed plate and the web plate and the ribbed plate and the wing plate in a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of planes,
fitting the point cloud by using a plane equation, extracting characteristic points of the intersection line of the rib plate and the web plate and the rib plate and the wing plate by using a method of solving the intersection line of a plane and a plane,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding seam, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
the angle posture of the welding gun is given through manual teaching, the welding is started,
and after one rib plate is welded, moving the robot arm to the adjacent rib plate, and repeating the process to finally complete the welding of all H steel rib plates.
5.2.4 when the welding workpiece is a civil air defense door leaf grid, adopting the following welding seam track scanning and extracting steps:
the door leaf to be welded is flatly placed on the workbench, the grid of the door leaf is manually spot-welded with the lower panel in advance, the tail end of the arm of the robot is moved to the central position above one grid of the door leaf, so that the measuring range of the scanning device covers four corners of the grid,
the rotary robot arm scans the three-dimensional shape of the surface of the work piece of the grid by adopting a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain point clouds of a plurality of planes,
fitting the point cloud by using a plane equation, extracting intersection characteristic points by using a method of solving an intersection between a plane and a plane,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding line, including the trace points of a flat welding line, a vertical welding line and a step welding line, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
the angle posture of the welding gun is given through manual teaching, the welding is started,
and after welding one grid, moving the robot arm to the adjacent rib plate, and repeating the process to finally complete the welding of all H steel rib plates.
When the welding workpiece is a civil air defense door leaf sealing frame, the following welding seam track scanning and extracting steps are adopted:
the door leaf to be welded is flatly placed on the workbench, the door leaf and the sealing frame are manually spot-welded in advance, the tail end of the robot arm is moved to the position above the door leaf,
the mobile robot arm scans the three-dimensional shape of the surface of the workpiece of the sealing frame in a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of point clouds of planes and curved surfaces,
fitting the point cloud by using a plane and curved surface equation, extracting intersection characteristic points by using a method of solving an intersection between a plane and solving an intersection between the plane and a curved surface,
fitting the intersection characteristic points to obtain the coordinates of the welding track points, including the track points of the linear welding line and the arc welding line of the sealing frame, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
the angle posture of the welding gun is given through manual teaching, the welding is started,
and after one side of the door leaf is welded, the door leaf is reversed to weld the other side of the sealing frame, and the process is repeated to finally complete the welding of all the door leaf sealing frames.
When the welding workpiece is a pipe plate, the following welding seam track scanning and extracting steps are adopted:
moving the tail end of the robot arm to a position above the tube plate piece to enable the scanning device to measure the range of the tube plate piece,
the mobile robot arm scans the three-dimensional shape of the surface of the workpiece of the pipe plate in a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plate plane and a plurality of point clouds on the end surfaces of the pipe fittings,
fitting the point cloud of the plate plane by using a plane equation, fitting the point cloud of the end surface of the pipe fitting by using a curved surface equation, extracting intersection characteristic points by using a method of solving an intersection between a plane and a curved surface,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding seam, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
and (3) giving the angle posture of the welding gun through manual teaching, and welding each pipe fitting and the plate welding seam one by one.
Repeating the above steps 1) -7) if the welding process has thermal deformation.
Advantageous effects
The invention adopts a structured light three-dimensional surface scanning technology for scanning the complex welding seam of the medium plate, extracts the welding seam track and width through the segmentation and fitting calculation of three-dimensional point cloud, provides the welding robot with fast and accurate welding seam locating and welding process parameters, and effectively helps the robot to realize the automatic welding of the medium plate such as carriage plates, steel structures, civil air defense doors, containers, engineering machinery, pipe plate parts and the like.
The invention has the obvious advantages of large shooting area, high efficiency, less manual intervention and high automation degree by adopting surface structure light scanning. The MEMS micro-mirror projector is adopted to realize the miniaturization of the scanning device, is convenient to install at the tail end of the robot arm and enlarges the scanning working range. The intersection line of geometric elements is calculated by adopting a point cloud segmentation and plane surface fitting algorithm, so that the accurate extraction of complex welding seam tracks is realized, the dependence on programming teaching of a robot is eliminated, and the field application of the medium plate welding robot is more intelligent and more convenient.
Drawings
FIG. 1 is a schematic view of a scanning apparatus;
FIG. 2 is a schematic view of a typical medium plate weld scan;
FIG. 3 is a schematic view of a weld scan of a bed plate;
FIG. 4 is a schematic view of a weld scan of H-section steel;
FIG. 5 is a schematic view of a civil air defense door check weld scan;
FIG. 6 is a schematic view of a weld scan of a fire damper panel;
FIG. 7 is a schematic view of a weld scan of a tube sheet;
wherein: a is a structured light projection unit, B is an image receiving unit, a is a semiconductor laser, B is a word line beam shaping focusing lens, c is a single-axis MEMS micro-mirror, 1 is a robot arm, 2 is a scanning device, 3 is the tail end of a robot welding gun, 4 is a workpiece to be welded, and 5 is a welding seam.
Detailed Description
The invention will be further described with reference to the accompanying drawings in which:
scanning device
The utility model provides a device is sought to structured light three-dimensional vision based cut deal robot welding vision, scanning device installs at the robot arm end, and the cut deal is placed in the robot arm motion range, and the region of being welded faces scanning device. Referring to fig. 1, the scanning apparatus includes a structured light projection unit a and an image receiving unit B, which is composed of one camera and another camera spaced apart by a certain distance; a consists of 1 semiconductor laser, a word line beam shaping focusing lens and a single-axis MEMS micro-mirror; the word line beam shaping focusing lens is a Powell prism, and the single-axis MEMS micromirror is an electrostatically-driven micromirror; the MEMS micro-mirror and the semiconductor laser are driven and controlled by an external circuit board, and the device executes the following scanning mode when in work:
the method comprises the following steps: the circuit board outputs periodic sine wave signals to the MEMS micro-mirror to drive the MEMS micro-mirror to scan at a high speed; the circuit board outputs another light intensity modulation signal to the laser to form brightness change;
step two: point laser emitted by a laser device is incident to the surface of an MEMS (micro-electromechanical system) micromirror through a word line beam shaping and focusing lens and is reflected to the surface of an imaged object through the micromirror to form a grating pattern;
step three: the circuit board outputs a synchronous trigger signal to the camera, and the camera receives the trigger signal to complete shooting of a plurality of grating patterns;
step five: the shot pattern is sent to a computer for calculation, and the three-dimensional information of the object is obtained by utilizing the imaging principle of a phase shift method.
Welding seam scanning and locating method
Referring to fig. 2, the following weld trace scanning extraction steps are adopted for a general medium plate:
the tail end of the robot arm is moved to be close to a workpiece to be welded,
the scanning device scans the three-dimensional shape of the surface of the workpiece in a surface scanning mode to obtain three-dimensional point cloud of the surface of the workpiece,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of plane and curved surface point clouds,
fitting the point cloud by using a plane and curved surface equation, extracting intersection characteristic points by using a method of solving intersection lines between planes, between planes and curved surfaces and between curved surfaces and curved surfaces,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding seam, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
and (5) giving the angle posture of the welding gun through manual teaching, and starting welding.
Referring to fig. 3, the following weld trace scanning extraction steps are adopted for the carriage plate:
the tail end of the robot arm is moved to the central position above the square frame of the carriage plate, so that the scanning device covers four corners of the square frame,
the rotary robot arm scans the three-dimensional shape of the surface of the workpiece at four corners of the square frame in a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of plane and curved surface point clouds,
fitting the point cloud by using a plane and curved surface equation, extracting intersection characteristic points by using a method of solving intersection lines between planes, between planes and curved surfaces and between curved surfaces and curved surfaces,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding seam, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
the angle posture of the welding gun is given through manual teaching, the welding is started,
and after one four-square frame is welded, moving the robot arm to the adjacent four square frames, repeating the process, and finally completing the welding of the four square frames of all the carriage plates.
Referring to fig. 4, the following weld trace scanning and extracting steps are adopted for H-shaped steel:
the tail end of the robot arm is moved to the position above the steel structure web plate, the scanning device is opposite to the ribbed slab for shooting,
rotating the robot arm, scanning the surface near the intersection line of the ribbed plate and the web plate and the ribbed plate and the wing plate by a scanning device in a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of planes,
fitting the point cloud by using a plane equation, extracting characteristic points of the intersection line of the rib plate and the web plate and the rib plate and the wing plate by using a method of solving the intersection line of a plane and a plane,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding seam, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
the angle posture of the welding gun is given through manual teaching, the welding is started,
and after one rib plate is welded, moving the robot arm to the adjacent rib plate, and repeating the process to finally complete the welding of all H steel rib plates.
Referring to fig. 5, the following welding seam track scanning extraction steps are adopted for the civil air defense latticed pattern:
the door leaf to be welded is flatly placed on the workbench, the grid of the door leaf is manually spot-welded with the lower panel in advance, the tail end of the arm of the robot is moved to the central position above one grid of the door leaf, so that the measuring range of the scanning device covers four corners of the grid,
rotating the robot arm, scanning the three-dimensional shape of the surface of the work piece of the grid by a scanning device in a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain point clouds of a plurality of planes,
fitting the point cloud by using a plane equation, extracting intersection characteristic points by using a method of solving an intersection between a plane and a plane,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding line, including the trace points of a flat welding line, a vertical welding line and a step welding line, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
the angle posture of the welding gun is given through manual teaching, the welding is started,
and after welding one grid, moving the robot arm to the adjacent rib plate, repeating the process, and finally completing the welding of all grids.
Referring to fig. 6, the following welding seam track scanning and extracting steps are adopted for the door leaf sealing frame of the civil air defense door:
the door leaf to be welded is flatly placed on the workbench, the door leaf and the sealing frame are manually spot-welded in advance, the tail end of the robot arm is moved to the position above the door leaf,
the mobile robot arm scans the three-dimensional shape of the surface of the workpiece of the sealing frame in a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of point clouds of planes and curved surfaces,
fitting the point cloud by using a plane and curved surface equation, extracting intersection characteristic points by using a method of solving an intersection between a plane and solving an intersection between the plane and a curved surface,
fitting the intersection characteristic points to obtain the coordinates of the welding track points, including the track points of the linear welding line and the arc welding line of the sealing frame, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
the angle posture of the welding gun is given through manual teaching, the welding is started,
and after one side of the door leaf is welded, the door leaf is reversed to weld the other side of the sealing frame, and the process is repeated to finally complete the welding of all the door leaf sealing frames.
Referring to fig. 7, the following weld trace scan extraction steps are employed for the tube sheet:
moving the tail end of the robot arm to a position above the tube plate piece to enable the scanning device to measure the range of the tube plate piece,
the mobile robot arm scans the three-dimensional shape of the surface of the workpiece of the pipe plate in a surface scanning mode to obtain three-dimensional point cloud,
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plate plane and a plurality of point clouds on the end surfaces of the pipe fittings,
fitting the point cloud of the plate plane by using a plane equation, fitting the point cloud of the end surface of the pipe fitting by using a curved surface equation, extracting intersection characteristic points by using a method of solving an intersection between a plane and a curved surface,
fitting the intersection characteristic points to obtain the coordinates of the trace points of the welding seam, outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct the welding track,
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
and (3) giving the angle posture of the welding gun through manual teaching, and welding each pipe fitting and the plate welding seam one by one.
If the welding process has thermal deformation, the above steps are repeated.
Claims (15)
1. The utility model provides a device is sought to cut deal robot welding vision based on three-dimensional vision of structured light which characterized in that:
the device consists of a robot arm and a scanning measuring head;
the robot arm is used for driving the scanning measuring head;
the scanning measuring head is used for carrying out point cloud collection on the workpiece; and identifying the welding seam track by a method of fitting a plane and a curved surface to solve an intersection line, and providing a robot locating coordinate.
2. The position searching method of the welding vision position searching device of the medium plate robot based on the claim 1 is characterized in that:
a robot arm is adopted to drive a scanning measuring head to carry out point cloud collection on a workpiece, and a welding seam track is identified by a method of fitting a plane and a curved surface to solve an intersection line, so that a robot locating coordinate is provided.
3. The welding vision locating device of the medium plate robot as claimed in claim 1, wherein: the scanning measuring head is arranged at the tail end of the robot arm and comprises a structured light projection unit and an image receiving unit, and the image receiving unit consists of a camera or two cameras with a certain distance; the medium plate is placed in the motion range of the robot arm, and the welded area faces the scanning device.
4. The welding vision locating device of the medium plate robot as claimed in claim 3, wherein: the types of structured light projection units include digital micromirror projectors, speckle projectors, MEMS projectors.
5. The welding vision locating device of the medium plate robot as claimed in claim 3, wherein: the structured light projection unit comprises a group of semiconductor lasers, a word line beam shaping focusing lens and a single-axis MEMS micro-mirror; the MEMS micro-mirror and the semiconductor laser are driven and controlled by an external circuit board, and the device executes the following scanning mode when in work:
the method comprises the following steps: the circuit board outputs periodic sine wave signals to the MEMS micro-mirror to drive the MEMS micro-mirror to scan at a high speed; the circuit board outputs another light intensity modulation signal to the laser to form brightness change;
step two: point laser emitted by a laser device is incident to the surface of an MEMS (micro-electromechanical system) micromirror through a word line beam shaping and focusing lens and is reflected to the surface of an imaged object through the micromirror to form a grating pattern;
step three: the circuit board outputs a synchronous trigger signal to the camera, and the camera receives the trigger signal to complete shooting of a plurality of grating patterns;
step five: the shot pattern is sent to a computer for calculation, and the three-dimensional information of the object is obtained by utilizing the imaging principle of a phase shift method.
6. The locating method of the welding vision locating device of the medium plate robot as claimed in claim 2, wherein: the MEMS micro-mirror driving mode comprises an electrostatic driving mode, an electromagnetic driving mode and an electric heating driving mode.
7. The welding vision locating device of the medium plate robot as claimed in claim 3, wherein: a beam shaping focusing lens is a Powell prism.
8. The welding vision locating device of the medium plate robot as claimed in claim 3, wherein: the wavelength range of the semiconductor laser is between 400-1000 nm.
9. The welding vision locating device of the medium plate robot as claimed in claim 3, wherein: the medium plate comprises a carriage plate, a steel structure, a civil air defense door, a container, engineering machinery and a pipe plate.
10. The locating method of the welding vision locating device of the medium plate robot as claimed in claim 2, wherein: the following welding seam track scanning and extracting steps are adopted:
moving the tail end of the robot arm to be close to a workpiece to be welded;
scanning the three-dimensional shape of the surface of the workpiece in a surface scanning mode to obtain three-dimensional point cloud of the surface of the workpiece;
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of plane and curved surface point clouds;
fitting the point cloud by using a plane and curved surface equation, and extracting intersection characteristic points by using a method of solving intersection lines between planes, between planes and curved surfaces and between curved surfaces and curved surfaces;
fitting intersection characteristic points to obtain welding track point coordinates, and outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct a welding track;
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current;
and (5) giving the angle posture of the welding gun through manual teaching, and starting welding.
11. The locating method of the welding vision locating device of the medium plate robot as claimed in claim 2, wherein: when the welding workpiece is a carriage plate, the following welding seam track scanning and extracting steps are adopted:
moving the tail end of the robot arm to the central position above the local square frame of the carriage plate, so that the measuring head measuring range covers four corners of the square frame;
the rotary robot arm scans the three-dimensional shape of the surface of the workpiece at four corners of the square frame in a surface scanning mode to obtain three-dimensional point cloud;
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of plane and curved surface point clouds;
fitting the point cloud by using a plane and curved surface equation, and extracting intersection characteristic points by using a method of solving intersection lines between planes, between planes and curved surfaces and between curved surfaces and curved surfaces;
fitting intersection characteristic points to obtain welding track point coordinates, and outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct a welding track;
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current;
giving the angle posture of the welding gun through manual teaching, and starting welding;
and after one four-square frame is welded, moving the robot arm to the adjacent four square frames, repeating the process, and finally completing the welding of the four square frames of all the carriage plates.
12. The locating method of the welding vision locating device of the medium plate robot as claimed in claim 2, wherein: when the welded workpiece is an H-shaped steel structure rib plate, the following welding seam track scanning and extracting steps are adopted:
moving the tail end of the robot arm to be above the steel structure web plate, and shooting by the measuring head right facing the rib plate;
the rotary robot arm scans the surfaces near the intersection lines of the rib plates and the web plates and the rib plates and the wing plates in a surface scanning mode to obtain three-dimensional point clouds;
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of planes;
fitting the point cloud by using a plane equation, and extracting characteristic points of intersection lines of the rib plates and the web plates and the rib plates and wing plates by using a method of solving intersection lines of planes and planes;
fitting intersection characteristic points to obtain welding track point coordinates, and outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct a welding track;
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current,
giving the angle posture of the welding gun through manual teaching, and starting welding;
and after one rib plate is welded, moving the robot arm to the adjacent rib plate, and repeating the process to finally complete the welding of all H steel rib plates.
13. The locating method of the welding vision locating device of the medium plate robot as claimed in claim 2, wherein: when the welding workpiece is a field-shaped lattice of a door leaf of the civil air defense door, the following welding seam track scanning and extracting steps are adopted:
horizontally placing the door leaf to be welded on the workbench, combining the grid of the door leaf with the lower panel by manual spot welding in advance, and moving the tail end of the arm of the robot to the central position above one grid of the door leaf to enable the measuring range of the measuring head to cover four corners of the grid;
scanning the three-dimensional shape of the surface of the work piece of the grid by the rotary robot arm in a surface scanning mode to obtain three-dimensional point cloud;
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain point clouds of a plurality of planes;
fitting the point cloud by using a plane equation, and extracting intersection characteristic points by using a method of solving an intersection between a plane and a plane;
fitting the characteristic points of the intersection line to obtain the coordinates of the track points of the welding line, including a flat welding line and a vertical welding line; outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help a robot correct a welding track;
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current;
giving the angle posture of the welding gun through manual teaching, and starting welding;
and after welding one grid, moving the robot arm to the adjacent rib plate, and repeating the process to finally complete the welding of all H steel rib plates.
14. The locating method of the welding vision locating device of the medium plate robot as claimed in claim 2, wherein: when the welding workpiece is a civil air defense door leaf sealing frame, the following welding seam track scanning and extracting steps are adopted:
horizontally placing the door leaf to be welded on the workbench, combining the door leaf with the sealing frame through manual spot welding in advance, and moving the tail end of the robot arm to the position above the door leaf;
scanning the three-dimensional shape of the surface of the workpiece of the sealing frame by the mobile robot arm in a surface scanning mode to obtain three-dimensional point cloud;
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plurality of point clouds of planes and curved surfaces;
fitting the point cloud by using a plane and curved surface equation, and extracting intersection characteristic points by using a method of solving an intersection line between a plane and solving an intersection line between the plane and a curved surface;
fitting the intersection characteristic points to obtain welding track point coordinates comprising track points of a sealing frame straight welding line and an arc welding line, and outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct a welding track;
giving the angle posture of the welding gun through manual teaching, and starting welding;
and after one side of the door leaf is welded, the door leaf is reversed to weld the other side of the sealing frame, and the process is repeated to finally complete the welding of all the door leaf sealing frames.
15. The locating method of the welding vision locating device of the medium plate robot as claimed in claim 2, wherein: when the welding workpiece is a pipe plate, the following welding seam track scanning and extracting steps are adopted:
moving the tail end of the robot arm to a position above the tube plate, so that the measuring head covers the tube plate within the measuring range;
scanning the three-dimensional shape of the surface of the pipe plate workpiece by a mobile robot arm in a surface scanning mode to obtain three-dimensional point cloud;
performing three-dimensional point cloud segmentation on the point cloud on the surface of the workpiece to obtain a plate plane and a plurality of pipe fitting end face point clouds;
fitting the point cloud of the plate plane by using a plane equation, fitting the point cloud of the end surface of the pipe fitting by using a curved surface equation, and extracting intersection characteristic points by using a method of solving an intersection between a plane and a curved surface;
fitting intersection characteristic points to obtain welding track point coordinates, and outputting XYZ coordinate values of a plurality of sampling points to a robot controller to help the robot to correct a welding track;
analyzing the three-dimensional point cloud of the welding seam track accessory to obtain the actual width of each welding seam for guiding the movement mode of the tail end of the welding gun of the robot and the arc welding current;
giving the angle posture of the welding gun through manual teaching, and welding each pipe fitting with the plate welding line one by one;
if the welding process has thermal deformation, the above steps are repeated.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110171580.0A CN112958973A (en) | 2021-02-08 | 2021-02-08 | Welding vision locating device of medium plate robot based on structured light three-dimensional vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110171580.0A CN112958973A (en) | 2021-02-08 | 2021-02-08 | Welding vision locating device of medium plate robot based on structured light three-dimensional vision |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112958973A true CN112958973A (en) | 2021-06-15 |
Family
ID=76275417
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110171580.0A Pending CN112958973A (en) | 2021-02-08 | 2021-02-08 | Welding vision locating device of medium plate robot based on structured light three-dimensional vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112958973A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114346528A (en) * | 2021-12-09 | 2022-04-15 | 成都卡诺普机器人技术股份有限公司 | Method for high-precision identification of plate pipe welding seam and acquisition of welding track based on laser scanning |
| CN114419046A (en) * | 2022-03-30 | 2022-04-29 | 季华实验室 | Method and device for recognizing weld of H-shaped steel, electronic equipment and storage medium |
| CN114670199A (en) * | 2022-03-29 | 2022-06-28 | 深圳市智流形机器人技术有限公司 | Identification positioning device, system and real-time tracking system |
| CN114789448A (en) * | 2022-05-31 | 2022-07-26 | 杭州固建机器人科技有限公司 | Steel member welding device and welding method thereof |
| CN114851188A (en) * | 2022-03-29 | 2022-08-05 | 深圳市智流形机器人技术有限公司 | Identification positioning method and device, and real-time tracking method and device |
| CN115032200A (en) * | 2022-08-10 | 2022-09-09 | 广东省智能机器人研究院 | Method and system for detecting 3D defects of sealing weld at top of new energy cylindrical battery |
| CN115026480A (en) * | 2022-07-08 | 2022-09-09 | 天津大学 | Full-automatic assembling and welding device and method for offshore platform jacket |
| CN115213600A (en) * | 2022-08-31 | 2022-10-21 | 深圳前海瑞集科技有限公司 | Method and device for identifying curved surface weld joint in welding workstation equipment |
| CN116690046A (en) * | 2023-08-04 | 2023-09-05 | 中国电力科学研究院有限公司 | Weld position indicator, method for positioning end point of fillet weld and welding method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102721378A (en) * | 2012-06-20 | 2012-10-10 | 北京航空航天大学 | Three-dimensional mirror object shape measurement system based on sinusoidal stripe projection |
| CN105783726A (en) * | 2016-04-29 | 2016-07-20 | 无锡科技职业学院 | Curve-welding-seam three-dimensional reconstruction method based on line structure light vision detection |
| CN106405829A (en) * | 2016-11-23 | 2017-02-15 | 青岛小优智能科技有限公司 | Laser structure light 3D imaging method |
| CN108453439A (en) * | 2018-03-14 | 2018-08-28 | 清华大学天津高端装备研究院洛阳先进制造产业研发基地 | The robot welding track self-programming system and method for view-based access control model sensing |
| CN110508906A (en) * | 2019-08-30 | 2019-11-29 | 上海发那科机器人有限公司 | A kind of method that robotic laser displacement sensor seeks position |
| CN110524581A (en) * | 2019-09-16 | 2019-12-03 | 西安中科光电精密工程有限公司 | A kind of flexible welding robot system and its welding method |
| CN111745266A (en) * | 2020-06-09 | 2020-10-09 | 宝冠科技(苏州)有限公司 | Corrugated board welding track generation method and system based on 3D vision position finding |
| CN112223294A (en) * | 2020-10-22 | 2021-01-15 | 湖南大学 | Mechanical arm machining track correction method based on three-dimensional vision |
-
2021
- 2021-02-08 CN CN202110171580.0A patent/CN112958973A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102721378A (en) * | 2012-06-20 | 2012-10-10 | 北京航空航天大学 | Three-dimensional mirror object shape measurement system based on sinusoidal stripe projection |
| CN105783726A (en) * | 2016-04-29 | 2016-07-20 | 无锡科技职业学院 | Curve-welding-seam three-dimensional reconstruction method based on line structure light vision detection |
| CN106405829A (en) * | 2016-11-23 | 2017-02-15 | 青岛小优智能科技有限公司 | Laser structure light 3D imaging method |
| CN108453439A (en) * | 2018-03-14 | 2018-08-28 | 清华大学天津高端装备研究院洛阳先进制造产业研发基地 | The robot welding track self-programming system and method for view-based access control model sensing |
| CN110508906A (en) * | 2019-08-30 | 2019-11-29 | 上海发那科机器人有限公司 | A kind of method that robotic laser displacement sensor seeks position |
| CN110524581A (en) * | 2019-09-16 | 2019-12-03 | 西安中科光电精密工程有限公司 | A kind of flexible welding robot system and its welding method |
| CN111745266A (en) * | 2020-06-09 | 2020-10-09 | 宝冠科技(苏州)有限公司 | Corrugated board welding track generation method and system based on 3D vision position finding |
| CN112223294A (en) * | 2020-10-22 | 2021-01-15 | 湖南大学 | Mechanical arm machining track correction method based on three-dimensional vision |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114346528A (en) * | 2021-12-09 | 2022-04-15 | 成都卡诺普机器人技术股份有限公司 | Method for high-precision identification of plate pipe welding seam and acquisition of welding track based on laser scanning |
| CN114670199A (en) * | 2022-03-29 | 2022-06-28 | 深圳市智流形机器人技术有限公司 | Identification positioning device, system and real-time tracking system |
| CN114851188A (en) * | 2022-03-29 | 2022-08-05 | 深圳市智流形机器人技术有限公司 | Identification positioning method and device, and real-time tracking method and device |
| CN114419046A (en) * | 2022-03-30 | 2022-04-29 | 季华实验室 | Method and device for recognizing weld of H-shaped steel, electronic equipment and storage medium |
| CN114419046B (en) * | 2022-03-30 | 2022-06-28 | 季华实验室 | Method and device for recognizing weld of H-shaped steel, electronic equipment and storage medium |
| CN114789448A (en) * | 2022-05-31 | 2022-07-26 | 杭州固建机器人科技有限公司 | Steel member welding device and welding method thereof |
| CN115026480A (en) * | 2022-07-08 | 2022-09-09 | 天津大学 | Full-automatic assembling and welding device and method for offshore platform jacket |
| CN115032200A (en) * | 2022-08-10 | 2022-09-09 | 广东省智能机器人研究院 | Method and system for detecting 3D defects of sealing weld at top of new energy cylindrical battery |
| CN115032200B (en) * | 2022-08-10 | 2022-10-21 | 广东省智能机器人研究院 | Method and system for detecting 3D defects of sealing weld at top of new energy cylindrical battery |
| CN115213600A (en) * | 2022-08-31 | 2022-10-21 | 深圳前海瑞集科技有限公司 | Method and device for identifying curved surface weld joint in welding workstation equipment |
| CN116690046A (en) * | 2023-08-04 | 2023-09-05 | 中国电力科学研究院有限公司 | Weld position indicator, method for positioning end point of fillet weld and welding method |
| CN116690046B (en) * | 2023-08-04 | 2023-09-29 | 中国电力科学研究院有限公司 | Weld position indicator, method for positioning end point of fillet weld and welding method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112958973A (en) | Welding vision locating device of medium plate robot based on structured light three-dimensional vision | |
| Lei et al. | A review of vision-aided robotic welding | |
| CN108747132B (en) | Autonomous mobile welding robot vision control system | |
| US7034249B2 (en) | Method of controlling the welding of a three-dimensional structure | |
| CN102303190B (en) | Method for visually tracking plane abut-jointed weld beam by linear laser | |
| US5001324A (en) | Precision joint tracking laser welding system | |
| CN112958958B (en) | MEMS micro-mirror scanning and line scanning mixed laser welding seam scanning device and scanning method | |
| CN105458483B (en) | Postwelding weld joint tracking robot automatic deviation correction and ultrasonic impact system | |
| Ryberg et al. | Stereo vision for path correction in off-line programmed robot welding | |
| CN108127217B (en) | Welding device and method for automatic guided welding of end face weld joint of spiral plate type heat exchanger | |
| WO2013148522A1 (en) | Lasergrammetry system and methods | |
| Chen et al. | Seam tracking of large pipe structures for an agile robotic welding system mounted on scaffold structures | |
| CN201693290U (en) | Laser processing device | |
| CN101750018A (en) | Non-contact real-time displacement measuring method and device in bending deformation process of work piece | |
| CN109865919A (en) | A kind of real-time weld seam tracing system of right angle welding robot line laser | |
| CN109128540B (en) | Method for determining laser welding focal track of T-shaped joint | |
| Tran et al. | An intelligent path planning of welding robot based on multisensor interaction | |
| JP2003114105A (en) | Construction method for large structure | |
| JPH1094874A (en) | Automatic welding method for tube joint | |
| Yang et al. | A fast calibration of laser vision robotic welding systems using automatic path planning | |
| CN113280790B (en) | Building mapping device based on laser positioning | |
| Yin et al. | Welding seam detection and tracking based on laser vision for robotic arc welding | |
| JPH0545117A (en) | Optical method for measuring three-dimensional position | |
| JPH10311706A (en) | Butt position detector | |
| CN112254652A (en) | Workpiece connecting point positioning device and method of single laser sensor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210615 |
|
| RJ01 | Rejection of invention patent application after publication |