CN112677159B - Mechanical arm operation path planning method and device based on visual positioning - Google Patents
Mechanical arm operation path planning method and device based on visual positioning Download PDFInfo
- Publication number
- CN112677159B CN112677159B CN202110301062.6A CN202110301062A CN112677159B CN 112677159 B CN112677159 B CN 112677159B CN 202110301062 A CN202110301062 A CN 202110301062A CN 112677159 B CN112677159 B CN 112677159B
- Authority
- CN
- China
- Prior art keywords
- operated
- manipulator
- target object
- obstacle
- position information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000002093 peripheral effect Effects 0.000 claims abstract description 36
- 230000005540 biological transmission Effects 0.000 claims description 38
- 238000012544 monitoring process Methods 0.000 claims description 14
- 238000004519 manufacturing process Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000002542 deteriorative effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Landscapes
- Manipulator (AREA)
- Numerical Control (AREA)
Abstract
The invention discloses a manipulator operation path planning method and a manipulator operation path planning device based on visual positioning, wherein the method comprises the following steps: starting visual equipment arranged on the manipulator to perform matching identification on a target object to be operated in a workbench within the operation range of the manipulator; positioning a target object to be operated to obtain position information of the target object to be operated; obtaining relative position information of a target object to be operated and each peripheral obstacle and the size relation of each obstacle based on visual equipment; forming an initial operation planning path; when the operation is carried out based on the initial operation planning path, the target object to be operated is positioned in real time based on the visual equipment, and the initial operation planning path is updated based on the real-time position information. The invention can automatically form the operation planning path according to the position of the target to be operated on the workbench and the condition of the obstacle, and update the operation planning path in real time in the operation process, thereby improving the operation precision.
Description
Technical Field
The invention relates to the technical field of automatic control, in particular to a manipulator operation path planning method and device based on visual positioning.
Background
The manipulator is an automatic operation device which can imitate certain motion functions of a human hand and an arm and is used for grabbing and carrying objects or operating tools according to a fixed program; the robot is characterized in that various expected operations can be completed through programming, and the robot has the advantages of both human and mechanical arm machines in structure and performance; in the conventional automated production line, the robot is basically applied to perform the automatic operation, but the path of the conventional robot operation is basically planned in advance, and the operation path is easily deviated with the lapse of operation time during the operation, thereby deteriorating the operation accuracy.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and provides a manipulator operation path planning method and device based on visual positioning, which can realize automatic formation of an operation planning path according to the position of a target to be operated on a workbench and the condition of an obstacle, and can update the operation planning path in real time in the operation process, thereby improving the operation precision.
In order to solve the technical problem, an embodiment of the present invention provides a manipulator operation path planning method based on visual positioning, where the method includes:
starting visual equipment arranged on a manipulator to perform matching identification on a target object to be operated in a workbench within the operation range of the manipulator;
when a target object to be operated is matched, positioning the target object to be operated to obtain position information of the target object to be operated;
obtaining relative position information of the target object to be operated and each peripheral obstacle and the size relation of each obstacle based on the visual equipment;
forming an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relation of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content;
and when the manipulator operates the target object to be operated based on the initial operation planned path, the target object to be operated is positioned in real time based on the visual equipment, and the initial operation planned path is updated based on real-time position information.
Optionally, the workbench is a working area controlled by the manipulator on the production line;
the vision equipment is double-camera equipment, and the vision equipment is arranged on the manipulator and forms a hand-eye system with the manipulator.
Optionally, the starting of the visual device arranged on the manipulator matches and identifies the target object to be operated in the workbench within the operation range of the manipulator, and the starting of the visual device includes:
a control module of the manipulator receives a control file sent by a control end based on a file transmission interface;
the control module of the manipulator analyzes the control file to obtain a target object to be operated and operation contents in the control file;
starting a visual device arranged on the manipulator to perform real-time video monitoring on a workbench in the operation range of the manipulator;
matching and identifying a target object in real-time video monitoring with a target object to be operated in the control file;
the file transmission interface comprises a video window interface and a file uploading interface.
Optionally, the positioning the target object to be operated to obtain the position information of the target object to be operated includes:
and positioning the target object to be operated based on a hand-eye system formed by the vision equipment and the manipulator to obtain the coordinate position information of the target object to be operated in the hand-eye system.
Optionally, the obtaining, based on the visual device, the relative position information between the target object to be operated and each peripheral obstacle and the size relationship between the target object to be operated and each peripheral obstacle includes:
carrying out relative position positioning identification on each obstacle around the target object to be operated based on a hand-eye system formed by the vision equipment and the manipulator, and obtaining relative coordinate position information of the target object to be operated and each obstacle around the target object to be operated in the hand-eye system;
performing three-dimensional modeling based on coordinate position information occupied by each obstacle in the hand-eye system to obtain a three-dimensional model of each obstacle;
and obtaining the size relation of each obstacle based on the three-dimensional model of each obstacle, wherein the size relation of each obstacle comprises the length, width and height relation of each obstacle.
Optionally, the forming an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator, and the operation content includes:
after obtaining the position information of a target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content, the control module of the manipulator forms an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content;
and the moving speed of the workbench in the working range of the manipulator is the moving speed of the production line corresponding to the working area controlled by the manipulator.
Optionally, when the manipulator performs operation on the target object to be operated based on the initial operation planned path, the positioning of the target object to be operated based on the visual device in real time includes:
and when the manipulator operates the target object to be operated based on the initial operation planning path, the target object to be operated is positioned in real time through a hand-eye system formed by the vision equipment and the manipulator.
Optionally, the updating the initial job planning path based on the real-time location information includes:
after the real-time position information is obtained, the control module of the manipulator updates the path in real time on the basis of the initial operation planned path on the basis of the real-time position information.
In addition, the embodiment of the invention also provides a manipulator operation path planning device based on visual positioning, which comprises:
a matching identification module: the visual equipment is used for starting the visual equipment arranged on the manipulator to perform matching identification on a target object to be operated in the workbench within the operation range of the manipulator;
a positioning processing module: the system comprises a target object to be operated, a positioning module and a processing module, wherein the target object to be operated is matched with the target object to be operated;
an obtaining module: the vision equipment is used for obtaining the relative position information of the target object to be operated and each peripheral obstacle and the size relation of each obstacle;
a path planning module: for determining the position of the target object based on the position information of the target object, the relative position information of the target object and surrounding obstacles forming an initial operation planning path by the size relation of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content;
a path update module: and the manipulator is used for positioning the target object to be operated in real time based on the visual equipment and updating the initial operation planning path based on real-time position information when the manipulator operates the target object to be operated based on the initial operation planning path.
In the embodiment of the invention, the vision equipment is arranged on the mechanical arm, the hand-eye system is formed between the vision equipment and the mechanical arm, the target object to be operated on the workbench is positioned in real time through the hand-eye system, and meanwhile, the obstacles and the like around the target object to be operated are identified and positioned, so that an initial operation planning path is formed, and in the operation process, the operation planning path is updated, so that the operation planning path can be automatically formed according to the position of the target object to be operated on the workbench and the obstacle condition, and the operation planning path is updated in real time in the operation process, so that the operation precision is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a method for planning a working path of a manipulator based on visual positioning according to an embodiment of the present invention;
fig. 2 is a schematic structural component diagram of a robot working path planning apparatus based on visual positioning according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The first embodiment is as follows:
referring to fig. 1, fig. 1 is a schematic flow chart illustrating a method for planning a working path of a robot based on visual positioning according to an embodiment of the present invention.
As shown in fig. 1, a method for planning a working path of a manipulator based on visual positioning includes:
s11: starting visual equipment arranged on a manipulator to perform matching identification on a target object to be operated in a workbench within the operation range of the manipulator;
in the specific implementation process of the invention, the workbench is a working area controlled by the manipulator on the production line; the vision equipment is double-camera equipment, and the vision equipment is arranged on the manipulator and forms a hand-eye system with the manipulator.
Further, the starting of the visual device arranged on the manipulator matches and identifies the target object to be operated in the workbench within the operation range of the manipulator, and the starting of the visual device comprises the following steps: the control module of the manipulator receives a control file sent by the control end based on the file transmission interface; the control module of the manipulator analyzes the control file to obtain a target object to be operated and operation contents in the control file; starting a visual device arranged on the manipulator to perform real-time video monitoring on a workbench in the operation range of the manipulator; matching and identifying a target object in real-time video monitoring with a target object to be operated in the control file; the file transmission interface comprises a video window interface and a file uploading interface.
Specifically, the workbench is a work area controlled by the manipulator on the production line, the vision device is a dual-camera device, both the dual-camera devices are CCD devices, and the dual-camera device is disposed on the manipulator to form a hand-eye system, which can adopt image-based vision control, position-based vision control, and hybrid vision control combining the two. For the vision control based on the image, because a closed loop is formed in the image space, the calibration error of the camera can be effectively overcome, and the requirement on the precision of the camera calibration is not high.
The manipulator is provided with a control module for controlling the manipulator to operate; the control module of the manipulator receives the control file sent by the control end, and then the control module analyzes the control file, so that a target object to be operated and operation contents in the control file are obtained; after a target object to be operated and operation contents in the control file are obtained, starting visual equipment arranged on the manipulator to carry out real-time video monitoring on a workbench in the operation range of the manipulator; simultaneously, matching and identifying a target object in the real-time video monitoring and a target object to be operated in the control file; the control end sends a control file to a control module of the manipulator through a file transmission interface, and the file transmission interface comprises a video window interface and a file uploading interface.
When the file transmission interface is a video window interface, the control end transmits a control file according to the DiiVA technology in an IP streaming media mode; specifically, the control end adopts an MPEG transport stream coding conforming to the DiiVA transmission mode, adopts a fine granular scalable coding (FGS) or progressive fine granular scalable coding (PFGS) technology in the MPEG-4 standard, and combines an h.264 technology with a network transmission protocol; real-time transmission error control is carried out on data by adopting a real-time transmission protocol or a transmission control protocol; and when the file transmission interface is a file uploading interface, the control end sends the control file according to the file uploading interface.
S12: when a target object to be operated is matched, positioning the target object to be operated to obtain position information of the target object to be operated;
in a specific implementation process of the present invention, the positioning the target object to be operated to obtain the position information of the target object to be operated includes: and positioning the target object to be operated based on a hand-eye system formed by the vision equipment and the manipulator to obtain the coordinate position information of the target object to be operated in the hand-eye system.
Specifically, when a target object to be operated is matched, a related coordinate system is constructed through a hand-eye system formed by the vision equipment and the manipulator, and it is assumed that one camera of the double-camera equipment in the vision equipment is marked as O1, and the other camera is marked as O2; one of the cameras is taken as a coordinate origin of a coordinate system, O1 is taken as the coordinate origin of the coordinate system, a connecting line between two camera devices is taken as an X axis, the distance between the two points is b, in the double-camera device, the X axis is vertical to the Y axis, a Z axis in the depth direction is determined by adopting a right hand rule, and the height of an effective view field of the double-camera device is the same as that of the single camera device; when two cameras are placed randomly, a line is connected in a Z-axis direction in a coordinate system XOZ plane to form an effective visual field of a binocular system, the width of the effective visual field is w, the projection of any target point P (x, y, Z) on images collected by the two cameras in the binocular camera equipment is respectively P1 (u 1, v 1), P2 (u 2, v 2), the field angles of the point P relative to perspective central points O1 and O2 of the binocular camera equipment are respectively a1 and a2, the model is simplified, the cameras are assumed to have the same configuration and are symmetrically placed, the coordinate P (x, y, Z) value of the point P can be calculated through the geometrical relationship, and the coordinate position information of an object to be worked in the hand-eye system can be obtained.
S13: obtaining relative position information of the target object to be operated and each peripheral obstacle and the size relation of each obstacle based on the visual equipment;
in the specific implementation process of the invention, based on a hand-eye system formed by the vision equipment and the manipulator, the relative position of each obstacle around the target object to be operated is positioned and identified, and the relative coordinate position information of the target object to be operated and each obstacle around the target object to be operated in the hand-eye system is obtained; performing three-dimensional modeling based on coordinate position information occupied by each obstacle in the hand-eye system to obtain a three-dimensional model of each obstacle; and obtaining the size relation of each obstacle based on the three-dimensional model of each obstacle, wherein the size relation of each obstacle comprises the length, width and height relation of each obstacle.
Specifically, a relative coordinate system established by a hand-eye system formed by vision equipment and a manipulator is utilized to perform relative position positioning identification on each obstacle around a target object to be operated, so that relative coordinate position information of the target object to be operated and each peripheral obstacle in the hand-eye system is obtained, and then three-dimensional modeling is performed through the coordinate position information occupied by each obstacle in the hand-eye system, so as to obtain a three-dimensional model of each obstacle; and finally, obtaining the size relation of each obstacle according to the three-dimensional model of each obstacle, wherein the size relation of each obstacle comprises the length, width and height relation of each obstacle.
S14: forming an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relation of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content;
in a specific implementation process of the present invention, the forming an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator, and the operation content includes: after acquiring the position information of a target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content, the control module of the manipulator forms an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content; and the moving speed of the workbench in the working range of the manipulator is the moving speed of the production line corresponding to the working area controlled by the manipulator.
Specifically, after the control module of the manipulator obtains the position information of the target to be operated, the relative position information of the target to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content, an initial operation planning path is formed according to the position information of the target to be operated, the relative position information of the target to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content according to preset operation requirements, and the moving speed of the workbench in the working range of the manipulator is the pipeline moving speed corresponding to the working area controlled by the manipulator.
S15: and when the manipulator operates the target object to be operated based on the initial operation planned path, the target object to be operated is positioned in real time based on the visual equipment, and the initial operation planned path is updated based on real-time position information.
In the specific implementation process of the present invention, when the manipulator performs operation on the target object to be operated based on the initial operation planned path, the real-time positioning of the target object to be operated based on the visual device includes: and when the manipulator operates the target object to be operated based on the initial operation planning path, the target object to be operated is positioned in real time through a hand-eye system formed by the vision equipment and the manipulator.
Further, the updating the initial operation planned path based on the real-time position information includes: after the real-time position information is obtained, the control module of the manipulator updates the path in real time on the basis of the initial operation planned path on the basis of the real-time position information.
Specifically, when the manipulator operates the target object to be operated according to the initial operation planning path, the target object to be operated is positioned in real time through a hand-eye system formed by the vision equipment and the manipulator; and after the real-time position information is obtained, the control module of the manipulator updates the path in real time on the basis of the initial operation path according to the real-time position information.
In the embodiment of the invention, the vision equipment is arranged on the mechanical arm, the hand-eye system is formed between the vision equipment and the mechanical arm, the target object to be operated on the workbench is positioned in real time through the hand-eye system, and meanwhile, the obstacles and the like around the target object to be operated are identified and positioned, so that an initial operation planning path is formed, and the operation planning path is updated in the operation process, so that the operation planning path can be automatically formed according to the position of the target object to be operated on the workbench and the obstacle condition, and the operation planning path is updated in real time in the operation process, so that the operation precision is improved.
Example two
Referring to fig. 2, fig. 2 is a schematic structural diagram of a robot operation path planning apparatus based on visual positioning according to an embodiment of the present invention.
As shown in fig. 2, a manipulator working path planning device based on visual positioning comprises:
the matching identification module 21: the system comprises a manipulator, a vision device, a detection device, a display device and a control device, wherein the vision device is used for starting the vision device arranged on the manipulator to perform matching identification on a target object to be operated in a workbench in the operation range of the manipulator;
in the specific implementation process of the invention, the workbench is a working area controlled by the manipulator on the production line; the vision equipment is double-camera equipment, and the vision equipment is arranged on the manipulator and forms a hand-eye system with the manipulator.
Further, the starting of the visual device arranged on the manipulator matches and identifies the target object to be operated in the workbench in the operation range of the manipulator, and the method includes: a control module of the manipulator receives a control file sent by a control end based on a file transmission interface; the control module of the manipulator analyzes the control file to obtain a target object to be operated and operation contents in the control file; starting a visual device arranged on the manipulator to perform real-time video monitoring on a workbench in the operation range of the manipulator; matching and identifying a target object in real-time video monitoring with a target object to be operated in the control file; the file transmission interface comprises a video window interface and a file uploading interface.
Specifically, the workbench is a work area controlled by the manipulator on the production line, the vision device is a dual-camera device, both the dual-camera devices are CCD devices, and the dual-camera device is disposed on the manipulator to form a hand-eye system, which can adopt image-based vision control, position-based vision control, and hybrid vision control combining the two. For the vision control based on the image, because a closed loop is formed in the image space, the calibration error of the camera can be effectively overcome, and the requirement on the precision of the camera calibration is not high.
A control module is arranged on the manipulator and used for controlling the manipulator to operate; the control module of the manipulator receives the control file sent by the control end, and then the control module analyzes the control file, so that a target object to be operated and operation contents in the control file are obtained; after a target object to be operated and operation contents in the control file are obtained, starting visual equipment arranged on the manipulator to carry out real-time video monitoring on a workbench in the operation range of the manipulator; simultaneously, matching and identifying a target object in the real-time video monitoring with a target object to be operated in the control file; the control end sends a control file to a control module of the manipulator through a file transmission interface, and the file transmission interface comprises a video window interface and a file uploading interface.
When the file transmission interface is a video window interface, the control end transmits a control file according to the DiiVA technology in an IP streaming media mode; specifically, the control end adopts MPEG transport stream coding conforming to DiiVA transmission mode, fine granular scalable coding (FGS) or progressive fine granular scalable coding (PFGS) technology is adopted in MPEG-4 standard, and H.264 technology is combined with network transmission protocol; real-time transmission error control is carried out on data by adopting a real-time transmission protocol or a transmission control protocol; and when the file transmission interface is a file uploading interface, the control end sends the control file according to the file uploading interface.
The positioning processing module 22: the system comprises a target object to be operated, a positioning module and a processing module, wherein the target object to be operated is matched with the target object to be operated;
in a specific implementation process of the present invention, the positioning the target object to be operated to obtain the position information of the target object to be operated includes: and positioning the target object to be operated based on a hand-eye system formed by the vision equipment and the manipulator to obtain the coordinate position information of the target object to be operated in the hand-eye system.
Specifically, when a target object to be operated is matched, a related coordinate system is constructed through a hand-eye system formed by the vision equipment and the manipulator, and it is assumed that one camera of the double-camera equipment in the vision equipment is marked as O1, and the other camera is marked as O2; one camera is used as a coordinate origin of a coordinate system, O1 is used as the coordinate origin of the coordinate system, a connecting line between two camera devices is an X axis, the distance between the two points is b, in the double-camera device, the X axis is vertical to the Y axis, the Z axis in the depth direction is determined by adopting a right hand criterion, and the height of an effective visual field of the double-camera device is the same as that of a single camera device; when two cameras are placed randomly, a line is connected in a Z-axis direction in a coordinate system XOZ plane to form an effective visual field of a binocular system, the width of the effective visual field is w, the projections of any target point P (x, y, Z) on images collected by the two cameras in the binocular camera equipment are respectively P1 (u 1, v 1) and P2 (u 2, v 2), the field angles of the point P relative to perspective central points O1 and O2 of the dual-camera equipment are respectively a1 and a2, the model is simplified, the cameras are assumed to have the same configuration and are symmetrically placed, the coordinate P (x, y, Z) value of the point P can be calculated through a geometrical relationship, and the coordinate position information of an object to be worked in a hand-eye system can be obtained.
The obtaining module 23: the vision equipment is used for acquiring relative position information of the target object to be operated and each peripheral obstacle and the size relation of each obstacle;
in the specific implementation process of the invention, based on a hand-eye system formed by the vision equipment and the manipulator, the relative position of each obstacle around the target object to be operated is positioned and identified, and the relative coordinate position information of the target object to be operated and each obstacle around the target object to be operated in the hand-eye system is obtained; performing three-dimensional modeling based on coordinate position information occupied by each obstacle in the hand-eye system to obtain a three-dimensional model of each obstacle; and obtaining the size relation of each obstacle based on the three-dimensional model of each obstacle, wherein the size relation of each obstacle comprises the length, width and height relation of each obstacle.
Specifically, a relative coordinate system established by a hand-eye system formed by vision equipment and a manipulator is utilized to perform relative position positioning identification on each obstacle around a target object to be operated, so that relative coordinate position information of the target object to be operated and each peripheral obstacle in the hand-eye system is obtained, and then three-dimensional modeling is performed through the coordinate position information occupied by each obstacle in the hand-eye system, so as to obtain a three-dimensional model of each obstacle; and finally, obtaining the size relation of each obstacle according to the three-dimensional model of each obstacle, wherein the size relation of each obstacle comprises the length, width and height relation of each obstacle.
The path planning module 24: for determining the position of the object to be worked based on the position information of the object to be worked, the relative position information of the object to be worked and each peripheral obstacle forming an initial operation planning path by the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content;
in a specific implementation process of the present invention, the forming an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator, and the operation content includes: after acquiring the position information of a target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content, the control module of the manipulator forms an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content; and the moving speed of the workbench in the working range of the manipulator is the moving speed of the production line corresponding to the working area controlled by the manipulator.
Specifically, after the control module of the manipulator obtains the position information of the target to be operated, the relative position information of the target to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content, an initial operation planning path is formed according to the position information of the target to be operated, the relative position information of the target to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content according to preset operation requirements, and the moving speed of the workbench in the working range of the manipulator is the pipeline moving speed corresponding to the working area controlled by the manipulator.
The path update module 25: the vision device is used for positioning the target object to be operated in real time based on the vision device and updating the initial operation planning path based on real-time position information when the manipulator operates the target object to be operated based on the initial operation planning path.
In a specific implementation process of the present invention, when the manipulator performs an operation on the target object to be operated based on the initial operation planned path, the positioning of the target object to be operated based on the visual device in real time includes: and when the manipulator operates the target object to be operated based on the initial operation planning path, the target object to be operated is positioned in real time through a hand-eye system formed by the vision equipment and the manipulator.
Further, the updating the initial operation planned path based on the real-time position information includes: after obtaining the real-time position information, a control module of the manipulator updates a path in real time based on the real-time position information on the basis of the initial working path.
Specifically, when the manipulator operates the target object to be operated according to the initial operation planning path, the target object to be operated is positioned in real time through a hand-eye system formed by the vision equipment and the manipulator; and after the real-time position information is obtained, the control module of the manipulator updates the path in real time on the basis of the initial operation planned path according to the real-time position information.
In the embodiment of the invention, the vision equipment is arranged on the mechanical arm, the hand-eye system is formed between the vision equipment and the mechanical arm, the target object to be operated on the workbench is positioned in real time through the hand-eye system, and meanwhile, the obstacles and the like around the target object to be operated are identified and positioned, so that an initial operation planning path is formed, and in the operation process, the operation planning path is updated, so that the operation planning path can be automatically formed according to the position of the target object to be operated on the workbench and the obstacle condition, and the operation planning path is updated in real time in the operation process, so that the operation precision is improved.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic or optical disk, or the like.
In addition, the method and the device for planning the operation path of the manipulator based on visual positioning according to the embodiments of the present invention are described in detail, and a specific example is used herein to explain the principle and the implementation of the present invention, and the description of the embodiments is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (4)
1. A manipulator operation path planning method based on visual positioning is characterized by comprising the following steps:
starting visual equipment arranged on a manipulator to perform matching identification on a target object to be operated in a workbench within the operation range of the manipulator;
when a target object to be operated is matched, positioning the target object to be operated to obtain position information of the target object to be operated;
obtaining relative position information of the target object to be operated and each peripheral obstacle and the size relation of each obstacle based on the visual equipment;
forming an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relation of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content;
when the manipulator operates the target object to be operated based on the initial operation planned path, the target object to be operated is positioned in real time based on the visual equipment, and the initial operation planned path is updated based on real-time position information;
the visual equipment that starts setting on the manipulator is to treat the operation target object in the workstation of manipulator operation within range and match the discernment, includes:
the control module of the manipulator receives a control file sent by the control end based on the file transmission interface;
the control module of the manipulator analyzes the control file to obtain a target object to be operated and operation contents in the control file;
starting a visual device arranged on the manipulator to perform real-time video monitoring on a workbench in the operation range of the manipulator;
matching and identifying a target object in real-time video monitoring with a target object to be operated in the control file;
the file transmission interface comprises a video window interface and a file uploading interface;
when the file transmission interface is a video window interface, the control end transmits a control file according to the DiiVA technology in an IP streaming media mode; when the file transmission interface is a file uploading interface, the control terminal sends a control file according to the file uploading interface; real-time transmission error control is carried out on data by adopting a real-time transmission protocol or a transmission control protocol;
the obtaining of the relative position information between the target object to be operated and each peripheral obstacle and the size relationship between the target object to be operated and each peripheral obstacle based on the visual equipment includes:
performing relative position positioning identification on each obstacle around the target object to be operated based on a hand-eye system formed by the vision equipment and the manipulator to obtain relative coordinate position information of the target object to be operated and each obstacle around the target object to be operated in the hand-eye system;
performing three-dimensional modeling based on coordinate position information occupied by each obstacle in the hand-eye system to obtain a three-dimensional model of each obstacle;
obtaining the size relation of each obstacle based on the three-dimensional model of each obstacle, wherein the size relation of each obstacle comprises the length, width and height relation of each obstacle;
the forming of the initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content comprises the following steps:
after acquiring the position information of a target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content, the control module of the manipulator forms an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content;
the moving speed of the workbench in the working range of the manipulator is the pipeline moving speed corresponding to the working area controlled by the manipulator;
the positioning the target object to be operated to obtain the position information of the target object to be operated comprises the following steps:
positioning the target object to be operated based on a hand-eye system formed by the visual equipment and the manipulator to obtain coordinate position information of the target object to be operated in the hand-eye system;
when a target object to be operated is matched, a related coordinate system is constructed through a hand-eye system formed by the vision equipment and the manipulator, and one camera of the double-camera equipment in the vision equipment is assumed to be marked as O1, and the other camera is assumed to be marked as O2; one camera is used as a coordinate origin of a coordinate system, O1 is used as the coordinate origin of the coordinate system, a connecting line between two camera devices is an X axis, the distance between the two points is b, in the double-camera device, the X axis is vertical to the Y axis, the Z axis in the depth direction is determined by adopting a right hand criterion, and the height of an effective visual field of the double-camera device is the same as that of a single camera device; when two cameras are placed randomly, connecting lines along the Z-axis direction in a coordinate system XOZ plane to form an effective visual field of a binocular system, wherein the width of the effective visual field is w, the projections of any target point P (x, y, Z) on images collected by the two cameras in the binocular camera equipment are respectively P1 (u 1, v 1) and P2 (u 2, v 2), the field angles of the point P relative to perspective central points O1 and O2 of the dual-camera equipment are respectively a1 and a2, the model is simplified, the cameras are assumed to have the same configuration and are symmetrically placed, the coordinate P (x, y, Z) value of the point P can be calculated through a geometrical relationship, and the coordinate position information of an object to be worked in a hand-eye system can be obtained;
when the manipulator operates the target object to be operated based on the initial operation planning path, the manipulator positions the target object to be operated in real time based on the visual equipment, and the method comprises the following steps:
and when the manipulator operates the target object to be operated based on the initial operation planning path, the target object to be operated is positioned in real time through a hand-eye system formed by the vision equipment and the manipulator.
2. The manipulator work path planning method according to claim 1, wherein the work table is a work area controlled on a production line by the manipulator;
the vision equipment is double-camera equipment, and the vision equipment is arranged on the manipulator and forms a hand-eye system with the manipulator.
3. The manipulator work path planning method according to claim 1, wherein the updating of the initial work planned path based on the real-time position information includes:
after obtaining the real-time position information, a control module of the manipulator performs real-time path update on the basis of the initial working path based on the real-time position information.
4. A vision positioning based manipulator work path planning apparatus, the apparatus comprising:
a matching identification module: the visual equipment is used for starting the visual equipment arranged on the manipulator to perform matching identification on a target object to be operated in the workbench within the operation range of the manipulator;
a positioning processing module: the system comprises a target object to be operated, a positioning module and a processing module, wherein the target object to be operated is matched with the target object to be operated;
an obtaining module: the vision equipment is used for obtaining the relative position information of the target object to be operated and each peripheral obstacle and the size relation of each obstacle;
a path planning module: for determining the position of the object to be worked based on the position information of the object to be worked, the relative position information of the object to be worked and each peripheral obstacle forming an initial operation planning path by the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content;
a path update module: the vision equipment is used for positioning the target object to be operated in real time based on the vision equipment and updating the initial operation planning path based on real-time position information when the manipulator operates the target object to be operated based on the initial operation planning path;
a matching identification module: the control module is also used for receiving a control file sent by the control end based on the file transmission interface; the control module of the manipulator analyzes the control file to obtain a target object to be operated and operation content in the control file; starting a visual device arranged on the manipulator to perform real-time video monitoring on a workbench in the operation range of the manipulator; matching and identifying a target object in real-time video monitoring and a target object to be operated in the control file; the file transmission interface comprises a video window interface and a file uploading interface; when the file transmission interface is a video window interface, the control end transmits a control file according to the DiiVA technology in an IP streaming media mode; when the file transmission interface is a file uploading interface, the control end sends a control file according to the file uploading interface; real-time transmission error control is carried out on data by adopting a real-time transmission protocol or a transmission control protocol;
the obtaining module: the robot is further used for carrying out relative position positioning identification on each obstacle around the target object to be operated based on a hand-eye system formed by the vision equipment and the manipulator, and obtaining relative coordinate position information of the target object to be operated and each obstacle around the target object to be operated in the hand-eye system; performing three-dimensional modeling based on coordinate position information occupied by each obstacle in the hand-eye system to obtain a three-dimensional model of each obstacle; obtaining the size relation of each obstacle based on the three-dimensional model of each obstacle, wherein the size relation of each obstacle comprises the length, width and height relation of each obstacle;
the path planning module: the control module is also used for forming an initial operation planning path based on the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content after the control module of the manipulator obtains the position information of the target to be operated, the relative position information of the target object to be operated and each peripheral obstacle, the size relationship of each obstacle, the moving speed of the workbench in the working range of the manipulator and the operation content; the moving speed of the workbench in the working range of the manipulator is the pipeline moving speed corresponding to the working area controlled by the manipulator;
the positioning the target object to be operated to obtain the position information of the target object to be operated comprises the following steps:
positioning the target object to be operated based on a hand-eye system formed by the visual equipment and the manipulator to obtain coordinate position information of the target object to be operated in the hand-eye system;
when a target object to be operated is matched, a related coordinate system is constructed through a hand-eye system formed by the vision equipment and the manipulator, and one camera of the double-camera equipment in the vision equipment is assumed to be marked as O1, and the other camera is assumed to be marked as O2; one of the cameras is taken as a coordinate origin of a coordinate system, O1 is taken as the coordinate origin of the coordinate system, a connecting line between two camera devices is taken as an X axis, the distance between the two points is b, in the double-camera device, the X axis is vertical to the Y axis, a Z axis in the depth direction is determined by adopting a right hand rule, and the height of an effective view field of the double-camera device is the same as that of the single camera device; when two cameras are placed randomly, connecting lines along the Z-axis direction in a coordinate system XOZ plane to form an effective visual field of a binocular system, wherein the width of the effective visual field is w, the projections of any target point P (x, y, Z) on images collected by the two cameras in the binocular camera equipment are respectively P1 (u 1, v 1) and P2 (u 2, v 2), the field angles of the point P relative to perspective central points O1 and O2 of the dual-camera equipment are respectively a1 and a2, the model is simplified, the cameras are assumed to have the same configuration and are symmetrically placed, the coordinate P (x, y, Z) value of the point P can be calculated through a geometrical relationship, and the coordinate position information of an object to be worked in a hand-eye system can be obtained;
when the manipulator operates the target object to be operated based on the initial operation planning path, the manipulator positions the target object to be operated in real time based on the visual equipment, and the method comprises the following steps:
and when the manipulator operates the target object to be operated based on the initial operation planning path, the target object to be operated is positioned in real time through a hand-eye system formed by the vision equipment and the manipulator.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110301062.6A CN112677159B (en) | 2021-03-22 | 2021-03-22 | Mechanical arm operation path planning method and device based on visual positioning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110301062.6A CN112677159B (en) | 2021-03-22 | 2021-03-22 | Mechanical arm operation path planning method and device based on visual positioning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112677159A CN112677159A (en) | 2021-04-20 |
| CN112677159B true CN112677159B (en) | 2023-02-28 |
Family
ID=75455722
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110301062.6A Active CN112677159B (en) | 2021-03-22 | 2021-03-22 | Mechanical arm operation path planning method and device based on visual positioning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112677159B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113180894B (en) * | 2021-04-27 | 2022-02-11 | 浙江大学 | Visual intelligence-based hand-eye coordination method and device for multiple-obstacle person |
| CN114160507B (en) * | 2021-11-24 | 2023-06-06 | 上海航翼高新技术发展研究院有限公司 | Laser cleaning path automatic planning method based on multiple sensing detection |
| CN116214531B (en) * | 2023-05-10 | 2023-06-30 | 佛山隆深机器人有限公司 | Path planning method and device for industrial robot |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104062902A (en) * | 2014-05-15 | 2014-09-24 | 江南大学 | Delta robot time optimal trajectory planning method |
| CN204866557U (en) * | 2015-05-20 | 2015-12-16 | 大连华工创新科技股份有限公司 | Express delivery of container formula and automatic sorting system |
| CN109261531A (en) * | 2018-10-30 | 2019-01-25 | 福建省亿顺机械设备有限公司 | Stacking baling line based on intelligent sorting |
| CN109911549A (en) * | 2019-01-25 | 2019-06-21 | 东华大学 | A kind of the Robotic Dynamic tracking grasping system and method for fragile goods |
| CN109926342A (en) * | 2019-03-27 | 2019-06-25 | 杭州翼道智能科技有限公司 | A kind of efficient sorting system of express delivery and its method for sorting |
| CN110976306A (en) * | 2019-10-22 | 2020-04-10 | 江苏苏宁物流有限公司 | Goods sorting method, system, control server and medium |
| US10793355B1 (en) * | 2018-04-06 | 2020-10-06 | Amazon Technologies, Inc. | Inventory storage module with robotic transport system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106444458B (en) * | 2016-11-11 | 2019-05-03 | 华南智能机器人创新研究院 | A kind of method and system of long-range control industrial robot |
-
2021
- 2021-03-22 CN CN202110301062.6A patent/CN112677159B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104062902A (en) * | 2014-05-15 | 2014-09-24 | 江南大学 | Delta robot time optimal trajectory planning method |
| CN204866557U (en) * | 2015-05-20 | 2015-12-16 | 大连华工创新科技股份有限公司 | Express delivery of container formula and automatic sorting system |
| US10793355B1 (en) * | 2018-04-06 | 2020-10-06 | Amazon Technologies, Inc. | Inventory storage module with robotic transport system |
| CN109261531A (en) * | 2018-10-30 | 2019-01-25 | 福建省亿顺机械设备有限公司 | Stacking baling line based on intelligent sorting |
| CN109911549A (en) * | 2019-01-25 | 2019-06-21 | 东华大学 | A kind of the Robotic Dynamic tracking grasping system and method for fragile goods |
| CN109926342A (en) * | 2019-03-27 | 2019-06-25 | 杭州翼道智能科技有限公司 | A kind of efficient sorting system of express delivery and its method for sorting |
| CN110976306A (en) * | 2019-10-22 | 2020-04-10 | 江苏苏宁物流有限公司 | Goods sorting method, system, control server and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112677159A (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112677159B (en) | Mechanical arm operation path planning method and device based on visual positioning | |
| CN107618030B (en) | Robot dynamic tracking grabbing method and system based on vision | |
| CN104400279B (en) | Pipeline space weld seam based on CCD identifies the method with trajectory planning automatically | |
| JP4167954B2 (en) | Robot and robot moving method | |
| WO2018043525A1 (en) | Robot system, robot system control device, and robot system control method | |
| CN113333998A (en) | Automatic welding system and method based on cooperative robot | |
| CN110640730A (en) | Method and system for generating three-dimensional model for robot scene | |
| CN104842361A (en) | Robotic system with 3d box location functionality | |
| CN109702290B (en) | Steel plate groove cutting method based on visual identification | |
| CN110900611A (en) | Novel mechanical arm target positioning and path planning method | |
| CN107103624B (en) | Stereoscopic vision conveying system and conveying method thereof | |
| CN111624994A (en) | Robot inspection method based on 5G communication | |
| CN111975200A (en) | Intelligent welding method and intelligent welding system based on visual teaching technology | |
| CN111958593B (en) | Vision servo method and system for inspection operation robot of semantic intelligent substation | |
| CN114299039B (en) | Robot and collision detection device and method thereof | |
| CN110039520B (en) | Teaching and processing system based on image contrast | |
| JP2007171018A (en) | Object position recognition method and device | |
| CN112059983A (en) | Method, device and computer readable medium for assembling workpiece | |
| CN206416179U (en) | A kind of motion target tracking positioning and grasping system based on binocular vision | |
| CN111625001B (en) | Robot control method and device and industrial robot | |
| JPS6334093A (en) | Visual device | |
| JP2005205519A (en) | Robot hand device | |
| CN114800524A (en) | System and method for actively avoiding collision of human-computer interaction cooperative robot | |
| JP2023508414A (en) | Multi-Drone Visual Content Ingestion System | |
| Sukumar et al. | Augmented reality-based tele-robotic system architecture for on-site construction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A method and device for robot arm operation path planning based on visual positioning Granted publication date: 20230228 Pledgee: Beijiao sub branch of Guangdong Shunde Rural Commercial Bank Co.,Ltd. Pledgor: FOSHAN LONGSHEN ROBOT Co.,Ltd. Registration number: Y2024980007296 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |