CN114407011A - Special-shaped workpiece grabbing planning method, planning device and special-shaped workpiece grabbing method - Google Patents
Special-shaped workpiece grabbing planning method, planning device and special-shaped workpiece grabbing method Download PDFInfo
- Publication number
- CN114407011A CN114407011A CN202210004305.4A CN202210004305A CN114407011A CN 114407011 A CN114407011 A CN 114407011A CN 202210004305 A CN202210004305 A CN 202210004305A CN 114407011 A CN114407011 A CN 114407011A
- Authority
- CN
- China
- Prior art keywords
- special
- shaped workpiece
- grabbing
- bounding box
- finger
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J15/00—Gripping heads and other end effectors
- B25J15/0033—Gripping heads and other end effectors with gripping surfaces having special shapes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0014—Image feed-back for automatic industrial control, e.g. robot with camera
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Manipulator (AREA)
Abstract
The application provides a method and a device for grabbing and planning a special-shaped workpiece, and a method and a system for grabbing the special-shaped workpiece, wherein the method for grabbing and planning the special-shaped workpiece comprises the following steps: acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image; calculating to obtain the total offset of the grabbing tool according to the mask image of the special-shaped workpiece and the bounding box information of the mask image; and planning to obtain a grabbing path of the grabbing tool according to the initial position of the grabbing tool, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system and the total offset. The grabbing path of the grabbing tool can be planned, so that the special-shaped workpiece can be stably and effectively grabbed.
Description
Technical Field
The application belongs to the field of robot grabbing planning, and particularly relates to a special-shaped workpiece grabbing planning method, a planning device, a special-shaped workpiece grabbing method and a special-shaped workpiece grabbing system.
Background
With the development of artificial intelligence technology, the application scene of the robot is more and more complex. For example, in industries such as logistics and industrial production, robots are required to autonomously perform operations such as gripping and carrying instead of human power.
At present, when a robot is used for grabbing a workpiece on an assembly line, a camera generally acquires image data of the workpiece, then the workpiece is positioned in an image processing mode, a grabbing plan scheme is established, and finally an end effector finishes grabbing. However, there are several disadvantages to this robot vision grasping method in the prior art. For example, first the robotic vision grasping method does not define the final grasping pose of the end effector. Secondly, for the special-shaped workpiece with complex structural characteristics, the robot is often difficult to accurately position the grabbing point of the special-shaped workpiece, can only grab roughly and is easy to damage the workpiece. Finally, when the common clamping jaw actuator or the common sucking disc actuator is used for grabbing the special-shaped workpiece with complex structural characteristics, the workpiece is easy to slide due to unstable grabbing.
Disclosure of Invention
In order to overcome the problems in the related art at least to a certain extent, the application provides a special-shaped workpiece grabbing planning method, a planning device, a special-shaped workpiece grabbing method and a special-shaped workpiece grabbing system.
According to a first aspect of embodiments of the present application, there is provided a method for planning grabbing of a special-shaped workpiece, including the following steps:
acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image;
calculating to obtain the total offset of the grabbing tool according to the mask image of the special-shaped workpiece and the bounding box information of the mask image;
and planning to obtain a grabbing path of the grabbing tool according to the initial position of the grabbing tool, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system and the total offset.
In the method for grabbing and planning the special-shaped workpiece, the specific process of obtaining and processing the image of the special-shaped workpiece to obtain the mask image of the special-shaped workpiece and the bounding box information of the mask image is as follows:
acquiring an original image of the special-shaped workpiece by using a camera;
processing an original image of the special-shaped workpiece by using an image processing algorithm to obtain a mask image of the special-shaped workpiece and a central point of the mask image under a camera pixel coordinate system;
and processing the mask image of the special-shaped workpiece to obtain an enclosure box of the mask image and an included angle between the main direction of the enclosure box and a horizontal axis in a camera pixel coordinate system.
Further, the specific process of calculating the total offset of the grasping tool according to the mask diagram of the special-shaped workpiece and the bounding box information of the mask diagram is as follows:
presetting an initial position of a gripping tool; the initial positions of the gripping tool are: the center point of the grabbing tool is positioned right above the center point of the mask image of the special-shaped workpiece under the camera pixel coordinate system; the center point of the cross section of each tail end in the grabbing tool is positioned right above a straight line which is coplanar and vertical to the main direction of the bounding box;
calculating a first offset of the grasping tool from the initial position to the position circumscribed with the bounding box;
calculating a second offset of the grabbing tool from the time of externally tangent with the bounding box to the time of externally tangent with the special-shaped workpiece;
and obtaining the total offset of the grabbing tool from the initial position to the external tangent of the special-shaped workpiece according to the first offset and the second offset.
Furthermore, in the grasping path of the grasping tool obtained according to the initial position of the grasping tool, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system, and the total offset, the grasping path of the grasping tool in the camera pixel coordinate system is:
the gripping tool is rotated until the main direction of the gripping tool is consistent with the main direction of the bounding box; and each tail end of the gripping tool moves towards the special-shaped workpiece by the total offset distance.
Furthermore, when the gripping tool adopts a four-finger gripping jaw with three degrees of freedom, the specific process of calculating the first offset of the gripping tool from the initial position to the position of the first offset when the gripping tool is circumscribed with the bounding box is as follows:
presetting initial offset between central points of cross sections of fingers in four-finger clamping jaw relative to central axis of main direction of bounding boxIs m1、m2、m3、m4；
Assuming that the central point of each finger cross section of the four-finger clamping jaw at the initial position is A01(x01,y01)、A02(x02,y02)、A03(x03,y03)、A04(x04,y04) (ii) a The central points of the cross sections of the four-finger clamping jaw are A when the four-finger clamping jaw is circumscribed with the bounding box G1(x1,y1)、A2(x2,y2)、A3(x3,y3)、A4(x4,y4) Then, the first offset of the four-finger clamping jaw from the initial position to the position where the four-finger clamping jaw is circumscribed with the bounding box is respectively as follows:
where r denotes the cross-sectional radius of each finger of the four finger jaw itself and w denotes the length of the side of the bounding box perpendicular to its main direction.
Furthermore, when the gripping tool adopts a four-finger gripping jaw with three degrees of freedom, the specific process of calculating the second offset from the time when the gripping tool is circumscribed with the bounding box to the time when the gripping tool is circumscribed with the special-shaped workpiece is as follows:
taking the central point of a mask image under a camera pixel coordinate system as an original point, making a straight line along the main direction of the bounding box, and respectively making a straight line perpendicular to the straight line at the position which is +/-l/2 away from the original point on the straight line; the straight line and four intersection points of the straight line and the mask diagram are approximate grabbing points of the four-finger clamping jaw; wherein l represents the distance between the center points of the cross sections of two fingers in the length direction in the four-finger clamping jaw;
setting the central point of the cross section of each finger as B when each finger in the four-finger clamping jaw is tangent to the mask graph at the approximate grabbing point1(x′1,y′1)、B2(x′2,y′2)、B3(x′3,y′3)、B4(x′4,y′4) Then the four fingers hold the slave and the bagThe second offset from the time when the enclosure is externally tangent to the special-shaped workpiece is respectively as follows:
furthermore, when the gripping tool adopts a four-finger gripping jaw with three degrees of freedom, the total offset of the gripping tool from the initial position to the external tangent of the special-shaped workpiece is obtained according to the first offset and the second offset, and the total offset of the four-finger gripping jaw from the initial position to the external tangent of the special-shaped workpiece is respectively as follows:
m1-w/2-r+p1、m2-w/2-r+p2、m3-w/2-r+p3、m4-w/2-r+p4。
according to a second aspect of the embodiments of the present application, there is provided a grabbing planning system based on any one of the above grabbing planning methods for the special-shaped workpiece, including an image processing module, a total offset calculation module, and a path planning module;
the image processing module is used for acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image;
the total offset calculation module is used for calculating the total offset of the grabbing tool according to the mask image of the special-shaped workpiece and the bounding box information of the mask image;
and the path planning module is used for planning to obtain the grabbing path of the grabbing tool according to the initial position of the grabbing tool, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system and the total offset.
According to a third aspect of the embodiments of the present application, the present application further provides a method for grabbing a special-shaped workpiece, which includes the following steps:
arranging a robot for grabbing the special-shaped workpiece, wherein an end effector of the robot is a three-degree-of-freedom four-finger clamping jaw;
acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image, wherein the specific process comprises the following steps:
acquiring an original image of the special-shaped workpiece by using a camera; processing an original image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image;
according to the mask image of the special-shaped workpiece and the bounding box information of the mask image, the total offset of the three-degree-of-freedom four-finger clamping jaw is obtained by calculation, and the specific process is as follows:
presetting initial positions of the four-finger clamping jaws;
calculating a first offset of the four-finger clamping jaw from the initial position to the position of the four-finger clamping jaw when the four-finger clamping jaw is circumscribed with the bounding box;
calculating a second offset of the grabbing tool from the time of externally tangent with the bounding box to the time of externally tangent with the special-shaped workpiece;
obtaining the total offset of the four-finger clamping jaw from the initial position to the external tangent of the special-shaped workpiece according to the first offset and the second offset;
planning to obtain a grabbing path of the four-finger clamping jaw according to the initial position of the four-finger clamping jaw, the included angle between the main direction of the bounding box and a horizontal axis in a camera pixel coordinate system and the total offset;
converting the coordinates of the center point of the mask image under the camera pixel coordinate system to the robot coordinate system;
and sending the included angle between the main direction of the bounding box and a horizontal axis in a camera pixel coordinate system and the total offset of the four-finger clamping jaws from the initial position to the circumscribed special-shaped workpiece to the robot, and controlling the four-finger clamping jaws to grab the special-shaped workpiece by the robot.
According to a fourth aspect of the embodiments of the present application, the present application further provides a special-shaped workpiece grabbing system based on the special-shaped workpiece grabbing method, which includes a camera and a robot, wherein the camera is used for acquiring an original image of a special-shaped workpiece; the robot is provided with an image processing module, a total offset calculation module, a path planning module, a coordinate transformation module and a control module; the end effector of the robot adopts a three-degree-of-freedom four-finger clamping jaw;
the image processing module is used for acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image;
the total offset calculation module is used for calculating the total offset of the four-finger clamping jaw according to the mask image of the special-shaped workpiece and the bounding box information of the mask image;
the path planning module is used for planning to obtain a grabbing path of the four-finger clamping jaw according to the initial position of the four-finger clamping jaw, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system and the total offset;
the coordinate transformation module is used for transforming the coordinates of the central point of the mask image under the camera pixel coordinate system to the robot coordinate system;
and the control module is used for controlling the four-finger clamping jaws to grab the special-shaped workpiece from the central point of the mask image under the robot coordinate system along the grabbing path of the four-finger clamping jaws.
According to the above embodiments of the present application, at least the following advantages are obtained: according to the method for grabbing and planning the special-shaped workpiece, the mask image of the special-shaped workpiece and the bounding box information of the mask image are obtained by obtaining and processing the image of the special-shaped workpiece; further calculating to obtain the total offset of the grabbing tool; and then planning to obtain a grabbing path of the grabbing tool according to the initial position of the grabbing tool, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system and the total offset.
According to the special-shaped workpiece grabbing method, the planned grabbing path is sent to the robot, and the robot can control the end effector to stably and effectively grab the special-shaped workpiece according to the planned path.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the scope of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of the specification of the application, illustrate embodiments of the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic structural diagram of a special-shaped workpiece according to an embodiment of the present application.
Fig. 2 is a flowchart of a method for planning grabbing of a special-shaped workpiece according to an embodiment of the present application.
Fig. 3 is a schematic structural diagram of a three-degree-of-freedom four-finger clamping jaw in the method for planning grabbing of the special-shaped workpiece according to the embodiment of the present application.
Fig. 4 is a schematic diagram of a relative position relationship between the special-shaped workpiece and the bounding box in the method for planning grabbing the special-shaped workpiece according to the embodiment of the present application.
Fig. 5 is a schematic view of a special-shaped workpiece grabbing system according to an embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
The special-shaped workpiece is different from a workpiece with a regular shape. As shown in fig. 1, it is often difficult to find a pose that can be effectively grasped. The existing image processing method outputs only the position information of the workpiece, but the grasping position is which point on the workpiece and is not accurate. In addition, the common two-degree-of-freedom clamping jaw or the sucking disc for the special-shaped workpiece cannot stably grab the special-shaped workpiece, and the workpiece is easy to slide.
Aiming at the problem that the special-shaped workpiece cannot be effectively grabbed and in order to improve grabbing stability of the special-shaped workpiece, the embodiment of the application provides a special-shaped workpiece grabbing planning method and device, a special-shaped workpiece grabbing method and a special-shaped workpiece grabbing system.
As shown in fig. 2, the method for planning grabbing of the special-shaped workpiece provided by the embodiment of the present application includes the following steps:
s1, acquiring and processing the image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image, wherein the specific process comprises the following steps:
and S11, acquiring an original image of the special-shaped workpiece.
In particular, a camera may be utilized to acquire an original image of the shaped workpiece. The camera can be arranged above the special-shaped workpiece so as to shoot the special-shaped workpiece, and therefore an original image of the special-shaped workpiece is obtained.
S12, processing the original image of the special-shaped workpiece, wherein the specific process is as follows:
and processing the original image of the special-shaped workpiece by using a Blob image processing algorithm or a depth learning image processing algorithm to obtain a mask image S of the special-shaped workpiece and a central point O (x, y) of the mask image S under a camera pixel coordinate system.
And processing the mask image S of the special-shaped workpiece by utilizing a Principal Component Analysis (PCA) algorithm or a circumscribed rectangle algorithm to obtain an enclosing box G of the mask image S and an included angle theta between the main direction of the enclosing box G and a horizontal axis in a camera pixel coordinate system. Wherein, the side of the bounding box G perpendicular to the main direction is a broadside, and the length of the broadside is w; the side of the bounding box G parallel to the main direction is a long side, the length of which is h.
The grabbing posture of the grabbing tool can be determined by acquiring various information such as a mask, a position, a bounding box, an angle and the like of the special-shaped workpiece.
S2, calculating the total offset of the grabbing tool according to the mask image of the special-shaped workpiece and the bounding box information of the mask image, wherein the specific process is as follows:
s21, presetting an initial position of the gripping tool;
the initial positions of the gripping tool are: the center point of the grabbing tool is positioned right above the center point O (x, y) of the mask image S of the special-shaped workpiece under the camera pixel coordinate system. The centre point of the cross-section of each end of the gripper tool is located directly above a line coplanar and perpendicular to the main direction of the bounding box G.
S22, calculating a first offset of the grasping tool from the initial position to the position of the grasping tool when the grasping tool is circumscribed with the bounding box G.
And S23, calculating a second offset of the grabbing tool from the time of the grabbing tool is circumscribed with the bounding box G to the time of the grabbing tool is circumscribed with the special-shaped workpiece.
When the grabbing tool is externally tangent to the special-shaped workpiece, namely the tail end of the grabbing tool can grab the special-shaped workpiece.
And S24, obtaining the total offset of the grabbing tool from the initial position to the outer tangent of the special-shaped workpiece according to the first offset and the second offset.
And S3, planning to obtain a grabbing path of the grabbing tool according to the initial position of the grabbing tool, the included angle theta between the main direction of the bounding box G and the horizontal axis in the camera pixel coordinate system and the total offset.
Specifically, the grabbing path of the grabbing tool under the camera pixel coordinate system is as follows:
the gripper tool is rotated from the initial position by an angle theta to bring the main direction of the gripper tool into line with the main direction of the bounding box. And each tail end of the grabbing tool moves to the special-shaped workpiece by the distance of the total offset so as to grab the special-shaped workpiece.
In this embodiment, as shown in fig. 3, the gripping tool may be a robot with an end effector, and the end effector may be a three-finger gripping jaw with three degrees of freedom, a four-finger gripping jaw with three degrees of freedom, a five-finger gripping jaw with three degrees of freedom, a six-finger gripping jaw with three degrees of freedom, and the like. Specifically, the three degrees of freedom refer to the transverse degree of freedom and the longitudinal degree of freedom of the special-shaped workpiece.
As shown in fig. 4, when the end effector employs a four-finger grip with three degrees of freedom, the specific process of step S21 is as follows:
the initial positions of the preset three-degree-of-freedom four-finger clamping jaw are as follows: the center points of the four-finger clamping jaws are located right above the center point O (x, y) of a mask diagram S of the special-shaped workpiece under a camera pixel coordinate system, the length direction of the four-finger clamping jaws is consistent with the main direction of the bounding box G, and the center points of the cross sections of the four-finger clamping jaws are located right above a straight line which is coplanar with and perpendicular to the main direction of the bounding box G.
As shown in fig. 4, when the end effector employs a four-finger grip with three degrees of freedom, the specific process of step S22 is as follows:
presetting the initial offset between the central point of each finger cross section in the four-finger clamping jaw relative to the central axis of the main direction of the bounding box G as m1、m2、m3、m4。
Assuming that the central point of each finger cross section of the four-finger clamping jaw at the initial position is A01(x01,y01)、A02(x02,y02)、A03(x03,y03)、A04(x04,y04) (ii) a The central points of the cross sections of the four-finger clamping jaw are A when the four-finger clamping jaw is circumscribed with the bounding box G1(x1,y1)、A2(x2,y2)、A3(x3,y3)、A4(x4,y4) When the four-finger clamping jaw is at the initial position, the distance between the central point of each finger cross section and the central point of each finger cross section when the corresponding four-finger clamping jaw is circumscribed with the bounding box G is a first offset from the initial position to the position when the four-finger clamping jaw is circumscribed with the bounding box G, and the first offset is respectively as follows:
where r represents the cross-sectional radius of each finger of the four-finger jaw itself.
As shown in fig. 4, when the end effector employs a four-finger grip with three degrees of freedom, the specific process of step S23 is as follows:
taking the central point O (x, y) of the mask image S under the camera pixel coordinate system as an origin, making a straight line along the main direction of the bounding box G, and respectively making a straight line L perpendicular to the straight line at the position with the distance of +/-L/2 from the origin on the straight line1And L2. Straight line L1And L2The four intersection points with the mask pattern S are the approximate grasping points of the four-finger gripping jaw.
Setting the central point of the cross section of each finger as B when each finger in the four-finger clamping jaw is tangent to the mask picture S at the approximate grabbing point1(x′1,y′1)、B2(x′2,y′2)、B3(x′3,y′3)、B4(x′4,y′4) And the distance moved by the four-finger clamping jaw from the state of externally tangent with the bounding box G to the approximate grabbing point is the second offset of the four-finger clamping jaw from the state of externally tangent with the bounding box G to the state of externally tangent with the special-shaped workpiece, and the second offset is respectively as follows:
as shown in fig. 4, when the end effector employs a four-finger grip with three degrees of freedom, the specific process of step S24 is as follows:
the total offset of the four-finger clamping jaw from the initial position to the external tangent of the special-shaped workpiece is respectively as follows:
m1-w/2-r+p1、m2-w/2-r+p2、m3-w/2-r+p3、m4-w/2-r+p4。
as shown in fig. 4, when the end effector employs a four-finger grip with three degrees of freedom, the specific process of step S3 is as follows:
the four-finger clamping jaw rotates by an angle theta from the initial position, and each finger of the four-finger clamping jaw moves towards the special-shaped workpiece by the corresponding total offset respectively so as to grab the special-shaped workpiece.
Based on the method for grabbing and planning the special-shaped workpiece, the application also provides a device for grabbing and planning the special-shaped workpiece, which comprises an image processing module, a total offset calculation module and a path planning module.
The image processing module is used for acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box G information of the mask image.
The total offset calculation module is used for calculating the total offset of the grabbing tool according to the mask image of the special-shaped workpiece and the bounding box information of the mask image;
and the path planning module is used for planning to obtain the grabbing path of the grabbing tool according to the initial position of the grabbing tool, the included angle between the main direction of the bounding box G and the horizontal axis in the camera pixel coordinate system and the total offset.
It should be noted that: the special-shaped workpiece grabbing and planning device provided in the above embodiment is only exemplified by the division of the program modules, and in practical applications, the processing allocation may be completed by different program modules according to needs, that is, the internal structure of the special-shaped workpiece grabbing and planning device is divided into different program modules to complete all or part of the above-described processing. In addition, the special-shaped workpiece grabbing planning device and the special-shaped workpiece grabbing planning method provided by the embodiment belong to the same concept, and specific implementation processes are described in the method embodiment and are not described herein again.
As shown in fig. 4, based on the method for planning grabbing of the special-shaped workpiece provided in the embodiment of the present application, the present application also provides a method for grabbing the special-shaped workpiece, which includes the following steps:
and S10, arranging a robot for grabbing the special-shaped workpiece, wherein the end effector of the robot is a three-degree-of-freedom four-finger clamping jaw. The distance between the central points of the cross sections of the two fingers in the length direction in the four-finger clamping jaw is l, and the radius of the cross section of each finger is r.
S20, acquiring and processing the image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image, wherein the specific process comprises the following steps:
s201, acquiring an original image of the special-shaped workpiece by using a camera.
Specifically, as shown in fig. 3, a camera may be installed above the special-shaped workpiece so as to photograph the special-shaped workpiece, thereby obtaining an original image of the special-shaped workpiece.
S202, processing the original image of the special-shaped workpiece, wherein the specific process is as follows:
and processing the original image of the special-shaped workpiece by using a Blob image processing algorithm or a depth learning image processing algorithm to obtain a mask image S of the special-shaped workpiece and a central point O (x, y) of the mask image S under a camera pixel coordinate system.
And processing the mask image S of the special-shaped workpiece by utilizing a Principal Component Analysis (PCA) algorithm or a circumscribed rectangle algorithm to obtain an enclosing box G of the mask image S and an included angle theta between the main direction of the enclosing box G and a horizontal axis in a camera pixel coordinate system. The side of the bounding box G perpendicular to the main direction is a wide side w, and the side of the bounding box G parallel to the main direction is a long side h.
S30, calculating the total offset of the three-degree-of-freedom four-finger clamping jaw according to the mask image of the special-shaped workpiece and the information of the bounding box G of the mask image, wherein the specific process is as follows:
s301, presetting initial positions of the four-finger clamping jaws;
the three-degree-of-freedom four-finger clamping jaw has the initial positions as follows: the center point of the four-finger clamping jaw is positioned right above the center point of a mask image S of the special-shaped workpiece under a camera pixel coordinate system, the length direction of the four-finger clamping jaw is consistent with the main direction of the bounding box G, and the center point of the cross section of each finger in the four-finger clamping jaw is positioned right above a straight line which is coplanar with and perpendicular to the main direction of the bounding box G.
S302, calculating a first offset of the four-finger clamping jaw from the initial position to the position circumscribed with the bounding box G, wherein the specific process is as follows:
presetting the initial offset between the central point of each finger cross section in the four-finger clamping jaw relative to the central axis of the main direction of the bounding box G as m1、m2、m3、m4。
Assuming that the central point of each finger cross section of the four-finger clamping jaw at the initial position is A01(x01,y01)、A02(x02,y02)、A03(x03,y03)、A04(x04,y04) (ii) a The central points of the cross sections of the four-finger clamping jaw are A when the four-finger clamping jaw is circumscribed with the bounding box G1(x1,y1)、A2(x2,y2)、A3(x3,y3)、A4(x4,y4) When the four-finger clamping jaws are in the initial position, the distance between the central point of each finger cross section and the central point of each finger cross section when the corresponding four-finger clamping jaws are circumscribed with the bounding box G is the first deviation of the four-finger clamping jaws from the initial position to the position when the four-finger clamping jaws are circumscribed with the bounding box GThe shift amount, the first offset amount is respectively:
where r represents the cross-sectional radius of each finger of the four-finger jaw itself.
S303, calculating a second offset from the time when the grasping tool is circumscribed with the bounding box G to the time when the grasping tool is circumscribed with the special-shaped workpiece, wherein the specific process is as follows:
taking the central point O (x, y) of the mask image S under the camera pixel coordinate system as an origin, making a straight line along the main direction of the bounding box G, and respectively making a straight line L perpendicular to the straight line at the position with the distance of +/-L/2 from the origin on the straight line1And L2. Straight line L1And L2The four intersection points with the mask pattern S are the approximate grasping points of the four-finger gripping jaw.
Setting the central point of the cross section of each finger as B when each finger in the four-finger clamping jaw is tangent to the mask picture S at the approximate grabbing point1(x′1,y′1)、B2(x′2,y′2)、B3(x′3,y′3)、B4(x′4,y′4) And the distance moved by the four-finger clamping jaw from the state of externally tangent with the bounding box G to the approximate grabbing point is the second offset of the four-finger clamping jaw from the state of externally tangent with the bounding box G to the state of externally tangent with the special-shaped workpiece, and the second offset is respectively as follows:
when the four-finger clamping jaw is externally tangent to the special-shaped workpiece, namely, the tail end of the four-finger clamping jaw can grab the special-shaped workpiece.
S304, obtaining the total offset of the four-finger clamping jaw from the initial position to the external tangent of the special-shaped workpiece according to the first offset and the second offset, wherein the total offset is as follows:
m1-w/2-r+p1、m2-w/2-r+p2、m3-w/2-r+p3、m4-w/2-r+p4。
and S40, planning to obtain the grabbing path of the four-finger clamping jaw according to the initial position of the four-finger clamping jaw, the included angle between the main direction of the bounding box G and the horizontal axis in the camera pixel coordinate system and the total offset.
Specifically, the grabbing path of the four-finger clamping jaw under the camera pixel coordinate system is as follows:
the four-finger clamping jaw rotates by an angle theta from the initial position, and the four-finger clamping jaw moves towards the special-shaped workpiece by the corresponding total offset to grab the special-shaped workpiece.
S50, converting the coordinates of the center point of the mask image S under the camera pixel coordinate system into the robot coordinate system, and the specific process is as follows:
s501, converting a camera pixel coordinate system into a robot coordinate system;
the coordinate system conversion formula is as follows:
wherein the content of the first and second substances,
representing a coordinate matrix in the robot coordinate system,
representing a coordinate matrix in the camera pixel coordinate system, R representing a matrix with respect to the parameters a, a ', b, b ', and M representing a matrix with respect to the parameters c and c '.
Selecting 4 known coordinates (X) in a camera pixel coordinate system1,Y1)、(X2,Y2)、(X3,Y3)、(X4,Y4) And substituting the coordinate system conversion formula with the coordinate system conversion formula to obtain:
and calculating to obtain parameters a, a ', b, b ', c and c ', and further obtaining matrixes R and M.
And S502, converting the coordinates of the central point O (x, y) of the mask image S under the camera pixel coordinate system into the robot coordinate system according to the matrixes R and M.
And S60, sending an included angle theta between the main direction of the bounding box G and a horizontal axis in a camera pixel coordinate system and a total offset when the four-finger clamping jaws are externally tangent from the initial position to the special-shaped workpiece to the robot, and controlling the four-finger clamping jaws to grab the special-shaped workpiece by the robot.
As shown in fig. 5, based on the method for grabbing the special-shaped workpiece provided by the embodiment of the present application, the present application further provides a special-shaped workpiece grabbing system, which includes a camera 1 and a robot 2, where the camera 1 is used to acquire an original image of the special-shaped workpiece; the robot is provided with an image processing module, a total offset calculation module, a path planning module, a coordinate transformation module and a control module. The end effector of the robot uses three-degree-of-freedom four-finger gripping jaws 21.
The image processing module is used for acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box G information of the mask image.
And the total offset calculation module is used for calculating the total offset of the four-finger clamping jaw according to the mask diagram of the special-shaped workpiece and the bounding box information of the mask diagram.
And the path planning module is used for planning to obtain a grabbing path of the four-finger clamping jaw according to the initial position of the four-finger clamping jaw, the included angle between the main direction of the bounding box G and the horizontal axis in the camera pixel coordinate system and the total offset.
And the coordinate transformation module is used for transforming the coordinates of the central point of the mask image under the camera pixel coordinate system to the robot coordinate system.
And the control module is used for controlling the four-finger clamping jaws to grab the special-shaped workpiece along the grabbing path of the four-finger clamping jaws from the central point of the mask image under the robot coordinate system.
It should be noted that: the special-shaped workpiece grabbing system provided by the above embodiment is only exemplified by the division of the above program modules, and in practical applications, the processing allocation may be completed by different program modules according to needs, that is, the internal structure of the robot in the special-shaped workpiece grabbing system is divided into different program modules to complete all or part of the above-described processing. In addition, the special-shaped workpiece grabbing system and the special-shaped workpiece grabbing method provided by the embodiment belong to the same concept, and specific implementation processes are detailed in the method embodiment and are not described again.
The problem that special-shaped workpiece is difficult to snatch can be effectively solved in this application, the stability that special-shaped workpiece snatched can be showing and improve.
The embodiments of the present application described above may be implemented in various hardware, software code, or a combination of both. For example, the embodiments of the present application may also be program code for executing the above-described method in a data signal processor. The present application may also relate to various functions performed by a computer processor, digital signal processor, microprocessor, or field programmable gate array. The processor described above may be configured in accordance with the present application to perform certain tasks by executing machine-readable software code or firmware code that defines certain methods disclosed herein. Software code or firmware code may be developed in different programming languages and in different formats or forms. Software code may also be compiled for different target platforms. However, different code styles, types, and languages of software code and other types of configuration code for performing tasks according to the present application do not depart from the spirit and scope of the present application.
The foregoing is merely an illustrative embodiment of the present application, and any equivalent changes and modifications made by those skilled in the art without departing from the spirit and principles of the present application shall fall within the protection scope of the present application.
Claims (10)
1. A method for grabbing and planning a special-shaped workpiece is characterized by comprising the following steps:
acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image;
calculating to obtain the total offset of the grabbing tool according to the mask image of the special-shaped workpiece and the bounding box information of the mask image;
and planning to obtain a grabbing path of the grabbing tool according to the initial position of the grabbing tool, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system and the total offset.
2. The method for grabbing and planning a special-shaped workpiece according to claim 1, wherein the specific process of obtaining and processing the image of the special-shaped workpiece to obtain the mask image of the special-shaped workpiece and the bounding box information of the mask image comprises:
acquiring an original image of the special-shaped workpiece by using a camera;
processing an original image of the special-shaped workpiece by using an image processing algorithm to obtain a mask image of the special-shaped workpiece and a central point of the mask image under a camera pixel coordinate system;
and processing the mask image of the special-shaped workpiece to obtain an enclosure box of the mask image and an included angle between the main direction of the enclosure box and a horizontal axis in a camera pixel coordinate system.
3. The method for planning grabbing of the special-shaped workpiece according to claim 2, wherein the specific process of calculating the total offset of the grabbing tool according to the mask diagram of the special-shaped workpiece and the bounding box information of the mask diagram is as follows:
presetting an initial position of a gripping tool; the initial positions of the gripping tool are: the center point of the grabbing tool is positioned right above the center point of the mask image of the special-shaped workpiece under the camera pixel coordinate system; the center point of the cross section of each tail end in the grabbing tool is positioned right above a straight line which is coplanar and vertical to the main direction of the bounding box;
calculating a first offset of the grasping tool from the initial position to the position circumscribed with the bounding box;
calculating a second offset of the grabbing tool from the time of externally tangent with the bounding box to the time of externally tangent with the special-shaped workpiece;
and obtaining the total offset of the grabbing tool from the initial position to the external tangent of the special-shaped workpiece according to the first offset and the second offset.
4. The method for planning grabbing of a special-shaped workpiece according to claim 3, wherein the grabbing path of the grabbing tool in the camera pixel coordinate system is planned according to the initial position of the grabbing tool, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system and the total offset:
the gripping tool is rotated until the main direction of the gripping tool is consistent with the main direction of the bounding box; and each tail end of the gripping tool moves towards the special-shaped workpiece by the total offset distance.
5. The method for planning grabbing of the special-shaped workpiece according to claim 3, wherein when the grabbing tool adopts a four-finger gripping jaw with three degrees of freedom, the specific process of calculating the first offset of the grabbing tool from the initial position to the state of being circumscribed with the bounding box is as follows:
presetting the initial offset between the central point of each finger cross section in the four-finger clamping jaw relative to the central axis of the main direction of the bounding box as m1、m2、m3、m4；
Assuming that the central point of each finger cross section of the four-finger clamping jaw at the initial position is A01(x01,y01)、A02(x02,y02)、A03(x03,y03)、A04(x04,y04) (ii) a The central points of the cross sections of the four-finger clamping jaw are A when the four-finger clamping jaw is circumscribed with the bounding box G1(x1,y1)、A2(x2,y2)、A3(x3,y3)、A4(x4,y4) Then, the first offset of the four-finger clamping jaw from the initial position to the position where the four-finger clamping jaw is circumscribed with the bounding box is respectively as follows:
where r denotes the cross-sectional radius of each finger of the four finger jaw itself and w denotes the length of the side of the bounding box perpendicular to its main direction.
6. The method for planning grabbing of the special-shaped workpiece according to claim 5, wherein when the grabbing tool adopts a four-finger gripping jaw with three degrees of freedom, the specific process of calculating the second offset from the time when the grabbing tool is circumscribed with the bounding box to the time when the grabbing tool is circumscribed with the special-shaped workpiece is as follows:
taking the central point of a mask image under a camera pixel coordinate system as an original point, making a straight line along the main direction of the bounding box, and respectively making a straight line perpendicular to the straight line at the position which is +/-l/2 away from the original point on the straight line; the straight line and four intersection points of the straight line and the mask diagram are approximate grabbing points of the four-finger clamping jaw; wherein l represents the distance between the center points of the cross sections of two fingers in the length direction in the four-finger clamping jaw;
setting the central point of the cross section of each finger as B when each finger in the four-finger clamping jaw is tangent to the mask graph at the approximate grabbing point1(x′1,y′1)、B2(x′2,y′2)、B3(x′3,y′3)、B4(x′4,y′4) And then the second offset of the four-finger clamping jaw from the time when the four-finger clamping jaw is circumscribed with the bounding box to the time when the four-finger clamping jaw is circumscribed with the special-shaped workpiece is respectively as follows:
7. the method for planning grabbing of the special-shaped workpiece according to claim 6, wherein when the grabbing tool adopts a four-finger gripper with three degrees of freedom, the total offset of the grabbing tool from the initial position to the outer tangent of the special-shaped workpiece is obtained according to the first offset and the second offset, and the total offset of the four-finger gripper from the initial position to the outer tangent of the special-shaped workpiece is respectively as follows:
m1-w/2-r+p1、m2-w/2-r+p2、m3-w/2-r+p3、m4-w/2-r+p4。
8. a grabbing planning system based on the method for grabbing and planning the special-shaped workpiece according to any one of claims 1-7, which is characterized by comprising an image processing module, a total offset calculation module and a path planning module;
the image processing module is used for acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image;
the total offset calculation module is used for calculating the total offset of the grabbing tool according to the mask image of the special-shaped workpiece and the bounding box information of the mask image;
and the path planning module is used for planning to obtain the grabbing path of the grabbing tool according to the initial position of the grabbing tool, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system and the total offset.
9. A method for grabbing a special-shaped workpiece is characterized by comprising the following steps:
arranging a robot for grabbing the special-shaped workpiece, wherein an end effector of the robot is a three-degree-of-freedom four-finger clamping jaw; the distance between the central points of the cross sections of the two fingers in the length direction in the four-finger clamping jaw is l, and the radius of the cross section of each finger is r;
acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image, wherein the specific process comprises the following steps:
acquiring an original image of the special-shaped workpiece by using a camera; processing an original image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image;
according to the mask image of the special-shaped workpiece and the bounding box information of the mask image, the total offset of the three-degree-of-freedom four-finger clamping jaw is obtained by calculation, and the specific process is as follows:
presetting initial positions of the four-finger clamping jaws;
calculating a first offset of the four-finger clamping jaw from the initial position to the position of the four-finger clamping jaw when the four-finger clamping jaw is circumscribed with the bounding box;
calculating a second offset of the grabbing tool from the time of externally tangent with the bounding box to the time of externally tangent with the special-shaped workpiece;
obtaining the total offset of the four-finger clamping jaw from the initial position to the external tangent of the special-shaped workpiece according to the first offset and the second offset;
planning to obtain a grabbing path of the four-finger clamping jaw according to the initial position of the four-finger clamping jaw, the included angle between the main direction of the bounding box and a horizontal axis in a camera pixel coordinate system and the total offset;
converting the coordinates of the center point of the mask image under the camera pixel coordinate system to the robot coordinate system;
and sending the included angle between the main direction of the bounding box and a horizontal axis in a camera pixel coordinate system and the total offset of the four-finger clamping jaws from the initial position to the circumscribed special-shaped workpiece to the robot, and controlling the four-finger clamping jaws to grab the special-shaped workpiece by the robot.
10. A special-shaped workpiece gripping system based on the special-shaped workpiece gripping method according to claim 9, comprising a camera and a robot, wherein the camera is used for acquiring an original image of the special-shaped workpiece; the robot is provided with an image processing module, a total offset calculation module, a path planning module, a coordinate transformation module and a control module; the end effector of the robot adopts a three-degree-of-freedom four-finger clamping jaw;
the image processing module is used for acquiring and processing an image of the special-shaped workpiece to obtain a mask image of the special-shaped workpiece and bounding box information of the mask image;
the total offset calculation module is used for calculating the total offset of the four-finger clamping jaw according to the mask image of the special-shaped workpiece and the bounding box information of the mask image;
the path planning module is used for planning to obtain a grabbing path of the four-finger clamping jaw according to the initial position of the four-finger clamping jaw, the included angle between the main direction of the bounding box and the horizontal axis in the camera pixel coordinate system and the total offset;
the coordinate transformation module is used for transforming the coordinates of the central point of the mask image under the camera pixel coordinate system to the robot coordinate system;
and the control module is used for controlling the four-finger clamping jaws to grab the special-shaped workpiece from the central point of the mask image under the robot coordinate system along the grabbing path of the four-finger clamping jaws.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210004305.4A CN114407011B (en) | 2022-01-05 | 2022-01-05 | Special-shaped workpiece grabbing planning method, planning device and special-shaped workpiece grabbing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210004305.4A CN114407011B (en) | 2022-01-05 | 2022-01-05 | Special-shaped workpiece grabbing planning method, planning device and special-shaped workpiece grabbing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114407011A true CN114407011A (en) | 2022-04-29 |
| CN114407011B CN114407011B (en) | 2023-10-13 |
Family
ID=81272175
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210004305.4A Active CN114407011B (en) | 2022-01-05 | 2022-01-05 | Special-shaped workpiece grabbing planning method, planning device and special-shaped workpiece grabbing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114407011B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115781665A (en) * | 2022-11-01 | 2023-03-14 | 深圳史河机器人科技有限公司 | Monocular camera-based mechanical arm control method and device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010012567A (en) * | 2008-07-04 | 2010-01-21 | Fanuc Ltd | Article picking device |
| US20190111571A1 (en) * | 2017-10-13 | 2019-04-18 | Soft Robotics, Inc. | End of arm tools for soft robotic systems |
| CN110238840A (en) * | 2019-04-24 | 2019-09-17 | 中山大学 | A kind of autonomous grasping means of the mechanical arm of view-based access control model |
| CN111968235A (en) * | 2020-07-08 | 2020-11-20 | 杭州易现先进科技有限公司 | Object attitude estimation method, device and system and computer equipment |
-
2022
- 2022-01-05 CN CN202210004305.4A patent/CN114407011B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010012567A (en) * | 2008-07-04 | 2010-01-21 | Fanuc Ltd | Article picking device |
| US20190111571A1 (en) * | 2017-10-13 | 2019-04-18 | Soft Robotics, Inc. | End of arm tools for soft robotic systems |
| CN110238840A (en) * | 2019-04-24 | 2019-09-17 | 中山大学 | A kind of autonomous grasping means of the mechanical arm of view-based access control model |
| CN111968235A (en) * | 2020-07-08 | 2020-11-20 | 杭州易现先进科技有限公司 | Object attitude estimation method, device and system and computer equipment |
Non-Patent Citations (1)
| Title |
|---|
| 葛俊彦等: "基于三维检测网络的机器人抓取方法", 仪器仪表学报 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115781665A (en) * | 2022-11-01 | 2023-03-14 | 深圳史河机器人科技有限公司 | Monocular camera-based mechanical arm control method and device and storage medium |
| CN115781665B (en) * | 2022-11-01 | 2023-08-08 | 深圳史河机器人科技有限公司 | Mechanical arm control method and device based on monocular camera and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114407011B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108818535B (en) | Robot 3D vision hand-eye calibration method | |
| CN110573308B (en) | Computer-based method and system for spatial programming of robotic devices | |
| CN113379849B (en) | Robot autonomous recognition intelligent grabbing method and system based on depth camera | |
| US10456917B2 (en) | Robot system including a plurality of robots, robot controller and robot control method | |
| CN110238849B (en) | Robot hand-eye calibration method and device | |
| JP5200772B2 (en) | Robot apparatus control method and robot apparatus | |
| WO2019116891A1 (en) | Robot system and robot control method | |
| US20150273689A1 (en) | Robot control device, robot, robotic system, teaching method, and program | |
| US20140031982A1 (en) | Robotic system and robot control device | |
| JP2019014030A (en) | Control device for robot, robot, robot system, and calibration method for camera | |
| JP2015213973A (en) | Picking device and picking method | |
| JP2019155556A (en) | Control device of robot, robot, robot system, and calibration method for camera | |
| US20220331964A1 (en) | Device and method for controlling a robot to insert an object into an insertion | |
| CN111360821A (en) | Picking control method, device and equipment and computer scale storage medium | |
| US20220335622A1 (en) | Device and method for training a neural network for controlling a robot for an inserting task | |
| JP5223407B2 (en) | Redundant robot teaching method | |
| CN114407011B (en) | Special-shaped workpiece grabbing planning method, planning device and special-shaped workpiece grabbing method | |
| JP7112528B2 (en) | Work coordinate creation device | |
| JPH1148179A (en) | Work line tracing control method of robot | |
| US20220335710A1 (en) | Device and method for training a neural network for controlling a robot for an inserting task | |
| CN115519536A (en) | System and method for error correction and compensation for 3D eye-hand coordination | |
| Gao et al. | Vision-based grasping and manipulation of flexible USB wires | |
| Vollmann et al. | Manipulator control by calibration-free stereo vision | |
| JP2006315128A (en) | Shifting from one hand to the other hand control method for robot hand | |
| Krishnan et al. | Adaptive gain control law for visual servoing of 6DOF robot manipulator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |