CN113538576A - Grabbing method and device based on double-arm robot and double-arm robot - Google Patents

Grabbing method and device based on double-arm robot and double-arm robot Download PDF

Info

Publication number
CN113538576A
CN113538576A CN202110594376.XA CN202110594376A CN113538576A CN 113538576 A CN113538576 A CN 113538576A CN 202110594376 A CN202110594376 A CN 202110594376A CN 113538576 A CN113538576 A CN 113538576A
Authority
CN
China
Prior art keywords
pose
sample
grabbing
target
vacuum adsorption
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110594376.XA
Other languages
Chinese (zh)
Inventor
王鹏
许广云
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Institute of Automation of Chinese Academy of Science
Original Assignee
Institute of Automation of Chinese Academy of Science
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Automation of Chinese Academy of Science filed Critical Institute of Automation of Chinese Academy of Science
Priority to CN202110594376.XA priority Critical patent/CN113538576A/en
Publication of CN113538576A publication Critical patent/CN113538576A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J15/00—Gripping heads and other end effectors
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J15/00—Gripping heads and other end effectors
    • B25J15/06—Gripping heads and other end effectors with vacuum or magnetic holding means
    • B25J15/0616—Gripping heads and other end effectors with vacuum or magnetic holding means with vacuum
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1679—Programme controls characterised by the tasks executed
    • B25J9/1682—Dual arm manipulator; Coordination of several manipulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/0014—Image feed-back for automatic industrial control, e.g. robot with camera
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Manipulator (AREA)

Abstract

The invention provides a grabbing method and device based on a double-arm robot and the double-arm robot, wherein the method comprises the following steps: inputting the target scene depth map into the example segmentation model to obtain a grabbed target mask and a vacuum adsorption target mask output by the example segmentation model; determining a pose of a grab target based on the grab target mask, and determining a pose of a vacuum adsorption target based on the vacuum adsorption target mask; and sending the grabbing target pose and the vacuum adsorption target pose to an actuating mechanism of the double-arm robot so that the actuating mechanism grabs the object in the target scene depth map. The gripping mechanical arm clamping jaw structure does not need to be collected for training when the example segmentation model is trained, so that the example segmentation model can be suitable for gripping mechanical arm clamping jaws of different models, and the generalization capability of the model is strong. According to the invention, after the grabbing target pose and the vacuum adsorption target pose are accurately obtained, the actuating mechanism of the double-arm robot can accurately grab the object in the target scene depth map, and the robustness is strong.

Description

Grabbing method and device based on double-arm robot and double-arm robot
Technical Field
The invention relates to the technical field of robot grabbing, in particular to a grabbing method and device based on a double-arm robot and the double-arm robot.
Background
Mechanical arm grabbing is basic and important basic operation of a robot, and is widely applied to various fields such as sorting, assembling and service robots of industrial parts. Although smart grabbing by robots has great application potential in the fields of e-commerce logistics, manufacturing, home service and the like, objects in these scenes are generally disordered or stacked, and the geometric structures of the objects are complex.
At present, in the conventional method, a machine learning model (such as CNN, LSTM, etc.) is mostly used to segment a multi-view RGB-D image, and the multi-view RGB-D image is registered with a three-dimensional model of a target object, so as to obtain a 6D pose of the target object, and then the target object is grabbed. Although the method solves the problem of smart grabbing of the robot to a certain extent, training data of the method is difficult to obtain. On one hand, a large number of scenes need to be manually set, and the method is very complicated; on the other hand, different two-finger clamping jaws have different three-dimensional structures, and the two-finger clamping jaws need to be trained again when replaced.
Disclosure of Invention
The invention provides a grabbing method and device based on a double-arm robot and the double-arm robot, which are used for overcoming the defects of low efficiency and robustness in grabbing objects in the prior art.
The invention provides a grabbing method based on a double-arm robot, which comprises the following steps:
inputting the target scene depth map into an example segmentation model to obtain a grab target mask and a vacuum adsorption target mask output by the example segmentation model; the target scene at least comprises one object to be grabbed;
determining a grab target pose based on the grab target mask, and determining a vacuum adsorption target pose based on the vacuum adsorption target mask;
sending the grabbing target pose and the vacuum adsorption target pose to an actuating mechanism of the double-arm robot so that the actuating mechanism grabs the object in the target scene depth map; the executing mechanism comprises a grabbing mechanical arm and a vacuum adsorption mechanical arm;
wherein the example segmentation model is trained based on a sample scene depth map, a grab sample mask of the sample scene depth map, a vacuum suction sample mask of the sample scene depth map, a score of the grab sample mask, and a score of the vacuum suction sample mask; the sample scene comprises at least one sample object to be grabbed.
According to the grabbing method based on the double-arm robot provided by the invention, the method for determining the pose of the grabbing target based on the grabbing target mask and determining the pose of the vacuum adsorption target based on the vacuum adsorption target mask comprises the following steps:
acquiring a plurality of candidate grabbing poses from a first object corresponding to the grabbing target mask, determining grabbing pose scores corresponding to the candidate grabbing poses based on the grabbing orientation information of the first object, and taking the candidate grabbing pose corresponding to the maximum grabbing pose score as the grabbing target pose;
and acquiring a plurality of candidate vacuum adsorption poses from a second object corresponding to the vacuum adsorption target mask, determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose based on the adsorption azimuth information of the second object, and taking the candidate vacuum adsorption pose corresponding to the maximum vacuum adsorption pose score as the vacuum adsorption target pose.
According to the grabbing method based on the double-arm robot provided by the invention, the grabbing azimuth information of the first object comprises a linear distance from the center of the first object to the grabbing direction and an included angle between the grabbing direction of the first object and the gravity direction of the first object;
the acquiring a plurality of candidate grab poses from the first object corresponding to the grab target mask code, and determining a grab pose score corresponding to each candidate grab pose based on the grab orientation information of the first object includes:
acquiring a plurality of initial grabbing poses from the first object corresponding to the grabbing target mask based on a grid method, and determining a clamping jaw point cloud corresponding to each initial grabbing pose based on each initial grabbing pose;
if the clamping jaw point cloud does not collide with the point cloud in the target scene depth map and at least one point cloud exists in a closed area of the clamping jaw point cloud, taking an initial grabbing pose corresponding to the clamping jaw point cloud as the candidate grabbing pose;
inputting the linear distance from the center of the first object to the grabbing direction and the included angle between the grabbing direction of the first object and the gravity direction of the first object into a grabbing pose scoring model, and determining grabbing pose scores corresponding to the candidate grabbing poses; the grabbing pose scoring model is as follows:
Sg＝1-(dg+θg)；
wherein S isgRepresenting the corresponding grab pose score of each candidate grab pose, dgRepresenting a linear distance, θ, from the center of the first object to the gripping directiongAnd the included angle between the grabbing direction of the first object and the gravity direction of the first object is represented.
According to the grabbing method based on the double-arm robot provided by the invention, the adsorption azimuth information of the second object comprises an included angle between the adsorption direction of the second object and the weight direction of the second object, a distance between the adsorption point of the second object and the center of the second object and a minimum distance between the adsorption point of the second object and a non-adsorption point of the second object; the non-adsorbable point of the second object refers to a point on the second object, the curvature of which is greater than a preset value;
the acquiring a plurality of candidate vacuum adsorption poses from a second object corresponding to the vacuum adsorption target mask, and determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose based on adsorption azimuth information of the second object includes:
determining the curvatures of all points on a second object corresponding to the vacuum adsorption target mask, and taking the corresponding positions with the curvatures less than or equal to a preset value as candidate vacuum adsorption positions;
inputting an included angle between the adsorption direction of the second object and the weight direction of the second object, a distance between the adsorption point of the second object and the center of the second object and a minimum distance between the adsorption point of the second object and the non-adsorption point of the second object into a vacuum adsorption pose scoring model, and determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose; the vacuum adsorption pose scoring model is as follows:
Figure BDA0003090620500000041
wherein S issVacuum adsorption pose scores representing the correspondence of each candidate vacuum adsorption pose, SaRepresents an angle between the second object adsorption direction and the second object gravity direction, SdRepresents the distance between the adsorption point of the second object and the center of the second object, SbRepresents a minimum distance between the adsorption point of the second object and the non-adsorption point of the second object.
According to the grabbing method based on the double-arm robot, the grabbing sample mask and the score of the grabbing sample mask are determined based on the following steps:
acquiring a plurality of candidate sample grabbing poses from each sample object, determining a grabbed sample pose score corresponding to each candidate sample grabbing pose based on the grabbing position information of each sample object, and taking the candidate sample grabbing pose corresponding to the maximum grabbed sample pose score as a grabbed sample target pose of each sample object;
and determining the grab score of each sample object based on the grab sample target pose of each sample object, taking the maximum grab score as the score of the grab sample mask code, and determining the grab sample mask code based on the grab sample target pose corresponding to the maximum grab score.
According to the grabbing method based on the double-arm robot, the vacuum adsorption sample mask and the score of the vacuum adsorption sample mask are determined based on the following steps:
acquiring a plurality of candidate sample vacuum adsorption poses from each sample object, determining a vacuum adsorption sample pose score corresponding to each candidate sample vacuum adsorption pose based on adsorption azimuth information of each sample object, and taking the candidate sample vacuum adsorption pose corresponding to the maximum vacuum adsorption sample pose score as a vacuum adsorption sample target pose of each sample object;
and determining a vacuum adsorption score of each sample object based on the vacuum adsorption sample target pose of each sample object, taking the maximum vacuum adsorption score as the score of the vacuum adsorption sample mask, and determining the vacuum adsorption sample mask based on the vacuum adsorption sample target pose corresponding to the maximum vacuum adsorption score.
The invention also provides a grabbing device based on the double-arm robot, which comprises:
the target mask determining unit is used for inputting the target scene depth map into the example segmentation model to obtain a grab target mask and a vacuum adsorption target mask output by the example segmentation model; the target scene at least comprises one object to be grabbed;
the target pose determining unit is used for determining a grab target pose based on the grab target mask and determining a vacuum adsorption target pose based on the vacuum adsorption target mask;
the object grabbing unit is used for sending the grabbing target pose and the vacuum adsorption target pose to an actuating mechanism of the double-arm robot so that the actuating mechanism grabs the object in the target scene depth map; the executing mechanism comprises a grabbing mechanical arm and a vacuum adsorption mechanical arm;
wherein the example segmentation model is trained based on a sample scene depth map, a grab sample mask of the sample scene depth map, a vacuum suction sample mask of the sample scene depth map, a score of the grab sample mask, and a score of the vacuum suction sample mask; the sample scene comprises at least one sample object to be grabbed.
The present invention also provides a dual-arm robot, comprising: the gripping device based on the two-arm robot as described above.
The invention also provides an electronic device, comprising a memory, a processor and a computer program stored on the memory and operable on the processor, wherein the processor executes the computer program to implement the steps of the two-arm robot-based grabbing method according to any one of the above.
The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the dual-arm robot-based gripping method as described in any of the above.
According to the grabbing method and device based on the double-arm robot and the double-arm robot, the grabbing target mask and the vacuum adsorption target mask can be rapidly and accurately obtained based on the example segmentation model, and meanwhile, the structure of the grabbing mechanical arm clamping jaw does not need to be acquired for training when the example segmentation model is trained, so that the example segmentation model can be suitable for grabbing mechanical arm clamping jaws of different models, and the generalization capability of the model is strong. After the grabbed target mask and the vacuum adsorption target mask are determined, the grabbed target pose and the vacuum adsorption target pose can be accurately obtained, so that the execution mechanism of the double-arm robot can accurately grab objects in the target scene depth map, and the robustness is high.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a grabbing method based on a double-arm robot provided by the invention;
FIG. 2 is a schematic flow chart of determining a grab target and a vacuum absorption target based on an example segmentation model provided by the present invention;
FIG. 3 is a schematic structural diagram of a grasping apparatus based on a two-arm robot according to the present invention;
fig. 4 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The robot smart grab has great application potential in the fields of e-commerce logistics, production and manufacturing, home service and the like, but objects in the scenes are generally disordered or stacked, and the geometric structures of the objects are complex. Therefore, how to improve the stability and efficiency of smart grabbing of complex objects by the robot in a stacking scene is a very worthy of research.
At present, the robot dexterously grasps and has developed small-scale application in the manufacturing field and the logistics field, and although corresponding dexterously grasps the algorithm to different scenes and objects at present, still there are many problems to be solved urgently in practical application, such as low grasping stability, poor grasping efficiency, small range of the graspable object, difficult sample collection and the like.
For example, at present, a machine learning model such as CNN is often used to segment a multi-view RGB-D image, and the multi-view RGB-D image is registered with a three-dimensional model of a target object, so as to obtain a 6D pose of the target object, and then the target object is grabbed. Although the method solves the problem of smart grabbing of the robot to a certain extent, data for training the model is difficult to obtain, namely, on one hand, a large number of scenes need to be set manually, and the method is very complicated; on the other hand, different two-finger clamping jaws have different three-dimensional structures, and training needs to be carried out again when the two-finger clamping jaws are replaced.
In view of the above, the present invention provides a grasping method based on a double-arm robot. Fig. 1 is a schematic flow chart of a grabbing method based on a two-arm robot provided by the invention, as shown in fig. 1, the grabbing method comprises the following steps:
step 110, inputting the target scene depth map into an example segmentation model to obtain a grabbed target mask and a vacuum adsorption target mask output by the example segmentation model; the target scene at least comprises an object to be grabbed;
the example segmentation model is obtained based on a sample scene depth map, a grabbed sample mask of the sample scene depth map, a vacuum adsorption sample mask of the sample scene depth map, a score of the grabbed sample mask and a score training of the vacuum adsorption sample mask; the sample scene includes at least one sample object to be grabbed.
Specifically, the target scene at least comprises one object to be grabbed, and the target scene depth map comprises point cloud information of the target scene and point cloud information of the object to be grabbed in the target scene. If the target scene only contains one object to be grabbed, namely other objects do not exist in the target scene, the double-arm robot cannot collide with other objects in the scene when grabbing the object, so that in this case, the target scene depth map is input into the example segmentation model, and the obtained grabbing target mask and the vacuum adsorption target mask both correspond to the object, namely the object can be regarded as the optimal grabbing target.
When a target scene contains a plurality of objects to be grabbed, due to the fact that the plurality of objects to be grabbed in the target scene may be in a messy stacked arrangement, if the two-arm robot randomly grabs or vacuum-adsorbs one object, collision may occur with other objects, and therefore, an optimal grabbed object and an optimal vacuum-adsorbed object need to be determined from the plurality of objects to be grabbed, namely point cloud information corresponding to the optimal grabbed object is a grabbed target mask, and point cloud information corresponding to the optimal vacuum-adsorbed object is a vacuum-adsorbed target mask. The grabbing target mask and the vacuum adsorption target mask can correspond to the same object or different objects.
In addition, before the target scene depth map is input into the example segmentation model, the example segmentation model may be trained in advance based on training samples (the sample scene depth map, the grab sample mask of the sample scene depth map, the vacuum adsorption sample mask of the sample scene depth map, the score of the grab sample mask, and the score of the vacuum adsorption sample mask), and the training process is as follows:
firstly, a simulation image (namely a sample scene depth map) of a sample scene and annotation information are generated through a simulation engine, and then an object which is most suitable for capturing and most suitable for vacuum adsorption is selected from the simulation image by utilizing the annotation information to serve as a sample of a training example segmentation model. And finally, training the initial model by a large number of samples by utilizing a deep learning technology to obtain an example segmentation model, and realizing robustness and generalization capability of flexibly grabbing objects by utilizing two fingers of clamping jaws and a vacuum chuck in a disordered stacking scene.
After the object which is most suitable for grabbing and most suitable for vacuum adsorption is selected, the score of the grabbing sample mask and the grabbing sample mask which are most suitable for grabbing the object and the score of the vacuum adsorption sample mask and the vacuum adsorption sample mask which are most suitable for vacuum adsorption of the object are determined. The above sample was obtained as follows:
(a) grab sample position appearance
Firstly, collecting N sampling points from each sample object to be grabbed in a sample scene depth map of a specified area. Then, for each sample point p, the eigenvectors of the matrix are calculated according to equation (1).
Figure BDA0003090620500000081
Wherein,
Figure BDA0003090620500000091
surface normal vectors representing unit length at p points, calculated by a common standard algorithm, Br(p)Indicating the radius of the reference frame truncated centered at point p. Using F (p) ═ v3(p)v2(p)v1(p)]An orthogonal coordinate system representing p points, where v1(p) maximum eigenvalue of correspondence matrix M (p), v3(p) the minimum eigenvalue of the corresponding matrix M (p). Vector v3(p) is a smoothed estimate of the normal vector at point p, vector v2(p) axis of large principal curvature at point p, vector v1(p) represents the axis of small principal curvature at point p. To make v3(p) pointing to the outside of the object surface, and rotating F (p) by 180 degrees. For each reference frame p ∈ C ≧ Br(p)Searching a pose which enables the clamping jaw not to collide with point clouds in a scene and at least one point cloud exists in a closed area of the clamping jaw by using a method based on a grid, and specifically:
first, a search is performed in a two-dimensional grid G — Y × Φ, where Y and Φ represent the degree of dispersion between the Y axis and the x axis in the coordinate system f (p) (Y is 10 and Φ is 8 in this embodiment). For each (y, Φ) eg, the robotic arm is translated and rotated according to coordinate system f (p). The robot arm is then pushed to move in the negative x-axis direction until the base of the robot arm or one of the robot arms contacts the surface of the object. Using T(x,y,Φ)∈R4×4A homogeneous transformation matrix representing translation along the x, y plane and rotation along the z axis, where x, y, Φ represent the transformed values along the x, y, z axes, respectively. Meanwhile, for each (y, Φ) ∈ G, F (h) is usedy,Φ) The grasp pose until the robot arm touches the surface of the object in the grid y, Φ is expressed as shown in equation (2):
Figure BDA0003090620500000092
wherein x is*Min x ∈ R such that B (h)x,y,Φ(p)). andgate.C.phi. And if the number of the point clouds in the closed area meets the condition 1, adding the grabbing poses into a candidate grabbing pose set.
Finally, calculating the score S of each grabbing pose according to the formula (3)gAnd selecting the grabbing pose with the highest score as the optimal grabbing pose. Wherein d isgIs the linear distance from the center of the object to the grabbing direction，θgIs the angle between the gripping direction and gravity.
Sg＝1-(dg+θg) (3)
(b) Vacuum adsorption sample pose
The method comprises the steps of firstly calculating the curvature of each point of each sample object to be grabbed in a sample scene depth map, then marking the points with the curvatures larger than a specified threshold value as non-adsorbable points, and marking the rest points as candidate adsorbable points.
Each candidate adsorption point score S is then calculated using equation (4)s. Wherein S isaIndicates the angle between the adsorption direction and the gravity direction corresponding to the adsorption point, SdDenotes the distance of the adsorption point from the center of the object, SbRepresenting the minimum distance of the adsorption site to the non-adsorption site.
Figure BDA0003090620500000101
(c) Optimal gripping and vacuum suction of sample objects
Calculating the grabbing score S of each sample object to be grabbed according to the formula (5)mgAnd calculating the vacuum adsorption score S of each sample object to be grabbed according to the formula (6)ms：
Smg＝dl+rv (5)
Sms＝dr+am (6)
Wherein d islIs the distance from the center of the object to the left reference point, drIs the distance from the center of the object to the right reference point, amIs the visible area of the object, rvIs the visible proportion (visible area/total area) of the object.
Then, for SmgAnd SmsSorting is carried out to respectively obtain a list LsAnd LgThe object with the highest score is selected from the list as the training sample, i.e. from the list LsMiddle picking SmgThe mask corresponding to the largest sample object is used as a capture sample mask, and the maximum S is usedmgAs a score of the grab sample mask; from the list LgMiddle picking SmsThe mask corresponding to the largest sample object is used as the vacuum adsorption sample mask, and the maximum S is usedmsScoring as a vacuum suction sample mask;
after obtaining the sample scene depth map, the grab sample mask of the sample scene depth map, the vacuum adsorption sample mask of the sample scene depth map, the score of the grab sample mask, and the score of the vacuum adsorption sample mask, the sample scene depth map may be regarded as a set of an optimal grab target mask and an optimal adsorption target mask to serve as a true value of the initial mask branch. The score of the grab sample mask and the score of the vacuum suction sample mask are used to serve as the true value of the policy confidence branch. The grab sample mask center offset and the vacuum suction sample mask center offset are used to serve as the true values for the center offset branch.
As shown in fig. 2, the example segmentation model obtains the grab target and the vacuum adsorption target as follows:
extracting features through a backbone network, wherein the backbone network takes the structured point cloud as input and then outputs a 64-dimensional feature map. In conjunction with this are three parallel convolution branches that predict three outputs: initial segmentation mask, policy confidence, center offset.
The initial segmentation mask divides the scene into two parts, foreground (the set of candidate grab objects) and background. And-ing the initial segmentation mask with the center-shifted map to obtain a center-shifted map of the foreground. The center shift map is a three channel image, each channel representing the x, y, z values of a point-to-object center point vector on the object, respectively. In the foreground, the center offset of each pixel points to a central point, the central points of the objects are clustered through a clustering algorithm, and the corresponding pixels on the corresponding images are found through indexes of the clustered points, so that candidate segmentation masks are obtained.
The grasping strategy confidence of each pixel in the scene can be obtained through the strategy confidence branch. After the segmentation masks are obtained, two grabbing scores of each candidate segmentation mask are respectively calculated by using grabbing strategy confidence degrees according to a formula (7)
Figure BDA0003090620500000111
And vacuum sorption scoring
Figure BDA0003090620500000112
Figure BDA0003090620500000113
Wherein N iskIs the number of pixels in the k-th Mask,
Figure BDA0003090620500000114
and
Figure BDA0003090620500000115
the score of the grabbed sample mask and the score of the vacuum suction sample mask of the ith pixel are respectively expressed. And respectively selecting the Mask with the maximum score of the grabbed sample Mask and the Mask with the maximum score of the vacuum adsorption sample Mask as the grabbed target and the vacuum adsorption target.
And 120, determining the pose of the grab target based on the grab target mask, and determining the pose of the vacuum adsorption target based on the vacuum adsorption target mask.
Specifically, after the grab target mask is determined, that is, the best grabbed object determined from the multiple objects to be grabbed in the target scene is sampled from multiple points on the best grabbed object to serve as candidate grabbing poses, the score of each candidate grabbing pose is judged, and the candidate grabbing pose with the largest score is taken as the grab target pose.
Similarly, after the vacuum adsorption target mask is determined, namely the best vacuum adsorption object determined from the multiple objects to be vacuum adsorbed in the target scene is sampled from multiple points on the best vacuum adsorption object to serve as candidate vacuum adsorption poses, the score of each candidate vacuum adsorption pose is judged, and the candidate vacuum adsorption pose with the largest score is taken as the vacuum adsorption target pose.
Step 130, sending the grabbing target pose and the vacuum adsorption target pose to an executing mechanism of the double-arm robot so that the executing mechanism grabs the object in the target scene depth map; the actuating mechanism comprises a grabbing mechanical arm and a vacuum adsorption mechanical arm.
Specifically, after the grabbing target pose and the vacuum adsorption target pose are determined, the grabbing target pose and the vacuum adsorption target pose are sent to an executing mechanism of the double-arm robot, so that the grabbing mechanical arm can grab a corresponding object based on the grabbing target pose, and the vacuum adsorption mechanical arm can grab the corresponding object based on the vacuum adsorption target pose in an adsorption mode.
Compared with the traditional method in which a large number of scenes need to be manually set to obtain training data, the method and the device provided by the embodiment of the invention adopt simulation, can quickly generate a large number of sample scenes, and are simple, convenient and quick. Compared with the traditional method that machine learning models need to be trained respectively according to different clamping jaws, the method provided by the embodiment of the invention does not need to acquire the structures of the clamping jaws when training the example segmentation model, and only needs to adjust corresponding parameters according to the clamping jaws of different models when determining the pose of the grabbing target, namely, the trained segmentation model can be suitable for the clamping jaws of different models, and the generalization capability of the model is strong.
According to the grabbing method based on the double-arm robot, the grabbing target mask and the vacuum adsorption target mask can be rapidly and accurately obtained based on the example segmentation model, and meanwhile, the structure of the grabbing mechanical arm clamping jaw does not need to be acquired for training when the example segmentation model is trained, so that the example segmentation model can be suitable for grabbing mechanical arm clamping jaws of different models, and the generalization capability of the model is strong. After the grabbed target mask and the vacuum adsorption target mask are determined, the grabbed target pose and the vacuum adsorption target pose can be accurately obtained, so that the execution mechanism of the double-arm robot can accurately grab objects in the target scene depth map, and the robustness is high.
Based on the above embodiment, determining the pose of the grab target based on the grab target mask, and determining the pose of the vacuum adsorption target based on the vacuum adsorption target mask includes:
acquiring a plurality of candidate grabbing poses from a first object corresponding to a grabbing target mask, determining grabbing pose scores corresponding to the candidate grabbing poses based on grabbing orientation information of the first object, and taking the candidate grabbing pose corresponding to the maximum grabbing pose score as a grabbing target pose;
and acquiring a plurality of candidate vacuum adsorption poses from a second object corresponding to the vacuum adsorption target mask, determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose based on the adsorption azimuth information of the second object, and taking the candidate vacuum adsorption pose corresponding to the maximum vacuum adsorption pose score as the vacuum adsorption target pose.
Specifically, the first object refers to an optimal grabbed object in the target scene, N sampling points are collected from the first object, and since there may be a point causing collision between the clamping jaw and the object in the N sampling points, a candidate grabbing pose needs to be screened out on the basis of the N sampling points, for example, the candidate grabbing pose can be screened out by using that the clamping jaw does not collide with other objects in the target scene and at least one point cloud exists in a closed area of the point cloud of the clamping jaw (namely, an object to be grabbed exists in the closed area of the point cloud of the clamping jaw). After a plurality of candidate grabbing poses are obtained, based on grabbing orientation information (such as a linear distance from a center of the first object to a grabbing direction) of the first object, grabbing pose scores corresponding to the candidate grabbing poses are determined, and the higher the grabbing pose score is, the more suitable the grabbing pose is for grabbing, so that the candidate grabbing pose corresponding to the maximum grabbing pose score is used as a grabbing target pose, and the object can be grabbed at the best pose.
The second object refers to an optimal vacuum adsorption object in a target scene, the curvatures of all points on the second object are different, if the curvatures are too large, the corresponding points are not suitable for vacuum adsorption, therefore, candidate vacuum adsorption poses need to be screened out on the second object, and for example, the corresponding poses with the curvatures less than or equal to a preset value can be used as the candidate vacuum adsorption poses. After a plurality of candidate vacuum adsorption poses are obtained, based on adsorption azimuth information (such as an included angle between the adsorption direction of the second object and the gravity direction) of the second object, a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose is determined, and the higher the vacuum adsorption pose score is, the more suitable the vacuum adsorption is to be performed with the pose, so that the candidate vacuum adsorption pose corresponding to the maximum vacuum adsorption pose score is used as a vacuum adsorption target pose, and the object can be vacuum-adsorbed with the best pose.
Based on any of the above embodiments, the grasping orientation information of the first object includes a linear distance from the center of the first object to the grasping direction, and an included angle between the grasping direction of the first object and the gravity direction of the first object;
acquiring a plurality of candidate grabbing poses from a first object corresponding to a grabbing target mask, and determining grabbing pose scores corresponding to the candidate grabbing poses based on grabbing orientation information of the first object, wherein the grabbing pose scores include:
acquiring a plurality of initial grabbing poses from a first object corresponding to a grabbing target mask code based on a grid method, and determining a clamping jaw point cloud corresponding to each initial grabbing pose based on each initial grabbing pose;
if the clamping jaw point cloud does not collide with the point cloud in the target scene depth map and at least one point cloud exists in a closed area of the clamping jaw point cloud, taking an initial grabbing pose corresponding to the clamping jaw point cloud as a candidate grabbing pose;
inputting the linear distance from the center of the first object to the grabbing direction and the included angle between the grabbing direction of the first object and the gravity direction of the first object into a grabbing pose score model, and determining grabbing pose scores corresponding to the candidate grabbing poses; the grabbing pose scoring model is as follows:
Sg＝1-(dg+θg)；
wherein S isgRepresenting the corresponding grab pose score of each candidate grab pose, dgRepresenting the linear distance, theta, from the center of the first object to the direction of capturegThe included angle between the grabbing direction of the first object and the gravity direction of the first object is shown.
Specifically, the method for acquiring pose of grab samples may be adopted to acquire a plurality of initial grab poses of the first object, that is, the plurality of initial grab poses of the first object are determined according to equations (1) and (2). In order to avoid that the initial grabbing pose enables the clamping jaw to collide with other objects or the initial grabbing pose cannot grab the objects, the embodiment of the invention screens each initial grabbing pose, specifically, based on each initial grabbing pose, clamping jaw point clouds corresponding to each initial grabbing pose are determined, and if the clamping jaw point clouds do not collide with the point clouds in the target scene depth map and at least one point cloud exists in a closed area of the clamping jaw point clouds, the initial grabbing pose corresponding to the clamping jaw point clouds is used as a candidate grabbing pose.
After candidate grabbing poses are obtained, the linear distance from the center of the first object to the grabbing direction and the included angle between the grabbing direction of the first object and the gravity direction of the first object are input into a grabbing pose scoring model, grabbing pose scores corresponding to the candidate grabbing poses are determined, and the higher the grabbing pose score is, the more suitable the grabbing pose is to grab the object, so that the candidate grabbing pose corresponding to the maximum grabbing pose score is used as a grabbing target pose in the embodiment of the invention, and the object can be grabbed at the best pose.
Based on any of the above embodiments, the adsorption azimuth information of the second object includes an included angle between the adsorption direction of the second object and the gravity direction of the second object, a distance between the adsorption point of the second object and the center of the second object, and a minimum distance between the adsorption point of the second object and the non-adsorption point of the second object; the non-adsorbable point of the second object refers to a point on the second object with a curvature greater than a preset value;
acquiring a plurality of candidate vacuum adsorption poses from a second object corresponding to the vacuum adsorption target mask, and determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose based on adsorption azimuth information of the second object, wherein the vacuum adsorption pose score comprises the following steps:
determining the curvatures of all points on a second object corresponding to the vacuum adsorption target mask, and taking the corresponding positions with the curvatures less than or equal to a preset value as candidate vacuum adsorption positions;
inputting an included angle between the adsorption direction of the second object and the gravity direction of the second object, a distance between an adsorption point of the second object and the center of the second object and a minimum distance between an adsorption point of the second object and a non-adsorption point of the second object into a vacuum adsorption pose scoring model, and determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose; the vacuum adsorption pose scoring model is as follows:
Figure BDA0003090620500000151
wherein S issVacuum adsorption pose scores representing the correspondence of each candidate vacuum adsorption pose, SaRepresents the angle between the second object adsorption direction and the second object gravity direction, SdDenotes the distance between the adsorption point of the second object and the center of the second object, SbRepresenting the minimum distance between the adsorption point of the second object and the non-adsorption point of the second object.
Specifically, curvatures of points on the second object are different, and if the curvatures are too large, the corresponding points are not suitable for vacuum adsorption, so that candidate vacuum adsorption poses need to be screened out on the second object; and taking the point with the curvature larger than the preset value as the non-adsorbable point of the second object.
After a plurality of candidate vacuum adsorption poses are obtained, an included angle between the adsorption direction of the second object and the weight direction of the second object, a distance between an adsorption point of the second object and the center of the second object and a minimum distance between an adsorption point of the second object and a non-adsorption point of the second object are input into a vacuum adsorption pose scoring model, a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose is determined, the higher the vacuum adsorption pose score is, the more suitable the vacuum adsorption is to be carried out at the pose, and therefore the candidate vacuum adsorption pose corresponding to the maximum vacuum adsorption pose score is used as a vacuum adsorption target pose, and the object can be vacuum-adsorbed at the best pose.
Based on any of the above embodiments, the score of the grab sample mask and the grab sample mask are determined based on the following steps:
acquiring a plurality of candidate sample grabbing poses from each sample object, determining a grabbed sample pose score corresponding to each candidate sample grabbing pose based on the grabbing position information of each sample object, and taking the candidate sample grabbing pose corresponding to the maximum grabbed sample pose score as a grabbed sample target pose of each sample object;
and determining the grab score of each sample object based on the grab sample target pose of each sample object, taking the maximum grab score as the score of the grab sample mask code, and determining the grab sample mask code based on the grab sample target pose corresponding to the maximum grab score.
Specifically, a plurality of candidate sample grabbing poses can be obtained from each sample object according to the formulas (1) and (2), grabbing sample pose scores corresponding to the candidate sample grabbing poses are determined based on the formula (3), and meanwhile, the candidate sample grabbing pose corresponding to the largest grabbing sample pose score is used as the grabbing sample target pose of each sample object, so that the corresponding optimal grabbing pose of each sample object can be screened out.
However, there may be multiple sample objects in the sample scene, and the sample objects are randomly stacked, so as to avoid the collision between the clamping jaw and other objects during the grabbing process, the best grabbing object needs to be confirmed from the multiple sample objects. Therefore, according to the embodiment of the invention, the grabbing score S of each sample object can be determined according to the formula (5)mgAnd determining the mask code of the grabbed sample based on the target pose of the grabbed sample corresponding to the maximum grabbing score.
In any of the above embodiments, the vacuum suction sample mask and the score of the vacuum suction sample mask are determined based on the following steps:
acquiring a plurality of candidate sample vacuum adsorption poses from each sample object, determining a vacuum adsorption sample pose score corresponding to each candidate sample vacuum adsorption pose based on adsorption azimuth information of each sample object, and taking the candidate sample vacuum adsorption pose corresponding to the maximum vacuum adsorption sample pose score as a vacuum adsorption sample target pose of each sample object;
and determining a vacuum adsorption score of each sample object based on the vacuum adsorption sample target pose of each sample object, taking the maximum vacuum adsorption score as the score of the vacuum adsorption sample mask, and determining the vacuum adsorption sample mask based on the vacuum adsorption sample target pose corresponding to the maximum vacuum adsorption score.
Specifically, based on the curvature of each point on each sample object, the candidate sample vacuum adsorption pose corresponding to each sample object can be determined, the vacuum adsorption sample pose score corresponding to each candidate sample vacuum adsorption pose is determined based on the formula (4), and meanwhile, the candidate sample vacuum adsorption pose corresponding to the maximum vacuum adsorption sample pose score is used as the vacuum adsorption sample target pose of each sample object, so that the corresponding optimal vacuum adsorption pose on each sample object can be screened out.
However, there may be a plurality of sample objects in the sample scene, and the sample objects are randomly stacked, so as to avoid collision of the vacuum chuck with other objects during the vacuum suction process, it is necessary to determine the best vacuum suction object from the plurality of sample objects. Therefore, according to the embodiment of the present invention, the vacuum adsorption score S of each sample object can be determined according to the formula (6)msAnd determining the mask of the vacuum adsorption sample based on the target pose of the vacuum adsorption sample corresponding to the maximum vacuum adsorption score.
The following describes the two-arm robot-based grasping apparatus according to the present invention, and the two-arm robot-based grasping apparatus described below and the two-arm robot-based grasping method described above may be referred to in correspondence with each other.
Based on any one of the above embodiments, the present invention provides a grabbing device based on a two-arm robot, as shown in fig. 3, the grabbing device includes:
a target mask determining unit 310, configured to input the target scene depth map into the example segmentation model, and obtain a grab target mask and a vacuum adsorption target mask output by the example segmentation model; the target scene at least comprises one object to be grabbed;
a target pose determination unit 320, configured to determine a grab target pose based on the grab target mask, and determine a vacuum adsorption target pose based on the vacuum adsorption target mask;
an object grabbing unit 330, configured to send the grabbing target pose and the vacuum adsorption target pose to an actuator of a two-arm robot, so that the actuator grabs an object in the target scene depth map; the executing mechanism comprises a grabbing mechanical arm and a vacuum adsorption mechanical arm;
wherein the example segmentation model is trained based on a sample scene depth map, a grab sample mask of the sample scene depth map, a vacuum suction sample mask of the sample scene depth map, a score of the grab sample mask, and a score of the vacuum suction sample mask; the sample scene comprises at least one sample object to be grabbed.
Based on any embodiment, the object pose determining unit 320 includes:
a grab target pose determining unit, configured to acquire multiple candidate grab poses from the first object corresponding to the grab target mask, determine a grab pose score corresponding to each candidate grab pose based on the grab orientation information of the first object, and use the candidate grab pose corresponding to the maximum grab pose score as the grab target pose;
and the adsorption target pose determining unit is used for acquiring a plurality of candidate vacuum adsorption poses from a second object corresponding to the vacuum adsorption target mask, determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose based on the adsorption azimuth information of the second object, and taking the candidate vacuum adsorption pose corresponding to the maximum vacuum adsorption pose score as the vacuum adsorption target pose.
Based on any one of the above embodiments, the grabbing position information of the first object includes a linear distance from the center of the first object to the grabbing direction, and an included angle between the grabbing direction of the first object and the gravity direction of the first object;
the grab target pose determination unit includes:
a clamping jaw point cloud determining unit, configured to obtain a plurality of initial grabbing poses from the first object corresponding to the grabbing target mask based on a grid method, and determine a clamping jaw point cloud corresponding to each initial grabbing pose based on each initial grabbing pose;
a candidate grabbing pose determining unit, configured to take an initial grabbing pose corresponding to the jaw point cloud as a candidate grabbing pose if the jaw point cloud does not collide with a point cloud in a target scene depth map and at least one point cloud exists in a closed area of the jaw point cloud;
the grabbing pose score determining unit is used for inputting the linear distance from the center of the first object to a grabbing direction and the included angle between the grabbing direction of the first object and the gravity direction of the first object into a grabbing pose score model and determining grabbing pose scores corresponding to the candidate grabbing poses; the grabbing pose scoring model is as follows:
Sg＝1-(dg+θg)；
wherein S isgRepresenting the corresponding grab pose score of each candidate grab pose, dgRepresenting a linear distance, θ, from the center of the first object to the gripping directiongAnd the included angle between the grabbing direction of the first object and the gravity direction of the first object is represented.
Based on any of the above embodiments, the adsorption azimuth information of the second object includes an included angle between the adsorption direction of the second object and the gravity direction of the second object, a distance between the adsorption point of the second object and the center of the second object, and a minimum distance between the adsorption point of the second object and the non-adsorption point of the second object; the non-adsorbable point of the second object refers to a point on the second object, the curvature of which is greater than a preset value;
the adsorption target pose determination unit includes:
the candidate vacuum adsorption pose determining unit is used for determining the curvatures of all points on the second object corresponding to the vacuum adsorption target masks, and taking the poses corresponding to the points with the curvatures less than or equal to a preset value as candidate vacuum adsorption poses;
the vacuum adsorption pose scoring determination unit is used for inputting an included angle between the adsorption direction of the second object and the weight direction of the second object, a distance between the adsorption point of the second object and the center of the second object and a minimum distance between the adsorption point of the second object and the non-adsorption point of the second object into the vacuum adsorption pose scoring model, and determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose; the vacuum adsorption pose scoring model is as follows:
Figure BDA0003090620500000201
wherein S isSVacuum adsorption pose scores representing the correspondence of each candidate vacuum adsorption pose, SaRepresents an angle between the second object adsorption direction and the second object gravity direction, SdRepresents the distance between the adsorption point of the second object and the center of the second object, SbRepresents a minimum distance between the adsorption point of the second object and the non-adsorption point of the second object.
Based on any embodiment above, still include:
the sample grabbing target pose determining unit is used for acquiring a plurality of candidate sample grabbing poses from each sample object, determining a sample grabbing pose score corresponding to each candidate sample grabbing pose based on the grabbing azimuth information of each sample object, and taking the candidate sample grabbing pose corresponding to the maximum sample grabbing pose score as the sample grabbing target pose of each sample object;
the first determining unit is used for determining a grab score of each sample object based on the grab sample target pose of each sample object, taking the maximum grab score as the score of the grab sample mask code, and determining the grab sample mask code based on the grab sample target pose corresponding to the maximum grab score.
Based on any embodiment above, still include:
the vacuum adsorption sample target pose determining unit is used for acquiring a plurality of candidate sample vacuum adsorption poses from each sample object, determining a vacuum adsorption sample pose score corresponding to each candidate sample vacuum adsorption pose based on adsorption azimuth information of each sample object, and taking the candidate sample vacuum adsorption pose corresponding to the maximum vacuum adsorption sample pose score as the vacuum adsorption sample target pose of each sample object;
and the second determining unit is used for determining a vacuum adsorption score of each sample object based on the vacuum adsorption sample target pose of each sample object, taking the maximum vacuum adsorption score as the score of the vacuum adsorption sample mask, and determining the vacuum adsorption sample mask based on the vacuum adsorption sample target pose corresponding to the maximum vacuum adsorption score.
Based on any one of the above embodiments, the present invention provides a dual-arm robot, including: the double-arm robot-based gripping device as described in any of the above embodiments.
In particular, the dual-arm robot may further include a sensor module and an execution module.
The sensor module can use a PhoXi 3D scanner to sense the 3-dimensional point cloud structure, the resolution of the scanner can be 1024 × 772, and the installation position can be 1.3m above the grabbing area. The grabbing device based on the double-arm robot is used for receiving the images acquired by the sensor module, calculating the grabbing target pose and the vacuum adsorption target pose in the scene, processing the grabbing target pose and the vacuum adsorption target pose into instructions required by the operation execution module of the double-arm robot, and sending the instructions to the execution module. The execution module is used for grabbing objects according to instructions, and the execution module comprises a robot body, two fingers and a vacuum chuck. Wherein the two fingers clamping jaws are fixedly connected on the tail end of one mechanical arm of the robot body, and the vacuum chuck is fixed on the tail end of the other mechanical arm of the robot body.
Fig. 4 is a schematic structural diagram of an electronic device provided in the present invention, and as shown in fig. 4, the electronic device may include: a processor (processor)410, a memory (memory)420, a communication Interface (Communications Interface)430 and a communication bus 440, wherein the processor 410, the memory 420 and the communication Interface 430 are configured to communicate with each other via the communication bus 440. The processor 410 may invoke logic instructions in the memory 420 to perform a dual-arm robot-based grasping method comprising: inputting the target scene depth map into an example segmentation model to obtain a grab target mask and a vacuum adsorption target mask output by the example segmentation model; the target scene at least comprises one object to be grabbed; determining a grab target pose based on the grab target mask, and determining a vacuum adsorption target pose based on the vacuum adsorption target mask; sending the grabbing target pose and the vacuum adsorption target pose to an actuating mechanism of the double-arm robot so that the actuating mechanism grabs the object in the target scene depth map; the executing mechanism comprises a grabbing mechanical arm and a vacuum adsorption mechanical arm; wherein the example segmentation model is trained based on a sample scene depth map, a grab sample mask of the sample scene depth map, a vacuum suction sample mask of the sample scene depth map, a score of the grab sample mask, and a score of the vacuum suction sample mask; the sample scene comprises at least one sample object to be grabbed.
Furthermore, the logic instructions in the memory 420 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the dual-arm robot-based grasping method provided by the above methods, the method comprising: inputting the target scene depth map into an example segmentation model to obtain a grab target mask and a vacuum adsorption target mask output by the example segmentation model; the target scene at least comprises one object to be grabbed; determining a grab target pose based on the grab target mask, and determining a vacuum adsorption target pose based on the vacuum adsorption target mask; sending the grabbing target pose and the vacuum adsorption target pose to an actuating mechanism of the double-arm robot so that the actuating mechanism grabs the object in the target scene depth map; the executing mechanism comprises a grabbing mechanical arm and a vacuum adsorption mechanical arm; wherein the example segmentation model is trained based on a sample scene depth map, a grab sample mask of the sample scene depth map, a vacuum suction sample mask of the sample scene depth map, a score of the grab sample mask, and a score of the vacuum suction sample mask; the sample scene comprises at least one sample object to be grabbed.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which when executed by a processor, is implemented to perform the above-provided two-arm robot-based grasping method, the method comprising: inputting the target scene depth map into an example segmentation model to obtain a grab target mask and a vacuum adsorption target mask output by the example segmentation model; the target scene at least comprises one object to be grabbed; determining a grab target pose based on the grab target mask, and determining a vacuum adsorption target pose based on the vacuum adsorption target mask; sending the grabbing target pose and the vacuum adsorption target pose to an actuating mechanism of the double-arm robot so that the actuating mechanism grabs the object in the target scene depth map; the executing mechanism comprises a grabbing mechanical arm and a vacuum adsorption mechanical arm; wherein the example segmentation model is trained based on a sample scene depth map, a grab sample mask of the sample scene depth map, a vacuum suction sample mask of the sample scene depth map, a score of the grab sample mask, and a score of the vacuum suction sample mask; the sample scene comprises at least one sample object to be grabbed.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A grabbing method based on a double-arm robot is characterized by comprising the following steps:
inputting the target scene depth map into an example segmentation model to obtain a grab target mask and a vacuum adsorption target mask output by the example segmentation model; the target scene at least comprises one object to be grabbed;
determining a grab target pose based on the grab target mask, and determining a vacuum adsorption target pose based on the vacuum adsorption target mask;
sending the grabbing target pose and the vacuum adsorption target pose to an actuating mechanism of the double-arm robot so that the actuating mechanism grabs the object in the target scene depth map; the executing mechanism comprises a grabbing mechanical arm and a vacuum adsorption mechanical arm;
wherein the example segmentation model is trained based on a sample scene depth map, a grab sample mask of the sample scene depth map, a vacuum suction sample mask of the sample scene depth map, a score of the grab sample mask, and a score of the vacuum suction sample mask; the sample scene comprises at least one sample object to be grabbed.
2. The dual-arm robot-based grabbing method according to claim 1, wherein said determining grabbing target poses based on the grabbing target mask and vacuum adsorption target poses based on the vacuum adsorption target mask comprises:
acquiring a plurality of candidate grabbing poses from a first object corresponding to the grabbing target mask, determining grabbing pose scores corresponding to the candidate grabbing poses based on the grabbing orientation information of the first object, and taking the candidate grabbing pose corresponding to the maximum grabbing pose score as the grabbing target pose;
and acquiring a plurality of candidate vacuum adsorption poses from a second object corresponding to the vacuum adsorption target mask, determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose based on the adsorption azimuth information of the second object, and taking the candidate vacuum adsorption pose corresponding to the maximum vacuum adsorption pose score as the vacuum adsorption target pose.
3. The dual-arm robot-based gripping method according to claim 2, wherein the gripping orientation information of the first object includes a linear distance from a center of the first object to a gripping direction, and an angle between the gripping direction of the first object and a direction of gravity of the first object;
the acquiring a plurality of candidate grab poses from the first object corresponding to the grab target mask code, and determining a grab pose score corresponding to each candidate grab pose based on the grab orientation information of the first object includes:
acquiring a plurality of initial grabbing poses from the first object corresponding to the grabbing target mask based on a grid method, and determining a clamping jaw point cloud corresponding to each initial grabbing pose based on each initial grabbing pose;
if the clamping jaw point cloud does not collide with the point cloud in the target scene depth map and at least one point cloud exists in a closed area of the clamping jaw point cloud, taking an initial grabbing pose corresponding to the clamping jaw point cloud as the candidate grabbing pose;
inputting the linear distance from the center of the first object to the grabbing direction and the included angle between the grabbing direction of the first object and the gravity direction of the first object into a grabbing pose scoring model, and determining grabbing pose scores corresponding to the candidate grabbing poses; the grabbing pose scoring model is as follows:
Sg＝1-(dg+θg)；
wherein S isgRepresenting the corresponding grab pose score of each candidate grab pose, dgRepresenting a linear distance, θ, from the center of the first object to the gripping directiongAnd the included angle between the grabbing direction of the first object and the gravity direction of the first object is represented.
4. The dual-arm robot-based gripping method according to claim 2, wherein the suction orientation information of the second object includes an angle between the second object suction direction and the second object weight direction, a distance between the suction point of the second object and the center of the second object, and a minimum distance between the suction point of the second object and the non-suction point of the second object; the non-adsorbable point of the second object refers to a point on the second object, the curvature of which is greater than a preset value;
the acquiring a plurality of candidate vacuum adsorption poses from a second object corresponding to the vacuum adsorption target mask, and determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose based on adsorption azimuth information of the second object includes:
determining the curvatures of all points on a second object corresponding to the vacuum adsorption target mask, and taking the corresponding positions with the curvatures less than or equal to a preset value as candidate vacuum adsorption positions;
inputting an included angle between the adsorption direction of the second object and the weight direction of the second object, a distance between the adsorption point of the second object and the center of the second object and a minimum distance between the adsorption point of the second object and the non-adsorption point of the second object into a vacuum adsorption pose scoring model, and determining a vacuum adsorption pose score corresponding to each candidate vacuum adsorption pose; the vacuum adsorption pose scoring model is as follows:
Figure FDA0003090620490000031
wherein S issVacuum adsorption pose scores representing the correspondence of each candidate vacuum adsorption pose, SaRepresents an angle between the second object adsorption direction and the second object gravity direction, SdRepresents the distance between the adsorption point of the second object and the center of the second object, SbRepresents a minimum distance between the adsorption point of the second object and the non-adsorption point of the second object.
5. The dual-arm robot-based grasping method according to claim 1, wherein the grasping sample mask and the score of the grasping sample mask are determined based on:
acquiring a plurality of candidate sample grabbing poses from each sample object, determining a grabbed sample pose score corresponding to each candidate sample grabbing pose based on the grabbing position information of each sample object, and taking the candidate sample grabbing pose corresponding to the maximum grabbed sample pose score as a grabbed sample target pose of each sample object;
and determining the grab score of each sample object based on the grab sample target pose of each sample object, taking the maximum grab score as the score of the grab sample mask code, and determining the grab sample mask code based on the grab sample target pose corresponding to the maximum grab score.
6. The dual-arm robot-based gripping method according to claim 1, wherein the vacuum adsorption sample mask and the score of the vacuum adsorption sample mask are determined based on the steps of:
acquiring a plurality of candidate sample vacuum adsorption poses from each sample object, determining a vacuum adsorption sample pose score corresponding to each candidate sample vacuum adsorption pose based on adsorption azimuth information of each sample object, and taking the candidate sample vacuum adsorption pose corresponding to the maximum vacuum adsorption sample pose score as a vacuum adsorption sample target pose of each sample object;
and determining a vacuum adsorption score of each sample object based on the vacuum adsorption sample target pose of each sample object, taking the maximum vacuum adsorption score as the score of the vacuum adsorption sample mask, and determining the vacuum adsorption sample mask based on the vacuum adsorption sample target pose corresponding to the maximum vacuum adsorption score.
7. A grabbing device based on two-arm robot, its characterized in that includes:
the target mask determining unit is used for inputting the target scene depth map into the example segmentation model to obtain a grab target mask and a vacuum adsorption target mask output by the example segmentation model; the target scene at least comprises one object to be grabbed;
the target pose determining unit is used for determining a grab target pose based on the grab target mask and determining a vacuum adsorption target pose based on the vacuum adsorption target mask;
the object grabbing unit is used for sending the grabbing target pose and the vacuum adsorption target pose to an actuating mechanism of the double-arm robot so that the actuating mechanism grabs the object in the target scene depth map; the executing mechanism comprises a grabbing mechanical arm and a vacuum adsorption mechanical arm;
wherein the example segmentation model is trained based on a sample scene depth map, a grab sample mask of the sample scene depth map, a vacuum suction sample mask of the sample scene depth map, a score of the grab sample mask, and a score of the vacuum suction sample mask; the sample scene comprises at least one sample object to be grabbed.
8. A dual-arm robot, comprising: the dual-arm robot-based gripping apparatus of claim 7.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor when executing the program realizes the steps of the two-arm robot-based gripping method according to any one of claims 1 to 6.
10. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the steps of the dual-arm robot-based grasping method according to any one of claims 1 to 6.
CN202110594376.XA 2021-05-28 2021-05-28 Grabbing method and device based on double-arm robot and double-arm robot Pending CN113538576A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110594376.XA CN113538576A (en) 2021-05-28 2021-05-28 Grabbing method and device based on double-arm robot and double-arm robot

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110594376.XA CN113538576A (en) 2021-05-28 2021-05-28 Grabbing method and device based on double-arm robot and double-arm robot

Publications (1)

Publication Number Publication Date
CN113538576A true CN113538576A (en) 2021-10-22

Family

ID=78095042

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110594376.XA Pending CN113538576A (en) 2021-05-28 2021-05-28 Grabbing method and device based on double-arm robot and double-arm robot

Country Status (1)

Country Link
CN (1) CN113538576A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114714365A (en) * 2022-06-08 2022-07-08 湖南大学 Disordered workpiece grabbing method and system based on cloud platform
WO2024104424A1 (en) * 2022-11-17 2024-05-23 北京有竹居网络技术有限公司 Object scraping method and apparatus, computer device, and storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040096096A1 (en) * 2002-10-30 2004-05-20 Metrica, Inc. Matching binary templates against range map derived silhouettes for object pose estimation
US20160005161A1 (en) * 2014-07-01 2016-01-07 Seiko Epson Corporation Robot system
GB201813197D0 (en) * 2018-08-13 2018-09-26 Imperial Innovations Ltd Mapping object instances using video data
US20190259170A1 (en) * 2018-02-21 2019-08-22 Beijing Jingdong Shangke Information Technology Co., Ltd. System and method for feature screening in slam
CN111192293A (en) * 2019-12-27 2020-05-22 深圳市越疆科技有限公司 Moving target pose tracking method and device
CN111251294A (en) * 2020-01-14 2020-06-09 北京航空航天大学 Robot grabbing method based on visual pose perception and deep reinforcement learning
CN112233181A (en) * 2020-10-29 2021-01-15 深圳市广宁股份有限公司 6D pose recognition method and device and computer storage medium

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040096096A1 (en) * 2002-10-30 2004-05-20 Metrica, Inc. Matching binary templates against range map derived silhouettes for object pose estimation
US20160005161A1 (en) * 2014-07-01 2016-01-07 Seiko Epson Corporation Robot system
US20190259170A1 (en) * 2018-02-21 2019-08-22 Beijing Jingdong Shangke Information Technology Co., Ltd. System and method for feature screening in slam
GB201813197D0 (en) * 2018-08-13 2018-09-26 Imperial Innovations Ltd Mapping object instances using video data
CN111192293A (en) * 2019-12-27 2020-05-22 深圳市越疆科技有限公司 Moving target pose tracking method and device
CN111251294A (en) * 2020-01-14 2020-06-09 北京航空航天大学 Robot grabbing method based on visual pose perception and deep reinforcement learning
CN112233181A (en) * 2020-10-29 2021-01-15 深圳市广宁股份有限公司 6D pose recognition method and device and computer storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
SIYI DU: "《Vision-Based Robotic Manipulation of Intelligent Wheelchair with Human-Computer Shared Control》", 《IEEE》, 24 May 2021 (2021-05-24) *
杨月全: "《基于深度卷积网络学习的位姿估计及机器人抓取研究：现状与挑战》", 《第33届中国控制与决策会议论文集》, 22 May 2021 (2021-05-22) *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114714365A (en) * 2022-06-08 2022-07-08 湖南大学 Disordered workpiece grabbing method and system based on cloud platform
WO2024104424A1 (en) * 2022-11-17 2024-05-23 北京有竹居网络技术有限公司 Object scraping method and apparatus, computer device, and storage medium

Similar Documents

Publication Publication Date Title
CN109801337B (en) 6D pose estimation method based on instance segmentation network and iterative optimization
WO2020119338A1 (en) Method for detecting grabbing position of robot for target object
CN111251295B (en) Visual mechanical arm grabbing method and device applied to parameterized parts
CN112070818A (en) Robot disordered grabbing method and system based on machine vision and storage medium
CN110298886B (en) Dexterous hand grabbing planning method based on four-stage convolutional neural network
CN113409384B (en) Pose estimation method and system of target object and robot
CN111553949B (en) Positioning and grabbing method for irregular workpiece based on single-frame RGB-D image deep learning
CN112894815B (en) Method for detecting optimal position and posture for article grabbing by visual servo mechanical arm
CN111085997A (en) Capturing training method and system based on point cloud acquisition and processing
CN113538576A (en) Grabbing method and device based on double-arm robot and double-arm robot
CN112669385A (en) Industrial robot workpiece identification and pose estimation method based on three-dimensional point cloud characteristics
CN113034600A (en) Non-texture planar structure industrial part identification and 6D pose estimation method based on template matching
WO2024148645A1 (en) Apparatus for estimating under monocular infrared thermal imaging vision pose of object grasped by manipulator, and method thereof
CN114140418A (en) Seven-degree-of-freedom grabbing posture detection method based on RGB image and depth image
CN114029941B (en) Robot grabbing method and device, electronic equipment and computer medium
CN114310892B (en) Object grabbing method, device and equipment based on point cloud data collision detection
CN113762159B (en) Target grabbing detection method and system based on directional arrow model
CN115284279A (en) Mechanical arm grabbing method and device based on aliasing workpiece and readable medium
CN113034526A (en) Grabbing method, grabbing device and robot
CN112070005A (en) Three-dimensional primitive data extraction method and device and storage medium
CN114211490B (en) Method for predicting pose of manipulator gripper based on transducer model
Yu et al. Object recognition and robot grasping technology based on RGB-D data
CN114998573A (en) Grabbing pose detection method based on RGB-D feature depth fusion
Zhang et al. Object detection and grabbing based on machine vision for service robot
Zhang et al. Robotic grasp detection using effective graspable feature selection and precise classification

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination