CN114310872B - Automatic vegetable-beating method for mechanical arm based on DGG point cloud segmentation network - Google Patents
Automatic vegetable-beating method for mechanical arm based on DGG point cloud segmentation network Download PDFInfo
- Publication number
- CN114310872B CN114310872B CN202111427395.XA CN202111427395A CN114310872B CN 114310872 B CN114310872 B CN 114310872B CN 202111427395 A CN202111427395 A CN 202111427395A CN 114310872 B CN114310872 B CN 114310872B
- Authority
- CN
- China
- Prior art keywords
- point
- spoon
- dish
- mechanical arm
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Image Analysis (AREA)
- Manipulator (AREA)
Abstract
The invention relates to an automatic vegetable-beating method of a mechanical arm based on a DGG point cloud segmentation network. The automatic vegetable beating is realized by estimating the vegetable beating position and position, selecting different vegetable beating actions according to different vegetable varieties and vegetable quantities, planning the path and the track of the six-axis mechanical arm, and finally digging the vegetable. At present, the dish-making work is mainly realized in the restaurant by manpower, and the restaurant has the defects of low efficiency, high labor cost and the like. According to the invention, point cloud data on the surface of dishes are collected through a depth camera, a point cloud segmentation network DGG based on graph convolution is provided to realize prediction of the dishes, gesture information of the dishes is obtained through calculation according to specific dishes, a track at the tail end of a mechanical arm is planned through a cubic B-spline interpolation algorithm and a square interpolation algorithm, and finally the mechanical arm is controlled to complete the dishes.
Description
Technical Field
The invention belongs to the field of machine vision, and particularly relates to an automatic vegetable-playing method of a mechanical arm based on a DGG point cloud segmentation network.
Background
Aiming at an automatic dish-beating robot used in a restaurant and a canteen, the dish-beating algorithm is generally calculated by analyzing the surface depth information characteristics of dishes, however, in the presence of various dishes, the traditional algorithm is difficult to obtain ideal results, lacks certain flexibility and universality, and lacks certain stability in the presence of noise; with the continuous deep learning application in 3D detection, the method has satisfactory results on the detection and segmentation accuracy of 3D data, and can cope with various complex and changeable data, so that the success rate of dish playing can be greatly increased by adopting the deep learning method to process dish data.
Disclosure of Invention
In view of the above problems, the invention collects point cloud data of the dish surface in real time through a depth camera, simultaneously provides a point cloud segmentation network DGG based on graph convolution, processes the dish point cloud data and predicts to obtain position information of dish-making points, selects corresponding dish-making actions according to distribution conditions of dish quantity among the dish-making points, calculates to obtain gesture information of each dish-making point according to a 3D space rotation principle, plans tail end tracks of a mechanical arm through a cubic B spline interpolation algorithm and a square interpolation algorithm, obtains motion information in joint space through inverse kinematics, and finally controls the mechanical arm to complete the dish-making actions.
The invention provides an automatic vegetable-beating method of a mechanical arm based on a DGG point cloud segmentation network, which comprises the following steps:
step one: building a DGG point cloud segmentation network; firstly, constructing a feature extraction module of a local point cloud, and defining a directed graph G= (v, epsilon) to represent the geometric structure of the local point cloud, wherein v and epsilon respectively represent vertexes and edges; randomly selecting a point from the input point cloud as the center of the graph G and marking as p i From R by K nearest neighbor algorithm 3×3 Space gets k nearest neighbors { p } i1 ,p i2 ,...,p ik Defining the position characteristics of the local point cloud as
P i,m ＝{p i1,m -p i,m ,p i2,m -p i,m ,...,p ik,m -p i }
Wherein p is ik,m An mth dimension feature representing a kth point in the neighborhood;
defining the density of local point clouds as
Where k represents the number of neighborhood points, F m (j) Representative point p i And to characterize ρ in the m-th dimension of i,m As a position feature P i,m Is a weight function of (2); at the same time, the shape characteristics of the local point cloud are defined as
Wherein the method comprises the steps ofRepresenting the inner product calculation between two vectors, < ->Is an edge within graph G; handle P i,m And S is i,m Further feature extraction is realized through the MLP structures of the two layers respectively, the problem of the disorder of the point cloud is solved through the maximum pooling layer, and finally, feature aggregation is carried out to obtain the integral feature of the local point cloud.
Inputting the integral characteristics of the local point cloud into a three-layer MLP network with residual connection, and then lifting the characteristic dimension to 1024; in order to train various dish data at the same time, one-dimensional dish variety vectors are added, and the number of convolution output channels of the last layer is the total number of the segmentation labels through a common three-layer MLP after the maximum pooling function. The step size of all convolution operations in DGG network is 1, the activation function uses the leak Relu, and Dropout is added before the output layer to avoid over-fitting during training, and the inactivation rate is set to 0.5.
Step two: acquiring point cloud data of the surfaces of various dishes by using a depth camera, wherein the camera is fixed at the tail end of a mechanical arm, and adopts a calibration mode that eyes are on hands; marking data by using a Semantic-segment-Editor tool, which is divided into three tags: a spoon inlet area, a spoon outlet area and an irrelevant area; the training set and the testing set are input into a DGG point cloud segmentation model constructed in the first step, and training parameters are set: iteration times 300, batch size 32, initial learning rate 0.001, and adjusting the weight parameters of the network model by using an SGD optimizer; and saving the network weight parameters when the loss value is the lowest.
Step three: transplanting the segmentation model built in the first step to an ROS development platform, loading the network weight parameters stored in the second step, predicting target points of dish point cloud data acquired in real time in the ROS, and screening the prediction results of the network; and eliminating the situation that the number of the predicted result areas is small or the areas are separated. The prediction result is issued through the ROS message mechanism.
Step four: the mechanical arm subscribes to topics released in the third step to obtain a prediction result, and the maximum depth extracted by the end effector is calculated by combining the dish surface depth information to obtain the attitude information of the end effector of the mechanical arm in Cartesian space; the specific calculation process is as follows:
a) The mechanical arm end effector is set as a general dish-beating spoon on the market, the center of a spoon opening is set as an origin of an end coordinate system, the upward direction of a vertical spoon is set as the Z-axis direction, the direction pointing to a spoon handle is set as the Y-axis, and the X-axis direction can be determined according to the right-hand rule;
b) Taking the central point of the predicted area of the network as the scoop-in point (x s ,y s ,z s ) And spoon out point (x) e ,y e ,z e ) If the distance between the spoon inlet point or the spoon outlet point and the boundary of the dish basin is smaller than the radius r of the dish-beating spoon spoon Executing the third step, and predicting the point cloud again; the spoon inlet point is used as a starting point of the dish-beating spoon, the spoon outlet point is used as a advancing direction of the dish-beating spoon, and the advancing distance is determined by different kinds of dish densities; dish with full spoon has quality of G dish According to the dish density ρ dish The volume of the obtained dishesCombining the point cloud depth information, taking a spoon inlet point as a starting point and a spoon outlet point as a forward direction, calculating and dividing the volume into 2V by using a convex hull approximation algorithm dish The point cloud area of (1) is used as a dish-making area, and the end coordinates of the dish-making area are used as new spoon outlet pointsSelecting (x) s ,y s ,z s ) And->Is used as a transition point (x i ,y i ) Its depth value z i The calculation formula of (2) is as follows:
wherein n is 1 Is the point number between the spoon entering point and the transition point in the dish taking area，n 2 Is the number of points between the transition point and the spoon outlet point; if the calculated depth value exceeds the depth of the vegetable basin, returning to the step three, and predicting again by using a DGG network.
c) After the space position of the dish-beating spoon road points is obtained, the gesture of each point needs to be calculated; setting according to the coordinate system in a), wherein the positive direction of the z-axis represents the advancing direction of the spoon; for the quantity of dishes, two different dishes beating actions are designed, wherein the first is to rotate and advance at the same time, and the second is to push and rotate in the advancing process, and the difference is that the gesture of the waypoint is changed; firstly, setting a z-axis of a spoon inlet point to a transition point, wherein the z-axis of the transition point points to a spoon outlet point, and the z-axis of the spoon outlet point is vertically upwards; the second kind of spoon-in point z axis is parallel to the world coordinate system and points to the spoon-out point direction, the transition point has the same gesture as the spoon-in point, and represents pushing action from the spoon-in point to the transition point, and the spoon-out point z axis is also vertically upwards; all the gestures are rotated based on a camera-based coordinate system, such as the gesture of a spoon-in point in the first case, and the Euler angle is:
wherein θ is z The angle between the spoon handle and the ground is regulated and is generally controlled to be between 35 and 50 degrees; the pose calculation method of other points is similar.
Because the camera is fixed at the tail end of the mechanical arm, the coordinate system of the camera can be changed along with the movement of the mechanical arm, and all the pose information of all the road points are converted into the world coordinate system; assume that the spatial position of a certain waypoint in the camera coordinate system is (x 0 ,y 0 ,z 0 ) Euler angle is (θ x ,θ y ,θ z ) The method comprises the steps of carrying out a first treatment on the surface of the The camera is positioned in the world coordinate system at (x c ,y c ,z c ) The rotation matrix isFirst converting Euler angles into rotation matrix form +.>The formula is
Further obtain pose information under world coordinate system as
Step five, obtaining pose information based on a world coordinate system according to the step four, and obtaining track information of an end effector of the mechanical arm by combining a cubic B spline interpolation algorithm and a square interpolation algorithm; the specific calculation process is as follows:
interpolation of the spatial position of the end of the arm was performed using cubic B-splines, the formula defined as follows:
S(u)＝∑P i N i,k (u)
wherein P is i Is a control point of a spline curve, wherein a scoop-in point, a transition point and a scoop-out point are taken as control points, N i,k Is a basis function of a spline curve, k is the number of curves, where k=3 is set; the spline basis function equation can be solved by a recursive formula:
interpolation is carried out on four-element information at the tail end of the mechanical arm by using a square algorithm, the rotation matrix obtained through calculation in the fourth step is firstly required to be converted into four-element, and a conversion formula is as follows:
wherein T is a rotation matrix, and q is a four-element obtained by conversion; assume that four elements of a scoop-in point, a transition point and a scoop-out point are q s ,q i ,q e The duration t, the square interpolation formula is as follows:
finally, combining the space position and the four elements by taking the time node as a reference to obtain complete track information of the tail end of the mechanical arm; and the motion information is converted into the motion information under the joint space through inverse kinematics, and the motion information is sent to the mechanical arm control module to complete the dish-making task.
The invention has the beneficial effects that: according to the invention, the sensor is used for collecting depth data on the surface of dishes, the dishes are identified through the DGG point cloud segmentation network, different dishes are selected according to the quantity of the dishes, pose information of the dishes is estimated, and finally, the running track of the mechanical arm is obtained through a track planning algorithm. The method can accurately estimate proper dish ordering points, rapidly plan corresponding dish ordering tracks and complete dish ordering tasks.
Drawings
Fig. 1: the local point cloud characteristic extraction module is used for extracting local point cloud characteristics;
fig. 2: the network structure is divided for DGG point cloud of the invention.
Specific implementation steps the technical scheme of the present invention is further specifically described below through specific embodiments and with reference to the accompanying drawings.
Example 1:
step one: constructing a DGG point cloud segmentation network shown in a second diagram; firstly, constructing a local point cloud characteristic extraction module shown in a first graph, and defining a directed graph G= (v, epsilon) to represent the geometric structure of the local point cloud, wherein v, epsilon respectively represent vertexes and edges. Randomly selecting a point from the input point cloud as the center of the graph G and marking as p i From R by K nearest neighbor algorithm 3×3 Space yields k=40 nearest neighbors { p } i1 ,p i2 ,...,p ik Defining the position characteristics of the local point cloud as
P i,m ＝{p i1,m -p i,m ,p i2,m -p i,m ,...,p ik,m -p i }
Wherein p is ik,m An mth dimension feature representing a kth point in the neighborhood;
defining the density of local point clouds as
Where k represents the number of neighborhood points, F m (j) Representative point p i Where m= {1,2,3}; and p is combined with i,m As a position feature P i,m Is a weight function of (2); at the same time, the shape characteristics of the local point cloud are defined as
Wherein the method comprises the steps ofRepresenting the inner product calculation between two vectors, < ->Is an edge within graph G; handle P i,m And S is i,m Further feature extraction is realized through the MLP (64, 128) structures of the two layers respectively, the problem of the disorder of the point cloud is solved through the maximum pooling layer, and finally, feature aggregation is carried out to obtain the integral feature of the local point cloud.
Then inputting the features of the local point cloud into a three-layer MLP (128, 256, 256) network with residual connections, and then lifting the feature dimensions to 1024; in order to train multiple dish data at the same time, one-dimensional dish variety vectors are added, and the maximum pooling function is carried out, and then the three layers of common MLPs (1024, 512, num) are adopted, wherein the number of convolution output channels num of the last layer is the total number of the segmentation labels. The step size of all convolution operations in DGG network is 1, the activation function uses the leak Relu, and Dropout is added before the output layer to avoid over-fitting during training, and the inactivation rate is set to 0.5.
Step two: acquiring point cloud data of the surfaces of various dishes by using a depth camera, wherein the camera is fixed at the tail end of a mechanical arm, and adopts a calibration mode that eyes are on hands; marking data by using a Semantic-segment-Editor tool, which is divided into three tags: a spoon inlet area, a spoon outlet area and an irrelevant area; dividing the model into a training set and a testing set according to the proportion of 9:1, inputting the training set and the testing set into the DGG point cloud segmentation model constructed in the step one, and setting training parameters: iteration times 300, batch size 32, initial learning rate 0.001, and adjusting the weight parameters of the network model by using an SGD optimizer; and saving the network weight parameters when the loss value is the lowest.
Step three: transplanting the segmentation model built in the first step to an ROS development platform, loading the network weight parameters stored in the second step, predicting target points of dish point cloud data acquired in real time in the ROS, and screening the prediction results of the network; and eliminating the situation that the number of the predicted result areas is small or the areas are separated. The prediction result is issued through the ROS message mechanism.
Step four: the mechanical arm subscribes to topics released in the third step to obtain a prediction result, and the maximum depth extracted by the end effector is calculated by combining the dish surface depth information to obtain the attitude information of the end effector of the mechanical arm in Cartesian space; the specific calculation process is as follows:
a) Assuming that the mechanical arm end effector is a universal dish-beating spoon on the market, providing that the center of a spoon opening is an origin of an end coordinate system, enabling the vertical spoon surface to face upwards to be in the Z-axis direction, enabling the direction pointing to a spoon handle to be taken as a Y-axis, and enabling the X-axis direction to be determined according to a right-hand rule;
b) Taking the central point of the predicted area of the network as the scoop-in point (x s ,y s ,z s ) And spoon out point (x) e ,y e ,z e ) If the distance between the spoon inlet point or the spoon outlet point and the boundary of the dish basin is smaller than the radius r of the dish-beating spoon spoon If the point cloud is=0.05m, executing the third step, and predicting the point cloud again; the spoon inlet point is used as a starting point of the dish-beating spoon, the spoon outlet point is used as a advancing direction of the dish-beating spoon, and the advancing distance is determined by different kinds of dish densities; suppose the quality of dishes on the full spoon is G dish According to the dish density ρ dish The volume of the obtained dishesCombining the point cloud depth information, taking a spoon inlet point as a starting point and a spoon outlet point as a forward direction, calculating and dividing the volume into 2V by using a convex hull approximation algorithm dish The point cloud area of (2) is used as a dish-making area, and the end coordinates of the dish-making area are used as new spoon outlet points +.>Selecting (x) s ,y s ,z s ) And->Is used as a transition point (x i ,y i ) Its depth value z i The calculation formula of (2) is as follows:
wherein n is 1 Is the number of points between the spoon entering point and the transition point in the dish beating area, n 2 Is the number of points between the transition point and the spoon outlet point; if the calculated depth value exceeds the depth of the vegetable basin, returning to the step three, and predicting again by using a DGG network.
c) After the space position of the dish-beating spoon road points is obtained, the gesture of each point needs to be calculated; setting according to the coordinate system in a), wherein the positive direction of the z-axis represents the advancing direction of the spoon; for the quantity of dishes, two different dishes beating actions are designed, one is to rotate while advancing, and the other is to push and rotate in the advancing process, wherein the difference is that the gesture of the waypoint is changed; for the first, setting the z-axis of the spoon inlet point to a transition point, and setting the z-axis of the transition point to the spoon outlet point, wherein the z-axis of the spoon outlet point is vertically upwards; the second kind of spoon-in point z axis is parallel to the world coordinate system and points to the spoon-out point direction, the transition point has the same gesture as the spoon-in point, and represents pushing action from the spoon-in point to the transition point, and the spoon-out point z axis is also vertically upwards; all the poses are rotated based on a camera-based coordinate system, for example, the pose of the spoon-in point in the first case, and the euler angle is:
wherein θ is z The angle between the spoon handle and the ground is regulated and is generally controlled to be between 35 and 50 degrees; the pose calculation method of other points is similar.
Because the camera is fixed at the tail end of the mechanical arm, the coordinate system of the camera can be changed along with the movement of the mechanical arm, and all the pose information of all the road points are converted into the world coordinate system; assume that the spatial position of a certain waypoint in the camera coordinate system is (x 0 ,y 0 ,z 0 ) Euler angle is (θ x ,θ y ,θ z ) The method comprises the steps of carrying out a first treatment on the surface of the The camera is positioned in the world coordinate system at (x c ,y c ,z c ) The rotation matrix isFirst converting Euler angles into rotation matrix form +.>The formula is
Further obtain pose information under world coordinate system as
Step five, obtaining pose information based on a world coordinate system according to the step four, and obtaining track information of an end effector of the mechanical arm by combining a cubic B spline interpolation algorithm and a square interpolation algorithm; the specific calculation process is as follows:
interpolation of the spatial position of the end of the arm was performed using cubic B-splines, the formula defined as follows:
S(u)＝∑P i N i,k (u)
wherein P is i Is a control point of a spline curve, wherein a scoop-in point, a transition point and a scoop-out point are taken as control points, N i,k Is a basis function of a spline curve, k is the number of curves, where k=3 is set; the spline basis function equation can be solved by a recursive formula:
interpolation is carried out on four-element information at the tail end of the mechanical arm by using a square algorithm, the rotation matrix obtained through calculation in the fourth step is firstly required to be converted into four-element, and a conversion formula is as follows:
wherein T is a rotation matrix, and q is a four-element obtained by conversion; assume that four elements of a scoop-in point, a transition point and a scoop-out point are q s ,q i ,q e The duration t, the square interpolation formula is as follows:
finally, combining the space position and the four elements by taking the time node as a reference to obtain complete track information of the tail end of the mechanical arm; and finally, converting motion information under the joint space through inverse kinematics, and sending the motion information to a mechanical arm control module to complete a dish-making task.
Claims (1)
1. An automatic vegetable-beating method of a mechanical arm based on a DGG point cloud segmentation network is characterized by comprising the following steps of: the method comprises the following steps:
step one: building a DGG point cloud segmentation network; constructing a feature extraction module of the local point cloud, and defining a directed graph G= (v, epsilon) to characterizeA geometry of a local point cloud, wherein v, ε represent the vertex and edge, respectively; randomly selecting a point from the input local point cloud as the center of the directed graph G, and recording as p i From R by K nearest neighbor algorithm 3×3 Space gets k neighboring points { p } i1 ,p i2 ,...,p ik Defining the position characteristics of the local point cloud as:
P i,m ＝{p i1,m -p i,m ,p i2,m -p i,m ,…,p ik,m -p i,m }
wherein p is ik,m An mth dimension feature representing a kth point in the neighborhood;
the degree of density of the local point cloud is defined as:
where k represents the number of neighborhood points, F m (j) Is the neighbor point p ij And to characterize ρ in the m-th dimension of i,m As a position feature P i,m Is a weight function of (2);
the shape characteristics of the local point cloud are defined as:
wherein the method comprises the steps ofRepresenting the inner product calculation between two vectors, < ->And->Are edges in the directed graph G; handle P i,m And S is i,m Further feature extraction is realized through the MLP structures of the two layers respectively, the problem of the disorder of the point cloud is solved through the maximum pooling layer,the integrated characteristic is taken as the integral characteristic of the local point cloud after characteristic aggregation;
inputting the integral characteristics of the local point cloud into a three-layer MLP network with residual connection, and improving the characteristic dimension to 1024; in order to train various dish data at the same time, adding one-dimensional dish variety vectors, and passing through a common three-layer MLP network after a maximum pooling function, wherein the number num of convolution output channels of the last layer is the total number of segmentation labels; the step length of all convolution operations in the DGG point cloud segmentation network is 1, the activation function uses a leak Relu, and in order to avoid over fitting in the training process, dropout is added in front of an output layer, and the inactivation rate is set to be 0.5;
step two: acquiring point cloud data of the surface of the dish by using a depth camera, wherein the camera is fixed at the tail end of the mechanical arm; marking data by using a Semantic-segment-Editor tool is divided into three labels: a spoon inlet area, a spoon outlet area and an irrelevant area; the data comprises a training set and a testing set, the training set and the testing set are input into the DGG point cloud segmentation network constructed in the first step, and training parameters are set: iteration times 300, batch size 32, initial learning rate 0.001, and adjusting the weight parameters of the network model by using an SGD optimizer; saving the network model weight parameters when the loss value is the lowest;
step three: transplanting the DGG point cloud segmentation network built in the first step to an ROS development platform, loading the network model weight parameters stored in the second step, predicting the point cloud data of the dish surface acquired in real time in the ROS, and screening the prediction result of the network; removing the situation that the number of the predicted result areas is small or the areas are separated; issuing a prediction result through an ROS message mechanism;
step four: the mechanical arm subscribes to topics released in the third step to obtain a prediction result, and the maximum depth of digging of the dish spoon at the tail end of the mechanical arm is calculated by combining the dish surface depth information to obtain the attitude information of the dish spoon at the tail end of the mechanical arm under a world coordinate system; the calculation process is as follows:
a) The center of the spoon opening is defined as the origin of a terminal coordinate system, the upward direction of the vertical spoon is defined as the Z-axis direction, the direction pointing to the spoon handle is defined as the Y-axis, and the X-axis direction can be determined according to the right-hand rule;
b) Taking the central point of the predicted result area of the network as the scoop-in point (x s ,y s ,z s ) And spoon out point (x) e ,y e ,z e ) If the distance between the spoon inlet point or the spoon outlet point and the boundary of the dish basin is smaller than the radius r of the dish-beating spoon spoon Executing the third step, and predicting the point cloud again; the spoon inlet point is used as a starting point of the dish-beating spoon, the spoon outlet point is used as a advancing direction of the dish-beating spoon, and the advancing distance is determined by different kinds of dish densities; dish with full spoon has quality of G dish According to the dish density ρ dish The volume of the obtained dishesCombining the point cloud depth information, taking a spoon inlet point as a starting point and a spoon outlet point as a forward direction, calculating and dividing the volume into 2V by using a convex hull approximation algorithm dish The point cloud area of (1) is used as a dish-making area, and the end coordinates of the dish-making area are used as new spoon outlet pointsSelecting (x) s ,y s ,z s ) And->Is used as a transition point (x i ,y i ) Its depth value z i The calculation formula of (2) is as follows:
wherein n is 1 Is the number of points between the spoon entering point and the transition point in the dish beating area, n 2 Is the number of points between the transition point and the spoon outlet point; if the calculated depth value exceeds the depth of the vegetable basin, returning to the third step, and predicting again by using a DGG point cloud segmentation network;
c) After the spatial position of the vegetable-beating spoon waypoints at the tail end of the mechanical arm is obtained, the gesture of each waypoint needs to be calculated; according to the coordinate system setting in a), the positive direction of the z-axis represents the advancing direction of the dish-beating spoon at the tail end of the mechanical arm; two different dish-beating actions are designed, wherein the first is to rotate and advance at the same time, and the second is to push and rotate in the advancing process, and the difference is that the gesture of the road point is changed;
firstly, setting a z-axis of a spoon inlet point to a transition point, wherein the z-axis of the transition point points to a spoon outlet point, and the z-axis of the spoon outlet point is vertically upwards;
the second kind of spoon-in point z axis is parallel to the world coordinate system and points to the spoon-out point direction, the transition point has the same gesture as the spoon-in point, and represents pushing action from the spoon-in point to the transition point, and the spoon-out point z axis is also vertically upwards; all the gestures are rotated based on a camera coordinate system, wherein in the first case, the gesture of the spoon-in point is as follows:
wherein θ is z The angle between the spoon handle and the ground is adjusted and controlled between 35 degrees and 50 degrees during the vegetable beating action;
because the camera is fixed at the tail end of the mechanical arm, the coordinate system of the camera can be changed along with the movement of the mechanical arm, and the pose information of all road points is converted into the world coordinate system; assume that the spatial position of a certain waypoint in the camera coordinate system is (x 0 ,y 0 ,z 0 ) Euler angle is (θ x ,θ y ,θ z ) The method comprises the steps of carrying out a first treatment on the surface of the The camera is positioned in the world coordinate system at (x c ,y c ,z c ) The rotation matrix isFirst converting Euler angles into rotation matrix form +.>The formula is:
and further obtaining pose information under a world coordinate system as follows:
step five, obtaining track information of the dish-taking ladle at the tail end of the mechanical arm according to the pose information of the dish-taking ladle at the tail end of the mechanical arm relative to the world coordinate system, which is obtained in the step four, by combining a cubic B spline interpolation algorithm and a square interpolation algorithm; the calculation process is as follows:
interpolation is carried out on the space position of the dish spoon at the tail end of the mechanical arm by using a cubic B spline, and the definition formula is as follows:
S(u)＝∑P i N i,k (u)
wherein P is i Is a control point of a spline curve, wherein a scoop-in point, a transition point and a scoop-out point are taken as control points, N i,k Is a basis function of spline curves, k is the number of times of the curves, and k=3 is set; the spline basis function equation can be solved by a recursive formula:
interpolation is carried out on quaternion information of the dish-taking spoon at the tail end of the mechanical arm by using a square algorithm, the rotation matrix calculated in the fourth step is converted into quaternion, and a conversion formula is as follows:
wherein T is a rotation matrix, and q is a quaternion obtained by conversion; set quaternion of scoop in point, transition point and scoop out point as q s ,q i ,q e The duration t, the square interpolation formula is as follows:
finally, combining the space position and the quaternion by taking the time node as a reference to obtain the complete track information of the dish spoon at the tail end of the mechanical arm; and the motion information is converted into the motion information under the joint space through inverse kinematics, and the motion information is sent to the mechanical arm control module to complete the dish-making task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111427395.XA CN114310872B (en) | 2021-11-29 | 2021-11-29 | Automatic vegetable-beating method for mechanical arm based on DGG point cloud segmentation network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111427395.XA CN114310872B (en) | 2021-11-29 | 2021-11-29 | Automatic vegetable-beating method for mechanical arm based on DGG point cloud segmentation network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114310872A CN114310872A (en) | 2022-04-12 |
| CN114310872B true CN114310872B (en) | 2023-08-22 |
Family
ID=81047623
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111427395.XA Active CN114310872B (en) | 2021-11-29 | 2021-11-29 | Automatic vegetable-beating method for mechanical arm based on DGG point cloud segmentation network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114310872B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115424701B (en) * | 2022-11-07 | 2023-07-21 | 杭州柳叶刀机器人有限公司 | Bone surface follow-up technology for optimal path planning |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102713671A (en) * | 2009-12-11 | 2012-10-03 | 株式会社拓普康 | Point group data processing device, point group data processing method, and point group data processing program |
| CN110428506A (en) * | 2019-08-09 | 2019-11-08 | 成都景中教育软件有限公司 | A kind of dynamic geometry 3-D graphic cutting implementation method based on parameter |
| CN112560918A (en) * | 2020-12-07 | 2021-03-26 | 杭州电子科技大学 | Dish identification method based on improved YOLO v3 |
| CN112809679A (en) * | 2021-01-25 | 2021-05-18 | 清华大学深圳国际研究生院 | Method and device for grabbing deformable object and computer readable storage medium |
| CN113066016A (en) * | 2021-02-18 | 2021-07-02 | 北京大学 | Three-dimensional dynamic point cloud repairing method based on image signal processing |
| CN113449736A (en) * | 2021-01-14 | 2021-09-28 | 浙江工业大学 | Photogrammetry point cloud semantic segmentation method based on deep learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109784333B (en) * | 2019-01-22 | 2021-09-28 | 中国科学院自动化研究所 | Three-dimensional target detection method and system based on point cloud weighted channel characteristics |
-
2021
- 2021-11-29 CN CN202111427395.XA patent/CN114310872B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102713671A (en) * | 2009-12-11 | 2012-10-03 | 株式会社拓普康 | Point group data processing device, point group data processing method, and point group data processing program |
| CN110428506A (en) * | 2019-08-09 | 2019-11-08 | 成都景中教育软件有限公司 | A kind of dynamic geometry 3-D graphic cutting implementation method based on parameter |
| CN112560918A (en) * | 2020-12-07 | 2021-03-26 | 杭州电子科技大学 | Dish identification method based on improved YOLO v3 |
| CN113449736A (en) * | 2021-01-14 | 2021-09-28 | 浙江工业大学 | Photogrammetry point cloud semantic segmentation method based on deep learning |
| CN112809679A (en) * | 2021-01-25 | 2021-05-18 | 清华大学深圳国际研究生院 | Method and device for grabbing deformable object and computer readable storage medium |
| CN113066016A (en) * | 2021-02-18 | 2021-07-02 | 北京大学 | Three-dimensional dynamic point cloud repairing method based on image signal processing |
Non-Patent Citations (1)
| Title |
|---|
| dynamic graph cnn for learning on point clouds;Yue Wang et al;《ACM TRANS. GRAPH》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114310872A (en) | 2022-04-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110703747B (en) | Robot autonomous exploration method based on simplified generalized Voronoi diagram | |
| Wu et al. | Plant phenotyping by deep-learning-based planner for multi-robots | |
| CN103778635B (en) | For the method and apparatus processing data | |
| CN109782779B (en) | AUV path planning method in ocean current environment based on population hyperheuristic algorithm | |
| Hamer et al. | An object-dependent hand pose prior from sparse training data | |
| CN110298886B (en) | Dexterous hand grabbing planning method based on four-stage convolutional neural network | |
| CN111260649B (en) | Close-range mechanical arm sensing and calibrating method | |
| CN113110513A (en) | ROS-based household arrangement mobile robot | |
| CN109108978B (en) | Three-degree-of-freedom space manipulator motion planning method based on learning generalization mechanism | |
| Tang et al. | Learning collaborative pushing and grasping policies in dense clutter | |
| CN114237235B (en) | Mobile robot obstacle avoidance method based on deep reinforcement learning | |
| CN111872934A (en) | Mechanical arm control method and system based on hidden semi-Markov model | |
| Zhang et al. | Sim2real learning of obstacle avoidance for robotic manipulators in uncertain environments | |
| Kim et al. | UAV-UGV cooperative 3D environmental mapping | |
| CN114310872B (en) | Automatic vegetable-beating method for mechanical arm based on DGG point cloud segmentation network | |
| Hueser et al. | Learning of demonstrated grasping skills by stereoscopic tracking of human head configuration | |
| CN113034575A (en) | Model construction method, pose estimation method and object picking device | |
| Menon et al. | NBV-SC: Next best view planning based on shape completion for fruit mapping and reconstruction | |
| CN105204511B (en) | A kind of decision-making technique of object autonomous | |
| CN118201742A (en) | Multi-robot coordination using a graph neural network | |
| CN114186112B (en) | Robot navigation method based on Bayesian optimization multiple information gain exploration strategy | |
| Breyer et al. | Closed-loop next-best-view planning for target-driven grasping | |
| Schaub et al. | 6-DoF grasp detection for unknown objects | |
| KR101107735B1 (en) | Camera pose decision method | |
| Sujan et al. | Visually built task models for robot teams in unstructured environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |