CN117456368A - Fruit and vegetable identification picking method, system and device - Google Patents
Fruit and vegetable identification picking method, system and device Download PDFInfo
- Publication number
- CN117456368A CN117456368A CN202311779603.1A CN202311779603A CN117456368A CN 117456368 A CN117456368 A CN 117456368A CN 202311779603 A CN202311779603 A CN 202311779603A CN 117456368 A CN117456368 A CN 117456368A
- Authority
- CN
- China
- Prior art keywords
- image
- fruit
- picking
- node
- representing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 235000012055 fruits and vegetables Nutrition 0.000 title claims abstract description 33
- 235000013399 edible fruits Nutrition 0.000 claims abstract description 104
- 238000000547 structure data Methods 0.000 claims abstract description 20
- 230000011218 segmentation Effects 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 8
- 239000013598 vector Substances 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 11
- 230000003287 optical effect Effects 0.000 claims description 8
- 238000007781 pre-processing Methods 0.000 claims description 4
- 230000004927 fusion Effects 0.000 claims description 3
- 235000013311 vegetables Nutrition 0.000 claims 1
- 238000013135 deep learning Methods 0.000 abstract description 4
- 230000009471 action Effects 0.000 abstract description 2
- 230000000007 visual effect Effects 0.000 abstract description 2
- 239000011159 matrix material Substances 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 235000004936 Bromus mango Nutrition 0.000 description 2
- 235000014826 Mangifera indica Nutrition 0.000 description 2
- 235000009184 Spondias indica Nutrition 0.000 description 2
- 240000006365 Vitis vinifera Species 0.000 description 2
- 235000014787 Vitis vinifera Nutrition 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000000149 penetrating effect Effects 0.000 description 2
- 241001093152 Mangifera Species 0.000 description 1
- 240000007228 Mangifera indica Species 0.000 description 1
- 235000009754 Vitis X bourquina Nutrition 0.000 description 1
- 235000012333 Vitis X labruscana Nutrition 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 208000002161 echolalia Diseases 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000010287 polarization Effects 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/188—Vegetation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/17—Terrestrial scenes taken from planes or by drones
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Remote Sensing (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of deep learning, and discloses a fruit and vegetable identification picking method, a system and a device, wherein the fruit and vegetable identification picking method comprises the following steps: collecting a first image and a second image of a picking area; processing the first image and the second image to obtain image structure data; inputting the graph structure data into a recognition model, and outputting coordinate values representing predicted positions of fruits corresponding to the nodes by the recognition model; the picking robot moves to the position of the fruit according to the predicted position of the fruit, and then a third image is acquired by an image acquisition unit on the picking robot; performing example segmentation on the third image through the contour recognition positioning model to determine picking points of fruits, and then controlling a picking robot to cut the picking points to finish picking the fruits; the invention ensures that the robot does not need to traverse the region and the visual angle when picking the fruit individuals, can save a great deal of recognition and action time and improve the picking speed.
Description
Technical Field
The invention relates to the technical field of deep learning, in particular to a fruit and vegetable identification and picking method, a fruit and vegetable identification and picking system and a fruit and vegetable identification and picking device.
Background
Machine learning and deep learning can enable robots to understand environments, pick crops and monitor crop health conditions, and for the problems of automatic picking in unstructured agricultural environments, mainly focus on fruit detection and picking point positioning, for example, chinese patent application publication No. CN106441098A discloses an identification positioning method for picking fruits and vegetables, which positions picking points based on a binocular vision system and a geometric method, and one obvious problem is poor performance in the case of fruit covering; chen Pengfei in Mask R-CNN-based mango instance segmentation and picking point detection study and implementation, a method for identifying mangoes and determining picking points based on Mask R-CNN instance segmentation performance is provided, and recognition accuracy of fruit covering in an image is improved through deep learning based on a convolutional neural network; however, the general method is based on optical pictures, and only surface features can be identified, so that the method is only suitable for identifying close-range pictures of the operation direction of a robot, the robot generally adopts a region traversing method to avoid omission, and if the robot wants to further find fruits hidden in the robot, the robot needs traversing with a larger range and view angle, and the picking speed is slower.
Disclosure of Invention
The invention provides a fruit and vegetable identification picking method, system and device, which solve the technical problem of slower picking speed in the related technology.
The invention provides a fruit and vegetable identification picking method, which comprises the following steps:
step 101, collecting a first image and a second image of a picking area;
the first image is a SAR image; the second image is an optical image or a SAR image;
step 102, processing the first image and the second image to obtain graph structure data, wherein the graph structure data comprises nodes, and one node is in data connection with one pixel of the first image or the second image; the conditions for establishing edges for two nodes include: the two node data associating two adjacent pixels on the first image; the two node data associating two adjacent pixels on the second image; one of the two nodes is connected with the pixel on the first image, the other is connected with the pixel on the second image, and the positions of the two pixels on the image coordinate systems of the first image and the second image are the same;
step 103, inputting the graph structure data into a recognition model, and outputting coordinate values representing predicted positions of fruits corresponding to the nodes by the recognition model; if the coordinate value of the predicted position of the fruit corresponding to the node is 0, the pixel of the first image representing the node data connection does not correspond to the fruit;
104, moving the picking robot to the position of the fruit according to the predicted position of the fruit, and then acquiring a third image which is a visible light image through an image acquisition unit on the picking robot;
and 105, performing example segmentation on the third image through the contour recognition positioning model to determine picking points of the fruits, and controlling the picking robot to cut the picking points to finish picking the fruits.
Further, the second image is a SAR image, acquired in a different frequency band from the first image, and the frequency of the acquired frequency band of the second image is lower than the acquired frequency band of the first image.
Further, the first image and the second image are superimposed on the same image coordinate system, so that the superimposed two pixels of the first image and the second image are the corresponding two pixels.
Further, the identification model comprises a first hidden layer and a fully connected layer;
the calculation formula of the first hidden layer is as follows:
；
；
；
；
；
；
；
wherein the method comprises the steps ofFor initially hiding features->Node characteristics representing the v-th node, +.>Representing fusion vectors +.>、、/>、/>Representing the first, second, third and fourth intermediate features of the v-th node at the t-th time step, respectively,/->And->Output characteristics of the v-th node at the t-th time step and t-1 time step are respectively represented,/->Representing the output characteristics of the jth node at t-1 time steps,/for>Representing a set of nodes directly connected by edges to a v-th node, the set comprising the v-th node,/the v-th node>Representing dot product->Representing vector concatenation->、/>、/>、/>、/>、/>、/>、/>Respectively represent the first, second, third, fourth, fifth, sixth, seventh and eighth weight parameters, ++>、/>、/>、/>Respectively representing the first, second, third and fourth bias parameters.
Further, the calculation formula of the full connection layer is as follows:
；
wherein the method comprises the steps ofA coordinate vector representing the predicted position of the fruit corresponding to the y-th node, the p-th component of the coordinate vector representing the p-th coordinate value of the predicted position of the fruit corresponding to the y-th node, ">Representing the output characteristics of the y-th node at the S-th time step, the y-th node establishing a data connection with the pixels of the first image,/a>Represents a ninth weight parameter,>representing a fifth bias parameter.
Further, the coordinate value of the predicted position of the fruit corresponding to the node output representing the recognition model is a three-dimensional coordinate in the world coordinate system, one node corresponds to three coordinate values, and the world coordinate system takes a certain point on the ground of the picking area as an origin.
Further, a calculation formula for calculating the loss value of the identification model with the coordinate values in the world coordinate system is as follows:
；
wherein Loss represents a Loss value, A represents the total number of nodes corresponding to which the coordinate value output by the recognition model is not 0, B represents the total number of actual fruits in the picking area,the g coordinate value of the predicted position of the fruit corresponding to the kth node,/for>A g-th coordinate value indicating a real fruit position corresponding to a predicted position of the fruit corresponding to the k-th node; />The g coordinate value of the d-th real fruit, < > is shown>And g coordinate values representing the predicted positions corresponding to the d-th real fruits.
Further, the contour recognition positioning model is Mask R-CNN.
The invention provides a fruit and vegetable identifying and picking system, which is used for executing the fruit and vegetable identifying and picking method, and comprises the following steps:
the first image acquisition module acquires a first image and a second image of the picking area;
the image preprocessing module is used for processing the first image and the second image to obtain image structure data, wherein the image structure data comprises nodes, and one node is in data connection with one pixel of the first image or the second image;
the target recognition module inputs the graph structure data into a recognition model, and the recognition model outputs coordinate values representing predicted positions of fruits corresponding to the nodes; if the coordinate value of the predicted position of the fruit corresponding to the node is 0, the pixel of the first image representing the node data connection does not correspond to the fruit;
the movement control module is used for enabling the picking robot to move to the position where the fruit is located according to the predicted position of the fruit;
the second image acquisition module acquires a third image on the front surface of the picking robot, wherein the third image is a visible light image;
the example segmentation module is used for carrying out example segmentation on the third image through the contour recognition positioning model to determine picking points of fruits;
and the picking control module is used for controlling the picking robot to cut the picking points to finish picking fruits.
The invention provides a fruit and vegetable identifying and picking device, which comprises:
a memory for storing machine executable instructions, an identification model, and a contour identification positioning model;
a processor for executing machine executable instructions stored in the memory, which when executed, is capable of performing the steps of a fruit and vegetable identification picking method as described above
The invention has the beneficial effects that: according to the invention, the fruit image features are obtained by penetrating through the leaf blade based on the micro synthetic aperture radar platform, and the fruit distribution prediction of the whole picking area is positioned by extracting the fused captured fruit image features based on the second image of the captured surface features, so that global guidance is provided for the picking of the robot, the robot does not need to traverse the area and the visual angle when picking the fruit individual, a large amount of recognition and action time is saved, the picking speed is improved, and the robot path planning method can be used for robot path planning.
Drawings
FIG. 1 is a flow chart of a fruit and vegetable identification picking method of the present invention;
fig. 2 is a schematic block diagram of a fruit and vegetable identification picking system according to the present invention.
In the figure: a first image acquisition module 201, an image preprocessing module 202, a target identification module 203, a movement control module 204, a second image acquisition module 205, an instance segmentation module 206 and a picking control module 207.
Detailed Description
The subject matter described herein will now be discussed with reference to example embodiments. It is to be understood that these embodiments are merely discussed so that those skilled in the art may better understand and implement the subject matter described herein and that changes may be made in the function and arrangement of the elements discussed without departing from the scope of the disclosure herein. Various examples may omit, replace, or add various procedures or components as desired. In addition, features described with respect to some examples may be combined in other examples as well.
As shown in fig. 1, the fruit and vegetable identification picking method comprises the following steps:
step 101, collecting a first image and a second image of a picking area;
the first image is an SAR image, and can be acquired by taking a miniature synthetic aperture radar based on an unmanned aerial vehicle as a platform as acquisition equipment; has the characteristics of close distance and high resolution;
the second image is an optical image or a SAR image, and if the second image is a SAR image, the same unmanned aerial vehicle platform as the first image is used, but it is acquired at a different frequency band from the first image, and the frequency of the acquisition frequency band of the second image is lower than that of the first image.
In one embodiment of the invention, the optical image is a laser radar image, and can be acquired by taking a laser radar based on an unmanned plane as a platform as acquisition equipment;
if the SAR image is used as the second image and the first image, the parameters corresponding to the pixels of the second image and the first image are the same, so that the consistency processing is not needed;
if the SAR image is used as the first image and the optical image is used as the second image, the parameters corresponding to the two pixels are different, and an unification process, such as conversion into a depth map, is required, and the depth value under the camera coordinate system is used as the parameter of the pixel.
Step 102, processing the first image and the second image to obtain graph structure data, wherein the graph structure data comprises nodes, and one node is in data connection with one pixel of the first image or the second image; the conditions for establishing edges for two nodes include: the two node data associating two adjacent pixels on the first image; the two node data associating two adjacent pixels on the second image; one of the two nodes is connected with the pixel on the first image, the other is connected with the pixel on the second image, and the positions of the two pixels on the image coordinate systems of the first image and the second image are the same;
the first image and the second image are superimposed on the same image coordinate system, so that the superimposed two pixels of the first image and the second image are the corresponding two pixels;
in one embodiment of the invention, the SAR image is taken as a second image and a first image, which are registered to the same space by registration prior to step 102.
In an embodiment of the invention, the second image is an optical image, typically having a resolution greater than the SAR image, and the pixels of the optical image may be subjected to a resolution reduction process.
Step 103, inputting the graph structure data into a recognition model, and outputting coordinate values representing predicted positions of fruits corresponding to the nodes by the recognition model; if the coordinate value of the predicted position of the fruit corresponding to the node is 0, the pixel of the first image representing the node data connection does not correspond to the fruit;
the recognition model comprises a first hidden layer and a full connection layer;
in one embodiment of the present invention, the calculation formula of the first hidden layer is as follows:
wherein the method comprises the steps ofFor initially hiding features->Node characteristics representing the v-th node, +.>Representing fusion vectors +.>、、/>、/>Representing the first, second, third and fourth intermediate features of the v-th node at the t-th time step, respectively,/->And->Output characteristics of the v-th node at the t-th time step and t-1 time step are respectively represented,/->Representing the output characteristics of the jth node at t-1 time steps,/for>Representing a set of nodes directly connected by edges to a v-th node, the set comprising the v-th node,/the v-th node>Representing dot product->Representing vector concatenation->、/>、/>、/>、/>、/>、/>、/>Respectively represent the first, second, third, fourth, fifth, sixth, seventh and eighth weight parameters, ++>、/>、/>、/>Respectively representing first, second, third and fourth bias parameters;
the nth component of the node characteristic of the nth node represents the nth attribute value of the pixel to which the data of the nth node is connected; the properties of the pixels of the SAR image include phase, amplitude, etc.
S is a super parameter, and the default value is 3.
In one embodiment of the present invention, the calculation formula of the first hidden layer is as follows:
wherein X represents a node feature matrix, the ith row vector of which represents node features of the ith node,representing an output feature matrix whose ith row vector represents the output feature of the ith node,/>Representing the sum of the adjacency matrix and the identity matrix, +.>Representation->Degree matrix of->Representing the tenth weight parameter, the element value of the nth row and the mth column of the adjacent matrix being 1 representing the mth and nth sectionsThere are edges between the points, otherwise the element value is 0.
In one embodiment of the present invention, the calculation formula of the full connection layer is as follows:
a coordinate vector representing the predicted position of the fruit corresponding to the y-th node, the p-th component of the coordinate vector representing the p-th coordinate value of the predicted position of the fruit corresponding to the y-th node, ">Representing the output characteristics of the y-th node at the S-th time step, the y-th node establishing a data connection with the pixels of the first image,/a>Represents a ninth weight parameter,>representing a fifth bias parameter.
The foregoingRepresenting sigmoid function->Representing a hyperbolic tangent function.
In one embodiment of the present invention, the coordinate value of the predicted position of the fruit corresponding to the node output representing the recognition model is a three-dimensional coordinate in the world coordinate system, one node corresponds to three coordinate values, and the world coordinate system takes a certain point on the ground of the picking area as the origin.
For the training process, for the output of the recognition model, it is feasible to independently measure the actual position of the fruit, but the cost is high; as a method for reducing the cost, the data for comparison with the output of the recognition model may be derived from the picking position of the robot arm at the time of actual picking as the actual fruit position, and the center point of the holding space of the robot arm holding the fruit at the time as the actual fruit centroid position. The data acquisition is used as the addition of the acquisition fruits, so that the cost is reduced compared with the data acquisition alone, and enough data which can meet the training of the recognition model can be acquired; it should be noted that the manipulator may be controlled automatically or manually or a combination of both.
For the training process, the calculation formula for calculating the loss value by the coordinate values under the world coordinate system is as follows:
wherein A represents the total number of nodes corresponding to which the coordinate value output by the recognition model is not 0, B represents the total number of actual fruits in the picking area,the g coordinate value of the predicted position of the fruit corresponding to the kth node,/for>A g-th coordinate value indicating a real fruit position corresponding to a predicted position of the fruit corresponding to the k-th node; />The g coordinate value of the d-th real fruit, < > is shown>A g-th coordinate value representing a predicted position corresponding to the d-th real fruit;
if the fruit exists in the spherical space with the radius L and the predicted position of the fruit corresponding to the kth node is taken as the center, the actual position of the actual fruit with the centroid closest to the predicted position of the R-th fruit is taken as the actual fruit position corresponding to the predicted position of the fruit corresponding to the kth node, otherwise, the definition is madeIs 0;
if the predicted position exists in the spherical space with the radius L and the true position of the d-th real fruit is taken as the predicted position corresponding to the d-th real fruit, the predicted position closest to the d-th real fruit is taken as the predicted position corresponding to the d-th real fruit, otherwise, the definition is madeIs 0.
The loss calculation method can unify the loss of the real fruits and the loss of the position deviation.
104, moving the picking robot to the position of the fruit according to the predicted position of the fruit, and then acquiring a third image which is a visible light image through an image acquisition unit on the picking robot;
and 105, performing example segmentation on the third image through the contour recognition positioning model to determine picking points of the fruits, and controlling the picking robot to cut the picking points to finish picking the fruits.
The contour recognition positioning model can adopt Mask R-CNN mentioned in the background art as another common deep learning model YOLO model, and can also be used for achieving the same purpose.
The fruit and vegetable identification picking method in one embodiment of the present invention further includes a step for planning a moving path of a plurality of picking robots, specifically, planning by combining coordinate values of the obtained predicted positions of all fruits with an ant colony algorithm, with the shortest picking time as an optimization target of the planning.
For grape vine, HV polarization is adopted in the W frequency band, so that a more proper penetrating effect can be obtained, and the identification accuracy and recall rate are optimal under the condition that other environments and parameters are the same.
It should be noted that, since the present invention utilizes the difference between the moisture content of the fruit and the moisture content of the leaf and the branch to generate different radar echo reactions, the method in the foregoing embodiment is preferably applied to fruits and vegetables with the moisture content of the fruit being more than 1.5 times of the moisture content of the leaf.
Since the first image adopts the SAR image, the method in the foregoing embodiment is preferably applied to fruits and vegetables having an average diameter of greater than 2.5cm or fruits having a string-like structure similar to grape, in view of resolution.
The aforementioned frequency band refers to an electromagnetic wave frequency band, which is defined by the IEEE (institute of electrical and electronics engineers) division method, and since the frequency of the electromagnetic wave frequency band used is not greater than the W frequency band, in the embodiment of the present invention, there is no cross-overlapping of frequency bands.
In one embodiment of the present invention, there is provided a fruit and vegetable identification picking system, as shown in fig. 2, comprising:
a first image acquisition module 201 that acquires a first image and a second image of a picking area;
an image preprocessing module 202 for processing the first image and the second image to obtain image structure data, wherein the image structure data comprises nodes, and one node is in data connection with one pixel of the first image or the second image;
the target recognition module 203 inputs the graph structure data into a recognition model, and the recognition model outputs coordinate values representing predicted positions of fruits corresponding to the nodes; if the coordinate value of the predicted position of the fruit corresponding to the node is 0, the pixel of the first image representing the node data connection does not correspond to the fruit;
a movement control module 204, wherein the picking robot moves to the position of the fruit according to the predicted position of the fruit;
the second image acquisition module 205 acquires a third image of the front surface of the picking robot, wherein the third image is a visible light image;
the example segmentation module 206 performs example segmentation on the third image through the contour recognition positioning model to determine picking points of fruits;
a picking control module 207 for controlling the picking robot to cut the picking points to complete picking of the fruit.
In one embodiment of the present invention, there is provided a fruit and vegetable identifying and picking device, comprising:
a memory for storing machine executable instructions, an identification model, and a contour identification positioning model;
and a processor for executing the machine executable instructions stored in the memory, which when executed, is capable of performing the steps of a fruit and vegetable identification picking method as described above.
The embodiment has been described above with reference to the embodiment, but the embodiment is not limited to the above-described specific implementation, which is only illustrative and not restrictive, and many forms can be made by those of ordinary skill in the art, given the benefit of this disclosure, are within the scope of this embodiment.
Claims (10)
1. The fruit and vegetable identification picking method is characterized by comprising the following steps of:
step 101, collecting a first image and a second image of a picking area;
the first image is a SAR image; the second image is an optical image or a SAR image;
step 102, processing the first image and the second image to obtain graph structure data, wherein the graph structure data comprises nodes, and one node is in data connection with one pixel of the first image or the second image; the conditions for establishing edges for two nodes include: the two node data associating two adjacent pixels on the first image; the two node data associating two adjacent pixels on the second image; one of the two nodes is connected with the pixel on the first image, the other is connected with the pixel on the second image, and the positions of the two pixels on the image coordinate systems of the first image and the second image are the same;
step 103, inputting the graph structure data into a recognition model, and outputting coordinate values representing predicted positions of fruits corresponding to the nodes by the recognition model; if the coordinate value of the predicted position of the fruit corresponding to the node is 0, the pixel of the first image representing the node data connection does not correspond to the fruit;
104, moving the picking robot to the position of the fruit according to the predicted position of the fruit, and then acquiring a third image which is a visible light image through an image acquisition unit on the picking robot;
and 105, performing example segmentation on the third image through the contour recognition positioning model to determine picking points of the fruits, and controlling the picking robot to cut the picking points to finish picking the fruits.
2. The fruit and vegetable identification picking method according to claim 1, wherein the second image is a SAR image, is acquired in a different frequency band from the first image, and the frequency of the acquisition frequency band of the second image is lower than the acquisition frequency band of the first image.
3. A fruit and vegetable identification picking method according to claim 1 wherein the first image and the second image are superimposed on the same image coordinate system such that the two pixels of the first image and the second image that are superimposed are the corresponding two pixels.
4. The fruit and vegetable identification picking method according to claim 1, wherein the identification model comprises a first hidden layer and a full connection layer;
the calculation formula of the first hidden layer is as follows:
；
；
；
；
；
；
；
wherein the method comprises the steps ofFor initially hiding features->Node characteristics representing the v-th node, +.>Representing fusion vectors +.>、/>、、/>Representing the first, second, third and fourth intermediate features of the v-th node at the t-th time step, respectively,/->And->Output characteristics of the v-th node at the t-th time step and t-1 time step are respectively represented,/->Representing the output characteristics of the jth node at t-1 time steps,/for>Representing a set of nodes directly connected by edges to a v-th node, the set comprising the v-th node,/the v-th node>Representing dot product->Representing vector concatenation->、/>、/>、/>、/>、/>、/>、/>Respectively represent the first, second, third, fourth, fifth, sixth, seventh and eighth weight parameters, ++>、/>、/>、/>Respectively representing the first, second, third and fourth bias parameters.
5. The fruit and vegetable identification picking method according to claim 4, wherein the calculation formula of the full-connection layer is as follows:
；
wherein the method comprises the steps ofA coordinate vector representing the predicted position of the fruit corresponding to the y-th node, the p-th component of the coordinate vector representing the p-th coordinate value of the predicted position of the fruit corresponding to the y-th node, ">Representing the output characteristics of the y-th node at the S-th time step, the y-th node establishing a data connection with the pixels of the first image,/a>Represents a ninth weight parameter,>representing a fifth bias parameter.
6. The method according to claim 1, 4 or 5, wherein the coordinate value of the predicted position of the fruit corresponding to the node output representing the output of the recognition model is a three-dimensional coordinate in a world coordinate system, one node corresponds to three coordinate values, and the world coordinate system is an origin point of a ground of the picking area.
7. The fruit and vegetable identification picking method according to claim 1, 4 or 5, wherein the calculation formula for calculating the loss value of the identification model by using the coordinate values in the world coordinate system is as follows:
；
wherein Loss represents a Loss value, A represents the total number of nodes corresponding to which the coordinate value output by the recognition model is not 0, B represents the total number of actual fruits in the picking area,the g coordinate value of the predicted position of the fruit corresponding to the kth node,/for>A g-th coordinate value indicating a real fruit position corresponding to a predicted position of the fruit corresponding to the k-th node; />The g coordinate value of the d-th real fruit, < > is shown>And g coordinate values representing the predicted positions corresponding to the d-th real fruits.
8. The fruit and vegetable identification picking method according to claim 1, wherein the contour identification positioning model is Mask R-CNN.
9. A fruit and vegetable identification picking system for performing a fruit and vegetable identification picking method according to any one of claims 1-8, comprising:
the first image acquisition module acquires a first image and a second image of the picking area;
the image preprocessing module is used for processing the first image and the second image to obtain image structure data, wherein the image structure data comprises nodes, and one node is in data connection with one pixel of the first image or the second image;
the target recognition module inputs the graph structure data into a recognition model, and the recognition model outputs coordinate values representing predicted positions of fruits corresponding to the nodes; if the coordinate value of the predicted position of the fruit corresponding to the node is 0, the pixel of the first image representing the node data connection does not correspond to the fruit;
the movement control module is used for enabling the picking robot to move to the position where the fruit is located according to the predicted position of the fruit;
the second image acquisition module acquires a third image on the front surface of the picking robot, wherein the third image is a visible light image;
the example segmentation module is used for carrying out example segmentation on the third image through the contour recognition positioning model to determine picking points of fruits;
and the picking control module is used for controlling the picking robot to cut the picking points to finish picking fruits.
10. The utility model provides a fruit vegetables discernment picking device which characterized in that includes:
a memory for storing machine executable instructions, an identification model, and a contour identification positioning model;
a processor for executing machine executable instructions stored in a memory, which when executed, is capable of performing the steps of a fruit and vegetable identification picking method as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311779603.1A CN117456368B (en) | 2023-12-22 | 2023-12-22 | Fruit and vegetable identification picking method, system and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311779603.1A CN117456368B (en) | 2023-12-22 | 2023-12-22 | Fruit and vegetable identification picking method, system and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117456368A true CN117456368A (en) | 2024-01-26 |
| CN117456368B CN117456368B (en) | 2024-03-08 |
Family
ID=89589558
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311779603.1A Active CN117456368B (en) | 2023-12-22 | 2023-12-22 | Fruit and vegetable identification picking method, system and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117456368B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2019194821A (en) * | 2018-05-06 | 2019-11-07 | 英俊 古川 | Target recognition device, target recognition method, and program |
| CN111783693A (en) * | 2020-07-06 | 2020-10-16 | 深圳市多彩汇通实业有限公司 | Intelligent identification method of fruit and vegetable picking robot |
| CN112990103A (en) * | 2021-04-16 | 2021-06-18 | 苏州大学 | String mining secondary positioning method based on machine vision |
| TWI744020B (en) * | 2020-10-06 | 2021-10-21 | 劉志俊 | Intelligent fruit bagging machine system |
| WO2022057319A1 (en) * | 2020-09-21 | 2022-03-24 | 河南大学 | Garlic crop recognition method based on coupled active/passive remote sensing images on cloud platform |
| CN115661569A (en) * | 2022-10-17 | 2023-01-31 | 北京航空航天大学 | High-precision fine-grained SAR target detection method |
| CN116363505A (en) * | 2023-03-07 | 2023-06-30 | 中科合肥智慧农业协同创新研究院 | Target picking method based on picking robot vision system |
| CN116385432A (en) * | 2023-06-01 | 2023-07-04 | 安徽大学 | Light-weight decoupling wheat scab spore detection method |
-
2023
- 2023-12-22 CN CN202311779603.1A patent/CN117456368B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2019194821A (en) * | 2018-05-06 | 2019-11-07 | 英俊 古川 | Target recognition device, target recognition method, and program |
| CN111783693A (en) * | 2020-07-06 | 2020-10-16 | 深圳市多彩汇通实业有限公司 | Intelligent identification method of fruit and vegetable picking robot |
| WO2022057319A1 (en) * | 2020-09-21 | 2022-03-24 | 河南大学 | Garlic crop recognition method based on coupled active/passive remote sensing images on cloud platform |
| TWI744020B (en) * | 2020-10-06 | 2021-10-21 | 劉志俊 | Intelligent fruit bagging machine system |
| CN112990103A (en) * | 2021-04-16 | 2021-06-18 | 苏州大学 | String mining secondary positioning method based on machine vision |
| CN115661569A (en) * | 2022-10-17 | 2023-01-31 | 北京航空航天大学 | High-precision fine-grained SAR target detection method |
| CN116363505A (en) * | 2023-03-07 | 2023-06-30 | 中科合肥智慧农业协同创新研究院 | Target picking method based on picking robot vision system |
| CN116385432A (en) * | 2023-06-01 | 2023-07-04 | 安徽大学 | Light-weight decoupling wheat scab spore detection method |
Non-Patent Citations (1)
| Title |
|---|
| 舒文灿: "基于YOLOv4的梨子检测算法研究及轻量化***实现", 中国优秀硕士学位论文全文数据库 农业科技辑, no. 3, 15 March 2022 (2022-03-15), pages 048 - 149 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117456368B (en) | 2024-03-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Tang et al. | Recognition and localization methods for vision-based fruit picking robots: A review | |
| CN111602517B (en) | Distributed visual active perception method for string-type fruits and application of distributed visual active perception method | |
| Bargoti et al. | Image segmentation for fruit detection and yield estimation in apple orchards | |
| Majeed et al. | Determining grapevine cordon shape for automated green shoot thinning using semantic segmentation-based deep learning networks | |
| CN111213155A (en) | Image processing method, device, movable platform, unmanned aerial vehicle and storage medium | |
| CN109255302A (en) | Object recognition methods and terminal, mobile device control method and terminal | |
| CN108765452A (en) | A kind of detection of mobile target in complex background and tracking | |
| CN110770791A (en) | Image boundary acquisition method and device based on point cloud map and aircraft | |
| Yoshida et al. | Automated harvesting by a dual-arm fruit harvesting robot | |
| Menon et al. | NBV-SC: Next best view planning based on shape completion for fruit mapping and reconstruction | |
| CN113065562A (en) | Crop ridge row extraction and leading route selection method based on semantic segmentation network | |
| Parhar et al. | A deep learning-based stalk grasping pipeline | |
| Basri et al. | Robots and drones in agriculture—a survey | |
| Sane et al. | Artificial intelligence and deep learning applications in crop harvesting robots-A survey | |
| Magistri et al. | Towards in-field phenotyping exploiting differentiable rendering with self-consistency loss | |
| Li et al. | Improved random sampling consensus algorithm for vision navigation of intelligent harvester robot | |
| Jin et al. | Far-near combined positioning of picking-point based on depth data features for horizontal-trellis cultivated grape | |
| Hafeez et al. | Crop monitoring and automatic weed detection using drone | |
| Suresh Kumar et al. | Selective fruit harvesting: Research, trends and developments towards fruit detection and localization–A review | |
| CN117456368B (en) | Fruit and vegetable identification picking method, system and device | |
| Mejia et al. | Strawberry localization in a ridge planting with an autonomous rover | |
| CN115294562B (en) | Intelligent sensing method for operation environment of plant protection robot | |
| Yu et al. | Design and implementation of an automatic peach-harvesting robot system | |
| CN115797397A (en) | Method and system for robot to autonomously follow target person in all weather | |
| Dang et al. | LiDAR-based online navigation algorithm for an autonomous agricultural robot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |