CN114830915A - Litchi vision picking robot based on laser radar navigation and implementation method thereof - Google Patents
Litchi vision picking robot based on laser radar navigation and implementation method thereof Download PDFInfo
- Publication number
- CN114830915A CN114830915A CN202210384198.2A CN202210384198A CN114830915A CN 114830915 A CN114830915 A CN 114830915A CN 202210384198 A CN202210384198 A CN 202210384198A CN 114830915 A CN114830915 A CN 114830915A
- Authority
- CN
- China
- Prior art keywords
- litchi
- picking
- vision
- camera
- laser radar
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D46/00—Picking of fruits, vegetables, hops, or the like; Devices for shaking trees or shrubs
- A01D46/30—Robotic devices for individually picking crops
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J11/00—Manipulators not otherwise provided for
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C11/04—Interpretation of pictures
- G01C11/30—Interpretation of pictures by triangulation
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Environmental Sciences (AREA)
- Electromagnetism (AREA)
- Computer Networks & Wireless Communication (AREA)
- Multimedia (AREA)
- Life Sciences & Earth Sciences (AREA)
- Manipulator (AREA)
Abstract
The invention discloses a litchi vision picking robot based on laser radar navigation and an implementation method thereof, wherein the litchi vision picking robot comprises a navigation mechanism, a carrying mechanism, a lifting mechanism, a power supply, a mechanical arm, a binocular vision mechanism, a tail end execution mechanism, a control cabinet and a fruit collection device; the navigation mechanism is arranged at the front end of a vehicle body bottom plate of the carrying mechanism; the lifting mechanism is arranged on a vehicle body bottom plate of the carrying mechanism; the mechanical arm, the power supply and the control cabinet are arranged on a working platform of the lifting mechanism; the binocular vision mechanism is arranged on the end effector fixing piece; the tail end executing mechanism is arranged at the tail end of the mechanical arm; the fruit collecting device is arranged on the side surface of the lifting mechanism. The automatic litchi picking machine can accurately identify the fruit stalks of litchi and pick the litchi automatically, and is high in picking efficiency, simple in overall structure and high in automation and intelligence degree.
Description
Technical Field
The invention belongs to the field of agricultural machinery, and particularly relates to a litchi vision picking robot based on laser radar navigation and an implementation method thereof.
Background
China is the most important litchi producing country in the world, but litchi harvesting needs a large amount of labor force, the litchi mature period is very short, and meanwhile, the weather in south of Lingnan is hot and rainy, so that serious economic loss can be caused if the litchi is not picked in time. At present, some litchi picking machines exist, for example, patent CN109566097A discloses a litchi picking machine, when picking claws of the litchi picking machine perform picking actions, a plurality of claw fingers arranged on a claw finger support in parallel can clamp branches without fruits in the picking process, the picking efficiency is low, autonomous navigation cannot be realized, and a visual module is not carried, so that automatic picking cannot be realized.
Disclosure of Invention
The invention aims to overcome the defects in the prior art, and provides a litchi vision picking robot based on laser radar navigation and an implementation method thereof, which can accurately identify litchi fruit stalks and automatically pick litchi fruits, and have the advantages of high picking efficiency, simple overall structure and high automation and intelligence degrees.
The purpose of the invention is realized by the following technical scheme:
a litchi vision picking robot based on laser radar navigation comprises a navigation mechanism 1, a carrying mechanism 2, a lifting mechanism 3, a power supply 4, a mechanical arm 5, a binocular vision mechanism 6, a tail end execution mechanism 7, a control cabinet 8 and a fruit collecting device 9; the navigation mechanism 1 is arranged at the front end of the bottom plate of the vehicle body of the carrying mechanism 2; the lifting mechanism 3 is arranged on the bottom plate of the vehicle body of the carrying mechanism 2; the mechanical arm 5, the power supply 4 and the control cabinet 8 are arranged on a working platform of the lifting mechanism 3; the binocular vision mechanism 6 is mounted on the end effector fixing member 34; the end executing mechanism 7 is arranged at the end of the mechanical arm; the fruit collecting device 9 is installed at the side of the elevating mechanism.
The lifting mechanism 3 comprises a base 11, a crossed lifting assembly 12, a hydraulic driving assembly 10 and a working platform 13; the hydraulic driving assembly 10 is fixed on the base 11, and the hydraulic driving assembly 10 is respectively connected with the cross type lifting assembly 12 and the control cabinet 8 and is used for driving the cross type lifting assembly 12 to do lifting movement; the working platform 13 is horizontally arranged on the crossed lifting assembly 12; the fruit collecting device 9 is arranged on the side surface of the lifting mechanism working platform.
The mechanical arm 5 comprises a first screw rod 14, a second screw rod 15, a third screw rod 16, a fourth screw rod 17, a fifth screw rod 18, a sixth screw rod 19, a first push rod 20, a second push rod 21, a third push rod 22, a fourth push rod 23, a first connecting piece 24 and a second connecting piece 25; the mechanical arms 5 are two sets of mechanical arms with the same structure and are respectively arranged at the diagonal positions of the lifting mechanism working platform 13; one set of mechanical arm is that a first screw rod 14 and a second screw rod 15 are vertically fixed on a lifting mechanism working platform 13 along the advancing direction of a carrying mechanism, two ends of a third screw rod 16 are respectively and horizontally fixed on sliding tables of the first screw rod 14 and the second screw rod 15, a first push rod 20 and a second push rod 21 are vertically fixed on the third screw rod 16 in a symmetrical mode, an end effector is connected to the tail end of the first push rod 20 through an end effector fixing part 34, and then the end effector fixing part is connected with the second push rod 21 through a first connecting part 24; the same structure is applied to another set of mechanical arms which are arranged diagonally to the other set of mechanical arms.
The end effector 7 comprises an end effector fixing part 34 and an end effector; the end effector comprises a motor 29, a meshing blade 30, a rubber clamping plate 31, a clamping plate fixing member 32, a structure fixing member 33, a screw rod nut 35 and a push rod 36; the meshing blade 30 is rigidly connected with a clamping plate fixing member 32, a rubber clamping plate 31 is arranged below the meshing blade 30 through the clamping plate fixing member 32, the motor 29 is connected with a structure fixing member 33, a screw rod nut 35 is connected with a screw rod of the motor 29, a push rod 36 is fixed on the screw rod nut 35, and the meshing blade 30 is connected with the tail end of the push rod 36. When picking operation is carried out, the meshing blade and the clamping component, namely the rubber clamping plate 31, are relatively static, when the motor works, the screw rod nut 35 realizes the mutual conversion of rotary motion and linear motion by the rotation of the screw rod of the motor 29, and the push rod 36 is driven by the reciprocating linear motion of the screw rod nut 35 to provide power for the meshing blade 30 so as to carry out shearing motion; the rubber clamping plate 31 completes the clamping of the litchi fruit stalks when the engagement blade 30 is engaged. The meshing blade 30 is designed by a saw-toothed structure imitating biological teeth, the rubber clamping plate 31 is designed into a shape similar to a biological lip, and a clamping and shearing integrated structure imitating a biological mouth structure has the advantages of realizing processability, and the saw-toothed structure of the meshing blade 30 is also beneficial to positioning the litchi stalks and preventing the stalks from sliding when being meshed and sheared. The end effector designed by the invention can be used for picking litchi fruits and branches without damaging the litchi fruits and branches, the integrity of the litchi fruits and branches is ensured, the end effector can be better suitable for litchi fruit stems with different thicknesses, the end effector is compact in structure, the litchi fruit stems can be prevented from being influenced by other branches during clamping and shearing, and litchi strings can be stably clamped by a clamp with a saw-toothed structure and made of rubber materials and cannot fall off.
The binocular vision mechanism 6 comprises a left camera 27, a right camera 28 and a camera fixing frame 26, wherein the left camera and the right camera are installed on the camera fixing frame, and the camera fixing frame 26 is installed on an end effector fixing piece 34.
A realization method of a litchi vision picking robot based on laser radar navigation comprises the following steps:
(1) laser radar navigation: the laser radar and the encoder are used for acquiring data information of a road, positioning, map construction and path planning of the robot are carried out through the acquired data information, and the carrying mechanism accurately reaches a specified position after receiving an instruction;
(2) the lifting mechanism is lifted: aiming at different picking environments and litchi varieties, the lifting mechanism is controlled to reach a proper height to carry out picking operation;
(3) data set preparation: firstly, a camera is used for collecting a large number of litchi images, then a marking tool is used for marking the collected images, namely, litchi fruit stems to be picked are selected out by a frame, and a litchi fruit stem data set is obtained;
(4) and (3) deep learning model: training the litchi fruit stem data set by using an improved YOLOV4 network to obtain a network model for identifying the litchi fruit stems;
(5) single and double eye calibration and hand-eye calibration of the double-eye camera: firstly, performing monocular calibration on two cameras respectively to obtain an internal reference matrix and a distortion matrix of each camera; then, performing binocular stereo vision calibration on the two cameras to obtain a reprojection matrix for binocular correction, and simultaneously obtaining a conversion relation between a real object distance and a camera pixel distance under a world coordinate system; performing binocular correction on the litchi image shot by the binocular camera to obtain a binocular corrected image; the hand-eye calibration is used for determining the coordinate system conversion relation between the robot and the camera;
(6) three-dimensional reconstruction: acquiring a color image of a litchi image through a binocular camera, obtaining a depth map through stereo matching, inputting the color image into the trained network model in the step (4), obtaining position coordinates of picking points of litchi fruit stems, and obtaining three-dimensional point cloud of the litchi fruit stems according to a triangular distance measurement principle;
(7) planning a motion track: three-dimensional point cloud information of the litchi fruit stalks is transmitted to a control cabinet, the control cabinet plans the motion track of the mechanical arm by analyzing spatial three-dimensional information contained in the point cloud information and adopting an obstacle avoidance algorithm to enable the mechanical arm to be close to a picking target;
(8) and (3) attitude estimation: because the stem direction of the litchi is not vertical downwards, the axial direction vector n of the stem and the direction vector m vertical to the n are determined by the acquired three-dimensional point cloud of the stem of the litchi;
(9) picking and collecting: the end effector clamps and shears the picking point from the direction of a vector m by controlling the extension of the first push rod and the second push rod, and the picked litchi fruit string is placed into a fruit collecting device.
In the step (1), the laser radar navigation means that the carrying mechanism utilizes a sensor thereof to identify the position of the carrying mechanism in the environment and construct a map based on the environment, and simultaneously, the positioning navigation function is realized based on the map, and the realization of the function is the technology of simultaneously positioning and constructing the map.
In the step (4), the improved YOLOV4 network is a feature map obtained by convolution with different receptive fields, and its kernel size is set to 1 × 1, 3 × 3, and 5 × 5, respectively. The original Yolov4 network adopts the operation of maximum pooling in the pyramid pooling structure to obtain the characteristic maps of different receptive fields, but the multi-scale object information can not be obtained through the maximum pooling, and a lot of detailed information about the image boundary can be lost, so the invention adopts convolution to realize the characteristic maps of different receptive fields, and after improvement, the invention not only solves the problem that the convolutional neural network has to fix the size of the input image, but also enlarges the receptive fields by introducing different void rates and fuses the output results.
In the step (5), the camera calibration refers to a process of solving camera model parameters. Firstly, performing monocular calibration to respectively obtain an external parameter matrix, an internal parameter matrix and a distortion matrix of a left camera and a right camera, performing binocular calibration on the basis of the monocular calibration to obtain matrix parameters such as a re-projection matrix and a mapping table of the binocular calibration, and finally obtaining the mutual relation between the three-dimensional geometric position of a certain point on the surface of the space object and the corresponding point in the image. The purpose of solving the transformation relationship between the camera coordinate system and the robot coordinate system is: and converting the litchi three-dimensional coordinates based on the camera coordinate system into litchi three-dimensional coordinates based on the robot coordinate system, calculating the motion postures of each screw rod and each push rod according to the coordinates, and controlling the end effector to reach the designated position.
In the step (6), the binocular vision-based three-dimensional reconstruction is an important component of the image processing technology and the machine vision. Because the image shot by the monocular camera is two-dimensional, a binocular camera is needed to simulate a human visual system, the distance from the position of a certain point on the surface of a space object to a connecting line between optical centers of two lens heads is calculated according to the parallax of the matching point pairs of two images shot by the left camera and the right camera, and the three-dimensional coordinate of the point cloud on the surface of the object is obtained. The three-dimensional reconstruction is a process of reconstructing the surface of a target object by using the three-dimensional coordinates of the point cloud on the surface of the object, so that the picking speed of the robot can be increased, the efficiency is improved, and the robot becomes more intelligent.
Compared with the prior art, the invention has the following advantages and effects:
(1) the invention uses the improved YOLOV4 network, and adopts convolution to replace the operation of maximum pooling in the original network space pyramid pooling structure to obtain the characteristic maps of different receptive fields. Because multi-scale object information cannot be acquired through maximum pooling, and much detailed information about image boundaries can be lost, the invention adopts convolution implementation, and kernel sizes of the kernel sizes are respectively set to be 1 × 1, 3 × 3 and 5 × 5, so that the problem that the input image size of a convolutional neural network needs to be fixed is solved after improvement, the receptive field is enlarged in a phase-changing manner by introducing different cavity rates, and output results are fused. In addition, in order to solve the problem of missed detection of the small target litchi by the YOLOV4, a method is provided for sampling and training only the small target litchi by using a high-resolution sample picture, and then performing mixed training with other pictures shot under a large visual field, so that the recognition rate of the YOLOV4 on the small target litchi is improved.
(2) The end effector used by the invention designs a clamping and shearing integrated structure that the engaging blades are engaged to shear the litchi fruit stalks and rubber materials are used to clamp the litchi fruit stalks by the zigzag structure of the bionic teeth, and has the capability of realizing the processing. The end effector with the bionic mouth structure can pick litchi fruits and branches and leaves on the premise of not damaging the litchi fruits and the branches and leaves, the completeness of the litchi fruits and the branches and leaves is guaranteed, the litchi fruit stem with different thicknesses can be better adapted, and the litchi string can be stably clamped by the saw-toothed structure and the rubber material clamp so as not to fall. The milling machine imitating type mechanical arm designed by the invention has the advantages of simple structure, good symmetrical load and force balance of the mechanical arms at the two sides, low use cost, capability of avoiding the defects of large working space, high cost, increased accumulated error, poor rigidity and the like required by the mechanical arm with 6 shafts connected in series, capability of realizing the simultaneous picking of fruits at the two sides by the mechanical arm, and improvement on picking efficiency.
(3) When the end effector is used for picking the litchi fruits through clamping and shearing, because the fruit stem direction of the litchi fruits is not vertical downwards, the axial direction vector n of the fruit stem and the direction vector m vertical to the axial direction vector n need to be determined through the acquired three-dimensional point cloud of the litchi fruit stem, and the picking angle of the end effector is changed by innovatively controlling the stretching amount of the push rod, so that the end clamping and shearing mechanism can pick the litchi fruits in the vertical direction of the axial direction of the obtained litchi stems.
Drawings
Fig. 1 is a schematic view of the overall structure of the litchi vision picking robot of the invention.
Fig. 2 is a front view of the litchi vision picking robot of the present invention.
Fig. 3 is a side view of the litchi vision picking robot of the present invention.
Fig. 4 is a top view of the litchi vision picking robot of the present invention.
Fig. 5 is a schematic structural view of the robot arm of the present invention.
Fig. 6 is a schematic structural view of an end effector of the present invention.
Fig. 7 is a schematic structural view of an end effector of the present invention.
FIG. 8 is a flow chart of lidar navigation of the present invention.
FIG. 9 is a flow chart of the vision recognition algorithm of the present invention.
Fig. 10 is a modified YOLOV4 network SPP architecture of the present invention.
Wherein, 1-a navigation mechanism, 2-a carrying mechanism, 3-a lifting mechanism, 4-a power supply, 5-a mechanical arm, 6-a binocular vision mechanism, 7-an end executing mechanism, 8-a control cabinet, 9-a fruit collecting device, 10-a hydraulic driving component, 11-a base, 12-a cross type lifting component, 13-a working platform, 14-a first screw rod, 15-a second screw rod, 16-a third screw rod, 17-a fourth screw rod, 18-a fifth screw rod, 19-a sixth screw rod, 20-a first push rod, 21-a second push rod, 22-a third push rod, 23-a fourth push rod, 24-a first connecting piece, 25-a second connecting piece, 26-a camera fixing frame, 27-a left camera and 28-a right camera, 29-motor, 30-meshing blade, 31-rubber splint, 32-splint fixing piece, 33-structure fixing piece, 34-end effector fixing piece, 35-screw rod nut and 36-push rod.
Detailed Description
In order that the invention may be readily understood, reference will now be made in detail to the specific embodiments of the invention. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that, for a person skilled in the art, many variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Unless defined otherwise, all technical and scientific terms used herein are to be interpreted as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Examples
As shown in fig. 1-4, an automatic litchi picking robot based on laser radar navigation comprises a navigation mechanism 1, a lifting mechanism 3, a binocular vision mechanism 6, a carrying mechanism 2, a mechanical arm 5, a tail end execution mechanism 7, a power supply 4, a control cabinet 8 and a fruit collection device 9; the laser radar for navigation is arranged at the front end of the bottom plate of the vehicle body of the carrying mechanism 2; the lifting mechanism 3 is arranged on the bottom plate of the vehicle body of the carrying mechanism 2; the mechanical arm 5, the power supply 4 and the control cabinet 8 are arranged on the working platform of the lifting mechanism 3; the binocular vision mechanism 6 is mounted on the end effector fixing part 34; the end executing mechanism 7 is arranged at the tail end of the mechanical arm; the fruit collecting device 9 is arranged on the side surface of the lifting platform. As shown in figure 1, the laser radar is installed at the front end of the bottom plate of the vehicle body of the carrying mechanism, the laser radar and the encoder are responsible for acquiring data information of roads, and the carrying mechanism is responsible for executing motion instructions. The robot performs functions such as positioning, map construction, and path planning of the robot using the obtained data. Elevating system installs on carrying mechanism vehicle body bottom plate, elevating system 3 includes base 11, crossing lifting unit 12, hydraulic drive subassembly 10 and work platform 13, hydraulic drive subassembly 10 is fixed on base 11, and hydraulic drive subassembly 10 is connected with crossing lifting unit 12, switch board 8 respectively for drive lifting unit 12 is elevating movement, 13 levels of work platform set up on crossing lifting unit 12, and fruit collection device 9 installs in elevating system work platform side. The mechanical arm 5, the power supply 4 and the control cabinet 8 are arranged on the working platform 13. As shown in fig. 5, the mechanical arm 5 includes a first lead screw 14, a second lead screw 15, a third lead screw 16, a fourth lead screw 17, a fifth lead screw 18, a sixth lead screw 19, a first push rod 20, a second push rod 21, a third push rod 22, a fourth push rod 23, a first connecting piece 24, a second connecting piece 25, an end effector fixing piece 34, and an end effector. Two sets of mechanical arms with the same structure are assembled by the parts and are respectively arranged at the diagonal positions of the lifting mechanism working platform 13. In recent years, agricultural planting in China especially fruit and vegetable planting tends to be standardized, so that the picking of fruits on two sides by the mechanical arm is realized, and the picking efficiency is greatly improved. One set of mechanical arms is that a first screw rod 14 and a second screw rod 15 are vertically fixed on a lifting mechanism working platform 13 along the advancing direction of a carrying mechanism, two ends of a third screw rod 16 are horizontally fixed on sliding tables of the first screw rod 14 and the second screw rod 15 respectively, a first push rod 20 and a second push rod 21 are vertically fixed with the third screw rod 16 in a symmetrical mode, an end effector is connected to the tail end of the first push rod 20 through an end effector fixing part 34, then the end effector fixing part is connected with the second push rod 21 through a first connecting part 24, and the mechanical arms arranged diagonally with the end effector are also of the same structure. As shown in fig. 6, the end effector 7 includes an end effector fixing member 34 and an end effector, as shown in fig. 7, the end effector is composed of a motor 29, a meshing blade 30, a rubber clamping plate 31, a clamping plate fixing member 32, a structural fixing member 33, a screw nut 35 and a push rod 36, the meshing blade 30 is fixed on the structural fixing member 33 through a bolt, the meshing blade 30 is driven to open or close by the rotation of the motor 29, the rubber clamping plate 31 is installed below the meshing blade 30 through the clamping plate fixing member 32, the motor 29 is connected with the structural fixing member 33, the screw nut 35 is connected with a screw rod of the motor 29, the push rod 36 is fixed on the screw nut 35, the meshing blade 30 is connected to the tail end of the push rod 36, and the rubber clamping plate 31 realizes shearing while clamping the litchi fruit stalks. The binocular vision mechanism 6 includes a left camera 27, a right camera 28, and a camera mount 26, the left and right cameras being mounted on the camera mount, the camera mount 26 being mounted on an end effector mount 34. The end effector clamps and shears the picking points from the direction vertical to the axes of the litchi fruit stems by controlling the extension of the first push rod and the second push rod, and the picked litchi fruit strings are placed into the collecting device 9.
The litchi vision picking robot based on laser radar navigation and the implementation method thereof comprise the following steps:
(1) laser radar navigation: the laser radar and the encoder are responsible for acquiring data information of roads, in a robot software system, the robot performs the functions of positioning, map construction, path planning and the like of the robot by using the acquired data, and the carrying mechanism can accurately reach a specified position after receiving an instruction; as shown in fig. 8;
(2) lifting the picking platform: due to the influence of different environments and litchi varieties, the lifting mechanism 3 needs to be controlled to reach a proper height to carry out picking operation;
(3) data set preparation: firstly, a camera is used for collecting a large number of litchi images, and then a marking tool is used for marking the collected images, namely, litchi fruit stems to be picked are selected out from a frame;
(4) deep learning model: training the data set in the step (3) by using an improved YOLOV4 network to obtain a network model for identifying fruit stalks better; as shown in fig. 10;
(5) single and double eye calibration and hand-eye calibration of the double-eye camera: firstly, performing monocular calibration on two cameras respectively to obtain an internal reference matrix and a distortion matrix of each camera; then, performing binocular stereo vision calibration on the two cameras to obtain a reprojection matrix for binocular correction, and simultaneously obtaining a conversion relation between a real object distance and a camera pixel distance under a world coordinate system; carrying out binocular correction on the litchi image shot by the binocular camera to obtain a binocular corrected image; the hand-eye calibration can be used for determining the coordinate system conversion relation between the robot and the camera;
(6) three-dimensional reconstruction: acquiring a color image of a litchi image through a binocular camera, obtaining a depth map through stereo matching, inputting the color image into the trained network model in the step (4), obtaining position coordinates of litchi fruit stem picking points, and obtaining three-dimensional point cloud of litchi fruit stems according to a triangular distance measurement principle; as shown in fig. 9;
(7) planning a motion track: three-dimensional point cloud information of the litchi fruit stalks is transmitted to the control cabinet 8, the control cabinet 8 plans the motion track of the mechanical arm 5 by adopting an obstacle avoidance algorithm through analyzing spatial three-dimensional information contained in the point cloud data, and the mechanical arm is made to approach a picking target;
(8) and (3) attitude estimation: because the stem direction of the litchi is not vertical downwards, the axial direction vector n of the stem and the direction vector m vertical to the n are determined by the acquired three-dimensional point cloud of the stem of the litchi;
(9) picking and collecting: the end effector clamps and shears the picking point from the direction of the vector m by controlling the elongation of the first push rod 20 and the second push rod 21, and the picked litchi fruit bunch is placed into the collecting device 9.
The working principle of the invention is as follows: the automatic driving of the picking robot in the orchard is realized through a laser radar, the height of a lifting platform is controlled according to the tree height, the litchi fruit stems are identified through an improved YOLOV4 network framework, a color image of a litchi image is obtained through a binocular camera, a depth map is obtained through three-dimensional matching, three-dimensional point clouds of the litchi fruit stems are obtained according to a triangular distance measurement principle, the position coordinates of the litchi fruit stem picking points are obtained, the tail end executing mechanism is controlled to be close to the picking points to clamp and cut the picking points in the direction perpendicular to the axis of the litchi fruit stems according to a detected target, and the picked litchi fruit strings are placed into a collecting device.
The above description is only an example of the present invention, but the present invention is not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention are all equivalent substitutions and are intended to be included within the scope of the present invention.
Claims (9)
1. The utility model provides a litchi vision picking robot based on laser radar navigation which characterized in that: the fruit picking device comprises a navigation mechanism, a carrying mechanism, a lifting mechanism, a power supply, a mechanical arm, a binocular vision mechanism, a tail end executing mechanism, a control cabinet and a fruit collecting device; the navigation mechanism is arranged at the front end of a vehicle body bottom plate of the carrying mechanism; the lifting mechanism is arranged on a vehicle body bottom plate of the carrying mechanism; the mechanical arm, the power supply and the control cabinet are arranged on a working platform of the lifting mechanism; the binocular vision mechanism is arranged on the end effector fixing piece; the tail end executing mechanism is arranged at the tail end of the mechanical arm; the fruit collecting device is arranged on the side surface of the lifting mechanism.
2. The litchi vision picking robot based on laser radar navigation of claim 1, characterized in that: the navigation mechanism comprises a laser radar and an encoder, the laser radar is installed at the front end of a vehicle body bottom plate of the carrying mechanism, and the laser radar and the encoder are used for acquiring data information of a road.
3. The litchi vision picking robot based on laser radar navigation of claim 1, characterized in that: the lifting mechanism comprises a base, a crossed lifting assembly, a hydraulic driving assembly and a working platform; the hydraulic driving assembly is fixed on the base, and is respectively connected with the crossed lifting assembly and the control cabinet and used for driving the crossed lifting assembly to do lifting motion; the working platform is horizontally arranged on the crossed lifting assembly.
4. The litchi vision picking robot based on laser radar navigation of claim 1, characterized in that: the mechanical arms are two sets of mechanical arms with the same structure and are respectively arranged at the diagonal positions of the lifting mechanism working platform; the mechanical arm sets vertically fix a first screw rod and a second screw rod on a lifting mechanism working platform along the advancing direction of a carrying mechanism, two ends of a third screw rod are horizontally fixed on sliding tables of the first screw rod and the second screw rod respectively, a first push rod and a second push rod are vertically fixed on the third screw rod symmetrically, an end effector is connected to the tail end of the first push rod through an end effector fixing part, and then the end effector fixing part is connected with the second push rod through a first connecting piece; the other set of mechanical arms is also of the same structure.
5. The litchi vision picking robot based on laser radar navigation of claim 1, characterized in that: the end effector comprises an end effector fixing part and an end effector; the end effector comprises a motor, a meshing blade, a rubber clamping plate, a clamping plate fixing piece, a structure fixing piece, a screw rod nut and a push rod; the meshing blade is rigidly connected with the clamping plate fixing piece, the rubber clamping plate is arranged below the meshing blade through the clamping plate fixing piece, the motor is connected with the structure fixing piece, the screw rod nut is connected with the screw rod of the motor, the push rod is fixed on the screw rod nut, and the meshing blade is connected to the tail end of the push rod.
6. The litchi vision picking robot based on laser radar navigation of claim 5, characterized in that: the engaging blade is a saw-toothed structure imitating biological teeth, and the rubber splint is in a shape imitating biological lips.
7. The litchi vision picking robot based on laser radar navigation of claim 1, characterized in that: the binocular vision mechanism comprises a left camera, a right camera and a camera fixing frame, wherein the left camera and the right camera are installed on the camera fixing frame, and the camera fixing frame is installed on the end effector fixing piece.
8. The implementation method of the litchi vision picking robot based on laser radar navigation as claimed in any one of claims 1-7, characterized by comprising the following steps:
(1) laser radar navigation: the laser radar and the encoder are used for acquiring data information of a road, positioning, map construction and path planning of the robot are carried out through the acquired data information, and the carrying mechanism accurately reaches a specified position after receiving an instruction;
(2) the lifting mechanism is lifted: aiming at different picking environments and litchi varieties, the lifting mechanism is controlled to reach a proper height to carry out picking operation;
(3) data set preparation: firstly, a camera is used for collecting a large number of litchi images, then a marking tool is used for marking the collected images, namely, litchi fruit stems to be picked are selected out by a frame, and a litchi fruit stem data set is obtained;
(4) deep learning model: training the litchi fruit stem data set by using an improved YOLOV4 network to obtain a network model for identifying the litchi fruit stems;
(5) single and double eye calibration and hand-eye calibration of the double-eye camera: firstly, performing monocular calibration on two cameras respectively to obtain an internal parameter matrix and a distortion matrix of each camera; then, performing binocular stereo vision calibration on the two cameras to obtain a reprojection matrix for binocular correction, and simultaneously obtaining a conversion relation between a real object distance and a camera pixel distance under a world coordinate system; performing binocular correction on the litchi image shot by the binocular camera to obtain a binocular corrected image; the hand-eye calibration is used for determining the coordinate system conversion relation between the robot and the camera;
(6) three-dimensional reconstruction: acquiring a color image of a litchi image through a binocular camera, obtaining a depth map through stereo matching, inputting the color image into the trained network model in the step (4), obtaining position coordinates of litchi fruit stem picking points, and obtaining three-dimensional point cloud of litchi fruit stems according to a triangular distance measurement principle;
(7) planning a motion track: three-dimensional point cloud information of the litchi fruit stalks is transmitted to a control cabinet, the control cabinet plans the motion track of the mechanical arm by analyzing spatial three-dimensional information contained in the point cloud information and adopting an obstacle avoidance algorithm to enable the mechanical arm to be close to a picking target;
(8) and (3) attitude estimation: because the stem direction of the litchi is not vertical downwards, the axial direction vector n of the stem and the direction vector m vertical to the n are determined by the acquired three-dimensional point cloud of the stem of the litchi;
(9) picking and collecting: the end effector clamps and shears the picking point from the direction of a vector m by controlling the extension of the first push rod and the second push rod, and the picked litchi fruit string is placed into a fruit collecting device.
9. The realization method of the litchi vision picking robot according to claim 8, characterized in that: in the step (4), the improved YOLOV4 network is a feature map obtained by convolution with different receptive fields, and its kernel size is set to 1 × 1, 3 × 3, and 5 × 5, respectively.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210384198.2A CN114830915B (en) | 2022-04-13 | 2022-04-13 | Litchi vision picking robot based on laser radar navigation and implementation method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210384198.2A CN114830915B (en) | 2022-04-13 | 2022-04-13 | Litchi vision picking robot based on laser radar navigation and implementation method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114830915A true CN114830915A (en) | 2022-08-02 |
| CN114830915B CN114830915B (en) | 2023-09-26 |
Family
ID=82563866
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210384198.2A Active CN114830915B (en) | 2022-04-13 | 2022-04-13 | Litchi vision picking robot based on laser radar navigation and implementation method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114830915B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115316129A (en) * | 2022-08-12 | 2022-11-11 | 华南农业大学 | Self-adaptive bionic picking device based on binocular vision recognition and cluster fruit picking method |
| CN115589849A (en) * | 2022-10-18 | 2023-01-13 | 大连理工大学（Cn） | Spherical fruit and vegetable picking device |
Citations (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004180554A (en) * | 2002-12-02 | 2004-07-02 | National Agriculture & Bio-Oriented Research Organization | Method and apparatus for selectively harvesting fruit vegetables |
| CN107094429A (en) * | 2017-07-03 | 2017-08-29 | 彭紫薇 | A kind of multiaxis fruit picking robot |
| CN207678305U (en) * | 2017-12-18 | 2018-08-03 | 艾力·艾散 | A kind of mechanical arm for walnut picking |
| CN108557143A (en) * | 2018-05-01 | 2018-09-21 | 浙江工业大学 | A kind of fruit picking packing machine |
| CN108575320A (en) * | 2018-05-23 | 2018-09-28 | 芮杰 | Strawberry picking robot |
| CN208047324U (en) * | 2018-04-17 | 2018-11-06 | 石河子大学 | A kind of safflower picking end effector |
| CN208079866U (en) * | 2018-04-16 | 2018-11-13 | 苏州大学 | A kind of fruit automatic picking apparatus |
| CN108834576A (en) * | 2018-06-07 | 2018-11-20 | 华南农业大学 | A kind of citrus picking robot and its implementation based on binocular vision |
| CN109328643A (en) * | 2018-12-17 | 2019-02-15 | 青岛科技大学 | A kind of self-balancing type apple-picking vehicle |
| JP2019037214A (en) * | 2017-08-29 | 2019-03-14 | 株式会社デンソー | End effector |
| CN109743966A (en) * | 2019-02-18 | 2019-05-14 | 德施普科技发展温州有限公司 | A kind of fruit picking |
| CN209047050U (en) * | 2018-08-17 | 2019-07-02 | 长沙理工大学 | A kind of Portable electric fruit picker |
| CN110832991A (en) * | 2019-10-24 | 2020-02-25 | 广州大学 | Picking machine based on image recognition and control method thereof |
| CN211020054U (en) * | 2019-09-16 | 2020-07-17 | 中国农业科学院农业信息研究所 | Fruit and vegetable picking device |
| CN211415245U (en) * | 2019-12-26 | 2020-09-04 | 青岛悟牛智能科技有限公司 | Picking robot |
| CN111673755A (en) * | 2020-06-24 | 2020-09-18 | 西北农林科技大学 | Picking robot control system and method based on visual servo |
| CN111709489A (en) * | 2020-06-24 | 2020-09-25 | 广西师范大学 | Citrus identification method based on improved YOLOv4 |
| CN111742709A (en) * | 2020-04-23 | 2020-10-09 | 华南农业大学 | Banana picking device and method based on depth vision |
| CN112425373A (en) * | 2020-12-02 | 2021-03-02 | 陕西中建建乐智能机器人股份有限公司 | Kiwi fruit picking and sorting robot and kiwi fruit sorting method thereof |
| CN113115638A (en) * | 2019-12-31 | 2021-07-16 | 季华实验室 | Intelligent tea picking robot |
| CN113273395A (en) * | 2021-05-21 | 2021-08-20 | 佛山市中科农业机器人与智慧农业创新研究院 | Cotton topping robot based on visual identification and implementation method thereof |
| CN113408410A (en) * | 2021-06-18 | 2021-09-17 | 重庆科技学院 | Traffic sign detection method based on YOLOv4 algorithm |
| CN214508055U (en) * | 2021-03-09 | 2021-10-29 | 甄圣超 | Mobile picking robot based on vision |
| CN113692854A (en) * | 2021-09-10 | 2021-11-26 | 江苏理工学院 | Tomato picking machine and control method thereof |
| EP3918904A1 (en) * | 2020-06-02 | 2021-12-08 | Exel Industries | Automated method for selecting and harvesting fruit and mechanical equipment implementing the method |
| CN113971775A (en) * | 2021-10-14 | 2022-01-25 | 江苏省电力试验研究院有限公司 | Optimized yolov4 algorithm-based violation behavior identification method and system |
| CN216087713U (en) * | 2021-11-10 | 2022-03-22 | 双髻鲨(威海)机器人科技有限公司 | Robot for automatically picking oranges |
-
2022
- 2022-04-13 CN CN202210384198.2A patent/CN114830915B/en active Active
Patent Citations (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004180554A (en) * | 2002-12-02 | 2004-07-02 | National Agriculture & Bio-Oriented Research Organization | Method and apparatus for selectively harvesting fruit vegetables |
| CN107094429A (en) * | 2017-07-03 | 2017-08-29 | 彭紫薇 | A kind of multiaxis fruit picking robot |
| JP2019037214A (en) * | 2017-08-29 | 2019-03-14 | 株式会社デンソー | End effector |
| CN207678305U (en) * | 2017-12-18 | 2018-08-03 | 艾力·艾散 | A kind of mechanical arm for walnut picking |
| CN208079866U (en) * | 2018-04-16 | 2018-11-13 | 苏州大学 | A kind of fruit automatic picking apparatus |
| CN208047324U (en) * | 2018-04-17 | 2018-11-06 | 石河子大学 | A kind of safflower picking end effector |
| CN108557143A (en) * | 2018-05-01 | 2018-09-21 | 浙江工业大学 | A kind of fruit picking packing machine |
| CN108575320A (en) * | 2018-05-23 | 2018-09-28 | 芮杰 | Strawberry picking robot |
| CN108834576A (en) * | 2018-06-07 | 2018-11-20 | 华南农业大学 | A kind of citrus picking robot and its implementation based on binocular vision |
| CN209047050U (en) * | 2018-08-17 | 2019-07-02 | 长沙理工大学 | A kind of Portable electric fruit picker |
| CN109328643A (en) * | 2018-12-17 | 2019-02-15 | 青岛科技大学 | A kind of self-balancing type apple-picking vehicle |
| CN109743966A (en) * | 2019-02-18 | 2019-05-14 | 德施普科技发展温州有限公司 | A kind of fruit picking |
| CN211020054U (en) * | 2019-09-16 | 2020-07-17 | 中国农业科学院农业信息研究所 | Fruit and vegetable picking device |
| CN110832991A (en) * | 2019-10-24 | 2020-02-25 | 广州大学 | Picking machine based on image recognition and control method thereof |
| CN211415245U (en) * | 2019-12-26 | 2020-09-04 | 青岛悟牛智能科技有限公司 | Picking robot |
| CN113115638A (en) * | 2019-12-31 | 2021-07-16 | 季华实验室 | Intelligent tea picking robot |
| CN111742709A (en) * | 2020-04-23 | 2020-10-09 | 华南农业大学 | Banana picking device and method based on depth vision |
| EP3918904A1 (en) * | 2020-06-02 | 2021-12-08 | Exel Industries | Automated method for selecting and harvesting fruit and mechanical equipment implementing the method |
| CN111673755A (en) * | 2020-06-24 | 2020-09-18 | 西北农林科技大学 | Picking robot control system and method based on visual servo |
| CN111709489A (en) * | 2020-06-24 | 2020-09-25 | 广西师范大学 | Citrus identification method based on improved YOLOv4 |
| CN112425373A (en) * | 2020-12-02 | 2021-03-02 | 陕西中建建乐智能机器人股份有限公司 | Kiwi fruit picking and sorting robot and kiwi fruit sorting method thereof |
| CN214508055U (en) * | 2021-03-09 | 2021-10-29 | 甄圣超 | Mobile picking robot based on vision |
| CN113273395A (en) * | 2021-05-21 | 2021-08-20 | 佛山市中科农业机器人与智慧农业创新研究院 | Cotton topping robot based on visual identification and implementation method thereof |
| CN113408410A (en) * | 2021-06-18 | 2021-09-17 | 重庆科技学院 | Traffic sign detection method based on YOLOv4 algorithm |
| CN113692854A (en) * | 2021-09-10 | 2021-11-26 | 江苏理工学院 | Tomato picking machine and control method thereof |
| CN113971775A (en) * | 2021-10-14 | 2022-01-25 | 江苏省电力试验研究院有限公司 | Optimized yolov4 algorithm-based violation behavior identification method and system |
| CN216087713U (en) * | 2021-11-10 | 2022-03-22 | 双髻鲨(威海)机器人科技有限公司 | Robot for automatically picking oranges |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115316129A (en) * | 2022-08-12 | 2022-11-11 | 华南农业大学 | Self-adaptive bionic picking device based on binocular vision recognition and cluster fruit picking method |
| CN115316129B (en) * | 2022-08-12 | 2024-03-12 | 华南农业大学 | Self-adaptive bionic picking device based on binocular vision recognition and fruit stringing picking method |
| CN115589849A (en) * | 2022-10-18 | 2023-01-13 | 大连理工大学（Cn） | Spherical fruit and vegetable picking device |
| CN115589849B (en) * | 2022-10-18 | 2023-06-30 | 大连理工大学 | Spherical fruit and vegetable picker |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114830915B (en) | 2023-09-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Mu et al. | Design and simulation of an integrated end-effector for picking kiwifruit by robot | |
| CN114830915B (en) | Litchi vision picking robot based on laser radar navigation and implementation method thereof | |
| US11304374B2 (en) | Selective automated blossom thinning | |
| Silwal et al. | Design, integration, and field evaluation of a robotic apple harvester | |
| Sarig | Robotics of fruit harvesting: A state-of-the-art review | |
| CN109197160B (en) | Guava picking robot and implementation method thereof | |
| CN108811766B (en) | Man-machine interactive greenhouse fruit and vegetable harvesting robot system and harvesting method thereof | |
| CN113330915B (en) | Self-adaptive cotton harvesting method based on binocular vision recognition and intelligent mechanical harvesting device | |
| CN115316129B (en) | Self-adaptive bionic picking device based on binocular vision recognition and fruit stringing picking method | |
| CN110089350A (en) | A kind of Mushroom Picking Robot system and picking method | |
| Zhaoxin et al. | Design a robot system for tomato picking based on YOLO v5 | |
| CN114260895B (en) | Method and system for determining grabbing obstacle avoidance direction of mechanical arm of picking robot | |
| Hu et al. | Simplified 4-DOF manipulator for rapid robotic apple harvesting | |
| CN115299245B (en) | Control method and control system of intelligent fruit picking robot | |
| CN108093868A (en) | A kind of picker system based on embedded intelligence control and binocular machine vision | |
| CN111684924A (en) | Strawberry picking robot based on raspberry group | |
| Ji et al. | Research on key technology of truss tomato harvesting robot in greenhouse | |
| CN116439018A (en) | Seven-degree-of-freedom fruit picking robot and picking method thereof | |
| Jin et al. | Detection method for table grape ears and stems based on a far-close-range combined vision system and hand-eye-coordinated picking test | |
| CN108182374A (en) | A kind of picking point recognition methods for fruit string | |
| Jin et al. | Intelligent tomato picking robot system based on multimodal depth feature analysis method | |
| CN206686630U (en) | One kind automation picker system | |
| Libin et al. | Overview of research on agricultural robot in China | |
| Hu et al. | Design and experiment of a new citrus harvesting robot | |
| CN212116264U (en) | Novel fruit is picked device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |