CN117016200A - Intelligent identification harvesting method suitable for broccoli - Google Patents

Intelligent identification harvesting method suitable for broccoli Download PDF

Info

Publication number
CN117016200A
CN117016200A CN202311043060.7A CN202311043060A CN117016200A CN 117016200 A CN117016200 A CN 117016200A CN 202311043060 A CN202311043060 A CN 202311043060A CN 117016200 A CN117016200 A CN 117016200A
Authority
CN
China
Prior art keywords
broccoli
harvesting
area
ball
balls
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311043060.7A
Other languages
Chinese (zh)
Inventor
王俊
康朔
杜冬冬
叶永凯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University ZJU
Original Assignee
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University ZJU filed Critical Zhejiang University ZJU
Priority to CN202311043060.7A priority Critical patent/CN117016200A/en
Publication of CN117016200A publication Critical patent/CN117016200A/en
Pending legal-status Critical Current

Links

Classifications

    • AHUMAN NECESSITIES
    • A01AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
    • A01DHARVESTING; MOWING
    • A01D46/00Picking of fruits, vegetables, hops, or the like; Devices for shaking trees or shrubs
    • A01D46/30Robotic devices for individually picking crops

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Environmental Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an intelligent identification and harvesting method suitable for broccoli, and belongs to the technical field of agricultural production. S1, collecting RGB-D images of a current harvesting area of a broccoli field, inputting the images into a target detection network, and identifying the maturity type of a broccoli ball; s2, determining the actual space position of the mature broccoli balls, and controlling a harvesting mechanism to harvest the broccoli balls positioned at the actual space position; s3, controlling the harvesting mechanism to move to a designated storage area, discharging according to the size of the broccoli balls in a grading manner, judging whether the current harvesting area has ripe broccoli which is not harvested, if so, returning to the step S2, otherwise, executing the step S4; and S4, moving the harvesting mechanism to the next harvesting area, and repeating the steps S1 to S3 to realize intelligent harvesting operation. The invention realizes the integration of detection, positioning, harvesting, grading, collecting, navigation and other processes in the selective harvesting operation of broccoli, and improves the harvesting efficiency and the intelligent degree.

Description

Intelligent identification harvesting method suitable for broccoli
Technical Field
The invention belongs to the technical field of agricultural production, and relates to an intelligent identification and harvesting method suitable for broccoli.
Background
Broccoli, also known as broccoli, green cauliflower, etc., belongs to the genus brassica of the family brassicaceae. Broccoli is mainly planted in a field, the consistency of maturity is generally poor, and the robot selective harvesting is a future trend for replacing a manual mode.
CN 20231008787367.0 discloses a broccoli combine harvester and a control method thereof, wherein an ultrasonic ranging sensor controls the cutting height, a stepping motor drives a disc to rotationally cut, but the mode can not carry out selective harvesting, and the harvesting method does not meet the agronomic requirements. CN 202210160153.7 discloses a broccoli harvesting device, but it cannot perform intelligent identification of the type of flower bulbs and automatic control of the whole harvesting process, and the degree of intellectualization is low.
Aiming at the problems of the existing broccoli harvesting mechanical control method, the intelligent, practical and efficient intelligent identification harvesting method suitable for broccoli is provided, the processes of detection, positioning, harvesting, grading, collecting, navigation and the like in operation are realized, the labor force is liberated, and the harvesting efficiency is improved, and the method has important significance.
Disclosure of Invention
The invention aims to overcome the problems in the prior art, and provides an intelligent identification harvesting method suitable for broccoli, which realizes the integration of detection, positioning, harvesting, grading, collection, navigation and other whole processes in selective harvesting operation, and improves the harvesting efficiency and the intelligent degree.
The technical scheme adopted for solving the technical problems is as follows:
an intelligent identifying and harvesting method suitable for broccoli comprises the following steps:
s1, acquiring RGB-D images of a current harvesting area of a broccoli field, inputting the RGB-D images into a target detection network, and identifying the maturity type of a broccoli ball; the target detection network comprises a target detection module, a maturity detection module and a size classification module;
s2, determining the actual space position of the mature broccoli balls, and controlling a harvesting mechanism to harvest the broccoli balls positioned at the actual space position;
s3, controlling the harvesting mechanism to move to a designated storage area, discharging according to the size of the broccoli balls in a grading manner, judging whether the current harvesting area has ripe broccoli which is not harvested, if so, returning to the step S2, otherwise, executing the step S4;
and S4, moving the harvesting mechanism to the next harvesting area, and repeating the steps S1 to S3 to realize intelligent harvesting operation.
Further, the RGB-D image of the current harvesting area of the broccoli field is obtained by vertically shooting at a position 60-90cm above crops, and the RGB-D image at least comprises one ridge of multiple rows of broccoli.
Further, the target detection module in the target detection network adopts an improved deep Lab-V3+ semantic segmentation network for identifying the broccoli and positioning the plane position coordinates of the top points of the broccoli;
the encoder of the semantic segmentation network consists of a trunk feature extraction network, a pooling module and an attention mechanism, wherein the trunk feature extraction network adopts a Mobile Net-V2, the pooling module adopts a densely connected cavity space pyramid pooling module, and the output of the pooling module is added with the space attention mechanism for adjusting the weight of a feature channel; the decoder of the semantic segmentation network adopts a decoder structure in DeepLab-V < 3+ >.
Further, the positioning broccoli sphere vertex plane position coordinates include:
marking the center point position of the broccoli ball in the RGB-D image according to the broccoli ball area identified by the target detection module;
the center point position of the broccoli ball in the RGB-D image is converted into the spatial position coordinates of the broccoli ball vertex by the hand-eye coordinate system conversion matrix and the camera parameter matrix for shooting the RGB-D image. The hand-eye coordinate system conversion matrix can be obtained through a hand-eye calibration method, is obtained by adopting an ax=xb model and performing optimization solution by using a solver based on a cost function, and belongs to common knowledge in the field, and the description of the hand-eye coordinate system conversion matrix is omitted.
Further, the maturity detection module and the size classification module in the target detection network take the broccoli area in the detection frame output by the target detection module as input, and the maturity detection module is used for classifying the maturity type of the pixel points of the broccoli area, and taking the maturity type with the most pixels as the maturity type of the broccoli; the size grading module is used for estimating the diameter size of the broccoli ball according to the pixel point area of the broccoli ball area and the shooting distance between the broccoli ball and the RGB-D image, and grading the size according to the estimated value.
Further, before the harvesting mechanism is controlled to harvest the broccoli balls positioned in the actual space position, all the mature broccoli balls identified in the current harvesting area are sequentially used as the broccoli balls to be harvested according to the sequence from left to right and from top to bottom.
Further, the harvesting mechanism is controlled to harvest the broccoli balls positioned at the actual space position, and the method comprises the following steps:
planning an optimal path from the bottom plane center position coordinate of a harvesting mechanism to the plane position coordinate of the top point of the broccoli to be harvested by adopting a fast search random tree algorithm, and re-planning the path if collision exists in the path planned for the first time;
controlling the harvesting mechanism to move to a target position along the planned path;
and after the harvesting mechanism reaches the target position, controlling the harvesting mechanism to vertically descend by 15cm, and clamping and cutting the broccoli balls to finish harvesting.
Further, the controlling the harvesting mechanism to move to the designated storage area comprises:
confirming the position coordinates of the corresponding designated storage area according to the diameter grade of the current picked broccoli;
planning an optimal path of a harvesting mechanism bottom plane center position coordinate to a position 5-15cm above the position coordinate of a designated storage area by adopting a fast search random tree algorithm;
controlling the harvesting mechanism to move to a target position along the planned path;
and discharging the broccoli balls after the harvesting mechanism reaches the target position.
Further, the moving the harvesting mechanism to the next harvesting area comprises:
shooting a front view image of the harvesting mechanism at an angle of-30 degrees to 0 degrees at 40 cm to 60cm above crops, wherein the front view image at least comprises one row of multiple rows of broccoli;
planning a navigation path with a forward distance not exceeding the width of the camera field of view for shooting RGB-D images based on the current harvesting area position;
and controlling the harvesting mechanism to advance to the next harvesting area along the navigation path, so as to realize unmanned and field sectional type circulating operation.
Further, a navigation line extraction algorithm and a Pure burst tracking algorithm of Lane Detection are adopted as navigation path planning algorithms.
Further, the harvesting mechanism is arranged on the mechanical arm, the mechanical arm drives the harvesting mechanism to move in space, and a camera-mechanical arm configuration with eyes outside hands is adopted; the camera field of view for taking RGB-D images should be smaller than the robotic harvesting workspace.
The invention has the beneficial effects that: the intelligent identifying and harvesting method is intelligent, practical and efficient, the target detection algorithm can rapidly and accurately obtain the maturity class of the broccoli and the size grade of the broccoli ball, the ripe broccoli is harvested and discharged to the designated position of the corresponding grade, collision is avoided through optimal path planning in discharging and harvesting processes, and the time is shortened; after the current harvesting area is finished, the automatic navigation is carried out to the next harvesting area, the integration of the processes of detection, positioning, harvesting, grading, collection, navigation and the like in the selective harvesting operation of the broccoli is realized, and the harvesting efficiency and the intelligent degree are improved.
Drawings
FIG. 1 is an overview of an intelligent identification harvesting method suitable for broccoli;
FIG. 2 is a specific flow chart of an intelligent identification harvesting method suitable for broccoli;
fig. 3 is a schematic diagram of a target detection network suitable for the intelligent recognition and harvesting method of broccoli.
Detailed Description
The invention is further illustrated in the following, but is not limited to, the accompanying drawings and specific examples:
as shown in fig. 1, an intelligent identification harvesting method suitable for broccoli comprises the following steps:
s1, acquiring RGB-D images of a current harvesting area of a broccoli field, inputting the RGB-D images into a target detection network, and identifying the maturity type of a broccoli ball;
s2, determining the actual space position of the mature broccoli balls, and controlling a harvesting mechanism to harvest the broccoli balls positioned at the actual space position;
s3, controlling the harvesting mechanism to move to a designated storage area, discharging according to the size of the broccoli balls in a grading manner, judging whether the current harvesting area has ripe broccoli which is not harvested, if so, returning to the step S2, otherwise, executing the step S4;
and S4, moving the harvesting mechanism to the next harvesting area, and repeating the steps S1 to S3 to realize intelligent harvesting operation.
As shown in fig. 2, in the step S1 of the intelligent identifying and harvesting method for broccoli, the type of the broccoli ball is identified through the target detection network, and an optional implementation process specifically includes the following steps:
s1.1, vertically shooting at a position 80cm above crops by adopting an RGB-D camera, wherein the field of view of the camera is equal to or smaller than a harvesting working space of a harvesting mechanism, taking the range in the field of view of the camera as a current harvesting area, and transmitting the shot image to a target detection network; the target detection network adopts improved deep Lab-V < 3+ >, takes broccoli balls as detection targets, and has maturity judging and size grading functions.
S1.2, outputting an identification result by the target detection network, wherein the identification result comprises the maturity types of all broccoli in the image, the plane position coordinates (X, Y) of the top points of the broccoli balls and the diameter D of the flower balls.
As shown in fig. 3, in the step S1.2, the target detection network mainly includes an improved deep lab-v3+ semantic segmentation network, a maturity determination module, and a size classification module. In some embodiments, in order to improve the accuracy and speed of identifying the maturity type of the broccoli balls, the encoder of the semantic segmentation network is composed of a trunk feature extraction network, a pooling module, an attention mechanism and the like, wherein the trunk feature extraction network is selected from Mobile Net-V2, the pooling module is replaced by a densely connected cavity space pyramid pooling module, the output with lower cavity rate is connected to a layer with higher cavity rate in a densely connected mode to be used as input, so that denser pixel sampling is obtained, and the network feature extraction capability is enhanced; the output of the pooling module is added with a spatial attention mechanism to adjust the weight share of the characteristic channel, so that the problem of weak expression of the broccoli image characteristics is solved, and the image segmentation accuracy of the model is improved; the decoder of the semantic segmentation network adopts a decoder structure in DeepLab-V < 3+ >.
And judging the maturity and the size grade according to the decoding result, wherein the maturity judging module judges the type of the current broccoli by comparing the number of the pixel point types, and the size grading module carries out conversion estimation of the actual size by using the pixel point area of the flower ball area and the distance between the flower ball and the camera.
In this embodiment, four maturity types are set: immature, semi-mature, over-mature, correspond to four pixel colors: green, blue, red, yellow; in the maturity judging module, classifying the maturity types of the pixel points of the broccoli ball areas, marking one broccoli ball area as a plurality of colors, and taking the maturity type with the most pixel points as the maturity type of the broccoli ball; and taking the broccoli balls classified as mature and overmature as objects to be harvested.
The size classification module estimates the diameter D of the broccoli ball according to the pixel point area of the broccoli ball area and the shooting distance between the broccoli ball and the RGB-D image, classifies the broccoli ball according to the diameter D of the broccoli ball, and distinguishes two commodity grades of the ripe broccoli ball by taking 15cm as a boundary.
As shown in fig. 2, in step S2 of the intelligent identifying and harvesting method for broccoli, a harvesting mechanism moving path is planned to complete harvesting of the broccoli balls at the positioned actual spatial position, and an optional implementation process specifically includes the following steps:
s2.1, selecting a mature broccoli ball identified in the current harvesting area as the broccoli ball to be harvested, and in the embodiment, sequentially taking all the broccoli balls identified in the current harvesting area as the broccoli ball to be harvested from left to right and from top to bottom.
S2.2, obtaining space position coordinates (X, Y, Z) of the vertexes of the broccoli balls to be harvested through a hand-eye coordinate system conversion matrix and RGB-D camera parameters. The hand-eye coordinate system conversion matrix is a conversion relation between the position of a plane pixel point in the RGB-D image and the actual space position in the harvesting mechanism coordinate system, and can be obtained through a TASI or Zhang Zhengyou calibration method.
S2.3, planning a path from the bottom of the harvesting mechanism to the position right above the spatial position of coordinates (X, Y, Z) according to the spatial position coordinates (X, Y, Z) of the top of the broccoli ball to be harvested and the position of the harvesting mechanism, and obtaining a collision-free path with optimal time and optimal path; in this embodiment, the harvesting mechanism is controlled to move by the mechanical arm, and the mechanical arm is controlled to move, so that the harvesting mechanism moves to the target harvesting position, and the optimal path is obtained by a fast search random tree algorithm in a path planning library of the mechanical arm operating system.
S2.4, controlling the harvesting mechanism to vertically descend by 15cm, wherein the height is the optimal cutting position of broccoli, and clamping and cutting the broccoli balls.
As shown in fig. 2, in the step S3, the harvesting mechanism is controlled to move to the designated storage area, and the harvesting mechanism is discharged in a grading manner according to the size of the broccoli balls, and an optional implementation process specifically includes the following steps:
s3.1, grading the diameter D of the flower ball according to the identification result of the target detection network, wherein in the embodiment, 15cm is used as a boundary, the diameter is larger than 15cm and is the mature first-stage broccoli, and the diameter is smaller than 15cm and is the mature second-stage broccoli;
s3.2, planning a path of the harvesting mechanism to reach a position 10cm above the corresponding grade collecting area according to the clamped and cut broccoli ball vertex space position coordinates (X, Y and Z) and the corresponding grade collecting area position; in this embodiment, the harvesting mechanism is controlled to move by the mechanical arm, and the mechanical arm is controlled to act, so that the harvesting mechanism moves to the position of the collection area of the corresponding grade, and the optimal path is obtained by a fast search random tree algorithm in a path planning library of an operating system of the mechanical arm.
And S3.3, the harvesting mechanism moves to 10cm above the corresponding grade collecting area according to the planned path, and discharging is completed.
And S3.4, circularly executing the harvesting action until the mature broccoli to be harvested does not exist in the field of view of the camera.
As shown in fig. 2, in the step S4, the recovery mechanism is moved to the next recovery area, and an optional implementation process specifically includes the following steps:
s4.1, the navigation camera acquires an image which is 50cm above the crop and is shot at-15 degrees, and the image field of view should contain the tops of broccoli balls on one row and two ridges, so that the crop line navigation line is conveniently extracted.
S4.2, generating a navigation path of the next harvesting area by adopting a field navigation algorithm according to the image shot in the step S4.1, wherein the field navigation algorithm comprises a navigation line extraction algorithm of Lane Detection and a Pure Pursuist tracking algorithm in the embodiment.
And S4.3, driving the harvesting mechanism to advance to the next harvesting area along the navigation path generated in the step S4.2, wherein the advancing distance is smaller than the width of the field of view of the camera.
The foregoing list is only illustrative of specific embodiments of the invention. Obviously, the invention is not limited to the above embodiments, but many variations are possible. All modifications directly derived or suggested to one skilled in the art from the present disclosure should be considered as being within the scope of the present invention.

Claims (3)

1. The intelligent identifying and harvesting method suitable for broccoli is characterized by comprising the following steps of:
s1, acquiring RGB-D images of a current harvesting area of a broccoli field, inputting the images into a target detection network, and identifying the maturity type of a broccoli ball; the target detection network comprises a target detection module, a maturity detection module and a size classification module; the RGB-D image of the current harvesting area of the broccoli field is obtained by vertically shooting at a position 60-90cm above crops, and the RGB-D image at least comprises one ridge of multiple rows of broccoli;
the target detection module in the target detection network adopts an improved DeepLab-V3+ semantic segmentation network, and is used for identifying the broccoli ball and positioning the plane position coordinates of the top point of the broccoli ball; the encoder of the semantic segmentation network consists of a trunk feature extraction network, a pooling module and an attention mechanism, wherein the trunk feature extraction network adopts a Mobile Net-V2, the pooling module adopts a densely connected cavity space pyramid pooling module, and the output of the pooling module is added with the space attention mechanism for adjusting the weight of a feature channel; the decoder of the semantic segmentation network adopts a decoder structure in DeepLab-V < 3+ >;
the maturity detection module and the size classification module in the target detection network take the broccoli area in the detection frame output by the target detection module as input, and the maturity detection module is used for classifying the maturity types of the pixel points of the broccoli area and taking the maturity type with the most pixels as the maturity type of the broccoli; the size grading module is used for estimating the diameter size of the broccoli ball according to the pixel point area of the broccoli ball area and the shooting distance between the broccoli ball and the RGB-D image, and grading the size according to the estimated value;
the positioning broccoli ball vertex plane position coordinates comprise: marking the center point position of the broccoli ball in the RGB-D image according to the broccoli ball area identified by the target detection module; converting the central point position of the broccoli ball in the RGB-D image into the spatial position coordinates of the broccoli ball vertex through a hand-eye coordinate system conversion matrix and a camera parameter matrix for shooting the RGB-D image;
s2, determining the actual space position of the mature broccoli balls, controlling a harvesting mechanism to harvest the broccoli balls positioned at the actual space position, and comprising the following steps:
planning an optimal path for the central position coordinate of the bottom plane of a harvesting mechanism to reach the position coordinate of the top plane of the broccoli ball to be harvested by adopting a fast search random tree algorithm;
controlling the harvesting mechanism to move to a target position along the planned path;
after the harvesting mechanism reaches the target position, controlling the harvesting mechanism to vertically descend by 15cm, and clamping and cutting the broccoli balls to finish harvesting;
s3, controlling the harvesting mechanism to move to a designated storage area, and discharging according to the size of the broccoli balls in a grading manner, wherein the harvesting mechanism comprises: confirming the position coordinates of the corresponding designated storage area according to the diameter grade of the current picked broccoli; planning an optimal path of a harvesting mechanism bottom plane center position coordinate to a position 5-15cm above the position coordinate of a designated storage area by adopting a fast search random tree algorithm; controlling the harvesting mechanism to move to a target position along the planned path; discharging the broccoli balls after the harvesting mechanism reaches the target position;
judging whether the current harvesting area has the ripe broccoli which is not harvested, if so, returning to the step S2, otherwise, executing the step S4;
s4, moving the harvesting mechanism to a next harvesting area, including: shooting a front view image of the harvesting mechanism at an angle of-30 degrees to 0 degrees at 40 cm to 60cm above crops, wherein the front view image at least comprises one row of multiple rows of broccoli; planning a navigation path with a forward distance not exceeding the width of the camera field of view for shooting RGB-D images based on the current harvesting area position; controlling the harvesting mechanism to advance to the next harvesting area along the navigation path;
and repeating the steps S1 to S3 to realize intelligent harvesting operation.
2. The intelligent identifying and harvesting method for broccoli according to claim 1, wherein all the ripe broccoli balls identified in the current harvesting area are sequentially used as the broccoli balls to be harvested in the order from left to right and from top to bottom before the harvesting mechanism is controlled to harvest the broccoli balls positioned at the actual space position.
3. The intelligent identifying and harvesting method for broccoli according to claim 1, wherein a navigation line extraction algorithm of Lane Detection and a Pure pulse tracking algorithm are adopted as a navigation path planning algorithm.
CN202311043060.7A 2023-08-18 2023-08-18 Intelligent identification harvesting method suitable for broccoli Pending CN117016200A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311043060.7A CN117016200A (en) 2023-08-18 2023-08-18 Intelligent identification harvesting method suitable for broccoli

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311043060.7A CN117016200A (en) 2023-08-18 2023-08-18 Intelligent identification harvesting method suitable for broccoli

Publications (1)

Publication Number Publication Date
CN117016200A true CN117016200A (en) 2023-11-10

Family

ID=88629712

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311043060.7A Pending CN117016200A (en) 2023-08-18 2023-08-18 Intelligent identification harvesting method suitable for broccoli

Country Status (1)

Country Link
CN (1) CN117016200A (en)

Similar Documents

Publication Publication Date Title
Yu et al. Real-time visual localization of the picking points for a ridge-planting strawberry harvesting robot
Sarig Robotics of fruit harvesting: A state-of-the-art review
Montoya-Cavero et al. Vision systems for harvesting robots: Produce detection and localization
Hayashi et al. Robotic harvesting system for eggplants
CN110122073B (en) Strawberry picking robot based on machine vision
CN112715162B (en) System for intelligent string type fruit of picking
CN108633482A (en) A kind of fruit picking aircraft
WO2023050783A1 (en) Weeding robot and method and apparatus for planning weeding path thereof, and medium
Gao et al. Image processing and analysis for autonomous grapevine pruning
You et al. Semiautonomous precision pruning of upright fruiting offshoot orchard systems: An integrated approach
Zhang et al. An automated apple harvesting robot—From system design to field evaluation
Jin et al. Far-near combined positioning of picking-point based on depth data features for horizontal-trellis cultivated grape
Parsa et al. Modular autonomous strawberry picking robotic system
Feng et al. Design and test of harvesting robot for table-top cultivated strawberry
CN117016200A (en) Intelligent identification harvesting method suitable for broccoli
Lehnert et al. A sweet pepper harvesting robot for protected cropping environments
Hu et al. Design and experiment of a new citrus harvesting robot
Goondram et al. Strawberry Detection using Mixed Training on Simulated and Real Data
Fujinaga Cutting point detection for strawberry fruit harvesting and truss pruning by agricultural robot
CN116391506A (en) Tomato fruit high-speed collection system and method and tomato fruit picking machine
Mhamed et al. Advances in apple’s automated orchard equipment: A comprehensive research
US20220264801A1 (en) Method of controlling a robotic harvesting device
CA3111952A1 (en) Mushroom harvesting vision system and method of harvesting mushrooms using said system
Shamshiri et al. An overview of visual servoing for robotic manipulators in digital agriculture
Wang et al. Study on Feedback and Correction of Tomato Picking Localization Information.

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination