US20210327133A1 - System and Method of Virtual Plant Field Modelling - Google Patents
System and Method of Virtual Plant Field Modelling Download PDFInfo
- Publication number
- US20210327133A1 US20210327133A1 US17/327,516 US202117327516A US2021327133A1 US 20210327133 A1 US20210327133 A1 US 20210327133A1 US 202117327516 A US202117327516 A US 202117327516A US 2021327133 A1 US2021327133 A1 US 2021327133A1
- Authority
- US
- United States
- Prior art keywords
- plant
- skeleton
- skeleton segment
- segment
- candidate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/05—Geographic models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/08—Volume rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
- G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
- G06T7/74—Determining position or orientation of objects or cameras using feature-based methods involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/56—Particle system, point based geometry or rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/20—Indexing scheme for editing of 3D models
- G06T2219/2008—Assembling, disassembling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
Definitions
- This description relates generally to the precision agriculture field, and more specifically to a new and useful virtual plant model generation system in the precision agriculture field.
- Identifying plants from captured images is beneficial for a number of agricultural purposes.
- individual plants are generally planted close in proximity to each other in order to maximize a desired outcome (e.g., maximize yield, protein percentage, or some other measurable quantity) while minimizing the amount of land that is needed to grow the crops.
- a desired outcome e.g., maximize yield, protein percentage, or some other measurable quantity
- existing image recognition systems experience difficulty when trying to identify plant matter than may appear to belong to multiple nearly overlapping plants. Often, they will mischaracterize plant growths as belonging to the wrong plant, or will misidentify how many plants are present in the field.
- FIG. 1A is a schematic representation of an example of generating a point cloud from plants identified within the set of images and generating skeleton segments from the point cloud.
- FIG. 1B is a schematic representation of an example of classifying generating skeleton segments as stalk bases using the set of images.
- FIGS. 2A, 2B, and 2C are schematic representations of a first, second, and third example of growing plant skeletons based on the skeleton segments classified as stalk bases.
- FIGS. 3 and 4 are schematic representations of a first and second example of downsampling the point cloud.
- FIG. 5 is a schematic representation of an example of recording a set of stereoimages for a plant, and performing the method using the stereoimages.
- FIG. 6 is a schematic representation of an example of the terrestrial system.
- FIG. 7 is a flow diagram representation of an example method for generating a virtual plant model.
- FIG. 8 is an example process for using labeled stalk bases of plants (or another unique plant feature) to carry out actions with respect to those plants.
- an example method for generating virtual models of plants in a field includes a number of steps that detect the presence and position of a unique plant feature within a 2D or 3D point or voxel space to serve as a starting point for creating a virtual model for an individual plant.
- the unique plant feature is a stalk base, however in other implementations other unique plant features may be used, such as the stalk itself, a location of where the stalk is planted in the ground, a flower, a trunk, an ear, a meristem, etc.
- the steps of the example method include recording a set of images of a set of plants in-situ; generating a point cloud from plants identified within the set of images; generating skeleton segments from the point cloud; classifying a subset of skeleton segments as stalk bases (or another unique plant feature) using the set of images; and growing plant skeletons from skeleton segments classified as stalk bases (or, again, another unique plant feature).
- the method may be used to generate a virtual model of a single, real plant, a portion of a real plant field (thus virtual models many real plants), and/or the entirety of the real plant field.
- the method builds a virtual model of a corn field, wherein the virtual model includes a virtual structure for all or most of the detected corn plants within the corn field.
- the virtual model can be analyzed to determine or estimate plant population parameters, such as density, number of plants within the plant field, average plant size, and plant uniformity, and terminal yield.
- the virtual model can also be analyzed to determine or estimate individual plant parameters, such as biomass, uniformity, susceptibility to nutrient, water, or biotic stress, leaf area index, biomass, plant temperature, stalk diameter and height, vigor, and possibly terminal yield on a plant-by-plant basis.
- the virtual model can also be used in aggregate across multiple plants to determine quantities such as stand count and aggregate yield.
- the virtual model can also be used for predictive purposes. In one example, one or more scenarios, such potential treatments or thinning practices, can be applied to the virtual model.
- the future values for yield, plant uniformity, or any other parameter can be determined from the projected results based on the virtual model.
- the method can be performed and/or the virtual model dynamically generated in real- or near-real time.
- the method can be performed as the recording and/or computing system for capturing images and performing the method travels along the field and/or as the imaging system is recording subsequent images, or 3D data.
- the method can be performed asynchronously with information collection (e.g., after all field images have been recorded), or be performed at any other suitable time.
- the virtual model can be generated by an in-situ system (e.g., the imaging system), by a remote system, or by any other suitable system.
- the method has a number of benefits.
- the method can decrease computation time and/or resources by identifying stalk segment candidates using the virtual model, but classifying the segments as stalks or non-stalks using the images (e.g., by using a hybrid approach).
- the method can enable quick navigation through a downsampled point cloud to identify the stalk candidates, where the downsampled point cloud can contain insufficient information for the image classification system, neural network, and/or other classification system to reliably classify the segments as stalks.
- the images, which have enough information for the neural network to reliably classify the segments, corresponding to the stalk segment candidates can then be used to classify the segments. This can increase the computation speed over pure virtual plant models because a downsampled point cloud can be used.
- the method can oversegment the plant when generating the initial skeleton segments. This can function to distinguish overlapping plants.
- An imaging system records a set of images of a set of plants in-situ to gather the raw information which can be used to generate point clouds, generate skeleton segments for classification, for plant parameter determination, or for any other suitable function.
- the images are preferably recorded in real time, as the system traverses through a real (physical) plant field (e.g., geographic region), but can alternatively be recorded after the system stops (e.g., within the field), or be recorded at any other suitable frequency or time.
- a recorded image can be a 2D image, a 3D image, a depth image, a scan (e.g., laser scan, ultrasound visualization, etc.), or be any other suitable image.
- the image can be collected using a variety of techniques, including ambient light imaging, laser imaging (LIDAR), hyperspectral imaging, radar imaging, etc.
- the wavelengths of light captured may include wavelengths in the visible range, near infrared range, and thermal (mid IR) range.
- the image can be a composite image (e.g., generated from a first and second image), a single image, a mosaicked image (e.g., compiled from multiple images), or be any other suitable image.
- the image preferably captures (and/or the imaging system field of view is arranged such that the recorded image includes) both a portion of the ground and a portion of the plant.
- the image captures a ground plane and the plant stalk or leaves proximal the ground.
- the image can capture the entirety of the plant, a subsection of the plant, or capture any other suitable portion of the plant.
- the set of images can only capture the portion of the plant proximal the ground.
- the method can include recording and processing a second set of images capturing the portions of the plant distal the ground, wherein the second set of images or resultant virtual model can be merged with the first set of images or first virtual model based on the geographic location of the images.
- Each image can capture features from one or more plants.
- the images are recorded by an imaging system.
- the imaging system can be arranged on a ground-based (terrestrial) system (e.g., as shown in FIG. 6 ), wherein the system can additionally include treatment mechanisms, computing systems, communication systems, or any other suitable subsystem.
- the imaging system can be an aerial system, aquatic system, or be any other suitable system.
- the imaging system can be a 2D imaging system, 3D imaging system, or be any other suitable imaging system. Examples of the imaging system include contact systems, non contact active systems (e.g., time of flight, triangulation, conoscopic holography, etc.), structured light systems, modulated light systems, non-contact passive systems (e.g., stereoscopic, photometric, silhouette, etc.) or any other suitable system.
- the imaging system may be configured to record a side view of the plants (e.g., wherein the imaging system is angled along a vector having a projection that is either perpendicular to a gravity vector or is at least at a non-zero angle with respect to the gravity vector), but can alternatively be configured to record a top-view of the plants, record both a side and a top view of the plants, or record an image of the plant from any other suitable angle.
- the system can include one or more imaging systems arranged in one or more positions.
- the system can additionally include a light shield or other ambient light shading element to create more controlled lighting environment.
- the system can be open to the ambient environment (e.g., record images at least partially using ambient light).
- the imaging system can be substantially similar to that disclosed in U.S. application Ser. No. 14/329,161, filed 11 Jul. 2014, incorporated herein in its entirety.
- the imaging system uses the set of images to generate a point cloud from plants identified within the set of images.
- the point cloud virtualizes/reduces an image to a set of virtual points, which functions as the basis from which the virtual model of a plant can be built.
- the point cloud may be a 3D point cloud (e.g., a set of points arranged in a 3D virtual space), a depth map, 2D point cloud, or be any other type of point cloud.
- the point cloud can be generated using pattern recognition on the image segments for which the cloud is to be generated, generated from a pattern projected onto the geographic region, or be otherwise generated.
- Generating the point cloud can additionally include identifying the ground within the image (identifying the ground plane), as shown in FIG. 1A .
- the ground plane is preferably represented in the virtual model, such as by a set of points, a vector, a plane, or a volume, but can alternatively remain unrepresented or be otherwise noted.
- the ground plane location can subsequently be used as a reference point for skeletonization and stalk candidate identification.
- the ground plane can be identified based on a set of predetermined features (e.g., determined by a trained machine learning module, determined as a substantially straight line, etc.), determined based on ground plane segmenting techniques (e.g., using a corner based approach, color cues, Gaussian Mixture Model based segmentation method, etc.), determined from the foreground, or identified in any other suitable manner.
- a set of predetermined features e.g., determined by a trained machine learning module, determined as a substantially straight line, etc.
- ground plane segmenting techniques e.g., using a corner based approach, color cues, Gaussian Mixture Model based segmentation method, etc.
- Generating the point cloud can additionally include identifying plants within the image, as shown in FIG. 1A .
- the plants can be identified as foreground features that do not qualify as the ground plane, identified based on a set of predetermined features (e.g., determined by a trained machine learning module, etc.), or be identified in any other suitable manner.
- the point cloud is generated for the entire image, wherein points corresponding to background, plant matter, and ground can be subsequently determined after the point cloud is generated.
- generating the point cloud includes segmenting the foreground from the background), generating the point cloud for the foreground, then identifying the ground plane and the plant.
- generating the point cloud includes identifying a ground plane within the set of images; identifying plants within the set of images; and generating point clouds from the plants identified within the images.
- Generating skeleton segments from the point cloud functions to generate preliminary plant segments, such that stalk candidates can be subsequently identified from the preliminary plant segments.
- Generating the skeleton segments preferably functions to generate a plurality of virtual skeleton segments separated by virtual boundaries, but can alternatively generate a single skeleton, or generate any suitable number of skeleton segments.
- Generating the skeleton segments preferably over segments the plants, wherein each plant is segmented into a plurality of segments, but can alternatively under-segment the plants (e.g., wherein two plants are part of the same skeleton segment), or otherwise segmented.
- Each segment preferably includes one or more connected nodes.
- the nodes can be points, voxels, or any other suitable virtual unit.
- the nodes are preferably connected by vectors, such that the segment forms a pointer structure, but can alternatively be connected by lines (e.g., without an associated direction), or be connected in any other suitable manner.
- the method can additionally include identifying nodes.
- the nodes can be the points of the generated point cloud.
- identifying nodes can include generating voxels from the point cloud, wherein the nodes can subsequently be one or more of the generated voxels. This can additionally function to downsample the point cloud.
- Each voxel preferably represents a volume of virtual space, but can alternatively represent a 2D section of virtual space or represent any other suitable unit of virtual space.
- the voxels can be generated by clustering points within a threshold distance of each other (e.g., within 1 millimeter, 1 centimeter, etc.) and defining a voxel about the clustered points, segmenting the virtual space covered by the point cloud into a uniform voxel grid (e.g., wherein each voxel is substantially the same size), segmenting the virtual space covered by the point cloud into a voxel grid, wherein each voxel encloses the same number of points (e.g., to normalize the points per volume between images of objects close to the camera and distal from the camera), or be generated in any other suitable manner.
- a threshold distance of each other e.g., within 1 millimeter, 1 centimeter, etc.
- Generating skeleton segments from the point cloud can include identifying starting nodes and growing the skeleton from the starting nodes, as shown in FIG. 1A .
- the skeleton segments can be generated using curve skeletonization (e.g., medial axis skeletonization of surface elements generated from the nodes, skeletonization of the distance field, skeletonization of the geodesic field, Voronoi diagram extraction, topology preserving iterative erosion, topological thinning, distance transform based technique, Laplacian based contraction, etc.), Blum's grassfire transform, morphological operators, morphological operators and shape-based pruning, intersections of distances from boundary sections, curve evolution, level sets, finding ridge points on the distance function, shape peeling until convergence without topology change, surface skeletonization, iterative skeleton expansion (e.g., by iteratively adding new branches that connect the furthest voxel to the current skeleton using a minimum cost geodesic path), or be generated in any other
- Identifying starting nodes functions to identify nodes from which the skeleton is grown.
- the starting node can be the node closest to the ground plane, such that the node is not connected to a second node on the side closest to the ground plane; nodes within a threshold virtual distance of the ground plane (e.g., 1 virtual meter, 1 virtual centimeter, etc.); edge nodes; nodes further than a threshold distance (virtual or real) from the next closest node (neighboring node) (e.g., 1 virtual centimeter, 1 virtual meter, etc.); nodes closest to or overlapping the ground plane; or be nodes having any other suitable characteristic.
- a threshold virtual distance of the ground plane e.g., 1 virtual meter, 1 virtual centimeter, etc.
- edge nodes nodes further than a threshold distance (virtual or real) from the next closest node (neighboring node) (e.g., 1 virtual centimeter, 1 virtual meter, etc.)
- the skeleton segment can be linear, be a surface, be a volume, or have any other suitable dimensionality.
- Each skeleton is preferably grown based on plant-specific characteristics (e.g., grown upwards away from the ground, only joined to nodes within a threshold distance, such as within 1 virtual centimeter, etc.), but can be grown based on any other suitable set of rules.
- the skeleton segment can be grown by iteratively joining nodes to the last node of the skeleton stemming from a starting node.
- the joined nodes are preferably virtually located within a threshold distance of the last node (e.g., within a virtual centimeter, meter, etc.), and located further away from the ground plane than the last node.
- One or more skeletons can be generated from each starting node.
- the skeletons from each starting node are preferably kept separate, but can alternatively join (e.g., such that a skeleton can stem from multiple starting nodes).
- the node is only joined to the skeleton if the vector joining the first and second nodes has a non-zero projection on a reference vector.
- the reference vector can be an measured or estimated gravity vector (e.g., as determined from the image, measured by an accelerometer, determined from the orientation of the recording image sensor, etc.), be a vector extending at an angle to the ground plane (e.g., normal to the ground plane, etc.), or be any other suitable reference vector.
- Skeleton growth can terminate when no other nodes satisfy these two conditions. This functions to generate virtual segments indicative of plant structures, such as stalks, leaves, and stems.
- the joined node can be the nearest node neighboring the last node of the skeleton segment, be joined using geodesic field generation, be joined based on a cost function, or be joined in any other suitable manner.
- the skeleton segments can be otherwise generated.
- the method can additionally include downsampling the point cloud, such that a reduced number of points and/or skeleton segments can be considered at later stages of the method.
- the point cloud can be downsampled before the skeleton segments are generated, after the skeleton segments are generated, or be downsampled at any other suitable time.
- point cloud can be downsampled by generating voxels from the point cloud, as discussed above (example shown in FIG. 4 ).
- the point cloud can be downsampled by clustering points within a threshold virtual distance together (example shown in FIG. 3 ).
- the point cloud can be downsampled by using a geodesic tracing technique, pruning, or applying any other suitable downsampling method.
- the plurality of skeletons generated from the starting nodes can be downsampled by selecting the best fit skeleton for the estimated plant section.
- the skeletons can be downsampled based on a cost function minimization, wherein a skeleton can have a higher cost if the skeleton is unbalanced (e.g., is biased heavily to one side), deviates from the set of typical profiles (e.g., straight lines, as exhibited by stems, loose curls as exhibited by leaves, etc.), or has any other suitable unusual plant characteristic.
- the skeletons can be otherwise downsampled.
- Classifying a subset of skeleton segments as stalk bases identifies the starting nodes and/or starting skeleton segments from which the virtual plant model should be generated. This enables the final plant skeleton to be generated from the bottom up (e.g., from the stalk up).
- Classifying a subset of skeleton segments as stalk bases can include: identifying the skeleton segments proximal the ground plane; identifying sections of the image corresponding to the identified skeleton segments; and identifying stalks from the identified image sections, an example of which is shown in FIG. 1B .
- the skeleton segments can be categorized as stalk bases based solely on the virtual model (e.g., based on the skeleton segment vector field, based on the geodesic structures formed by connecting the virtual nodes, etc.) or be categorized as stalk bases in any other suitable manner.
- the method can include classifying a subset of skeleton segments as plant apexes (e.g., plant tops), wherein the skeleton segments and subsequent final plant skeleton can be determined from the top down.
- the skeleton segments can be otherwise classified, and the virtual plant model otherwise generated.
- Identifying the skeleton segments proximal the ground plane functions to identify skeleton segments that potentially represent real plant stalks within the virtual space.
- the stalk candidates are preferably identified in the virtual space because the virtual model is faster and requires less computational power to sort through, as compared to stalk candidate identification from images, but the stalk candidates can be identified in any other suitable manner.
- the skeleton segments identified as potential candidates can be: skeleton segments at least partially located within a threshold virtual distance of the ground plane; skeleton segments that intersect with or touch the ground plane; skeleton segments satisfying a set of stalk conditions (e.g., have a threshold thickness, variable dependent upon the stage of plant growth, have a threshold straightness, etc.); or be any otherwise selected.
- Identifying sections of the image corresponding to the identified skeleton segments functions to identify high-resolution pieces of information suitable for use in classification. This can be desirable because, in some variations, the skeleton segments are downsampled to increase candidate identification speed, and therefore do not contain sufficient information for the classification module to reliably classify the segment. Furthermore, because the potential stalk candidates and/or corresponding volumes have already been identified using the virtual model, the method can reduce the number of image sections that need to be processed by focusing only on the image sections corresponding to the stalk candidates and surrounding areas, and ignoring the remainder of the image. This can translate to faster computation time and/or a smaller computation power requirement.
- an entire image corresponding to the virtual stalk candidate can be identified.
- a section of the image (e.g., less than the entire image) corresponding to the virtual stalk candidate can be identified.
- all images associated with the geographic volume associated with the stalk candidate can be identified.
- any other suitable image portion can be identified.
- the identified image section preferably encompasses the entirety of the corresponding virtual stalk candidate (identified skeleton segment), but can alternatively encompass a portion of the corresponding virtual stalk candidate (e.g., the top, bottom, or middle), or encompass any other suitable portion of the virtual stalk candidate.
- the image section preferably includes a portion of the ground plane proximal the corresponding virtual stalk candidate, but can alternatively exclude the ground plane.
- the image section is preferably wider than the corresponding virtual stalk candidate, such that the image includes a portion of the ambient environment next to the virtual stalk candidate, but can alternatively be as wide as the corresponding virtual stalk candidate (e.g., cropped at the virtually defined stalk boundaries) or be thinner than the corresponding virtual stalk candidate.
- the image section size can be dynamically determined based on the virtual dimensions of the virtual stock candidate.
- the image section size can be predetermined (e.g., fixed). However, any other suitable portion of the image can be identified as the image section.
- the identified image sections preferably correspond to the virtual stalk candidate.
- the image section correspondence with the identified skeleton segment is determined based on substantially matching respective geographic identifiers.
- a virtual stalk candidate can be associated with a set of geographic locations (e.g., latitude/longitude coordinates, altitude, etc.) based on the base image(s) from which the model was generated, wherein the identified image section is also associated with substantially the same or a larger set of geographic locations.
- the geographic locations can be associated with the virtual model and/or images using odometry systems, such as the one disclosed in U.S. application Ser. No. 14/629,361 filed 23 Feb. 2015, incorporated herein in its entirety by this reference.
- the image section correspondence with the identified skeleton segment is determined based on the location of the skeleton segment within the source image(s) from which the nodes in skeleton segment was generated.
- the image section can be identified by aligning a virtual reference point with a corresponding reference point on the source image and identifying the image section that overlaps with the stalk candidate.
- the virtual reference point can be a vector, point, volume, or other reference point.
- the image reference point can be an image edge, image pixel (e.g., arranged at a predetermined location relative to the image edges or center), or be any other suitable reference point.
- the reference point can be a plant feature represented in the virtual space and the image. However, any other suitable reference point can be used.
- the identified image is preferably a set of images, including a left image and a right image of the plant section corresponding to the virtual stalk candidate.
- the left and right images can cooperatively form a depth image, wherein a segment of the depth image can be identified and used to classify the stalk candidate.
- the set of images can additionally or alternatively include a depth image (e.g., a scan, stereoscopic image, etc.) composited with the other images, wherein the depth image can be recorded from the same or similar angle (relative to the plant) as the right and left images, or be recorded from a different angle (e.g., from the top of the plant).
- a single image or any other suitable set of images can be used.
- the images are preferably those used to generate the point cloud, but can alternatively be different images.
- Identifying stalks from the identified image sections functions to classify each virtual stalk candidate as a plant stalk or non-stalk.
- the candidate skeleton segments can additionally or alternatively be classified as dualities (e.g., as single plants, two plants, multiple plants, etc.), leaves, or be otherwise classified.
- the virtual stalk candidates are preferably automatically classified by a classification module running on the in-field system or a remote system, but can alternatively be classified in any other suitable manner.
- the classification module can simultaneously classify multiple stalk candidates, a single stalk candidate, or concurrently classify any number of stalk candidates.
- the classification module is preferably a convolutional neural network (CNN), but can alternatively or additionally be a deep neural network (DNN), feedforward neural network, radial basis function (RBF) network, Kohonen self organizing network, Learning Vector Quantization, recurrent neural network, modular neural network, physical neural network, holographic associative memory, instantaneously trained network, spiking neural network, dynamic neural network, cascading neural network, neuro-fuzzy network, compositional pattern-producing network, or be any other suitable artificial neural network.
- the classification module can be any other suitable machine learning module (e.g., including SIFT features with the support vector machine [SVM]), or be any other suitable classification module.
- the plant skeleton is preferably virtually grown from the skeleton segments classified as stalk bases by the classification module (examples shown in FIG. 2A , FIG. 2C ), but can alternatively be grown from any other suitable set of skeleton segments.
- the plant skeleton can be virtually grown from the starting nodes of the skeleton segments classified as stalk bases (example shown in FIG. 2B ), or be grown from any other suitable virtual structure.
- the plant skeleton can be iteratively generated, selected from multiple different plant skeletons candidates (e.g., based on the respective probability that the skeleton candidate represents a plant), or determined in any other suitable manner.
- the plant skeleton can be selected from the plurality of possible skeletons to maximize the plant probability for each plant, for the virtual plant population, or be selected in any other suitable manner.
- the finalized plant skeleton is generated from the previously generated skeleton segments.
- the plant skeleton can be iteratively generated upwards from the stalk skeleton by joining skeleton segments within a threshold distance of the last plant skeleton segment (e.g., within a virtual centimeter, meter, number of voxels, such as one voxel, etc.).
- subsequent skeleton segments can be iteratively joined (e.g., virtually annealed) to the last segments of the skeleton when the subsequent segment is within a threshold virtual or real distance (different or similar to the threshold distance used to generate the segments) of the last skeleton segment, and/or when the subsequent segment is further than the last segment from the ground plane.
- the subsequent segment can be joined when the subsequent segment is further from the starting segment than the last segment.
- the subsequent segment can be joined when the probability that the resultant skeleton represents a plant (e.g., based on the skeleton shape uniformity, position of the resultant center of mass, resultant plant size as compared to an expected plant size based on the actual or estimated growth stage, etc.) is beyond a threshold probability (e.g., 75%, 90%, etc.).
- a threshold probability e.g. 75%, 90%, etc.
- the plant skeleton can be entirely re-generated, wherein a new skeleton can be virtually grown, independent of the previously generated skeleton segments (example shown in FIG. 2A ).
- the new skeleton can be grown from the starting node of the old skeleton segment that was classified as a stalk, grown from the skeleton segment classified as a stalk (e.g., wherein the skeleton segment is the only re-used virtual component), or be otherwise generated.
- Growing the virtual plant skeleton can additionally include joining point clouds or voxel groups representative of leaves, fruit, or other plant components to the plant skeleton (example shown in FIG. 2A ).
- the virtual plant component representation can be joined as part of the final plant skeletonization process, or be joined after the plant skeleton representative of the stalk and/or stems is finalized.
- the virtual plant component representation is treated the same as all other plant segments.
- the virtual plant component representations are classified as non-stalk or non-stem (e.g., by a classification module), segmented out of the virtual model used to generate the plant skeleton, then reconsidered after the plant skeleton has been generated.
- Each plant cluster is preferably joined to a single plant skeleton, but can alternatively be joined to multiple plant skeletons.
- the plant clusters can be joined to the closest plant skeleton (e.g., touching or within a threshold distance), be joined based on the probability that the resultant virtual plant is representative of a real plant (e.g., based on the resultant center of mass, shape uniformity, size of the cluster relative to the size of the stalk, etc.), or be joined based on the satisfaction of any other suitable characteristic.
- multiple virtual plant options (each with a different combination of joined clusters) can be generated for each plant skeleton, wherein the plant option with the highest probability of representing a real plant can be selected. Alternatively, the plant options maximizing the probability that the virtual plant population represent real plants are selected.
- a single virtual plant with a single cluster combination can be generated. However, the clusters can be otherwise joined to the virtual plant skeleton.
- FIG. 8 is an example process 800 for using labeled stalks of plants to carry out actions with respect to those plants.
- the positions of the unique plant features such as stalks or stalk bases
- other analyses or processes may be carried out based on the information. These include, for example, generating three-dimensional models of plants growing in the field, assessing plant growth and phenotypic features, determining what kinds of treatments to apply including both where to apply the treatments and how much, determining whether to remove weeds or other undesirable plants, and so on.
- the position in the field of unique plant features, and thus each individual plant, is the starting point for performing any of these processes on a plant-by-plant basis.
- FIG. 8 illustrates that the locations of the unique plant features 802 may be passed as an input to plant modeling computer code 804 as described above.
- Modeling may further include mosaicking the generated models, such that a virtual model of a geographic region can be generated from multiple images of geographic region sub-plots.
- the images can be mosaicked (e.g., based on geographic location, estimated image position relative to the preceding image, reference points within the image, etc.), such that the point cloud on which the method is performed is already representative of the entire geographic region to be modeled.
- the virtual plant models can be individually generated or be generated in subsets. Multiple individual virtual plant models and/or plant model subsets can be subsequently merged or arranged within the virtual space based on the relative locations of the respective underlying images. However, the virtual model of the plant field can be otherwise generated.
- the virtual model is accessed by plant analysis computer code that analyzes 806 the virtual model to identify plant parameters of individual plants, and/or aggregate statistics regarding the modeled plants.
- the plant parameters are preferably determined solely from the virtual model, but can alternatively be verified or cooperatively determined based on the underlying source images.
- Aggregate statistics may be generated for a plant and a subset of its nearest neighbors, an entire row of plants, an entire field of plants, or any sub-zone thereof. Examples of aggregate plant parameters that may be identified include plant density, a number of plants within the plant field, plant size (e.g., average height, median height, height distribution over a population of plants), plant greenness (e.g., based on the colors in the image pixels corresponding to the virtual plant), and plant uniformity, and terminal yield.
- Examples of individual plant parameters include biomass, greenness, uniformity, leaf area or volume (e.g., based on the volume or surface area of the point clusters representative of leaves), susceptibility to nutrient, water, or biotic stress, leaf area, stalk diameter (e.g., based on the virtual diameter of the plant skeleton), height, and terminal yield on a plant-by-plant basis.
- the individual or aggregate plant parameters are accessed by plant action computer code that analyzes 808 the parameters to determine what action to take with respect to the plants in the field. These determinations may be made on the basis of parameters of individual plants, such that separate and different action can be taken on each individual plant in the field. These determinations may also be made on the basis on aggregate parameters of more than one plant, such as the parameters of a plant and its nearby neighbors, or in aggregate across an entire row, field, or any sub-zone thereof.
- the determined actions 808 may include potential treatments, such as the application of fertilizer, pesticides or other chemicals.
- the determined actions 908 may also include thinning practices, such as the removal of plants identified as weeds, or thinning to remove undesired plants.
- the determined actions may be provided 810 to a physical implement (not shown) attached to a device (e.g., tractor, combine, other vehicle, UAV, etc.), to carry out the determined action.
- a physical implement e.g., tractor, combine, other vehicle, UAV, etc.
- the provided action may include timing of when to spray, what chemical to spray, how much to spray, for how long, and at what flow rate.
- the implement may then carry out 810 the determined action.
- the providing step may include transmitting the determined action from one device/system to another. This may be a wired or wireless transmission, according to any known transmission protocol, such as by radio frequency transmission (e.g., WiFi, Bluetooth), infrared transmission, Ethernet or CAN bus transmission, etc.
- all of the process steps of FIG. 8 can be performed in real- or near-real time.
- the method can be performed as the device/vehicle/drone etc. travels along a row of the field.
- a camera may capture images, which may be passed to an onboard system to identify the unique plant features 802 , model plants 804 , identify parameters 806 , and determine actions 808 , and provide 810 those actions to an implement of the vehicle.
- All steps 802 - 810 are capable of being performed in a time between when the cameras capture the images of the plants for whom action 808 is to be determined, and when the implement to carry out those action passes by those same plants along the field.
- the vehicle may be traveling at a speed between 1 and 20 miles per hour, and the camera may be capturing images at a frequency within an order of magnitude of 30 Hertz.
- the physical distance between the cameras and the implement that performs the action may be approximately a foot, and/or within an order of magnitude of a foot.
- the process of FIG. 8 can be carried out before the vehicle travels the distance between the cameras and the implement.
- the device/vehicle can completely carry out the steps FIG. 8 in a time range between 40 and 100 milliseconds. This entirely eliminates the need for offboard processing, and allows action while the device/vehicle is in the field, rather than requiring a first pass to identify what to do, and a second pass to actually do it.
- the process of FIG. 8 can be carried out over different time periods. For example, image collection by the cameras may occur during a first time period as the device travels through the field. Unique feature detection 802 , and the other processing steps 804 - 808 may occur over a second time period later in time, such as in an office or away from the field. The action to be taken 810 may be performed during a third time, such as when the same or a different device/vehicle/UAV travels through the field to carry out the action.
- the processes described above may be embodied in computer program instructions stored within a non-transitory computer-readable storage medium, and designed to be executed by one or more computer processors within one or more computing devices.
- the non-transitory computer-readable medium can be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device.
- the computer-executable component is preferably a processor but the instructions may alternatively or additionally be executed by any suitable dedicated hardware device.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Remote Sensing (AREA)
- Image Analysis (AREA)
- Multimedia (AREA)
Abstract
Description
- This application is a continuation of U.S. patent application Ser. No. 16/984,026, filed on Aug. 3, 2020, which is a continuation of U.S. patent application Ser. No. 15/996,810, filed on Jun. 4, 2018, which is a continuation of application U.S. patent application Ser. No. 15/157,997, filed on May 18, 2016, which claims the benefit of U.S. Provisional Application No. 62/163,147, filed May 18, 2015, all of these applications are incorporated herein by reference in their entirety for all purposes.
- This invention was made with government support under Phase II SBIR contract NSF #1256596 with the National Science Foundation. The government has certain rights in the invention.
- This description relates generally to the precision agriculture field, and more specifically to a new and useful virtual plant model generation system in the precision agriculture field.
- Identifying plants from captured images is beneficial for a number of agricultural purposes. However, individual plants are generally planted close in proximity to each other in order to maximize a desired outcome (e.g., maximize yield, protein percentage, or some other measurable quantity) while minimizing the amount of land that is needed to grow the crops. Based on this, it is common for the leaves, branches, and other growths of a plant to overlap with other nearby plants. As these growths are usually both numerous and roughly similar in appearance from plant to plant, existing image recognition systems experience difficulty when trying to identify plant matter than may appear to belong to multiple nearly overlapping plants. Often, they will mischaracterize plant growths as belonging to the wrong plant, or will misidentify how many plants are present in the field.
-
FIG. 1A is a schematic representation of an example of generating a point cloud from plants identified within the set of images and generating skeleton segments from the point cloud. -
FIG. 1B is a schematic representation of an example of classifying generating skeleton segments as stalk bases using the set of images. -
FIGS. 2A, 2B, and 2C are schematic representations of a first, second, and third example of growing plant skeletons based on the skeleton segments classified as stalk bases. -
FIGS. 3 and 4 are schematic representations of a first and second example of downsampling the point cloud. -
FIG. 5 is a schematic representation of an example of recording a set of stereoimages for a plant, and performing the method using the stereoimages. -
FIG. 6 is a schematic representation of an example of the terrestrial system. -
FIG. 7 is a flow diagram representation of an example method for generating a virtual plant model. -
FIG. 8 is an example process for using labeled stalk bases of plants (or another unique plant feature) to carry out actions with respect to those plants. - As shown in
FIG. 7 , an example method for generating virtual models of plants in a field includes a number of steps that detect the presence and position of a unique plant feature within a 2D or 3D point or voxel space to serve as a starting point for creating a virtual model for an individual plant. In the example of this description, the unique plant feature is a stalk base, however in other implementations other unique plant features may be used, such as the stalk itself, a location of where the stalk is planted in the ground, a flower, a trunk, an ear, a meristem, etc. - The steps of the example method include recording a set of images of a set of plants in-situ; generating a point cloud from plants identified within the set of images; generating skeleton segments from the point cloud; classifying a subset of skeleton segments as stalk bases (or another unique plant feature) using the set of images; and growing plant skeletons from skeleton segments classified as stalk bases (or, again, another unique plant feature).
- The method may be used to generate a virtual model of a single, real plant, a portion of a real plant field (thus virtual models many real plants), and/or the entirety of the real plant field. In a specific example, the method builds a virtual model of a corn field, wherein the virtual model includes a virtual structure for all or most of the detected corn plants within the corn field.
- The virtual model can be analyzed to determine or estimate plant population parameters, such as density, number of plants within the plant field, average plant size, and plant uniformity, and terminal yield. The virtual model can also be analyzed to determine or estimate individual plant parameters, such as biomass, uniformity, susceptibility to nutrient, water, or biotic stress, leaf area index, biomass, plant temperature, stalk diameter and height, vigor, and possibly terminal yield on a plant-by-plant basis. The virtual model can also be used in aggregate across multiple plants to determine quantities such as stand count and aggregate yield. The virtual model can also be used for predictive purposes. In one example, one or more scenarios, such potential treatments or thinning practices, can be applied to the virtual model. The future values for yield, plant uniformity, or any other parameter can be determined from the projected results based on the virtual model.
- The method can be performed and/or the virtual model dynamically generated in real- or near-real time. For example, the method can be performed as the recording and/or computing system for capturing images and performing the method travels along the field and/or as the imaging system is recording subsequent images, or 3D data. Alternatively, the method can be performed asynchronously with information collection (e.g., after all field images have been recorded), or be performed at any other suitable time. The virtual model can be generated by an in-situ system (e.g., the imaging system), by a remote system, or by any other suitable system.
- The method has a number of benefits. As will be described further below, the method can decrease computation time and/or resources by identifying stalk segment candidates using the virtual model, but classifying the segments as stalks or non-stalks using the images (e.g., by using a hybrid approach). In particular, the method can enable quick navigation through a downsampled point cloud to identify the stalk candidates, where the downsampled point cloud can contain insufficient information for the image classification system, neural network, and/or other classification system to reliably classify the segments as stalks. The images, which have enough information for the neural network to reliably classify the segments, corresponding to the stalk segment candidates can then be used to classify the segments. This can increase the computation speed over pure virtual plant models because a downsampled point cloud can be used. This can increase the computation speed over pure image-based methods because the virtual model is used to decrease the number of image segments to be considered by the neural network. Second, the method can oversegment the plant when generating the initial skeleton segments. This can function to distinguish overlapping plants.
- An imaging system records a set of images of a set of plants in-situ to gather the raw information which can be used to generate point clouds, generate skeleton segments for classification, for plant parameter determination, or for any other suitable function. The images are preferably recorded in real time, as the system traverses through a real (physical) plant field (e.g., geographic region), but can alternatively be recorded after the system stops (e.g., within the field), or be recorded at any other suitable frequency or time.
- A recorded image can be a 2D image, a 3D image, a depth image, a scan (e.g., laser scan, ultrasound visualization, etc.), or be any other suitable image. The image can be collected using a variety of techniques, including ambient light imaging, laser imaging (LIDAR), hyperspectral imaging, radar imaging, etc. The wavelengths of light captured may include wavelengths in the visible range, near infrared range, and thermal (mid IR) range. The image can be a composite image (e.g., generated from a first and second image), a single image, a mosaicked image (e.g., compiled from multiple images), or be any other suitable image. The image preferably captures (and/or the imaging system field of view is arranged such that the recorded image includes) both a portion of the ground and a portion of the plant. In one example, the image captures a ground plane and the plant stalk or leaves proximal the ground. The image can capture the entirety of the plant, a subsection of the plant, or capture any other suitable portion of the plant. In one variation, the set of images can only capture the portion of the plant proximal the ground. In a second variation, the method can include recording and processing a second set of images capturing the portions of the plant distal the ground, wherein the second set of images or resultant virtual model can be merged with the first set of images or first virtual model based on the geographic location of the images. Each image can capture features from one or more plants.
- The images are recorded by an imaging system. The imaging system can be arranged on a ground-based (terrestrial) system (e.g., as shown in
FIG. 6 ), wherein the system can additionally include treatment mechanisms, computing systems, communication systems, or any other suitable subsystem. Alternatively, the imaging system can be an aerial system, aquatic system, or be any other suitable system. The imaging system can be a 2D imaging system, 3D imaging system, or be any other suitable imaging system. Examples of the imaging system include contact systems, non contact active systems (e.g., time of flight, triangulation, conoscopic holography, etc.), structured light systems, modulated light systems, non-contact passive systems (e.g., stereoscopic, photometric, silhouette, etc.) or any other suitable system. The imaging system may be configured to record a side view of the plants (e.g., wherein the imaging system is angled along a vector having a projection that is either perpendicular to a gravity vector or is at least at a non-zero angle with respect to the gravity vector), but can alternatively be configured to record a top-view of the plants, record both a side and a top view of the plants, or record an image of the plant from any other suitable angle. The system can include one or more imaging systems arranged in one or more positions. The system can additionally include a light shield or other ambient light shading element to create more controlled lighting environment. Alternatively, the system can be open to the ambient environment (e.g., record images at least partially using ambient light). In one variation, the imaging system can be substantially similar to that disclosed in U.S. application Ser. No. 14/329,161, filed 11 Jul. 2014, incorporated herein in its entirety. - As shown in
FIG. 1A , the imaging system uses the set of images to generate a point cloud from plants identified within the set of images. The point cloud virtualizes/reduces an image to a set of virtual points, which functions as the basis from which the virtual model of a plant can be built. The point cloud may be a 3D point cloud (e.g., a set of points arranged in a 3D virtual space), a depth map, 2D point cloud, or be any other type of point cloud. The point cloud can be generated using pattern recognition on the image segments for which the cloud is to be generated, generated from a pattern projected onto the geographic region, or be otherwise generated. - Generating the point cloud can additionally include identifying the ground within the image (identifying the ground plane), as shown in
FIG. 1A . The ground plane is preferably represented in the virtual model, such as by a set of points, a vector, a plane, or a volume, but can alternatively remain unrepresented or be otherwise noted. The ground plane location can subsequently be used as a reference point for skeletonization and stalk candidate identification. The ground plane can be identified based on a set of predetermined features (e.g., determined by a trained machine learning module, determined as a substantially straight line, etc.), determined based on ground plane segmenting techniques (e.g., using a corner based approach, color cues, Gaussian Mixture Model based segmentation method, etc.), determined from the foreground, or identified in any other suitable manner. - Generating the point cloud can additionally include identifying plants within the image, as shown in
FIG. 1A . The plants can be identified as foreground features that do not qualify as the ground plane, identified based on a set of predetermined features (e.g., determined by a trained machine learning module, etc.), or be identified in any other suitable manner. - In one variation, the point cloud is generated for the entire image, wherein points corresponding to background, plant matter, and ground can be subsequently determined after the point cloud is generated. In a second variation, generating the point cloud includes segmenting the foreground from the background), generating the point cloud for the foreground, then identifying the ground plane and the plant. In a third variation, generating the point cloud includes identifying a ground plane within the set of images; identifying plants within the set of images; and generating point clouds from the plants identified within the images.
- Generating skeleton segments from the point cloud functions to generate preliminary plant segments, such that stalk candidates can be subsequently identified from the preliminary plant segments. Generating the skeleton segments preferably functions to generate a plurality of virtual skeleton segments separated by virtual boundaries, but can alternatively generate a single skeleton, or generate any suitable number of skeleton segments. Generating the skeleton segments preferably over segments the plants, wherein each plant is segmented into a plurality of segments, but can alternatively under-segment the plants (e.g., wherein two plants are part of the same skeleton segment), or otherwise segmented. Each segment preferably includes one or more connected nodes. The nodes can be points, voxels, or any other suitable virtual unit. The nodes are preferably connected by vectors, such that the segment forms a pointer structure, but can alternatively be connected by lines (e.g., without an associated direction), or be connected in any other suitable manner.
- In one variation, the method can additionally include identifying nodes. In a first variation, the nodes can be the points of the generated point cloud. In a second variation, identifying nodes can include generating voxels from the point cloud, wherein the nodes can subsequently be one or more of the generated voxels. This can additionally function to downsample the point cloud. Each voxel preferably represents a volume of virtual space, but can alternatively represent a 2D section of virtual space or represent any other suitable unit of virtual space. The voxels can be generated by clustering points within a threshold distance of each other (e.g., within 1 millimeter, 1 centimeter, etc.) and defining a voxel about the clustered points, segmenting the virtual space covered by the point cloud into a uniform voxel grid (e.g., wherein each voxel is substantially the same size), segmenting the virtual space covered by the point cloud into a voxel grid, wherein each voxel encloses the same number of points (e.g., to normalize the points per volume between images of objects close to the camera and distal from the camera), or be generated in any other suitable manner.
- Generating skeleton segments from the point cloud can include identifying starting nodes and growing the skeleton from the starting nodes, as shown in
FIG. 1A . However, the skeleton segments can be generated using curve skeletonization (e.g., medial axis skeletonization of surface elements generated from the nodes, skeletonization of the distance field, skeletonization of the geodesic field, Voronoi diagram extraction, topology preserving iterative erosion, topological thinning, distance transform based technique, Laplacian based contraction, etc.), Blum's grassfire transform, morphological operators, morphological operators and shape-based pruning, intersections of distances from boundary sections, curve evolution, level sets, finding ridge points on the distance function, shape peeling until convergence without topology change, surface skeletonization, iterative skeleton expansion (e.g., by iteratively adding new branches that connect the furthest voxel to the current skeleton using a minimum cost geodesic path), or be generated in any other suitable manner. - Identifying starting nodes functions to identify nodes from which the skeleton is grown. The starting node can be the node closest to the ground plane, such that the node is not connected to a second node on the side closest to the ground plane; nodes within a threshold virtual distance of the ground plane (e.g., 1 virtual meter, 1 virtual centimeter, etc.); edge nodes; nodes further than a threshold distance (virtual or real) from the next closest node (neighboring node) (e.g., 1 virtual centimeter, 1 virtual meter, etc.); nodes closest to or overlapping the ground plane; or be nodes having any other suitable characteristic.
- Growing a skeleton segment from a starting node functions to generate a set of preliminary plant segments. The skeleton segment can be linear, be a surface, be a volume, or have any other suitable dimensionality. Each skeleton is preferably grown based on plant-specific characteristics (e.g., grown upwards away from the ground, only joined to nodes within a threshold distance, such as within 1 virtual centimeter, etc.), but can be grown based on any other suitable set of rules. The skeleton segment can be grown by iteratively joining nodes to the last node of the skeleton stemming from a starting node. The joined nodes are preferably virtually located within a threshold distance of the last node (e.g., within a virtual centimeter, meter, etc.), and located further away from the ground plane than the last node. One or more skeletons can be generated from each starting node. The skeletons from each starting node are preferably kept separate, but can alternatively join (e.g., such that a skeleton can stem from multiple starting nodes).
- In one example, as shown in
FIG. 1A , the node is only joined to the skeleton if the vector joining the first and second nodes has a non-zero projection on a reference vector. The reference vector can be an measured or estimated gravity vector (e.g., as determined from the image, measured by an accelerometer, determined from the orientation of the recording image sensor, etc.), be a vector extending at an angle to the ground plane (e.g., normal to the ground plane, etc.), or be any other suitable reference vector. Skeleton growth can terminate when no other nodes satisfy these two conditions. This functions to generate virtual segments indicative of plant structures, such as stalks, leaves, and stems. Alternatively, the joined node can be the nearest node neighboring the last node of the skeleton segment, be joined using geodesic field generation, be joined based on a cost function, or be joined in any other suitable manner. However, the skeleton segments can be otherwise generated. - The method can additionally include downsampling the point cloud, such that a reduced number of points and/or skeleton segments can be considered at later stages of the method. The point cloud can be downsampled before the skeleton segments are generated, after the skeleton segments are generated, or be downsampled at any other suitable time. In a first variation, point cloud can be downsampled by generating voxels from the point cloud, as discussed above (example shown in
FIG. 4 ). In a second variation, the point cloud can be downsampled by clustering points within a threshold virtual distance together (example shown inFIG. 3 ). However, the point cloud can be downsampled by using a geodesic tracing technique, pruning, or applying any other suitable downsampling method. In a first variation, the plurality of skeletons generated from the starting nodes can be downsampled by selecting the best fit skeleton for the estimated plant section. In a second variation, the skeletons can be downsampled based on a cost function minimization, wherein a skeleton can have a higher cost if the skeleton is unbalanced (e.g., is biased heavily to one side), deviates from the set of typical profiles (e.g., straight lines, as exhibited by stems, loose curls as exhibited by leaves, etc.), or has any other suitable unusual plant characteristic. However, the skeletons can be otherwise downsampled. - Classifying a subset of skeleton segments as stalk bases identifies the starting nodes and/or starting skeleton segments from which the virtual plant model should be generated. This enables the final plant skeleton to be generated from the bottom up (e.g., from the stalk up). Classifying a subset of skeleton segments as stalk bases can include: identifying the skeleton segments proximal the ground plane; identifying sections of the image corresponding to the identified skeleton segments; and identifying stalks from the identified image sections, an example of which is shown in
FIG. 1B . However, the skeleton segments can be categorized as stalk bases based solely on the virtual model (e.g., based on the skeleton segment vector field, based on the geodesic structures formed by connecting the virtual nodes, etc.) or be categorized as stalk bases in any other suitable manner. Alternatively, the method can include classifying a subset of skeleton segments as plant apexes (e.g., plant tops), wherein the skeleton segments and subsequent final plant skeleton can be determined from the top down. However, the skeleton segments can be otherwise classified, and the virtual plant model otherwise generated. - Identifying the skeleton segments proximal the ground plane functions to identify skeleton segments that potentially represent real plant stalks within the virtual space. The stalk candidates are preferably identified in the virtual space because the virtual model is faster and requires less computational power to sort through, as compared to stalk candidate identification from images, but the stalk candidates can be identified in any other suitable manner. The skeleton segments identified as potential candidates can be: skeleton segments at least partially located within a threshold virtual distance of the ground plane; skeleton segments that intersect with or touch the ground plane; skeleton segments satisfying a set of stalk conditions (e.g., have a threshold thickness, variable dependent upon the stage of plant growth, have a threshold straightness, etc.); or be any otherwise selected.
- Identifying sections of the image corresponding to the identified skeleton segments functions to identify high-resolution pieces of information suitable for use in classification. This can be desirable because, in some variations, the skeleton segments are downsampled to increase candidate identification speed, and therefore do not contain sufficient information for the classification module to reliably classify the segment. Furthermore, because the potential stalk candidates and/or corresponding volumes have already been identified using the virtual model, the method can reduce the number of image sections that need to be processed by focusing only on the image sections corresponding to the stalk candidates and surrounding areas, and ignoring the remainder of the image. This can translate to faster computation time and/or a smaller computation power requirement.
- In one variation, an entire image corresponding to the virtual stalk candidate can be identified. In a second variation, a section of the image (e.g., less than the entire image) corresponding to the virtual stalk candidate can be identified. In a third variation, all images associated with the geographic volume associated with the stalk candidate can be identified. However, any other suitable image portion can be identified.
- The identified image section preferably encompasses the entirety of the corresponding virtual stalk candidate (identified skeleton segment), but can alternatively encompass a portion of the corresponding virtual stalk candidate (e.g., the top, bottom, or middle), or encompass any other suitable portion of the virtual stalk candidate. The image section preferably includes a portion of the ground plane proximal the corresponding virtual stalk candidate, but can alternatively exclude the ground plane. The image section is preferably wider than the corresponding virtual stalk candidate, such that the image includes a portion of the ambient environment next to the virtual stalk candidate, but can alternatively be as wide as the corresponding virtual stalk candidate (e.g., cropped at the virtually defined stalk boundaries) or be thinner than the corresponding virtual stalk candidate.
- In one variation, the image section size can be dynamically determined based on the virtual dimensions of the virtual stock candidate. In a second variation, the image section size can be predetermined (e.g., fixed). However, any other suitable portion of the image can be identified as the image section.
- The identified image sections preferably correspond to the virtual stalk candidate. In one variation, the image section correspondence with the identified skeleton segment is determined based on substantially matching respective geographic identifiers. For example, a virtual stalk candidate can be associated with a set of geographic locations (e.g., latitude/longitude coordinates, altitude, etc.) based on the base image(s) from which the model was generated, wherein the identified image section is also associated with substantially the same or a larger set of geographic locations. The geographic locations can be associated with the virtual model and/or images using odometry systems, such as the one disclosed in U.S. application Ser. No. 14/629,361 filed 23 Feb. 2015, incorporated herein in its entirety by this reference. However, geographic locations can be otherwise associated with the image section and virtual model. In a second variation, the image section correspondence with the identified skeleton segment is determined based on the location of the skeleton segment within the source image(s) from which the nodes in skeleton segment was generated. For example, the image section can be identified by aligning a virtual reference point with a corresponding reference point on the source image and identifying the image section that overlaps with the stalk candidate. The virtual reference point can be a vector, point, volume, or other reference point. The image reference point can be an image edge, image pixel (e.g., arranged at a predetermined location relative to the image edges or center), or be any other suitable reference point. Alternatively, the reference point can be a plant feature represented in the virtual space and the image. However, any other suitable reference point can be used.
- The identified image is preferably a set of images, including a left image and a right image of the plant section corresponding to the virtual stalk candidate. The left and right images can cooperatively form a depth image, wherein a segment of the depth image can be identified and used to classify the stalk candidate. Alternatively, the set of images can additionally or alternatively include a depth image (e.g., a scan, stereoscopic image, etc.) composited with the other images, wherein the depth image can be recorded from the same or similar angle (relative to the plant) as the right and left images, or be recorded from a different angle (e.g., from the top of the plant). Alternatively, a single image or any other suitable set of images can be used. The images are preferably those used to generate the point cloud, but can alternatively be different images.
- Identifying stalks from the identified image sections functions to classify each virtual stalk candidate as a plant stalk or non-stalk. However, the candidate skeleton segments can additionally or alternatively be classified as dualities (e.g., as single plants, two plants, multiple plants, etc.), leaves, or be otherwise classified. The virtual stalk candidates are preferably automatically classified by a classification module running on the in-field system or a remote system, but can alternatively be classified in any other suitable manner. The classification module can simultaneously classify multiple stalk candidates, a single stalk candidate, or concurrently classify any number of stalk candidates. The classification module is preferably a convolutional neural network (CNN), but can alternatively or additionally be a deep neural network (DNN), feedforward neural network, radial basis function (RBF) network, Kohonen self organizing network, Learning Vector Quantization, recurrent neural network, modular neural network, physical neural network, holographic associative memory, instantaneously trained network, spiking neural network, dynamic neural network, cascading neural network, neuro-fuzzy network, compositional pattern-producing network, or be any other suitable artificial neural network. However, the classification module can be any other suitable machine learning module (e.g., including SIFT features with the support vector machine [SVM]), or be any other suitable classification module.
- Growing plant skeletons from skeleton segments classified as stalk bases functions to generate the finalized virtual plant representations. The plant skeleton is preferably virtually grown from the skeleton segments classified as stalk bases by the classification module (examples shown in
FIG. 2A ,FIG. 2C ), but can alternatively be grown from any other suitable set of skeleton segments. Alternatively, the plant skeleton can be virtually grown from the starting nodes of the skeleton segments classified as stalk bases (example shown inFIG. 2B ), or be grown from any other suitable virtual structure. The plant skeleton can be iteratively generated, selected from multiple different plant skeletons candidates (e.g., based on the respective probability that the skeleton candidate represents a plant), or determined in any other suitable manner. The plant skeleton can be selected from the plurality of possible skeletons to maximize the plant probability for each plant, for the virtual plant population, or be selected in any other suitable manner. - In one variation, an example of which is shown in
FIG. 2C , the finalized plant skeleton is generated from the previously generated skeleton segments. The plant skeleton can be iteratively generated upwards from the stalk skeleton by joining skeleton segments within a threshold distance of the last plant skeleton segment (e.g., within a virtual centimeter, meter, number of voxels, such as one voxel, etc.). In a specific example, subsequent skeleton segments can be iteratively joined (e.g., virtually annealed) to the last segments of the skeleton when the subsequent segment is within a threshold virtual or real distance (different or similar to the threshold distance used to generate the segments) of the last skeleton segment, and/or when the subsequent segment is further than the last segment from the ground plane. Alternatively, the subsequent segment can be joined when the subsequent segment is further from the starting segment than the last segment. Additionally or alternatively, the subsequent segment can be joined when the probability that the resultant skeleton represents a plant (e.g., based on the skeleton shape uniformity, position of the resultant center of mass, resultant plant size as compared to an expected plant size based on the actual or estimated growth stage, etc.) is beyond a threshold probability (e.g., 75%, 90%, etc.). This can enable the virtual model to represent plants that do not stand entirely upright. However, the plant skeleton can be otherwise annealed. Subsequent skeleton segment joining can be halted when the next segment is beyond the threshold distance, or when any other suitable joining condition is failed. - In a second variation, the plant skeleton can be entirely re-generated, wherein a new skeleton can be virtually grown, independent of the previously generated skeleton segments (example shown in
FIG. 2A ). The new skeleton can be grown from the starting node of the old skeleton segment that was classified as a stalk, grown from the skeleton segment classified as a stalk (e.g., wherein the skeleton segment is the only re-used virtual component), or be otherwise generated. - Growing the virtual plant skeleton can additionally include joining point clouds or voxel groups representative of leaves, fruit, or other plant components to the plant skeleton (example shown in
FIG. 2A ). The virtual plant component representation can be joined as part of the final plant skeletonization process, or be joined after the plant skeleton representative of the stalk and/or stems is finalized. In one variation, the virtual plant component representation is treated the same as all other plant segments. In a second variation, the virtual plant component representations are classified as non-stalk or non-stem (e.g., by a classification module), segmented out of the virtual model used to generate the plant skeleton, then reconsidered after the plant skeleton has been generated. Each plant cluster is preferably joined to a single plant skeleton, but can alternatively be joined to multiple plant skeletons. The plant clusters can be joined to the closest plant skeleton (e.g., touching or within a threshold distance), be joined based on the probability that the resultant virtual plant is representative of a real plant (e.g., based on the resultant center of mass, shape uniformity, size of the cluster relative to the size of the stalk, etc.), or be joined based on the satisfaction of any other suitable characteristic. In one variation, multiple virtual plant options (each with a different combination of joined clusters) can be generated for each plant skeleton, wherein the plant option with the highest probability of representing a real plant can be selected. Alternatively, the plant options maximizing the probability that the virtual plant population represent real plants are selected. In a second variation, a single virtual plant with a single cluster combination can be generated. However, the clusters can be otherwise joined to the virtual plant skeleton. -
FIG. 8 is anexample process 800 for using labeled stalks of plants to carry out actions with respect to those plants. As discussed in the overview, once the positions of the unique plant features, such as stalks or stalk bases, are known 802 other analyses or processes may be carried out based on the information. These include, for example, generating three-dimensional models of plants growing in the field, assessing plant growth and phenotypic features, determining what kinds of treatments to apply including both where to apply the treatments and how much, determining whether to remove weeds or other undesirable plants, and so on. The position in the field of unique plant features, and thus each individual plant, is the starting point for performing any of these processes on a plant-by-plant basis. - As one exemplary embodiment,
FIG. 8 illustrates that the locations of the unique plant features 802 may be passed as an input to plant modeling computer code 804 as described above. Modeling may further include mosaicking the generated models, such that a virtual model of a geographic region can be generated from multiple images of geographic region sub-plots. In one variation, the images can be mosaicked (e.g., based on geographic location, estimated image position relative to the preceding image, reference points within the image, etc.), such that the point cloud on which the method is performed is already representative of the entire geographic region to be modeled. In a second variation, an example of which is shown inFIG. 5 , the virtual plant models can be individually generated or be generated in subsets. Multiple individual virtual plant models and/or plant model subsets can be subsequently merged or arranged within the virtual space based on the relative locations of the respective underlying images. However, the virtual model of the plant field can be otherwise generated. - The virtual model is accessed by plant analysis computer code that analyzes 806 the virtual model to identify plant parameters of individual plants, and/or aggregate statistics regarding the modeled plants. The plant parameters are preferably determined solely from the virtual model, but can alternatively be verified or cooperatively determined based on the underlying source images. Aggregate statistics may be generated for a plant and a subset of its nearest neighbors, an entire row of plants, an entire field of plants, or any sub-zone thereof. Examples of aggregate plant parameters that may be identified include plant density, a number of plants within the plant field, plant size (e.g., average height, median height, height distribution over a population of plants), plant greenness (e.g., based on the colors in the image pixels corresponding to the virtual plant), and plant uniformity, and terminal yield. Examples of individual plant parameters include biomass, greenness, uniformity, leaf area or volume (e.g., based on the volume or surface area of the point clusters representative of leaves), susceptibility to nutrient, water, or biotic stress, leaf area, stalk diameter (e.g., based on the virtual diameter of the plant skeleton), height, and terminal yield on a plant-by-plant basis.
- The individual or aggregate plant parameters are accessed by plant action computer code that analyzes 808 the parameters to determine what action to take with respect to the plants in the field. These determinations may be made on the basis of parameters of individual plants, such that separate and different action can be taken on each individual plant in the field. These determinations may also be made on the basis on aggregate parameters of more than one plant, such as the parameters of a plant and its nearby neighbors, or in aggregate across an entire row, field, or any sub-zone thereof. The
determined actions 808 may include potential treatments, such as the application of fertilizer, pesticides or other chemicals. The determined actions 908 may also include thinning practices, such as the removal of plants identified as weeds, or thinning to remove undesired plants. - The determined actions may be provided 810 to a physical implement (not shown) attached to a device (e.g., tractor, combine, other vehicle, UAV, etc.), to carry out the determined action. For example, if the determined action is a chemical application, the provided action may include timing of when to spray, what chemical to spray, how much to spray, for how long, and at what flow rate. The implement may then carry out 810 the determined action. Alternatively, in an embodiment where the device that will carry out the action is physically remote or otherwise separate from the device/system implementing this process to determine the action to be taken, the providing step may include transmitting the determined action from one device/system to another. This may be a wired or wireless transmission, according to any known transmission protocol, such as by radio frequency transmission (e.g., WiFi, Bluetooth), infrared transmission, Ethernet or CAN bus transmission, etc.
- In one embodiment, all of the process steps of
FIG. 8 can be performed in real- or near-real time. For example, the method can be performed as the device/vehicle/drone etc. travels along a row of the field. A camera may capture images, which may be passed to an onboard system to identify the unique plant features 802, model plants 804, identifyparameters 806, and determineactions 808, and provide 810 those actions to an implement of the vehicle. All steps 802-810 are capable of being performed in a time between when the cameras capture the images of the plants for whomaction 808 is to be determined, and when the implement to carry out those action passes by those same plants along the field. For example, in various embodiments, the vehicle may be traveling at a speed between 1 and 20 miles per hour, and the camera may be capturing images at a frequency within an order of magnitude of 30 Hertz. Given the length of the device or vehicle, the physical distance between the cameras and the implement that performs the action may be approximately a foot, and/or within an order of magnitude of a foot. As a consequence, the process ofFIG. 8 can be carried out before the vehicle travels the distance between the cameras and the implement. This means that the device/vehicle can completely carry out the stepsFIG. 8 in a time range between 40 and 100 milliseconds. This entirely eliminates the need for offboard processing, and allows action while the device/vehicle is in the field, rather than requiring a first pass to identify what to do, and a second pass to actually do it. - In another embodiment, the process of
FIG. 8 can be carried out over different time periods. For example, image collection by the cameras may occur during a first time period as the device travels through the field.Unique feature detection 802, and the other processing steps 804-808 may occur over a second time period later in time, such as in an office or away from the field. The action to be taken 810 may be performed during a third time, such as when the same or a different device/vehicle/UAV travels through the field to carry out the action. - The processes described above may be embodied in computer program instructions stored within a non-transitory computer-readable storage medium, and designed to be executed by one or more computer processors within one or more computing devices. The non-transitory computer-readable medium can be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device. The computer-executable component is preferably a processor but the instructions may alternatively or additionally be executed by any suitable dedicated hardware device.
- Although omitted for conciseness, the preferred embodiments include every combination and permutation of the various system components and the various method processes described above. As a person skilled in the art will recognize from the previous detailed description and from the figures and claims, modifications and changes can be made to the preferred embodiments of the invention without departing from the scope of this invention, one implementation of which is set forth in the following claims.
Claims (21)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/327,516 US20210327133A1 (en) | 2015-05-18 | 2021-05-21 | System and Method of Virtual Plant Field Modelling |
Applications Claiming Priority (5)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562163147P | 2015-05-18 | 2015-05-18 | |
US15/157,997 US10008035B1 (en) | 2015-05-18 | 2016-05-18 | System and method of virtual plant field modelling |
US15/996,810 US10769845B2 (en) | 2015-05-18 | 2018-06-04 | System and method of virtual plant field modelling |
US16/984,026 US11030804B2 (en) | 2015-05-18 | 2020-08-03 | System and method of virtual plant field modelling |
US17/327,516 US20210327133A1 (en) | 2015-05-18 | 2021-05-21 | System and Method of Virtual Plant Field Modelling |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/984,026 Continuation US11030804B2 (en) | 2015-05-18 | 2020-08-03 | System and method of virtual plant field modelling |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210327133A1 true US20210327133A1 (en) | 2021-10-21 |
Family
ID=62623930
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/157,997 Active US10008035B1 (en) | 2015-05-18 | 2016-05-18 | System and method of virtual plant field modelling |
US15/996,810 Active US10769845B2 (en) | 2015-05-18 | 2018-06-04 | System and method of virtual plant field modelling |
US16/984,026 Active US11030804B2 (en) | 2015-05-18 | 2020-08-03 | System and method of virtual plant field modelling |
US17/327,516 Abandoned US20210327133A1 (en) | 2015-05-18 | 2021-05-21 | System and Method of Virtual Plant Field Modelling |
Family Applications Before (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/157,997 Active US10008035B1 (en) | 2015-05-18 | 2016-05-18 | System and method of virtual plant field modelling |
US15/996,810 Active US10769845B2 (en) | 2015-05-18 | 2018-06-04 | System and method of virtual plant field modelling |
US16/984,026 Active US11030804B2 (en) | 2015-05-18 | 2020-08-03 | System and method of virtual plant field modelling |
Country Status (1)
Country | Link |
---|---|
US (4) | US10008035B1 (en) |
Families Citing this family (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10008035B1 (en) * | 2015-05-18 | 2018-06-26 | Blue River Technology Inc. | System and method of virtual plant field modelling |
KR101956158B1 (en) * | 2017-08-22 | 2019-03-08 | 한국과학기술연구원 | System and method for plant leaf identification |
US11127202B2 (en) * | 2017-12-18 | 2021-09-21 | Parthiv Krishna | Search and rescue unmanned aerial system |
US11481972B2 (en) * | 2018-01-17 | 2022-10-25 | United States Of America As Represented By The Secretary Of Agriculture | Method of performing dendrometry and forest mapping |
US11275941B2 (en) * | 2018-03-08 | 2022-03-15 | Regents Of The University Of Minnesota | Crop models and biometrics |
CN108537814B (en) * | 2018-03-14 | 2019-09-03 | 浙江大学 | A kind of three-dimensional sonar point cloud chart based on ViBe is as dividing method |
WO2020076735A1 (en) * | 2018-10-08 | 2020-04-16 | Mjnn Llc | Control of latent and sensible loads in controlled environment agriculture |
CN109522901B (en) * | 2018-11-27 | 2020-11-03 | 中国计量大学 | Tomato plant stem edge identification method based on edge dual relation |
CN109685797B (en) * | 2018-12-25 | 2021-08-10 | 北京旷视科技有限公司 | Bone point detection method, device, processing equipment and storage medium |
CN109887020B (en) * | 2019-02-25 | 2020-08-04 | 中国农业科学院农业信息研究所 | Plant organ separation method and system |
US11270112B2 (en) * | 2019-04-16 | 2022-03-08 | Precision Silver, LLC | Systems and methods for rating vegetation health and biomass from remotely sensed morphological and radiometric data |
CN110232698B (en) * | 2019-04-20 | 2021-09-17 | 北京工业大学 | Based on model segmentation and L1Method for extracting continuous frame three-dimensional human body curve skeleton of median skeleton |
CN109975250B (en) * | 2019-04-24 | 2020-03-24 | 中国科学院遥感与数字地球研究所 | Leaf area index inversion method and device |
US11610158B2 (en) | 2019-05-02 | 2023-03-21 | Mjnn Llc | Automated placement of plant varieties for optimum performance within a grow space subject to environmental condition variability |
US11803172B2 (en) | 2019-05-10 | 2023-10-31 | Mjnn Llc | Efficient selection of experiments for enhancing performance in controlled environment agriculture |
CN110728739B (en) * | 2019-09-30 | 2023-04-14 | 杭州师范大学 | Virtual human control and interaction method based on video stream |
CN110880204B (en) * | 2019-11-21 | 2022-08-16 | 腾讯科技(深圳)有限公司 | Virtual vegetation display method and device, computer equipment and storage medium |
US11521382B2 (en) | 2020-02-09 | 2022-12-06 | Stout Industrial Technology, Inc. | Machine vision plant tracking system for precision agriculture |
CA3167221A1 (en) | 2020-02-09 | 2021-08-12 | Jeffrey L. Antle | Smart tool arm for precision agriculture |
US11113525B1 (en) | 2020-05-18 | 2021-09-07 | X Development Llc | Using empirical evidence to generate synthetic training data for plant detection |
US11398028B2 (en) * | 2020-06-08 | 2022-07-26 | X Development Llc | Generating and using synthetic training data for plant disease detection |
US11593597B2 (en) * | 2020-11-16 | 2023-02-28 | GM Global Technology Operations LLC | Object detection in vehicles using cross-modality sensors |
US11760379B2 (en) * | 2021-01-11 | 2023-09-19 | Toyota Research Institute, Inc. | Navigating an autonomous vehicle through an intersection |
CN112802089B (en) * | 2021-02-05 | 2023-09-01 | 重庆大学 | Point cloud skeleton line extraction method and system based on automatic bifurcation quantity estimation |
EP4060608A1 (en) * | 2021-03-17 | 2022-09-21 | Robovision | Improved vision-based measuring |
US12001512B2 (en) | 2021-05-25 | 2024-06-04 | Mineral Earth Sciences Llc | Generating labeled synthetic training data |
CN113205536B (en) * | 2021-05-28 | 2022-04-29 | 湖北工业大学 | Method for drawing aggregate density cloud picture |
US11995859B2 (en) | 2021-10-28 | 2024-05-28 | Mineral Earth Sciences Llc | Sparse depth estimation from plant traits |
US20230133026A1 (en) * | 2021-10-28 | 2023-05-04 | X Development Llc | Sparse and/or dense depth estimation from stereoscopic imaging |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5797976A (en) * | 1988-09-09 | 1998-08-25 | Yamashita; Thomas T. | Method and composition for promoting and controlling growth of plants |
US6309440B1 (en) * | 1998-08-25 | 2001-10-30 | Thomas T. Yamashita | Method and composition for promoting and controlling growth of plants |
US9238821B2 (en) * | 2009-04-14 | 2016-01-19 | The Samuel Roberts Noble Foundation, Inc. | Metabolic engineering for plant disease resistance |
US10008035B1 (en) * | 2015-05-18 | 2018-06-26 | Blue River Technology Inc. | System and method of virtual plant field modelling |
US20190246586A1 (en) * | 2018-02-15 | 2019-08-15 | Monsanto Technology Llc | Methods for Hybrid Corn Seed Production |
US20190278988A1 (en) * | 2018-03-08 | 2019-09-12 | Regents Of The University Of Minnesota | Crop models and biometrics |
US20200048156A1 (en) * | 2018-08-09 | 2020-02-13 | Thomas T. Yamashita | Foliar Fertilizer Compositions and Methods for Using Same |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5224179A (en) * | 1988-12-20 | 1993-06-29 | At&T Bell Laboratories | Image skeletonization method |
US8855405B2 (en) * | 2003-04-30 | 2014-10-07 | Deere & Company | System and method for detecting and analyzing features in an agricultural field for vehicle guidance |
US20090174703A1 (en) * | 2008-01-07 | 2009-07-09 | Disney Enterprises, Inc. | Particle-based method of generating and animating three-dimensional vegetation |
US7917346B2 (en) * | 2008-02-19 | 2011-03-29 | Harris Corporation | Geospatial modeling system providing simulated tree trunks and branches for groups of tree crown vegetation points and related methods |
US8610712B2 (en) * | 2011-03-01 | 2013-12-17 | Adobe Systems Incorporated | Object selection in stereo image pairs |
KR20140142470A (en) * | 2013-06-04 | 2014-12-12 | 한국전자통신연구원 | Method for generating a tree model and a forest model and apparatus for the same |
WO2015171954A2 (en) * | 2014-05-09 | 2015-11-12 | Raven Industries, Inc. | Refined row guidance parameterization with hough transform |
ES2886865T3 (en) | 2014-12-10 | 2021-12-21 | Univ Sydney | Automatic target recognition and dispensing system |
WO2016123656A1 (en) | 2015-02-05 | 2016-08-11 | The University Of Sydney | Horticultural harvesting system and apparatus using revolving shells |
US10491879B2 (en) * | 2016-01-15 | 2019-11-26 | Blue River Technology Inc. | Plant feature detection using captured images |
-
2016
- 2016-05-18 US US15/157,997 patent/US10008035B1/en active Active
-
2018
- 2018-06-04 US US15/996,810 patent/US10769845B2/en active Active
-
2020
- 2020-08-03 US US16/984,026 patent/US11030804B2/en active Active
-
2021
- 2021-05-21 US US17/327,516 patent/US20210327133A1/en not_active Abandoned
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5797976A (en) * | 1988-09-09 | 1998-08-25 | Yamashita; Thomas T. | Method and composition for promoting and controlling growth of plants |
US6309440B1 (en) * | 1998-08-25 | 2001-10-30 | Thomas T. Yamashita | Method and composition for promoting and controlling growth of plants |
US9238821B2 (en) * | 2009-04-14 | 2016-01-19 | The Samuel Roberts Noble Foundation, Inc. | Metabolic engineering for plant disease resistance |
US10008035B1 (en) * | 2015-05-18 | 2018-06-26 | Blue River Technology Inc. | System and method of virtual plant field modelling |
US20200364932A1 (en) * | 2015-05-18 | 2020-11-19 | Blue River Technology Inc. | System and Method of Virtual Plant Field Modelling |
US20190246586A1 (en) * | 2018-02-15 | 2019-08-15 | Monsanto Technology Llc | Methods for Hybrid Corn Seed Production |
US20190278988A1 (en) * | 2018-03-08 | 2019-09-12 | Regents Of The University Of Minnesota | Crop models and biometrics |
US20200048156A1 (en) * | 2018-08-09 | 2020-02-13 | Thomas T. Yamashita | Foliar Fertilizer Compositions and Methods for Using Same |
Non-Patent Citations (6)
Title |
---|
Chaudhury A, Godin C. Skeletonization of Plant Point Cloud Data Using Stochastic Optimization Framework. Frontiers in Plant Science. 2020 Jun 16;11:773-. * |
Livny Y, Yan F, Olson M, Chen B, Zhang H, El-Sana J. Automatic reconstruction of tree skeletal structures from point clouds. InACM SIGGRAPH Asia 2010 papers 2010 Dec 15 (pp. 1-8). * |
Sandy Perry et al., Forest Pest Management: A Guide for Commercial Applicators, Category 2; Michigan State University, 09-2000, pp. 1-111. * |
Wu S, Wen W, Xiao B, Guo X, Du J, Wang C, Wang Y. An accurate skeleton extraction approach from 3D point clouds of maize plants. Frontiers in plant science. 2019 Mar 7;10:248. * |
Xu H, Gossett N, Chen B. Knowledge and heuristic-based modeling of laser-scanned trees. ACM Transactions on Graphics (TOG). 2007 Oct 1;26(4):19-es. * |
YAN HP, Kang MZ, De Reffye P, Dingkuhn M. A dynamic, architectural plant model simulating resource‐dependent growth. Annals of botany. 2004 May 1;93(5):591-602. * |
Also Published As
Publication number | Publication date |
---|---|
US10008035B1 (en) | 2018-06-26 |
US20180286117A1 (en) | 2018-10-04 |
US10769845B2 (en) | 2020-09-08 |
US20200364932A1 (en) | 2020-11-19 |
US11030804B2 (en) | 2021-06-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11030804B2 (en) | System and method of virtual plant field modelling | |
US10614562B2 (en) | Inventory, growth, and risk prediction using image processing | |
JP7307743B2 (en) | Method and Apparatus for Acquiring Work Area Boundary and Method for Planning Work Path | |
Montoya-Cavero et al. | Vision systems for harvesting robots: Produce detection and localization | |
CN109886155B (en) | Single-plant rice detection and positioning method, system, equipment and medium based on deep learning | |
Karkee et al. | A method for three-dimensional reconstruction of apple trees for automated pruning | |
Zhang et al. | A method for organs classification and fruit counting on pomegranate trees based on multi-features fusion and support vector machine by 3D point cloud | |
Zermas et al. | Estimating the leaf area index of crops through the evaluation of 3D models | |
Kraemer et al. | From plants to landmarks: Time-invariant plant localization that uses deep pose regression in agricultural fields | |
Xiang et al. | Field‐based robotic leaf angle detection and characterization of maize plants using stereo vision and deep convolutional neural networks | |
Zhang et al. | Automatic flower cluster estimation in apple orchards using aerial and ground based point clouds | |
Li et al. | Multi-source data fusion improves time-series phenotype accuracy in maize under a field high-throughput phenotyping platform | |
Zou et al. | Density estimation method of mature wheat based on point cloud segmentation and clustering | |
Mahmud et al. | Unmanned aerial vehicle based tree canopy characteristics measurement for precision spray applications | |
Dong et al. | Drone-based three-dimensional photogrammetry and concave hull by slices algorithm for apple tree volume mapping | |
Treboux et al. | Towards retraining of machine learning algorithms: an efficiency analysis applied to smart agriculture | |
Ma et al. | Banana individual segmentation and phenotypic parameter measurements using deep learning and terrestrial LiDAR | |
Mancini et al. | Soil/crop segmentation from remotely sensed data acquired by Unmanned Aerial System | |
Henrich et al. | Treelearn: A comprehensive deep learning method for segmenting individual trees from forest point clouds | |
Riccardi et al. | Fruit Tracking Over Time Using High-Precision Point Clouds | |
Joey et al. | Automated pruning of greenhouse indeterminate tomato plants | |
Bono et al. | Biomass characterization with semantic segmentation models and point cloud analysis for precision viticulture | |
Akhtar et al. | Unlocking plant secrets: A systematic review of 3D imaging in plant phenotyping techniques | |
Roggiolani et al. | Unsupervised Pre-Training for 3D Leaf Instance Segmentation | |
CN116935235B (en) | Fresh tea leaf identification method and related device based on unmanned tea picking machine |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: BLUE RIVER TECHNOLOGY INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:REDDEN, LEE KAMP;APOSTOLOFF, NICHOLAS;SIGNING DATES FROM 20160815 TO 20170716;REEL/FRAME:057290/0128 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |