CN113763465A - Garbage determination system, model training method, determination method and determination device - Google Patents
Garbage determination system, model training method, determination method and determination device Download PDFInfo
- Publication number
- CN113763465A CN113763465A CN202010489021.XA CN202010489021A CN113763465A CN 113763465 A CN113763465 A CN 113763465A CN 202010489021 A CN202010489021 A CN 202010489021A CN 113763465 A CN113763465 A CN 113763465A
- Authority
- CN
- China
- Prior art keywords
- feature map
- point cloud
- frame
- determined
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the invention discloses a garbage determination system, a model training method, a determination method and a determination device. The method comprises the following steps: acquiring image data from an image sensor and point cloud data from a radar sensor; respectively extracting feature data in the image data and the point cloud data to generate a first image feature map and a first point cloud feature map; carrying out element fusion on object frames with the same size in the first image feature map and the first point cloud feature map to generate a first frame to be detected; and determining the garbage determination model according to the first frame to be measured and a real frame preset by the object to be determined. According to the embodiment of the invention, the attribute and the position of the object to be determined are determined by acquiring the image data from the image sensor and the depth data from the radar sensor, so that the data perception capability is improved, and the implementation logic is optimized.
Description
Technical Field
The invention relates to the field of image processing, in particular to a garbage determination system, a model training method, a determination device, equipment and a storage medium.
Background
In the existing garbage determination method, a binocular Coupled Device (CCD) camera is often used to collect a target image, and then the target image is decoded by a video decoder and transmitted to a Digital Signal Processor (DSP) to convert an analog Signal into a Digital Signal and to primarily process the image and store the Digital Signal and the image by an image data memory, and then the Digital Signal is transmitted to a central processing unit to perform Digital image reprocessing and binocular stereo vision recognition and positioning, so as to calculate the specific three-dimensional size and position information of the target.
In the existing technical scheme, because the DSP has relatively limited computing power, and the CCD binocular camera also has the defects of low visual precision, limited visual field and inaccurate depth data, the existing technical scheme for determining the garbage has the problems of weak data sensing capability and poor logic implementation.
Disclosure of Invention
Embodiments of the present invention provide a garbage determination system, a model training method, a determining device, a device, and a storage medium, which solve the problems of weak data sensing capability and poor logic implementation in the existing garbage determination technical scheme, improve data sensing capability, and optimize implementation logic.
In a first aspect, a garbage determination system is provided, which includes:
the image sensor is used for acquiring image data of an object to be determined;
the radar sensor is used for acquiring point cloud data of an object to be determined;
the Field Programmable Gate Array (FPGA) is used for preprocessing the acquired image data and point cloud data of the object to be determined;
the Graphics Processing Unit (GPU) is used for performing feature extraction on the preprocessed image data and point cloud data of the object to be determined, training a garbage determination model, and determining the attribute of the object to be determined and the boundary frame of the object to be determined according to the trained garbage determination model;
the communication module is used for communicating the garbage determination system with the outside;
the Central Processing Unit (CPU) is used for connecting a (5th generation mobile networks, 5G) module, a Graphics Processing Unit (GPU) and a Field Programmable Gate Array (FPGA), and sending instructions to the Graphics Processing Unit (GPU) and the 5G module.
In a second aspect, a method for training a garbage determination model is provided, the method comprising:
acquiring image data from an image sensor and point cloud data from a radar sensor;
respectively extracting feature data in the image data and the point cloud data, and correspondingly generating a first image feature map and a first point cloud feature map;
generating an object frame in the first image feature map and an object frame in the first point cloud feature map which have the same size according to the object frame in the first image feature map and the object frame in the first point cloud feature map respectively;
performing element fusion on an object frame in the first image feature map and an object frame in the first point cloud feature map which are the same in size to generate a first frame to be detected, wherein the first frame to be detected comprises a three-dimensional boundary frame of an object to be determined;
determining a coordinate difference value between a first frame to be measured and a real frame preset by an object to be determined, and adjusting initial parameters of a garbage determination model according to the coordinate difference value;
and when the coordinate difference value meets a first preset condition, saving the garbage determination model after the parameters are adjusted.
In a third aspect, a method for determining garbage is provided, where the method includes:
acquiring image data from an image sensor and point cloud data from a radar sensor;
inputting the image data and the point cloud data into the garbage determination model after parameter adjustment, and outputting the attribute of the object to be determined and the three-dimensional bounding box of the object to be determined, wherein the garbage determination model after parameter adjustment is obtained by the method for training the garbage determination model in the second aspect.
In a fourth aspect, an apparatus for training a garbage determination model is provided, the apparatus comprising:
the acquisition module is used for acquiring image data from the image sensor and point cloud data from the radar sensor;
the processing module is used for respectively extracting feature data in the image data and the point cloud data and correspondingly generating a first image feature map and a first point cloud feature map;
the processing module is further used for generating an object frame in the first image feature map and an object frame in the first point cloud feature map which are the same in size according to the object frame in the first image feature map and the object frame in the first point cloud feature map respectively;
the processing module is further used for carrying out element fusion on an object frame in the first image feature map and an object frame in the first point cloud feature map which are the same in size to generate a first frame to be detected, wherein the first frame to be detected comprises a three-dimensional boundary frame of an object to be determined;
the processing module is also used for determining a coordinate difference value between the first frame to be measured and a real frame preset by the object to be determined, and adjusting initial parameters of the garbage determination model according to the coordinate difference value;
and the processing module is further used for saving the garbage determination model after the parameters are adjusted when the coordinate difference value meets a first preset condition.
In a fifth aspect, there is provided an apparatus for garbage determination, the apparatus comprising:
the acquisition module is used for acquiring image data from the image sensor and point cloud data from the radar sensor;
and the processing module is used for inputting the image data and the point cloud data into the garbage determination model after the parameters are adjusted and outputting the attribute of the object to be determined and the three-dimensional bounding box of the object to be determined, wherein the garbage determination model after the parameters are adjusted is obtained by the method for training the garbage determination model in the second aspect.
In a sixth aspect, there is provided a device for garbage determination, the device comprising: a processor and a memory storing computer program instructions;
the processor, when executing the computer program instructions, implements the method of training the spam determination model of the second aspect, or the method of spam determination of the third aspect.
In a seventh aspect, a computer storage medium is provided, having computer program instructions stored thereon, which, when executed by a processor, implement the method of training a spam determination model of the second aspect, or the method of spam determination of the third aspect.
The embodiment of the invention provides a garbage determination system, a model training method, a determination device, equipment and a storage medium.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a garbage determination system according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a method for training a garbage determination model according to an embodiment of the present invention;
fig. 3 is a schematic diagram of Feature Pyramid Network (FPN) feature map extraction based on target detection according to an embodiment of the present invention;
fig. 4 is a schematic flowchart of a method for determining garbage according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of an apparatus for training a garbage determination model according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a device for determining garbage according to an embodiment of the present invention;
fig. 7 is a block diagram of an exemplary hardware architecture of a computing device provided by an embodiment of the present invention.
Detailed Description
Features and exemplary embodiments of various aspects of the present invention will be described in detail below, and in order to make objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not to be construed as limiting the invention. It will be apparent to one skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present invention by illustrating examples of the present invention.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
At present, in a method for determining garbage, a binocular Charge Coupled Device (CCD) camera is often used to collect a target image, and then the target image is decoded by a video decoder and transmitted to a Digital Signal Processor (DSP) to convert an analog Signal into a Digital Signal, and the Digital Signal is primarily processed and then stored in an image data memory. Then the image is transmitted to a central processing unit for digital image reprocessing, binocular stereo vision identification and positioning, the specific three-dimensional size and position information of the target are calculated, and preset model matching is carried out. And finally, the central processing unit transmits the information to the DSP processor, and the DSP can send an instruction to the robot sorting structure to determine the garbage after the robot sorting operation.
Because the processing capacity of a DSP processor in the current garbage determination method is limited, the support capacity of an image recognition algorithm is insufficient, and the recognition capacity is often poor. In addition, the CCD binocular camera also has the problems of low visual precision, limited visual field and inaccurate depth data, so that the technical scheme of the existing garbage determination has the problems of weak data sensing capability and poor logic implementation.
In order to solve the problems of weak data sensing capability and poor logic implementation existing in the existing technical scheme, embodiments of the present application provide a garbage determination system, a model training method, a determination method, an apparatus, a device, and a storage medium, image data and point cloud data are obtained from an image sensor and a radar sensor, a three-dimensional bounding box of an object to be determined is obtained from the image data and the point cloud data, and a parameter of a garbage determination model is adjusted by a difference between the three-dimensional bounding box of the object to be determined and a real frame to obtain a garbage determination model.
The following describes aspects of embodiments of the present application with reference to the drawings.
Fig. 1 shows a schematic structural diagram of a garbage determination system according to an embodiment of the present invention. As shown in fig. 1, the garbage determination system may include: image sensor, radar sensor, Field Programmable Gate Array (FPGA), Graphics Processing Unit (GPU), communication module, Central Processing Unit (CPU), wherein,
101: the image sensor is used for acquiring image data of the object to be determined.
Wherein the image sensor may comprise a camera.
Specifically, the image sensor may be configured to acquire Red Green Blue (RGB) digital data of an image of an object to be determined.
102: and the radar sensor is used for acquiring point cloud data of the object to be determined.
Wherein the radar sensor may comprise a lidar.
In particular, the radar sensor may be used to acquire Bird's-Eye View (BEV) digital data of the object to be determined, which may be a type of point cloud data.
103: and the field programmable gate array FPGA is used for preprocessing the acquired image data and point cloud data of the object to be determined.
Specifically, the FPGA may be configured to perform encoding and decoding, filtering, edge extraction, and the like on the acquired RGB data and BEV data of the object to be determined.
104: and the graphics processing unit GPU is used for performing feature extraction on the preprocessed image data and point cloud data of the object to be determined, performing a training model for determining garbage, and determining the attribute of the object to be determined and the boundary frame of the object to be determined according to the trained garbage determination model.
Specifically, the GPU may acquire image data from the image sensor and point cloud data from the radar sensor based on a floating point computing capability provided by a common parallel computing Architecture (CUDA) of the same Device computing Architecture; respectively extracting feature data in the image data and the point cloud data, and correspondingly generating a first image feature map and a first point cloud feature map; generating an object frame in the first image feature map and an object frame in the first point cloud feature map which have the same size according to the object frame in the first image feature map and the object frame in the first point cloud feature map respectively; performing element fusion on an object frame in the first image feature map and an object frame in the first point cloud feature map which are the same in size to generate a first frame to be detected, wherein the first frame to be detected comprises a three-dimensional boundary frame of an object to be determined; determining a coordinate difference value between a first frame to be measured and a real frame preset by an object to be determined, and adjusting initial parameters of a garbage determination model according to the coordinate difference value; when the coordinate difference value meets a preset condition, saving the garbage determination model after the parameters are adjusted; and determining the attribute of the object to be determined and the three-dimensional bounding box of the object to be determined by using the garbage determination model after the parameters are adjusted.
105: and the communication module is used for communicating the garbage determination system with the outside.
Specifically, the communication module may include a fifth generation mobile communication technology 5G module and an SFP optical module.
106: and the central processing unit CPU is used for connecting the 5G module, the GPU and the FPGA and sending instructions to the GPU and the 5G module.
Specifically, the CPU can implement peripheral driving and display of the system, and in addition, the CPU can be connected to a third-generation Double Data Rate Synchronous Dynamic Random Access Memory (DDR 3 SDRAM), a FLASH Memory, a display, and a debug interface, wherein the CPU can form a minimum hardware system together with the DDR3 SDRAM and the FLASH Memory, and Boot an embedded operating system after power-on to complete necessary peripheral driving and logic control; the display can realize the type prediction and classification data display of the object to be detected; the debugging interface can provide running display and instruction receiving and sending in the debugging process of the CPU and the GPU.
The garbage determination system provided by the embodiment of the invention obtains the image data and the point cloud data from the image sensor and the radar sensor, obtains the three-dimensional boundary frame of the object to be determined from the image data and the point cloud data, and adjusts the parameters of the garbage determination model through the difference between the three-dimensional boundary frame of the object to be determined and the real frame to obtain the garbage determination model, thereby solving the problems of weak data sensing capability and poor logic realization existing in the prior art, improving the data sensing capability and optimizing the realization logic.
Fig. 2 is a schematic flow chart illustrating a training method of a garbage determination model according to an embodiment of the present invention.
As shown in fig. 2, the training method of the garbage determination model may include:
s201: image data is acquired from an image sensor and point cloud data is acquired from a radar sensor.
The image data may include Red Green Blue (RGB) digital data of an image of an object to be determined, which is acquired using an image sensor, and the point cloud data may include Bird's-Eye View (BEV) digital data of the object to be determined, which is acquired using a radar sensor.
Alternatively, in one embodiment, the image data and depth data acquired by the image sensor and radar sensor may be extracted using a full resolution feature extractor comprised of an encoder and decoder. Specifically, the encoder may model after (Visual Geometry Group-16, VGG-16), reduce the number of channels by half, and cut the network on the 4 th convolutional layer (conv-4). Therefore, the encoder takes an R × G × B image or a BEV map as input, and generates a feature map F of M/8 × N/8 × D/8. The feature map F has a higher representation capability but a resolution 8 times lower than the input. An 8-fold down-sampling (data decimation) will result in these subclasses occupying less than one pixel in the output signature to increase the speed of the computation. Based on a Feature Pyramid Network (FPN) for target detection, the extraction process can be as shown in fig. 3, where the decoder is designed to decode from bottom to top, and performs up-sampling (reduction) on the input through a convolution transpose operation, and up-samples the input to the original input size, and can keep the running speed. That is, the decoder takes the F output from the encoder as input, generates a new M 'xn' xd 'feature map F', connects the corresponding feature maps from the encoder, merges the two through a 3 × 3 convolution operation, finally generates data with high resolution and representativeness, and is shared by a multiple model fusion area candidate Network (RPN) in the garbage determination model and the second stage detection Network.
S202: and respectively extracting feature data in the image data and the point cloud data, and correspondingly generating a first image feature map and a first point cloud feature map.
Specifically, independent 1x1 convolution operation may be performed on the first image feature map and the first point cloud feature map to generate a bounding box of the object to be determined in the first image feature map and a bounding box of the object to be determined in the first point cloud feature map, and the 1x1 convolution operation may effectively reduce the dimensionality of the input data to reduce the data pressure of the bounding box of the object to be determined in the subsequent generation.
Then, determining an object frame in the first image feature map according to a boundary frame of the object to be determined in the first image feature map; and determining an object frame in the first point cloud feature map according to the boundary frame of the object to be determined in the first point cloud feature map.
Specifically, in the process of determining the object frame according to the bounding box, anchor frames of various sizes can be generated according to feature points in the first image feature map and the first point cloud feature map through a preset three-dimensional anchor point coordinate reference system, where the anchor frames may be referred to as bounding boxes.
When the detection evaluation function (IOU) of the anchor frame of the object to be determined in the first image feature map is greater than a first preset threshold, determining that the anchor frame of the object to be determined in the first image feature map is the object frame in the first image feature map.
And when the detection evaluation function IOU of the anchor frame of the object to be determined in the first point cloud characteristic diagram is larger than a first preset threshold value, determining the anchor frame of the object to be determined in the first point cloud characteristic diagram as the object frame in the first point cloud characteristic diagram.
Optionally, in an embodiment, the first preset threshold may be 0.5, that is, when the detection evaluation function of the anchor frame of the object to be determined in the first image feature map or the first point cloud feature map is greater than 0.5, the anchor frame of the object to be determined in the first image feature map or the first point cloud feature map is determined to be the object frame in the first image feature map or the first point cloud feature map.
In addition, optionally, when the detection evaluation function IOU is less than 0.3, the anchor frame of the object to be determined in the first image feature map or the first point cloud feature map may be considered as the background frame.
After determining the object frame in the first image feature map and the object frame in the first point cloud feature map, the process may proceed to S203 to perform further clipping and cropping on the object frames.
S203: and generating the object frame in the first image feature map and the object frame in the first point cloud feature map which have the same size according to the object frame in the first image feature map and the object frame in the first point cloud feature map respectively.
Specifically, the object frame in the first image feature map and the object frame in the first point cloud feature map may be cut out and clipped out, and the object frames in the first image feature map and the first point cloud feature map may be adjusted to be the object frames with the same size.
S204: and performing element fusion on the object frame in the first image feature map and the object frame in the first point cloud feature map which have the same size to generate a first frame to be detected, wherein the first frame to be detected comprises a three-dimensional boundary frame of the object to be determined, and the element fusion can be average fusion of elements.
S205: and determining a coordinate difference value of the first frame to be measured and a real frame preset by the object to be determined, and adjusting initial parameters of the garbage determination model according to the coordinate difference value.
Specifically, the coordinate difference value may be a coordinate difference value (Δ t _ x, Δ t _ y, Δ t _ z, Δ d _ x, Δ d _ y, Δ d _ z) of 6 coordinate points between the first frame to be measured and the real frame, which is determined by the Loss function Smooth L1 Loss. The difference values of the central point of the first frame to be detected and the central point of the real frame preset by the object to be determined in the three axes of x, y and z are respectively represented by delta t _ x, delta t _ y and delta t _ z, and the difference values of the length, the width and the height of the first frame to be detected and the real frame preset by the object to be determined are respectively represented by delta d _ x, delta d _ y and delta d _ z.
The real frame preset by the object to be determined may refer to a real position of the object to be determined in the image data acquired from the image sensor and the point cloud data acquired from the radar sensor, and the position may be understood as a coordinate.
S206: and when the coordinate difference value meets a preset condition, saving the garbage determination model after the parameters are adjusted.
The preset condition may include that the coordinate difference is a minimum value, that is, when the coordinate difference is the minimum value, the garbage determination model after the parameter adjustment is stored.
Optionally, in an embodiment, the first frames to be detected output by the first 1024 RPN networks may be determined according to descending order of the detection evaluation functions IOU of the bounding boxes of the object, and then the coordinates of the obtained 1024 first frames to be detected may be represented by 4 foot coordinates and 2 height values (distances between the top and bottom feet and the reference plane), so that the data amount may be reduced to 6, and the calculated data amount may be further reduced.
Optionally, in an embodiment, according to the 1024 first frames to be measured, a second image feature map and a second point cloud feature map are correspondingly obtained from the first image feature map and the first point cloud feature map.
And then performing pixel-level fusion and superposition on the second image characteristic diagram and the second point cloud characteristic diagram by using convolution operation of 7x7 to generate a second frame to be detected, wherein the second frame to be detected can be called a final version of the garbage frame to be detected.
After the second frame to be measured is obtained, calculating coordinate difference values (delta t _ x, delta t _ y, delta t _ z, delta d _ x, delta d _ y and delta d _ z) of 6 coordinate points between the second frame to be measured and a real frame preset by the object to be determined through a Smooth L1 Loss function;
and when the coordinate difference value meets a second preset condition, saving the garbage determination model after the parameters are adjusted.
The second preset condition may also include that the coordinate difference is a minimum value, and when the garbage determination model satisfies that the coordinate difference is the minimum value, the output of the garbage determination model after training and adjusting the parameters may be made as close as possible to the real frame, so as to perform object recognition on the data obtained by the image sensor and the radar sensor.
Optionally, in an embodiment, in the training method of the garbage determination model, each training process may be performed on an object to be recognized, where the object to be recognized may include a glass bottle, a plastic bottle, a paper box, and the like.
For example, after the training of the waste determination model for identifying the glass bottles is completed, the waste determination model for identifying the plastic bottles may be trained, and after the training of the waste determination model for identifying the plastic bottles is completed, the waste determination model for identifying the paper boxes may be trained. That is to say, the training method of the garbage determination model in S201-S206 may be used to train one object to be recognized, and after the training is completed, train another object to be recognized. The training can be completed for all the object types to be recognized, that is, the garbage determination model after the final training can recognize various objects including glass bottles, plastic bottles, paper boxes, and the like.
According to the training method for the garbage determination model, provided by the embodiment of the invention, the image data and the point cloud data are obtained from the image sensor and the radar sensor, the three-dimensional boundary frame of the object to be determined is obtained from the image data and the point cloud data, and the parameters of the garbage determination model are adjusted according to the difference between the three-dimensional boundary frame of the object to be determined and the real frame, so that the garbage determination model is obtained, the problems of weak data sensing capability and poor logic realization in the prior art are solved, the data sensing capability is improved, and the realization logic is optimized.
Fig. 4 is a flowchart illustrating a method for determining spam according to an embodiment of the present invention.
As shown in fig. 4, the method of garbage determination may include:
s401: image data is acquired from an image sensor and point cloud data is acquired from a radar sensor.
S402: and inputting the image data and the point cloud data into the garbage determination model after the parameters are adjusted, and outputting the attribute of the object to be determined and the three-dimensional bounding box of the object to be determined.
The garbage determination model may be obtained by any one of the methods for training the garbage determination model in fig. 2, and the image data and the point cloud data are input into the garbage determination model after the parameters are adjusted, so that the attributes of the object to be determined (for example, a glass bottle, a plastic bottle, and a paper box) and the three-dimensional bounding box (position) of the object to be determined may be determined.
According to the method for determining the garbage, provided by the embodiment of the invention, the image data and the depth data are obtained from the image sensor and the radar sensor and are input into the garbage determination model after the parameters are adjusted, and the garbage determination model determines the attribute and the three-dimensional boundary frame of the object to be determined from the image data and the depth data, so that the problems of weak data sensing capability and poor logic realization existing in the prior art are solved, the data sensing capability is improved, and the realization logic is optimized.
Corresponding to the embodiment of the training method of the garbage determination model, the embodiment of the invention also provides a training device of the garbage determination model.
As shown in fig. 5, fig. 5 is a schematic structural diagram of an apparatus for training a garbage determination model according to an embodiment of the present invention.
The apparatus for garbage determination model training may include: an acquisition module 501 and a processing module 502, wherein,
the acquiring module 501 may be configured to acquire image data from an image sensor and point cloud data from a radar sensor.
The processing module 502 may be configured to extract feature data in the image data and the point cloud data, and generate a first image feature map and a first point cloud feature map correspondingly.
The processing module 502 may be further configured to generate an object frame in the first image feature map and an object frame in the first point cloud feature map, which have the same size, according to the object frame in the first image feature map and the object frame in the first point cloud feature map, respectively.
The processing module 502 may further be configured to perform element fusion on an object frame in the first image feature map and an object frame in the first point cloud feature map, which have the same size, to generate a first frame to be detected, where the first frame to be detected includes a three-dimensional bounding box of the object to be determined.
The processing module 502 may further be configured to determine a coordinate difference between the first frame to be measured and a real frame preset by the object to be determined, and adjust an initial parameter of the garbage determination model according to the coordinate difference.
The processing module 502 may further be configured to store the garbage determination model after adjusting the parameter when the coordinate difference value meets a first preset condition.
The processing module 502 may further be configured to perform convolution operation on the first image feature map and the first point cloud feature map respectively, and generate a preprocessed image feature map and a preprocessed point cloud feature map correspondingly.
The processing module 502 may further be configured to generate a bounding box of the object to be determined in the first image feature map and a bounding box of the object to be determined in the first point cloud feature map according to the preprocessed image feature map and the preprocessed point cloud feature map, respectively.
The processing module 502 may further be configured to determine an object frame in the first image feature map according to a bounding box of an object to be determined in the first image feature map.
The processing module 502 may further be configured to determine an object frame in the first point cloud feature map according to a bounding box of the object to be determined in the first point cloud feature map.
The processing module 502 may further be configured to determine, when the detection evaluation function IOU of the bounding box of the object to be determined in the first image feature map is greater than a first preset threshold, that the bounding box of the object to be determined in the first image feature map is the object frame in the first image feature map.
The processing module 502 may further be configured to determine that the bounding box of the object to be determined in the first point cloud feature map is the object box in the first point cloud feature map when the detection evaluation function IOU of the bounding box of the object to be determined in the first point cloud feature map is greater than a first preset threshold.
The processing module 502 may further be configured to determine the first frame to be detected according to the detection evaluation function IOU of the bounding box of the object to be determined.
The processing module 502 may further be configured to correspondingly obtain a second image feature map and a second point cloud feature map from the first image feature map and the first point cloud feature map respectively according to the first frame to be measured.
The processing module 502 may further be configured to superimpose the second image feature map and the second point cloud feature map to generate a second frame to be detected.
The processing module 502 may further be configured to determine a coordinate difference between the second frame to be detected and a preset real frame of the object to be determined.
The processing module 502 may further be configured to store the garbage determination model after adjusting the parameter when the coordinate difference value meets a second preset condition.
The device for training the garbage determination model, provided by the embodiment of the invention, obtains the image data and the point cloud data from the image sensor and the radar sensor, obtains the three-dimensional boundary frame of the object to be determined from the image data and the point cloud data, and adjusts the parameters of the garbage determination model according to the difference between the three-dimensional boundary frame of the object to be determined and the real frame to obtain the garbage determination model, so that the problems of weak data sensing capability and poor logic realization in the prior art are solved, the data sensing capability is improved, and the realization logic is optimized.
Corresponding to the embodiment of the method for determining the garbage, the embodiment of the invention also provides a device for determining the garbage.
As shown in fig. 6, fig. 6 is a schematic structural diagram of a device for determining garbage according to an embodiment of the present invention.
The garbage determination apparatus may include: an acquisition module 601, and a processing module 602, wherein,
the acquisition module 601 may be used to acquire image data from an image sensor and point cloud data from a radar sensor.
The processing module 602 may be configured to input the image data and the point cloud data into the garbage determination model after the parameters are adjusted, and output the attribute of the object to be determined and the three-dimensional bounding box of the object to be determined.
The garbage determination model may be obtained by any one of the methods for training the garbage determination model in fig. 2, and the image data and the point cloud data are input into the garbage determination model after the parameters are adjusted, so that the attributes of the object to be determined (for example, a glass bottle, a plastic bottle, and a paper box) and the three-dimensional bounding box (position) of the object to be determined may be determined.
According to the device for determining the garbage, provided by the embodiment of the invention, the image data and the depth data are obtained from the image sensor and the radar sensor and are input into the garbage determination model after the parameters are adjusted, and the garbage determination model determines the attribute and the three-dimensional boundary frame of the object to be determined from the image data and the depth data, so that the problems of weak data sensing capability and poor logic realization existing in the prior art are solved, the data sensing capability is improved, and the realization logic is optimized.
FIG. 7 sets forth a block diagram of an exemplary hardware architecture of a computing device capable of implementing a method of spam determination model training, or a method of spam determination, according to embodiments of the present invention. As shown in fig. 7, computing device 700 includes an input device 701, an input interface 702, a central processor 703, a memory 704, an output interface 705, and an output device 706. The input interface 702, the central processing unit 703, the memory 704, and the output interface 705 are connected to each other via a bus 710, and the input device 701 and the output device 706 are connected to the bus 710 via the input interface 702 and the output interface 705, respectively, and further connected to other components of the computing device 700.
Specifically, the input device 701 receives input information from the outside, and transmits the input information to the central processor 703 through the input interface 702; the central processor 703 processes input information based on computer-executable instructions stored in the memory 704 to generate output information, stores the output information temporarily or permanently in the memory 704, and then transmits the output information to the output device 706 through the output interface 705; the output device 706 outputs output information external to the computing device 700 for use by a user.
That is, the computing device shown in fig. 7 may also be implemented as, or include: a memory storing computer-executable instructions; and a processor, which when executing the computer-executable instructions, may implement the method for training the garbage determination model or the method for garbage determination provided by the embodiments of the present invention.
An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium has computer program instructions stored thereon; the computer program instructions, when executed by a processor, implement a method for training a spam determination model, or a method for spam determination, provided by embodiments of the present invention.
It is to be understood that the invention is not limited to the specific arrangements and instrumentality described above and shown in the drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications and additions or change the order between the steps after comprehending the spirit of the present invention.
The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" may include any medium that can store or transfer information. Examples of a machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, Erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, Radio Frequency (RF) links, and so forth. The code segments may be downloaded via computer networks such as the internet, intranet, etc.
It should also be noted that the exemplary embodiments mentioned in this patent describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, may be performed in an order different from the order in the embodiments, or may be performed simultaneously.
As described above, only the specific embodiments of the present invention are provided, and it can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the module and the unit described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. It should be understood that the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various equivalent modifications or substitutions within the technical scope of the present invention, and these modifications or substitutions should be covered within the scope of the present invention.
Claims (13)
1. A garbage determination system, comprising:
the image sensor is used for acquiring image data of an object to be determined;
the radar sensor is used for acquiring point cloud data of an object to be determined;
the field programmable gate array FPGA is used for preprocessing the acquired image data and point cloud data of the object to be determined;
the image processor GPU is used for carrying out feature extraction on the preprocessed image data and point cloud data of the object to be determined, carrying out training on a garbage determination model, and determining the attribute of the object to be determined and the boundary frame of the object to be determined according to the trained garbage determination model;
the communication module is used for communicating the garbage determination system with the outside;
and the central processing unit CPU is used for connecting the communication module, the GPU and the FPGA and sending instructions to the GPU and the communication module.
2. A method of spam determination model training, the method comprising:
acquiring image data from an image sensor and point cloud data from a radar sensor;
respectively extracting feature data in the image data and the point cloud data, and correspondingly generating a first image feature map and a first point cloud feature map;
generating an object frame in the first image feature map and an object frame in the first point cloud feature map which have the same size according to the object frame in the first image feature map and the object frame in the first point cloud feature map respectively;
performing element fusion on the object frame in the first image feature map and the object frame in the first point cloud feature map which are the same in size to generate a first frame to be detected, wherein the first frame to be detected comprises a three-dimensional boundary frame of an object to be determined;
determining a coordinate difference value between the first frame to be measured and a real frame preset by the object to be determined, and adjusting initial parameters of a garbage determination model according to the coordinate difference value;
and when the coordinate difference value meets a first preset condition, saving the garbage determination model after the parameters are adjusted.
3. The method of claim 2, further comprising:
performing convolution operation on the first image feature map and the first point cloud feature map respectively, and correspondingly generating a preprocessed image feature map and a preprocessed point cloud feature map;
respectively generating a boundary frame of an object to be determined in the first image characteristic diagram and a boundary frame of the object to be determined in the first point cloud characteristic diagram according to the preprocessed image characteristic diagram and the preprocessed point cloud characteristic diagram;
determining an object frame in the first image feature map according to a boundary frame of an object to be determined in the first image feature map;
and determining an object frame in the first point cloud feature map according to a boundary frame of the object to be determined in the first point cloud feature map.
4. The method of claim 3,
when the detection evaluation function IOU of the boundary frame of the object to be determined in the first image feature map is larger than a first preset threshold value, determining the boundary frame of the object to be determined in the first image feature map as the object frame in the first image feature map;
when the detection evaluation function IOU of the boundary box of the object to be determined in the first point cloud feature map is larger than a first preset threshold value, determining the boundary box of the object to be determined in the first point cloud feature map as the object box in the first point cloud feature map.
5. The method according to claim 2, wherein when the coordinate difference satisfies a preset condition, saving the garbage determination model after adjusting the parameters comprises:
determining a first frame to be detected according to the detection evaluation function IOU of the boundary frame of the object to be determined;
according to the first frame to be detected, correspondingly obtaining a second image feature map and a second point cloud feature map from the first image feature map and the first point cloud feature map respectively;
superposing the second image characteristic diagram and the second point cloud characteristic diagram to generate a second frame to be detected;
determining a coordinate difference value between the second frame to be determined and a real frame preset by the object to be determined;
and when the coordinate difference value meets a second preset condition, saving the garbage determination model after the parameters are adjusted.
6. A method of garbage determination, the method comprising:
acquiring image data from an image sensor and point cloud data from a radar sensor;
inputting the image data and the point cloud data into a garbage determination model after parameter adjustment, and outputting the attribute of the object to be determined and a three-dimensional bounding box of the object to be determined, wherein the garbage determination model after parameter adjustment is obtained by a method for training the garbage determination model according to any one of claims 1 to 5.
7. An apparatus for garbage determination model training, the apparatus comprising:
the acquisition module is used for acquiring image data from the image sensor and point cloud data from the radar sensor;
the processing module is used for respectively extracting the feature data in the image data and the point cloud data and correspondingly generating a first image feature map and a first point cloud feature map;
the processing module is further configured to generate an object frame in the first image feature map and an object frame in the first point cloud feature map, which are the same in size, according to the object frame in the first image feature map and the object frame in the first point cloud feature map, respectively;
the processing module is further configured to perform element fusion on the object frame in the first image feature map and the object frame in the first point cloud feature map which are the same in size, so as to generate a first frame to be detected, where the first frame to be detected includes a three-dimensional bounding box of an object to be determined;
the processing module is further used for determining a coordinate difference value between the first frame to be detected and a real frame preset by the object to be determined, and adjusting initial parameters of the garbage determination model according to the coordinate difference value;
and the processing module is further used for saving the garbage determination model after the parameters are adjusted when the coordinate difference value meets a first preset condition.
8. The apparatus of claim 7,
the processing module is further configured to perform convolution operation on the first image feature map and the first point cloud feature map respectively, and generate a preprocessed image feature map and a preprocessed point cloud feature map correspondingly;
the processing module is further used for generating a boundary frame of an object to be determined in the first image feature map and a boundary frame of the object to be determined in the first point cloud feature map according to the preprocessed image feature map and the preprocessed point cloud feature map respectively;
the processing module is further configured to determine an object frame in the first image feature map according to a bounding box of an object to be determined in the first image feature map;
the processing module is further configured to determine an object frame in the first point cloud feature map according to a bounding box of an object to be determined in the first point cloud feature map.
9. The apparatus of claim 8,
the processing module is further configured to determine that the bounding box of the object to be determined in the first image feature map is the object box in the first image feature map when a detection evaluation function IOU of the bounding box of the object to be determined in the first image feature map is greater than a first preset threshold;
the processing module is further configured to determine that the bounding box of the object to be determined in the first point cloud feature map is the object box in the first point cloud feature map when a detection evaluation function IOU of the bounding box of the object to be determined in the first point cloud feature map is greater than a first preset threshold.
10. The apparatus of claim 7,
the processing module is further configured to determine a first frame to be detected according to the detection evaluation function IOU of the bounding box of the object to be determined;
the processing module is further configured to correspondingly acquire a second image feature map and a second point cloud feature map from the first image feature map and the first point cloud feature map respectively according to the first frame to be detected;
the processing module is further used for superposing the second image feature map and the second point cloud feature map to generate a second frame to be detected;
the processing module is further configured to determine a coordinate difference between the second frame to be determined and a real frame preset by the object to be determined;
and the processing module is further used for saving the garbage determination model after the parameters are adjusted when the coordinate difference value meets a second preset condition.
11. An apparatus for garbage determination, the apparatus comprising:
the acquisition module is used for acquiring image data from the image sensor and point cloud data from the radar sensor;
a processing module, configured to input the image data and the point cloud data into a garbage determination model with adjusted parameters, and output attributes of an object to be determined and a three-dimensional bounding box of the object to be determined, where the garbage determination model with adjusted parameters is obtained by the method for training the garbage determination model according to any one of claims 2 to 5.
12. A device for garbage determination, the device comprising: a processor and a memory storing computer program instructions;
the processor when executing the computer program instructions implements a method of spam determination model training as claimed in any of claims 2-5 or the processor when executing the computer program instructions implements a method of spam determination as claimed in claim 6.
13. A computer storage medium having stored thereon computer program instructions which, when executed by a processor, implement a method of spam determination model training as claimed in any of claims 2-5, or which, when executed by the processor, implement a spam determination method as claimed in claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010489021.XA CN113763465A (en) | 2020-06-02 | 2020-06-02 | Garbage determination system, model training method, determination method and determination device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010489021.XA CN113763465A (en) | 2020-06-02 | 2020-06-02 | Garbage determination system, model training method, determination method and determination device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113763465A true CN113763465A (en) | 2021-12-07 |
Family
ID=78782702
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010489021.XA Pending CN113763465A (en) | 2020-06-02 | 2020-06-02 | Garbage determination system, model training method, determination method and determination device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113763465A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106969721A (en) * | 2017-02-20 | 2017-07-21 | 深圳大学 | A kind of method for three-dimensional measurement and its measurement apparatus |
| CN109118564A (en) * | 2018-08-01 | 2019-01-01 | 湖南拓视觉信息技术有限公司 | A kind of three-dimensional point cloud labeling method and device based on fusion voxel |
| CN109948661A (en) * | 2019-02-27 | 2019-06-28 | 江苏大学 | A kind of 3D vehicle checking method based on Multi-sensor Fusion |
| CN110032949A (en) * | 2019-03-22 | 2019-07-19 | 北京理工大学 | A kind of target detection and localization method based on lightweight convolutional neural networks |
| CN110427917A (en) * | 2019-08-14 | 2019-11-08 | 北京百度网讯科技有限公司 | Method and apparatus for detecting key point |
| CN111027401A (en) * | 2019-11-15 | 2020-04-17 | 电子科技大学 | End-to-end target detection method with integration of camera and laser radar |
| CN111201451A (en) * | 2018-02-09 | 2020-05-26 | 宝马股份公司 | Method and device for detecting object in scene based on laser data and radar data of scene |
-
2020
- 2020-06-02 CN CN202010489021.XA patent/CN113763465A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106969721A (en) * | 2017-02-20 | 2017-07-21 | 深圳大学 | A kind of method for three-dimensional measurement and its measurement apparatus |
| CN111201451A (en) * | 2018-02-09 | 2020-05-26 | 宝马股份公司 | Method and device for detecting object in scene based on laser data and radar data of scene |
| CN109118564A (en) * | 2018-08-01 | 2019-01-01 | 湖南拓视觉信息技术有限公司 | A kind of three-dimensional point cloud labeling method and device based on fusion voxel |
| CN109948661A (en) * | 2019-02-27 | 2019-06-28 | 江苏大学 | A kind of 3D vehicle checking method based on Multi-sensor Fusion |
| CN110032949A (en) * | 2019-03-22 | 2019-07-19 | 北京理工大学 | A kind of target detection and localization method based on lightweight convolutional neural networks |
| CN110427917A (en) * | 2019-08-14 | 2019-11-08 | 北京百度网讯科技有限公司 | Method and apparatus for detecting key point |
| CN111027401A (en) * | 2019-11-15 | 2020-04-17 | 电子科技大学 | End-to-end target detection method with integration of camera and laser radar |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7073247B2 (en) | Methods for generating lane boundary detection models, methods for detecting lane boundaries, devices for generating lane boundary detection models, devices for detecting lane boundaries, equipment, computers readable Storage media and computer programs | |
| CN110674829B (en) | Three-dimensional target detection method based on graph convolution attention network | |
| CN113139543B (en) | Training method of target object detection model, target object detection method and equipment | |
| CN113378760A (en) | Training target detection model and method and device for detecting target | |
| CN103839258A (en) | Depth perception method of binarized laser speckle images | |
| CN111914843B (en) | Character detection method, system, equipment and storage medium | |
| US20240193923A1 (en) | Method of training target object detection model, method of detecting target object, electronic device and storage medium | |
| US20190164259A1 (en) | Method and apparatus for removing turbid objects in an image | |
| CN108197567B (en) | Method, apparatus and computer readable medium for image processing | |
| CN112989995B (en) | Text detection method and device and electronic equipment | |
| CN113267761B (en) | Laser radar target detection and identification method, system and computer readable storage medium | |
| CN113591566A (en) | Training method and device of image recognition model, electronic equipment and storage medium | |
| CN112819796A (en) | Tobacco shred foreign matter identification method and equipment | |
| CN113657274A (en) | Table generation method and device, electronic equipment, storage medium and product | |
| CN113409461A (en) | Method and device for constructing landform map, electronic equipment and readable storage medium | |
| CN112750139A (en) | Image processing method and device, computing equipment and storage medium | |
| CN114283343B (en) | Map updating method, training method and device based on remote sensing satellite image | |
| CN109492697B (en) | Picture detection network training method and picture detection network training device | |
| CN117315406B (en) | Sample image processing method, device and equipment | |
| CN113409340A (en) | Semantic segmentation model training method, semantic segmentation device and electronic equipment | |
| CN114638996A (en) | Model training method, device, equipment and storage medium based on counterstudy | |
| CN113763465A (en) | Garbage determination system, model training method, determination method and determination device | |
| CN111144361A (en) | Road lane detection method based on binaryzation CGAN network | |
| CN111079749A (en) | End-to-end commodity price tag character recognition method and system with attitude correction function | |
| CN113706705B (en) | Image processing method, device, equipment and storage medium for high-precision map |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |