CN114882299B - Fruit sorting method and device, picking equipment and storage medium - Google Patents
Fruit sorting method and device, picking equipment and storage medium Download PDFInfo
- Publication number
- CN114882299B CN114882299B CN202210809892.4A CN202210809892A CN114882299B CN 114882299 B CN114882299 B CN 114882299B CN 202210809892 A CN202210809892 A CN 202210809892A CN 114882299 B CN114882299 B CN 114882299B
- Authority
- CN
- China
- Prior art keywords
- fruit
- classified
- network model
- ith
- quality information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/68—Food, e.g. fruit or vegetables
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Sorting Of Articles (AREA)
- Image Processing (AREA)
Abstract
The application relates to a fruit sorting method, a fruit sorting device, picking equipment and a storage medium, wherein the method comprises the following steps: s1, obtaining ith image information corresponding to fruits to be classified, and inputting the ith image information into an ith pre-trained network model to obtain quality information of ith dimensionality of the fruits to be classified; s2, classifying the fruits to be classified according to the ith dimension quality information to obtain an ith classification result; s3, determining any fruit in the ith classification result as a fruit to be classified, and repeatedly executing S1 to S3 to obtain quality information of the Nth dimension; s4, classifying the fruits to be classified according to the quality information of the Nth dimension; the quantity of the network models corresponds to the dimensionality of the quality information of the fruits to be classified, and different network models are used for predicting the quality information of different dimensionalities of the fruits to be classified. Therefore, the classification result of the fruits to be classified can be obtained, manual classification is not needed, and the labor cost and the time cost are saved.
Description
Technical Field
The application relates to the technical field of deep learning, in particular to a fruit classification method and device, picking equipment and a storage medium.
Background
At present, fruit picking is gradually mechanized, and various fruit picking devices appear on the market, so that the convenience of fruit picking is greatly improved, and the danger of high-altitude picking can be effectively reduced. However, the existing fruit picking device cannot classify the fruits in different dimensions such as shape, size and defects, and still needs to sort the fruits manually, so that the labor cost and the time cost are high.
Disclosure of Invention
The application provides a fruit classifying method, a fruit classifying device, picking equipment and a storage medium, which are used for solving the problem that the existing fruit picking device cannot classify different dimensions such as the shape, the size and the defects of fruits and still needs manual sorting, so that the labor cost and the time cost are high.
In a first aspect, the present application provides a method of fruit classification, the method comprising:
s1, obtaining ith image information corresponding to a fruit to be classified, inputting the ith image information into an ith network model trained in advance, and obtaining quality information of an ith dimension of the fruit to be classified, wherein i is a positive integer which is greater than or equal to 1 and less than or equal to N, and N is any integer which is greater than 1;
s2, classifying the fruits to be classified according to the quality information of the ith dimension to obtain an ith classification result;
s3, determining any type of fruit in the ith classification result as the fruit to be classified, and repeatedly executing the S1 to the S3 to obtain the quality information of the Nth dimension; the execution of the S1 to the S3 is repeatedly completed every time when the value of i starts from 1, and the value of i is added with 1 until the value of i is N;
s4, classifying the fruits to be classified according to the quality information of the Nth dimension;
the quantity of the network models corresponds to the dimensionality of the quality information of the fruit to be classified, and different network models are used for predicting the quality information of different dimensionalities of the fruit to be classified.
Optionally, the ith network model comprises an input layer, an output layer and a plurality of hidden layers;
the step of inputting the ith image information into an ith network model trained in advance to obtain the ith dimensional quality information of the fruits to be classified comprises the following steps:
inputting the ith image information into the input layer for feature digitization to obtain first intermediate feature information;
sequentially inputting the first intermediate feature information to each hidden layer in the multiple hidden layers for feature extraction to obtain second intermediate feature information;
and inputting the second intermediate characteristic information into the output layer for quality classification to obtain the quality information of the ith dimension of the fruit to be classified.
Optionally, the ith network model is a first network model, a second network model, a third network model or a fourth network model, the first network model is used for predicting the shape of the fruit to be classified, the second network model is used for predicting the size of the fruit to be classified, the third network model is used for predicting the defect of the fruit to be classified, and the fourth network model is used for predicting the taste of the fruit to be classified.
Optionally, each hidden layer in the first network model, the second network model, and the third network model includes a pooling layer and a plurality of convolutional layers, each convolutional layer includes a plurality of nodes, the number of convolutional kernels of each node is multiplied, and the sizes of the convolutional kernels corresponding to each node are equal.
Optionally, the mouthfeel of the fruit to be classified comprises sweetness of the fruit to be classified, and the fourth network model is used for fitting the sweetness of the fruit to be classified according to the size and weight of the fruit to be classified.
Optionally, before the inputting the ith image information into a pre-trained ith network model to obtain quality information of an ith dimension of the fruit to be classified, the method further includes:
acquiring N types of sample image information corresponding to the quality information of N dimensions of the fruit;
respectively inputting the N types of sample image information into N models to be trained for training, and training to obtain N network models under the condition that the loss values of the N models to be trained are all smaller than the corresponding preset threshold values;
the number of the network models corresponds to the dimensionality of the quality information of the fruit, and different network models are used for predicting the quality information of different dimensionalities of the fruit.
Optionally, after the obtaining N types of sample image information corresponding to the N dimensions of quality information of the fruit, the method further includes:
carrying out image centering processing on the N types of sample image information;
carrying out image enhancement processing on the image information of the N types of samples after image centering processing;
cutting the N types of sample image information subjected to image enhancement processing to obtain N types of sample image information with preset size;
respectively inputting the N types of sample image information into N models to be trained for training, and obtaining N network models by training under the condition that the loss values of the N models to be trained are smaller than the corresponding preset threshold values, wherein the training comprises the following steps:
and respectively inputting the N types of sample image information with the preset size into N models to be trained, and training to obtain N network models under the condition that the loss values of the N models to be trained are smaller than the corresponding preset threshold value.
In a second aspect, the present application also provides a fruit sorting device, the device comprising:
the first processing module is used for executing the step S1, obtaining ith image information corresponding to the fruit to be classified, inputting the ith image information to an ith pre-trained network model, and obtaining quality information of the ith dimension of the fruit to be classified, wherein i is a positive integer which is greater than or equal to 1 and less than or equal to N, and N is any integer which is greater than 1;
the first classification module is used for executing the step S2 and classifying the fruits to be classified according to the quality information of the ith dimension to obtain an ith classification result;
a repeated execution module, configured to execute step S3, determine any type of fruit in the ith classification result as the fruit to be classified, and repeatedly execute the steps S1 to S3 to obtain quality information of an nth dimension; the execution of the S1 to the S3 is repeatedly completed every time when the value of i starts from 1, and the value of i is added with 1 until the value of i is N;
a circulation module, configured to determine any type of fruit in the ith classification result as the fruit to be classified, and repeatedly execute the steps S1 to S3 to obtain quality information of an nth dimension; the execution of the S1 to the S3 is repeatedly completed every time the value of the i starts from 1, and the value of the i is added with 1 until the value of the i is N;
a second classification module, configured to perform step S4, classify the fruit to be classified according to the quality information of the nth dimension;
the quantity of the network models corresponds to the dimensionality of the quality information of the fruits to be classified, and different network models are used for predicting the quality information of different dimensionalities of the fruits to be classified.
In a third aspect, the present application further provides a picking apparatus, including the fruit classifying device according to any one of the embodiments of the second aspect and a plurality of conveying channels, where the fruit classifying device is configured to predict N classification results corresponding to N-dimensional quality information of a fruit to be classified, and convey the fruit to be classified to the plurality of conveying channels according to the N classification results.
In a fourth aspect, a computer-readable storage medium is provided, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of fruit sorting according to any one of the embodiments of the first aspect.
In the embodiment of the application, S1, obtaining ith image information corresponding to a fruit to be classified, and inputting the ith image information into an ith pre-trained network model to obtain quality information of an ith dimension of the fruit to be classified, wherein i is a positive integer which is greater than or equal to 1 and less than or equal to N, and N is any integer which is greater than 1; s2, classifying the fruits to be classified according to the ith dimension quality information to obtain an ith classification result; s3, determining any fruit in the ith classification result as the fruit to be classified, and repeatedly executing the S1 to the S3 to obtain quality information of the Nth dimension; the execution of the S1 to the S3 is repeatedly completed every time when the value of i starts from 1, and the value of i is added with 1 until the value of i is N; s4, classifying the fruits to be classified according to the quality information of the Nth dimension; the quantity of the network models corresponds to the dimensionality of the quality information of the fruits to be classified, and different network models are used for predicting the quality information of different dimensionalities of the fruits to be classified. Through the execution of the steps S1 to S4, the quality information of N dimensions of the fruit to be classified can be predicted, and N times of classification can be carried out according to the predicted quality information of the N dimensions, so that a refined classification result of the fruit to be classified is obtained, manual classification of the fruit to be classified is not needed, and the labor cost and the time cost are saved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a schematic flow chart of a fruit classification method provided in an embodiment of the present application;
fig. 2 is a schematic structural diagram of a model structure of each network model provided in the embodiment of the present application;
fig. 3 is a flowchart for classifying fruits to be classified respectively by using a first network model, a second network model, a third network model and a fourth network model according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a fruit sorting device provided in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making creative efforts shall fall within the protection scope of the present application.
Referring to fig. 1, fig. 1 is a schematic flow chart of a fruit sorting method provided in an embodiment of the present application. As shown in fig. 1, the fruit sorting method includes:
s1, obtaining ith image information corresponding to the fruit to be classified, inputting the ith image information into an ith network model trained in advance, and obtaining quality information of the ith dimension of the fruit to be classified, wherein i is a positive integer which is greater than or equal to 1 and less than or equal to N, and N is any integer which is greater than 1.
Specifically, the fruits to be classified may be the same kind of fruits or different kinds of fruits, and the application is not particularly limited. For example, when the fruit to be classified is only apples, the dimensions of the apples, such as shape, size, defects, and the like, can be classified; when the fruits to be classified comprise apples, oranges and pears, the apples, the oranges and the pears can be classified firstly, and then dimensions such as shapes, sizes, defects and the like of the apples, the oranges and the pears are classified respectively. When i is equal to 1, the ith image information can be understood as image information corresponding to fruits which are just picked up and are not classified temporarily; when i is greater than 1, the ith image information can be understood as image information corresponding to any classified fruit after the classification of the (i-1) th time is completed. The mode of obtaining the image information can be obtained by shooting through the cameras, and in order to enable the obtained characteristic information to be more complete, the multiple cameras can be arranged to obtain the all-dimensional image information of the fruits. The ith network model is a pre-selected trained deep learning model and is used for predicting the quality information of the ith dimension of the fruits to be classified. The quality information here can include, but is not limited to, dimension information of shape, size, defect type, etc. of the fruit, where i can be any positive integer greater than or equal to 1 and less than or equal to N.
And S2, classifying the fruits to be classified according to the quality information of the ith dimension to obtain an ith classification result.
After the quality information of the ith dimension is obtained, classifying the fruits to be classified based on the quality information of the ith dimension to obtain an ith classification result. It should be noted that, assuming that the i-1 th classification result obtained by the i-1 st classification includes three types, namely, a result 1, a result 2, and a result 3, in the i-th classification, image information needs to be acquired for the three types respectively, and then the acquired image information is input to the i-th network model respectively to obtain classification results corresponding to the three types, which are used as the i-th classification result.
S3, determining any fruit in the ith classification result as a fruit to be classified, and repeatedly executing S1 to S3 to obtain quality information of the Nth dimension; and repeating the execution of S1 to S3 from 1 to 1, and adding 1 to the value of i until the value of i is N.
After the ith classification is completed, any type of fruit in the ith classification result can be determined as a fruit to be classified, and the steps S1 to S3 are repeatedly executed, so that the quality information of the nth dimension can be obtained. That is, after the ith classification is completed, assuming that the obtained ith classification result includes three types, namely, a result 1, a result 2 and a result 3, in the (i + 1) th classification, image information needs to be obtained for the three types respectively, and then the obtained image information is input to the (i + 1) th network model respectively to obtain classification results corresponding to the three types, which are taken as the (i + 1) th classification result. And circulating the steps until the quality information of the Nth dimension of the fruit to be classified is obtained.
And S4, classifying the fruits to be classified according to the quality information of the Nth dimension, wherein the number of the network models corresponds to the dimension of the quality information of the fruits to be classified, and different network models are used for predicting the quality information of the fruits to be classified in different dimensions.
It should be noted that the network models correspond to the dimensions of the quality information of the fruits to be classified one by one, and different network models are used for predicting the quality information of different dimensions of the fruits to be classified, so that the fruits to be classified are sequentially input into the network models, the quality information of the fruits to be classified can be sequentially obtained, and then the quality information is classified, so that the classification of the fruits to be classified is continuously refined, and the effect of accurate classification is realized.
In this embodiment, by executing the steps S1 to S4, the quality information of N dimensions of the fruit to be classified can be predicted, and N times of classification can be performed according to the predicted quality information of N dimensions, so as to obtain a refined classification result of the fruit to be classified, and the fruit to be classified does not need to be classified manually, so that the labor cost and the time cost are saved.
Further, the ith network model comprises an input layer, an output layer and a plurality of hidden layers;
inputting the ith image information into an ith network model trained in advance to obtain the quality information of the ith dimensionality of the fruit to be classified, wherein the quality information comprises the following steps:
inputting the ith image information into an input layer for feature numeralization to obtain first intermediate feature information;
sequentially inputting the first intermediate feature information to each hidden layer of the multiple hidden layers for feature extraction to obtain second intermediate feature information;
and inputting the second intermediate characteristic information to an output layer for quality classification to obtain the quality information of the ith dimension of the fruit to be classified.
In an embodiment, each network model is a deep learning model, and the model structure of each network model may include an input layer, an output layer, and a plurality of hidden layers, as shown in fig. 2. When the quality information of the ith dimension of the fruit to be classified is obtained, the ith image information of the fruit to be classified can be input into an input layer to be subjected to feature digitization to obtain first intermediate feature information, the first intermediate feature information is sequentially input into each hidden layer of a plurality of hidden layers to be subjected to feature extraction to obtain second intermediate feature information, and finally the second intermediate feature information is input into an output layer to be subjected to quality classification to obtain the quality information of the ith dimension of the fruit to be classified.
It should be noted that, because each network model is used for predicting quality information of different dimensions of the fruit to be classified, the model parameters used by each network model are different. For example, the N network models are respectively used for predicting quality information of N dimensions of fruits to be classified, so that the N models to be trained need to be trained by using sample images of the N dimensions, and as extracted feature data are different dimensions, trained model parameters thereof are different inevitably. In addition, the activation function, the loss function, and the like used in each network model may be flexibly set as needed, and may be the same or different.
Furthermore, the ith network model is a first network model, a second network model, a third network model or a fourth network model, the first network model is used for predicting the shape of the fruit to be classified, the second network model is used for predicting the size of the fruit to be classified, the third network model is used for predicting the defects of the fruit to be classified, and the fourth network model is used for predicting the taste of the fruit to be classified.
In an embodiment, fruits to be classified can be classified by using a first network model, a second network model, a third network model and a fourth network model, wherein the first network model is used for predicting the shape of the fruits to be classified, the second network model is used for predicting the size of the fruits to be classified, the third network model is used for predicting the defects of the fruits to be classified, and the fourth network model is used for predicting the mouthfeel of the fruits to be classified, wherein the mouthfeel includes but is not limited to sweetness, softness and the like. In this embodiment, a first network model, a second network model, a third network model and a fourth network model may be used to classify fruits to be classified respectively, and a flowchart thereof is as shown in fig. 3, wherein 1 st image information of the fruits to be classified is obtained and input into the first network model, assuming that when the shape of the fruits to be classified is predicted by using the first network model to obtain shape 1, shape 2 and shape 3 (e.g. very beautiful shape, more beautiful shape and non-beautiful shape), then the fruits in shape 1, shape 2 and shape 3 may be classified and photographed, and then the obtained 2 nd image information of the fruits to be classified is input into the second network model to predict the size of the fruits to be classified, assuming that the fruits in each shape type can be predicted to obtain size 1, size 2 and size 3 (e.g. large size, common size, small size), then 9 prediction results can be obtained, then fruits corresponding to the 9 prediction results are classified and photographed, then the obtained 3 rd image information of the fruits to be classified is input into a third network model to predict defects of the fruits to be classified, if the fruits in each prediction result can be predicted to obtain defects 1, 2 and 3 (such as rotten surfaces, bump surfaces and wormholes on the surfaces), then 27 prediction results can be obtained, finally the fruits corresponding to the 27 prediction results are classified and photographed, then the obtained 4 th image information of the fruits to be classified is input into a fourth network model to predict the mouthfeel of the fruits to be classified, and if the fruits in each prediction result can be predicted to obtain 1, 2 and 3 mouthfeel (such as very good taste), taste is normal, taste is poor), 81 predictions can be made. Like this, through will treat categorised fruit and input 4 network models in proper order, can obtain in proper order and treat categorised fruit in the classification on shape, size, defect and taste for the classification of treating categorised fruit constantly refines, realizes accurate categorised effect. Of course, in practical applications, the output type corresponding to each network model may be any positive integer greater than 1, such as 2, 4, 5, and so on, and is not limited to the above example 3.
Further, each hidden layer in the first network model, the second network model and the third network model comprises a pooling layer and a plurality of convolution layers, each convolution layer comprises a plurality of nodes, the number of convolution kernels of each node is multiplied, and the sizes of the convolution kernels corresponding to each node are equal.
In an embodiment, the first network model, the second network model and the third network model have the same model structure, and each of the first network model, the second network model and the third network model includes a plurality of hidden layers, and each hidden layer includes a pooling layer and a plurality of convolutional layers. Each convolution layer includes a plurality of nodes, the number of convolution kernels of each node is multiplied, and the sizes of the convolution kernels corresponding to the nodes are equal. As an optional implementation manner, the first network model, the second network model, and the third network model may each include a 1-layer input layer, a 5-layer hidden layer, and a 1-layer output layer, where the input layer is configured to perform feature digitization on image data and then input the image data to the hidden layer. The output layer is used for outputting the result, the number of neurons of the output layer is generally set as the number of categories, if the number of fruit categories is 10, the number of neurons (i.e. nodes) of the output layer is 10, and the pictures are classified according to the probability values of the 10 categories in the pictures. Each hidden layer may include 5 convolutional layers and 1 pooling layer, each convolutional layer may include 5 nodes, the number of convolutional kernels of each convolutional layer corresponding to node 1 is 64, and the size of the convolutional kernels is set to 3 × 3; the number of convolution kernels of each convolution layer corresponding to the node 2 is 128, and the size of each convolution kernel is set to be 3 x 3; the number of convolution kernels of the node 3 is 256, and the size of the convolution kernels is set to be 3 x 3; the number of convolution kernels of the node 4 is 512, and the size of the convolution kernels is set to be 3 x 3; the number of convolution kernels of node 5 is 512, and the convolution kernel size is set to 3 × 3. Padding = "same" is set for each node, so that after each convolution, the data size will be filled to be consistent with the original size. Furthermore, for a pooling layer, the pooling layer window of each node is 2 x2, with a step size of 2 x 2. Padding = "same" is set, which can make the size of the input picture still keep unchanged during the transmission process.
Further, the taste of the fruit to be classified comprises the sweetness of the fruit to be classified, and the fourth network model is used for fitting the sweetness of the fruit to be classified according to the size and the weight of the fruit to be classified.
In an embodiment, the sweetness of the fruit may be classified by using the fourth network model, and specifically, the size (i.e. volume) of the fruit to be classified may be predicted by using the second network model, and the weight of the fruit to be classified may be measured. And inputting the volume and the weight of the fruit to be classified into the fourth network model for prediction, so as to obtain the sweetness of the fruit to be classified. It should be noted that, since the volume and weight of the fruit to be classified and the sweetness of the fruit to be classified are in a non-linear relationship, the following formula can be adopted as the activation function of the fourth network model:
wherein S (x) represents an activation function for the fourth network model, x1 and x2 represent the size and weight of the fruit, respectively, a and b are both coefficients, and c is a constant.
In this way, the activation function may introduce a non-linear factor, deriving the sweetness of the fruit to be classified from the volume and weight of the fruit to be classified.
Further, before the above step, inputting the ith image information into the ith pre-trained network model to obtain the quality information of the ith dimension of the fruit to be classified, the method further includes:
acquiring N types of sample image information corresponding to the quality information of N dimensions of the fruit;
respectively inputting the N types of sample image information into N models to be trained for training, and training to obtain N network models under the condition that the loss values of the N models to be trained are smaller than the corresponding preset threshold value;
the number of the network models corresponds to the dimensionality of the quality information of the fruits, and different network models are used for predicting the quality information of different dimensionalities of the fruits.
In an embodiment, before predicting quality information of N dimensions of a fruit to be classified by using N network models, N models to be trained need to be trained to obtain N network models. Specifically, N types of sample image information corresponding to N dimensions of quality information of a fruit may be obtained, where N is any positive integer greater than 1, and each type of sample image information carries a label of the quality information of one dimension of the fruit, such as a shape, a size, a defect, or sweetness of the fruit. In this way, the N types of sample image information are respectively input to the N models to be trained for training, and the N network models are obtained through training under the condition that the loss values of the N models to be trained are smaller than the corresponding preset threshold value. And the model structures of the N models to be trained correspond to the model structures of the N network models one by one. In the training process, a cross-entropy cost function can be adopted to calculate the loss value, and the formula of the cross-entropy cost function is as follows:
wherein L is cee Representing the cost, y the real data, and a the model output value. Iteration is carried out continuously in model training, loss values are enabled to be as small as possible in each iteration, and the smaller the loss value is, the better the model performance is represented. And when the loss values of the N to-be-trained models are all smaller than the corresponding preset threshold value or the iteration training of the preset times is completed, obtaining N network models. As an alternative, the final N network models may be obtained after 50epoch iterations (1 epoch equals one training using all samples in the training set, in colloquial terms the value of epoch is the entire data set is rotated several times).
Further, after the above step of obtaining N types of sample image information corresponding to N dimensions of quality information of the fruit, the method further includes:
carrying out image centering processing on the N types of sample image information;
carrying out image enhancement processing on the image information of the N types of samples after image centering processing;
cutting the N types of sample image information subjected to image enhancement processing to obtain N types of sample image information with preset sizes;
the above steps, respectively inputting the N types of sample image information to N models to be trained for training, and training to obtain N network models when the loss values of the N models to be trained are all smaller than the corresponding preset threshold value, including:
and respectively inputting N types of sample image information with preset sizes into N models to be trained for training, and training to obtain N network models under the condition that the loss values of the N models to be trained are smaller than the corresponding preset threshold values.
Specifically, the image centering processing refers to selecting a region with obvious features from the ith image information, and is used for matching images of which parts of the ith image information are fruits. The image enhancement processing is image processing for making an original unclear image clear or emphasizing some interesting features, suppressing the uninteresting features, improving the image quality, enriching the information content, and enhancing the image interpretation and recognition effects. The cropping means cropping according to a certain size to obtain an image with a uniform size.
In an embodiment, image centering processing may be performed on the N-type sample image information, image enhancement processing may be performed on the N-type sample image information after the image centering processing, then the N-type sample image information after the image enhancement processing is cut to obtain N-type sample image information with a preset size, and finally the obtained N-type sample image information with the preset size is respectively input to the N models to be trained for training, and when the loss values of the N models to be trained are all smaller than the preset threshold value corresponding to the N models, the N network models are trained to obtain. In this way, the training efficiency and precision of the model can be improved by performing image centering processing, image enhancement processing and clipping on the N types of sample image information.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a fruit sorting device provided in an embodiment of the present application. As shown in fig. 4, the fruit sorting apparatus 400 includes:
a first processing module 401, configured to execute step S1, obtain ith image information corresponding to a fruit to be classified, and input the ith image information to an ith network model trained in advance, to obtain quality information of an ith dimension of the fruit to be classified, where i is a positive integer greater than or equal to 1 and less than or equal to N, and N is any integer greater than 1;
a first classification module 402, configured to execute step S2, classify the fruit to be classified according to the quality information of the ith dimension, and obtain an ith classification result;
a repeated execution module 403, configured to execute step S3, determine any type of fruit in the ith classification result as a fruit to be classified, and repeatedly execute S1 to S3 to obtain quality information of the nth dimension; the execution of S1 to S3 is repeatedly completed every time when the value of i starts from 1, and the value of i is added with 1 until the value of i is N;
a loop module 404, configured to determine any type of fruit in the ith classification result as a fruit to be classified, and repeatedly execute S1 to S3 to obtain quality information of the nth dimension; the execution of S1 to S3 is repeatedly completed every time when the value of i starts from 1, and the value of i is added with 1 until the value of i is N;
the second classification module is used for executing the step S4 and classifying the fruits to be classified according to the quality information of the Nth dimension;
the quantity of the network models corresponds to the dimensionality of the quality information of the fruits to be classified, and different network models are used for predicting the quality information of different dimensionalities of the fruits to be classified.
Further, the ith network model comprises an input layer, an output layer and a plurality of hidden layers;
the first processing module 401 includes:
the numeralization submodule is used for inputting the ith image information into the input layer for characteristic numeralization to obtain first intermediate characteristic information;
the characteristic extraction submodule is used for sequentially inputting the first intermediate characteristic information to each hidden layer of the hidden layers to carry out characteristic extraction so as to obtain second intermediate characteristic information;
and the quality classification submodule is used for inputting the second intermediate characteristic information to the output layer for quality classification to obtain the ith dimension quality information of the fruits to be classified.
Furthermore, the ith network model is a first network model, a second network model, a third network model or a fourth network model, the first network model is used for predicting the shape of the fruit to be classified, the second network model is used for predicting the size of the fruit to be classified, the third network model is used for predicting the defects of the fruit to be classified, and the fourth network model is used for predicting the taste of the fruit to be classified.
Further, each hidden layer in the first network model, the second network model and the third network model comprises a pooling layer and a plurality of convolution layers, each convolution layer comprises a plurality of nodes, the number of convolution kernels of each node is multiplied, and the sizes of the convolution kernels corresponding to each node are equal.
Further, the taste of the fruit to be classified comprises the sweetness of the fruit to be classified, and the fourth network model is used for fitting the sweetness of the fruit to be classified according to the size and the weight of the fruit to be classified.
Further, the apparatus 400 further comprises:
the acquisition module is used for acquiring N types of sample image information corresponding to the quality information of N dimensions of the fruits;
the training module is used for inputting the N types of sample image information into N models to be trained respectively for training, and under the condition that the loss values of the N models to be trained are smaller than the corresponding preset threshold values, N network models are obtained through training;
the number of the network models corresponds to the dimensionality of the quality information of the fruits, and different network models are used for predicting the quality information of different dimensionalities of the fruits.
Further, the apparatus 400 further comprises:
the image centering processing module is used for carrying out image centering processing on the N types of sample image information;
the image enhancement processing module is used for carrying out image enhancement processing on the N types of sample image information after image centering processing;
the cutting module is used for cutting the N types of sample image information subjected to image enhancement processing to obtain N types of sample image information with preset size;
and the training module is further used for inputting N types of sample image information with preset sizes into the N models to be trained respectively for training, and under the condition that the loss values of the N models to be trained are smaller than the corresponding preset threshold values, the N network models are obtained through training.
It should be noted that the fruit sorting apparatus 400 can implement the steps of the fruit sorting method provided in any one of the foregoing method embodiments, and can achieve the same technical effects, which are not described in detail herein.
The picking equipment provided by the embodiment of the application comprises the fruit classifying device and the plurality of conveying channels, wherein the fruit classifying device is used for predicting N classifying results corresponding to N dimensionalities of fruits to be classified, and conveying the fruits to be classified to the plurality of conveying channels according to the N classifying results.
It should be noted that the picking device can implement the fruit sorting device provided in any of the foregoing embodiments, and can achieve the same technical effects, which are not described in detail herein.
Embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the fruit sorting method provided in any one of the method embodiments described above.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.
The above description is merely illustrative of particular embodiments of the invention that enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (7)
1. A fruit sorting method, for use in a picking apparatus, the method comprising:
s1, obtaining ith image information corresponding to a fruit to be classified, inputting the ith image information into an ith network model trained in advance, and obtaining quality information of an ith dimension of the fruit to be classified, wherein i is a positive integer which is greater than or equal to 1 and less than or equal to N, and N is any integer which is greater than 1;
s2, classifying the fruits to be classified according to the ith dimension quality information to obtain an ith classification result;
s3, determining any fruit in the ith classification result as the fruit to be classified, and repeatedly executing the S1 to the S3 to obtain quality information of the Nth dimension; the execution of the S1 to the S3 is repeatedly completed every time when the value of i starts from 1, and the value of i is added with 1 until the value of i is N;
s4, classifying the fruits to be classified according to the quality information of the Nth dimension;
the quantity of the network models corresponds to the dimensionality of the quality information of the fruit to be classified, and different network models are used for predicting the quality information of different dimensionalities of the fruit to be classified;
the ith network model is a first network model, a second network model, a third network model or a fourth network model, the first network model is used for predicting the shape of the fruit to be classified, the second network model is used for predicting the size of the fruit to be classified, the third network model is used for predicting the defects of the fruit to be classified, and the fourth network model is used for predicting the taste of the fruit to be classified; the mouthfeel of the fruit to be classified comprises the sweetness of the fruit to be classified, and the fourth network model is used for fitting the sweetness of the fruit to be classified according to the size and the weight of the fruit to be classified;
the ith network model comprises an input layer, an output layer and a plurality of hidden layers, each hidden layer in the first network model, the second network model and the third network model comprises a pooling layer and a plurality of convolution layers, each convolution layer comprises a plurality of nodes, the number of convolution kernels of each node is multiplied, and the sizes of the convolution kernels corresponding to each node are equal.
2. The method according to claim 1, wherein the inputting the ith image information into an ith network model trained in advance to obtain the quality information of the ith dimension of the fruit to be classified comprises:
inputting the ith image information into the input layer for feature digitization to obtain first intermediate feature information;
sequentially inputting the first intermediate feature information to each hidden layer of the plurality of hidden layers for feature extraction to obtain second intermediate feature information;
and inputting the second intermediate characteristic information into the output layer for quality classification to obtain the quality information of the ith dimension of the fruit to be classified.
3. The method according to claim 1, wherein before the inputting the ith image information into a pre-trained ith network model to obtain quality information of an ith dimension of the fruit to be classified, the method further comprises:
acquiring N types of sample image information corresponding to the quality information of N dimensions of the fruit;
respectively inputting the N types of sample image information into N models to be trained for training, and training to obtain N network models under the condition that the loss values of the N models to be trained are smaller than the corresponding preset threshold values;
the number of the network models corresponds to the dimensionality of the quality information of the fruit, and different network models are used for predicting the quality information of different dimensionalities of the fruit.
4. The method according to claim 3, wherein after the obtaining N types of sample image information corresponding to N dimensions of quality information of fruit, the method further comprises:
carrying out image centering processing on the N types of sample image information;
carrying out image enhancement processing on the N types of sample image information after image centering processing;
cutting the N types of sample image information subjected to image enhancement processing to obtain N types of sample image information with preset size;
respectively inputting the N types of sample image information into N models to be trained for training, and obtaining N network models by training under the condition that the loss values of the N models to be trained are smaller than the corresponding preset threshold values, wherein the training comprises the following steps:
and respectively inputting the N types of sample image information with the preset size into N models to be trained for training, and training to obtain N network models under the condition that the loss values of the N models to be trained are all smaller than the corresponding preset threshold values.
5. A fruit sorting device for use in a picking apparatus, the device comprising:
the first processing module is used for executing the step S1, obtaining ith image information corresponding to the fruit to be classified, inputting the ith image information to an ith pre-trained network model, and obtaining quality information of the ith dimension of the fruit to be classified, wherein i is a positive integer which is greater than or equal to 1 and less than or equal to N, and N is any integer which is greater than 1;
the first classification module is used for executing the step S2 and classifying the fruits to be classified according to the quality information of the ith dimension to obtain an ith classification result;
a repeated execution module, configured to execute step S3, determine any type of fruit in the ith classification result as the fruit to be classified, and repeatedly execute the steps S1 to S3 to obtain quality information of an nth dimension; the execution of the S1 to the S3 is repeatedly completed every time when the value of i starts from 1, and the value of i is added with 1 until the value of i is N;
a circulation module, configured to determine any type of fruit in the ith classification result as the fruit to be classified, and repeatedly execute the steps S1 to S3 to obtain quality information of an nth dimension; the execution of the S1 to the S3 is repeatedly completed every time when the value of the i starts from 1, and the value of the i is added with 1 until the value of the i is N;
a second classification module, configured to perform step S4, classify the fruit to be classified according to the quality information of the nth dimension;
the quantity of the network models corresponds to the dimensionality of the quality information of the fruits to be classified, and different network models are used for predicting the quality information of different dimensionalities of the fruits to be classified;
the ith network model is a first network model, a second network model, a third network model or a fourth network model, the first network model is used for predicting the shape of the fruit to be classified, the second network model is used for predicting the size of the fruit to be classified, the third network model is used for predicting the defects of the fruit to be classified, and the fourth network model is used for predicting the taste of the fruit to be classified; wherein the mouthfeel of the fruit to be classified comprises the sweetness of the fruit to be classified, and the fourth network model is used for fitting the sweetness of the fruit to be classified according to the size and the weight of the fruit to be classified;
the ith network model comprises an input layer, an output layer and a plurality of hidden layers, each hidden layer in the first network model, the second network model and the third network model comprises a pooling layer and a plurality of convolution layers, each convolution layer comprises a plurality of nodes, the number of convolution kernels of each node is multiplied, and the sizes of the convolution kernels corresponding to each node are equal.
6. A picking apparatus comprising a fruit sorting device according to claim 5 and a plurality of conveying channels, the fruit sorting device being configured to predict N sorting results corresponding to N dimensions of quality information of fruit to be sorted and convey the fruit to be sorted to the plurality of conveying channels according to the N sorting results.
7. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the fruit sorting method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210809892.4A CN114882299B (en) | 2022-07-11 | 2022-07-11 | Fruit sorting method and device, picking equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210809892.4A CN114882299B (en) | 2022-07-11 | 2022-07-11 | Fruit sorting method and device, picking equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114882299A CN114882299A (en) | 2022-08-09 |
| CN114882299B true CN114882299B (en) | 2022-11-15 |
Family
ID=82683599
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210809892.4A Active CN114882299B (en) | 2022-07-11 | 2022-07-11 | Fruit sorting method and device, picking equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114882299B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110807495A (en) * | 2019-11-08 | 2020-02-18 | 腾讯科技（深圳）有限公司 | Multi-label classification method and device, electronic equipment and storage medium |
| US10664722B1 (en) * | 2016-10-05 | 2020-05-26 | Digimarc Corporation | Image processing arrangements |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11914674B2 (en) * | 2011-09-24 | 2024-02-27 | Z Advanced Computing, Inc. | System and method for extremely efficient image and pattern recognition and artificial intelligence platform |
| CN206430750U (en) * | 2016-09-29 | 2017-08-22 | 西北农林科技大学 | A kind of small-sized kiwifruit fruit detection grading plant |
| US10813295B2 (en) * | 2017-04-17 | 2020-10-27 | Iron Ox, Inc. | Method for monitoring growth of plants and generating a plant grow schedule |
| CN109359636B (en) * | 2018-12-14 | 2023-04-28 | 腾讯科技（深圳）有限公司 | Video classification method, device and server |
| CN111222747A (en) * | 2019-11-22 | 2020-06-02 | 南宁职业技术学院 | Picking method and device for fresh goods |
| CN111062439A (en) * | 2019-12-17 | 2020-04-24 | 腾讯科技（深圳）有限公司 | Video definition classification method, device, equipment and storage medium |
| CN111553439A (en) * | 2020-05-08 | 2020-08-18 | 视睿（杭州）信息科技有限公司 | Agricultural product quality classification method and device, computer equipment and storage medium |
| CN111340148B (en) * | 2020-05-22 | 2020-09-22 | 支付宝(杭州)信息技术有限公司 | Training method of business classification model, business classification method and terminal |
| CN111753794B (en) * | 2020-06-30 | 2024-02-27 | 创新奇智(成都)科技有限公司 | Fruit quality classification method, device, electronic equipment and readable storage medium |
| CN112966541B (en) * | 2020-09-23 | 2023-12-05 | 北京豆牛网络科技有限公司 | Fruit and vegetable automatic checking method, system, electronic equipment and computer readable medium |
-
2022
- 2022-07-11 CN CN202210809892.4A patent/CN114882299B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10664722B1 (en) * | 2016-10-05 | 2020-05-26 | Digimarc Corporation | Image processing arrangements |
| CN110807495A (en) * | 2019-11-08 | 2020-02-18 | 腾讯科技（深圳）有限公司 | Multi-label classification method and device, electronic equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| Multi-Model CNN-RNN-LSTM Based Fruit Recognition and Classification;Harmandeep Singh Gill等;《Intelligent Automation & Soft Computing》;20220105;第33卷(第1期);637-650 * |
| 基于图像处理的苹果外观品质分级研究;刘亚东;《中国优秀硕士学位论文全文数据库_农业科技辑》;20220415;D048-126 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114882299A (en) | 2022-08-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Raikar et al. | Classification and Grading of Okra-ladies finger using Deep Learning | |
| US20190228268A1 (en) | Method and system for cell image segmentation using multi-stage convolutional neural networks | |
| CN109308696B (en) | No-reference image quality evaluation method based on hierarchical feature fusion network | |
| CN111611924B (en) | Mushroom identification method based on deep migration learning model | |
| US20180181867A1 (en) | Artificial neural network class-based pruning | |
| CN111709371B (en) | Classification method, device, server and storage medium based on artificial intelligence | |
| CN113066065B (en) | No-reference image quality detection method, system, terminal and medium | |
| CN110264407B (en) | Image super-resolution model training and reconstruction method, device, equipment and storage medium | |
| CN112381763A (en) | Surface defect detection method | |
| CN113569881A (en) | Self-adaptive semantic segmentation method based on chain residual error and attention mechanism | |
| CN110348503A (en) | A kind of apple quality detection method based on convolutional neural networks | |
| Jassmann et al. | Leaf classification utilizing a convolutional neural network | |
| CN110751072A (en) | Double-person interactive identification method based on knowledge embedded graph convolution network | |
| CN114022878A (en) | Improved YOLOv 5-based string-type tomato real-time detection method | |
| CN112200241A (en) | Automatic sorting method for fish varieties based on ResNet transfer learning | |
| CN111898614B (en) | Neural network system and image signal and data processing method | |
| Correa et al. | Design and Implementation of a CNN architecture to classify images of banana leaves with diseases | |
| CN111652307A (en) | Intelligent nondestructive identification method and device for redwood furniture based on convolutional neural network | |
| CN107220707A (en) | Dynamic neural network model training method and device based on 2-D data | |
| CN108920623B (en) | Data mining method and device | |
| CN114882299B (en) | Fruit sorting method and device, picking equipment and storage medium | |
| CN110956201A (en) | Image distortion type classification method based on convolutional neural network | |
| Buczkowski et al. | Convolutional neural network-based image distortion classification | |
| CN117456230A (en) | Data classification method, system and electronic equipment | |
| CN112801220B (en) | Agricultural product quality classification method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |