CN115223043A - Strawberry defect detection method and device, computer equipment and storage medium - Google Patents
Strawberry defect detection method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN115223043A CN115223043A CN202210788872.3A CN202210788872A CN115223043A CN 115223043 A CN115223043 A CN 115223043A CN 202210788872 A CN202210788872 A CN 202210788872A CN 115223043 A CN115223043 A CN 115223043A
- Authority
- CN
- China
- Prior art keywords
- strawberry
- image
- detected
- defect detection
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/188—Vegetation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Image Analysis (AREA)
Abstract
The application belongs to the technical field of computer vision, and discloses a strawberry defect detection method, a strawberry defect detection device, a strawberry defect detection computer device and a strawberry defect storage medium, wherein the strawberry defect detection method comprises the following steps: acquiring an original image containing the strawberry to be detected; preprocessing an original image to obtain an image to be detected; inputting an image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model; and obtaining a defect detection result of the strawberry to be detected according to the surface defect detection information and the overall outline detection information of the strawberry to be detected. The method and the device can achieve the effect of accurately and efficiently detecting the defects of the strawberries in the industrial sorting scene.
Description
Technical Field
The application relates to the technical field of computer vision, in particular to a strawberry defect detection method and device, computer equipment and a storage medium.
Background
At present, the detection methods of strawberry defects mainly comprise two types: one is a manual screening method, namely strawberries with deformed shapes and rotten or contusion on the surfaces are screened by visual identification, the method not only consumes a great deal of manpower and time, but also leads to wrong screening and low efficiency because the prior knowledge and the energy of screening personnel are reduced; the other method is a machine vision method, and detection is performed through the combination of an artificial intelligence algorithm and a vision sensor, so that the detection efficiency can be improved. The main application scenes of the existing machine vision method are outdoor and greenhouse culture areas, detection is usually only carried out according to strawberry disease classification and positioning, and the machine vision method is used for monitoring the growth condition of strawberries. However, in an industrial sorting scene, the quality of strawberries on a production line needs to be detected frequently, and the prior art does not consider the quality problem of strawberries caused by insufficient preservation in the industrial sorting process, and neglects the shape of the strawberries and is part of the quality screening. Therefore, the prior art has the problem that the defects of the strawberries cannot be accurately and efficiently detected in an industrial sorting scene.
Disclosure of Invention
The application provides a strawberry defect detection method and device, computer equipment and a storage medium, which can accurately and efficiently detect strawberries in an industrial sorting scene.
In a first aspect, an embodiment of the present application provides a method for detecting strawberry defects, where the method includes:
acquiring an original image containing the strawberry to be detected;
preprocessing an original image to obtain an image to be detected;
inputting an image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model;
and obtaining a defect detection result of the strawberry to be detected according to the surface defect detection information and the overall outline detection information of the strawberry to be detected.
In one embodiment, inputting an image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of a strawberry to be detected, including:
inputting an image to be detected into a pre-trained target detection and semantic segmentation network model, so that the pre-trained target detection and semantic segmentation network model outputs overall contour detection information and a plurality of surface defect prediction results of the strawberry to be detected;
and screening an optimal surface defect prediction result from the plurality of surface defect prediction results through a non-maximum suppression algorithm to obtain the surface defect detection information of the strawberry to be detected.
In one embodiment, the preprocessing the original image to obtain an image to be detected includes:
scaling an original image into an image with a preset resolution according to a preset proportion;
and carrying out normalization processing on the image with the preset resolution ratio to obtain the image to be detected.
In one embodiment, the strawberry defect detection model further comprises a classification network model; obtaining a defect detection result of the strawberry to be detected according to the surface defect detection information and the overall outline detection information of the strawberry to be detected, comprising the following steps:
generating a defect detection image of the strawberry to be detected according to the original image, the surface defect detection information and the overall contour detection information of the strawberry to be detected;
inputting the overall contour detection information of the strawberry to be detected into a pre-trained classification network model to obtain a size defect detection result of the strawberry to be detected;
and obtaining a defect detection result of the strawberry to be detected according to the defect detection image and the size defect detection result of the strawberry to be detected.
In one embodiment, the surface defect detection information of the strawberry to be detected is a surface defect detection image, and the overall contour detection information of the strawberry to be detected is an overall contour detection image;
generating a defect detection image of the strawberry to be detected according to the original image, the surface defect detection information and the overall contour detection information of the strawberry to be detected, comprising the following steps:
converting the surface defect detection image into a surface defect image according to a preset proportion, and converting the overall contour detection image into an overall contour image according to the preset proportion, wherein the size of the surface defect image and the overall contour image is the same as that of the original image;
and splicing the surface defect image, the overall outline image and the original image to generate a defect detection image.
In one embodiment, the pre-trained strawberry defect detection model is deployed in an embedded device.
In one embodiment, before acquiring the original image containing the strawberry to be detected, the method further includes:
constructing a strawberry defect detection model, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model and a classification network model, and the target detection and semantic segmentation network model comprises a target detection network and a semantic segmentation network;
acquiring a plurality of sample images containing strawberries on an industrial sorting line, and performing defect marking and contour calibration on the strawberries in each sample image to obtain a training image set;
training the strawberry defect detection model based on the training image set to obtain a pre-trained strawberry defect detection model;
and deploying the pre-trained strawberry defect detection model in the embedded equipment.
In a second aspect, an embodiment of the present application provides a strawberry defect detecting device, which includes:
the image acquisition module is used for acquiring an original image containing the strawberry to be detected;
the image preprocessing module is used for preprocessing the original image to obtain an image to be detected;
the model detection module is used for inputting the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, and the strawberry defect detection model comprises a target detection and semantic segmentation network model;
and the result output module is used for obtaining the defect detection result of the strawberry to be detected according to the surface defect detection information and the overall contour detection information of the strawberry to be detected.
In a third aspect, an embodiment of the present application provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the strawberry defect detecting method according to any of the above embodiments.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the steps of the strawberry defect detecting method according to any one of the above embodiments.
In summary, compared with the prior art, the beneficial effects brought by the technical scheme provided by the embodiment of the application at least include:
according to the strawberry defect detection method provided by the embodiment of the application, an original image containing strawberries to be detected can be obtained; preprocessing an original image to obtain an image to be detected; inputting an image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model, and can realize target detection and semantic segmentation of the image to be detected, so that the surface defect of the strawberry is detected and the overall shape of the strawberry is segmented, and the surface defect detection information and the overall contour detection information of the strawberry are obtained; according to the surface defect detection information and the overall contour detection information of the strawberries to be detected, the defect detection result of the strawberries to be detected is obtained, the surface defects and the shape defects of the strawberries can be detected, and the method is suitable for detecting the defects of the strawberries under the industrial sorting scene. According to the method, the image containing the strawberries can be detected through the pre-trained strawberry defect detection model, so that the surface defects of the strawberries can be accurately detected, the overall contour of the strawberries can be detected to judge whether the strawberries have shape defects, and the defects of the strawberries in an industrial sorting scene can be accurately and efficiently detected.
Drawings
Fig. 1 is a flowchart of a strawberry defect detection method according to an exemplary embodiment of the present application.
Fig. 2 is an exemplary diagram of a strawberry defect detection process according to an exemplary embodiment of the present application.
FIG. 3 is a flowchart of model training steps provided in an exemplary embodiment of the present application.
Fig. 4 is a block diagram of a strawberry defect detection model according to an exemplary embodiment of the present application.
FIG. 5 is an exemplary diagram of data annotation provided in an exemplary embodiment of the present application.
Fig. 6 is an application scenario diagram of a strawberry defect detection method according to an exemplary embodiment of the present application.
Fig. 7 is a structural diagram of a strawberry defect detecting apparatus according to an exemplary embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
Referring to fig. 1, the present application provides a method for detecting strawberry defects, which uses a terminal device as an execution main body for application in an industrial sorting occasion. The method specifically comprises the following steps:
s1, acquiring an original image containing the strawberry to be detected.
Wherein, the strawberries to be detected can be strawberries transported on an industrial production line; the original image is a strawberry image acquired by a video acquisition device, and each original image can contain one or more strawberries; in particular, in order to detect the quality of the strawberries more accurately, the video capture device may be configured to take a picture of only one strawberry at a time, so that each original image includes only one strawberry; above-mentioned video acquisition device mountable is directly over the assembly line, when setting up installation angle, in order to acquire clear and accurate strawberry image, can establish video acquisition device's shooting angle as perpendicular with the assembly line, face the strawberry position promptly.
Specifically, in an industrial sorting scene, a video acquisition device acquires images of single strawberries transmitted on a production line, the acquired images are sent to a terminal device, and the terminal device takes the acquired images containing the strawberries as original images. The terminal equipment can be connected with the video acquisition device through a data line and can also be connected with the video acquisition device in a wireless communication mode.
And S2, preprocessing the original image to obtain an image to be detected.
The image to be detected is an image obtained by preprocessing an original image, and the size of the image to be detected is usually smaller than that of the original image, so that the data volume needing to be processed is favorably reduced; meanwhile, the image to be detected meets the input requirements of the strawberry defect detection model in the step S3.
Specifically, the original image is preprocessed and converted into the image to be detected which meets the input requirement of the strawberry defect detection model. In specific implementation, mature image preprocessing modes in the prior art such as zooming, cropping and graying can be adopted, for example: the original image is processed according to the preprocessing procedures of graying, geometric transformation and image enhancement, so that the data is simplified to the maximum extent, and the reliability of subsequent target detection and semantic segmentation is improved. In addition to the several preprocessing methods listed in this embodiment, other common image preprocessing methods can be adopted in this embodiment.
And S3, inputting the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model.
The target detection and semantic segmentation network model can be a model combining a target detection network and a semantic segmentation network, the target detection network can adopt an RCNN series network, an SSD series network or a YOLO series network, and the semantic segmentation network can adopt an FCN network, a SegNet network, a U-Net network or a DeepLab series network; the strawberry defect detection model is generally deployed in terminal equipment. Besides the above-listed target detection networks and semantic segmentation networks, other common target detection network models and semantic segmentation network models can be combined with each other to form the strawberry defect detection model, so as to realize the functions of target detection and semantic segmentation simultaneously. It is noted that those not described in detail in this specification are prior art known to those skilled in the art.
Specifically, the pre-trained target detection and semantic segmentation network model can perform target detection on the input image to be detected through the target detection network to obtain the surface defect detection information of the strawberry, and can perform semantic segmentation on the image to be detected through the semantic segmentation network to obtain the overall contour detection information of the strawberry. In specific implementation, the surface defect detection information may include a position and a defect type of a surface defect of the strawberry, and the overall contour detection information may include an overall shape contour of the strawberry, for example, different pixel values may be used to distinguish a foreground and a background of the strawberry, to form a binary grayscale image representing the contour of the strawberry, and the binary grayscale image is used as the overall contour detection information.
And S4, obtaining a defect detection result of the strawberry to be detected according to the surface defect detection information and the overall contour detection information of the strawberry to be detected.
The defect detection result can comprise a visual defect detection image obtained based on surface defect detection information and overall contour detection information of the strawberry to be detected, and can also comprise a size defect detection result; the size defect detection result can be obtained from the overall contour detection information, and all surface defects of the strawberries in the defect detection image are framed and the corresponding defect types are displayed. Specifically, the terminal device may obtain a defect detection result of the strawberry to be detected according to the surface defect detection information and the overall profile detection information of the strawberry to be detected. In some embodiments, the terminal device may visualize the defect detection result of the strawberry to be detected through the display device, and the visualized defect detection result may be a defect detection image or a defect detection image marked with a size defect detection result.
The strawberry defect detection method provided in the above embodiment can acquire an original image containing a strawberry to be detected; preprocessing an original image to obtain an image to be detected; inputting an image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model, and can realize target detection and semantic segmentation of the image to be detected, so that the surface defect of the strawberry is detected and the overall shape of the strawberry is segmented, and the surface defect detection information and the overall contour detection information of the strawberry are obtained; the method and the device have the advantages that the defect detection result of the strawberry to be detected is obtained according to the surface defect detection information and the overall outline detection information of the strawberry to be detected, the surface defect and the shape defect of the strawberry can be detected, and the method and the device are suitable for detecting the strawberry defect in an industrial sorting scene. According to the method, the pre-trained strawberry defect detection model is used for detecting the image containing the strawberries, so that the surface defects of the strawberries can be accurately detected, the overall contour of the strawberries can be detected to judge whether the strawberries have shape defects, and the defects of the strawberries in an industrial sorting scene can be accurately and efficiently detected.
In some embodiments, step S3 specifically includes the following steps:
and inputting the image to be detected into the pre-trained target detection and semantic segmentation network model, so that the pre-trained target detection and semantic segmentation network model outputs the overall contour detection information and a plurality of surface defect prediction results of the strawberry to be detected.
And screening out an optimal surface defect prediction result from the plurality of surface defect prediction results through a non-maximum suppression algorithm to obtain the surface defect detection information of the strawberry to be detected.
The surface defect prediction result may be a plurality of candidate frames corresponding to each defect of the strawberry in the image output by the target detection network. In the process of target detection, a large number of candidate frames are generated at the same target position, and the candidate frames may overlap with each other, so that the optimal candidate frame is found by adopting a non-maximum suppression algorithm, redundant candidate frames are eliminated, and the obtained image of the optimal candidate frame only retaining each surface defect is used as the surface defect detection information of the strawberry to be detected. The Non-Maximum Suppression algorithm is called NMS algorithm for short, and is called Non-Maximum Suppression in English, and is used for searching for a local Maximum and suppressing a Maximum.
The embodiment can eliminate redundant surface defect prediction results and output the optimal surface defect prediction result, thereby improving the accuracy of the surface defect detection information.
In the specific implementation process, the quality of the image to be detected can directly affect the detection effect of the model, so that before the image is analyzed (feature extraction, segmentation, and the like), the original image needs to be preprocessed to obtain an image meeting the input requirement of the strawberry defect detection model in step S3.
In some embodiments, step S2 specifically includes the following steps:
and scaling the original image into an image with a preset resolution according to a preset proportion.
And carrying out normalization processing on the image with the preset resolution ratio to obtain the image to be detected.
The preset resolution is an input image resolution meeting the requirements of the strawberry defect detection model, and is generally smaller than the resolution of the original image, for example, the preset resolution is 640 × 640.
The embodiment can process the original image into the image to be detected through scaling and normalization, the original image is firstly scaled into the image with lower resolution, the data volume to be processed is reduced, then the image with lower resolution is normalized to obtain the image to be detected, irrelevant information in the image can be eliminated, useful real information is recovered, the detectability of relevant information is enhanced, and therefore the image data is simplified, and the reliability of subsequent feature extraction, segmentation and the like is improved.
In some embodiments, the strawberry defect detection model further comprises a classification network model; step S4 specifically includes the following steps:
and generating a defect detection image of the strawberry to be detected according to the original image, the surface defect detection information and the overall contour detection information of the strawberry to be detected.
And inputting the overall contour detection information of the strawberries to be detected into a pre-trained classification network model to obtain the size defect detection result of the strawberries to be detected.
And obtaining a defect detection result of the strawberry to be detected according to the defect detection image and the size defect detection result of the strawberry to be detected.
The defect detection image can be an image obtained by integrating an original image, surface defect detection information and overall contour detection information, can be used for marking the defect position and defect type on the surface of the strawberry, can perform blurring to a certain extent on the image background outside the strawberry removing field, and can reduce the data volume during subsequent storage or display processing; the size defect detection result can have normal and abnormal conditions, and the abnormal condition of the strawberry size can comprise malformation, undersize or oversize.
The embodiment can classify the shapes of the strawberries according to the overall contour detection information of the strawberries by utilizing the classification network model, and can visualize the defect detection images, thereby facilitating the observation of the defects of the strawberries.
In some implementations of the above embodiments, the surface defect detection information of the strawberry to be detected is a surface defect detection image, and the overall contour detection information of the strawberry to be detected is an overall contour detection image. In order to generate a defect detection image convenient to observe, the step of generating the defect detection image of the strawberry to be detected according to the original image, the surface defect detection information and the overall contour detection information of the strawberry to be detected specifically comprises the following steps:
and converting the surface defect detection image into a surface defect image according to a preset proportion, and converting the overall contour detection image into an overall contour image according to the preset proportion, wherein the size of the surface defect image, the overall contour image and the size of the original image are the same.
And splicing the surface defect image, the overall outline image and the original image to generate a defect detection image.
The surface defect detection image is a target detection result output by the pre-trained strawberry defect detection model, and the overall contour detection image is a semantic segmentation result output by the pre-trained strawberry defect detection model.
Specifically, referring to an example of a strawberry defect detection process in fig. 2, after the image preprocessing in step S2, the original image is input to a pre-trained model, the model outputs a target detection result and a semantic segmentation result, and after the target detection result and the semantic segmentation result are respectively processed, the generated defect detection image is visualized.
In some embodiments, in order to make the pre-trained strawberry defect detection model applicable to an actual industrial scenario, the pre-trained strawberry defect detection model is deployed in an embedded device.
In some embodiments, before step S1, a pre-trained strawberry defect detection model needs to be obtained, please refer to fig. 3, and the method further includes the following steps:
and S5, constructing a strawberry defect detection model, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model and a classification network model, and the target detection and semantic segmentation network model comprises a target detection network and a semantic segmentation network.
In the above embodiment, the strawberry defect detection model includes two components, namely a target detection and semantic segmentation network model and a classification network model. Most machine learning algorithms have low operation efficiency and slow processing speed, so that real-time detection on an industrial production line cannot be met. Therefore, in a specific implementation process, the target detection and semantic segmentation network model can adopt an improved YOLOv5 network model. The YOLOv5 network belongs to a single-stage detection algorithm, has a higher detection speed and lighter weight compared with a two-stage detection algorithm of Faster-RCNN, and is suitable for deployment of various scenes. In this embodiment, the problem of positioning and analyzing the surface rot of the strawberry needs to be solved, and the YOLOv5 network can only position the rot position of the strawberry, so the YOLOv5 network needs to be improved, a semantic segmentation network is added to extract foreground information of the strawberry by performing semantic segmentation, and finally the foreground information of the strawberry is input into a classification network model to classify the strawberry shape (such as malformation, over-size, or under-size) to obtain the size defect detection result of the strawberry. Wherein, the foreground information of the strawberry is the overall contour detection information of the strawberry; the original Yolov5 network model comprises a feature extraction network, a feature fusion network and a detection network, wherein the feature fusion network can input a feature graph with larger fused features into a semantic segmentation network.
Referring to fig. 4, the yolovv 5 network includes three major parts, the first part is a feature extraction network backbone, and is used to extract feature information of an image to obtain a corresponding feature map F; the second part is a feature fusion network head which is used for fusing feature information of different layers in the feature map F to obtain a feature map F' with richer feature information so as to improve the performance of the model; the third part is a detection network detect, a final feature map F' is converted into feature vectors through three branch structures for model training, the output vector comprises three parts, namely output1, output2 and output3, the output sizes of the three parts are respectively 1/8, 1/16 and 1/32 of the input sizes, the receptive field of the output1 is small, small objects can be mainly recognized, the receptive field of the output3 is maximum, larger objects can be recognized, and the output2 is between the two parts. The Mask network in fig. 4 is an improved part of the YOLOv5 network, which refers to a segmentation head module of the Deeplabv3 network, and adopts a feature map with a large feature map after the head part is fused as an input, the number of channels output by two layers of convolution layers is adjusted to be the number of categories (the number of categories of the invention is 2, and the categories are strawberry and background respectively), and the size of an input image is restored by an 8 times upper sampling layer of bilinear interpolation, so as to obtain a semantic segmentation result, and the specific process is as shown in formulas (1) and (2); and taking the segmentation result as the input of a classification network, and judging the shape of the strawberry by the classification network according to the foreground information. In the training process of the Mask network, cross entropy loss is calculated from pixel points to original images, and is added with loss obtained through detection in a weighting mode to carry out back propagation.
x l ＝Conv2d(W l ,x l-1 )+b l (1)
o l ＝Upsample(x l ,s) (2)
Wherein x is l Two-layer convolution operation, x, representing Mask network l-1 ∈R C×H×W Is the output of the C3 module, x l ∈R 2×H×W For the convolved output, W l 、b l E.r denotes the convolution parameter. o. o l Denotes the upsampling operation of the Mask network, s denotes the scaling factor, o l ∈R 2×sH×sW Representing the output mask.
And S6, acquiring a plurality of sample images containing strawberries on the industrial sorting line, and performing defect marking and contour calibration on the strawberries in each sample image to obtain a training image set.
Referring to an example of data labeling in fig. 5, the defect detection is labeled as frame coordinates and defect types of strawberry surface defects in a sample image; the contour calibration is to calibrate the contour of the strawberry in the sample image.
Specifically, in the data labeling process of the sample image, the detection data and the segmentation data need to be labeled. In specific implementation, for the detection data, as the main source of the defect is rotten caused by insufficient preservation in the sorting process, the defect can be positioned only by the embodiment. Generally, in the strawberry detection process, the situations such as contusion, plant diseases and insect pests are less, and the situations can not be considered, but can be adjusted according to actual situations; for semantic segmentation data, the contour of the strawberry can be calibrated to form a binary gray image, the foreground and the background of the strawberry are distinguished through different pixel values, the same group of sample images as the defect positioning are adopted, and parallel training and feature extraction of a model are facilitated.
And S7, training the strawberry defect detection model based on the training image set to obtain a pre-trained strawberry defect detection model.
And S8, deploying the pre-trained strawberry defect detection model in the embedded equipment.
The embodiment can obtain the pre-trained strawberry defect detection model, the pre-trained strawberry defect detection model can not only position the surface rot defect of the strawberry, but also analyze the shape of the strawberry, and can increase the functionality of the model and improve the detection efficiency.
At present, the application of a detection model is generally based on the reasoning operation of a CPU and a GPU on a computer or a server, and because the computer and the server have large volumes, the detection model is inconvenient and high-cost as edge computing equipment in an actual scene, and is difficult to be suitable for the actual scene. In some embodiments, in order to make the improved YOLOv5 model suitable for practical industrial scenarios, step S104 specifically includes the following steps:
and deploying the pre-trained strawberry defect detection model in the embedded device through an OpenVINO tool library.
Specifically, the pre-trained strawberry defect detection model can be converted into an ONNX expression format; converting the model of the ONNX expression format into a TensorRT format through an OpenVINO tool library; and deploying the obtained model in the TensrT format in the embedded equipment. As shown in fig. 6, the OpenVINO tool library can be used to transplant a pre-trained strawberry defect detection model (improved YOLOv5 model) into an embedded device, and implement the pre-trained strawberry defect detection model into a specific application scenario.
The strawberry defect detection model is built on the basis of a pyrrch framework; the pyrtch framework is a deep learning framework; the ONNX expression format is an Open Neural Network Exchange (ONNX) format, is a standard for expressing a deep learning model, and can transfer the model among different frameworks; the TensorRT format is a common model format; the OpenVINO tool library is a tool kit which is developed by Intel based on the existing hardware platform of the Intel, can accelerate the application development speed of high-performance computer vision and deep learning vision, supports deep learning on hardware accelerators of various Intel platforms, and allows direct heterogeneous execution.
According to the embodiment, the model can be deployed on the embedded device through the OpenVINO tool library, so that the detection speed of the deployed model can be increased, and the effect of real-time detection is achieved.
Referring to fig. 7, another embodiment of the present application provides a strawberry defect detecting apparatus, which includes:
the image obtaining module 101 is configured to obtain an original image including a strawberry to be detected.
The image preprocessing module 102 is configured to preprocess the original image to obtain an image to be detected.
The model detection module 103 is configured to input the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, where the strawberry defect detection model includes a target detection and semantic segmentation network model.
And the result output module 104 is used for obtaining the defect detection result of the strawberry to be detected according to the surface defect detection information and the overall contour detection information of the strawberry to be detected.
In some embodiments, the model detection module 103 is specifically configured to: inputting the image to be detected into a pre-trained target detection and semantic segmentation network model, so that the pre-trained target detection and semantic segmentation network model outputs the overall contour detection information and a plurality of surface defect prediction results of the strawberry to be detected; and screening out an optimal surface defect prediction result from the plurality of surface defect prediction results through a non-maximum suppression algorithm to obtain the surface defect detection information of the strawberry to be detected.
In some embodiments, the image pre-processing module 102 is specifically configured to: scaling an original image into an image with a preset resolution according to a preset proportion; and carrying out normalization processing on the image with the preset resolution ratio to obtain the image to be detected.
In some embodiments, the strawberry defect detection model further comprises a classification network model; the result output module 104 is specifically configured to:
generating a defect detection image of the strawberry to be detected according to the original image, the surface defect detection information and the overall contour detection information of the strawberry to be detected; inputting the overall contour detection information of the strawberry to be detected into a pre-trained classification network model to obtain a size defect detection result of the strawberry to be detected; and obtaining a defect detection result of the strawberry to be detected according to the defect detection image and the size defect detection result of the strawberry to be detected.
In some embodiments, the surface defect detection information of the strawberries to be detected is a surface defect detection image, and the overall contour detection information of the strawberries to be detected is an overall contour detection image; the result output module 104 is further specifically configured to: converting the surface defect detection image into a surface defect image according to a preset proportion, and converting the overall contour detection image into an overall contour image according to the preset proportion, wherein the size of the surface defect image and the overall contour image is the same as that of the original image; and splicing the surface defect image, the overall outline image and the original image to generate a defect detection image.
In some embodiments, the pre-trained strawberry defect detection model is deployed in an embedded device.
In some embodiments, the apparatus further comprises a model training module, the model training module being specifically configured to:
constructing a strawberry defect detection model, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model and a classification network model, and the target detection and semantic segmentation network model comprises a target detection network and a semantic segmentation network; acquiring a plurality of sample images containing strawberries on an industrial sorting line, and performing defect marking and contour calibration on the strawberries in each sample image to obtain a training image set; training the strawberry defect detection model based on the training image set to obtain a pre-trained strawberry defect detection model; and deploying the pre-trained strawberry defect detection model in the embedded equipment.
For specific limitations of the strawberry defect detecting apparatus provided in this embodiment, reference may be made to the above embodiments of the strawberry defect detecting method, which is not described herein again. All or part of the modules in the strawberry defect detecting device can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
Embodiments of the present application provide a computer device that may include a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. When executed by a processor, the computer program causes the processor to perform the steps of the strawberry defect detecting method according to any of the embodiments described above.
For the working process, the working details, and the technical effects of the computer device provided in this embodiment, reference may be made to the above embodiment of the method for detecting strawberry defects, and details are not described herein again.
The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of the strawberry defect detection method according to any one of the embodiments. The computer-readable storage medium refers to a carrier for storing data, and may include, but is not limited to, a floppy disk, an optical disk, a hard disk, a flash Memory, a flash disk and/or a Memory Stick (Memory Stick), etc., and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.
For the working process, the working details, and the technical effects of the computer-readable storage medium provided in this embodiment, reference may be made to the above embodiments of the method for detecting strawberry defects, which are not described herein again.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, and these are all within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A strawberry defect detection method is characterized by comprising the following steps:
acquiring an original image containing the strawberry to be detected;
preprocessing the original image to obtain an image to be detected;
inputting the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model;
and obtaining a defect detection result of the strawberry to be detected according to the surface defect detection information and the overall outline detection information of the strawberry to be detected.
2. The method according to claim 1, wherein the inputting the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected comprises:
inputting the image to be detected into the pre-trained target detection and semantic segmentation network model, so that the pre-trained target detection and semantic segmentation network model outputs the overall contour detection information and a plurality of surface defect prediction results of the strawberry to be detected;
and screening out an optimal surface defect prediction result from the plurality of surface defect prediction results through a non-maximum suppression algorithm to obtain the surface defect detection information of the strawberry to be detected.
3. The method according to claim 1, wherein the preprocessing the original image to obtain an image to be detected comprises:
scaling the original image into an image with a preset resolution according to a preset proportion;
and carrying out normalization processing on the image with the preset resolution ratio to obtain an image to be detected.
4. The method of claim 3, wherein the strawberry defect detection model further comprises a classification network model; the obtaining of the defect detection result of the strawberry to be detected according to the surface defect detection information and the overall contour detection information of the strawberry to be detected comprises the following steps:
generating a defect detection image of the strawberry to be detected according to the original image, the surface defect detection information and the overall contour detection information of the strawberry to be detected;
inputting the overall contour detection information of the strawberries to be detected into the pre-trained classification network model to obtain the size defect detection result of the strawberries to be detected;
and obtaining a defect detection result of the strawberry to be detected according to the defect detection image and the size defect detection result of the strawberry to be detected.
5. The method according to claim 4, wherein the surface defect detection information of the strawberries to be detected is a surface defect detection image, and the overall contour detection information of the strawberries to be detected is an overall contour detection image;
generating a defect detection image of the strawberry to be detected according to the original image, the surface defect detection information and the overall contour detection information of the strawberry to be detected, including:
converting the surface defect detection image into a surface defect image according to the preset proportion, and converting the overall contour detection image into an overall contour image according to the preset proportion, wherein the size of the surface defect image, the size of the overall contour image and the size of the original image are the same;
and splicing the surface defect image, the overall outline image and the original image to generate a defect detection image.
6. The method of any of claims 1 to 5, wherein the pre-trained strawberry defect detection model is deployed in an embedded device.
7. The method according to any one of claims 1 to 5, wherein before said acquiring an original image containing a strawberry to be detected, the method further comprises:
constructing a strawberry defect detection model, wherein the strawberry defect detection model comprises a target detection and semantic segmentation network model and a classification network model, and the target detection and semantic segmentation network model comprises a target detection network and a semantic segmentation network;
acquiring a plurality of sample images containing strawberries on an industrial sorting line, and carrying out defect marking and contour calibration on the strawberries in each sample image to obtain a training image set;
training the strawberry defect detection model based on the training image set to obtain a pre-trained strawberry defect detection model;
and deploying the pre-trained strawberry defect detection model in embedded equipment.
8. A strawberry defect detecting device, characterized in that the device includes:
the image acquisition module is used for acquiring an original image containing the strawberry to be detected;
the image preprocessing module is used for preprocessing the original image to obtain an image to be detected;
the model detection module is used for inputting the image to be detected into a pre-trained strawberry defect detection model to obtain surface defect detection information and overall contour detection information of the strawberry to be detected, and the strawberry defect detection model comprises a target detection and semantic segmentation network model;
and the result output module is used for obtaining the defect detection result of the strawberry to be detected according to the surface defect detection information and the overall outline detection information of the strawberry to be detected.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method according to any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210788872.3A CN115223043A (en) | 2022-07-06 | 2022-07-06 | Strawberry defect detection method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210788872.3A CN115223043A (en) | 2022-07-06 | 2022-07-06 | Strawberry defect detection method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115223043A true CN115223043A (en) | 2022-10-21 |
Family
ID=83609917
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210788872.3A Pending CN115223043A (en) | 2022-07-06 | 2022-07-06 | Strawberry defect detection method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115223043A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116577473A (en) * | 2023-07-14 | 2023-08-11 | 北京市农林科学院 | Detection method and device for strawberry mechanical damage occurrence time |
| CN116758045A (en) * | 2023-07-05 | 2023-09-15 | 日照鲁光电子科技有限公司 | Surface defect detection method and system for semiconductor light-emitting diode |
-
2022
- 2022-07-06 CN CN202210788872.3A patent/CN115223043A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116758045A (en) * | 2023-07-05 | 2023-09-15 | 日照鲁光电子科技有限公司 | Surface defect detection method and system for semiconductor light-emitting diode |
| CN116758045B (en) * | 2023-07-05 | 2024-01-23 | 日照鲁光电子科技有限公司 | Surface defect detection method and system for semiconductor light-emitting diode |
| CN116577473A (en) * | 2023-07-14 | 2023-08-11 | 北京市农林科学院 | Detection method and device for strawberry mechanical damage occurrence time |
| CN116577473B (en) * | 2023-07-14 | 2023-11-17 | 北京市农林科学院 | Detection method and device for strawberry mechanical damage occurrence time |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110059558B (en) | Orchard obstacle real-time detection method based on improved SSD network | |
| CN112967243B (en) | Deep learning chip packaging crack defect detection method based on YOLO | |
| CN110059694B (en) | Intelligent identification method for character data in complex scene of power industry | |
| CN111680746B (en) | Vehicle damage detection model training, vehicle damage detection method, device, equipment and medium | |
| CN111814741B (en) | Method for detecting embryo-sheltered pronucleus and blastomere based on attention mechanism | |
| CN115223043A (en) | Strawberry defect detection method and device, computer equipment and storage medium | |
| CN110516541B (en) | Text positioning method and device, computer readable storage medium and computer equipment | |
| CN111368758B (en) | Face ambiguity detection method, face ambiguity detection device, computer equipment and storage medium | |
| CN113487610B (en) | Herpes image recognition method and device, computer equipment and storage medium | |
| CN113077419A (en) | Information processing method and device for hip joint CT image recognition | |
| CN113537244A (en) | Livestock image target detection method and device based on light-weight YOLOv4 | |
| CN112487913A (en) | Labeling method and device based on neural network and electronic equipment | |
| CN113706481A (en) | Sperm quality detection method, sperm quality detection device, computer equipment and storage medium | |
| CN112419202A (en) | Wild animal image automatic identification system based on big data and deep learning | |
| CN114782410A (en) | Insulator defect detection method and system based on lightweight model | |
| CN112418033B (en) | Landslide slope surface segmentation recognition method based on mask rcnn neural network | |
| CN114331961A (en) | Method for defect detection of an object | |
| CN111915565B (en) | Method for analyzing cracks of porcelain insulator of power transmission and transformation line in real time based on YOLACT algorithm | |
| CN115830302B (en) | Multi-scale feature extraction fusion power distribution network equipment positioning identification method | |
| CN114155246B (en) | Deformable convolution-based power transmission tower pin defect detection method | |
| CN113901944B (en) | Marine organism target detection method based on improved YOLO algorithm | |
| CN115620083A (en) | Model training method, face image quality evaluation method, device and medium | |
| CN115761667A (en) | Unmanned vehicle carried camera target detection method based on improved FCOS algorithm | |
| CN115359091A (en) | Armor plate detection tracking method for mobile robot | |
| CN117274843B (en) | Unmanned aerial vehicle front end defect identification method and system based on lightweight edge calculation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |