CN116934674A - Large-granularity silicon material detection and removal system based on YOLOv8 network - Google Patents
Large-granularity silicon material detection and removal system based on YOLOv8 network Download PDFInfo
- Publication number
- CN116934674A CN116934674A CN202310432318.6A CN202310432318A CN116934674A CN 116934674 A CN116934674 A CN 116934674A CN 202310432318 A CN202310432318 A CN 202310432318A CN 116934674 A CN116934674 A CN 116934674A
- Authority
- CN
- China
- Prior art keywords
- silicon material
- module
- threshold value
- early warning
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 239000002210 silicon-based material Substances 0.000 title claims abstract description 126
- 238000001514 detection method Methods 0.000 title claims abstract description 41
- 230000002159 abnormal effect Effects 0.000 claims abstract description 39
- 238000012549 training Methods 0.000 claims abstract description 37
- 238000013135 deep learning Methods 0.000 claims abstract description 14
- 238000004140 cleaning Methods 0.000 claims abstract description 11
- 238000012795 verification Methods 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 39
- 238000012360 testing method Methods 0.000 claims description 20
- 238000000034 method Methods 0.000 claims description 14
- 238000004519 manufacturing process Methods 0.000 claims description 10
- 230000004913 activation Effects 0.000 claims description 9
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 229910052710 silicon Inorganic materials 0.000 claims description 6
- 239000010703 silicon Substances 0.000 claims description 6
- 230000001360 synchronised effect Effects 0.000 claims description 6
- 230000005856 abnormality Effects 0.000 claims description 5
- 238000007781 pre-processing Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 230000006978 adaptation Effects 0.000 claims description 3
- 238000007792 addition Methods 0.000 claims description 3
- 239000000443 aerosol Substances 0.000 claims description 3
- 239000000470 constituent Substances 0.000 claims description 3
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000009472 formulation Methods 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 claims description 3
- 239000000203 mixture Substances 0.000 claims description 3
- 238000012986 modification Methods 0.000 claims description 3
- 230000004048 modification Effects 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims 1
- 238000005065 mining Methods 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 8
- 201000006292 polyarteritis nodosa Diseases 0.000 description 3
- 208000028571 Occupational disease Diseases 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B07—SEPARATING SOLIDS FROM SOLIDS; SORTING
- B07C—POSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING
- B07C5/00—Sorting according to a characteristic or feature of the articles or material being sorted, e.g. by control effected by devices which detect or measure such characteristic or feature; Sorting by manually actuated devices, e.g. switches
- B07C5/04—Sorting according to size
- B07C5/10—Sorting according to size measured by light-responsive means
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B07—SEPARATING SOLIDS FROM SOLIDS; SORTING
- B07C—POSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING
- B07C5/00—Sorting according to a characteristic or feature of the articles or material being sorted, e.g. by control effected by devices which detect or measure such characteristic or feature; Sorting by manually actuated devices, e.g. switches
- B07C5/36—Sorting apparatus characterised by the means used for distribution
- B07C5/363—Sorting apparatus characterised by the means used for distribution by means of air
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30148—Semiconductor; IC; Wafer
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biomedical Technology (AREA)
- General Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Geometry (AREA)
- Quality & Reliability (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The application provides a deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network, which is suitable for the field of silicon material subdivision sorting. The system acquires the silicon material image on the conveyor belt through the high-frame-rate camera, and uses labelme for manual marking to remove abnormal silicon material pictures. The image samples are divided into a training set and a verification set according to the ratio of 9:1, and the data enhancement is carried out on the pictures so as to improve the generalization capability. The yolo_si algorithm is created with YOLO v8 as backbone network and a training dataset is used to get the network model. All silicon materials in the image are detected by using the algorithm, size early warning and tracking marking are carried out according to a preset threshold value, and a cleaning instruction is sent to a lower computer when a cleaning position is reached. Through installing the air jet at the preset position, the lower computer sends a command to the air jet device, and the marked and tracked large-granularity silicon materials are removed orderly. The application can efficiently detect and remove the silicon material, has high automation degree, effectively reduces the manual dependence and meets the mining sorting requirement.
Description
Technical Field
The application relates to an image detection method, in particular to a deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network.
Background
Silicon is an important material in the semiconductor industry, which is built on top of silicon materials. Silicon occupies an important position in the development process of China, and the yield and the consumption of the silicon also mark the electronic industry level of one country. In order to ensure the purity of the silicon material, the silicon material needs to be prevented from contacting as much as possible in the production process. For silicon materials with larger granularity, the machine body is easily damaged in the production process, and a large amount of economic loss is finally caused. The automatic detection and removal of the large-granularity silicon material can improve the production efficiency, reduce the number of workers, reduce the loss of lives and property under occupational diseases and dangerous working environments, and has great significance for ensuring the safe and orderly operation and upgrading of the silicon material production line.
Disclosure of Invention
1. A deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network is used for detecting the silicon material size, sending out early warning when the size exceeds a threshold value, and performing intelligent removal, and comprises the following specific steps:
s1, a data acquisition stage: acquiring a silicon material image on a conveyor belt through a high-frame-rate camera, and intercepting and retaining a video key frame as a picture;
s2, a data preprocessing stage: manually marking by using labelme, removing abnormal silicon material pictures, dividing an image sample into a training set and a testing set according to a ratio of 9:1, and carrying out data enhancement on the pictures to improve generalization capability; multiple groups of training sets can be created according to requirements, and multiple initial models are generated through training;
s3, building and training a network: creating a yolo_si algorithm by taking YOLOv8 as a backbone network, and obtaining a network model by using a training data set;
s4, testing: selecting the best model by using the test set, recording and using the test set parameters to train out a new model in the whole data set as a final model yolo_si;
s5, detection early warning stage: in field work, detecting all silicon materials in an image by using a YOLO_SI model, calibrating by a camera, using the focal length of a camera and the distance between the camera and a belt, and using the obtained prediction frame for positioning the silicon material position and calculating the real size of the silicon material; pre-alarming when the size exceeds a threshold value according to a preset threshold value of the system;
s6, tracking: setting a tracking mark on the silicon material exceeding the threshold value, updating and tracking the position of the silicon material exceeding the threshold value in real time by a tracking module, and sending a cleaning instruction to a lower computer when the position reaches a cleaning position;
s7, eliminating: and installing an air jet at a preset position, and sending a command to an air jet device by a lower computer to orderly and concurrently reject the marked and tracked large-granularity silicon material without conflict.
Further, a deep learning large-granularity silicon material detecting and removing system based on a YOLOv8 network specifically comprises:
(1) Acquiring a silicon material picture video data stream through a high frame rate camera arranged above a conveyor belt;
(2) And intercepting key frames from the acquired silicon material video data stream as silicon material picture data.
Further, the data preprocessing stage specifically includes:
(i1) Marking the position and the size of the silicon material in the acquired picture data by using a marking tool Lableme, wherein a rectangular mark is adopted in the marking process;
(i2) Removing the detected non-compliant silicon material picture data;
(i3) Dividing the image sample into a training set and a testing set according to the ratio of 9:1;
(i4) Data enhancement of training pictures, including: scaling the silicon material picture; carrying out random horizontal and vertical inversion on the silicon material pictures at different angles; adding random noise; converting the image from RGB to HSV color space, adjusting brightness, and performing normalization operation; adjusting the pixel values of the image to be uniformly distributed through histogram equalization; processing original image noise;
(i5) Multiple groups of training sets can be created according to requirements, and multiple initial models are generated through training;
further, in the stage of building and training the network, the method specifically includes:
(A) The YOLOv8 network is used as a base line network, the further light weight is realized by adopting a C2f module, the ideas of an SPPF module and a PAN are reserved, and a coupled-Head, anchor-Free and Task-Aligned Assigner matching mode is adopted;
(B) Using the VFL loss function as a classification loss function; using CIOULoss+DFL as a regression Loss function; using hard-Swish as an activation function;
(C) The VFL as a class loss function is formulated as follows:
q is label, ioU for positive samples q is bbox and gt, q=0 for negative samples, FL is not used for positive samples, BCE is used, and only a large number of adaptations IoU are weighted for highlighting the main samples. The standard FL is negative.
(D) The formulation of CIOU as part of the regression loss function is as follows:
(E) The formula of the DFL as a constituent part of the regression loss function is as follows:
DFL(S i ,S i+1 )＝-((y i+1 -y)log(S i )+(y-y i )log(S i+1 ))
(F) The hard-Swish formula as an activation function is as follows:
wherein ReLU (x) =max (0, x)
(G) The YOLO v8 model containing the VFL Loss function and the cious+dfl Loss function and the activation function hard-Swish is called yolo_si;
(H) Training based on the yolo_si network model;
further, in the testing stage, the method specifically includes:
(T1) training to generate a plurality of initial models, and selecting the best model from the test set, wherein the best model is free of error detection;
(T2) recording each item setting of the best model, including iteration times, learning rate, regularization mode, coefficients and the like;
(T3) retraining a new model using the entire dataset as the final model yolo_si;
further, in the detection early warning stage, the method specifically includes:
converting the rough scale of the silicon material by utilizing the angle and the distance of the camera and the edge information of the detection result, and setting an alarm threshold value and an error threshold value; setting the system threshold value as the sum of an alarm threshold value and an error threshold value, and comparing the real size of the converted silicon material with the system threshold value; if the real size is larger than the threshold value, abnormal information is generated and sent to the real-time early warning module;
further, the real-time early warning module specifically includes:
(M1) a user and equipment management module which is responsible for registration, login and management of the user, personal information and provides super user management authority; providing an industrial personal computer equipment management mechanism;
(M2) a silicon material video stream acquisition and display module, wherein a video data stream of a silicon material picture is acquired through a high frame rate camera arranged above a conveyor belt, and the acquired image is sent to a real-time early warning module for real-time display; (M3) judging an abnormality module, adopting TensorRT deployment to provide low delay and high throughput, receiving a silicon material image sent by a silicon material video stream acquisition module, transmitting the silicon material image into a trained YOLO_SI model to obtain the position and the size of the silicon material, and comparing the calculated real size of the silicon material with a system threshold value; if the real size is larger than the threshold value, generating and pushing abnormal information to the page, and writing the abnormal information into a database;
(M4) a setting module for setting system parameters and log parameters, such as the early warning of the size of a silicon block;
and (M5) a log module for receiving the abnormal record of the real-time system and displaying the abnormal record, wherein the system stores the detection result and the running state of the system in a local log and synchronizes to a cloud server.
The user and equipment management module of the early warning system specifically comprises:
(U1) providing registration, login and user name verification functions for management personnel, endowing corresponding rights, operating the whole system and storing information to a cloud server;
(U2) providing supervisors with operating system high-level rights to operate the entire system while other users may be managed, including obtaining and updating user information, logging out users, etc.;
(U3) the management of the industrial personal computer equipment, including the addition, deletion, modification and queue display of the industrial personal computer equipment, and starting and stopping the operation of the industrial personal computer;
further, the silicon material video stream acquisition and display module of the early warning system specifically comprises:
(V1) capturing belt images primarily by means of a high frame rate camera mounted above the conveyor belt;
(V2) transmitting the image to the detection and identification system through the message queue;
(V3) providing a visual management interface, and displaying a conveyor belt picture of the industrial personal computer equipment in the monitoring list in real time;
the abnormal judgment module of the early warning system specifically comprises:
(R1) TensorRT deployment is employed to provide low latency and high throughput. And receiving the silicon material image sent by the silicon material video stream acquisition module, and transmitting the silicon material image into a trained yolo_si model for judgment.
(R2) when abnormal information of the corresponding industrial personal computer system exceeding a threshold value is detected, pushing the abnormal information to a monitoring page, and writing abnormal time and pictures into a database and a log module, wherein a cloud server is synchronous; the setting module of the early warning system specifically comprises:
setting camera angle, distance, focal length, alarm threshold and error threshold parameters, server port and the like;
the log module of the early warning system specifically comprises:
(Z1) writing of abnormal records, and selecting whether the abnormal records are synchronized to a cloud end or not;
(Z2) providing an early warning record searching function, and searching early warning records according to information such as industrial personal computer equipment, time, batch, threshold value, operators on duty and the like;
and (Z3) a remote service running on the cloud server is responsible for collecting and managing all detection log data, so that an administrator can conveniently monitor and manage production information remotely.
Further, in the tracking stage, the method specifically includes:
(F1) The method comprises the steps that a deep SORT tracking algorithm is adopted to realize multi-target tracking, when a real-time early warning module detects abnormal information of a corresponding industrial personal computer system exceeding a threshold value, a target positioning detection module positions a super-threshold silicon material in a picture and distributes an ID, and the ID is kept until the life cycle of the silicon material in the frame;
(F2) The motion predictor is responsible for simply predicting future motion by using the past motion information of the silicon material and assisting in overcoming the interference of the aerosol environment and shielding;
(F3) When each ID reaches the end clearing device of the conveyor belt, immediately sending a clearing signal to a lower computer of the rejection module, and entering a clearing stage, wherein in order to ensure orderly and concurrent clearing of clearing information, a message queue and multithreading are used for sending control commands in the stage;
further, in the rejecting stage, the method specifically includes:
(C1) The cleaning device is positioned at the tail end of the conveyor belt, a plurality of groups of air ejector tubes which are arranged tightly are preset, the air inlet end of the pulse valve is connected with the compressed air assembly, the air outlet end of the pulse valve is connected with the air ejector tubes, an electromagnetic control valve is arranged on the pulse valve, and the electromagnetic control valve is connected with the lower computer;
(C2) The lower computer receives the clearing signal, and starts the clearing device to realize ordered concurrency, no conflict and no contact rejection.
Drawings
FIG. 1 is a schematic diagram of a deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network;
FIG. 2 is a flowchart showing the step S1 of FIG. 1 in one embodiment;
FIG. 3 is a flowchart showing the step S2 of FIG. 1 in one embodiment;
FIG. 4 is a schematic diagram of the structure of the Yolov8 skeleton network Yolov8 of the present application;
FIG. 5 is a flowchart showing the step S5 of FIG. 1 in one embodiment;
FIG. 6 is a block diagram showing the user and device management module M1 of FIG. 5 in one embodiment;
FIG. 7 is a block diagram showing the video stream acquisition and display module M2 of FIG. 5 according to one embodiment;
FIG. 8 is a block diagram illustrating the abnormality determination module M3 of FIG. 5 according to an embodiment;
FIG. 9 is a block diagram showing the setup module M4 of FIG. 5 in one embodiment;
FIG. 10 is a block diagram showing the log module M5 of FIG. 5 in one embodiment;
FIG. 11 is a flowchart showing step S6 of FIG. 1 in one embodiment;
FIG. 12 is a flowchart showing step S7 of FIG. 1 in one embodiment;
fig. 13 shows a live simulation illustration in operation.
Detailed Description
In order that the above-recited objects, features and advantages of the present application will become more readily apparent, a more particular description of the application will be rendered by reference to the appended drawings and appended detailed description. Therefore, the realization process of how to apply the technical means to solve the technical problems and achieve the technical effects can be fully understood and implemented.
According to the application, a YOLOv8 network is adopted to design a YOLO_SI model and a detection system, so that the purity of the silicon material is ensured to the greatest extent, and contact is avoided in the production process. The automation of large-granularity silicon material detection and removal can improve the production efficiency, reduce the number of operators and ensure the safe and orderly operation and upgrading of a silicon material production line.
According to fig. 1 and 13, a schematic diagram of a deep learning large-granularity silicon material detecting and removing system based on YOLOv8 network according to the present application includes the following steps:
s1, a data acquisition stage: acquiring a silicon material image on a conveyor belt through a high-frame-rate camera, and intercepting and retaining a video key frame as a picture;
s2, a data preprocessing stage: manually marking by using labelme, removing abnormal silicon material pictures, dividing an image sample into a training set and a testing set according to a ratio of 9:1, and carrying out data enhancement on the pictures to improve generalization capability; multiple groups of training sets can be created according to requirements, and multiple initial models are generated through training;
s3, building and training a network: creating a yolo_si algorithm by taking YOLOv8 as a backbone network, and obtaining a network model by using a training data set;
s4, testing: selecting the best model by using the test set, recording and using the test set parameters to train out a new model in the whole data set as a final model yolo_si;
s5, detection early warning stage: in field work, detecting all silicon materials in an image by using a YOLO_SI model, calibrating by a camera, using the focal length of a camera and the distance between the camera and a belt, and using the obtained prediction frame for positioning the silicon material position and calculating the real size of the silicon material; pre-alarming when the size exceeds a threshold value according to a preset threshold value of the system;
s6, tracking: setting a tracking mark on the silicon material exceeding the threshold value, updating and tracking the position of the silicon material exceeding the threshold value in real time by a tracking module, and sending a cleaning instruction to a lower computer when the position reaches a cleaning position;
s7, eliminating: and installing an air jet at a preset position, and sending a command to an air jet device by a lower computer to orderly and concurrently reject the marked and tracked large-granularity silicon material without conflict.
According to fig. 2, in step S1, the following steps are specifically included:
(1) Acquiring a silicon material picture video data stream through a high frame rate camera arranged above a conveyor belt;
(2) And intercepting key frames from the acquired silicon material video data stream as silicon material picture data.
According to fig. 3, in step S2, the following steps are specifically included:
(i1) Marking the position and the size of the silicon material in the acquired picture data by using a marking tool Lableme, wherein a rectangular mark is adopted in the marking process;
(i2) Removing the detected non-compliant silicon material picture data;
(i3) Dividing the image sample into a training set and a testing set according to the ratio of 9:1;
(i4) Data enhancement of training pictures, including: scaling the silicon material picture; carrying out random horizontal and vertical inversion on the silicon material pictures at different angles; adding random noise; converting the image from RGB to HSV color space, adjusting brightness, and performing normalization operation; adjusting the pixel values of the image to be uniformly distributed through histogram equalization; processing original image noise;
(i5) Multiple groups of training sets can be created according to requirements, and multiple initial models are generated through training;
according to fig. 4, in step S3, the following steps are specifically included:
(A) And a YOLOv8 network is used as a base line network, and a C2f module is adopted to realize further light weight and obtain more abundant gradient flow information. Deleting a convolution structure in an up-sampling stage of PAN-FPN in YOLOv5, and replacing a C3 module with a C2f module; reserving an SPPF module; the idea of PAN is reserved; adopting a coupled-Head, and simultaneously using a DFL concept, wherein the number of channels of the regression Head is changed into a form of 4 x reg_max; an Anchor-Free and Task-Aligned Assigner matching mode;
(B) Using the VFL loss function as a classification loss function; using CIOULoss+DFL as a regression Loss function; using hard-Swish as an activation function;
(C) The VFL focuses on asymmetric weighting operations as a classification loss function as follows:
q is label, ioU for positive samples q is bbox and gt, q=0 for negative samples, FL is not used for positive samples, BCE is used, and only a large number of adaptations IoU are weighted for highlighting the main samples. The standard FL is negative.
(D) The formulation of CIOU as part of the regression loss function is as follows:
(E) The formula of the DFL as a constituent part of the regression loss function is as follows:
DFL(S i ,S i+1 )＝-((y i+1 -y)log(S i )+(y-y i )log(S i+1 ))
(F) The hard-Swish formula as an activation function is as follows:
wherein ReLU (x) =max (0, x)
(G) The YOLO v8 model containing the VFL Loss function and the cious+dfl Loss function and the activation function hard-Swish is called yolo_si;
(H) Training based on the yolo_si network model;
in step S4, the method specifically includes the following steps:
(T1) training to generate a plurality of initial models, and selecting the best model from the test set, wherein the best model is free of error detection;
(T2) recording each item setting of the best model, including iteration times, learning rate, regularization mode, coefficients and the like;
(T3) retraining a new model using the entire dataset as the final model yolo_si;
according to fig. 5, in step S5, the following steps are specifically included:
converting the rough scale of the silicon material by utilizing the angle and the distance of the camera and the edge information of the detection result, and setting an alarm threshold value and an error threshold value; setting the system threshold value as the sum of an alarm threshold value and an error threshold value, and comparing the real size of the converted silicon material with the system threshold value; if the real size is larger than the threshold value, abnormal information is generated and sent to the real-time early warning module;
the real-time early warning module according to claim, characterized in that it specifically comprises:
(M1) a user and equipment management module which is responsible for registration, login and management of the user, personal information and provides super user management authority; providing an industrial personal computer equipment management mechanism;
(M2) a silicon material video stream acquisition and display module, wherein a video data stream of a silicon material picture is acquired through a high frame rate camera arranged above a conveyor belt, and the acquired image is sent to a real-time early warning module for real-time display; (M3) judging an abnormality module, adopting TensorRT deployment to provide low delay and high throughput, receiving a silicon material image sent by a silicon material video stream acquisition module, transmitting the silicon material image into a trained YOLO_SI model to obtain the position and the size of the silicon material, and comparing the calculated real size of the silicon material with a system threshold value; if the real size is larger than the threshold value, generating and pushing abnormal information to the page, and writing the abnormal information into a database;
(M4) a setting module for setting system parameters and log parameters, such as the early warning of the size of a silicon block;
and (M5) a log module for receiving the abnormal record of the real-time system and displaying the abnormal record, wherein the system stores the detection result and the running state of the system in a local log and synchronizes to a cloud server.
According to fig. 6, the user and device management module of the early warning system specifically includes:
(U1) providing registration, login and user name verification functions for management personnel, endowing corresponding rights, operating the whole system and storing information to a cloud server;
(U2) providing supervisors with operating system high-level rights to operate the entire system while other users may be managed, including obtaining and updating user information, logging out users, etc.;
(U3) the management of the industrial personal computer equipment, including the addition, deletion, modification and queue display of the industrial personal computer equipment, and starting and stopping the operation of the industrial personal computer;
according to fig. 7, the silicon material video stream acquisition and display module of the early warning system specifically includes:
(V1) capturing belt images primarily by means of a high frame rate camera mounted above the conveyor belt;
(V2) transmitting the image to the detection and identification system through the message queue;
(V3) providing a visual management interface, and displaying a conveyor belt picture of the industrial personal computer equipment in the monitoring list in real time;
according to fig. 8, the abnormality judging module of the early warning system specifically includes:
(R1) TensorRT deployment is employed to provide low latency and high throughput. And receiving the silicon material image sent by the silicon material video stream acquisition module, and transmitting the silicon material image into a trained yolo_si model for judgment.
(R2) when abnormal information of the corresponding industrial personal computer system exceeding a threshold value is detected, pushing the abnormal information to a monitoring page, and writing abnormal time and pictures into a database and a log module, wherein a cloud server is synchronous; according to fig. 9, the setup module of the early warning system specifically includes:
setting camera angle, distance, focal length, alarm threshold and error threshold parameters, server port and the like;
according to fig. 10, the log module of the early warning system specifically includes:
(Z1) writing of abnormal records, and selecting whether the abnormal records are synchronized to a cloud end or not;
(Z2) providing an early warning record searching function, and searching early warning records according to information such as industrial personal computer equipment, time, batch, threshold value, operators on duty and the like;
and (Z3) a remote service running on the cloud server is responsible for collecting and managing all detection log data, so that an administrator can conveniently monitor and manage production information remotely.
According to fig. 11, in step S6, the following steps are specifically included:
(F1) The method comprises the steps that a deep SORT tracking algorithm is adopted to realize multi-target tracking, when a real-time early warning module detects abnormal information of a corresponding industrial personal computer system exceeding a threshold value, a target positioning detection module positions a super-threshold silicon material in a picture and distributes an ID, and the ID is kept until the life cycle of the silicon material in the frame;
(F2) The motion predictor is responsible for simply predicting future motion by using the past motion information of the silicon material and assisting in overcoming the interference of the aerosol environment and shielding;
(F3) When each ID reaches the end clearing device of the conveyor belt, immediately sending a clearing signal to a lower computer of the rejection module, and entering a clearing stage, wherein in order to ensure orderly and concurrent clearing of clearing information, a message queue and multithreading are used for sending control commands in the stage;
according to fig. 12, in step S7, the following steps are specifically included:
(C1) The cleaning device is positioned at the tail end of the conveyor belt, a plurality of groups of air ejector tubes which are arranged tightly are preset, the air inlet end of the pulse valve is connected with the compressed air assembly, the air outlet end of the pulse valve is connected with the air ejector tubes, an electromagnetic control valve is arranged on the pulse valve, and the electromagnetic control valve is connected with the lower computer;
(C2) The lower computer receives the clearing signal, and starts the clearing device to realize ordered concurrency, no conflict and no contact rejection.
Claims (8)
1. A deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network is used for detecting the silicon material size, sending out early warning when the size exceeds a threshold value, and performing intelligent removal, and is characterized by comprising the following specific steps:
s1, a data acquisition stage: acquiring a silicon material image on a conveyor belt through a high-frame-rate camera, and intercepting and retaining a video key frame as a picture;
s2, a data preprocessing stage: manually marking by using labelme, removing abnormal silicon material pictures, dividing an image sample into a training set and a testing set according to a ratio of 9:1, and carrying out data enhancement on the pictures to improve generalization capability; multiple groups of training sets can be created according to requirements, and multiple initial models are generated through training;
s3, building and training a network: creating a yolo_si algorithm by taking YOLOv8 as a backbone network, and obtaining a network model by using a training data set;
s4, testing: selecting the best model by using the test set, recording and using the test set parameters to train out a new model in the whole data set as a final model yolo_si;
s5, detection early warning stage: in field work, detecting all silicon materials in an image by using a YOLO_SI model, calibrating by a camera, using the focal length of a camera and the distance between the camera and a belt, and using the obtained prediction frame for positioning the silicon material position and calculating the real size of the silicon material; pre-alarming when the size exceeds a threshold value according to a preset threshold value of the system;
s6, tracking: setting a tracking mark on the silicon material exceeding the threshold value, updating and tracking the position of the silicon material exceeding the threshold value in real time by a tracking module, and sending a cleaning instruction to a lower computer when the position reaches a cleaning position;
s7, eliminating: and installing an air jet at a preset position, and sending a command to an air jet device by a lower computer to orderly and concurrently reject the marked and tracked large-granularity silicon material without conflict.
2. The deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network according to claim 1, wherein the data acquisition stage specifically comprises:
(1) Acquiring a silicon material picture video data stream through a high frame rate camera arranged above a conveyor belt;
(2) And intercepting key frames from the acquired silicon material video data stream as silicon material picture data.
3. The deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network according to claim 1, wherein the data preprocessing stage specifically comprises:
(i1) Marking the position and the size of the silicon material in the acquired picture data by using a marking tool Lableme, wherein a rectangular mark is adopted in the marking process;
(i2) Removing the detected non-compliant silicon material picture data;
(i3) Dividing the image sample into a training set and a testing set according to the ratio of 9:1;
(i4) Data enhancement of training pictures, including: scaling the silicon material picture; carrying out random horizontal and vertical inversion on the silicon material pictures at different angles; adding random noise; converting the image from RGB to HSV color space, adjusting brightness, and performing normalization operation; adjusting the pixel values of the image to be uniformly distributed through histogram equalization; processing original image noise;
(i5) Multiple sets of training sets can be created according to requirements, and multiple initial models are generated through training.
4. The deep learning large-granularity silicon material detection and removal system based on the YOLOv8 network according to claim 1, wherein the network construction and training stage specifically comprises:
(A) The YOLOv8 network is used as a base line network, the further light weight is realized by adopting a C2f module, the ideas of an SPPF module and a PAN are reserved, and a coupled-Head, anchor-Free and Task-Aligned Assigner matching mode is adopted;
(B) Using the VFL loss function as a classification loss function; using cious+dfl as a regression loss function; using hard-Swish as an activation function;
(C) The VFL as a class loss function is formulated as follows:
q is label, ioU for positive samples q is bbox and gt, q=0 for negative samples, FL is not used for positive samples, BCE is used, and only a large number of adaptations IoU are weighted for highlighting the main samples. FL, standard when negative;
(D) The formulation of CIOU as part of the regression loss function is as follows:
(E) The formula of the DFL as a constituent part of the regression loss function is as follows:
DFL(S i ,S i+1 )＝-((y i+1 -y)log(S i )+(y-y i )log(S i+1 ))
(F) The hard-Swish formula as an activation function is as follows:
wherein ReLU (x) =max (0, x)
(G) The YOLO v8 model containing the VFL loss function and the ciouloss+dfl loss function and the activation function hard-Swish is called yolo_si;
(H) Training is performed based on the yolo_si network model.
5. The deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network according to claim 1, wherein the testing stage specifically comprises:
(T1) training to generate a plurality of initial models, and selecting the best model from the test set, wherein the best model is free of error detection;
(T2) recording each item setting of the best model, including iteration times, learning rate, regularization mode, coefficients and the like;
(T3) retraining a new model using the entire dataset as the final model yolo_si.
6. The deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network according to claim 1, wherein the detection early-warning stage specifically comprises:
converting the rough scale of the silicon material by utilizing the angle and the distance of the camera and the edge information of the detection result, and setting an alarm threshold value and an error threshold value; setting the system threshold value as the sum of an alarm threshold value and an error threshold value, and comparing the real size of the converted silicon material with the system threshold value; if the real size is larger than the threshold value, abnormal information is generated and sent to the real-time early warning module;
the real-time early warning module according to claim, characterized in that it specifically comprises:
(M1) a user and equipment management module which is responsible for registration, login and management of the user, personal information and provides super user management authority; providing an industrial personal computer equipment management mechanism;
(M2) a silicon material video stream acquisition and display module, wherein a video data stream of a silicon material picture is acquired through a high frame rate camera arranged above a conveyor belt, and the acquired image is sent to a real-time early warning module for real-time display; (M3) judging an abnormality module, adopting TensorRT deployment to provide low delay and high throughput, receiving a silicon material image sent by a silicon material video stream acquisition module, transmitting the silicon material image into a trained YOLO_SI model to obtain the position and the size of the silicon material, and comparing the calculated real size of the silicon material with a system threshold value; if the real size is larger than the threshold value, generating and pushing abnormal information to the page, and writing the abnormal information into a database;
(M4) a setting module for setting system parameters and log parameters, such as the early warning of the size of a silicon block;
and (M5) a log module for receiving the abnormal record of the real-time system and displaying the abnormal record, wherein the system stores the detection result and the running state of the system in a local log and synchronizes to a cloud server.
The user and equipment management module of the early warning system specifically comprises:
(U1) providing registration, login and user name verification functions for management personnel, endowing corresponding rights, operating the whole system and storing information to a cloud server;
(U2) providing supervisors with operating system high-level rights to operate the entire system while other users may be managed, including obtaining and updating user information, logging out users, etc.;
(U3) the management of the industrial personal computer equipment, including the addition, deletion, modification and queue display of the industrial personal computer equipment, and starting and stopping the operation of the industrial personal computer;
the silicon material video stream acquisition and display module of the early warning system specifically comprises:
(V1) capturing belt images primarily by means of a high frame rate camera mounted above the conveyor belt;
(V2) transmitting the image to the detection and identification system through the message queue;
(V3) providing a visual management interface, and displaying a conveyor belt picture of the industrial personal computer equipment in the monitoring list in real time;
the abnormal judgment module of the early warning system specifically comprises:
(R1) TensorRT deployment is employed to provide low latency and high throughput. Receiving a silicon material image sent by a silicon material video stream acquisition module, and transmitting the silicon material image into a trained yolo_si model for judgment;
(R2) when abnormal information of the corresponding industrial personal computer system exceeding a threshold value is detected, pushing the abnormal information to a monitoring page, and writing abnormal time and pictures into a database and a log module, wherein a cloud server is synchronous;
the setting module of the early warning system specifically comprises:
setting camera angle, distance, focal length, alarm threshold and error threshold parameters, server port and the like;
the log module of the early warning system specifically comprises:
(Z1) writing of abnormal records, and selecting whether the abnormal records are synchronized to a cloud end or not;
(Z2) providing an early warning record searching function, and searching early warning records according to information such as industrial personal computer equipment, time, batch, threshold value, operators on duty and the like;
and (Z3) a remote service running on the cloud server is responsible for collecting and managing all detection log data, so that an administrator can conveniently monitor and manage production information remotely.
7. The deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network according to claim 1, wherein the tracking stage specifically comprises:
(F1) The method comprises the steps that a deep SORT tracking algorithm is adopted to realize multi-target tracking, when a real-time early warning module detects abnormal information of a corresponding industrial personal computer system exceeding a threshold value, a target positioning detection module positions a super-threshold silicon material in a picture and distributes an ID, and the ID is kept until the life cycle of the silicon material in the frame;
(F2) The motion predictor is responsible for simply predicting future motion by using the past motion information of the silicon material and assisting in overcoming the interference of the aerosol environment and shielding;
(F3) When each ID reaches the end clearing device of the conveyor belt, a clearing signal is immediately sent to a lower computer of the rejection module, and a clearing stage is entered, wherein in order to ensure orderly and concurrent clearing of clearing information, a message queue and multithreading are used for sending control commands in the stage.
8. The deep learning large-granularity silicon material detection and removal system based on a YOLOv8 network according to claim 1, wherein the rejection stage specifically comprises:
(C1) The cleaning device is positioned at the tail end of the conveyor belt, a plurality of groups of air ejector tubes which are arranged tightly are preset, the air inlet end of the pulse valve is connected with the compressed air assembly, the air outlet end of the pulse valve is connected with the air ejector tubes, an electromagnetic control valve is arranged on the pulse valve, and the electromagnetic control valve is connected with the lower computer;
(C2) The lower computer receives the clearing signal, and starts the clearing device to realize ordered concurrency, no conflict and no contact rejection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310432318.6A CN116934674A (en) | 2023-07-25 | 2023-07-25 | Large-granularity silicon material detection and removal system based on YOLOv8 network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310432318.6A CN116934674A (en) | 2023-07-25 | 2023-07-25 | Large-granularity silicon material detection and removal system based on YOLOv8 network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116934674A true CN116934674A (en) | 2023-10-24 |
Family
ID=88388445
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310432318.6A Pending CN116934674A (en) | 2023-07-25 | 2023-07-25 | Large-granularity silicon material detection and removal system based on YOLOv8 network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116934674A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117876905A (en) * | 2024-01-20 | 2024-04-12 | 广州市图南软件科技有限公司 | Quick high-accuracy unmanned aerial vehicle aerial photographing target detection method |
-
2023
- 2023-07-25 CN CN202310432318.6A patent/CN116934674A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117876905A (en) * | 2024-01-20 | 2024-04-12 | 广州市图南软件科技有限公司 | Quick high-accuracy unmanned aerial vehicle aerial photographing target detection method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhang et al. | Deep convolutional neural networks for forest fire detection | |
| CN111179249A (en) | Power equipment detection method and device based on deep convolutional neural network | |
| EP3382643A1 (en) | Automated object tracking in a video feed using machine learning | |
| CN111754498A (en) | Conveyor belt carrier roller detection method based on YOLOv3 | |
| CN109284729A (en) | Method, apparatus and medium based on video acquisition human face recognition model training data | |
| CN109685075A (en) | A kind of power equipment recognition methods based on image, apparatus and system | |
| CN110569843B (en) | Intelligent detection and identification method for mine target | |
| CN116934674A (en) | Large-granularity silicon material detection and removal system based on YOLOv8 network | |
| CN110458794B (en) | Quality detection method and device for accessories of rail train | |
| CN111160432A (en) | Automatic classification method and system for panel production defects | |
| CN112287915B (en) | Equipment fault warning method and system based on deep learning | |
| US20230186634A1 (en) | Vision-based monitoring of site safety compliance based on worker re-identification and personal protective equipment classification | |
| CN114862832A (en) | Method, device and equipment for optimizing defect detection model and storage medium | |
| CN113158829A (en) | Deep learning ore size measuring method and early warning system based on EfficientDet network | |
| CN115035088A (en) | Helmet wearing detection method based on yolov5 and posture estimation | |
| CN112687022A (en) | Intelligent building inspection method and system based on video | |
| CN114708537A (en) | Multi-view-angle-based system and method for analyzing abnormal behaviors of complex places | |
| CN110059076A (en) | A kind of Mishap Database semi-automation method for building up of power transmission and transformation line equipment | |
| CN103425958B (en) | A kind of method of motionless analyte detection in video | |
| Bhargava et al. | A study on potential of big visual data analytics in construction Arena | |
| CN117114420B (en) | Image recognition-based industrial and trade safety accident risk management and control system and method | |
| CN117593689A (en) | Mine personnel dangerous behavior identification method and system based on computer vision | |
| Wang et al. | Visual defect detection for substation equipment based on joint inspection data of camera and robot | |
| CN114387564A (en) | Head-knocking engine-off pumping-stopping detection method based on YOLOv5 | |
| CN112115876A (en) | Water-soluble method experimental process identification method based on 3D convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |