CN113838015B - Electrical product appearance defect detection method based on network cooperation - Google Patents
Electrical product appearance defect detection method based on network cooperation Download PDFInfo
- Publication number
- CN113838015B CN113838015B CN202111079409.3A CN202111079409A CN113838015B CN 113838015 B CN113838015 B CN 113838015B CN 202111079409 A CN202111079409 A CN 202111079409A CN 113838015 B CN113838015 B CN 113838015B
- Authority
- CN
- China
- Prior art keywords
- appearance
- defect
- model
- deep learning
- equipment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/0008—Industrial image inspection checking presence/absence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The application relates to an electrical product appearance defect detection method based on network cooperation. According to the application, in the appearance detection of the electrical product, a human-computer cooperative technology is utilized to collect a detection sample and a deep learning mechanism is overlapped to dynamically optimize an appearance detection model of the electrical product, and finally, an appearance detection model expert library of the electrical product is formed. At present, the traditional appearance detection system in the electrical appliance industry has higher requirements and needs professionally trained personnel to operate, the system adopts detection data to automatically learn, an intelligent quality management analysis system is realized, detection standardization is realized, the influence of human factors on detection results is reduced, the quality management level is improved, personnel deployment is reduced, and the production cost is saved.
Description
Technical Field
The application relates to a network collaboration-based method for detecting appearance defects of electrical products, and belongs to the technical fields of artificial intelligence and industrial control automation.
Background
With the deep development and promotion of intelligent manufacturing, industrial Internet and artificial intelligence in China, the application of various electrical equipment is increased, and the corresponding product quality and use safety problems are not ignored. At present, the intellectualization of inspection means and tools in the production and manufacturing links of the electrical equipment at the user end is still in the primary stage in China, and the requirement of rapid development of industry cannot be met.
Currently, a product appearance defect detection system and a device thereof based on deep learning generally comprise an image display device, a server, a controller and the like. In most detection systems carrying a server, a deep learning module is included at the server end, and iterative computation is performed on collected image data to obtain an updated detection model, but defect types of the detection model are mostly fixed and do not have expansion of new types, and meanwhile, for the model being detected and a product detection result detected by the model, a user at the edge side cannot inform the machine learning module of the new defect types and product characteristics, and further cannot form a new detection mechanism and update a product detection expert library according to the new situation
The application patent application with the application number of 202010107000.7 discloses an intelligent control system of an assembly production line based on an expert database, which comprises the expert database, an upper computer, a robot, a screw machine, a detection unit, a driving unit, a visual identification unit, a nail feeder, an intelligent assembly unit and the like. The patent application does not carry out iterative optimization on the detection model through deep machine learning, and meanwhile, the generated expert database cannot meet the requirement of updating under a similar distributed system at the edge side and the far end, so that a new detection mechanism is not formed.
The application patent application with the application number of 202010888429.4 discloses a deep learning device and a deep learning application method, wherein the deep learning device comprises the following steps: model library, operating device, executive device. And the user selects and operates the page visualization component according to the application requirement, and invokes the corresponding deep learning model to process the input data, so that the required deep learning task for the input data can be realized. The iterative training of the model is not embodied in the method disclosed in the patent application, and the breadth of deep learning and the precision of the produced model are limited.
The application patent application with the application number 202010829978.4 discloses intelligent steel belt visual detection equipment, which is more in detection by using a model which is trained and completed, has no steps based on network coordination and machine learning, and belongs to a traditional visual recognition detection solution.
Disclosure of Invention
The purpose of the application is that: and introducing a man-machine cooperative mechanism into a product appearance defect detection method based on deep learning.
In order to achieve the above purpose, the technical scheme of the application is to provide a network collaboration-based method for detecting appearance defects of electrical products, which is characterized by comprising the following steps:
step 1, an appearance detection expert model library in the electrical appliance field is built in a remote training server, the remote training server trains a machine deep learning algorithm by utilizing training data sets which are uploaded by edge side man-machine interaction equipment and correspond to different electrical appliance equipment models, a plurality of appearance detection deep learning models which correspond to the different electrical appliance equipment models are formed, all the appearance detection deep learning models are stored in the appearance detection expert model library in the electrical appliance field, and a mapping relation is built between each appearance detection deep learning model and the corresponding electrical appliance equipment model;
when the edge side production equipment needs to produce an electrical product with a new electrical equipment model, the remote training server trains to obtain a new appearance detection deep learning model and stores the new appearance detection deep learning model in an appearance detection expert model library in the electrical equipment field, and then the remote training server informs the edge side man-machine interaction equipment of model information of the new appearance detection deep learning model, so that a user can select the new appearance detection deep learning model in the appearance detection expert model library in the electrical equipment field based on the electrical equipment model through the edge side man-machine interaction equipment;
when the appearance detection deep learning model detects appearance defects of an electrical product, firstly, carrying out standardization operation on received image data with any size, and unifying the received image data with the input size of a CNN convolutional neural network, thereby obtaining standardized image data; then inputting the standardized image data into a CNN convolutional neural network, and performing convolutional operation on the standardized image data by convolutional check of different dimensions and types to respectively generate a small target feature map, a medium target feature map and a large target feature map, wherein the sampling receptive field of the small target feature map is smaller than that of the medium target feature map, and the sampling receptive field of the medium target feature map is smaller than that of the large target feature map; the appearance detection deep learning model detects small target defects based on a small target feature map, detects large target defects based on a medium target feature map, and detects large target defects based on a large target feature map, when detecting, the appearance detection deep learning model carries out frame regression prediction of target detection objects and classification of target objects under a frame by using a frame regression algorithm and a multi-classification algorithm on vector groups corresponding to the small target feature map, the medium target feature map and the large target feature map, if judging that defects exist, a small target defect frame and/or a medium target defect frame and/or a large target defect frame and a defect category for marking the positions of the small target defects and/or the positions of the medium target defects and/or the positions of the large target defects respectively are obtained, and output parameters X, Y, w, h and confidence are obtained, wherein X and Y represent X-axis offset and Y-axis offset of the central position of the small target defect frame or the medium target defect frame relative to the left upper corner position of the current defect frame, and w and h represent the width and the height of the whole image respectively, and the confidence value of the width and the height of the large target defect frame or the large target defect frame are not larger than 1;
step 2, a user generates model calling information related to the electric appliance model by utilizing edge side man-machine interaction equipment according to the electric appliance model of the electric appliance actually produced by the edge side production equipment; after the man-machine interaction equipment uploads the model calling information to a remote training server, the remote training server calls an appearance detection deep learning model corresponding to the current electrical equipment model stored in an appearance detection expert model library in the electrical equipment field according to the model calling information;
step 3, after the edge side production equipment obtains a real-time appearance picture of the current electrical product through the image acquisition equipment, uploading the real-time appearance picture to a far-end training server, inputting the received real-time appearance picture into a called appearance detection deep learning model by the far-end training server, judging whether the current electrical product has defects by using the appearance detection deep learning model by using the real-time appearance picture, if the current electrical product has defects, outputting predicted small target defect frames and/or middle target defect frames and/or large target defect frames and defect types, and outputting corresponding output parameters x, y, w, h, confidence degrees and defect types;
step 4, the edge side man-machine interaction equipment frames a corresponding defect area on the real-time appearance picture by utilizing the received output parameters x, y, w, h, and displays the corresponding confidence coefficient and defect category;
step 5, if the confidence coefficient displayed in the step 4 is lower than a preset threshold value, or a new defect type is judged to appear according to the real-time appearance picture, drawing a defect frame on the real-time appearance picture by using edge side man-machine interaction equipment, inputting the corresponding defect type, obtaining a corresponding defect frame parameter x, y, w, h by using the drawn defect frame by using the edge side man-machine interaction equipment, and setting the confidence coefficient to be 1 by using the edge side man-machine interaction equipment; the edge side man-machine interaction equipment stores the defect frame parameters x, y, w, h obtained before and the corresponding defect types, confidence values and real-time appearance pictures as new training data;
if the confidence coefficient displayed in the step 4 is not lower than the preset threshold value and no new defect type exists, the edge side man-machine interaction equipment stores the defect frame parameters x, y, w, h obtained in the step 4, the corresponding defect type, the confidence coefficient value and the real-time appearance picture as new training data;
and 6, uploading all new training data collected by the edge side human-computer interaction equipment according to a set period to a training server, forming a new training data set by the training server through all new training data collected by each period step, retraining the appearance detection deep learning model used in the steps 2 to 5 based on the training data set to obtain an updated and optimized appearance detection deep learning model, and storing the appearance detection deep learning model in an appearance detection expert model library in the electrical appliance field instead of the existing appearance detection deep learning model.
Preferably, in step 1, the training data set of the current electrical equipment model is obtained by the following method:
the edge side man-machine interaction equipment obtains an appearance picture of the electrical product of the current electrical equipment model through the image acquisition equipment, then judges whether the appearance picture of the electrical product has appearance defects, and manually marks the appearance picture of the electrical product with the appearance defects; when manual marking is carried out, marking the sampling picture based on the appearance detection standard in the field of the electrical appliance industry, and marking a defect frame and defect types on the appearance picture of the electrical appliance product; the edge side man-machine interaction equipment uploads the appearance pictures of the electrical products which are manually marked and the appearance pictures of the electrical products which do not need to be manually marked to a far-end training server, the far-end training server builds a training data set based on the appearance pictures of the electrical products which are received in a certain time period, and a machine deep learning algorithm is trained by using the training data set, so that an appearance detection deep learning model corresponding to the current electrical equipment model is obtained.
According to the application, man-machine cooperation and deep learning are combined in the research visual detection technology to jointly form the product detection model expert library, so that the detection quality is improved. According to the application, field data are collected through a visual detection technology and a man-machine interaction technology, and deep learning of the traditional electrical appliance appearance detection model at a server side is realized through a cloud platform technology. Finally, the application stores the optimized models based on a large number of electrical appliance appearance sample data at the edge side into a special database to form an expert model library for electrical appliance appearance detection, and forms a highly intelligent detection standard in the electrical appliance industry in practical application through the models stored in the expert model library.
According to the application, in the appearance detection of the electrical product, a human-computer cooperative technology is utilized to collect a detection sample and a deep learning mechanism is overlapped to dynamically optimize an appearance detection model of the electrical product, and finally, an appearance detection model expert library of the electrical product is formed. At present, the traditional appearance detection system in the electrical appliance industry has higher requirements and needs professionally trained personnel to operate, the system adopts detection data to automatically learn, an intelligent quality management analysis system is realized, detection standardization is realized, the influence of human factors on detection results is reduced, the quality management level is improved, personnel deployment is reduced, and the production cost is saved.
Drawings
FIG. 1 is a schematic diagram of a product appearance defect detection method based on network collaboration;
FIG. 2 is a schematic diagram of an application of a product appearance defect detection method based on network collaboration;
FIG. 3 is a flow chart of model warehousing after training of the deep machine learning algorithm;
FIG. 4 is a flow chart of a human-computer interaction interface for human-computer collaborative detection;
FIG. 5 is a flow chart of a visual WEB interface control deep machine learning service.
Detailed Description
The application will be further illustrated with reference to specific examples. It is to be understood that these examples are illustrative of the present application and are not intended to limit the scope of the present application. Furthermore, it should be understood that various changes and modifications can be made by one skilled in the art after reading the teachings of the present application, and such equivalents are intended to fall within the scope of the application as defined in the appended claims.
The application provides a network collaboration-based method for detecting appearance defects of an electrical product, which comprises the following steps:
step 1, an appearance detection expert model library in the electrical appliance field is built in a remote training server, the remote training server trains a machine deep learning algorithm by utilizing training data sets which are uploaded by edge side man-machine interaction equipment and correspond to different electrical appliance equipment models, a plurality of appearance detection deep learning models which correspond to the different electrical appliance equipment models are formed, all the appearance detection deep learning models are stored in the appearance detection expert model library in the electrical appliance field, and a mapping relation is built between each appearance detection deep learning model and the corresponding electrical appliance equipment model.
In the step, a training data set of the current electrical equipment model is obtained by adopting the following method:
the edge side man-machine interaction equipment obtains appearance pictures of the electrical products of the current electrical equipment model through the image acquisition equipment, then judges whether the appearance pictures of the electrical products have appearance defects, and manually marks the appearance pictures of the electrical products with the appearance defects. And when the manual marking is carried out, marking the sampling picture based on the appearance detection standard in the field of the electrical appliance industry, and marking a defect frame and defect types on the appearance picture of the electrical product. The edge side man-machine interaction equipment uploads the appearance pictures of the electrical products which are manually marked and the appearance pictures of the electrical products which do not need to be manually marked to a far-end training server, the far-end training server builds a training data set based on the appearance pictures of the electrical products which are received in a certain time period, and a machine deep learning algorithm is trained by using the training data set, so that an appearance detection deep learning model corresponding to the current electrical equipment model is obtained.
When the edge side production equipment needs to produce an electrical product with a new electrical equipment model, the remote training server trains based on the method to obtain a new appearance detection deep learning model and stores the new appearance detection deep learning model in an appearance detection expert model library in the electrical equipment field, and then the remote training server informs the edge side man-machine interaction equipment of model information of the new appearance detection deep learning model, so that a user can select the new appearance detection deep learning model in the appearance detection expert model library in the electrical equipment field based on the electrical equipment model through the edge side man-machine interaction equipment.
In this embodiment, when the appearance detection deep learning model detects appearance defects of an electrical product, firstly, standardized operation is performed on received image data with any size, and the received image data is unified into an input size of a CNN convolutional neural network through scaling and pad (extended pixel value 0), so as to obtain standardized image data. And then inputting the standardized image data into a CNN convolutional neural network, and performing convolution operation on the standardized image data by convolution checks of different dimensions and types to respectively generate a small target feature map with the 32 times of downsampling receptive field of 13X 13pixels, a medium target feature map with the 16 times of downsampling receptive field of 26X 26pixels and a large target feature map with the 8 times of downsampling receptive field of 52X 52 pixels. Different size targets are detected under the condition of multiple sizes through three feature maps with different sizes, such as: detecting small target defects, such as scratches, flaws, paint drops, based on the small target feature map; detecting middle target defects such as nameplate flaws and electrical appliance surface knob flaws based on the middle target feature map; a large target defect, such as a wiring terminal abnormality, is detected based on the large target feature map. Finally, the appearance detection deep learning model uses a frame regression algorithm (BBoxReg) and a multi-classification algorithm (SVMs) to carry out frame regression prediction of a target detection object and classification of a target object under a frame on vector groups corresponding to a small target feature image, a middle target feature image and a large target feature image under three scales, if the defects are judged to exist, a small target defect frame and/or a middle target defect frame and/or a large target defect frame and a defect category for marking the positions of the small target defect and/or the positions of the middle target defect and/or the positions of the large target defect respectively are obtained, and output parameters X, Y, w, h and confidence are obtained, wherein X and Y represent X-axis offset and Y-axis offset of the central position of the small target defect frame or the middle target defect frame or the large target defect frame relative to the left upper corner position of the current defect, w and h represent the ratio of the width and the height of the small target defect frame or the middle target defect frame or the large target defect frame to the whole picture, and the confidence value is not larger than 1. The appearance detection deep learning model performs normalization processing on an input image, and the marked parameters x, y, w and h are output according to the center point coordinates x, y, w and h of the normalized defect frame, wherein the output parameters are x, y, w and h occupy the proportion of image pixels, and finally, the reduction operation is needed when the defect frame is displayed by the terminal equipment.
And 2, generating model calling information related to the electric appliance model by using the edge side man-machine interaction equipment according to the electric appliance model of the electric appliance actually produced by the edge side production equipment by a user. And after the man-machine interaction equipment uploads the model calling information to the remote training server, the remote training server calls an appearance detection deep learning model corresponding to the current electrical equipment model stored in an appearance detection expert model library in the electrical equipment field according to the model calling information.
Step 3, after the edge side production equipment obtains a real-time appearance picture of the current electrical product through the image acquisition equipment, uploading the real-time appearance picture to a far-end training server, inputting the received real-time appearance picture into a called appearance detection deep learning model by the far-end training server, judging whether the current electrical product has defects by using the appearance detection deep learning model by using the real-time appearance picture, if the current electrical product has defects, outputting predicted small target defect frames and/or middle target defect frames and/or large target defect frames and defect types, and outputting corresponding output parameters x, y, w, h, confidence and defect types.
And 4, the edge side man-machine interaction equipment frames a corresponding defect area on the real-time appearance picture by utilizing the received output parameters x, y, w, h, and displays the corresponding confidence and defect category.
And 5, if the confidence coefficient displayed in the step 4 is lower than a preset threshold value, or if the new defect type appears according to the real-time appearance picture, drawing a defect frame on the real-time appearance picture by using edge side man-machine interaction equipment, inputting the corresponding defect type, obtaining a corresponding defect frame parameter x, y, w, h by using the drawn defect frame by using the edge side man-machine interaction equipment, and setting the confidence coefficient to be 1 by using the edge side man-machine interaction equipment. The edge side man-machine interaction device stores the defect frame parameters x, y, w, h obtained before, the corresponding defect types, the confidence value and the real-time appearance picture as a new training data.
If the confidence coefficient displayed in the step 4 is not lower than the preset threshold value and no new defect type appears, the edge side man-machine interaction device stores the defect frame parameters x, y, w, h obtained in the step 4, the corresponding defect type, the confidence coefficient value and the real-time appearance picture as new training data.
And 6, uploading all new training data collected by the edge side human-computer interaction equipment according to a set period to a training server, forming a new training data set by the training server through all new training data collected by each period step, retraining the appearance detection deep learning model used in the steps 2 to 5 based on the training data set to obtain an updated and optimized appearance detection deep learning model, and storing the appearance detection deep learning model in an appearance detection expert model library in the electrical appliance field instead of the existing appearance detection deep learning model.
In this embodiment, the man-machine interaction device needs to be equipped with an Intel siren J1800 processor as a main chip, a 2g DDR3 memory, an SSD hard disk, and a pre-installed ubuntu16.04 and above version operating system, and is equipped with a display screen having a touch function. As shown in fig. 2, the man-machine interaction device needs to be provided with a configuration function interface for completing necessary configuration options, a display interface for displaying the detected image, uploading a data set and updating a detection model.
The built-in hardware standard of the training server needs to be provided with an Intel to strong 5128 processor as a main chip, a 32G DDR3 memory and an SSD hard disk are built in the training server, a GPU unit is required to be configured, and Ubuntu16.04 and operating systems with the versions are preloaded. In order to be able to achieve model optimization, the operating system must also be preloaded with relational databases, deep machine learning frameworks, web server frameworks. Training server
The operation of the training server is realized by adopting a visual WEB operation interface, and the operation process is as follows:
1. the visual operation interface is a visual operation interface which can be deployed at the cloud end and provided by a WEB service built in a server.
2. The visual operation interface is connected with the man-machine interaction equipment to check parameters of the man-machine interaction equipment, the visual operation interface is connected with the server to specify training parameters, the electric appliance appearance detection model training is started, and the model library information of the generator appearance detection model is transmitted to the man-machine interaction equipment.
In the system, a Web interface is developed based on a flash framework and is responsible for communication with man-machine interaction equipment and a server. As shown in fig. 1, the Web interface needs to be connected with the server, so that training can be started by issuing a training instruction from the visual interface to the training server.
As shown in fig. 1, the Web interface needs to be connected with the man-machine interaction device, so that a user can query parameters of a detected product in the visual interface and change the setting of the man-machine interaction device, and the whole system is configured and monitored at the cloud end to conveniently manage the running state of the system.
Claims (2)
1. The method for detecting the appearance defects of the electrical products based on network cooperation is characterized by comprising the following steps of:
step 1, an appearance detection expert model library in the electrical appliance field is built in a remote training server, the remote training server trains a machine deep learning algorithm by utilizing training data sets which are uploaded by edge side man-machine interaction equipment and correspond to different electrical appliance equipment models, a plurality of appearance detection deep learning models which correspond to the different electrical appliance equipment models are formed, all the appearance detection deep learning models are stored in the appearance detection expert model library in the electrical appliance field, and a mapping relation is built between each appearance detection deep learning model and the corresponding electrical appliance equipment model;
when the edge side production equipment needs to produce an electrical product with a new electrical equipment model, the remote training server trains to obtain a new appearance detection deep learning model and stores the new appearance detection deep learning model in an appearance detection expert model library in the electrical equipment field, and then the remote training server informs the edge side man-machine interaction equipment of model information of the new appearance detection deep learning model, so that a user can select the new appearance detection deep learning model in the appearance detection expert model library in the electrical equipment field based on the electrical equipment model through the edge side man-machine interaction equipment;
when the appearance detection deep learning model detects appearance defects of an electrical product, firstly, carrying out standardization operation on received image data with any size, and unifying the received image data with the input size of a CNN convolutional neural network, thereby obtaining standardized image data; then inputting the standardized image data into a CNN convolutional neural network, and performing convolutional operation on the standardized image data by convolutional check of different dimensions and types to respectively generate a small target feature map, a medium target feature map and a large target feature map, wherein the sampling receptive field of the small target feature map is smaller than that of the medium target feature map, and the sampling receptive field of the medium target feature map is smaller than that of the large target feature map; the appearance detection deep learning model detects small target defects based on a small target feature map, detects large target defects based on a medium target feature map, and detects large target defects based on a large target feature map, when detecting, the appearance detection deep learning model carries out frame regression prediction of target detection objects and classification of target objects under a frame by using a frame regression algorithm and a multi-classification algorithm on vector groups corresponding to the small target feature map, the medium target feature map and the large target feature map, if judging that defects exist, a small target defect frame and/or a medium target defect frame and/or a large target defect frame and a defect category for marking the positions of the small target defects and/or the positions of the medium target defects and/or the positions of the large target defects respectively are obtained, and output parameters X, Y, w, h and confidence are obtained, wherein X and Y represent X-axis offset and Y-axis offset of the central position of the small target defect frame or the medium target defect frame relative to the left upper corner position of the current defect frame, and w and h represent the width and the height of the whole image respectively, and the confidence value of the width and the height of the large target defect frame or the large target defect frame are not larger than 1;
step 2, a user generates model calling information related to the electric appliance model by utilizing edge side man-machine interaction equipment according to the electric appliance model of the electric appliance actually produced by the edge side production equipment; after the man-machine interaction equipment uploads the model calling information to a remote training server, the remote training server calls an appearance detection deep learning model corresponding to the current electrical equipment model stored in an appearance detection expert model library in the electrical equipment field according to the model calling information;
step 3, after the edge side production equipment obtains a real-time appearance picture of the current electrical product through the image acquisition equipment, uploading the real-time appearance picture to a far-end training server, inputting the received real-time appearance picture into a called appearance detection deep learning model by the far-end training server, judging whether the current electrical product has defects by using the appearance detection deep learning model by using the real-time appearance picture, if the current electrical product has defects, outputting predicted small target defect frames and/or middle target defect frames and/or large target defect frames and defect types, and outputting corresponding output parameters x, y, w, h, confidence degrees and defect types;
step 4, the edge side man-machine interaction equipment frames a corresponding defect area on the real-time appearance picture by utilizing the received output parameters x, y, w, h, and displays the corresponding confidence coefficient and defect category;
step 5, if the confidence coefficient displayed in the step 4 is lower than a preset threshold value, or a new defect type is judged to appear according to the real-time appearance picture, drawing a defect frame on the real-time appearance picture by using edge side man-machine interaction equipment, inputting the corresponding defect type, obtaining a corresponding defect frame parameter x, y, w, h by using the drawn defect frame by using the edge side man-machine interaction equipment, and setting the confidence coefficient to be 1 by using the edge side man-machine interaction equipment; the edge side man-machine interaction equipment stores the defect frame parameters x, y, w, h obtained before and the corresponding defect types, confidence values and real-time appearance pictures as new training data;
if the confidence coefficient displayed in the step 4 is not lower than the preset threshold value and no new defect type exists, the edge side man-machine interaction equipment stores the defect frame parameters x, y, w, h obtained in the step 4, the corresponding defect type, the confidence coefficient value and the real-time appearance picture as new training data;
and 6, uploading all new training data collected by the edge side human-computer interaction equipment according to a set period to a training server, forming a new training data set by the training server through all new training data collected by each period step, retraining the appearance detection deep learning model used in the steps 2 to 5 based on the training data set to obtain an updated and optimized appearance detection deep learning model, and storing the appearance detection deep learning model in an appearance detection expert model library in the electrical appliance field instead of the existing appearance detection deep learning model.
2. The method for detecting the appearance defects of the electrical products based on network coordination as claimed in claim 1, wherein in step 1, the training data set of the current electrical equipment model is obtained by adopting the following method:
the edge side man-machine interaction equipment obtains an appearance picture of the electrical product of the current electrical equipment model through the image acquisition equipment, then judges whether the appearance picture of the electrical product has appearance defects, and manually marks the appearance picture of the electrical product with the appearance defects; when manual marking is carried out, marking the sampling picture based on the appearance detection standard in the field of the electrical appliance industry, and marking a defect frame and defect types on the appearance picture of the electrical appliance product; the edge side man-machine interaction equipment uploads the appearance pictures of the electrical products which are manually marked and the appearance pictures of the electrical products which do not need to be manually marked to a far-end training server, the far-end training server builds a training data set based on the appearance pictures of the electrical products which are received in a certain time period, and a machine deep learning algorithm is trained by using the training data set, so that an appearance detection deep learning model corresponding to the current electrical equipment model is obtained.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111079409.3A CN113838015B (en) | 2021-09-15 | 2021-09-15 | Electrical product appearance defect detection method based on network cooperation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111079409.3A CN113838015B (en) | 2021-09-15 | 2021-09-15 | Electrical product appearance defect detection method based on network cooperation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113838015A CN113838015A (en) | 2021-12-24 |
| CN113838015B true CN113838015B (en) | 2023-09-22 |
Family
ID=78959322
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111079409.3A Active CN113838015B (en) | 2021-09-15 | 2021-09-15 | Electrical product appearance defect detection method based on network cooperation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113838015B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114445344A (en) * | 2021-12-29 | 2022-05-06 | 广州瑞松视觉技术有限公司 | Battery box appearance detection method and device |
| CN114994046A (en) * | 2022-04-19 | 2022-09-02 | 深圳格芯集成电路装备有限公司 | Defect detection system based on deep learning model |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5544256A (en) * | 1993-10-22 | 1996-08-06 | International Business Machines Corporation | Automated defect classification system |
| WO2018165753A1 (en) * | 2017-03-14 | 2018-09-20 | University Of Manitoba | Structure defect detection using machine learning algorithms |
| CN109598287A (en) * | 2018-10-30 | 2019-04-09 | 中国科学院自动化研究所 | The apparent flaws detection method that confrontation network sample generates is generated based on depth convolution |
| CN109993094A (en) * | 2019-03-26 | 2019-07-09 | 苏州富莱智能科技有限公司 | Fault in material intelligent checking system and method based on machine vision |
| CN110378869A (en) * | 2019-06-05 | 2019-10-25 | 北京交通大学 | A kind of rail fastening method for detecting abnormality of sample automatic marking |
| WO2020007096A1 (en) * | 2018-07-02 | 2020-01-09 | 北京百度网讯科技有限公司 | Method and device for detecting quality of display screen, electronic device, and storage medium |
| CA3128957A1 (en) * | 2019-03-04 | 2020-03-03 | Bhaskar Bhattacharyya | Near real-time detection and classification of machine anomalies using machine learning and artificial intelligence |
| CN110910353A (en) * | 2019-11-06 | 2020-03-24 | 成都数之联科技有限公司 | Industrial false failure detection method and system |
| CN111179223A (en) * | 2019-12-12 | 2020-05-19 | 天津大学 | Deep learning-based industrial automatic defect detection method |
| CN111223093A (en) * | 2020-03-04 | 2020-06-02 | 武汉精立电子技术有限公司 | AOI defect detection method |
| GB202007344D0 (en) * | 2020-03-17 | 2020-07-01 | Apical Ltd | Model-based machine-learning and inferencing |
| CN111489326A (en) * | 2020-01-13 | 2020-08-04 | 杭州电子科技大学 | Copper foil substrate surface defect detection method based on semi-supervised deep learning |
| CN111754456A (en) * | 2020-05-15 | 2020-10-09 | 清华大学 | Two-dimensional PCB appearance defect real-time automatic detection technology based on deep learning |
| CN111798419A (en) * | 2020-06-27 | 2020-10-20 | 上海工程技术大学 | Metal paint spraying surface defect detection method |
| CN112132776A (en) * | 2020-08-11 | 2020-12-25 | 苏州跨视科技有限公司 | Visual inspection method and system based on federal learning, storage medium and equipment |
| CN113096098A (en) * | 2021-04-14 | 2021-07-09 | 大连理工大学 | Casting appearance defect detection method based on deep learning |
| WO2021140483A1 (en) * | 2020-01-10 | 2021-07-15 | Everseen Limited | System and method for detecting scan and non-scan events in a self check out process |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109544548A (en) * | 2018-11-30 | 2019-03-29 | 北京百度网讯科技有限公司 | Defect inspection method, device, server, equipment and the storage medium of cutlery box |
-
2021
- 2021-09-15 CN CN202111079409.3A patent/CN113838015B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5544256A (en) * | 1993-10-22 | 1996-08-06 | International Business Machines Corporation | Automated defect classification system |
| WO2018165753A1 (en) * | 2017-03-14 | 2018-09-20 | University Of Manitoba | Structure defect detection using machine learning algorithms |
| WO2020007096A1 (en) * | 2018-07-02 | 2020-01-09 | 北京百度网讯科技有限公司 | Method and device for detecting quality of display screen, electronic device, and storage medium |
| CN109598287A (en) * | 2018-10-30 | 2019-04-09 | 中国科学院自动化研究所 | The apparent flaws detection method that confrontation network sample generates is generated based on depth convolution |
| CA3128957A1 (en) * | 2019-03-04 | 2020-03-03 | Bhaskar Bhattacharyya | Near real-time detection and classification of machine anomalies using machine learning and artificial intelligence |
| CN109993094A (en) * | 2019-03-26 | 2019-07-09 | 苏州富莱智能科技有限公司 | Fault in material intelligent checking system and method based on machine vision |
| CN110378869A (en) * | 2019-06-05 | 2019-10-25 | 北京交通大学 | A kind of rail fastening method for detecting abnormality of sample automatic marking |
| CN110910353A (en) * | 2019-11-06 | 2020-03-24 | 成都数之联科技有限公司 | Industrial false failure detection method and system |
| CN111179223A (en) * | 2019-12-12 | 2020-05-19 | 天津大学 | Deep learning-based industrial automatic defect detection method |
| WO2021140483A1 (en) * | 2020-01-10 | 2021-07-15 | Everseen Limited | System and method for detecting scan and non-scan events in a self check out process |
| CN111489326A (en) * | 2020-01-13 | 2020-08-04 | 杭州电子科技大学 | Copper foil substrate surface defect detection method based on semi-supervised deep learning |
| CN111223093A (en) * | 2020-03-04 | 2020-06-02 | 武汉精立电子技术有限公司 | AOI defect detection method |
| GB202007344D0 (en) * | 2020-03-17 | 2020-07-01 | Apical Ltd | Model-based machine-learning and inferencing |
| CN111754456A (en) * | 2020-05-15 | 2020-10-09 | 清华大学 | Two-dimensional PCB appearance defect real-time automatic detection technology based on deep learning |
| CN111798419A (en) * | 2020-06-27 | 2020-10-20 | 上海工程技术大学 | Metal paint spraying surface defect detection method |
| CN112132776A (en) * | 2020-08-11 | 2020-12-25 | 苏州跨视科技有限公司 | Visual inspection method and system based on federal learning, storage medium and equipment |
| CN113096098A (en) * | 2021-04-14 | 2021-07-09 | 大连理工大学 | Casting appearance defect detection method based on deep learning |
Non-Patent Citations (2)
| Title |
|---|
| Defect Image Sample Generation With GAN for Improving Defect Recognition;S. Niu, B. Li, X. Wang and H. Lin;《IEEE Transactions on Automation Science and Engineering》;全文 * |
| Weakly-Supervised Defect Segmentation on Periodic Textures Using CycleGAN;M. Kim, H. Jo, M. Ra and W. -Y. Kim;《IEEE Access》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113838015A (en) | 2021-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113838015B (en) | Electrical product appearance defect detection method based on network cooperation | |
| Subakti et al. | Indoor augmented reality using deep learning for industry 4.0 smart factories | |
| CN110796018A (en) | Hand motion recognition method based on depth image and color image | |
| US11881015B2 (en) | High-precision identification method and system for substations | |
| CN115184361B (en) | Real-time workpiece surface defect detection and evaluation system and method based on machine vision | |
| CN114155610B (en) | Panel assembly key action identification method based on upper half body posture estimation | |
| CN115330734A (en) | Automatic robot repair welding system based on three-dimensional target detection and point cloud defect completion | |
| CN113723300A (en) | Artificial intelligence-based fire monitoring method and device and storage medium | |
| CN112017172A (en) | System and method for detecting defects of deep learning product based on raspberry group | |
| CN111507325B (en) | Industrial visual OCR recognition system and method based on deep learning | |
| CN112686322A (en) | Part difference identification method, device, equipment and storage medium | |
| CN114972246A (en) | Die-cutting product surface defect detection method based on deep learning | |
| KR102366396B1 (en) | RGB-D Data and Deep Learning Based 3D Instance Segmentation Method and System | |
| Lyu et al. | Automated visual inspection expert system for multivariate statistical process control chart | |
| CN112710671A (en) | Intelligent visual inspection device and inspection method for defects of trolley overhead rail trolley line | |
| CN111583225A (en) | Defect detection method, device and storage medium | |
| CN112581472B (en) | Target surface defect detection method facing human-computer interaction | |
| CN113158632B (en) | Table reconstruction method for CAD drawing and computer readable storage medium | |
| CN115631374A (en) | Control operation method, control detection model training method, device and equipment | |
| CN117315368B (en) | Intelligent operation and maintenance inspection method for large-scale data center | |
| CN216747439U (en) | Appearance detection system based on visual identification | |
| CN117671379A (en) | Intelligent recognition system and method based on CCD visual detection | |
| CN117975359A (en) | Intelligent image recognition method and system for intelligent inspection robot | |
| CN116453223A (en) | Intelligent pilot control system and working method thereof | |
| CN116798293A (en) | Industrial Internet desktop type practical training teaching system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |