CN113361487A

CN113361487A - Foreign matter detection method, device, equipment and computer readable storage medium

Info

Publication number: CN113361487A
Application number: CN202110777006.XA
Authority: CN
Inventors: 张隽华; 黄雷
Original assignee: Wuxi Ea Medical Instruments Technologies Ltd
Current assignee: Wuxi Ea Medical Instruments Technologies Ltd
Priority date: 2021-07-09
Filing date: 2021-07-09
Publication date: 2021-09-07

Abstract

The invention provides a foreign matter detection method, a foreign matter detection device, foreign matter detection equipment and a computer-readable storage medium. According to the embodiment of the invention, the image to be detected of the tray with the appliance is obtained, then the target detection model obtained through pre-training is utilized to perform target detection on the image to be detected, the regional image including the detected target is cut out from the image to be detected in response to the detection of the target from the image to be detected, then the classification model obtained through pre-training is utilized to classify the regional image, the classification result whether the regional image includes the foreign matter or not is obtained, and further, whether the foreign matter is mixed in the tray or not is determined based on the classification result, so that the foreign matter possibly mixed in the inner packaging bag of the appliance in the sorting and packaging stage can be effectively detected.

Description

Foreign matter detection method, device, equipment and computer readable storage medium

[ technical field ] A method for producing a semiconductor device

The invention relates to the field of medical instruments, in particular to a foreign matter detection technology for medical instrument packages, and particularly relates to a foreign matter detection method, a device, equipment and a computer readable storage medium.

[ background of the invention ]

Shell-like orthodontic appliances (e.g., invisible appliances) based on polymeric materials are becoming increasingly popular because of their aesthetic, convenient, and cleaning benefits. Shell orthodontic appliances are devices that use the resilience of a deformation to reposition teeth from one configuration to another.

The process of producing shell-shaped orthodontic appliances (hereinafter referred to as appliances) generally requires the following steps: and (3) forming the photocuring mold, pressing the film of the corrector by using the photocuring mold, cutting the corrector, shelling, grinding the corrector, cleaning, sorting the corrector and packaging.

Because the production flow of correcting the ware is longer, and production environmental factor influences complicatedly, in the letter sorting packing stage of production line, foreign matter such as hair, filament have minimum probability to sneak into in the inner wrapping bag, although do not influence the wearing of correcting the ware, bring negative experience for the user to bring adverse effect for the brand image. Therefore, it is necessary to identify the foreign object in time and to clean it accordingly so as not to bring it into the inner packaging bag of the orthosis.

However, the foreign matters such as hair and filament are very difficult to judge or are very fine and thus difficult to detect, so that the prior art cannot effectively detect the foreign matters possibly mixed into the inner packaging bag in the sorting and packaging stage by manually and visually detecting the foreign matters, and the prior art needs a large amount of manpower resources, and has high cost, long time and low efficiency.

[ summary of the invention ]

Aspects of the present invention provide a method, an apparatus, a device and a computer readable storage medium for detecting foreign objects, which may be mixed into an inner bag of a orthosis during a sorting and packaging stage.

In one aspect of the present invention, there is provided a foreign object detection method including:

acquiring an image to be detected comprising a tray with a correction device;

carrying out target detection on the image to be detected by utilizing a target detection model obtained by pre-training;

in response to the target being detected from the image to be detected, cutting out a region image including the detected target from the image to be detected;

classifying the region images by using a classification model obtained by pre-training to obtain a classification result of whether the region images comprise foreign matters or not;

determining whether foreign matter is mixed in the tray based on the classification result.

As to the above-mentioned aspects and any possible implementation manner, there is further provided an implementation manner, the acquiring an image to be detected including a tray on which an appliance is placed, including:

and adopting a mode of overlooking and photographing to acquire the image of the tray with the appliance, so as to obtain the image to be detected of the tray with the appliance.

The above-described aspect and any possible implementation manner further provide an implementation manner, in which cropping out an area image including a detected target from the image to be detected includes:

and cutting out an area image which is larger than the target detection frame of the target and smaller than the image to be detected from the image to be detected according to a preset size expansion ratio.

The above-mentioned aspect and any possible implementation manner further provide an implementation manner, where the cutting out, according to a preset size expansion ratio, an area image that is larger than a target detection frame of the target and smaller than the image to be detected from the image to be detected includes:

and cutting out an area image which is larger than the target detection frame of the target and smaller than the image to be detected from the image to be detected according to a preset size expansion proportion of 1.0-1.2 times of the size of the target detection frame by taking the central position of the target detection frame as a center.

The above-described aspects and any possible implementation further provide an implementation, further including:

and sending a reminding message or an alarm message of the foreign matters mixed in the tray in response to the fact that the foreign matters mixed in the tray are determined based on the classification result.

acquiring a training set, wherein the training set comprises a plurality of sample images; the multiple sample images comprise positive sample images with foreign matters in trays for placing the orthodontic appliances and negative sample images without foreign matters in trays for placing the orthodontic appliances, the positive sample images are marked with foreign matter marking information, and the foreign matter marking information marked by the positive sample images comprises foreign matter positions and size marking information

Carrying out target detection on the sample images in the training set by using a target detection model to be trained to obtain a target detection result, wherein the target detection result comprises the position and the size of a foreign body;

and training the target detection model to be trained according to the target detection result of the sample image and the corresponding foreign matter marking information until a first preset training completion condition is met, and obtaining the target detection model.

The above-described aspect and any possible implementation further provide an implementation, where the obtaining a training set includes:

adopting a overlook photographing mode to acquire images of a plurality of trays which do not have foreign matters and are provided with the correcting appliances, so as to obtain a negative sample image set;

adopting a overlooking and photographing mode to acquire images of a plurality of trays with foreign matters and provided with the orthodontic appliances so as to obtain a positive sample image set;

and performing down-sampling on a negative sample image set, performing over-sampling on a positive sample image set, and obtaining the training set based on the negative sample image set and the positive sample image set.

The above-described aspect and any possible implementation manner further provide an implementation manner, where the acquiring a training set further includes:

performing data expansion based on the existing sample images in the training set by a preset data expansion mode to obtain expanded sample images, and adding the expanded sample images into the training set;

the preset data expansion mode comprises any one or more of the following modes: adjusting the brightness of the image, adjusting the contrast of the image, and performing translation, mirroring, rotation, noise addition, mosaic splicing and shielding operations on the image.

The above-mentioned aspects and any possible implementation manners further provide an implementation manner, training the target detection model to be trained until a first preset training completion condition is met, further including:

carrying out target detection on the sample images in the training set by using the target detection model;

in response to the detection of the target from the sample image, cutting out an area image sample which is larger than a target detection frame of the target and smaller than the sample image from the sample image according to a preset size expansion ratio;

classifying the region image sample by using a classification model to be trained to obtain a classification result of whether the region image sample comprises foreign matters;

and training the classification model to be trained according to the classification result of the regional image sample and the corresponding foreign matter marking information until a second preset training completion condition is met, and obtaining the classification model.

The above-mentioned aspect and any possible implementation manner further provide an implementation manner, where the second preset training completion condition includes:

and the training times of the classification model to be trained reach preset times, and/or the function value of the cross entropy loss function with the weight calculated based on the classification result of the regional image sample and the corresponding foreign object labeling information is smaller than a preset threshold value.

In another aspect of the present invention, there is provided a foreign object detection apparatus including:

the first acquisition module is used for acquiring an image to be detected comprising a tray with an appliance;

the detection module is used for carrying out target detection on the image to be detected by utilizing a target detection model obtained by pre-training;

the cutting module is used for responding to the detection of the target in the image to be detected and cutting out a regional image comprising the detected target from the image to be detected;

the classification module is used for classifying the region images by using a classification model obtained by pre-training to obtain a classification result of whether the region images comprise foreign matters or not;

a determination module for determining whether foreign matter is mixed in the tray based on the classification result.

As for the above-mentioned aspect and any possible implementation manner, an implementation manner is further provided, where the first obtaining module is specifically configured to:

The above-described aspect and any possible implementation further provide an implementation, where the clipping module is specifically configured to:

and the sending module is used for responding to the fact that the foreign matters are mixed in the tray according to the classification result and sending a reminding message or an alarm message of the foreign matters mixed in the tray.

the second acquisition module is used for acquiring a training set, and the training set comprises a plurality of sample images; the method comprises the following steps that a plurality of sample images are obtained, wherein the sample images comprise a positive sample image and a negative sample image, foreign matters exist in a tray with an appliance, the negative sample image does not exist in the tray with the appliance, foreign matter marking information is marked on the positive sample image, and the foreign matter marking information comprises foreign matter position and size marking information;

the first training module is used for carrying out target detection on the sample images in the training set by using a target detection model to be trained to obtain a target detection result, wherein the target detection result comprises the position and the size of a foreign matter; and training the target detection model to be trained according to the target detection result of the sample image and the corresponding foreign matter marking information until a first preset training completion condition is met, and obtaining the target detection model.

As for the above-mentioned aspect and any possible implementation manner, an implementation manner is further provided, where the second obtaining module is specifically configured to:

performing image oversampling on a plurality of trays with foreign matters and provided with the correcting devices by adopting a overlooking and photographing mode to obtain a positive sample image set;

and performing down-sampling on the negative sample image set, performing over-sampling on the positive sample image set, and obtaining the training set based on the down-sampled negative sample image set and the over-sampled positive sample image set.

the expansion module is used for performing data expansion on the basis of the existing sample images in the training set in a preset data expansion mode to obtain expanded sample images, and adding the expanded sample images into the training set;

The above-mentioned aspects and any possible implementation manners further provide an implementation manner, where the detection module is further configured to perform target detection on the sample images in the training set by using the target detection model;

the cutting module is further used for cutting out an area image sample which is larger than a target detection frame of the target and smaller than the sample image from the sample image according to a preset size expansion ratio in response to the target being detected from the sample image;

the device further comprises:

the second training module is used for classifying the regional image samples by using the classification model to be trained to obtain a classification result of whether the regional image samples comprise foreign matters or not; and training the classification model to be trained according to the classification result of the regional image sample and the corresponding foreign matter marking information until a second preset training completion condition is met, and obtaining the classification model.

In another aspect of the present invention, there is provided an apparatus comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement a foreign object detection method as provided in an aspect above.

In another aspect of the present invention, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the foreign object detection method provided in the above-described aspect.

According to the technical scheme, the image to be detected of the tray with the appliance is obtained, the image to be detected is subjected to target detection by utilizing a target detection model obtained through pre-training, the region image including the detected target is cut out from the image to be detected in response to the detection of the target from the image to be detected, the region image is classified by utilizing a classification model obtained through pre-training, the classification result of whether the region image includes the foreign matters or not is obtained, and whether the foreign matters are mixed in the tray or not is determined based on the classification result, so that the foreign matters (hairs, filaments and the like) which are difficult to identify for human eyes in the tray can be effectively identified based on a deep learning mode, and the possible foreign matters mixed in the inner packaging bag in the sorting and packaging stage can be effectively checked, so as to clean correspondingly in time and avoid bringing foreign matters into the inner packaging bag of the appliance.

In addition, by adopting the technical scheme provided by the invention, based on a deep learning mode, foreign matters which are difficult to identify can be effectively identified through a robust and reliable target detection model and a classification model, and the foreign matter identification accuracy is improved.

In addition, by adopting the technical scheme provided by the invention, based on a deep learning mode, whether foreign matters (hairs, filaments and the like) which are difficult to identify for human eyes are mixed in the tray or not can be identified through a robust and reliable target detection model and a classification model, so that occupation of operators and quality inspectors can be avoided, manpower resources are saved, quality inspection cost is reduced, quality inspection time is shortened, quality inspection efficiency is improved, workload of the operators and the quality inspectors is reduced, production efficiency is improved, and boosting capacity is favorably improved.

In addition, by adopting the technical scheme provided by the invention, based on a deep learning mode, whether foreign matters (hairs, filaments and the like) which are difficult to identify for human eyes are mixed in the tray or not can be identified through a robust and reliable target detection model and a classification model, the identification accuracy is high, and the method is reliable in a complex production environment.

In addition, by adopting the technical scheme provided by the invention, the user experience can be effectively improved.

[ description of the drawings ]

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed in the embodiments or the prior art descriptions will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without inventive labor.

Fig. 1 is a schematic flow chart illustrating a foreign object detection method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram illustrating a method for detecting a foreign object according to an embodiment of the present invention;

FIG. 3 is a schematic flow chart illustrating a process of training a target detection model according to an embodiment of the present invention;

FIG. 4 is a schematic flow chart illustrating a process of training a classification model according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of a foreign object detection apparatus according to another embodiment of the present invention;

fig. 6 is a schematic structural diagram of a foreign object detection apparatus according to another embodiment of the present invention;

FIG. 7 is a block diagram of an exemplary computer system/server 12 suitable for use in implementing embodiments of the present invention.

[ detailed description ] embodiments

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.

It should be noted that the terminal according to the embodiment of the present invention may include, but is not limited to, a mobile phone, a Personal Digital Assistant (PDA), a wireless handheld device, a Tablet Computer (Tablet Computer), a Personal Computer (PC), an MP3 player, an MP4 player, a wearable device (e.g., smart glasses, smart watch, smart bracelet, etc.), and the like.

In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.

The invention has the main idea that a deep learning mode is adopted, the target detection is firstly carried out on the image to be detected by using the target detection model, then the classification is carried out by using the classification model, so as to detect whether the tray comprises the foreign matters or not, and the foreign matters (hair, filaments and the like) which are difficult to identify for human eyes in the tray can be effectively identified, so that the foreign matters possibly mixed into the inner packaging bag in the sorting and packaging stage can be effectively checked.

Fig. 1 is a schematic flow chart of a foreign object detection method according to an embodiment of the present invention, as shown in fig. 1.

101. And acquiring an image to be detected of the tray with the appliance.

102. And carrying out target detection on the image to be detected by utilizing a target detection model obtained by pre-training.

The result of the target detection on the image to be detected can be that a target is detected or not detected, wherein the target can be a foreign matter needing to be detected. If the target is not detected, the target detection result information obtained based on 102 is null; if the target is detected, the target detection result information obtained based on 102 may include the size and position of the target detection frame.

103. In response to detecting a target from the image to be detected, a region image including the detected target is cut out from the image to be detected.

104. And classifying the region images by using a classification model obtained by pre-training to obtain a classification result of whether the region images comprise the foreign matters.

The foreign matter in the embodiment of the present invention is an object of a preset category, such as hair, filament, etc., and the specific definition and range of the foreign matter in the embodiment of the present invention are not limited.

The classification result may be that the target is the foreign object or the target is not the foreign object (i.e. the target is other objects), and may also include a probability that the target is the foreign object and a probability that the target is not the foreign object, which is not limited in the embodiment of the present invention.

105. Determining whether foreign matter is mixed in the tray based on the classification result.

Wherein the classification result is that the foreign matter is targeted or that the foreign matter is not targeted, whether the foreign matter is mixed in the tray may be determined accordingly based on the classification result.

If the classification result is the probability that the target is the foreign object and the probability that the target is not the foreign object, in step 105, the probability that the target is the foreign object and the probability that the target is not the foreign object may be compared, and the classification result with the larger probability value is selected to determine whether the foreign object is mixed in the tray, specifically, if the probability that the target is the foreign object is larger than the probability that the target is not the foreign object, the foreign object is mixed in the tray; determining that the foreign matter is not mixed in the tray if the probability that the target is the foreign matter is smaller than the probability that the target is not the foreign matter; if the probability that the object is the foreign matter is equal to the probability that the object is not the foreign matter, determining that the foreign matter is mixed in the tray, or determining that the foreign matter is not mixed in the tray, or giving an alarm prompt according to a preset mode.

It should be noted that part or all of the execution subjects 101 to 105 may be an application located in a local terminal, that is, a terminal device of a service provider, or may also be a functional unit such as a plug-in or Software Development Kit (SDK) set in the application located in the local terminal, or may also be a processing engine located in a server on a network side, or may also be a distributed system located on the network side, which is not particularly limited in this embodiment.

It is to be understood that the application may be a native app (native app) installed on the terminal, or may also be a web page program (webApp) of a browser on the terminal, and this embodiment is not particularly limited thereto.

In the packaging stage of the shell-shaped tooth appliance, a batch of shell-shaped tooth appliances are required to be respectively placed in corresponding cells of the tray according to a preset sequence, and then subsequent packaging is carried out. In the process of implementing the present invention, the inventors found through analysis that the probability of mixing foreign substances in the tray stage of the appliance packaging line is relatively high.

In the embodiment, the foreign matters (hairs, filaments and the like) which are difficult to recognize for human eyes in the tray can be effectively recognized based on a deep learning mode, so that the foreign matters possibly mixed into the inner packaging bag in the sorting and packaging stage can be effectively checked, corresponding cleaning can be timely performed, the foreign matters are prevented from being brought into the inner packaging bag of the appliance, and the user experience is effectively improved; in addition, foreign matters which are difficult to identify can be effectively identified through a robust and reliable target detection model and a classification model, the foreign matter identification accuracy is improved, and the method is reliable in a complex production environment; in addition, occupation of operators and quality testing personnel can be avoided, manpower resources are saved, quality testing cost is reduced, quality testing time is shortened, quality testing efficiency is improved, and workload of the operators and the quality testing personnel is reduced, so that production efficiency is improved, and boosting capacity is promoted.

Optionally, in a possible implementation manner of this embodiment, in 101, a top-view photographing manner may be adopted, and an image of the tray on which the appliance is placed is acquired, so as to obtain an image to be detected of the tray on which the appliance is placed, which is not particularly limited in this embodiment.

In concrete realization, can arrange the tray of the ware of correcting in placing into being shaded, adopt suitable illumination, adjust the light and shade and the color of light source, adjust the light ring of industry camera, focus, adopt the mode of overlooking and shooing, carry out image acquisition to the tray of placing the ware of correcting for the definition of the grey scale image on the ware surface of correcting of industry camera collection reaches the best, obtains waiting to detect the image, so that improve follow-up target detection and categorised accuracy.

Optionally, in a possible implementation manner of this embodiment, in 103, in response to detecting a target from the image to be detected, a region image that is larger than the target detection frame and smaller than the image to be detected may be cut out from the image to be detected according to a preset size expansion ratio.

Specifically, an area image which is larger than the target detection frame of the target and smaller than the image to be detected can be cut out from the image to be detected according to a preset size expansion ratio which is 1.0-1.2 times of the size of the target detection frame by taking the central position of the target detection frame as the center.

In the embodiment, according to the preset size expansion proportion of 1.0-1.2 times of the size of the target detection frame, the area image which is larger than the target detection frame of the target and smaller than the image to be detected is cut out from the image to be detected, so that the area image can not only comprise the complete target, but also can enlarge the target to a greater extent, and therefore when subsequent classification and identification are carried out based on the area image, an accurate classification result can be obtained, and the accuracy of foreign matter detection is improved.

Fig. 2 is a schematic diagram of a foreign object detection method according to an embodiment of the present invention. As shown in fig. 2, in the image to be detected 21, the target in the detection frame 201 is a scratch, and is not preset with foreign hair; in the image 22 to be detected, the target in the detection frame 202 is foreign hair. If foreign matter detection is directly performed on the images to be detected 21 and 22, false detection may occur, for example, a scratch in the detection frame 201 is detected as hair. Based on the embodiment of the invention, the corresponding area image is cut out from the target detection frame output by the target detection model, and then a classification model is utilized to classify, so that the probability that the target in the target detection frame is the preset foreign matter (hair) and the probability of other objects except the preset foreign matter can be obtained, and other objects except the preset foreign matter such as scratches, scribbles and the like can be excluded from the foreign matter, thereby correctly distinguishing whether the target in the target detection frame is the preset foreign matter.

Optionally, in a possible implementation manner of this embodiment, after 105, the method may further include: and sending a reminding message or an alarm message of the foreign matters mixed in the tray in response to the determination of the foreign matters mixed in the tray based on the classification result so as to timely make corresponding cleaning, avoid bringing the foreign matters into an inner packaging bag of the appliance and effectively improve the user experience.

Optionally, in a possible implementation manner of this embodiment, the target detection model and the classification model may be implemented based on deep learning, for example, may be implemented by a neural network, and the embodiment of the present invention is not limited thereto.

Optionally, in a possible implementation manner of this embodiment, a training set may also be used to train the target detection model to be trained in advance, so as to obtain the target detection model. Fig. 3 is a schematic flow chart of a target detection model obtained by training in an embodiment of the present invention, as shown in fig. 3.

301. A training set is obtained, the training set including a plurality of sample images.

The multi-sample image comprises a positive sample image and a negative sample image, wherein the positive sample image is used for placing a foreign body in a tray of the appliance, the negative sample image is used for placing a tray of the appliance, the foreign body labeling information is respectively labeled on the positive sample image and the negative sample image, the foreign body labeling information is labeled on the positive sample image, and the foreign body labeling information comprises a foreign body position and size labeling information.

302. And carrying out target detection on the sample images in the training set by using a target detection model to be trained to obtain a target detection result, wherein the target detection result comprises the position and the size of the foreign matter.

Optionally, the target detection result may further include a confidence of the alien material, i.e., a probability of belonging to the alien material.

303. And training the target detection model to be trained according to the target detection result of the sample image and the corresponding foreign matter marking information until a first preset training completion condition is met, and obtaining the target detection model.

Optionally, in a possible implementation manner of this embodiment, the first preset training completion condition may include, but is not limited to: the training times of the target detection model to be trained reach preset times, and/or the difference between the target detection result of the sample image and the corresponding foreign object marking information is smaller than a preset threshold value.

In this embodiment, the target detection model to be trained may be trained in advance through the training set to obtain the target detection model, so as to be used for performing target detection on the image to be detected.

Optionally, in a possible implementation manner of this embodiment, in 301, a top-view photographing manner may be adopted to perform image acquisition on a plurality of trays on which the appliances are placed and which do not have the foreign object, so as to obtain a negative sample image; performing image oversampling on a plurality of trays which are provided with foreign matters and provided with the correcting devices by adopting a overlooking and photographing mode to obtain positive sample images; then, the negative sample image set is subjected to down-sampling, the positive sample image set is subjected to over-sampling, and the training set is obtained based on the down-sampled negative sample image set and the over-sampled positive sample image set.

In this embodiment, the sample image may be collected in a manner similar to the above-described collection of the image to be detected.

Because the probability of mixing foreign matters is extremely low and is less than about 0.1 percent on a real production line, the problem of serious sample data imbalance exists. In this embodiment, the tray image set with the foreign object and the appliance placed therein is oversampled, and the image set without the foreign object is downsampled, so that the number of positive and negative samples is distributed in a balanced manner, and the performance of the trained target detection model and classification model is improved.

Optionally, in another possible implementation manner of this embodiment, in 301, after the training set is obtained through the above implementation manner, data expansion may be performed based on an existing sample image in the training set through a preset data expansion manner to obtain an expanded sample image, and the expanded sample image is added into the training set.

The preset data expansion mode may include, but is not limited to, any one or more of the following: adjusting the brightness of the image, adjusting the contrast of the image, and performing translation, mirroring, rotation, noise addition, mosaic splicing, shielding operation and the like on the image.

In the embodiment, the data expansion is performed based on the existing sample image in the training set by a preset data expansion mode to obtain an expanded sample image, and the expanded sample image is added into the training set, so that the diversity and richness of the expanded sample image are facilitated, and the robustness of the trained target detection model is improved.

Optionally, in a possible implementation manner of this embodiment, the classification model to be trained may also be trained in advance by using a training set, so as to obtain the classification model. Fig. 4 is a schematic flowchart of a process of training to obtain a classification model according to an embodiment of the present invention, and as shown in fig. 4, after 303, the classification model may be trained as follows.

401. And carrying out target detection on the sample images in the training set by using a target detection model.

402. In response to the detection of the target from the sample image, cutting out an area image sample which is larger than the target detection frame of the target and smaller than the sample image from the sample image according to a preset size expansion ratio.

403. And classifying the region image samples by using a classification model to be trained to obtain a classification result of whether the region image samples comprise foreign matters or not.

404. And training the classification model to be trained according to the classification result of the regional image sample and the corresponding foreign matter marking information until a second preset training completion condition is met, and obtaining the classification model.

In the embodiment, in order to reduce the false alarm rate (that is, the influence of factors such as scratches and scribbles needs to be eliminated), the target detection model and the classification model are combined, after the target is detected by using the target detection model, a corresponding area image is cut out according to the size proportion of 1.0-1.2 times of the target detection frame, and the image classification model is trained by combining the foreign object type labeling information, so that the image classification model is used for determining whether the area image sample includes the foreign object classification result.

Optionally, in another possible implementation manner of this embodiment, the second preset training completion condition in 404 may include, but is not limited to: the training times of the classification model to be trained reach preset times, and/or the function value of a weighted cross-entropy loss function (WCE loss) calculated based on the classification result of the region image sample and the corresponding foreign matter labeling information is smaller than a preset threshold value.

Optionally, in a possible implementation manner of this embodiment, the function value of the weighted cross entropy loss function may be calculated based on the following manner:

wherein WCE represents a function value of a cross entropy loss function with weight, w_pA weight value corresponding to a positive sample (i.e., a positive sample image in which a foreign object is present) is a preset constant not less than 0 and not more than 1; y is_iMarking information for the foreign body; p is a radical of_iIs the classification result.

In practical application, the sizes of the foreign matters such as hair and thin threads are small, the proportion of the pixels of the foreign matters such as hair and thin threads occupying the pixels of the area image is small, and the foreign matters such as hair and thin threads are submerged by the background in the gradient feedback of the loss function.

Optionally, in a possible implementation manner of this embodiment, in the process of training the target detection model, an existing target detection framework, for example, a detection library mmdetect based on PyTorch, may be used to refer to a publicly published target detection model (such as an RCNN series or a YOLO series), a migration learning method is adopted to initialize parameters of the target detection model to be trained to parameter values of ImageNet or MSCOCO pre-training, and on the basis, the target detection model to be trained is trained by using a training set. In the process of training the classification model, a publicly published classification model (entering ResNet series) can be used for reference, a transfer learning mode is adopted, parameters of the classification model to be trained are initialized to ImageNet pre-training parameter values, and on the basis, a training set is used for training the classification model.

In the deep learning-based model training, because the model contains many parameters, a large amount of training data (millions of data) is required to achieve the purposes of training convergence and good effect, and such a huge data set cannot be collected and labeled generally. ImageNet and MSCOCO are two public relatively large data sets, researchers do a lot of research work on the two data sets, classification and detection models with good effects are trained on the two data sets, in the embodiment, a migration learning mode is adopted, firstly, parameters of a to-be-trained target detection model are initialized to be pre-trained parameter values on the ImageNet and MSCOCO public data sets, parameters of the to-be-trained classification model are initialized to be pre-trained parameter values of ImageNet, and on the basis, the data sets of a targeted task are used for updating the parameters, so that convergence of the to-be-trained target detection model and the to-be-trained classification detection model can be accelerated, and training effects are improved.

The technical scheme based on deep learning in the embodiment of the invention can be generally realized by the following procedures when in application:

a) acquiring a specimen (appliance or mold), acquiring an electronic representation (i.e., image) of the specimen;

b) marking corresponding foreign matter marking information on the image of the sample to obtain a plurality of positive sample images with foreign matters and negative sample images without foreign matters, forming a data set, and dividing the data set into a training set, a verification set and a test set according to a certain proportion;

c) designing a deep learning model (namely a target detection model to be trained and a classification model to be trained), and training the deep learning model by using a training set; setting evaluation indexes, and screening out an optimal deep learning model through a verification set;

d) and setting evaluation indexes and evaluating the performance of the screened deep learning model test set. If the deployment requirement is not met, returning to the step a) to continuously collect samples to continuously execute the process, or returning to the step c) to modify the design deep learning model to continuously execute the process; if the deployment requirement is met, entering a deployment link of the step e);

e) deploying a deep learning model, acquiring an image to be detected, and applying the deep learning model deployed in d) to carry out quality detection.

f) And performing post-processing on the quality detection result, and performing character feedback or alarm if the tray is confirmed to be mixed with foreign matters.

Optionally, in a possible implementation manner of this embodiment, the sample images in the training set may be updated according to the requirements of the production line, and the iterative training is performed to update the target detection model in a first preset period; and iteratively training to update the classification model in a second preset period. Wherein the first preset period is longer than the second preset period.

Because the number of detected images is large, the probability of foreign matter mixing is extremely low, and the probability of false alarm is increased along with the time lapse, in the embodiment, a stage type model updating iteration method is adopted, sample images in a training set are updated according to the detection requirement of the production line, a target detection model and a classification model are iteratively trained, and the detection requirement of the production line is met. The training time of the classification model is short, and the classification model is updated in a short period, so that the classification model can reflect the real-time production line condition; the training time of the target detection model is longer, and the target detection model is updated in a longer period. The stage type model updating iteration mode is relatively efficient, the foreign matter detection rate is guaranteed, meanwhile, the false detection rate can be effectively reduced, and the detection accuracy rate is improved. In the embodiment, the target detection model is combined with the classification model and the stage model is used for updating the iterative strategy, so that the problem of high false detection rate caused by extremely low foreign matter mixing probability can be effectively solved.

In the embodiment, the image to be detected including the tray with the appliance is obtained, then the target detection model obtained through pre-training is utilized to perform target detection on the image to be detected, the region image including the detected target is cut out from the image to be detected in response to the detection of the target from the image to be detected, then the classification model obtained through pre-training is utilized to classify the region image to obtain the classification result of whether the region image includes the foreign matters or not, and further, whether the foreign matters are mixed in the tray or not is determined based on the classification result, therefore, the foreign matters (hairs, filaments and the like) which are difficult to identify for human eyes in the tray can be effectively identified based on the deep learning mode, so that the foreign matters possibly mixed in the inner packing bag in the sorting and packaging stage can be effectively checked so as to timely perform corresponding cleaning, the foreign bodies are prevented from being brought into the inner packaging bag of the appliance.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

Fig. 5 is a schematic structural diagram of a foreign object detection apparatus according to another embodiment of the present invention, as shown in fig. 5. The foreign object detection apparatus of the present embodiment may include a first acquisition module 501, a detection module 502, a cropping module 503, a classification module 504, and a determination module 505. The first obtaining module 501 is configured to obtain an image to be detected, which includes a tray on which an appliance is placed; a detection module 502, configured to perform target detection on the image to be detected by using a target detection model obtained through pre-training; a cropping module 503, configured to crop, in response to a target being detected from the image to be detected, a region image including the detected target from the image to be detected; a classification module 504, configured to classify the region image by using a classification model obtained through pre-training, so as to obtain a classification result of whether the region image includes a foreign object; a determining module 505, configured to determine whether foreign matter is mixed in the tray based on the classification result.

It should be noted that, part or all of the foreign object detection apparatus provided in this embodiment may be an application located at the local terminal, or may also be a functional unit such as a plug-in or Software Development Kit (SDK) set in the application located at the local terminal, or may also be a search engine located in a server on the network side, or may also be a distributed system located on the network side, which is not particularly limited in this embodiment.

Like this, based on the mode of degree of depth study, can effectively discern the great foreign matter of the degree of difficulty of discerning (hair, filament etc.) in the tray to the foreign matter that inner packing bag probably is sneaked into in the effective investigation letter sorting packing stage, so that in time make corresponding cleanness, avoid bringing the foreign matter into the inner packing bag of correcting the ware.

Optionally, in a possible implementation manner of this embodiment, the first obtaining module 501 is specifically configured to: and adopting a mode of overlooking and photographing to acquire the image of the tray with the appliance, so as to obtain the image to be detected of the tray with the appliance.

Optionally, in a possible implementation manner of this embodiment, the clipping module 503 is specifically configured to: and cutting out an area image which is larger than the target detection frame of the target and smaller than the image to be detected from the image to be detected according to a preset size expansion ratio.

Optionally, in a possible implementation manner of this embodiment, the clipping module 503 is specifically configured to: and cutting out an area image which is larger than the target detection frame of the target and smaller than the image to be detected from the image to be detected according to a preset size expansion proportion of 1.0-1.2 times of the size of the target detection frame by taking the central position of the target detection frame as a center.

Fig. 6 is a schematic structural diagram of a foreign object detection apparatus according to another embodiment of the present invention, and as shown in fig. 6, the foreign object detection apparatus according to the embodiment further includes: a sending module 601, configured to send a reminding message or an alarm message of the foreign object mixed in the tray in response to determining that the foreign object is mixed in the tray based on the classification result.

In addition, referring back to fig. 6, the foreign object detection apparatus of the above embodiment may further include: a second acquisition module 602 and a first training module 603. The second obtaining module 602 is configured to obtain a training set, where the training set includes multiple sample images; the method comprises the following steps that a plurality of sample images are obtained, wherein the sample images comprise a positive sample image and a negative sample image, foreign matters exist in a tray with an appliance, the negative sample image does not exist in the tray with the appliance, foreign matter marking information is marked on the positive sample image, and the foreign matter marking information comprises foreign matter position and size marking information; a first training module 603, configured to perform target detection on the sample images in the training set by using a target detection model to be trained, to obtain a target detection result, where the target detection result includes a position and a size of a foreign object; and training the target detection model to be trained according to the target detection result of the sample image and the corresponding foreign matter marking information until a first preset training completion condition is met, and obtaining the target detection model.

Optionally, in a possible implementation manner of this embodiment, the second obtaining module 602 is specifically configured to: adopting a overlook photographing mode to acquire images of a plurality of trays which do not have foreign matters and are provided with the orthodontic appliances, so as to obtain negative sample images; performing image oversampling on a plurality of trays with foreign matters and provided with the correcting devices by adopting a overlooking and photographing mode to obtain a positive sample image; and performing down-sampling on the negative sample image set, performing over-sampling on the positive sample image set, and obtaining the training set based on the down-sampled negative sample image set and the over-sampled positive sample image set.

In addition, referring back to fig. 6, the foreign object detection apparatus of the above embodiment may further include: an expansion module 604, configured to perform data expansion based on an existing sample image in the training set in a preset data expansion manner to obtain an expanded sample image, and add the expanded sample image to the training set; the preset data expansion mode may include, but is not limited to, any one or more of the following: adjusting the brightness of the image, adjusting the contrast of the image, and performing translation, mirroring, rotation, noise addition, mosaic splicing and shielding operations on the image.

Optionally, in a possible implementation manner of this embodiment, the detection module 502 is further configured to perform target detection on the sample images in the training set by using a target detection model; the cropping module 503 is further configured to crop out, in response to detecting the target from the sample image, a region image sample that is larger than the target detection frame of the target and smaller than the sample image according to a preset size expansion ratio. Accordingly, referring again to fig. 6, the foreign object detection apparatus of the above embodiment may further include: a second training module 605, configured to classify the region image sample by using a classification model to be trained, so as to obtain a classification result of whether the region image sample includes a foreign object; and training the classification model to be trained according to the classification result of the regional image sample and the corresponding foreign matter marking information until a second preset training completion condition is met, and obtaining the classification model.

Optionally, in another possible implementation manner of this embodiment, the second preset training completion condition may include, but is not limited to: and the training times of the classification model to be trained reach preset times, and/or the function value of the cross entropy loss function with the weight calculated based on the classification result of the regional image sample and the corresponding foreign object labeling information is smaller than a preset threshold value.

It should be noted that the method in the embodiment corresponding to fig. 1 to 4 can be implemented by the foreign object detection device provided in the embodiment shown in fig. 5 to 6. For detailed description, reference may be made to relevant contents in the embodiments corresponding to fig. 1 to fig. 4, and details are not described here.

FIG. 7 illustrates a block diagram of an exemplary computer system/server 12 suitable for use in implementing embodiments of the present invention. The computer system/server 12 shown in FIG. 7 is only one example and should not be taken to limit the scope of use or functionality of embodiments of the present invention.

As shown in FIG. 7, computer system/server 12 is in the form of a general purpose computing device. The components of computer system/server 12 may include, but are not limited to: one or more processors or processing units 16, a storage device or system memory 28, and a bus 18 that couples various system components including the system memory 28 and the processing unit 16.

Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Computer system/server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system/server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)30 and/or cache memory 32. The computer system/server 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 7, and commonly referred to as a "hard drive"). Although not shown in FIG. 7, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 by one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 42 generally carry out the functions and/or methodologies of the described embodiments of the invention.

The computer system/server 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), with one or more devices that enable a user to interact with the computer system/server 12, and/or with any devices (e.g., network card, modem, etc.) that enable the computer system/server 12 to communicate with one or more other computing devices. Such communication may be through an input/output (I/O) interface 44. Also, the computer system/server 12 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet) via the network adapter 20. As shown, network adapter 20 communicates with the other modules of computer system/server 12 via bus 18. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the computer system/server 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processing unit 16 executes various functional applications and data processing by executing programs stored in the system memory 28, for example, implementing the image processing method provided in the embodiment corresponding to fig. 1.

Another embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the image processing method provided by the embodiment corresponding to fig. 1.

In particular, any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or page components may be combined or integrated into another system, or some features may be omitted or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A foreign object detection method, comprising:

acquiring an image to be detected comprising a tray with a correction device;

2. The method of claim 1, wherein the obtaining of the image to be inspected including the tray with the appliance placed thereon comprises:

3. The method according to claim 1, wherein said cropping out an image of an area comprising the detected object from the image to be detected comprises:

4. The method according to claim 2, wherein the cutting out the area image larger than the target detection frame of the target and smaller than the image to be detected from the image to be detected according to the preset size expansion ratio comprises:

5. The method of claim 1, further comprising:

6. The method of any one of claims 1 to 5, further comprising:

acquiring a training set, wherein the training set comprises a plurality of sample images; the method comprises the following steps that a plurality of sample images are obtained, wherein the sample images comprise a positive sample image and a negative sample image, foreign matters exist in a tray with an appliance, the negative sample image does not exist in the tray with the appliance, foreign matter marking information is marked on the positive sample image, and the foreign matter marking information comprises foreign matter position and size marking information;

7. The method of claim 6, wherein the obtaining the training set comprises:

8. The method of claim 7, wherein the obtaining the training set further comprises:

9. The method according to claim 6, wherein the training of the target detection model to be trained is performed until a first preset training completion condition is met, and further comprising:

10. The method according to claim 9, wherein the second preset training completion condition comprises:

11. A foreign matter detection device, characterized by comprising:

12. The apparatus of claim 11, wherein the first obtaining module is specifically configured to:

13. The apparatus of claim 11, wherein the cropping module is specifically configured to:

14. The apparatus of claim 12, wherein the cropping module is specifically configured to:

15. The apparatus of claim 11, further comprising:

16. The apparatus of any one of claims 11 to 15, further comprising:

17. The apparatus of claim 16, wherein the second obtaining module is specifically configured to:

18. The apparatus of claim 17, further comprising:

19. The apparatus of claim 16,

the detection module is further configured to perform target detection on the sample images in the training set by using the target detection model;

the device further comprises:

20. The apparatus of claim 19, wherein the second preset training completion condition comprises:

21. An apparatus, characterized in that the apparatus comprises:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement a method as claimed in any one of claims 1 to 10.

22. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, carries out the method of any one of claims 1 to 10.