CN115439392A

CN115439392A - Object abnormal position detection method, device, electronic device and storage medium

Info

Publication number: CN115439392A
Application number: CN202110612562.1A
Authority: CN
Inventors: 李秀阳
Original assignee: Wuhan TCL Group Industrial Research Institute Co Ltd
Current assignee: Wuhan TCL Group Industrial Research Institute Co Ltd
Priority date: 2021-06-02
Filing date: 2021-06-02
Publication date: 2022-12-06

Abstract

The application provides a method and a device for detecting an abnormal position of an object, electronic equipment and a computer-readable storage medium. The object abnormal position detection method includes: acquiring an original image of a target object; calling a preset target generator to generate a restored image of the target object according to the original image; and determining abnormal information of the original image according to the restored image and the original image, wherein the abnormal information is used for indicating the abnormal position of the target object. The problem that a target detection model for detecting the abnormal position is poor in generalization capability is avoided, the time for marking the abnormal position of the sample is shortened, and the training time cost is reduced.

Description

Object abnormal position detection method, device, electronic device and storage medium

Technical Field

The present application relates to the field of computer vision technologies, and in particular, to a method and an apparatus for detecting an abnormal position of an object, an electronic device, and a computer-readable storage medium.

Background

With the rapid development of computer vision technology, computer vision technology has been widely applied in many fields such as security monitoring, medical health, industry, etc. In the detection of abnormal positions of objects, for example, defect detection of an industrial PCB, a target detection model, for example, a model based on the fast-RCNN series and the YOLO series, is trained after marking the abnormal positions of samples, and then the trained target detection model is used to implement a target detection task.

However, existing object detection models, such as those based on the fast-RCNN series and the YOLO series, have poor generalization ability; moreover, because a detection model with a good effect can be trained only by relying on a large number of labeled samples, the abnormal positions of the samples need to be labeled in the training process, and the labeling of a large number of abnormal positions of the samples needs a large time cost.

Disclosure of Invention

The application provides a method and a device for detecting an abnormal position of an object, electronic equipment and a computer-readable storage medium, and aims to solve the problems that an existing target detection model for detecting the abnormal position is poor in generalization capability and high in training time cost.

In a first aspect, the present application provides a method for detecting an abnormal position of an object, the method including:

acquiring an original image of a target object;

calling a preset target generator to generate a restored image of the target object according to the original image;

and determining abnormal information of the original image according to the restored image and the original image, wherein the abnormal information is used for indicating the abnormal position of the target object.

In a possible implementation manner of the present application, the target generator is obtained based on a preset generative countermeasure network training, where the generative countermeasure network includes a generator and a discriminator, and the target generator is obtained by the following training steps:

acquiring a training data set, wherein the training data set comprises at least a first training sample and a second training sample;

adjusting network parameters of the generator by using the first training sample, and taking the generator as a pre-training generator when a preset first training stopping condition is met, wherein the first training sample comprises defective real images of a plurality of sample objects;

and adjusting the network parameters of the pre-training generator and the network parameters of the discriminator by using the second training sample, and taking the pre-training generator as the target generator when the second training sample meets a preset second training stopping condition, wherein the second training sample comprises defective real images of a plurality of sample objects.

In one possible implementation manner of the present application, in the process of adjusting the network parameters of the generator by using the first training sample, the training loss of the generative confrontation network is the first generative loss of the generator; in the process of adjusting the network parameters of the pre-training generator and the network parameters of the discriminator by using a second training sample, the training loss of the generative confrontation network comprises a second generation loss of the generator and a confrontation loss of the discriminator;

when the network parameters of the generator are adjusted by using the first training sample until a preset first training stopping condition is met, the generator is used as a pre-training generator, and the method comprises the following steps:

determining the first generation loss from a predicted repair image generated by the generator based on defective real images in the first training sample;

adjusting the network parameters of the generator according to the first generation loss until the generator is in accordance with a preset first training stopping condition, and taking the generator as a pre-training generator;

the adjusting the network parameters of the pre-training generator and the network parameters of the discriminator by using a second training sample until a preset second training stopping condition is met, and using the pre-training generator as the target generator includes:

determining the second generation loss and the counter loss from a predicted repair image generated by the pre-training generator based on a defective real image in the second training sample;

and adjusting the network parameters of the pre-training generator and the network parameters of the discriminator according to the second generation loss and the antagonistic loss until the pre-training generator is taken as the target generator when a preset second training stopping condition is met.

In one possible implementation manner of the present application, the determining the first generation loss according to the predicted repair image generated by the generator based on the defective real image in the first training sample includes:

calling the generator to generate a corresponding predicted repair image according to the defective real image;

and determining the first generation loss according to the predicted repairing image, the defect-free real image and a preset first loss function.

In one possible implementation manner of the present application, the determining the second generation loss and the counter loss according to the predicted repair image generated by the pre-training generator based on the defective real image in the second training sample includes:

calling the pre-training generator to generate a corresponding prediction repairing image according to the defective real image;

calling the discriminator to determine a discrimination result of the predicted repaired image, wherein the discrimination result is used for indicating whether the predicted repaired image is a real image;

determining the second generation loss according to the predicted repaired image, the defect-free real image and a preset second loss function;

and determining the countermeasure loss according to the judgment result and a preset third loss function.

In one possible implementation manner of the present application, the generator of the generative confrontation network is a UNet network, and the arbiter of the generative confrontation network is a convolutional neural network structure.

In a possible implementation manner of the present application, the target object is a target PCB, the original image is a defect image of the target PCB, and the invoking a preset target generator to generate a restored image of the target object according to the original image includes:

calling the target generator to generate a restored image of the target PCB according to the defect image;

determining abnormal information of the original image according to the restored image and the original image, wherein the determining abnormal information of the original image comprises the following steps:

and according to the restored image and the defect image, determining abnormal information of the defect image, wherein the abnormal information is used for indicating the abnormal position of the target PCB.

In one possible implementation manner of the present application, the determining, according to the restored image and the original image, the abnormal information of the original image includes:

determining the abnormal position of the target object according to the restored image and the original image;

according to the abnormal position, determining abnormal information of the original image, wherein the abnormal information comprises at least one of the following information: coordinate information of the abnormal position, an area indication frame of the abnormal position, and an area image of the abnormal position.

In one possible implementation manner of the present application, the determining an abnormal position of the target object according to the restored image and the original image includes:

comparing the restored image with the original image to obtain difference pixel points of the original image;

and taking the position of the difference pixel point as the abnormal position of the target object.

In a second aspect, the present application provides an object abnormal position detecting device including:

an acquisition unit configured to acquire an original image of a target object;

the generating unit is used for calling a preset target generator to generate a restored image of the target object according to the original image;

and a determining unit configured to determine, based on the restored image and the original image, abnormality information of the original image, where the abnormality information is used to indicate an abnormal position of the target object.

In a possible implementation manner of the present application, the target generator is obtained based on a preset generative confrontation network training, the generative confrontation network includes a generator and an arbiter, the apparatus for detecting an abnormal position of an object further includes a training unit, and the training unit is specifically configured to:

and adjusting the network parameters of the pre-training generator and the network parameters of the discriminator by using the second training sample, and taking the pre-training generator as the target generator when a preset second training stopping condition is met, wherein the second training sample comprises defective real images of a plurality of sample objects.

In one possible implementation manner of the present application, in the process of adjusting the network parameters of the generator by using the first training sample, the training loss of the generative confrontation network is the first generative loss of the generator; in the process of adjusting the network parameters of the pre-training generator and the network parameters of the discriminator by using a second training sample, the training loss of the generative confrontation network includes a second generation loss of the generator and a confrontation loss of the discriminator, and the training unit is specifically configured to:

determining the first generation loss from a predicted restoration image generated by the generator based on a defective real image in the first training sample;

determining the second generation loss and the countermeasure loss from a predicted repair image generated by the pre-training generator based on defective real images in the second training sample;

and adjusting the network parameters of the pre-training generator and the network parameters of the discriminator according to the second generation loss and the confrontation loss until the pre-training generator is taken as the target generator when the pre-training generator meets a preset second training stopping condition.

In one possible implementation manner of the present application, the first training sample includes at least one pair of sample images, each pair of sample images includes a defective real image and a non-defective real image of a sample object, the non-defective real image is used as a label of the defective real image, and the training unit is specifically configured to:

In one possible implementation manner of the present application, the second training sample includes at least one pair of sample images, each pair of sample images includes a defective real image and a non-defective real image of a sample object, the non-defective real image is used as a label of the defective real image, and the training unit is specifically configured to:

determining the second generation loss according to the predicted and repaired image, the non-defective real image and a preset second loss function;

In one possible implementation manner of the present application, the generator of the generative confrontation network is a UNet network, and the discriminator of the generative confrontation network is a convolutional neural network structure.

In a possible implementation manner of the present application, the target object is a target PCB, the original image is a defect image of the target PCB, and the generating unit is specifically configured to:

in a possible implementation manner of the present application, the determining unit is specifically configured to:

and determining abnormal information of the defect image according to the restored image and the defect image, wherein the abnormal information is used for indicating the abnormal position of the target PCB.

In a third aspect, the present application further provides an electronic device, where the electronic device includes a processor and a memory, where the memory stores a computer program, and the processor executes any one of the steps in the method for detecting an abnormal position of an object provided in the present application when calling the computer program in the memory.

In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, the computer program being loaded by a processor to execute the steps in the method for detecting an abnormal position of an object.

The method comprises the steps of restoring an original image of a target object by calling a preset target generator to generate a restored image after restoring abnormal conditions of damage, defect and the like of the target object; determining abnormal information for indicating the abnormal position of the target object through the original image and the restored image; on one hand, the abnormal position of the object is detected without depending on a target detection model trained by a large number of samples marked with abnormal positions, so that the problem of poor generalization capability of the target detection model for detecting the abnormal position is avoided. On the other hand, the target generator does not need to be trained aiming at the sample abnormal position, and the sample abnormal position does not need to be labeled, so that the time for labeling the sample abnormal position is reduced, and the training time cost is reduced.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic view of a scene of an object abnormal position detection system provided in an embodiment of the present application;

fig. 2 is a schematic flowchart of a method for detecting an abnormal position of an object according to an embodiment of the present application;

FIG. 3 is a schematic diagram illustrating a training process of a generative confrontation network provided in an embodiment of the present application;

FIG. 4 is a schematic flow chart diagram illustrating one embodiment of step 203 provided in an embodiment of the present application;

FIG. 5 is a schematic diagram illustrating a PCB defect detection process according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an embodiment of an object abnormal position detection apparatus provided in an embodiment of the present application;

fig. 7 is a schematic structural diagram of an embodiment of an electronic device provided in the embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In the description of the embodiments of the present application, it should be understood that the terms "first", "second", and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, features defined as "first", "second", may explicitly or implicitly include one or more of the described features. In the description of the embodiments of the present application, "a plurality" means two or more unless specifically defined otherwise.

The following description is presented to enable any person skilled in the art to make and use the application. In the following description, details are set forth for the purpose of explanation. It will be apparent to one of ordinary skill in the art that the present application may be practiced without these specific details. In other instances, well-known processes have not been described in detail so as not to obscure the description of the embodiments of the present application with unnecessary detail. Thus, the present application is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed in the embodiments herein.

An execution main body of the method for detecting an abnormal object position in the embodiment of the present application may be the device for detecting an abnormal object position provided in the embodiment of the present application, or different types of electronic devices such as a server device, a physical host, or a User Equipment (UE) integrated with the device for detecting an abnormal object position, where the device for detecting an abnormal object position may be implemented in a hardware or software manner, and the UE may specifically be a terminal device such as a smart phone, a tablet computer, a notebook computer, a palm computer, a desktop computer, or a Personal Digital Assistant (PDA).

The electronic equipment can adopt a working mode of independent operation or a working mode of equipment cluster, and by applying the method for detecting the abnormal position of the object provided by the embodiment of the application, on one hand, the abnormal position of the object is detected without depending on a target detection model trained by a large number of samples marked with the abnormal position, so that the problem of poor generalization capability of the target detection model for detecting the abnormal position is avoided. On the other hand, the abnormal positions of the samples do not need to be marked, so that the time for marking the abnormal positions of the samples is reduced, and the training time cost is reduced.

Referring to fig. 1, fig. 1 is a schematic view of a scene of an object abnormal position detection system according to an embodiment of the present disclosure. The system for detecting the abnormal position of the object may include an electronic device 100, and an abnormal position detecting device of the object is integrated in the electronic device 100. For example, the electronic device may acquire an original image of the target object; calling a preset target generator to generate a restored image of the target object according to the original image; and determining abnormal information of the original image according to the restored image and the original image, wherein the abnormal information is used for indicating the abnormal position of the target object.

In addition, as shown in fig. 1, the system for detecting an abnormal position of an object may further include a memory 200 for storing data, such as image data.

It should be noted that the scene schematic diagram of the object abnormal position detection system shown in fig. 1 is only an example, and the object abnormal position detection system and the scene described in the embodiment of the present application are for more clearly illustrating the technical solution of the embodiment of the present application, and do not form a limitation on the technical solution provided in the embodiment of the present application.

In the following, an object abnormal position detecting method provided in an embodiment of the present application is described, where an electronic device is used as an executing subject, and for simplicity and convenience of description, the executing subject will be omitted in subsequent embodiments of the method, and the object abnormal position detecting method includes: acquiring an original image of a target object; calling a preset target generator to generate a restored image of the target object according to the original image; and determining abnormal information of the original image according to the restored image and the original image, wherein the abnormal information is used for indicating the abnormal position of the target object.

Referring to fig. 2, fig. 2 is a schematic flowchart of a method for detecting an abnormal position of an object according to an embodiment of the present application. It should be noted that although the embodiments of the present application show a logical order in the flowcharts, in some cases, the steps shown or described may be performed in an order different from that described herein. The method for detecting the abnormal position of the object comprises steps 201 to 203, wherein:

201. an original image of the target object is acquired.

The target object is an object to be subjected to abnormal position detection, and in an actual service scene, the target object may be an object having abnormality such as damage and deformity, for example, the target object may be a defective PCB, a damaged product in industrial production, or the like.

The original image refers to an image of the target object before restoration, and in an actual service scene, the original image may be specifically an image of the target object when the target object is damaged or incomplete.

Specifically, in practical application, the electronic device to which the method for detecting an abnormal position of an object provided by the embodiment of the present application is applied may directly include a camera (the camera is mainly used for acquiring an image of a target object) on hardware, and locally store the image captured by the camera, and may directly read the image inside the electronic device; or the electronic device can also establish network connection with the camera and acquire the image obtained by the camera on line from the camera according to the network connection; alternatively, the electronic device may also read the image captured by the camera from a related storage medium storing the image captured by the camera, and the specific acquisition mode is not limited herein.

The camera can shoot images according to a preset shooting mode, for example, shooting height, shooting direction or shooting distance can be set, the specific shooting mode can be adjusted according to the camera, and the camera is not limited specifically. The multi-frame images shot by the camera can form a video through a time line.

202. And calling a preset target generator to generate a restored image of the target object according to the original image.

The restored image is an image which is restored from the original image and in which the degree of abnormality such as damage or chipping of the target object is smaller than the degree of abnormality such as damage or chipping of the target object in the original image.

The target generator is used for repairing according to the original image of the target object and generating the image of which the degree that the target object does not have abnormal conditions such as damage and deformity is smaller than the degree that the target object in the original image has abnormal conditions such as damage and deformity. The target generator is trained, for example, by a self-encoder or a generative confrontation network. The training of the target generator will be described in detail later, and will not be described here for the sake of simplicity.

203. And determining abnormal information of the original image according to the restored image and the original image.

The abnormal position is a position where an abnormality such as a damage or a defect exists in the target object. For example, the target object is a PCB, and the abnormal position of the PCB is a position where the PCB has a defect.

The abnormality information is used to indicate an abnormal position of the target object.

Illustratively, step 203 may specifically include the following steps 2031 to 2032:

2031. and determining the abnormal position of the target object according to the restored image and the original image.

Specifically, the pixel points in the original image that are different from the restored image may be used as the abnormal positions of the target object by comparing the difference between the pixel points in the original image and the pixel points at the corresponding positions in the restored image.

2032. And determining the abnormal information of the original image according to the abnormal position.

The expression form of the abnormal information can be various, and the abnormal information exemplarily comprises at least one of the following information: coordinate information of the abnormal position, an area indication frame of the abnormal position, and an area image of the abnormal position. The following are exemplified separately:

(1) In some embodiments, the anomaly information refers to coordinate information of the location of the anomaly. Step 2032 may specifically include: and acquiring the coordinates of the abnormal position of the target object in the original image as the abnormal information of the original image. The coordinate information of the abnormal position specifically refers to a coordinate of the abnormal position of the target object in the original image.

(2) In some embodiments, the anomaly information refers to an area indication box of the anomaly location. Step 2032 may specifically include: and acquiring a minimum rectangular surrounding frame or an edge surrounding frame of the abnormal position of the target object in the original image as the abnormal information of the original image. The area indication frame of the abnormal position may specifically be a minimum rectangular enclosure frame of the abnormal position of the target object in the original image, or may also be an edge enclosure frame of the abnormal position of the target object in the original image, and specifically, the shape and size of the area indication frame of the abnormal position may be adjusted according to the actual service scene requirement, which is only an example, and is not limited thereto.

(3) In some embodiments, the anomaly information refers to an area image of the anomaly location. Step 2032 may specifically include: and intercepting an image of the abnormal position area of the target object in the original image as abnormal information of the original image. The area image of the abnormal position refers to an image of the abnormal position area of the target object captured in the original image.

In the embodiment of the application, a preset target generator is called to restore an original image of a target object, and a restored image which is obtained by restoring abnormal conditions of damage, deformity and the like of the target object is generated; determining abnormal information for indicating the abnormal position of the target object through the original image and the restored image; on one hand, the abnormal position of the object is detected without depending on a target detection model trained by a large number of samples marked with abnormal positions, so that the problem of poor generalization capability of the target detection model for detecting the abnormal position is avoided. On the other hand, the target generator does not need to be trained aiming at the sample abnormal position, and the sample abnormal position does not need to be labeled, so that the time for labeling the sample abnormal position is reduced, and the training time cost is reduced.

The target generator in step 202 is obtained by training, and for example, the target generator is obtained by training through a preset Generative Adaptive Networks (GAN). GAN is a deep neural network architecture, which includes a generator network and a discriminator, the input of the generator is a set of random variables (denoted as z), and the output is generated data (i.e., false data, denoted as G (z)); the discriminator is responsible for scoring the generated data or the real data (marked as y), and the output is the confidence coefficient of the generated data or the real data which is judged as the real data. The generator produces false data and attempts to fool the arbiter; the discriminator performs authenticity discrimination on the generated data and tries to correctly recognize all the false data. During the training iteration, the generator and the discriminator continue to evolve and compete until an equilibrium state is reached, i.e. the training ends until the discriminator determines the image generated by the generator as a real image. When the training reaches equilibrium, G (z) is the final generated image for an input noise z.

In some embodiments, the generator of the predetermined generative countermeasure network is a UNet network, and the discriminator is a convolutional neural network structure.

The following describes how to train a preset generative confrontation network to obtain a target generator.

Generally, the generator and the discriminator of the generative countermeasure network are jointly trained at the same stage, but the training is difficult because an abnormal image of an object is directly input, and an abnormal image of the object corresponding to the abnormal image of the object is generated and output. The difficulty in training the generator is greatly increased if the generator and the arbiter are updated from scratch at the same time.

Therefore, in the embodiment of the application, the generative confrontation network is trained by two stages, wherein the first stage is mainly to preferentially train the generator, and the second stage is started when the generator is close to the optimal state; the second phase trains the generator and the arbiter simultaneously. At this time, as shown in fig. 3, the training process of the preset generative confrontation network includes the following steps 301 to 303, wherein:

301. a training data set is acquired.

Wherein the training data set comprises at least two training samples. For example, the training data set comprises at least a first training sample and a second training sample.

The first training sample in the training dataset is used for training optimization of the generator in the first stage, and the second training sample is used for simultaneous training optimization of the generator and the arbiter in the second stage.

The first training sample includes defective real images of a plurality of sample objects. The second training sample also includes defective real images of the plurality of sample objects.

302. And adjusting the network parameters of the generator by using the first training sample until the generator meets a preset first training stopping condition, and taking the generator as a pre-training generator.

Wherein the first training sample comprises defective real images of a plurality of sample objects.

For example, the parameters of the discriminator may be fixed and the generator may be invoked to generate a predicted inpainting image according to the defective real image in the first training sample, with the goal of minimizing the training loss of the generator in the first stage; determining the training loss of the generator in the first stage according to the predicted restoration image and a preset loss function in the first stage; and adjusting the network parameters of the generator according to the training loss of the generator in the first stage until the network parameters meet a preset first training stopping condition, and taking the generator as a pre-training generator.

The preset first training stopping condition can be set according to actual requirements. For example, when the training loss value of the first-stage generation type countermeasure network is smaller than a preset value, or the training loss value of the first-stage generation type countermeasure network does not change basically, that is, the difference value of the training loss values corresponding to adjacent training times is smaller than the preset value; or when the number of iterations of the training of the generator of the generative countermeasure network in the first stage reaches the maximum number of iterations.

303. And adjusting the network parameters of the pre-training generator and the network parameters of the discriminator by using the second training sample until the pre-training generator is in accordance with a preset second training stopping condition, and taking the pre-training generator as the target generator.

Wherein the second training sample comprises defective real images of a plurality of sample objects.

Illustratively, the parameters of the arbiter or the parameters of the pre-training generator may be fixed and unchanged, and the arbiter and the pre-training generator may be iteratively trained alternately. For example:

1. firstly, fixing parameters of a pre-trainer, calling a pre-training generator to generate a prediction repairing image based on a defective image in a second training sample; calling a discriminator to discriminate the predicted repaired image to obtain a discrimination result of the predicted repaired image; determining the training loss of the discriminator at the second stage according to the discrimination result of the predicted repaired image and the loss function set for the discriminator at the second stage; and adjusting the network parameters of the discriminator according to the training loss of the discriminator in the second stage until the discriminator can discriminate the predicted repaired image as a fake image, thereby completing the training of the discriminator. Wherein the loss function is set to determine whether the output of the discriminator is true.

2. After the training of the discriminator is finished, fixing the parameters of the discriminator, and calling a pre-training generator to generate a prediction repairing image based on the defective image in the second training sample; determining the training loss of the pre-training generator in the second stage according to the predicted repairing image and the preset loss function in the second stage; adjusting network parameters of the pre-training generator according to the training loss of the pre-training generator in the second stage until the predictor can judge the predicted repaired image as a real image;

3. and after the training of the pre-training generator is finished, fixing the parameters of the pre-training generator again, training the discriminator, and continuously repeating the process until the pre-training generator meets a preset second training stopping condition, wherein the pre-training generator is used as the target generator.

Wherein, the preset second training stopping condition can be set according to the actual requirement. For example, when the training loss value of the second-stage generation type countermeasure network is smaller than the preset value, or when the training loss value of the second-stage generation type countermeasure network does not change basically, that is, the difference value of the training loss values corresponding to the adjacent training times is smaller than the preset value; or when the number of iterations of the training of the generator of the generative confrontation network reaches the maximum number of iterations in the second stage.

In the embodiment of the application, the generative confrontation network is trained by being divided into two stages, wherein the first stage is mainly to train a generator preferentially, and the second stage is entered when the generator converges; and in the second stage, the generator and the discriminator are trained simultaneously, so that the training time of the whole confrontation type generation network can be shortened on the basis of ensuring the generation effect of the generator, and the training difficulty of the generator is reduced.

The training process of the generative confrontation network in the first stage and the training process in the second stage will be described in detail below.

The training process of the first stage.

In some embodiments of the present application, in the process of adjusting the network parameters of the generator using the first training sample (referred to as a first stage for short), the training loss of the generative confrontation network is the first generative loss of the generator. Step 302 may specifically include the following steps 3021 to 3022:

3021. determining the first generation loss from a predicted restoration image generated by the generator based on defective real images in the first training sample.

Wherein the first generation loss is a training loss of the generator in the first phase.

3022. And adjusting the network parameters of the generator according to the first generation loss until the generator meets a preset first training stopping condition, and taking the generator as a pre-training generator.

In step 3021, the "determining the first generation loss according to the predicted restoration image generated by the generator based on the defective real image in the first training sample" may be performed in various ways, and includes, for example:

1) In some embodiments, the training loss of the generator in the first stage can be determined by a preset loss function of the first stage as shown in the following equation (1).

In formula (1), D (G (z)) represents the discrimination result of the discriminator at the first stage based on the predicted repaired image G (z) generated by the generator,

representing a first generation penalty of the generator.

The defective real image is a real image when the sample object has an abnormality.

At this time, in step 3021, "determining the first generation loss from the predicted inpainting image generated by the generator based on the defective real image in the first training sample" specifically includes: inputting the defective real image (noted as z) in the first training sample into a generative countermeasure network to invoke the generator to generate a predicted repairing image G (z) according to the defective real image in the first training sample; calling the discriminator to discriminate according to the predicted repaired image to obtain a discrimination result of the predicted repaired image; and determining the first generation loss according to the judgment result of the predicted repaired image and a preset loss function of a first stage.

Wherein the discrimination result of the predicted repair image is used to instruct the generator to discriminate whether the predicted repair image generated from the defective real image is a real image or a fake image.

The preset loss function in the first stage is used to indicate a training loss of the generator in the first stage, and the preset loss function in the first stage is only an example, and may be specifically set according to an actual requirement, and is not limited thereto.

2) In some embodiments, the training penalty of the generator in the first stage is set as the penalty between the predicted fix image generated by the generator from a defective real image and a non-defective real image. At this time, the first generation loss refers to a loss between the predicted repair image generated by the generator from the defective real image in the first stage and the non-defective real image. The defective real image is a real image when the sample object is abnormal, and the non-defective real image is a real image when the sample object is not abnormal.

At this time, the first training sample includes at least one pair of sample images, each pair of the sample images including a defective real image (denoted as x) and a non-defective real image (denoted as y) of a sample object, the non-defective real image serving as a label of the defective real image. In step 3021, the "determining the first generation loss from the predicted repair image generated by the generator based on the defective real image in the first training sample" specifically includes: calling the generator to generate a corresponding prediction repair image according to the defective real image; and determining the first generation loss according to the predicted repairing image, the defect-free real image and a preset first loss function.

Specifically, inputting a defective real image x of the sample object into a generative countermeasure network to invoke a generator to generate a predicted repair image G (x) of the sample object from the defective real image x; then, the training loss of the generator in the first stage is determined according to a preset first loss function, the generated predicted repair image G (x), and the defect-free real image y.

In a first stage of training the generator alone, the generator is trained by determining the generation loss of the generator using the non-defective real image y and a predicted repair image G (x) generated from the defective real image x; the original non-defective real image y is potentially used as supervision information, so that the generator can learn the capability of generating a non-defective image better, and the capability of the generator for generating a non-defective image can be improved more effectively, so that the restored image generated by the target generator is more accurate.

For example, the sample object is a PCB, the defective real image is a real image x of the PCB captured when there is an abnormal condition such as a breakage or a defect, and the non-defective real image is a real image y of the PCB captured when there is no abnormal condition such as a breakage or a defect. Inputting a real image x into the generative countermeasure network, and generating a predicted repairing image G (x) of the PCB according to the real image x by a generator; then, the training loss of the generator in the first stage can be determined according to the nondefective real image y of the PCB, the predicted repaired image G (x) and a preset first loss function. Wherein the preset first loss function is shown as the following formula (2),

L' _G1 ＝∑[||y _i -G(x _i )||]formula (2)

In the formula (2), y _i The ith pixel point G (x) in the defect-free real image y of the PCB _i ) The ith pixel point L 'in the predicted repair image G (x) of the PCB is represented' _G1 Representing a first generation loss of the generator.

The first loss function is used to indicate a training loss of the generator in the first stage, and the first loss function is only an example, and may be specifically set according to an actual requirement, which is not limited to this.

And (II) a training process of a second stage.

In some embodiments of the present application, in the adjusting process of the network parameters of the pre-training generator and the network parameters of the discriminator by using the second training sample (referred to as the second stage), the training loss of the generative confrontation network includes the second generation loss of the generator and the confrontation loss of the discriminator. Step 303 may specifically include the following steps 3031 to 3032:

3031. determining the second generation loss and the counter loss from a predicted repair image generated by the pre-training generator based on defective real images in the second training sample.

Wherein the second generation loss is a training loss of the generator in the second stage, and the counter loss is a training loss of the arbiter in the second stage.

In some embodiments, the training penalty of the generator in the second stage is set as the penalty between an image generated by the generator from a defective real image and a non-defective real image. At this time, the second generation loss refers to a loss between the image generated by the generator from the defective real image at the second stage and the non-defective real image. The defective real image is a real image when the sample object is abnormal, and the non-defective real image is a real image when the sample object is not abnormal.

In this case, the second training sample includes at least one pair of sample images, each pair of sample images includes a defective real image and a non-defective real image of the sample object, and step 3031 may specifically include the following steps a to D:

A. and calling the pre-training generator to generate a corresponding prediction repairing image according to the defective real image.

Specifically, a defective real image x of the sample object is input into the generative countermeasure network to invoke the pre-training generator to generate a predicted repair image G (x) of the sample object from the defective real image x.

B. And calling the discriminator to determine the discrimination result of the predicted repaired image.

And the judgment result of the predicted repaired image is used for indicating whether the predicted repaired image is a real image or not.

Specifically, the discriminator discriminates the predicted restored image G (x) obtained in the above step a, and outputs the discrimination result of the predicted restored image G (x).

C. And determining the second generation loss according to the predicted repaired image, the defect-free real image and a preset second loss function.

Specifically, the training loss of the pre-training generator in the second stage is determined according to a preset second loss function, the generated predicted repairing image G (x) and the defect-free real image y.

For example, the sample object is a PCB, the defective real image is a real image x of the PCB captured when there is an abnormal condition such as a breakage or a defect, and the non-defective real image is a real image y of the PCB captured when there is no abnormal condition such as a breakage or a defect. Inputting a real image x into the generative confrontation network, and generating a predicted repairing image G (x) of the PCB according to the real image x by a pre-training generator; then, the predicted repair image G (x), and the preset second loss function may be based on the defect-free real image y of the PCB panel. Wherein the preset second loss function is shown in the following formula (3),

L' _G2 ＝∑[||y _i -G(x _i )||]formula (3)

In the formula (3), y _i The ith pixel point in the nondefective real image y of the PCB, G (x) _i ) Ith pixel point L 'in predicted repair image G (x) representing PCB' _G2 Representing a second generation penalty of the pre-training generator.

D. And determining the countermeasure loss according to the judgment result of the predicted repaired image and a preset third loss function.

The discriminator discriminates the predicted repairing image G (x) and outputs the discrimination result of the predicted repairing image G (x); then, according to the judgment result of the predicted repairing image G (x), the probability that the generated image of the pre-training generator is judged to be a real image by the discriminator is counted; and then determining the training loss of the discriminator at the second stage according to the probability that the generated image of the pre-training generator is wrongly discriminated as a real image by the discriminator and a preset third loss function. Wherein, the preset third loss function is shown as the following formula (4),

L _D = D (G (x), y) formula (4)

In the formula (4), D (G (x), y) represents the probability that the discriminator erroneously discriminates the predicted restored image G (x) as a true image, and L _D Indicating the loss of confrontation of the arbiter.

Further, the discriminator also discriminates the defective real image to obtain the discrimination result of the defective real image. Wherein, the discrimination result of the defective real image is used to indicate whether the non-defective real image is a real image. At this time, the training loss of the discriminator at the second stage is set to include the probability that the discriminator erroneously discriminates the predicted repair image G (x) generated by the pre-training generator as a real image and the probability that the defective real image x is erroneously discriminated as a non-defective real image y. For example, the training loss of the arbiter in the second stage may be further determined according to a third loss function as shown in the following equation (5):

L _D = D (G (x), y) + D (x, y) equation (5)

In the formula (5), D (G (x), y) represents the probability that the discriminator misjudges the predicted repair image G (x) as a real image, D (x, y) represents the probability that the discriminator misjudges the defective real image x as a non-defective real image y, and L (x, y) represents the probability that the discriminator misjudges the defective real image x as a non-defective real image y _D Indicating the loss of confrontation of the arbiter.

The second loss function is used to indicate a training loss of the pre-training generator in the second stage, and the second loss function is only an example, and may be specifically set according to an actual requirement, and is not limited thereto.

The third loss function is used to indicate a training loss of the discriminator in the second stage, and the preset third loss function is only an example, and may be specifically set according to an actual requirement, and is not limited thereto.

3032. And adjusting the network parameters of the pre-training generator and the network parameters of the discriminator according to the second generation loss and the confrontation loss until the pre-training generator is taken as the target generator when the pre-training generator meets a preset second training stopping condition.

For example, the second generation loss and the confrontation loss may be added according to a certain weight ratio to serve as the training loss of the generative confrontation network at the second stage; and adjusting the network parameters of the pre-training generator and the network parameters of the discriminator according to the training loss of the generative confrontation network in the second stage until the generated confrontation network converges. At this time, a generator in the converged generation countermeasure network may be adopted as the target generator in step 202 for generating a restored image of the target image from the original image.

In a second stage of training the pre-training generator and the discriminator at the same time, determining the generation loss of the pre-training generator by adopting the non-defective real image y and the predicted repair image G (x) generated according to the defective real image x to train the pre-training generator; the original non-defective real image y is potentially used as the supervision information, so that the pre-training generator can learn the capability of generating non-defective images better, and the capability of the pre-training generator for generating non-defective images can be improved more effectively, so that the restored images generated by the target generator are more accurate.

In step 2031, there are various ways to determine the abnormal position of the target object based on the restored image and the original image, and as shown in fig. 4, the method may include the following steps 401 to 402:

401. and comparing the restored image with the original image to obtain the difference pixel points in the original image.

Specifically, the difference between the positions of the same pixel points in the original image and the restored image may be determined first to determine the difference pixel points in the original image.

For example, interpolation calculation may be performed on the restored image and the original image to determine the difference between the positions of the same pixel point in the restored image and the position of the same pixel point in the original image, so as to determine the difference pixel point in the original image.

In some embodiments, the difference pixel points are pixel points in the original image that have a difference from pixel points in a corresponding position of the restored image. For example, if there is a difference between the ith pixel point with coordinates (x, y) in the original image and the jth pixel point with coordinates (x, y) in the restored image, the ith pixel point in the original image is a difference pixel point.

In some embodiments, the difference pixel point is a pixel point with a larger difference (for example, the difference discrimination score is larger than a preset score threshold) between the pixel points in the original image and the corresponding positions of the restored image. For example, if the difference determination score between the i1 th pixel point with coordinates (x 1, y 1) in the original image and the j1 th pixel point with coordinates (x 1, y 1) in the restored image is greater than the preset score threshold, the i1 th pixel point in the original image is a difference pixel point. For another example, if the difference determination score between the i2 th pixel point with coordinates (x 2, y 2) in the original image and the j2 th pixel point with coordinates (x 2, y 2) in the restored image is smaller than the preset score threshold, the i2 th pixel point in the original image is not a difference pixel point.

402. And taking the position of the difference pixel point as the abnormal position of the target object.

For example, the positions of the differential pixels are represented by coordinates, the differential pixels in the original image are differential pixels 1 to 6, and the positions of the pixels 1 to 6 are respectively: (1, 1), (2, 1), (3, 1), (2, 2), (2, 3), the abnormal position of the target object is the position of the difference pixel points 1-6: (1, 1), (2, 1), (3, 1), (2, 2), (2, 3).

As shown in fig. 5, the following takes as an example that the target object is a PCB on the production line, the original image is a defect image of the PCB, the detection of the abnormal position of the object is to detect the defect of the PCB, and the target generator is a generator in the generative countermeasure network trained in the above step 303, to exemplify how to implement the detection of the abnormal position of the object.

(1) And acquiring defective images of the PCB and non-defective images of the PCB as a training data set, and training a preset generation type countermeasure network by taking the non-defective images of the PCB as supervision information according to the steps 301 to 303 to obtain a target generator for detecting defects of the PCB.

(2) And acquiring a defect image of the PCB through an industrial camera on a production line.

(3) Inputting the defect image of the PCB into the generation countermeasure network trained in the step 303, calling the generator in the generation countermeasure network trained in the step (1), and repairing according to the defect image of the PCB to generate a restored image of the PCB.

(4) And (3) comparing the defect image of the PCB acquired in the step (2) with the restored image of the PCB generated in the step (3), for example, performing interpolation calculation, comparing the difference between the defect image of the PCB and the restored image of the PCB, and taking the difference position between the defect image of the PCB and the restored image of the PCB as the defect position of the PCB to obtain the abnormal position of the PCB.

(5) And intercepting an image of the abnormal position area of the PCB in the defect image of the PCB as abnormal information of the defect image of the PCB.

The PCB board circuit is complicated changeable, and the defect of PCB board is detected out to the difficult accurate ground of pure target detection, compares through the original image that restores the original image that generates the PCB board and PCB board, can detect out the defect of PCB board more accurately fast.

In order to better implement the method for detecting an abnormal position of an object in the embodiment of the present application, based on the method for detecting an abnormal position of an object, an apparatus for detecting an abnormal position of an object is further provided in the embodiment of the present application, as shown in fig. 6, which is a schematic structural diagram of an embodiment of the apparatus for detecting an abnormal position of an object in the embodiment of the present application, and the apparatus 600 for detecting an abnormal position of an object includes:

an acquisition unit 601 configured to acquire an original image of a target object;

a generating unit 602, configured to invoke a preset target generator to generate a restored image of the target object according to the original image;

a determining unit 603 configured to determine, from the restored image and the original image, abnormality information of the original image, where the abnormality information is used to indicate an abnormal position of the target object.

In a possible implementation manner of the present application, the target generator is obtained by training based on a preset generative countermeasure network, where the generative countermeasure network includes a generator and a discriminator, and the apparatus 600 for detecting an abnormal position of an object further includes a training unit (not shown in the figure), where the training unit is specifically configured to:

In one possible implementation manner of the present application, in the adjusting process of the network parameter of the generator by using the first training sample, the training loss of the generative confrontation network is the first generative loss of the generator; in the process of adjusting the network parameters of the pre-training generator and the network parameters of the discriminator by using a second training sample, the training loss of the generative confrontation network includes a second generation loss of the generator and a confrontation loss of the discriminator, and the training unit is specifically configured to:

and determining the first generation loss according to the predicted repairing image, the non-defective real image and a preset first loss function.

and determining the antagonistic loss according to the judgment result and a preset third loss function.

In a possible implementation manner of the present application, the target object is a target PCB, the original image is a defect image of the target PCB, and the generating unit 602 is specifically configured to:

in a possible implementation manner of the present application, the determining unit 603 is specifically configured to:

In specific implementation, the above units may be implemented as independent entities, or may be combined arbitrarily, and implemented as the same or several entities, and specific implementations of the above units may refer to the foregoing method embodiment, which is not described herein again.

Since the apparatus for detecting an abnormal position of an object can perform the steps of the method for detecting an abnormal position of an object in any embodiment corresponding to fig. 1 to 5 of the present application, the advantageous effects that can be achieved by the method for detecting an abnormal position of an object in any embodiment corresponding to fig. 1 to 5 of the present application can be achieved, for which, the foregoing description is omitted for brevity.

In addition, in order to better implement the method for detecting the abnormal position of the object in the embodiment of the present application, based on the method for detecting the abnormal position of the object, the embodiment of the present application further provides an electronic device, referring to fig. 7, fig. 7 shows a schematic structural diagram of the electronic device in the embodiment of the present application, specifically, the electronic device in the embodiment of the present application includes a processor 701, and when the processor 701 is used for executing a computer program stored in a memory 702, each step of the method for detecting the abnormal position of the object in any embodiment corresponding to fig. 1 to 5 is implemented; alternatively, the processor 701 is configured to implement the functions of the units in the corresponding embodiment of fig. 6 when executing the computer program stored in the memory 702.

Illustratively, a computer program may be partitioned into one or more modules/units, which are stored in the memory 702 and executed by the processor 701 to implement embodiments of the present application. One or more modules/units may be a series of computer program instruction segments capable of performing certain functions, the instruction segments being used to describe the execution of the computer program in the computer apparatus.

The electronic device may include, but is not limited to, a processor 701, a memory 702. Those skilled in the art will appreciate that the illustration is merely an example of an electronic device and does not constitute a limitation of the electronic device, and may include more or less components than those illustrated, or combine some components, or different components, for example, the electronic device may further include an input output device, a network access device, a bus, etc., and the processor 701, the memory 702, the input output device, the network access device, etc., are connected via the bus.

The Processor 701 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, the processor being the control center for the electronic device and the various interfaces and lines connecting the various parts of the overall electronic device.

The memory 702 may be used to store computer programs and/or modules, and the processor 701 may implement various functions of the computer apparatus by running or executing the computer programs and/or modules stored in the memory 702 and invoking data stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, application programs (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like; the storage data area may store data (such as audio data, video data, etc.) created according to the use of the electronic device, and the like. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described object abnormal position detection apparatus, the electronic device and the corresponding units thereof may refer to the descriptions of the object abnormal position detection method in any embodiment corresponding to fig. 1 to 5, and are not described herein again in detail.

It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.

To this end, an embodiment of the present application provides a computer-readable storage medium, where multiple instructions are stored, and the instructions can be loaded by a processor to execute steps in the method for detecting an abnormal position of an object in any embodiment of the present application corresponding to fig. 1 to 5, and for specific operations, reference may be made to descriptions of the method for detecting an abnormal position of an object in any embodiment corresponding to fig. 1 to 5, which are not repeated herein.

Wherein the computer-readable storage medium may include: read Only Memory (ROM), random Access Memory (RAM), magnetic or optical disks, and the like.

Since the instructions stored in the computer-readable storage medium can execute the steps of the method for detecting an abnormal position of an object in any embodiment corresponding to fig. 1 to 5 in the present application, the beneficial effects that can be achieved by the method for detecting an abnormal position of an object in any embodiment corresponding to fig. 1 to 5 in the present application can be achieved, for details, see the foregoing description, and are not repeated here.

The foregoing describes in detail a method, an apparatus, an electronic device, and a computer-readable storage medium for detecting an abnormal position of an object provided in an embodiment of the present application, where a specific example is applied to explain the principle and the implementation of the present application, and the description of the foregoing embodiment is only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A method for detecting an abnormal position of an object, the method comprising:

acquiring an original image of a target object;

and according to the restored image and the original image, determining abnormal information of the original image, wherein the abnormal information is used for indicating the abnormal position of the target object.

2. The method for detecting the abnormal position of the object according to claim 1, wherein the target generator is trained based on a preset generative countermeasure network, the generative countermeasure network comprises a generator and an arbiter, and the target generator is trained by the following steps:

3. The method according to claim 2, wherein in the adjusting of the network parameters of the generator using the first training samples, the training loss of the generative confrontation network is a first generative loss of the generator; in the process of adjusting the network parameters of the pre-training generator and the network parameters of the discriminator by using a second training sample, the training loss of the generative confrontation network comprises a second generation loss of the generator and a confrontation loss of the discriminator;

when the network parameters of the generator are adjusted by using the first training sample until a preset first training stopping condition is met, taking the generator as a pre-training generator, including:

the using a second training sample to adjust the network parameter of the pre-training generator and the network parameter of the discriminator until a preset second training stopping condition is met, and using the pre-training generator as the target generator includes:

4. The method according to claim 3, wherein the first training sample includes at least one pair of sample images, each pair of sample images including a defective real image and a non-defective real image of a sample object, the non-defective real image serving as a label for the defective real image, and the determining the first generation loss from the predicted repair image generated by the generator based on the defective real image in the first training sample includes:

5. The method according to claim 3, wherein the second training sample includes at least one pair of sample images, each pair of the sample images including a defective real image and a non-defective real image of a sample object, the non-defective real image serving as a label of the defective real image, and the determining the second generation loss and the countermeasure loss according to the predicted repair image generated by the pre-training generator based on the defective real image in the second training sample includes:

calling the discriminator to determine a discrimination result of the predicted repaired image, wherein the discrimination result is used for indicating whether the predicted repaired image is a real image or not;

6. The method according to claim 2, wherein the generator of the generative countermeasure network is UNet network, and the discriminator of the generative countermeasure network is convolutional neural network structure.

7. The method for detecting the abnormal position of the object according to claim 1, wherein the target object is a target PCB board, the original image is a defect image of the target PCB board, and the invoking of a preset target generator generates a restored image of the target object from the original image includes:

the determining the abnormal information of the original image according to the restored image and the original image includes:

8. The method according to any one of claims 1 to 7, wherein the determining abnormality information of the original image from the restored image and the original image includes:

9. The method according to claim 8, wherein the determining the abnormal position of the target object from the restored image and the original image includes:

10. An object abnormal position detecting device, characterized by comprising:

an acquisition unit configured to acquire an original image of a target object;

and a determining unit configured to determine abnormality information of the original image based on the restored image and the original image, the abnormality information indicating an abnormal position of the target object.

11. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the object abnormal position detecting method according to any one of claims 1 to 9 when calling the computer program in the memory.

12. A computer-readable storage medium, having stored thereon a computer program which is loaded by a processor to perform the steps of the object abnormal position detecting method according to any one of claims 1 to 9.