CN118212175A

CN118212175A - Defect detection method, device, electronic equipment and storage medium

Info

Publication number: CN118212175A
Application number: CN202211644938.8A
Authority: CN
Inventors: 陈路燕; 聂磊
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-12-16
Filing date: 2022-12-16
Publication date: 2024-06-18

Abstract

The disclosure provides a defect detection method, a defect detection device, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, image processing, mechanical equipment and the like. The specific implementation scheme is as follows: performing target recognition on the equipment image to be detected, and determining a component image in the equipment image to be detected; performing component defect detection on the component image, and determining an initial component defect detection result of the component in the component image; determining a class confidence of the component from the initial component defect detection result in the case that the initial component defect detection result is determined to be used for characterizing the component as a predetermined class of the component; and determining a component defect detection result of the component based on the class confidence of the component and a predetermined confidence threshold.

Description

Defect detection method, device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of artificial intelligence, and in particular, to the technical fields of deep learning, image processing, mechanical equipment, and the like, and in particular, to a defect detection method, apparatus, electronic device, storage medium, and program product.

Background

Machine defect detection has significant advantages over human vision, such as "seeing" fast moving objects that cannot be seen by the human eye, and higher stability and controllability. In addition, the information can be integrated and stored, and the tracing is convenient. But there is also a large rise space in terms of improving accuracy and comprehensiveness of defect detection, and the like.

Disclosure of Invention

The present disclosure provides a defect detection method, apparatus, electronic device, storage medium, and program product.

According to an aspect of the present disclosure, there is provided a defect detection method including: performing target recognition on an image of the equipment to be detected, and determining a part image in the image of the equipment to be detected; performing component defect detection on the component image, and determining an initial component defect detection result of a component in the component image; determining a class confidence of the component from the initial component defect detection result when the initial component defect detection result is determined to be used for representing the component as a predetermined class of component; and determining a component defect detection result of the component based on the class confidence of the component and a predetermined confidence threshold.

According to another aspect of the present disclosure, there is provided a defect detecting apparatus including: the identification module is used for carrying out target identification on the equipment image to be detected and determining the component image in the equipment image to be detected; the first detection module is used for detecting the part defects of the part images and determining initial part defect detection results of the parts in the part images; a first determining module configured to determine a class confidence of the component from the initial component defect detection result when it is determined that the initial component defect detection result is used to characterize the component as a predetermined class of components; and a second determining module for determining a component defect detection result of the component based on the class confidence of the component and a predetermined confidence threshold.

According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method as disclosed herein.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer as described above to perform a method as disclosed herein.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a method as disclosed herein.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 schematically illustrates an exemplary system architecture to which defect detection methods and apparatus may be applied, according to embodiments of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a defect detection method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a component defect detection result of a determined component according to another embodiment of the present disclosure;

FIG. 4 schematically illustrates a schematic diagram of determining a part image according to an embodiment of the disclosure;

FIG. 5 schematically illustrates a flow chart of determining target defect detection results according to an embodiment of the disclosure;

FIG. 6 schematically illustrates a block diagram of a defect detection apparatus according to an embodiment of the present disclosure; and

Fig. 7 schematically illustrates a block diagram of an electronic device adapted to implement a defect detection method according to an embodiment of the disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

According to an embodiment of the present disclosure, there is provided a defect detection method including: performing target recognition on the equipment image to be detected, and determining a component image in the equipment image to be detected; performing component defect detection on the component image, and determining an initial component defect detection result of the component in the component image; determining a class confidence of the component from the initial component defect detection result in the case that the initial component defect detection result is determined to be used for characterizing the component as a predetermined class of the component; and determining a component defect detection result of the component based on the class confidence of the component and a predetermined confidence threshold.

In the technical scheme of the disclosure, the related processes of collecting, storing, using, processing, transmitting, providing, disclosing, applying and the like of the personal information of the user all conform to the regulations of related laws and regulations, necessary security measures are adopted, and the public order harmony is not violated.

In the technical scheme of the disclosure, the authorization or consent of the user is obtained before the personal information of the user is obtained or acquired.

Fig. 1 schematically illustrates an exemplary system architecture to which defect detection methods and apparatus may be applied, according to embodiments of the present disclosure.

It should be noted that fig. 1 is only an example of a system architecture to which embodiments of the present disclosure may be applied to assist those skilled in the art in understanding the technical content of the present disclosure, but does not mean that embodiments of the present disclosure may not be used in other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the defect detection method and apparatus may be applied may include a terminal device, but the terminal device may implement the defect detection method and apparatus provided by the embodiments of the present disclosure without interaction with a server.

As shown in fig. 1, a system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and/or wireless communication links, and the like.

The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as a knowledge reading class application, a web browser application, a search class application, an instant messaging tool, a mailbox client and/or social platform software, etc. (as examples only).

The terminal devices 101, 102, 103 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for content browsed by the user using the terminal devices 101, 102, 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that, the defect detection method provided by the embodiments of the present disclosure may be generally performed by the terminal device 101, 102, or 103. Accordingly, the defect detecting apparatus provided by the embodiments of the present disclosure may also be provided in the terminal device 101, 102, or 103.

Or the defect detection method provided by the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the defect detection apparatus provided by the embodiments of the present disclosure may be generally disposed in the server 105. The defect detection method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the defect detection apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

It should be noted that the sequence numbers of the respective operations in the following methods are merely representative of the operations for the purpose of description, and should not be construed as representing the order of execution of the respective operations. The method need not be performed in the exact order shown unless explicitly stated.

Fig. 2 schematically illustrates a flow chart of a defect detection method according to an embodiment of the present disclosure.

As shown in fig. 2, the method includes operations S210 to S240.

In operation S210, object recognition is performed on the device image to be detected, and a component image in the device image to be detected is determined.

In operation S220, component defect detection is performed on the component image, and an initial component defect detection result of the component in the component image is determined.

In operation S230, in the case where it is determined that the initial component defect detection result is used to characterize the component as a predetermined class of components, a class confidence of the component is determined from the initial component defect detection result.

In operation S240, a component defect detection result of the component is determined based on the class confidence of the component and a predetermined confidence threshold.

According to an embodiment of the present disclosure, the device to be detected image may be an image of the device to be detected as an object. The device to be detected may comprise at least one of: vehicles, manufacturing equipment, robots, aircraft. But is not limited thereto. As long as it is an apparatus composed of a plurality of components.

According to an embodiment of the present disclosure, the component image may be one partial image of the device image to be detected. The component image may be an image with one or more components in the device to be detected.

According to an embodiment of the present disclosure, performing object recognition on an image of a device to be detected, and determining a component image in the image of the device to be detected may include: and identifying at least one component in the device image to be detected to obtain a component image. The component image in the device image to be detected may be identified by using an example segmentation method, but is not limited thereto, and a template matching method may be used as long as the component image is obtained from the device image to be detected.

According to an embodiment of the present disclosure, performing component defect detection on a component image, determining an initial component defect detection result of a component in the component image may include: and detecting the part defects of the part image according to a part defect detection method matched with the part type of the part, and determining an initial part defect detection result of the part in the part image. But is not limited thereto. The method can also be used for detecting the defects of the components by using a general component defect detection method, and determining the initial component defect detection result of the components.

Compared with the common component defect detection method, the component defect detection method matched with the component type of the component is used for detecting the component defect, so that the component defect detection is accurate and effective, and the problem of low generalization caused by the use of the common component defect detection method is solved.

According to an embodiment of the present disclosure, the initial component defect detection result may refer to an initial component defect detection result of directly determined components by performing component defect detection on the component image. The initial component defect detection result may include: and detecting the defect type. It may be determined whether a component in the component image is defective based on the defect class detection result, and in the case of defect, a specific defect class. The initial component defect detection result may be directly used as the component defect detection result.

According to an alternative embodiment of the present disclosure, the initial component defect detection result may be taken as a component defect detection result in case it is determined that the initial component defect detection result of the component is used to characterize the component as a component of a non-predetermined category. In the case where the initial component defect detection result is determined to characterize the component as a predetermined class component, the class confidence of the component is determined from the initial component defect detection result, and may include a class confidence corresponding to the predetermined class, but is not limited thereto, and may include class confidence corresponding to a plurality of non-predetermined classes and predetermined classes, respectively.

According to embodiments of the present disclosure, in the case where an initial component defect detection result is determined to be used to characterize a component as a predetermined class of components, the component defect detection result of the component may be determined based on the class confidence of the component and a predetermined confidence threshold. For example, the predetermined class is a defect class a, and in the event that the initial component defect detection result is determined to be indicative of a component of defect class a, it is determined whether a class confidence of the component corresponding to defect class a is greater than a predetermined confidence threshold. In the case where it is determined that the class confidence of the component corresponding to the defect class a is greater than the predetermined confidence threshold, the component defect detection result of the component is determined to be defective and to be a defect of the defect class a. In the case where it is determined that the class confidence of the component corresponding to the defect class a is less than or equal to the predetermined confidence threshold, the component defect detection result of the component is determined to be a defect of the defective class B or to be a normal class such as no defect.

Compared with a defect detection method directly taking an initial part defect detection result as a part defect detection result, by utilizing the defect detection method provided by the embodiment of the disclosure, under the condition that the initial part defect detection result is determined to be used for representing a part of a preset category, the initial part defect detection result of the preset category can be re-verified by utilizing the preset confidence threshold value, the effect of repeated calibration is achieved, and the accuracy of the part defect detection result is further improved.

The defect detection method provided by the embodiment of the disclosure is applied to the industrial field, and can strengthen quality inspection, improve the technical level of the process and standardize production operation based on the detection result of the defects of the components, further improve the quality of products and realize the jump of the competitiveness of the products. In addition, production safety accidents caused by faults and unknowns of equipment to be detected can be avoided based on the detection result of the part defects in the production, manufacturing and equipment operation processes.

According to an alternative embodiment of the present disclosure, the predetermined category may include a normal category.

According to an embodiment of the present disclosure, for operation S240 as shown in fig. 2, determining the component defect detection result of the component based on the class confidence of the component and the predetermined confidence threshold may include the following operations.

For example, in the case where it is determined that the class confidence of the component corresponding to the normal class is greater than the predetermined confidence threshold, the component defect detection result of the component is determined to be the first component defect detection result. The first component defect detection result is used for representing the component as a normal class component. And determining that the component defect detection result of the component is a second component defect detection result in the case that the class confidence of the component corresponding to the normal class is determined to be less than or equal to the predetermined confidence threshold. The second component defect detection result is used to characterize the component as a defect class component.

According to an embodiment of the present disclosure, in a case where it is determined that an initial component defect detection result is used to characterize a component as a normal class component, a class confidence corresponding to the normal class is compared with a predetermined confidence threshold, and a confidence that the component is the normal class component is determined based on the class confidence and the predetermined confidence threshold. In the case where it is determined that the class confidence of the component corresponding to the normal class is greater than the predetermined confidence threshold, it is indicated that the component is highly reliable for the normal class component, and the initial component defect detection result can be used as the component defect detection result. And when the confidence coefficient of the category of the component corresponding to the normal category is less than or equal to the preset confidence coefficient threshold value, the reliability of the component in the normal category is lower, and the reliability of the component in the defect category is higher.

According to the embodiment of the disclosure, the defect detection result of the component is determined in the above manner, so that the problem of identifying defective components with endless new types of defects, such as unknown defect types, can be solved.

According to an embodiment of the present disclosure, a component image may be processed using a component defect detection model, outputting an initial component defect detection result. The component defect detection model may include an instance segmentation model such as Faster R-CNN or Mask R-CNN, etc. Any network model is possible that can use the component image as input data and the initial component defect detection result as output data.

According to the embodiment of the disclosure, a training sample, a training detection model or an example segmentation model may be utilized to obtain a component defect detection model, and the component defect detection model may be utilized to obtain an initial component defect detection result of a component in a component image. The training samples may include part images of defect class a, part images of defect class B, part images of defect class C, and part images of normal class. In the actual application process, the category of the initial component defect detection result is one of five categories of defect category a, defect category B, defect category C and normal category.

According to an exemplary embodiment of the present disclosure, in an actual defect detection process, it is found that the technical problem exists is: new classes of defects are endless and models are not trained by using training samples of the defect class; or defects of the defect class rarely occur, without training samples of the defect class. The component defect detection model thus does not learn the relevant knowledge and features of the defect class, which in turn results in the component defect detection model not being able to accurately identify defective components with unknown defect classes.

According to a related example, a component image is input to a component defect detection model to obtain an initial component defect detection result, and the initial component defect detection result is taken as a component defect detection result. It may result in defective parts having a new defect class E being incorrectly identified as, for example, defect class a or normal class.

According to embodiments of the present disclosure, it is found during actual defect detection that a component defect detection model readily identifies an unknown defect-class component as a normal-class component if the number of training samples for the known defect class is sufficient. The normal class may be used as a reference, e.g., a predetermined class, and the class confidence corresponding to the normal class and a predetermined confidence threshold may be used to ultimately determine whether a component in the component image is a normal class component or a defect class component of an unknown defect class. Therefore, the precision of the defect detection result is improved, the generalization of the part defect detection model is enlarged, and the cost and time of optimization training are reduced.

Fig. 3 schematically illustrates a flowchart of determining a component defect detection result of a component according to an embodiment of the present disclosure.

As shown in fig. 3, component defect detection is performed on the component image 310, and an initial component defect detection result 320 of the component in the component image 310 is determined, for example, as a normal category. In the event that the initial part defect detection result is determined to be used to characterize the part as a normal class of parts, a class confidence 330 for the part is determined. The class confidence corresponding to the defect class a is 0.3, the class confidence corresponding to the defect class B is 0.2, the class confidence corresponding to the defect class C is 0.1, and the class confidence corresponding to the normal class D is 0.4. In the case where it is determined that the class confidence level 0.4 corresponding to the normal class D is less than the predetermined confidence threshold value, e.g., 0.6, the component defect detection result 340 of the component in the component image 310 is determined, e.g., as an unknown defect class.

According to an embodiment of the present disclosure, for operation S210 shown in fig. 2, performing object recognition on a device image to be detected, determining a component image in the device image to be detected may include the following operations.

For example, object recognition is performed on the device image to be detected, and a reference component in the device image to be detected is determined. The reference component is a predetermined component of the device to be detected in the image of the device to be detected. A part image is determined based on the reference part and the template image. The template image is an image matched with the image of the equipment to be detected.

According to the embodiment of the disclosure, the device image to be detected can be input into the target recognition model, and the detection result of the reference component can be obtained. The detection result may include data of a detection frame of the reference component, a component category, a confidence level, and the like. The reference component in the image of the device to be detected may be determined based on the detection result of the reference component. The reference member may be one predetermined member of the device to be detected in the device to be detected image, but is not limited thereto, and may be a plurality of predetermined members. Any member may be used as long as it can exert a positioning effect.

According to embodiments of the present disclosure, the object recognition model may include an instance segmentation model such as Faster R-CNN or Mask R-CNN, or the like. As long as it is a network model that can take an image of a device to be detected as input data and a detection result of a reference member as output data.

According to an embodiment of the present disclosure, determining a part image based on a reference part and a template image may include: based on the reference member, a relative positional relationship between the template member and the reference member is determined from the template image. Based on the relative positional relationship, a component image is determined from the device image to be detected. But is not limited thereto. The template image and the device image to be detected may also be aligned based on the reference component, and the component image of the device image to be detected may be determined based on the template component image of the template image.

According to other embodiments of the present disclosure, an image of a device to be detected may be input into a component detection model, to obtain respective component detection results of at least one component in the image of the device to be detected. At least one component image is determined based on the respective component detection results of the at least one component. The component detection model may be identical to the model structure of the object recognition model and will not be described in detail herein.

Compared with the way of determining the component image by using the component detection model, the template image is used for determining the component image from the to-be-detected device image, so that the problem of missing identification caused by the identification of the component detection model can be avoided under the condition that the number of the components in the to-be-detected device image is large.

For example, when the imaging quality of the image of the device to be detected is poor, the definition is low or the noise is large, the image of the device to be detected is input into the component detection model, the accuracy of the detection result of the obtained component is poor, and the problem of missing detection or false detection is likely to occur. For example, in the case where a component of the device to be detected in the image of the device to be detected is lost or falls off, even if the imaging quality of the image of the device to be detected is high, the component detection model cannot recognize the detection frame of the component, and thus a problem of missing detection occurs.

By using the template image-based mode for determining the component image from the to-be-detected equipment image, the problem of missing detection or false detection can be avoided, and the component image is detected comprehensively and accurately.

According to an embodiment of the present disclosure, determining a part image based on a reference part and a template image may include the following operations.

For example, a template reference component that matches the reference component is determined from the template image. The template image and the device image to be detected are aligned based on the template reference member and the reference member. Position information of a component of the device to be detected is determined based on position information of a template component of the template image. The components of the device to be detected are matched to the template components of the template image. Based on the position information of the component of the device to be detected, a component image of the device to be detected is determined.

According to an embodiment of the present disclosure, the position information labeling and the part name labeling may be performed on a plurality of parts in the template image in advance so as to determine the template reference part matching the reference part from the template image directly based on the identification information of the reference part. But is not limited thereto. The template image may also be input into the target recognition model to obtain a detection result regarding the template reference member. And further determining a template reference component from the template image that matches the reference component.

According to an embodiment of the present disclosure, aligning a template image and an image of a device to be detected based on a template reference part and a reference part may include: the template reference component and the reference component are aligned by utilizing the principle of non-collinear multipoint constituent surfaces so as to align the template image and the device image to be detected. The number of reference members may include one, but is not limited thereto, and the number of reference members may also include a plurality. The greater the number of reference features, the more accurate the alignment between the template image and the image of the device to be inspected.

According to the embodiments of the present disclosure, a template image and a device image to be detected may be overlapped and aligned, and a component image matching the template component may be determined from the device image to be detected based on position information, such as contour information, of the template component of the template image. For example, according to the outline information of the template component, cutting the image of the equipment to be detected, and obtaining the component image corresponding to the template component. But is not limited thereto. The relative positional relationship between the template component and the reference template component may also be determined from the template image, and the component image may be determined from the device image to be detected based on the relative positional relationship. For example, positional information such as a relative positional relationship of the plurality of vertices of the template member from the reference template member is known. Positional information, such as a relative positional relationship, of the plurality of vertices of the part corresponding to the template part from the reference part, respectively, is determined. Based on the position information of the component of the device to be detected, a component image of the device to be detected is determined.

Fig. 4 schematically illustrates a schematic diagram of determining a part image according to an embodiment of the present disclosure.

As shown in fig. 4, the target recognition is performed on the device image 410 to be detected, and a reference component 411 in the device image to be detected, for example, a vehicle is the device to be detected, and the reference component is the left front wheel of the vehicle. A template reference component 421 that matches the reference component 411 is determined from the template image 420. The template image 420 and the device to be detected image 410 may be aligned in an overlapping manner. Based on the position information, such as contour information, of the template component 422 of the template image, such as the front right wheel, the rear left wheel, the rear right wheel, the fuel tank, etc., of the vehicle identified by the broken line, a plurality of component images 413 matching the template component 422 are determined from the device image to be detected 410.

But is not limited thereto. The relative positional relationship between the template part 422 and the reference template part 421 may also be determined from the template image 420, and the part image 413 may be determined from the device image 410 to be detected based on the relative positional relationship. For example, based on positional information such as a relative positional relationship of the plurality of vertexes of the template member such as the front right wheel, respectively, from the reference template member 421, positional information such as a relative positional relationship of the plurality of vertexes of the member 412 corresponding to the template member, respectively, from the reference member 411 in the device image to be detected is determined. Based on the positional information of the component 412 of the device to be detected, a component image of the device to be detected corresponding to the component 412 is determined.

According to a related example, template matching algorithms may be utilized, such as Fast corner detection, SIFT (Scale-INVARIANT FEATURE TRANSFORM ) matching, and the like. Features of the template component may be extracted from the template image to obtain a template feature vector that matches the template component. And extracting features from the image of the equipment to be detected to obtain feature vectors of the equipment to be detected. A component image is determined from the device image to be detected based on the vector similarity between the template feature vector and the feature vector of the device to be detected.

Compared with the mode of utilizing the template matching algorithm, the template image and the to-be-detected equipment image are aligned based on the template reference component and the reference component, so that the accuracy of the component image can be improved, the processing capacity can be reduced, and the processing efficiency can be improved.

According to an embodiment of the present disclosure, aligning a template image and a device image to be detected based on a template reference part and a reference part may include the following operations.

For example, the device image to be detected is subjected to transformation processing based on the template reference component and the reference component, and the transformed device image to be detected is obtained. And aligning the template image with the transformed device image to be detected based on the template reference component and the reference component.

According to the embodiment of the disclosure, the device image to be detected is subjected to transformation processing, and the transformed device image to be detected is obtained. The transformation process may include at least one of: rotation, scaling, stretching, binarization. As long as the transformation processing can be performed, a transformation method is sufficient in which the device image to be detected is corrected.

According to the embodiment of the disclosure, the device image to be detected is transformed, so that the transformed device image to be detected and the template image are aligned accurately, and the problem that the determined part image lacks local information due to the fact that the local part is not aligned is avoided.

According to an embodiment of the present disclosure, in a case where the device to be detected includes a plurality of devices, and the plurality of devices to be detected are different in device type from each other. A plurality of template images may be generated in advance, the device type in each template image being matched with the device type of one device to be detected.

According to an embodiment of the present disclosure, before performing the operation S210 shown in fig. 2, performing object recognition on the device image to be detected, and determining the component image in the device image to be detected, the defect detection method may further include: an operation of determining a template image matching the device to be detected from among the plurality of template images.

According to an embodiment of the present disclosure, determining a template image matching a device to be detected from a plurality of template images may include: based on the reference component, a device type of the device to be detected is determined. Based on the device type, a template image is determined that matches the device image to be detected.

According to an embodiment of the present disclosure, determining a device type of a device to be detected based on a reference component may include: a first mapping relationship between the reference component and the device type and a second mapping relationship between the device type and the template image are generated. The device type of the device to be detected may be determined based on the reference component and the first mapping relation. And determining a template image matched with the device image to be detected from the plurality of template images based on the device type and the second mapping relation.

According to an embodiment of the present disclosure, generating a first mapping relationship between a reference component and a device type may include: a first mapping relationship between the reference component and the device type is generated based on the profile information of the reference component. But is not limited thereto. The first mapping relationship between the reference component and the device type may also be generated based on the feature information of the reference component.

According to an embodiment of the present disclosure, the contour information of the reference component may refer to an outer boundary contour of the reference component. The first mapping relation may be generated based on the profile information of the reference component in a case where the profile information of the reference component is different in the device type. The feature information of the reference member may be feature information of the reference member obtained by extracting features from the reference member image. In the case where the feature information of the reference component, such as texture, color, and the like, is related to the device type, a first mapping relationship is generated based on the feature information of the reference component.

According to an embodiment of the present disclosure, determining a template image matching a device to be detected from among a plurality of template images may further include: and extracting device type identification information from the device image to be detected, and determining the device type of the device to be detected based on the device type identification information. Based on the device type, a template image is determined that matches the device image to be detected.

According to the embodiment of the disclosure, the component image in the device image to be detected is determined by using the template image matched with the device image to be detected, so that the component image determination is accurate and effective. In addition, the application range of the defect detection method can be expanded by providing a plurality of template images.

According to an embodiment of the present disclosure, before performing the object recognition of the device image to be detected and determining the component image in the device image to be detected, the defect detection method may further include the following operations, as illustrated in fig. 2, at operation S210. For example, a plurality of images are spliced according to the acquisition time sequence to obtain the image of the equipment to be detected.

According to embodiments of the present disclosure, the plurality of images may be acquired using the same information acquisition device. But is not limited thereto. And the information can be acquired by a plurality of information acquisition devices. And splicing the plurality of images to obtain the image of the equipment to be detected. The method can be applied to huge equipment to be detected, and the image of the equipment to be detected comprises the whole picture of the equipment to be detected by utilizing the splicing of a plurality of images.

According to the embodiment of the disclosure, the plurality of images can be spliced according to the acquisition time sequence, so that the spliced images are images of equipment to be detected. But is not limited thereto. And the azimuth sequence of the information acquisition equipment can be utilized to splice a plurality of images so as to obtain the image of the equipment to be detected. The splicing mode can be any mode as long as the splicing mode can be used for splicing a plurality of images to obtain the images of the equipment to be detected.

For example, the device to be detected may be a train. A plurality of cameras are erected on the ground of the train bottom to respectively shoot the running trains so as to solve the problem that the shooting of the cameras in narrow space is limited. The cameras can shoot in the running process of the train to obtain a plurality of images, and the images are spliced according to the acquisition time sequence to obtain a complete image of equipment to be detected on the bottom of the train.

According to the embodiment of the disclosure, the image of the equipment to be detected is obtained by utilizing a mode of splicing a plurality of images, so that the equipment to be detected in the image of the equipment to be detected is complete, and the problem of lack of parts or incomplete local parts is avoided. In addition, the method can also pre-process a plurality of images, and eliminate unclear, shielded and noisy images in advance, so that the images of the equipment to be detected are clear, and the method is favorable for carrying out target identification on the images of the equipment to be detected subsequently.

According to an embodiment of the present disclosure, the defect detection method may further include the operation of identifying a defect class independent of the component of the device to be detected. Such as wear defects, impact defects, etc. For defect categories that are not related to the component, merely detecting the component image will lead to the problem that the device to be detected detects incompleteness. The following operation can be adopted to solve the problem.

For example, a device image to be detected is cut to obtain a plurality of image blocks. And respectively carrying out general defect detection on the plurality of image blocks to obtain a general defect detection result of the equipment to be detected in the equipment image to be detected.

According to the embodiment of the disclosure, the image of the device to be detected can be cut according to the preset size, so that a plurality of image blocks are obtained. And respectively carrying out general defect detection on the plurality of image blocks. But is not limited thereto. The universal defect detection can also be directly carried out on the equipment image to be detected. As long as the general defect detection result of the device to be detected can be determined.

Compared with a mode of directly carrying out general defect detection on an image of the equipment to be detected, the method has the advantages that general defect detection is carried out on a plurality of image blocks respectively, a general defect detection model can be friendly, and the problems of overlarge data size and large model processing capacity caused by taking the image of the equipment to be detected directly as input data are avoided. The generic defect detection model may include an instance segmentation model such as Faster R-CNN or Mask R-CNN, etc. Any network model that can use an image block as input data and a general defect detection result as output data may be used.

According to the embodiment of the disclosure, the combination of the component defect detection on the component image and the general defect detection on the image block can expand the defect detection range of the equipment to be detected and improve the detection richness.

According to the embodiment of the present disclosure, the general defect detection result and the component defect detection result can be directly notified to the related personnel as target defect detection results, respectively. But is not limited thereto. The target defect detection result of the device to be detected may also be determined based on the general defect detection result and the component defect detection result.

According to an embodiment of the present disclosure, determining a target defect detection result of a device to be detected based on a general defect detection result and a component defect detection result may include: and under the condition that the universal defect detection result is used for representing that the equipment to be detected is normal type equipment and the component defect detection result is the first component defect detection result, determining the target defect detection result of the equipment to be detected is used for representing that the equipment to be detected is normal type equipment. And under the condition that the universal defect detection result is used for representing that the equipment to be detected is the defect type equipment or the component defect detection result is used for representing that the component is the defect type component, determining the target defect detection result of the equipment to be detected is used for representing that the equipment to be detected is the defect type equipment.

According to the embodiment of the disclosure, based on the general defect detection result and the component defect detection result, if one of the general defect detection result and the component defect detection result is determined to be used for representing that the equipment to be detected is the defect type equipment, the equipment to be detected is determined to be the defect type equipment, so that the safety of the equipment to be detected can be ensured, and the operation safety of the equipment is improved.

Fig. 5 schematically illustrates a flow chart of determining target defect detection results according to an embodiment of the disclosure.

As shown in fig. 5, a plurality of images 510 may be stitched to obtain a device image 520 to be detected. The device image 520 to be detected is cut to obtain a plurality of image blocks 530. And respectively performing general defect detection on the plurality of image blocks 530 to obtain a general defect detection result 540 of the equipment to be detected in the equipment image to be detected. Target recognition is performed on the device image to be detected 520, and a component image 550 in the device image to be detected is determined. Component defect detection is performed on the component image 550, and a component defect detection result 560 is determined. Based on the general defect detection result 540 and the component defect detection result 560, a target defect detection result 570 of the device to be inspected is determined.

By using the defect detection method provided by the embodiment of the disclosure, starting from the characteristics of industrial equipment defect detection and application scenes, aiming at the characteristics of few defect category data in the defect detection scenes, the operation of calibrating the initial part defect detection result through the category confidence coefficient and the preset confidence coefficient threshold corresponding to the preset category under the condition that the initial part defect detection result is the preset category is provided, and the identification precision of the unknown defect category is improved. And the robustness and generalization of defect detection are improved while the dependence on rules is reduced. In addition, the general defect detection result and the component defect detection result are utilized to comprehensively determine the target defect detection result, so that the operation safety of equipment to be detected is improved while the comprehensiveness of defect detection is improved.

Fig. 6 schematically illustrates a block diagram of a defect detection apparatus according to an embodiment of the present disclosure.

As shown in fig. 6, the defect detecting apparatus 600 includes: the identification module 610, the first detection module 620, the first determination module 630, and the second determination module 640.

The identifying module 610 is configured to perform object identification on the device image to be detected, and determine a component image in the device image to be detected.

The first detection module 620 is configured to perform component defect detection on the component image, and determine an initial component defect detection result of the component in the component image.

The first determining module 630 is configured to determine a class confidence of the component from the initial component defect detection result in a case where the initial component defect detection result is determined to be used to characterize the component as a predetermined class of component.

A second determination module 640 for determining a component defect detection result for the component based on the class confidence of the component and a predetermined confidence threshold.

According to an embodiment of the present disclosure, the predetermined category is a normal category.

According to an embodiment of the present disclosure, the second determining module includes: the first determination sub-module and the second determination sub-module.

The first determining submodule is used for determining that the component defect detection result of the component is a first component defect detection result when the class confidence coefficient of the component corresponding to the normal class is larger than the preset confidence coefficient threshold value, wherein the first component defect detection result is used for representing that the component is the normal class component.

And the second determining submodule is used for determining that the component defect detection result of the component is a second component defect detection result when the class confidence coefficient of the component corresponding to the normal class is smaller than or equal to a preset confidence coefficient threshold value, wherein the second component defect detection result is used for representing that the component is a defect class component.

According to an embodiment of the present disclosure, an identification module includes: the sub-module is identified and a third determination sub-module.

The identification sub-module is used for carrying out target identification on the equipment image to be detected and determining a reference component in the equipment image to be detected, wherein the reference component is a preset component of the equipment to be detected in the equipment image to be detected.

And the third determination submodule is used for determining a part image based on the reference part and a template image, wherein the template image is an image matched with the image of the equipment to be detected.

According to an embodiment of the present disclosure, the defect detecting apparatus further includes: the third determination module and the fourth determination module.

And a third determining module for determining the device type of the device to be detected based on the reference component.

And a fourth determining module, configured to determine, based on the device type, a template image that matches the device image to be detected.

According to an embodiment of the present disclosure, the third determination submodule includes: a first determination unit, an alignment unit, a second determination unit, and a third determination unit.

And a first determining unit for determining a template reference component matched with the reference component from the template image.

And an alignment unit for aligning the template image with the device image to be detected based on the template reference member and the reference member.

And a second determining unit configured to determine position information of a component of the device to be detected based on position information of a template component of the template image, wherein the component of the device to be detected matches the template component of the template image.

And a third determining unit configured to determine a component image of the device to be detected based on the position information of the component of the device to be detected.

According to an embodiment of the present disclosure, an alignment unit includes: a transform subunit and an alignment subunit.

And the transformation subunit is used for transforming the to-be-detected equipment image based on the template reference component and the reference component to obtain a transformed to-be-detected equipment image.

And the alignment subunit is used for aligning the template image with the transformed device image to be detected based on the template reference component and the reference component.

According to an embodiment of the present disclosure, the defect detecting apparatus further includes: and (5) splicing the modules.

And the splicing module is used for splicing the plurality of images according to the acquisition time sequence to obtain the image of the equipment to be detected.

According to an embodiment of the present disclosure, the defect detecting apparatus further includes: and the image cutting module and the second detection module.

And the image cutting module is used for cutting the image of the equipment to be detected to obtain a plurality of image blocks.

And the second detection module is used for respectively carrying out general defect detection on the plurality of image blocks to obtain a general defect detection result of the equipment to be detected in the equipment image to be detected.

According to an embodiment of the present disclosure, the defect detecting apparatus further includes: and a fifth determination module.

And a fifth determining module, configured to determine a target defect detection result of the device to be detected based on the general defect detection result and the component defect detection result.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method as in an embodiment of the present disclosure.

According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a method as in an embodiment of the present disclosure.

According to an embodiment of the present disclosure, a computer program product comprising a computer program which, when executed by a processor, implements a method as an embodiment of the present disclosure.

Fig. 7 illustrates a schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 7, the apparatus 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 may also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

Various components in device 700 are connected to I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, etc.; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, an optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The computing unit 701 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 701 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the respective methods and processes described above, such as a defect detection method. For example, in some embodiments, the defect detection method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 700 via ROM 702 and/or communication unit 709. When a computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the defect detection method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the defect detection method by any other suitable means (e.g. by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A defect detection method, comprising:

performing target recognition on an image of equipment to be detected, and determining a component image in the image of the equipment to be detected;

Performing component defect detection on the component image, and determining an initial component defect detection result of a component in the component image;

Determining a class confidence of the component from the initial component defect detection result in the case that the initial component defect detection result is determined to be used for representing the component as a predetermined class of component; and

And determining a component defect detection result of the component based on the class confidence of the component and a predetermined confidence threshold.

2. The method of claim 1, wherein the predetermined category is a normal category;

The determining a component defect detection result of the component based on the class confidence of the component and a predetermined confidence threshold value comprises:

Determining that the component defect detection result of the component is a first component defect detection result under the condition that the class confidence of the component corresponding to the normal class is greater than the preset confidence threshold, wherein the first component defect detection result is used for representing that the component is a normal class component; and

And determining that the component defect detection result of the component is a second component defect detection result under the condition that the class confidence of the component corresponding to the normal class is smaller than or equal to the preset confidence threshold, wherein the second component defect detection result is used for representing that the component is a defect class component.

3. The method of claim 1, wherein the object recognition of the device image to be detected, determining a component image in the device image to be detected, comprises:

Performing target recognition on the equipment image to be detected, and determining a reference component in the equipment image to be detected, wherein the reference component is a preset component of equipment to be detected in the equipment image to be detected; and

And determining the component image based on the reference component and a template image, wherein the template image is an image matched with the to-be-detected equipment image.

4. A method according to claim 3, further comprising:

determining the equipment type of the equipment to be detected based on the reference component; and

And determining the template image matched with the device image to be detected based on the device type.

5. A method according to claim 3, wherein the determining the part image based on the reference part and template images comprises:

Determining a template reference component from the template image that matches the reference component;

Aligning the template image and the device image to be detected based on the template reference component and the reference component;

Determining position information of a component of the equipment to be detected based on the position information of the template component of the template image, wherein the component of the equipment to be detected is matched with the template component of the template image; and

And determining the component image of the equipment to be detected based on the position information of the component of the equipment to be detected.

6. The method of claim 5, wherein the aligning the template image and the device image to be detected based on the template reference component and the reference component comprises:

Based on the template reference component and the reference component, carrying out transformation processing on the to-be-detected equipment image to obtain a transformed to-be-detected equipment image; and

And aligning the template image with the transformed device image to be detected based on the template reference component and the reference component.

7. The method of claim 1, further comprising:

And splicing the plurality of images according to the acquisition time sequence to obtain the image of the equipment to be detected.

8. The method of claim 1, further comprising:

Cutting the image of the equipment to be detected to obtain a plurality of image blocks; and

And respectively carrying out general defect detection on the plurality of image blocks to obtain a general defect detection result of the equipment to be detected in the equipment image to be detected.

9. The method of claim 8, further comprising:

And determining a target defect detection result of the equipment to be detected based on the general defect detection result and the component defect detection result.

10. A defect detection apparatus comprising:

the identification module is used for carrying out target identification on the equipment image to be detected and determining a component image in the equipment image to be detected;

the first detection module is used for detecting the part defects of the part images and determining initial part defect detection results of the parts in the part images;

A first determining module configured to determine a category confidence of the component from the initial component defect detection result if it is determined that the initial component defect detection result is used to characterize the component as a predetermined category of the component; and

And a second determining module for determining a component defect detection result of the component based on the class confidence of the component and a predetermined confidence threshold.

11. The apparatus of claim 10, wherein the predetermined category is a normal category;

the second determining module includes:

A first determining sub-module configured to determine, when it is determined that a class confidence of the component corresponding to the normal class is greater than the predetermined confidence threshold, that the component defect detection result of the component is a first component defect detection result, where the first component defect detection result is used to characterize the component as a normal class component; and

And a second determining sub-module, configured to determine that the component defect detection result of the component is a second component defect detection result if it is determined that the class confidence of the component corresponding to the normal class is less than or equal to the predetermined confidence threshold, where the second component defect detection result is used to characterize the component as a defect class component.

12. The apparatus of claim 10, wherein the identification module comprises:

The identification sub-module is used for carrying out target identification on the equipment image to be detected and determining a reference component in the equipment image to be detected, wherein the reference component is a preset component of equipment to be detected in the equipment image to be detected; and

And a third determining sub-module, configured to determine the component image based on the reference component and a template image, where the template image is an image that matches the to-be-detected device image.

13. The apparatus of claim 12, further comprising:

A third determining module, configured to determine a device type of the device to be detected based on the reference component; and

And a fourth determining module, configured to determine, based on the device type, the template image that matches the device image to be detected.

14. The apparatus of claim 12, wherein the third determination submodule comprises:

a first determining unit configured to determine a template reference component that matches the reference component from the template image;

An alignment unit configured to align the template image and the device image to be detected based on the template reference member and the reference member;

a second determining unit configured to determine position information of a component of the device to be detected based on position information of a template component of the template image, where the component of the device to be detected matches the template component of the template image; and

And a third determining unit configured to determine the component image of the device to be detected based on position information of the component of the device to be detected.

15. The apparatus of claim 14, wherein the alignment unit comprises:

The transformation subunit is used for carrying out transformation processing on the to-be-detected equipment image based on the template reference component and the reference component to obtain a transformed to-be-detected equipment image; and

And the alignment subunit is used for aligning the template image and the transformed equipment image to be detected based on the template reference component and the reference component.

16. The apparatus of claim 10, further comprising:

17. The apparatus of claim 10, further comprising:

the image cutting module is used for cutting images of the equipment to be detected to obtain a plurality of image blocks; and

18. The apparatus of claim 17, further comprising:

And a fifth determining module, configured to determine a target defect detection result of the device to be detected based on the generic defect detection result and the component defect detection result.

19. An electronic device, comprising:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.

20. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1 to 9.

21. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.