CN115830004A

CN115830004A - Surface defect detection method, device, computer equipment and storage medium

Info

Publication number: CN115830004A
Application number: CN202211689323.7A
Authority: CN
Inventors: 杨溪; 唐永亮; 彭斌; 杨艺
Original assignee: Luster LightTech Co Ltd
Current assignee: Luster LightTech Co Ltd
Priority date: 2022-12-27
Filing date: 2022-12-27
Publication date: 2023-03-21

Abstract

The application provides a surface defect detection method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a surface image to be detected and a standard surface image of a product to be detected; performing image segmentation on the surface image to be detected and the standard surface image to obtain at least one to-be-detected image block corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image; inputting the image block to be detected and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block; each defect division image block is used for representing pixel difference information between one to-be-detected image block and one standard image block; and dividing the image block based on the defects, and determining whether the product to be detected has surface defects. So, this application utilizes the defect segmentation model that trains well to carry out the defect and cuts apart to this judges whether there is surface defect in waiting to examine the product, has improved the detection efficiency that surface defect detected and the degree of accuracy of testing result.

Description

Surface defect detection method, surface defect detection device, computer equipment and storage medium

Technical Field

The present application relates to the field of printed matter detection technologies, and in particular, to a method and an apparatus for detecting surface defects, a computer device, and a storage medium.

Background

With the increasing requirements of industrial product users and production enterprises on product quality, the product needs to have good surface quality in addition to meeting the service performance. However, surface defects are often unavoidable in the product manufacturing process, and therefore, it is necessary to perform surface defect detection on the product. The surface defect detection refers to detecting defects such as scratches, foreign matter blocking, wrinkles, speckles, color pollution, holes and the like on the surface of a product, so as to obtain a series of information such as types, outlines, positions, sizes and the like of the defects on the surface of the detected product.

In the detection of the surface defects of the printed matters, a manual detection mode is generally adopted, in which random extraction is firstly performed in a batch of printed matters at random according to a preset extraction rate, and then whether the surface of the extracted printed matters has defects is manually observed. However, the manual detection has the problems of low accuracy, poor real-time performance, low efficiency, high labor intensity and larger influence of manual experience and subjective factors on the detection result.

Disclosure of Invention

The application provides a surface defect detection method, a surface defect detection device, computer equipment and a storage medium, which can improve the detection efficiency of the surface defects of products.

In a first aspect, the present application provides a surface defect detection method, comprising:

acquiring a surface image to be detected and a standard surface image of a product to be detected;

performing image segmentation on the surface image to be detected and the standard surface image to obtain at least one to-be-detected image block corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image; the number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same;

inputting the image block to be detected and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block; each defect division image block is used for expressing pixel difference information between one to-be-detected image block and one standard image block;

and dividing the image block based on the defects, and determining whether the product to be detected has surface defects.

In a possible implementation manner, performing image segmentation on a surface image to be detected and a standard surface image to obtain at least one to-be-detected image block corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image, includes:

the pixel value subtraction operation is performed on the pixels at the same positions in the surface image to be detected and the standard surface image, obtaining an intermediate surface image;

analyzing a connected region of the intermediate surface image to obtain at least one candidate defect feature map;

and according to the corresponding regional position information of the candidate defect characteristic diagram in the intermediate surface image, performing image segmentation on the surface image to be detected and the standard surface image to obtain an image block to be detected and a standard image block.

In a possible implementation manner, dividing the image block based on the defect, and determining whether the product to be detected has a surface defect, includes:

acquiring pixel intersection information between the defect segmentation image block and the candidate defect feature map;

if the pixel intersection information is null, determining that no surface defect exists on the surface of the product to be detected;

and if the pixel intersection information is not null, determining that the surface of the product to be detected has surface defects.

In one possible implementation, the defect segmentation model includes an encoder and a decoder;

inputting the to-be-detected image block and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block, wherein the defect segmentation image block comprises:

inputting an image block to be detected and a standard image block into an encoder, and acquiring a first multi-scale feature map of the image block to be detected under multiple preset resolutions and a second multi-scale feature map of the standard image block under multiple resolutions through the encoder;

and inputting the first multi-scale feature map and the second multi-scale feature map into a decoder, performing feature fusion processing on the first multi-scale feature map and the second multi-scale feature map through the decoder, and outputting a defect segmentation image block.

In one possible implementation, acquiring a surface image to be inspected and a standard surface image of a product to be inspected includes:

acquiring an initial to-be-detected image and an initial standard image of a product to be detected;

based on the initial standard image, carrying out image registration processing on the initial image to be detected to obtain an intermediate image to be detected;

and respectively carrying out image preprocessing on the intermediate image to be detected and the initial standard image to obtain a surface image to be detected and a standard surface image.

In a possible implementation manner, the image preprocessing is performed on the intermediate image to be inspected and the initial standard image, so as to obtain the image of the surface to be inspected and the image of the standard surface, and the method includes:

performing edge detection on the intermediate image to be detected and the initial standard image to obtain first edge information of the intermediate image to be detected and second edge information of the initial standard image;

performing edge expansion processing on the middle image to be detected based on the first edge information to obtain a surface image to be detected;

and performing edge expansion processing on the initial standard image based on the second edge information to obtain a standard surface image.

In one possible implementation, the training process of the defect segmentation model includes:

acquiring a training sample image of a detected product; the training sample image comprises a standard sample image and a plurality of defect sample images, each sample image carries a sample label, and the sample label is used for indicating whether the sample image has surface defects or not;

generating a plurality of training sample pairs according to the training sample images; each training sample pair comprises two sample images;

inputting a plurality of training sample pairs into an initial defect segmentation model to be trained, and calculating a training loss value of a target loss function of the initial defect segmentation model according to the output of the initial defect segmentation model;

and if the training loss value does not meet the preset convergence condition, adjusting parameters of the initial defect segmentation model, inputting a plurality of training sample pairs to the initial defect segmentation model after the parameters are adjusted again, and ending the training until the training loss value meets the preset convergence condition to obtain the trained defect segmentation model.

In one possible implementation, generating a plurality of training samples from training sample images includes:

combining the standard sample image with a plurality of defect sample images in sequence to obtain a plurality of initial sample pairs;

performing data enhancement processing on the plurality of initial sample pairs randomly to obtain a plurality of training sample pairs;

wherein the data enhancement processing comprises at least one of:

randomly exchanging two sample images in the initial sample pair to obtain a plurality of first sample pairs;

performing pixel offset processing on any sample image in the initial sample pair according to a preset pixel offset range to obtain a plurality of second sample pairs;

randomly copying the standard sample images in the initial sample pairs into defect sample images to obtain a plurality of third sample pairs;

pasting the pre-extracted image defects to the standard sample images of the initial sample pairs to obtain a plurality of fourth sample pairs;

the plurality of training sample pairs includes at least one of an initial sample pair, a first sample pair, a second sample pair, a third sample pair, and a fourth sample pair.

In one possible implementation manner, the objective loss function of the initial defect segmentation model includes a Dice loss function and a cross entropy loss function, and the Dice loss function and the cross entropy loss function have different calculation coefficients in the objective loss function.

In one possible implementation, the surface defects include light stick-flower defects if the product to be inspected is a print.

In a second aspect of the present invention, the application provides a surface defect detection device, includes:

the image acquisition module is used for acquiring a surface image to be detected and a standard surface image of a product to be detected;

the image segmentation module is used for carrying out image segmentation on the surface image to be detected and the standard surface image to obtain at least one to-be-detected image block corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image; the number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same;

the defect segmentation module is used for inputting the image block to be detected and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block; each defect division image block is used for representing pixel difference information between one to-be-detected image block and one standard image block;

and the defect detection module is used for dividing the image block based on the defects and determining whether the product to be detected has surface defects.

In a third aspect, the present application provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor, when calling from the memory and executing the computer program, implements the steps of the surface defect detection method shown in any one of the above first aspects.

In a fourth aspect, the present application provides a computer storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the surface defect detection method of any one of the first aspect described above.

In a fifth aspect, the present application provides a computer program product comprising a computer program that, when executed by a processor, performs the steps of the surface defect detection method of any one of the first aspects described above.

The technical scheme provided by the application can at least achieve the following beneficial effects:

the application provides a surface defect detection method, a surface defect detection device, computer equipment and a storage medium, for acquiring a to-be-detected surface image and a standard surface image of a to-be-detected product, image segmentation is firstly carried out on the to-be-detected surface image and the standard surface image, and at least one to-be-detected image block corresponding to the to-be-detected surface image and at least one standard image block corresponding to the standard surface image are obtained. The number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same. Then, the image block to be detected and the standard image block are input into a defect segmentation model trained in advance, a defect segmentation image block corresponding to the image block to be detected and the standard image block is obtained through the defect segmentation model, and the defect segmentation image block can reflect pixel difference information between the image block to be detected and the standard image block. And finally, dividing the image block based on the defects, and determining whether the product to be detected has surface defects. Therefore, the defect segmentation is carried out by using the trained defect segmentation model, and compared with manual detection, the detection efficiency is higher. Moreover, the defect segmentation model is a dual-input neural network model, the input of the model comprises a standard image block corresponding to a standard surface image and a standard image block corresponding to a surface image to be detected, the problem of weak generalization capability of a deep learning model is solved, model training is not needed for different scenes and different products, and the model training cost is reduced. The dual-input defect segmentation model can maintain a good defect detection result even if encountering products which are not in the training data set. In addition, the method and the device perform image segmentation on the surface image to be detected and the standard surface image in advance to obtain at least one image block to be detected corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image, and then process each image block through the defect segmentation model, so that the resource consumption in the image processing process is reduced, the calculation performance requirements of the defect segmentation model on computer equipment when the defect segmentation is performed on the image blocks are reduced, the high-speed real-time operation can be realized on low-end operation equipment, and the surface defect detection efficiency is improved.

Drawings

FIG. 1 is a schematic diagram illustrating a computer device according to an exemplary embodiment of the present application;

FIG. 2 is a schematic flow chart diagram illustrating a method for surface defect detection in accordance with an exemplary embodiment of the present application;

FIG. 3 is a schematic diagram of an image processing flow shown in an exemplary embodiment of the present application;

FIG. 4 is a schematic diagram illustrating a structure of a defect inspection model according to an exemplary embodiment of the present application;

FIG. 5 is a schematic diagram illustrating a training process of a defect segmentation model according to an exemplary embodiment of the present application;

FIG. 6 is a schematic flow chart illustrating the generation of training sample pairs according to an exemplary embodiment of the present application;

FIG. 7 is a schematic flow chart diagram illustrating another method of surface defect detection according to an exemplary embodiment of the present application;

FIG. 8 is a schematic diagram of a surface defect inspection apparatus according to an exemplary embodiment of the present application;

fig. 9 is a schematic structural diagram of another surface defect detecting apparatus according to an exemplary embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more clearly understood, the technical solutions of the present application will be further described in detail with reference to the accompanying drawings and embodiments.

Before explaining the surface defect detection method provided by the embodiment of the present application, an application scenario and an implementation environment of the embodiment of the present application are introduced.

With the increasing requirements of users and manufacturers of industrial products on product quality, the industrial products need to have good appearance, i.e., good surface quality, in addition to satisfying the service performance. However, the generation of surface defects is often unavoidable during the manufacturing process of the product. Defects are generally understood to mean surface defects, surface defects or deviations in area compared with normal products. The surface defect detection refers to detecting defects such as scratches, defects, foreign matter shielding, color pollution, holes and the like on the surface of a product, so that a series of information capable of describing the defects such as the defect type, the outline, the position, the size and the like of the surface defects of the product to be detected can be obtained.

Moreover, the surface defects not only affect the beauty and comfort of the product, but also bring adverse effects to the service performance of the product generally, so that the production enterprises pay great attention to the surface defect detection of the product so as to find the surface defects in time and effectively control the product quality. Furthermore, the technical/operation problems in the production process can be analyzed according to the surface defect detection result, so that the generation of defective products is avoided or reduced, potential trade disputes are prevented, and the honor of enterprises is maintained.

The manual detection is a traditional detection method for the surface defects of the products, but the method has the defects of low random inspection rate, low accuracy, poor real-time performance, low efficiency, high labor intensity and great influence by manual experience and subjective factors.

In addition, the conditions of false alarm and missing detection exist in the detection process of the shallow sticky flower defect of the printed matter based on the traditional image processing algorithm, and the parameters need to be adjusted manually and repeatedly. The surface defect detection method based on the single image input of the deep learning has the problems of low reasoning speed, weak generalization capability and the like; and specific neural network models need to be trained aiming at different scenes and different types of products, so that the detection cost is increased.

Although the surface defect detection technology based on deep learning can overcome the defects of manual detection to a great extent, the types of products to be detected are various, and the surface defects are relatively fine along with the improvement of the process. Therefore, how to construct a stable and efficient surface defect detection method based on machine vision becomes the biggest difficult problem when the machine vision is applied in the field of surface defect detection at present.

Based on the method, the device, the computer equipment and the storage medium, a deep learning technology is adopted to be matched with a proper data enhancement processing and training strategy to train a defect segmentation model, and the defect segmentation model is used for carrying out comparative analysis on the image blocks to be detected and the standard image blocks after image preprocessing during actual detection, so that whether the surface defects exist in the products to be detected or not is determined. So, the defect segmentation model that this application training obtained possesses very strong stability, generalization nature and high efficiency, consequently, combines image preprocessing and defect segmentation model, can improve the detection efficiency of waiting to examine product surface defect, and the homoenergetic can reach stable detection effect under multiple product, the multi-scene.

In an exemplary embodiment, the application can utilize a computer device to perform image preprocessing on a surface image to be inspected of a product to be inspected and a standard surface image of the product to be inspected, and analyze pixel differences of the surface image to be inspected and the standard surface image through a defect segmentation model trained in advance, so as to determine whether the surface defect exists in the product to be inspected.

In one possible implementation, the computer device is configured as shown in FIG. 1, and the computer device 100 includes at least one processor 110, a memory 120, a communication bus 130, and at least one communication interface 140.

The Processor 110 may be a general-purpose Central Processing Unit (CPU), a Network Processor (NP), a microprocessor, or one or more Integrated circuits for implementing the present invention, such as an Application-Specific Integrated Circuit (asic), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a Field-Programmable Gate Array (FPGA), general Array Logic (GAL), or any combination thereof.

Alternatively, processor 110 may include one or more CPUs. The computer device 100 may include a plurality of processors 110. Each of the processors 110 may be a single-Core Processor (CPU) or a multi-Core Processor (CPU).

It should be noted that, in the following description, processor 110 may refer herein to one or more devices, circuits, and/or processing cores that may be configured to process data, such as computer program instructions.

The Memory 120 may be a Read-Only Memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RA M) or other type of dynamic storage device that can store information and instructions, an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical Disc storage, optical Disc storage (including Compact Disc, laser Disc, optical Disc, digital versatile Disc, blu-ray Disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to these.

Alternatively, the memory 120 may be stand-alone and coupled to the processor 110 via the communication bus 130; memory 120 may also be integrated with processor 110.

Communication bus 130 is used to communicate information between components, such as between a processor and memory, and communication bus 120 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one communication bus is shown in FIG. 1, but that does not indicate only one bus or one type of bus.

The communication interface 140 is used for the computer device 100 to communicate with other devices or communication networks. The communication interface 140 includes a wired communication interface or a wireless communication interface. The wired communication interface may be an ethernet interface, for example. The ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The Wireless communication interface may be a Wireless Local Area Network (WLAN) interface, a cellular network communication interface, a combination thereof, or the like.

In some embodiments, the computer device 100 may also include output devices and input devices (not shown in FIG. 1). An output device, which is in communication with the processor 110, may display information in a variety of ways. For example, the output device may be a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) Display device, a Cathode Ray Tube (CRT) Display device, a projector (projector), or the like. The input device is in communication with the processor 110 and may receive user input in a variety of ways. For example, the input device may be a mouse, a keyboard, a touch screen device, or a sensing device, among others.

In some embodiments, the memory 120 is used to store computer programs for performing aspects of the present application, and the processor 110 may execute the computer programs stored in the memory 120. For example, the computer device 100 may invoke and execute computer programs stored in the memory 120 via the processor 110, the steps of the surface defect detection method provided by the embodiment of the application are realized.

It should be understood that the surface defect detecting method provided in the present application may be applied to a surface defect detecting apparatus, which may be implemented as part or all of the processor 110 by software, hardware or a combination of software and hardware, and integrated in the computer device 100.

Next, the technical solutions of the present application and how to solve the above technical problems will be specifically described by embodiments with reference to the accompanying drawings. Various embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. It is to be understood that the embodiments described are only a subset of the embodiments in the present application and not all embodiments.

In an exemplary embodiment, as shown in fig. 2, the present application provides a surface defect detection method, which is exemplified by being applied to the computer device 100 shown in fig. 1, and which may include the following steps:

step 210: and acquiring a surface image to be detected and a standard surface image of a product to be detected.

The product to be detected is any product produced on an industrial production line, and the surface image to be detected is a surface image of the product to be detected shot in real time. Correspondingly, the standard surface image is the same type of surface image as the product to be inspected and has no surface defects.

As an example, if the product to be inspected is a print, the surface defects include light stickers.

It should be noted that, for a batch of products of the same type, the corresponding surface images to be inspected of a plurality of products to be inspected obtained by production can be sequentially obtained in the production process. For the batch of products, the surface defect detection method provided by the present application may be performed by using the same standard surface image, or may be performed by using a plurality of standard surface images that are pre-inspected in the batch of products and have no surface defect, which is not limited in the embodiment of the present application.

It should be understood that the surface of the product to be inspected may or may not be defective. If the surface of the product to be detected has defects, under the condition that the surface image to be detected and the standard surface image are shot by the same image acquisition equipment, the image area with the defects in the surface image to be detected and the corresponding image area in the standard surface image have pixel difference. Based on this, the present application analyzes the pixel difference between the inspected surface image and the standard surface image by the following

steps

220 and 230.

As an example, the image capturing device that captures the surface image to be inspected and the standard surface image may be an RG B camera.

Because the initial image to be detected of the product to be detected shot in real time on the production line has redundant information and noise, if the initial image to be detected is directly used as the image of the surface to be detected of the product to be detected, the information processing amount in the subsequent model processing process can be increased, and the final defect detection result can be influenced.

Therefore, the initial image to be detected and the initial standard image of the product to be detected, which are obtained by actual shooting, can be preprocessed, so that the surface image to be detected and the standard surface image of the product to be detected are obtained.

In one possible implementation, as shown in fig. 3, the implementation process of step 210 may include the following sub-steps:

step 211: and acquiring an initial to-be-detected image and an initial standard image of the to-be-detected product.

Optionally, the initial suspect image and the initial standard image may be denoised to eliminate noise information and redundant information in the images before performing step 213 described below.

The denoising process may be implemented by a filtering algorithm. The filtering algorithm may be mean filtering, median filtering, gaussian filtering, and the like, which is not limited in this application.

Optionally, for convenience of calculation, the initial to-be-inspected image and the initial standard image may be subjected to a graying process to convert the color image captured by the RGB camera into a grayscale image.

It should be understood that the values of the three components R, G, B in the color image determine a specific pixel, and a pixel can have tens of millions of colors; the gray image is also a color image, but the gray image is characterized in that the specific values of the three components R, G, B are consistent, and the pixel value change interval of each pixel point in the gray image is 0 to 255.

The initial image to be detected and the initial standard image are converted into corresponding gray level images, and on the basis of keeping the image contour and the characteristics, the gray level images can still reflect the contour and the texture of the whole image, so that the subsequent defect detection is not influenced.

Step 213: and carrying out image registration processing on the initial image to be detected based on the initial standard image to obtain an intermediate image to be detected.

It should be appreciated that the initial suspect image is subjected to an image registration process to obtain an intermediate suspect image, and that no processing of the initial standard image is required in this step 213. In other words, after step 213 is performed, an intermediate suspect image and an initial standard image are obtained.

In an actual implementation process, due to a shooting angle, a shooting light and the like, the initial to-be-detected image and the initial standard image may not be aligned at a pixel level, and a pixel offset may exist between the two images.

Therefore, it is necessary to perform image registration processing on the initial suspect image and the initial standard image so that the initial suspect image and the initial standard image are aligned in terms of image size, pixel position, and the like.

Step 215: and respectively carrying out image preprocessing on the intermediate image to be detected and the initial standard image to obtain a surface image to be detected and a standard surface image.

The image preprocessing can include at least one of down-sampling processing, redundant point elimination, edge detection, corrosion processing and expansion processing.

In one possible implementation manner, the implementation process of step 215 is: performing edge detection on the intermediate image to be detected and the initial standard image to obtain first edge information of the intermediate image to be detected and second edge information of the initial standard image; performing edge expansion processing on the middle to-be-detected image based on the first edge information to obtain a to-be-detected surface image; and performing edge expansion processing on the initial standard image based on the second edge information to obtain a standard surface image.

The image Edge is the most basic feature of the image, and the Edge (Edge) refers to discontinuity of local characteristics of the image, and a sudden change of information such as gray scale or structure is called as an Edge. Such as abrupt changes in gray scale, abrupt changes in color, abrupt changes in texture, etc.

Specifically, the edge detection operator used in the edge detection operation may be: roberts operator, prewitt operator, sobel operator, laplacian (Laplacian) operator, loG operator (also known as Marr edge detector, or gaussian-Laplacian operator), canny edge detection.

The Roberts operator is also called a cross differential algorithm, which is a gradient algorithm based on cross difference, and edge lines are detected through local difference calculation; the template of the Roberts operator is divided into a horizontal direction and a vertical direction, so that the image edge of plus or minus 45 degrees can be better enhanced. The principle of the Prewitt operator is to utilize the difference generated by the gray value of the pixel point in a specific area to realize edge detection; the Prewitt operator adopts a template to calculate the pixel value of each pixel point in the region, so that the edge detection result is more obvious than a Robert operator in the horizontal direction and the vertical direction. The laplacian operator is a second-order differential operator in an n-dimensional euclidean space and is divided into a four-neighborhood region and an eight-neighborhood region, wherein the four-neighborhood region is used for solving the gradient of four directions of neighborhood center pixels, and the eight-neighborhood region is used for solving the gradient of eight directions. The LoG operator is typically gaussian low pass filtered before applying the laplacian operator. The basic idea of Canny edge detection is to select a certain Gauss filter for smoothing filtering on an image, and then to process by adopting a non-extremum inhibition technology to obtain a final edge image.

As an example, the Sobel operator is adopted to carry out edge detection on the intermediate image to be detected and the initial standard image so as to obtain image edge information.

The Sobel operator adds a weight concept on the basis of the Prewitt operator, the distance between adjacent pixels is considered to be far and near, the influence on the current pixel is different, the closer pixel has a larger influence on the current pixel, and the farther pixel has a smaller influence on the current pixel, so that the image sharpening is realized and the edge contour is highlighted.

Therefore, mutual influence among pixel points can be fully considered by adopting the Sobel operator, the calculated edge information is more accurate, the real outline in the middle of the image to be detected can be better described, and a clear outline edge is displayed.

Further, the image edge dilation is to add pixel values to the edge of an image to expand the entire pixel values, so as to achieve the dilation effect of the image, which may be referred to as pixel interpolation processing.

According to the position relation of the adjacent pixel points referred to in the expansion processing, the image expansion processing comprises horizontal expansion, vertical expansion and omnidirectional expansion.

As an example, the embodiment of the present application may perform an all-around dilation process on edges of the intermediate inspection image and the initial standard image to obtain an inspection surface image and a standard surface image.

It should be understood that the omni-directional expansion process is to traverse the target region in the target image, and only consider the gray values of the target pixel and its neighboring upper, lower, left, right four pixels to determine whether there is an intersection point with the expanded structural element, i.e. whether there is at least one corresponding gray value equal. If the intersection point exists, the pixel point is processed; otherwise, deleting the pixel point.

Therefore, through the omnibearing expansion processing, not only the image boundary point can be expanded, but also the background point contacted with the image edge can be merged into the edge information, so that the real boundary between the image edge and the background information can be better reflected after the expansion processing, and the expansion processing effect is better.

Step 220: and performing image segmentation on the surface image to be detected and the standard surface image to obtain at least one to-be-detected image block corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image.

The number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same.

In other words, when the image segmentation is performed on the surface image to be detected and the standard surface image, the step 220 performs segmentation based on the same image position, so that after the segmentation is completed, the position coordinates of the image area of the image block to be detected in the surface image to be detected are the same as the position coordinates of the image area of the standard image block in the standard surface image; meanwhile, the number of the to-be-detected image blocks and the number of the standard image blocks obtained after division are the same.

In one possible implementation manner, the implementation process of step 220 may be: carrying out pixel value subtraction operation on pixels at the same positions in the surface image to be detected and the standard surface image to obtain an intermediate surface image; analyzing a connected region of the intermediate surface image to obtain at least one candidate defect feature map; and according to the corresponding regional position information of the candidate defect characteristic diagram in the intermediate surface image, performing image segmentation on the surface image to be detected and the standard surface image to obtain the image block to be detected and the standard image block.

Because the surface image to be detected and the standard surface image are an image pair after image registration processing, pixels at corresponding positions in the two surface images describe the same surface characteristic of a product to be detected, and pixel values can be directly subtracted.

It should be understood that if the surface image to be inspected and the standard surface image are both gray scale images, an intermediate surface image obtained after the processing is also a gray scale image. The pixel value of each pixel point in the intermediate surface image is specifically the absolute value of the difference between two pixel values of corresponding positions in the surface image to be detected and the standard surface image.

As an example, at least one candidate defect feature map may be obtained by performing Blob analysis on the intermediate surface image.

In the embodiment of the present application, blob Analysis (Blob Analysis) is to analyze a connected region of the same pixel in the intermediate surface image (the connected region is referred to as a Blob block), so as to obtain a feature map of the connected region, that is, a candidate defect feature map.

Specifically, blob analysis is to perform binarization processing on pixel values of each pixel point in the intermediate surface image to segment a foreground and a background; then, detecting a communication area to obtain a Blob block; and further extracting pixel characteristics from each Blob block to obtain a candidate defect characteristic diagram.

It should be noted that the number of connected components in the intermediate surface image is the same as the number of candidate defect feature maps. That is, if only one connected domain exists in the intermediate surface image, a candidate defect feature map is obtained after Blob analysis; and if a plurality of connected domains exist in the intermediate surface image, obtaining candidate defect feature maps corresponding to the connected domains after Blob analysis.

Further, taking a candidate defect feature map as an example, based on the candidate defect feature map, performing image segmentation on the to-be-detected surface image and the standard surface image to obtain the to-be-detected image block and the standard image block, and the implementation process of obtaining the to-be-detected image block and the standard image block may be as follows: acquiring corresponding region position information of the candidate defect characteristic diagram in the middle surface image to obtain region position information; performing image segmentation on the surface image to be detected according to the region position information to segment a local image corresponding to the region position information from the surface image to be detected to obtain an image block to be detected; and similarly, performing image segmentation on the standard surface image according to the area position information to segment a local image corresponding to the area position information from the standard surface image to obtain a standard image block.

Therefore, the embodiment of the application does not directly input the surface image to be detected and the standard surface image into the trained defect segmentation model, but combines image preprocessing and image segmentation, reduces the processing information amount in the image block, reduces the requirement of the defect segmentation on the operation resource amount of the defect segmentation model, and can realize high-speed real-time operation of the defect segmentation method based on the surface image on low-end operation equipment.

Step 230: and inputting the to-be-detected image block and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block.

Wherein each defect divided image block is used to represent pixel difference information between one image block to be inspected and one standard image block.

The defect segmentation model provided by the present application is a dual-input neural network model, and when performing defect segmentation, the input of the model is required to be two corresponding images, and the order of inputting the two images is not required.

Optionally, a large number of non-defect features exist in the candidate defect feature map, and after small-size image blocks are collected at corresponding positions on the to-be-detected surface image and the standard surface image according to the region position information of each candidate defect feature map, the to-be-detected image blocks and the standard image blocks obtained by dividing can be sorted according to the area size of the candidate defect feature map, so that the to-be-detected image blocks and the standard image blocks corresponding to the large-area candidate defect feature map can be preferentially processed by the defect division model.

As an example, taking a shallow sticky flower defect existing in a printed matter as an example, in order to improve the detection effect of the shallow sticky flower defect and achieve better defect detection accuracy, an encoder of a Seg Former (a semantic segmentation framework that unifies a transform and a lightweight multi-layer perceptron decoder) based on a transform model (a deep learning model that adopts a self-attention mechanism that can allocate different weights according to different importance of each part of input data) is used as a backbone network (backbone) of a defect segmentation model in the embodiments of the present application. The SegFormer uses a hierarchical encoder structure, and can output multi-scale feature maps and fuse the multi-scale feature maps together in a decoder.

Among them, segFormer avoids a complex decoder, and the proposed MLP decoder can aggregate information from different layers. In other words, segFormer is similar to the way of fusing a shallow feature map and a deep feature map in a convolutional neural network model, and aims to capture high-resolution coarse-grained features and low-resolution fine-grained features together so as to optimize a segmentation result.

Moreover, the embodiments of the present application abandon the transform model and position coding in a self-attention-based encoder (SETR), and when the size of the input picture is different from the picture size used in training, it is not necessary to interpolate the position coding vector (when the test resolution is different from the training resolution, performance degradation may result).

Therefore, the simple and effective decoder designed based on the application can achieve the defect detection accuracy rate of more than 90% for the shallow sticky flower defects.

Based on this, the defect segmentation model provided in this application includes an encoder and a decoder, and the implementation process of step 230 may be: inputting an image block to be detected and a standard image block into an encoder, and acquiring a first multi-scale feature map of the image block to be detected under multiple preset resolutions and a second multi-scale feature map of the standard image block under multiple resolutions through the encoder; and inputting the first multi-scale feature map and the second multi-scale feature map into a decoder, performing feature fusion processing on the first multi-scale feature map and the second multi-scale feature map through the decoder, and outputting a defect segmentation image block.

It should be noted that each resolution corresponds to one first feature map and one second feature map, and a plurality of resolutions correspond to a plurality of first feature maps (i.e., first multi-scale feature maps) and a plurality of second feature maps (second multi-scale feature maps).

As an example, refer to the structure diagram of the defect detection model shown in fig. 4, which only exemplifies a processing flow of one to-be-detected image block and one standard image block, and the specific processing flow after the plurality of to-be-detected image blocks and the plurality of standard image blocks are input to the defect segmentation model is similar to this, and is not described again here.

Wherein, the encoder is composed of 4 Transformer blocks (Transformer block 1, transformer block 2, transformer block 3 and Transformer block 4), and the feature map is down-sampled to 1/4, 1/8, 1/16 and 1/32 after passing through each Transformer block respectively. These four different resolution profiles are then merged in the decoder. The decoder can be realized by a convolutional neural network, and all characteristic diagrams in the encoder are respectively subjected to bilinear difference values and convolution operation of 3x3, so that the effect of transposition convolution is achieved, and meanwhile, the influence of a checkerboard effect on a network model is avoided.

Specifically, in the decoder, feature maps of different sizes are first up-sampled to the original size of the image, and then a stitching operation is performed to obtain a stitched feature map including features of high resolution and coarse granularity and features of low resolution and fine granularity. Further, the spliced feature map is input into a subsequent convolution network, and three convolution operations are used for reducing the dimension of the feature map to 64 dimensions, 32 dimensions and 16 dimensions respectively. And finally, inputting the 16-dimensional convolution feature map into the final convolution layer, and reducing the dimension to 2 dimensions.

Wherein 2-dimension here indicates whether the current pixel has a surface defect. Such as whether a light sticking defect exists.

Based on the above explanation, as shown in fig. 5, the process of training the defect segmentation model in advance may include the following steps:

step 510: and acquiring a training sample image of the inspected product.

The inspected product is a product marked with surface defects, and the inspected product and the product to be detected can be the same type of product or different types of products.

In the step, the training sample image comprises a standard sample image and a plurality of defect sample images, each sample image carries a sample label, and the sample label is used for indicating whether the sample image has surface defects or not;

it should be noted that, for a plurality of defect sample images of the same type of product, a plurality of standard sample images may also be combined to form a training sample pair, and the number of the standard sample images is not limited in the embodiment of the present application.

As an example, for the inspected product a, a plurality of defect sample images detected in advance by the inspected product a may be acquired, and two standard sample images of the inspected product a may be acquired at the same time. The two standard sample images are slightly different in imaging angle, lighting information, and the like.

Step 520: a plurality of training sample pairs are generated from the training sample images.

Wherein each training sample pair comprises two sample images.

In one possible implementation manner, as shown in fig. 6, the implementation process of step 520 may be:

step 521: combining the standard sample image with a plurality of defect sample images in sequence to obtain a plurality of initial sample pairs;

that is, each of the initial sample pairs includes one defect sample image and one standard sample image.

Step 523: and randomly performing data enhancement processing on the plurality of initial sample pairs to obtain a plurality of training sample pairs.

In one possible implementation, the data enhancement process includes at least one of:

(1) Randomly exchanging two sample images in the initial sample pair to obtain a plurality of first sample pairs;

it should be noted that each initial sample pair may be represented as (sample image 1, sample image 2), that is, for the defect segmentation model, there is a precedence order between two sample images in the initial sample pair.

As an example, to complete modeling of dual-input image difference from the training strategy, when the defect sample image D and the standard sample image R are input, the initial sample pair (D, R) is randomly exchanged, and no change is made to the defect label, so that f (D, R) = f (R, D) is true, thereby conforming to dual-input difference modeling.

(2) And according to a preset pixel offset range, carrying out pixel offset processing on any sample image in the initial sample pair to obtain a plurality of second sample pairs.

In the sample image registration preprocessing, due to the stability of the used algorithm, hardware limitations, speed requirements and the like, the defective sample image and the standard sample image cannot be aligned at a pixel level, and an offset larger than 5 pixels occurs.

Based on this, in order to make the deep learning network model adaptable to such a situation, when the initial defect detection model is trained, random offset operation is performed on the defect sample image or the standard sample image in the initial sample pair, so that a position offset exists between the two sample images.

Therefore, the robustness of the finally trained defect segmentation model can be stronger by simulating the common image offset phenomenon in the actual scene.

(3) And randomly copying the standard sample images in the initial sample pairs into defect sample images to obtain a plurality of third sample pairs.

That is, the third sample pair includes two identical defect sample images, and in this case, no matter how the images are shifted and rotated, the defect label corresponding to the third sample pair should be a matrix of all zeros, which indicates that there is no difference between the two sample images, i.e., there is no defect.

(4) And pasting the pre-extracted image defects to the standard sample images of the initial sample pairs to obtain a plurality of fourth sample pairs.

Specifically, the marked surface defects in the historical detection data set are extracted and randomly pasted to the standard sample images of the initial sample pair, so that randomly generated defect sample images are obtained. And forming a fourth sample pair through the original standard sample image in the initial sample pair and the randomly generated defect sample image so as to relieve the requirement of the deep learning model on the data volume.

Based on this, the plurality of training sample pairs used for training the initial defect segmentation model in the embodiment of the present application include: at least one of the pair of initial samples, the pair of first samples, the pair of second samples, the pair of third samples, and the pair of fourth samples.

Step 530: inputting a plurality of training sample pairs into an initial defect segmentation model to be trained, and calculating a training loss value of a target loss function of the initial defect segmentation model according to the output of the initial defect segmentation model;

in one possible implementation manner, the implementation process of step 530 may be: and dividing a plurality of training sample pairs into a training set and a verification set, and inputting the training sample pairs in the training set into the initial defect segmentation model in batches, so that the initial defect segmentation model can fully learn the characteristic information of the standard sample image and the defect sample image. And after the training set is adopted for presetting the turn according to the preset iteration training turn, the training loss value of the target loss function of the initial defect segmentation model is calculated by adopting the learning effect of the training samples in the verification set on the inspection initial defect segmentation model.

The training set and the verification set both include a plurality of training sample pairs, and the number of the training sample pairs in the two sets may be the same or different, which is not limited in the embodiments of the present application.

It should be understood that the training loss is determined according to the difference between the pixel information in the defect segmentation image block predicted by the initial defect segmentation model and the input training sample pair of the pre-labeled defect label, and if the difference is reduced, the training result approaches the labeled information, and the training is ended.

As an example, the preset iteration training round may be any number, such as 10 rounds, 20 rounds, and so on.

Further, for a light sticky flower defect, it occupies only a small portion of the entire single color box image, and therefore, there may be a problem of class imbalance when performing classification at the pixel level. Although the foregoing data enhancement process may alleviate some of the problems, the embodiments of the present application still need to constrain the imbalance on the category through an objective loss function, so as to avoid the occurrence of the miss detection situation.

In a possible implementation manner, the objective loss function of the initial defect segmentation model includes a Dice loss function and a cross entropy loss function, and the Dice loss function and the cross entropy loss function have different calculation coefficients in the objective loss function.

It should be understood that Dice is a loss function related to a region, is used for calculating the similarity of two samples, has good performance in a scene where positive and negative samples are seriously unbalanced, and focuses on mining a foreground region in a training process; the cross entropy can measure the difference degree of two different probability distributions in the same random variable, and is expressed as the difference between the real probability distribution and the model prediction probability distribution in machine learning, and the smaller the value of the cross entropy is, the better the model prediction effect is.

As an example, a Dice loss function and a cross entropy (cross entropy) loss function are used to set a target loss function of an initial defect segmentation model, as shown in the following formula (1):

Loss＝1.0*Dice(A，A′)+0.1*CrossEntropy(A，A′) (1)

wherein, the Dice loss function and the cross entropy loss function calculate the pixel position deviation, see the following formula (2) and formula (3)

Dice(A，A′)＝1-2|A&A′|/(|A|+|A′|) (2)

CrossEntropy(A，A′)＝-\sum A′log(A) (3)

In the formula, A' represents a defect segmentation result output by an initial defect segmentation model, namely pixel position information of the shallow sticky flower defect on a defect sample image; a represents the position information of the shallow sticky flower defect manually marked in advance in the defect sample image; the computational coefficient of the Dice loss function is 1, and the computational coefficient of the cross entropy loss function is 0.1.

Therefore, the corresponding pixel position deviation is respectively calculated through the Dice loss function and the cross entropy loss function, and the loss weight (namely, the calculation coefficient) is superposed to set the target loss function of the application, so that the problem of serious unbalance of the positive and negative samples can be solved.

Step 540: and if the training loss value does not meet the preset convergence condition, adjusting parameters of the initial defect segmentation model, inputting a plurality of training sample pairs to the initial defect segmentation model after the parameters are adjusted again, and ending the training until the training loss value meets the preset convergence condition to obtain the trained defect segmentation model.

Adjusting parameters of the initial defect segmentation model comprises adjusting processing parameters of each transform block in the initial defect segmentation model and network parameters of the convolutional neural network.

As an example, the preset convergence condition includes: the training loss value reaches a preset loss threshold value; the training loss value does not change in the preset iteration training turn any more; the total training times reach the preset total training times.

For example, the loss threshold may be 0.1, the iterative training round may be 10 rounds, and the total number of training rounds may be 100 rounds.

Optionally, for the trained defect segmentation model, before online use, some product surface images may be obtained, and used as a test set to test the accuracy of the defect segmentation image block predicted by the defect segmentation model, and under the condition that the test result meets the requirement, the trained defect segmentation model may assist in surface defect detection in an actual scene.

In the process of training the defect segmentation model, the embodiment of the application acquires a plurality of training sample pairs by randomly performing data enhancement processing on the initial sample pairs, so that the diversity of the training sample pairs is improved, the number of the training sample pairs is increased, and the defect segmentation model obtained by final training has better robustness. Meanwhile, the method and the device adopt the Dice loss function and the cross entropy loss function to set the target loss function of the initial defect segmentation model, improve the segmentation accuracy of the defect segmentation model, and avoid the condition of missing detection.

Step 240: and dividing the image block based on the defects, and determining whether the product to be detected has surface defects.

Wherein the defect divided image blocks are used to represent pixel difference information between a to-be-detected image block and a standard image block. In other words, the pixel points in the defect segmentation map can reflect whether the surface defect information exists in the surface image to be detected compared with the standard surface image, and the surface defect information comprises information such as defect position, size and outline.

It should be understood that the number of the defect-divided image blocks is the same as the number of the to-be-detected image blocks or the standard image blocks input into the defect-divided model, and for a defect-divided image block, the number of pixels included therein may or may not be 0.

If the number of the pixels in the defect segmentation image block is 0, the defect segmentation model judges that no pixel difference exists between the corresponding to-be-detected image block and the standard image block; if the number of the pixels in the defect segmentation image block is not 0, the defect segmentation model judges that differences exist between the corresponding to-be-detected image block and the standard image block at the pixels which are not 0.

In one possible implementation manner, the implementation procedure of step 240 may be: judging whether pixel points for describing defect information exist in each defect segmentation image block or not according to the defect segmentation image blocks; if the pixel points describing the defect information do not exist in the defect segmentation image blocks, namely the defect segmentation image blocks are empty, determining that the defect segmentation image blocks do not have surface defects in the corresponding image areas of the to-be-detected surface image; and if the pixel points for describing the defect information exist in the defect segmentation image blocks, namely the defect segmentation image blocks are not empty, determining that the surface defects exist in the corresponding image areas of the defect segmentation image blocks in the surface image to be detected.

Specifically, when all the defect segmentation image blocks corresponding to the to-be-detected image blocks of the to-be-detected surface image do not have pixel points for describing defect information, it is determined that no surface defect exists in the to-be-detected surface image, the corresponding to-be-detected product also does not have surface defect, and the to-be-detected product is a good product.

Further, if there are pixels describing defect information in the defect segmentation image block, the position information of the pixels in the defect segmentation image block can be determined, and the position information is mapped to the surface image to be detected, so that the defect information such as the specific position, size, contour and the like of the surface defect can be determined.

In another possible implementation manner, the implementation procedure of step 240 may be: acquiring pixel intersection information between the defect segmentation image block and the candidate defect feature map; if the pixel intersection information is null, determining that no surface defect exists on the surface of the product to be detected; and if the pixel intersection information is not null, determining that the surface of the product to be detected has surface defects.

It should be understood that, when the pixel values of two pixels are different, the difference between the pixel values can reflect the difference between the two pixels. Therefore, when judging whether the surface defect exists in the product to be detected based on the defect segmentation image block, the candidate defect characteristic diagram obtained by carrying out pixel value subtraction operation and connected region analysis on the pixels at the same position in the surface image to be detected and the standard surface image can be combined, so that the accuracy of the surface defect detection result is improved.

Therefore, if the pixel points describing the defect information exist between the defect segmentation image block and the candidate defect feature map, the pixel intersection information includes the pixel points describing the defect information after the intersection is taken.

Further, if the pixel intersection information is not null, the position information of each pixel point in the intersection in the defect segmentation image block or the candidate defect feature map can be determined, and the position information is mapped to the surface image to be detected, so that the defect information such as the specific position, size, contour and the like of the surface defect can be determined.

In the embodiment of the application, after acquiring the surface image to be detected and the standard surface image of the product to be detected, the computer device performs image segmentation on the surface image to be detected and the standard surface image to obtain at least one image block to be detected corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image. The number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same. Then, the image block to be detected and the standard image block are input into a defect segmentation model trained in advance, a defect segmentation image block corresponding to the image block to be detected and the standard image block is obtained through the defect segmentation model, and the defect segmentation image block can reflect pixel difference information between the image block to be detected and the standard image block. And finally, dividing the image block based on the defects, and determining whether the product to be detected has surface defects. Therefore, the defect segmentation is carried out by using the trained defect segmentation model, and compared with manual detection, the detection efficiency is higher. Moreover, the defect segmentation model is a dual-input neural network model, the input of the model comprises a standard image block corresponding to a standard surface image and a standard image block corresponding to a surface image to be detected, the problem of weak generalization capability of a deep learning model is solved, model training is not needed for different scenes and different products, and the model training cost is reduced. The dual-input defect segmentation model can maintain a good defect detection result even if encountering products which are not in the training data set. In addition, the method and the device perform image segmentation on the surface image to be detected and the standard surface image in advance to obtain at least one to-be-detected image block corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image, and then process each image block through the defect segmentation model, so that the resource consumption in the image processing process is reduced, the calculation performance requirements of the defect segmentation model on computer equipment when performing defect segmentation on the image blocks are reduced, high-speed real-time operation can be realized on low-end operation equipment, and the surface defect detection efficiency is improved.

Based on the foregoing embodiments, in an exemplary embodiment, as shown in fig. 7, the present application further provides another surface defect detecting method, which is exemplified by applying the method to the computer device 100 shown in fig. 1, and the surface defect detecting method is implemented by:

the method comprises the steps of obtaining an initial to-be-detected image and an initial standard image of a product to be detected, and carrying out image registration processing on the initial to-be-detected image based on the initial standard image to obtain an intermediate to-be-detected image. And then, carrying out edge detection and edge expansion processing on the intermediate image to be detected and the initial standard image by using a Sobel operator to obtain a surface image to be detected and a standard surface image. Then, performing feature comparison and Blob analysis on the surface image to be detected and the standard surface image to obtain N candidate defect feature maps; wherein N is more than or equal to 1.

Further, based on the region position information of each candidate defect feature map, sorting each candidate defect feature map according to the size of the image area, and performing image segmentation on the surface image to be detected and the standard surface image based on each candidate defect feature map to obtain N to-be-detected image blocks and N standard image blocks.

The image blocks to be detected and the standard image blocks are in one-to-one correspondence, and the corresponding image blocks to be detected and the standard image blocks describe the same surface area on a product to be detected.

Further, the N to-be-detected image blocks and the N standard image blocks are input into the dual-input defect segmentation model provided by the application according to the sequence of the image block pairs, so that defect segmentation image blocks corresponding to each pair of input to-be-detected image blocks and standard image blocks are obtained.

The defect division image block is used for describing pixel difference information between a to-be-detected image block and a standard image block, wherein the included pixels can be empty, namely, no pixel difference exists between the to-be-detected image block and the standard image block; the pixels included therein may not be empty, i.e. there is a pixel difference between the image block to be examined and the standard image block.

It should be understood that after the N to-be-detected image blocks and the N standard image blocks are processed by the defect segmentation model, N defect-segmented image blocks are obtained.

And further, taking the intersection of the N defect division image blocks and the corresponding N candidate defect feature maps, and judging whether each defect division image block has a surface defect.

And finally, screening the surface defect judgment results of the defect segmentation image blocks to determine whether the product to be detected has surface defects.

It should be noted that, when the above defect detection method is implemented in the embodiment of the present application, the implementation principle and technical effect of the method may refer to the relevant contents of step 210 to step 240 in the above embodiment, and are not described herein again.

It should be understood that, although the steps in the flowcharts related to the above embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the above embodiments may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the surface defect detection method and adopting the same technical concept, the embodiment of the application also provides a surface defect detection device corresponding to the surface defect detection method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method embodiments, so specific limitations in one or more surface defect detection device embodiments provided below can be referred to the limitations on the surface defect detection method in the above, and details are not described herein.

In an exemplary embodiment, as shown in fig. 8, the surface defect detecting apparatus 800 includes:

the image acquisition module 810 is used for acquiring a surface image to be detected and a standard surface image of a product to be detected;

an image segmentation module 820, configured to perform image segmentation on the surface image to be detected and the standard surface image, to obtain at least one to-be-detected image block corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image; the number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same;

the defect segmentation module 830 is configured to input the to-be-detected image block and the standard image block into a defect segmentation model trained in advance, and acquire at least one defect segmentation image block; each defect division image block is used for representing pixel difference information between one to-be-detected image block and one standard image block.

And the defect detection module 840 is used for dividing the image blocks based on the defects and determining whether the product to be detected has surface defects.

In one possible implementation, the image segmentation module 820 includes:

the pixel processing unit is used for carrying out pixel value subtraction operation on pixels at the same positions in the surface image to be detected and the standard surface image to obtain an intermediate surface image;

the area analysis unit is used for carrying out connected area analysis on the intermediate surface image to obtain at least one candidate defect characteristic map;

and the image segmentation unit is used for carrying out image segmentation on the surface image to be detected and the standard surface image according to the corresponding region position information of the candidate defect characteristic diagram in the middle surface image to obtain the image block to be detected and the standard image block.

In one possible implementation, the defect detection module 840 includes:

the information acquisition unit is used for acquiring pixel intersection information between the defect segmentation image block and the candidate defect feature map;

the first determining unit is used for determining that no surface defect exists on the surface of the product to be detected if the pixel intersection information is empty;

and the second determining unit is used for determining that the surface of the product to be detected has surface defects if the pixel intersection information is not null.

In one possible implementation, the defect segmentation model includes an encoder and a decoder; the defect segmentation module 830 includes:

the characteristic extraction unit is used for inputting the image block to be detected and the standard image block into the encoder, and acquiring a first multi-scale characteristic diagram of the image block to be detected under multiple preset resolutions and a second multi-scale characteristic diagram of the standard image block under multiple resolutions through the encoder;

and the feature fusion unit is used for inputting the first multi-scale feature map and the second multi-scale feature map into the decoder, performing feature fusion processing on the first multi-scale feature map and the second multi-scale feature map through the decoder, and outputting the defect segmentation image block.

In one possible implementation, the image obtaining module 810 includes:

the image acquisition unit is used for acquiring an initial to-be-detected image and an initial standard image of a to-be-detected product;

the image registration unit is used for carrying out image registration processing on the initial image to be detected based on the initial standard image to obtain an intermediate image to be detected;

and the image processing unit is used for respectively carrying out image preprocessing on the intermediate image to be detected and the initial standard image to obtain a surface image to be detected and a standard surface image.

In a possible implementation manner, the image processing unit is specifically configured to:

In one possible implementation manner, as shown in fig. 9, the surface defect detecting apparatus 800 further includes:

a sample acquiring module 850, configured to acquire a training sample image of a detected product; the training sample image comprises a standard sample image and a plurality of defect sample images, each sample image carries a sample label, and the sample label is used for indicating whether the sample image has surface defects or not;

a sample pair preparation module 860 for generating a plurality of training sample pairs according to the training sample images; each training sample pair comprises two sample images;

the loss calculation module 870 is configured to input the plurality of training sample pairs into an initial defect segmentation model to be trained, and calculate a training loss value of a target loss function of the initial defect segmentation model according to an output of the initial defect segmentation model;

and a model training module 880, configured to adjust parameters of the initial defect segmentation model if the training loss value does not meet the preset convergence condition, and input the multiple training sample pairs to the initial defect segmentation model after the parameters are adjusted again until the training loss value meets the preset convergence condition, and then end the training to obtain the trained defect segmentation model.

In one possible implementation, the sample pair preparation module includes:

the sample combination unit is used for combining the standard sample image with the plurality of defect sample images in sequence to obtain a plurality of initial sample pairs;

the sample pair processing unit is used for carrying out data enhancement processing on the plurality of initial sample pairs randomly to obtain a plurality of training sample pairs;

wherein the data enhancement processing comprises at least one of:

It should be noted that the respective modules in the surface defect detecting apparatus 800 shown in fig. 8 and 9 can be implemented in whole or in part by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

Further, it should be understood that embodiments of the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product comprises a computer program. The procedures or functions shown in the embodiments according to the present application are wholly or partially generated when the computer program is loaded and executed on a computer device.

The computer program may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, and for example, the computer program may be transmitted from one website, terminal, server, or data center to another website, terminal, server, or data center through a wired or wireless manner.

The computer-readable storage medium can be any available medium that can be accessed by a computing device or a data storage device, such as a server, a data center, etc., that includes one or more of the available media.

It should be understood that the above description is only a specific implementation of the embodiments of the present application, and is not intended to limit the scope of the embodiments of the present application. Any modification, equivalent replacement, improvement and the like made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method of surface defect detection, comprising:

performing image segmentation on the surface image to be detected and the standard surface image to obtain at least one image block to be detected corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image; the number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same;

inputting the to-be-detected image block and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block; each defect division image block is used for representing pixel difference information between one to-be-detected image block and one standard image block;

and determining whether the product to be detected has surface defects or not based on the defect segmentation image blocks.

2. The method according to claim 1, wherein the image segmentation of the surface image to be inspected and the standard surface image to obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image comprises:

carrying out pixel value subtraction operation on pixels at the same positions in the surface image to be detected and the standard surface image to obtain an intermediate surface image;

and according to the corresponding region position information of the candidate defect characteristic diagram in the intermediate surface image, carrying out image segmentation on the surface image to be detected and the standard surface image to obtain the image block to be detected and the standard image block.

3. The method as claimed in claim 2, wherein said dividing image blocks based on said defects to determine whether said product to be inspected has surface defects comprises:

if the pixel intersection information is empty, determining that no surface defect exists on the surface of the product to be detected;

4. The method of claim 1, wherein the defect segmentation model comprises an encoder and a decoder;

inputting the image block to be detected and the standard image block into the encoder, and acquiring a first multi-scale feature map of the image block to be detected under multiple preset resolutions and a second multi-scale feature map of the standard image block under the multiple resolutions through the encoder;

inputting the first multi-scale feature map and the second multi-scale feature map into the decoder, performing feature fusion processing on the first multi-scale feature map and the second multi-scale feature map through the decoder, and outputting the defect segmentation image block.

5. The method according to any one of claims 1 to 4, characterized in that said acquiring of the surface image to be inspected and the standard surface image of the product to be inspected comprises:

acquiring an initial to-be-detected image and an initial standard image of the product to be detected;

and respectively carrying out image preprocessing on the intermediate image to be detected and the initial standard image to obtain the surface image to be detected and the standard surface image.

6. The method of claim 5, wherein said image preprocessing said intermediate inspection image and said initial standard image to obtain said inspection surface image and said standard surface image comprises:

performing edge expansion processing on the intermediate image to be detected based on the first edge information to obtain the surface image to be detected;

and performing edge expansion processing on the initial standard image based on the second edge information to obtain the standard surface image.

7. The method according to any one of claims 1 to 4, wherein the training process of the defect segmentation model comprises:

inputting the training sample pairs into an initial defect segmentation model to be trained, and calculating a training loss value of a target loss function of the initial defect segmentation model according to the output of the initial defect segmentation model;

and if the training loss value does not meet the preset convergence condition, adjusting parameters of the initial defect segmentation model, inputting the training sample pairs into the initial defect segmentation model after the parameters are adjusted again, and ending the training until the training loss value meets the preset convergence condition to obtain the trained defect segmentation model.

8. The method of claim 7, wherein generating a plurality of training samples from the training sample images comprises:

combining the standard sample image with the plurality of defect sample images in sequence to obtain a plurality of initial sample pairs;

wherein the data enhancement processing comprises at least one of:

pasting pre-extracted image defects to the standard sample images of the initial sample pairs to obtain a plurality of fourth sample pairs;

the plurality of training sample pairs includes at least one of the initial sample pair, the first sample pair, the second sample pair, the third sample pair, and the fourth sample pair.

9. The method according to claim 7, wherein the objective loss function of the initial defect segmentation model comprises a Dice loss function and a cross entropy loss function, and the Dice loss function and the cross entropy loss function have different calculation coefficients in the objective loss function.

10. The method according to any one of claims 1 to 4, characterized in that the surface defects comprise low-tack defects if the product to be examined is a print.

11. A surface defect detecting apparatus, comprising:

the image segmentation module is used for carrying out image segmentation on the surface image to be detected and the standard surface image to obtain at least one image block to be detected corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image; the number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same;

the defect segmentation module is used for inputting the to-be-detected image block and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block; each defect division image block is used for representing pixel difference information between one to-be-detected image block and one standard image block;

and the defect detection module is used for dividing the image blocks based on the defects and determining whether the product to be detected has surface defects.

12. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor when retrieving from the memory and executing the computer program implements the steps of the method of any of the preceding claims 1 to 10.

13. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of one of the preceding claims 1 to 10.