CN116777896B - Negative migration inhibition method for cross-domain classification and identification of apparent defects - Google Patents

Negative migration inhibition method for cross-domain classification and identification of apparent defects Download PDF

Info

Publication number
CN116777896B
CN116777896B CN202310830147.2A CN202310830147A CN116777896B CN 116777896 B CN116777896 B CN 116777896B CN 202310830147 A CN202310830147 A CN 202310830147A CN 116777896 B CN116777896 B CN 116777896B
Authority
CN
China
Prior art keywords
domain
source
source domain
cluster
sample
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310830147.2A
Other languages
Chinese (zh)
Other versions
CN116777896A (en
Inventor
伊国栋
曾威
李琎
王阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang University ZJU
Original Assignee
Zhejiang University ZJU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang University ZJU filed Critical Zhejiang University ZJU
Priority to CN202310830147.2A priority Critical patent/CN116777896B/en
Publication of CN116777896A publication Critical patent/CN116777896A/en
Application granted granted Critical
Publication of CN116777896B publication Critical patent/CN116777896B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/096—Transfer learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/50—Extraction of image or video features by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
    • G06V10/763—Non-hierarchical techniques, e.g. based on statistics of modelling distributions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30136—Metal
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Quality & Reliability (AREA)
  • Probability & Statistics with Applications (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a negative migration inhibition method for cross-domain classification and identification of apparent defects. And taking the apparent defect images of various non-injection-molded products as a source domain data set, taking the apparent defect images of the injection-molded products as a target domain data set, and setting an intermediate domain to extract corresponding features. Performing domain information labeling on the source domain and the intermediate domain images; extracting source domain and target domain features by using HOG; double clustering is carried out based on a K-means method; constructing a negative migration inhibition strategy; and establishing a multi-target domain task generalization model based on a multi-domain adaptive algorithm. The invention can effectively inhibit the negative migration phenomenon in the cross-domain classification and identification of apparent defects, and effectively realize the classification and identification of apparent defects of injection molding products.

Description

Negative migration inhibition method for cross-domain classification and identification of apparent defects
Technical Field
The invention belongs to the technical field of machine vision and industrial automation, and particularly relates to a negative migration inhibition method for cross-domain classification and identification of apparent defects.
Background
The surface quality of the injection molding product has a plurality of influencing factors, is closely related to technological parameters, processing environment, cooling conditions and post-treatment process, is easy to generate various defects, and is difficult to quantify because of the complex defect forming principle, and the surface defect identification is always a difficult problem, so that the surface defect detection task is often and efficiently and automatically realized by transferring and learning data in other fields.
The degree of association between different data domains is an important factor limiting the wider application of transfer learning. If the degree of association between the two domains is small, the knowledge learned from the source domain can have very limited effect and even negative impact, namely negative migration phenomenon. When the cross-domain migration learning is performed, the data distribution among different fields is quite different, and the characteristics corresponding to the target domain cannot be effectively extracted from the source domain, so that the difficulty of the cross-domain migration is quite high.
Therefore, how to use the existing remote data source to extract the corresponding features and transfer the corresponding features to the target domain correctly, effectively avoid negative transfer, improve the efficiency and accuracy of transfer learning, and have important significance for the development of transfer learning and the development of industrial automation technology.
Disclosure of Invention
The invention provides a negative migration inhibition method for cross-domain classification and identification of apparent defects, which aims to solve the problems of overlarge negative migration degree and low migration learning accuracy and efficiency in the long-distance migration process.
Aiming at the characteristic of sparse characteristics of the surface defects of the metal product, the simplified HOG characteristics are adopted to describe the surface defects; based on the characteristics, the training of the source domain is divided into a plurality of intermediate domains by adopting a double clustering method, so that the transfer learning task is completed.
The technical scheme of the invention comprises the following steps:
step 1: feature extraction of a source domain and a target domain is carried out by using HOG, and an image set of low-dimensional vector description is obtained;
step 2: performing defect feature division on the target domain and the source domain based on a K-means dual clustering algorithm;
step 3: an automatic encoder is constructed by using an unsupervised forward propagation neural network and supervised learning, is used for capturing important characteristics of input data, and updates parameters of the automatic encoder according to a data set of a source domain and an intermediate domain in an iterative mode;
the source domain is a metal product, the target domain is an injection molding product, samples which do not accord with the target domain are screened out from the source domain and the middle domain according to the weight w, the interference of data which do not accord with the samples of the target domain on the migration learning effect is restrained, the model on the target domain is ensured to learn favorable knowledge, and the remote migration process is completed;
step 4: and constructing a multi-domain task generalization model based on multi-domain and self-adaptive algorithms.
The learned models can achieve better effects on the target domain by knowledge migration from multiple domains, the losses of the learned models are measured based on the distance between the target domain and the training sample domain, and the final models can accurately label the data on the target domain through the characteristics of the source domain by knowledge migration from multiple domains.
The step 1) specifically comprises the following steps:
1.1 Inputting pictures in a source domain and a target domain;
the source domain is a metal product image set and other material product image sets with finished defect labeling, and the target domain is an injection molding product image with defects to be identified;
1.2 Graying the image, and removing RGB information with small defect characteristic effect, namely the gray information is out of the set range, according to the gray range;
1.3 Normalization of the image to reduce the effect of image noise (effects of local shadows, illumination variation, illumination intensity) on the results;
1.4 Traversing pixel points in the image, calculating gradient direction values of the pixel points in the image, acquiring defect and edge information of the image, and weakening the influence of noise;
1.5 Dividing the image into a plurality of small unit blocks, and carrying out weighted projection on each pixel in the unit blocks on the histogram of the gradient direction so as to obtain the gradient direction histogram of each unit block;
the size of the cell block selected here is 20 x 20, the two-dimensional signal is converted into a one-dimensional vector, and a combination of these histograms is used to characterize the target (defect) in the image.
1.6 Recording the average value of the gradient amplitude of each unit block, selecting the unit block with the largest gradient amplitude as the characteristic of the image (to realize the combination of the areas), obtaining the image described by using the low-dimensional vector, and finishing the main characteristic extraction of the image in the source domain and the target domain.
The step 2) specifically comprises the following steps:
2.1 Determining the size of k according to the number of defect types of the target domain and the source domain, wherein k represents the number of defect types;
2.2 Determining experimental parameters: the cluster number is k, and the iteration number is T 1 ,T 2 ；t 1 ,T 2 The iteration times are respectively used as the iteration times of the cluster 1 and the iteration times of the cluster 2;
2.3 Randomly selecting K images from the sample set D as the initial source domain mean vector set { μ } 1 ,μ 2 ,…,μ i ,…,μ K }；
Wherein mu i An ith image in the K images; sample set D is the image set of the low-dimensional vector description in the source domain obtained in step 1), sample set D is For the j-th image in sample set D, m is the number of imagesQuantity j e {1,2, …, m };
2.4 Cluster one: traversing sample set D, calculating samples in sample set DDistance from each source domain mean vector:
determining a cluster c= { C of a sample set from a source domain mean vector 1 ,C 2 ,…,C i ,…,C k And cluster labeling according to the distance nearest principle, i.e. d ji Minimum, sampleDrawing in a cluster C sample corresponding to the nearest source domain mean vector;
the initial quantity of the source domain mean vector set is the initial source domain mean vector set { mu ] obtained in the step 2.3) 1 ,μ 2 ,…,μ i ,…,μ K }；
2.5 Updating the source domain mean vector of step 2.4) by:
obtaining a new source domain mean vector set { mu' 1 ,μ′ 2 ,…,μ′ i ,…,μ′ k }；
2.6 Repeating T) 1 Sub-steps 2.4) and 2.5) to obtain an updated source domain mean vector setAnd outputs the updated clusterCluster C contains all +.>
2.7 Clustering two: traversing sample set E, calculating new samples in sample set EAnd the source domain mean vector set updated in step 2.6)>Distance of (2):
wherein, the sample set E is the image set of the low-dimensional vector description in the target domain obtained in the step 1), and the sample set E is P epsilon {1,2, …, l } for the p-th image in sample set E;
initial amount of +.>Is a source domain mean vector setI-th sample of (a)>On the nth iteration, < >>Is a mean vector setI-th sample of (a)>
From a set of source domain mean vectorsDetermining the cluster a= { a of the new sample set 1 ,A 2 ,…A k And cluster marking according to the distance nearest principle, i.e. d pi Minimum, sample->And (3) drawing in the cluster A sample corresponding to the nearest mean vector:
2.8 Calculating all samples and source domain mean vectorsIs the average of the distances of:
the obtained mean vector setFor updating a set of source domain mean vectors
2.9 Repeating T) 2 Sub-steps 2.7) and 2.8) to obtain a final updated set of mean vectorsOutput k clusters +.>And finishing defect characteristic classification of the target domain and the source domain.
In the step 3), the input data are a source domain data set S, a target domain data set T and an intermediate domain data set I with cluster marks, respectively:
I. source field data set with cluster markersThe data set S is the initial mean vector { mu ] in step 2 1 ,μ 2 ,…,μ i ,…,μ K Determined clusters with cluster markersA set size of n s ＝k；
Wherein,for the sample set D, but after calculating the mean vector, the cluster C with cluster labels is marked>
II, setting a target domain data set as follows:aggregate size n t ＝l；
Wherein,for +.>
Setting a middle domain data set I as follows:
wherein I is j Is the updated cluster in step 2)J e {1,2,…,k}；/>respectively are sample I 1 、I j 、I k Is not limited to the aggregate size of (a).
In the step 3), the automatic encoder specifically comprises the following steps:
3.1 Setting an objective function to perform unsupervised learning training on the automatic encoder:
wherein n is s And n t The set sizes of the source domain data set S and the target domain data set T respectively,the size of the intermediate domain set after clustering one; />And->Reconstructing data for the source domain, the target domain and the intermediate domain, respectively; />Representing samples in cluster C with cluster labels; />Representing samples in the target domain dataset; />Representing samples I in a mid-domain dataset j In (a) and (b)
λx is the weight coefficient of the target domain reconstruction error;
τ′＝{τ′ 1 ,τ′ 2 ,…,τ′ j ,…,τ′ k the factor τ 'is used to represent the importance of different intermediate domains' j ∈[0,1]；
Re(w s ,w I ) Is regularization term [ in order to control the sample selection factor of the source domain and the intermediate domain not to be 0, ensure that enough data can be screened out for knowledge migration, the regularization term in the objective function ] is defined as:
wherein w is s Selecting factors for source domain samples
Selecting factors for mid-domain samplesAnd->
λ s 、λ I Are super parameters for controlling the influence degree of the regular term on the whole;
through the above objective optimization, the automatic encoder can learn the encoding and decoding functions and implicit characteristic representations with more robustness, and can screen more useful data from the source domain and the intermediate domain.
3.2 In order to solve the problem that the implicit feature domain classification task of the sample is not relevant enough caused by the unsupervised learning, the supervised learning is added, and a minimized loss function is provided:
wherein l (·) represents a multi-class cross entropy loss function, f cls (. Cndot.) represents the classification result of the modulo prediction on the samples in brackets;
3.3 Combining the two objective functions of the step 3.1) and the step 3.2) to obtain a final objective function for solving the long-distance transfer learning:
wherein w= { w s ,w I θ is a parameter of the auto encoder, τ '= { τ' 1 ,τ′ 2 ,…,τ′ j ,…,τ′ k }
3.4 Setting iterative epoch, and updating three parameters of the automatic encoder in two steps of iteration:
the random gradient is reduced to update the parameter theta through back propagation;
fixing the parameter theta, and updating the selection factors w and tau';
thereby completing the training of the automatic encoder.
The step 3.4) specifically comprises the following steps:
3.4.1 Input training data set S, T, I and set corresponding hyper-parameters lambda s 、λ t 、λ I ；
3.4.2 Initializing CNN network parameters with data of the source domain:
θ，
3.4.3 Automatic encoder parameter iterative update:
constructing a CNN network, and determining parameters step by the following method:
firstly, fixing a selection factor w and tau', and optimizing CNN parameters theta by using a back propagation and random gradient descent mode;
fixing CNN parameter theta, updating selection factor w, tau':
wherein the sigmoid function is an activation function.
The step 4) is specifically as follows:
4.1 Determining the source domain:
taking a defect data field of a metal product as a source field to obtain N source fields, wherein all the source fields comprise m images;
dividing a source domain into k categories according to label categories, wherein the aggregate size of each category is m j ＝β j m；
Wherein m is j Representing the number of images of the jth source domain, j e {1,2, …, k }; beta= { beta 1 ,β 2 ,…,β k All elements in β add up to 1;
4.2 Defining multi-domain joint experience error:
defining a plurality of intermediate domains of joint generalization errors according to the joint generalization errors of the target domain and the source domain;
given a vector The sum of all elements in (2) is 1, i.e. +.>
The joint empirical error for multiple domains is expressed as:
the multi-domain joint generalization error is expressed as:
wherein,represents a single-domain empirical error based on the assumption h, ε j (h) Representing a single-domain generalization error based on a hypothesis h, h being a hypothesis that the automatic encoder herein can accurately identify a picture defect in the target domain;
4.3 Defining a probability model for a single source domain):
the weight of each source domain distribution is the same as the weight of the experience error; for any j ε {1,2, …, N }, S j Representing a size beta j m is the data domain sample of the source domain, from D j Sampling to obtain; d (D) j For D α Sample D of (2) α Representing a hybrid distribution of N source domains
For a specific weight alpha j ，And epsilon j (h) Respectively represent that the assumption h is in the data field S j Experience errors and generalization errors above; then:
wherein ε is a value infinitely close to zero;
by taking probabilitiesThe minimum value of the total number of the components,the experience error and the generalization error are closest; the condition for taking the minimum value is alpha j ＝β j The generalization capability of the model is best at this time;
4.4 Determining the total number of sample training and performing model training:
the total number of training samples is:
alpha is then j And beta j The target values of (2) are:
will be alpha j ＝β j As a constraint for retraining the step 3) automatic encoder model, let α at training j And beta j Approach to target valueThe generalization capability of the model on training samples is better; the larger m is, the more approximate the generalization error of the model is to the empirical error;
and obtaining a multi-target domain task generalization model based on a multi-domain and self-adaptive algorithm through training.
The invention has the beneficial effects that:
the feature extraction method based on HOG can convert two-dimensional signals of the image into one-dimensional vectors, improves the accuracy of clustering division and reduces the computational complexity;
the invention is based on the double clustering of the K mean value method, characterized by that can divide the training of the source domain into several intermediate domains through the method of the double clustering, thus finish the transfer learning task of the transmission, inhibit the emergence of the negative migration;
the multi-target domain task generalization model based on the multi-domain and self-adaptive algorithm is characterized in that the characteristics of a plurality of remote source domains are obtained, the generalization capability of the model is improved, training data on non-target domains are reduced, the generalization error of the model on target domain tasks is reduced to a certain extent, and the occurrence of negative migration is restrained.
Drawings
FIG. 1 is a gradient direction division diagram;
FIG. 2 is HOG feature extraction of an image;
FIG. 3 is a graph of K value versus model accuracy;
FIG. 4 is a CNN network architecture;
fig. 5 is an effect diagram of the transfer learning, (a) corresponds to the "dent" defect, and (b) corresponds to the "flash" defect.
Detailed Description
The invention will be further described with reference to the drawings and examples,
Step 1: source domain and target domain feature extraction using HOG:
dividing the whole image into a plurality of small unit blocks, generating a direction gradient histogram by each unit block, extracting the characteristics of the image and calculating the weight;
the size of the cell block selected here is 20 x 20, the two-dimensional signal is converted into a one-dimensional vector, and a combination of these histograms is used to characterize the target (defect) in the image.
The HOG-based feature extraction method in step 1 specifically includes the following steps:
1.1 inputting pictures in a source domain and a target domain;
1.2 graying the image, and removing RGB information with small defect characteristic effect (gray information outside a specific range) according to a gray range;
1.3 reducing the influence of image noise (local shadows, illumination changes, influence of illumination intensity) on the result, normalizing the whole image;
1.4 traversing pixel points in the image, calculating gradient direction values of the pixel points in the image, acquiring defect and edge information of the image, and weakening the influence of noise;
1.5 dividing the image into small unit blocks, and carrying out weighted projection on the histogram of each pixel gradient direction in the unit blocks so as to obtain the gradient direction histogram of the unit blocks.
1.6 recording the average value of the gradient amplitude of each unit block, selecting the unit block with the largest gradient amplitude as the characteristic of the image (thereby realizing the combination of areas), obtaining the image described by using the low-dimensional vector, and completing the extraction of the main characteristic of the image.
As shown in fig. 1, it is characterized as a 9-dimensional feature map; the HOG feature extraction flow is shown in figure 2.
Step 2: double clustering based on K-means method: dividing the sample division cluster in the source domain data set into C= { C based on the K-means method by utilizing the merging region obtained in the step 1 1 ,C 2 ,…,C k And minimizes the square error of:
wherein the method comprises the steps ofIs cluster C i Is a mean vector of (c).
The K-means dual clustering algorithm in the step 2 specifically comprises the following steps:
2.1, determining the size of k according to the number of defect types of the target domain and the source domain, wherein k represents the number of defect types;
2.2 input sample set:
the sample set is an image set of low-dimensional vector descriptions in the source domain obtained in the step 1);for the j-th image in sample set D, m is the imageNumber of pieces;
2.3 determining experimental parameters: cluster number k, iteration number T 1 ,T 2 ；
2.4 randomly selecting K images from the sample set D as the initial mean vector { μ } 1 ,μ 2 ,…,μ i ,…,μ K }；
μ i An image selected from the set of images of the low-dimensional vector description;
2.5 Cluster one: traversing the sample set D, and calculating the distance between the sample and each initial mean vector:
from the initial mean vector { mu } 1 ,μ 2 ,…,μ i ,…,μ K Determining clusters { C } of sample sets 1 ,C 2 ,…,C k Marking it, and grouping the samples according to the distance nearest principleGrouping into corresponding clusters:
2.6 calculate a new mean vector from 2.5:
2.7 clustering two: traversing sample set E:calculate new sample->Distance from each new mean vector:
the sample set E is an image set of low-dimensional vector descriptions in the target domain obtained in the step 1);
calculating the average value of the distances between the samples and each new average value vector:
determining clusters and labels of a new sample set according to the new mean value vector and determining the distance mean valueThe latest principle is to sample->Grouping into corresponding clusters:
2.8 output of n clusters A 1 ,A 2 ,…A k 。
The effect of K value selection on model accuracy is shown in FIG. 3.
Step 3: constructing a negative migration inhibition strategy: three data sets were set as follows:
I. the source domain data set with cluster mark obtained from step 2With a set size of n s ；
Wherein,for +.>
II, setting a target domain data set as follows:the size of n t ；
Wherein,for sample set E/>
Setting a middle domain data set as follows:
is large enough; wherein:
p s (x)≠p t (x)≠p I (x),p I (y|x)≠p s (y|x)
an unsupervised forward propagation neural network and supervised learning are used for constructing an automatic encoder, the automatic encoder is used for capturing important characteristics of input data, parameters of the automatic encoder are updated according to a data set of a source domain and an intermediate domain in an iterative mode, samples which do not accord with a target domain are screened out from the source domain and the intermediate domain according to a weight w, interference of the data on a transfer learning effect is restrained, and models on the target domain are guaranteed to learn favorable knowledge, so that a remote transfer process is completed;
the automatic encoder in step 3 specifically includes the following steps:
3.1. setting an objective function to perform unsupervised learning training on the automatic encoder:
wherein n is s And n t The aggregate sizes of the source domain and the target domain respectively,for the mid-domain set size after cluster one, and->Reconstructing data for the self-encoders of different fields, respectively;
w s ,sample selection factors that are the source domain and intermediate domain;
and->
λ t And reconstructing the weight coefficient of the error for the target domain.
τ′＝{τ′ 1 ,τ′ 2 ,…,τ′ j ,…,τ′ k The factor τ 'is used to represent the importance of different intermediate domains' j ∈[0,1]. End Re (w) s ,w I ) Is a regularization term that, in order to control the selection factor of these domains to be not all 0, keeps enough data to be screened out for knowledge migration, the regularization term in the objective function is defined as:
wherein lambda is s 、λ I Are super-parameters for controlling the degree of influence of the regularization term on the whole. Through the above objective optimization, the self-encoder can learn more robust codec functions and implicit feature representations from the sourceThe domain and the middle domain screen more useful data.
3.2 adding supervised learning to solve the problem that unsupervised learning results in insufficient correlation of sample implicit feature field classification tasks, and additionally providing a minimization loss function:
wherein l (·) represents a multi-class cross entropy loss function, f cls (. Cndot.) represents the classification result of a model for a sample prediction.
And 3.3, combining the two objective functions to obtain a final objective function for solving the remote transfer learning:
wherein w= { w s ,w I θ is a parameter of the auto encoder, τ '= { τ' 1 ,τ′ 2 ,…,τ′ j ,…,τ′ k }
3.4 iterative update of auto encoder parameters:
a CNN network was constructed as shown in fig. 4, taking two slabs, one of which is w, τ' and the other of which is θ. When the selection factor w, tau' is fixed, optimizing the parameter theta in a counter-propagation mode;
conversely, when the CNN parameter θ is fixed, iterate w, τ' with the following method:
wherein the sigmoid function is an activation function.
The step 3.4 specifically comprises the following steps:
3.4.1 inputting training data S, T, I and setting corresponding super parameter lambda s 、λ t 、λ I ；
3.4.2 data initialization network parameters using source domain:
θ，
3.4.3 setting iterative epoch, and updating three parameters in two steps:
the random gradient is reduced to update the parameter theta through back propagation;
fixing the parameter theta, and updating the selection factors w and tau';
3.4.4. and outputting the trained automatic encoder.
Step 4: constructing a multi-domain task generalization model based on a multi-domain and self-adaptive algorithm: the learned models can achieve better effects on the target domain by knowledge migration from multiple domains, the losses of the learned models are measured based on the distance between the target domain and the training sample domain, and the final models can accurately label the data on the target domain through the characteristics of the source domain by knowledge migration from multiple domains.
The multi-target domain task generalization model based on the adaptive algorithm in the step 4 specifically comprises the following steps:
4.1 determining a source domain:
n independent data domains are used as source domains, each data and target domain S j With a corresponding data distribution D j . A total of m marked data, each data field having a set size of m j ＝β j m；
4.2 define multi-domain joint experience error:
defining the joint generalization errors of a plurality of data domains according to the joint generalization errors of the target domain and the source domain, and giving a vectorThe sum of all elements is 1, i.e. +.>The joint empirical error for multiple domains is expressed as:
the multi-domain joint generalization error is expressed as:
wherein,represents a single-domain empirical error, ε, based on the coding function h j (h) Representing a single-domain generalization error based on the encoding function h;
4.3 define probability models for a single source domain:
using D α To represent a hybrid distribution of N data fields, each data field distribution having the same weight as the empirical error. For any j ε {1,2, …, N }, S j Representing a size beta j m, from D j And (5) sampling. For a particular weight vectorAnd epsilon j (h) The empirical error and the generalization error of the coding function h over the data domain are represented, respectively. Then:
the empirical error and the generalization error are closest to each other, and probability is calculatedThe minimum of the two pieces of the material is required to be taken, the condition that the minimum value can be obtained is alpha j ＝β j The generalization ability of the model is best. Epsilon is a value infinitely close to zero.
4.4, determining the total number of sample training and performing model training:
the total number of training samples is
Alpha is then j And beta j The target values of (2) are:
step 4.3) as a constraint for retraining the automatic encoder model of step 3), let α at training j And beta j Approaching the target value, the generalization ability of the model 3) on the training samples is better.
And (3) inputting the source domain samples of 4.1 into the model to perform fitting training, wherein the larger m is, the more approximate the generalization error of the model is to the empirical error. And obtaining a final model through training.
And inputting the injection molding product picture to be tested into the trained model, and outputting an injection molding product picture defect identification picture, wherein the implementation effect is shown in figure 5.
While the invention has been described with reference to preferred embodiments, it is not intended to be limiting. Those skilled in the art will appreciate that various modifications and adaptations can be made without departing from the spirit and scope of the present invention. Accordingly, the scope of the invention is defined by the appended claims.

Claims (6)

1. A negative migration inhibition method for cross-domain classification and identification of apparent defects is characterized by comprising the following steps,
step 1: feature extraction of a source domain and a target domain is carried out by using HOG, and an image set of low-dimensional vector description is obtained;
step 2: performing defect feature division on the target domain and the source domain based on a K-means dual clustering algorithm;
step 3: an automatic encoder is constructed by using an unsupervised forward propagation neural network and supervised learning, is used for capturing important characteristics of input data, and updates parameters of the automatic encoder according to a data set of a source domain and an intermediate domain in an iterative mode;
step 4: constructing a multi-domain task generalization model based on a multi-domain and self-adaptive algorithm;
the step 2) specifically comprises the following steps:
2.1 Determining the size of k according to the number of defect types of the target domain and the source domain, wherein k represents the number of defect types;
2.2 Determining experimental parameters: the cluster number is k, and the iteration number is T 1 ,T 2 ；
2.3 Randomly selecting K images from the sample set D as the initial source domain mean vector set { μ } 1 ,μ 2 ,…,μ i ,…,μ K }；
Wherein mu i An ith image in the K images; sample set D is the image set of the low-dimensional vector description in the source domain obtained in step 1), sample set D is For the j-th image in sample set D, m is the number of images, j ε {1,2, …, m };
2.4 Cluster one: traversing sample set D, calculating samples in sample set DDistance from each source domain mean vector:
determining a cluster c= { C of a sample set from a source domain mean vector 1 ,C 2 ,…,C i ,…,C k And cluster labeling according to the distance nearest principle, i.e. d ji Minimum, sampleDrawing in a cluster C sample corresponding to the nearest source domain mean vector;
the initial quantity of the source domain mean vector set is the initial source domain mean vector set { mu ] obtained in the step 2.3) 1 ,μ 2 ,…,μ i ，…,μ K }；
2.5 Updating the source domain mean vector of step 2.4) by:
obtaining a new source domain mean vector set { mu' 1 ,μ′ 2 ,…,μ′ i ,…,μ′ k }；
2.6 Repeating T) 1 Sub-steps 2.4) and 2.5) to obtain an updated source domain mean vector setAnd outputs the updated clusterCluster C contains all +.>
2.7 Clustering two: traversing sample set E, calculating new samples in sample set EAnd the source domain mean vector set updated in step 2.6)>Distance of (2):
wherein, the sample set E is the image set of the low-dimensional vector description in the target domain obtained in the step 1), and the sample set E is P epsilon {1,2, …, l } for the p-th image in sample set E;
from a set of source domain mean vectorsDetermining the cluster a= { a of the new sample set 1 ,A 2 ,…A k And cluster marking according to the distance nearest principle, i.e. d pi Minimum, sample->And (3) drawing in the cluster A sample corresponding to the nearest mean vector:
calculating average vectors of all samples and source domains in sample set EIs the average of the distances of:
the obtained mean vector setFor updating a set of source domain mean vectors
2.8 Repeating T) 2 Sub step 2.7), obtaining the final updated mean vector setOutput k clusters +.>And finishing defect characteristic classification of the target domain and the source domain.
2. The negative migration suppression method for cross-domain classification and identification of apparent defects according to claim 1, wherein the step 1) specifically comprises:
1.1 Inputting pictures in a source domain and a target domain;
the source domain is a metal product image set with finished defect labeling, and the target domain is an injection molding product image with defects to be identified;
1.2 Graying the image, and removing RGB information of which the gray information is out of a set range according to the gray range;
1.3 Normalizing the image;
1.4 Traversing pixel points in the image, and calculating gradient direction values of the pixel points in the image;
1.5 Dividing the image into a plurality of small unit blocks, and carrying out weighted projection on each pixel in the unit blocks on the histogram of the gradient direction so as to obtain the gradient direction histogram of each unit block;
1.6 Recording the average value of the gradient amplitude of each unit block, selecting the unit block with the largest gradient amplitude as the characteristic of the image, obtaining the image described by using the low-dimensional vector, and finishing the main characteristic extraction of the image in the source domain and the target domain.
3. The negative migration suppression method for cross-domain classification recognition of apparent defects according to claim 1, wherein in the step 3), the input data are a source domain data set S, a target domain data set T and an intermediate domain data set I with cluster marks, respectively:
I. source field data set with cluster markersThe data set S is the initial mean vector { mu ] in step 2 1 ,μ 2 ，…，μ i ，…，μ K Determined clusters with cluster markersA set size of n s ＝k；
Wherein,marking the original sample set D with cluster C with cluster mark +.>
II, setting a target domain data set as follows:aggregate size n t ＝l；
Wherein,for +.>
Setting a middle domain data set I as follows:
wherein I is j Is the updated cluster in step 2)J e {1,2, …, k }; />Respectively are sample I 1 、I j 、I k Is not limited to the aggregate size of (a).
4. The negative migration suppression method for cross-domain classification recognition of apparent defects according to claim 1, wherein in the step 3), the automatic encoder comprises the following steps:
3.1 Setting an objective function to perform unsupervised learning training on the automatic encoder:
wherein n is s And n t The set sizes of the source domain data set S and the target domain data set T respectively,the size of the intermediate domain set after clustering one; />And->Reconstructing data for the source domain, the target domain and the intermediate domain, respectively; />Representing samples in cluster C with cluster labels; />Representing samples in the target domain dataset; />Representing samples I in a mid-domain dataset j Is->
λ t Reconstructing a weight coefficient of the error for the target domain;
τ′＝{τ′ 1 ，τ′ 2 ，…,τ′ j ，…，τ′ k the factor τ 'is used to represent the importance of different intermediate domains' j ∈[0，1]；
Re(w s ,w I ) Is a regularization term defined as:
wherein w is s Selecting factors for source domain samples
Selecting a factor for mid-domain samples>And is also provided with
λ s 、λ I Are super parameters for controlling the influence degree of the regular term on the whole;
3.2 Adding supervised learning, another proposed minimization of the loss function:
wherein l (·) represents a multi-class cross entropy loss function, f cls (. Cndot.) represents the classification result of the modulo prediction on the samples in brackets;
3.3 Combining the two objective functions of the step 3.1) and the step 3.2) to obtain a final objective function for solving the long-distance transfer learning:
wherein w= { w s ,w I θ is a parameter of the auto encoder, τ '= { τ' 1 ,τ′ 2 ,…,τ′ j ,…,τ′ k }
3.4 Setting iterative epoch, and updating three parameters of the automatic encoder in two steps of iteration:
the random gradient is reduced to update the parameter theta through back propagation;
fixing the parameter theta, and updating the selection factors w and tau';
thereby completing the training of the automatic encoder.
5. The negative migration suppression method for cross-domain classification and identification of apparent defects according to claim 4, wherein the step 3.4) specifically comprises:
3.4.1 Input training data set S, T, I and set corresponding hyper-parameters lambda s 、λ t 、λ I ；
3.4.2 Initializing CNN network parameters with data of the source domain:
θ，
3.4.3 Automatic encoder parameter iterative update:
constructing a CNN network, and determining parameters step by the following method:
firstly, fixing a selection factor w and tau', and optimizing CNN parameters theta by using a back propagation and random gradient descent mode;
fixing CNN parameter theta, updating selection factor w, tau':
wherein the sigmoid function is an activation function.
6. The negative migration suppression method for cross-domain classification and identification of apparent defects according to claim 1, wherein the step 4) specifically comprises:
4.1 Determining the source domain:
taking a defect data field of a metal product as a source field to obtain N source fields, wherein all the source fields comprise m images;
dividing a source domain into k categories according to label categories, wherein the aggregate size of each category is m j ＝β j m；
Wherein m is j Representing the number of images of the jth source domain, j e {1,2, …, k }; beta= { beta 1 ,β 2 ,…,β k All elements in β add up to 1;
4.2 Defining multi-domain joint experience error:
defining a plurality of intermediate domains of joint generalization errors according to the joint generalization errors of the target domain and the source domain;
given a vector The sum of all elements in (2) is 1, i.e. +.>
The joint empirical error for multiple domains is expressed as:
the multi-domain joint generalization error is expressed as:
wherein,represents a single-domain empirical error based on the assumption h, ε j (h) Representing a single-domain generalization error based on a hypothesis h, h being a hypothesis that the automatic encoder herein can accurately identify a picture defect in the target domain;
4.3 Defining a probability model for a single source domain):
for any j ε {1,2, …, N }, S j Representing a size beta j m is the data domain sample of the source domain, from D j Sampling to obtain; d (D) j For D α Sample D of (2) α Representing a hybrid distribution of N source domains
For a specific weight alpha j ，And epsilon j (h) Respectively represent that the assumption h is in the data field S j Experience errors and generalization errors above; then:
wherein ε is a value infinitely close to zero;
by taking probabilitiesMinimizing the experience error and generalization error to be closest; /> The condition for taking the minimum value is alpha j ＝β j ；
4.4 Determining the total number of sample training and performing model training:
the total number of training samples is:
alpha is then j And beta j The target values of (2) are:
will be alpha j ＝β j As a constraint for retraining the step 3) automatic encoder model, let α at training j And beta j Approach to target valueThe generalization capability of the model on training samples is better;
and obtaining a multi-target domain task generalization model based on a multi-domain and self-adaptive algorithm through training.
CN202310830147.2A 2023-07-07 2023-07-07 Negative migration inhibition method for cross-domain classification and identification of apparent defects Active CN116777896B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310830147.2A CN116777896B (en) 2023-07-07 2023-07-07 Negative migration inhibition method for cross-domain classification and identification of apparent defects

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310830147.2A CN116777896B (en) 2023-07-07 2023-07-07 Negative migration inhibition method for cross-domain classification and identification of apparent defects

Publications (2)

Publication Number Publication Date
CN116777896A CN116777896A (en) 2023-09-19
CN116777896B true CN116777896B (en) 2024-03-19

Family

ID=87994548

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310830147.2A Active CN116777896B (en) 2023-07-07 2023-07-07 Negative migration inhibition method for cross-domain classification and identification of apparent defects

Country Status (1)

Country Link
CN (1) CN116777896B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117788467B (en) * 2024-02-26 2024-04-26 宝鸡百润万德钛业有限公司 Defect image classification method for titanium metal plate

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110111297A (en) * 2019-03-15 2019-08-09 浙江大学 A kind of injection-molded item surface image defect identification method based on transfer learning
CN110647904A (en) * 2019-08-01 2020-01-03 中国科学院信息工程研究所 Cross-modal retrieval method and system based on unmarked data migration
CN110728294A (en) * 2019-08-30 2020-01-24 北京影谱科技股份有限公司 Cross-domain image classification model construction method and device based on transfer learning
CN111860592A (en) * 2020-06-16 2020-10-30 江苏大学 Solar cell defect classification detection method under condition of few samples
CN114912012A (en) * 2021-12-27 2022-08-16 天翼数字生活科技有限公司 Method and system for content recommendation through transfer learning based on source domain features
US11544796B1 (en) * 2019-10-11 2023-01-03 Amazon Technologies, Inc. Cross-domain machine learning for imbalanced domains
CN116051479A (en) * 2022-12-26 2023-05-02 福州大学 Textile defect identification method integrating cross-domain migration and anomaly detection
CN116385808A (en) * 2023-06-02 2023-07-04 合肥城市云数据中心股份有限公司 Big data cross-domain image classification model training method, image classification method and system

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10635979B2 (en) * 2018-07-20 2020-04-28 Google Llc Category learning neural networks
CN113361549A (en) * 2020-03-04 2021-09-07 华为技术有限公司 Model updating method and related device
US11423307B2 (en) * 2020-06-03 2022-08-23 International Business Machines Corporation Taxonomy construction via graph-based cross-domain knowledge transfer
CN113221905B (en) * 2021-05-18 2022-05-17 浙江大学 Semantic segmentation unsupervised domain adaptation method, device and system based on uniform clustering and storage medium
KR102387663B1 (en) * 2021-08-30 2022-04-19 서울대학교산학협력단 Apparatus for fault diagnosis using domain adaptation with semantic clustering algorithm and method for fault diagnosis using the same

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110111297A (en) * 2019-03-15 2019-08-09 浙江大学 A kind of injection-molded item surface image defect identification method based on transfer learning
CN110647904A (en) * 2019-08-01 2020-01-03 中国科学院信息工程研究所 Cross-modal retrieval method and system based on unmarked data migration
CN110728294A (en) * 2019-08-30 2020-01-24 北京影谱科技股份有限公司 Cross-domain image classification model construction method and device based on transfer learning
US11544796B1 (en) * 2019-10-11 2023-01-03 Amazon Technologies, Inc. Cross-domain machine learning for imbalanced domains
CN111860592A (en) * 2020-06-16 2020-10-30 江苏大学 Solar cell defect classification detection method under condition of few samples
CN114912012A (en) * 2021-12-27 2022-08-16 天翼数字生活科技有限公司 Method and system for content recommendation through transfer learning based on source domain features
CN116051479A (en) * 2022-12-26 2023-05-02 福州大学 Textile defect identification method integrating cross-domain migration and anomaly detection
CN116385808A (en) * 2023-06-02 2023-07-04 合肥城市云数据中心股份有限公司 Big data cross-domain image classification model training method, image classification method and system

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
A Survey on Negative Transfer;Wen Zhang et.al;《IEEE/CAA Journal of Automatica Sinica》;第10卷(第2期);305 - 329 *
Reducing Negative Transfer Learning via Clustering for Dynamic Multiobjective Optimization;Jianqiang Li et.al;《IEEE Transactions on Evolutionary Computation》;第26卷(第5期);1102 - 1116 *
基于数据特征对齐的迁移学习方法研究;杨洪伟;《万方学位论文》;全文 *

Also Published As

Publication number Publication date
CN116777896A (en) 2023-09-19

Similar Documents

Publication Publication Date Title
CN110111297B (en) Injection molding product surface image defect identification method based on transfer learning
Zeng et al. Traffic sign recognition using kernel extreme learning machines with deep perceptual features
Springenberg et al. Improving deep neural networks with probabilistic maxout units
CN111723674B (en) Remote sensing image scene classification method based on Markov chain Monte Carlo and variation deduction and semi-Bayesian deep learning
CN114492574A (en) Pseudo label loss unsupervised countermeasure domain adaptive picture classification method based on Gaussian uniform mixing model
CN116777896B (en) Negative migration inhibition method for cross-domain classification and identification of apparent defects
CN110188827B (en) Scene recognition method based on convolutional neural network and recursive automatic encoder model
CN112733965B (en) Label-free image classification method based on small sample learning
Yang et al. Underwater acoustic research trends with machine learning: general background
CN113159072B (en) Online ultralimit learning machine target identification method and system based on consistency regularization
CN115761240B (en) Image semantic segmentation method and device for chaotic back propagation graph neural network
CN116342942A (en) Cross-domain target detection method based on multistage domain adaptation weak supervision learning
CN114357221B (en) Self-supervision active learning method based on image classification
CN116912568A (en) Noise-containing label image recognition method based on self-adaptive class equalization
CN116051410A (en) Wool cashmere fiber surface morphology structure diagram identification method based on image enhancement
CN116844217A (en) Image processing system and method for generating face data
Xu et al. An SVM-based AdaBoost cascade classifier for sonar image
Eerola et al. Survey of automatic plankton image recognition: challenges, existing solutions and future perspectives
CN113313179A (en) Noise image classification method based on l2p norm robust least square method
CN113297964A (en) Video target recognition model and method based on deep migration learning
CN110717402B (en) Pedestrian re-identification method based on hierarchical optimization metric learning
CN116665039A (en) Small sample target identification method based on two-stage causal intervention
CN116563602A (en) Fine granularity image classification model training method based on category-level soft target supervision
Ristiawanto et al. Wood classification with transfer learning method and bottleneck features
CN108985371B (en) Image multi-resolution dictionary learning method and application thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant