CN116935388A - Skin acne image auxiliary labeling method and system, and grading method and system - Google Patents

Abstract

The invention discloses an auxiliary labeling method and system, a grading method and system for skin acne images, belongs to the technical field of image processing, and aims to solve the technical problems that in the prior art, a new model and an old model are compared to introduce additional training expenditure to cause low training efficiency, and an excellent trained deep learning model is often based on a labeled specific high-quality data set and has no better migration learning effect. The method acquires high-quality data through an Oracle labeling period, and is different from a mode of focusing on changing a screening method to improve the quality of a labeling sample so as to improve the accuracy of a model; by introducing supervised learning and semi-supervised learning, the unlabeled data to be labeled can be better utilized, so that the accuracy of the model is improved, and the problem of low model prediction accuracy caused by insufficient high-quality data sets is solved to a certain extent.

Description

Skin acne image auxiliary labeling method and system, and grading method and system

Technical Field

The invention belongs to the technical field of image processing, relates to auxiliary labeling and grading of skin acne, and particularly relates to an auxiliary labeling method and system, and a grading method and system of skin acne images.

Background

Acne is a common skin disorder in adults and young people. Acne is a multifactorial disease of the pilo-sebaceous unit, with clinical manifestations ranging from mild acne type to acne with systemic symptoms. The main manifestations of acne are comedones, papules, pustules, nodules, cysts, and the like. According to the investigation, 80% of adolescents suffer from acne, and this chronic disease may in some cases persist to adulthood. More seriously, the face of a patient suffering from severe acne disease inevitably leaves scars and pigmentation, which can affect the appearance and mood of the patient. Thus, an accurate graded assessment of acne severity and corresponding treatment regimen is essential to the patient.

Acne severity grading typically combines a standard-based lesion count with an empirically based whole-map assessment, a classification method that is divided by analyzing the number of lesions on a patient's face and combining the physician's experience of judging the patient's whole face. The criteria for judgment are different, and the classification of the same patient is not identical. Common classification methods such as Hayashi classification classify patient facial pictures into four classes, including: "Cold" (light), "Modrate" (medium), "Severe" (severe) and "very Severe" (very severe) four grades.

With the development of artificial intelligence technology, more and more artificial intelligence technology is applied to the medical field, and the technology is used for identifying, classifying and the like of medical images.

The application patent application with application number 201910250510.7 discloses an automatic grading method for acnes based on deep learning, which comprises the following steps: s1, making a face sample data set, and training a deep convolutional neural network hierarchical model; s2, acquiring a face image to be detected, identifying face feature points in the image by using a face feature point detection network, cutting areas, and removing invalid areas; and S3, splicing the cut images to obtain skin region images, and inputting the skin region images into a deep convolutional neural network grading model to obtain grading results. The face images of the front face, the left face and the right face of the patient are acquired through the camera, and the severity of the facial acne is automatically graded by the computer through a pre-trained deep convolutional neural network grading model, so that accurate auxiliary information is provided for diagnosis of the illness state of the patient.

In addition to implementing automatic classification based on deep learning methods, there are semi-supervised learning-based methods. The application patent application with the application number of 2022113867194 discloses an acne grading method, system, equipment and storage medium based on semi-supervised learning, wherein a semi-supervised learning network model is built, and when the model is trained, first, labeled sample data are adopted to supervise a student network, and then non-labeled sample data are adopted to semi-supervise the student network and a teacher network; in the training process, the weight of the teacher network is updated by adopting a sliding index average mode through the weight of the student network. The application is based on semi-supervised facial acne classification, does not need excessive labeled data during training, and can automatically learn features from a large amount of unlabeled data, thereby achieving complementation between labeled and unlabeled data.

In the prior art, since the number of the labeling data is continuously increased, the model for deep learning training is too complex for the small data amount in the initial stage. The complex network not only increases training cost, but also easily causes the phenomenon of over fitting. Therefore, the existing idea to solve this problem is to increase the complexity of the network with the increase of the data volume. According to the solution idea, the former proposes a solution, namely, continuously training a new model with increased depth along with the increase of the labeling quantity, comparing the new model with the previous model, and selecting a better model.

However, the above solution may introduce additional training overhead, resulting in inefficient training, since the new model is continually compared to the old model, so that the old model may be discarded. Meanwhile, an excellent trained deep learning model is often based on a marked specific high-quality data set, and has no good migration learning effect.

Disclosure of Invention

The invention aims at: in order to solve the technical problems that in the prior art, the new model and the old model are compared, additional training expenditure is introduced, so that training efficiency is low, and an excellent trained deep learning model is often based on a marked specific high-quality data set and has no better migration learning effect, the invention provides a skin acne grading method, a skin acne grading system, skin acne grading equipment and a skin acne grading storage medium.

The invention adopts the following technical scheme for realizing the purposes:

an auxiliary labeling method for skin acne images comprises the following steps:

step S1-1, when no data is stored in the database or no skin image sample data exists in the data stored in the database, maintaining an initial state and repeating refreshing data input; if the skin image sample data is stored in the database and the skin image sample data comprises skin image sample data with original labels and skin image sample data without labels, the step S1-2 is carried out;

s1-2, performing supervised learning training on an infinite depth neural network model by using skin image sample data with an original label; if the specified training period is not met or the accuracy rate is not met, continuing to conduct supervised learning training; if the specified training period is reached or the accuracy rate reaches the requirement, the step S1-3 is entered;

in the process of supervised learning training, if a better model is searched, updating the architecture of the infinite depth neural network model;

s1-3, performing semi-supervised learning training on an infinite depth neural network model by using labeled skin image sample data and pseudo labels of unlabeled skin image sample data; if the specified training period is not met or the accuracy rate is not met, continuing to perform semi-supervised learning training; if the specified training period is met or the accuracy rate meets the requirement and the unlabeled skin image sample data still exists, then the inquiry is carried out by utilizing an inquiry strategy based on uncertainty through active learning, the high-value unlabeled skin image sample data to be recommended is generated, and the step S1-4 is carried out; if the specified training period is reached or the accuracy rate reaches the requirement and no label-free skin image sample data exists, the step S1-5 is entered;

In the semi-supervised learning training process, if a better model is searched, updating the architecture of the infinite depth neural network model;

s1-4, labeling the recommended unlabeled skin image sample data by a labeling person; if the quantity marked by the marker does not reach the budget quantity, repeatedly sending marking requests to the marker; if the number of labels of the labels reaches the budget, updating a labeled data set of the labeled skin image sample data, and entering a step S1-2;

s1-5, entering a static state, and waiting for storing new unlabeled skin image sample data; if new unlabeled skin image sample data is stored, the process proceeds to step S1-4.

A skin acne image assisted labeling system, comprising:

an initial state module for maintaining an initial state and repeating refreshing data input when no data is stored in the database or no skin image sample data is present in the data stored in the database; if skin image sample data is stored in the database and the skin image sample data comprises skin image sample data with original labels and skin image sample data without labels, entering a network updating and supervising learning module;

The network updating and supervising learning module is used for supervising learning training of the infinite depth neural network model by using skin image sample data with an original label; if the specified training period is not met or the accuracy rate is not met, continuing to conduct supervised learning training; if the specified training period is reached or the accuracy rate reaches the requirement, entering a network updating and semi-supervised learning module;

in the process of supervised learning training, if a better model is searched, updating the architecture of the infinite depth neural network model;

the network updating and semi-supervised learning module is used for performing semi-supervised learning training on the infinite depth neural network model by using the labeled skin image sample data and the pseudo label of the unlabeled skin image sample data; if the specified training period is not met or the accuracy rate is not met, continuing to perform semi-supervised learning training; if the specified training period is met or the accuracy rate meets the requirement and the unlabeled skin image sample data still exists, then through active learning, inquiring by utilizing an inquiry strategy based on uncertainty and generating high-value unlabeled skin image sample data to be recommended, and entering a data request module; if the specified training period is reached or the accuracy rate reaches the requirement and no label-free skin image sample data exists, entering a static state module;

In the semi-supervised learning training process, if a better model is searched, updating the architecture of the infinite depth neural network model;

the data request module is used for labeling the recommended unlabeled skin image sample data by the labeling person; if the quantity marked by the marker does not reach the budget quantity, repeatedly sending marking requests to the marker; if the number of labels of the labels reaches the budget, updating the labeled data set of the labeled skin image sample data, and entering a network updating and supervising learning module;

the static state module is used for entering a static state and waiting for storing new unlabeled skin image sample data; if new unlabeled skin image sample data is stored, a data request module is entered.

A method of grading skin acne comprising the steps of:

step S1, acquiring sample data;

acquiring skin image sample data, wherein the skin image sample data comprises skin image sample data with labels and skin image sample data without labels;

s2, constructing an infinite depth neural network model;

constructing an infinite depth neural network model, wherein each layer of the infinite depth neural network model comprises a hidden layer and an output layer;

S3, training an infinite depth neural network model;

inputting the skin image sample data of the step S1 into the infinite depth neural network model constructed in the step S2, and training the infinite depth neural network model;

during training, the label-free skin image sample data is subjected to auxiliary labeling by adopting the auxiliary labeling method for the skin acne images;

s4, grading in real time;

and (3) acquiring skin real-time image data, and inputting the skin real-time image data into the infinite depth neural network model obtained through training in the step (S3) to obtain a grading result.

Further, in the infinite depth neural network model in step S2, the hidden layer of the first layer performs a convolution operation with a convolution kernel size of 3, and the hidden layers of each of the other layers have two convolution operations with convolution kernels of 3, and the feature pattern sizes of the second layer, the fifth layer and the eighth layer are halved compared with the feature pattern size of the corresponding previous layer, and the channel numbers of the fifth layer and the eighth layer are doubled compared with the channel numbers of the corresponding previous layer;

the output of each layer and the corresponding probability weight are summed to be used as the output of the infinite depth neural network model.

Further, in step S3, when each training round is performed on the infinite depth neural network model, firstly, the labeled skin image sample data is adopted to perform supervised learning, then, after pseudo labels are given to the unlabeled skin image sample data, semi-supervised learning is performed, then, through active learning, an inquiry is performed by using an inquiry strategy based on uncertainty, and high-value unlabeled skin image sample data is recommended to a annotator, and finally, the annotator marks the recommended unlabeled skin image sample data and updates the labeled data set of the labeled skin image sample data, and then, the next training round is performed.

Further, when training the infinite depth neural network model in step S3, for each round of supervised learning and semi-supervised learning, the distribution of the infinite depth neural network model is fitted using the truncated poisson distribution, and the network weight parameter distribution of the infinite depth neural network model is fitted using the poisson distribution.

Further, during infinite depth neural network model training, a loss functionThe method comprises the following steps:

wherein ,parameters representing the depth profile of the neural network, +.>Parameters representing the distribution of the weight parameters of the neural network, +.>Representing neural network depth, ++>Represents the->Layer (S)>Representing the current network depth, +.>Representing the input of the i-th layer,represents the output of the ith layer,/-)>A variation approximation distribution representing the depth of the neural network, +.>Representing a priori distribution of neural network depth, +.>Representing the approximate distribution of variation of the weight parameters of the neural network, < ->Representing a priori distribution of neural network depth, +.>Representing a probability distribution of the output y under the neural network model f and the input x;

when training an infinite depth neural network model, adopting a uniform sampling method to perform target optimization on shared parameters, wherein the target is expressed as:

the overall optimization objective is achieved by a gradient descent method, and is expressed as:

wherein ,parameters representing the depth profile of the neural network, +.>Parameters representing the distribution of the weight parameters of the neural network, +.>Representing a labeled dataset->Representing neural network depth, ++>Representing a uniform distribution over the depth selection set, < >>Representing the desire to obey a uniform distribution of d, +.>Is expressed as d at depth and +.>Cross entropy between model output and real label under model of (a)>Representing a loss function.

A skin acne grading system, comprising:

the sample data acquisition module is used for acquiring skin image sample data, wherein the skin image sample data comprises labeled skin image sample data and unlabeled skin image sample data;

the infinite depth neural network model building module is used for building an infinite depth neural network model, and each layer of the infinite depth neural network model comprises a hidden layer and an output layer;

the infinite depth neural network model training module is used for inputting the skin image sample data of the sample data acquisition module into the infinite depth neural network model constructed by the infinite depth neural network model construction module to train the infinite depth neural network model;

during training, the label-free skin image sample data is subjected to auxiliary labeling by adopting the auxiliary labeling method for the skin acne images;

And the real-time grading module is used for acquiring the skin real-time image data, inputting the skin real-time image data into the infinite depth neural network model obtained through training of the infinite depth neural network model training module, and obtaining grading results.

The beneficial effects of the invention are as follows:

in the invention, aiming at the problem of small initial data volume of the existing algorithm, the project acquires high-quality data through an Oracle labeling period, and is different from a mode of focusing on improving the quality of a labeled sample by replacing a screening method (screening method), so that the accuracy of a model is improved; by introducing supervised learning and semi-supervised learning, the unlabeled data to be labeled can be better utilized, so that the accuracy of the model is improved, and the problem of low model prediction accuracy caused by insufficient high-quality data sets is solved to a certain extent.

In the invention, aiming at the deep learning model solidification problem, a neural network architecture search technology is introduced to perform micro-search on the model depth, parameters related to the model depth are added on the basis of an infinite layer network model, depth probability distribution is determined through the learned depth distribution parameters, and then corresponding probability weighted summation is given to output results of different layers as output, so that the model can adaptively change along with the increase of data quantity, an automatic network design mode without subjective interference is formed, the problems that the traditional deep learning model is fixed and cannot be updated along with the increase of data quantity are solved, and meanwhile, the problem that the traditional neural network needs expert manual design and has limitation is also solved.

According to the method, the method and the device, the active learning, the deep learning and the neural network architecture searching are combined in the active learning period, so that the marking efficiency is improved, the marking cost is reduced, the problem that the deep learning model can show the best performance only by a large amount of labeled data is solved, and the blank of a learning strategy with less initial marked data and an auxiliary marking model generating algorithm in a high-precision marking scene is filled.

Drawings

FIG. 1 is a flow chart of an auxiliary labeling method of the present invention;

FIG. 2 is a flow chart of the skin acne classification method of the present invention;

FIG. 3 is a schematic diagram of the structure of an infinite depth neural network model according to the present invention;

FIG. 4 is a schematic flow chart of the infinite depth neural network model according to the present invention when training;

FIG. 5 is a flow chart of the active learning of the present invention;

fig. 6 is a representation of a dataset of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.

Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Example 1

The embodiment provides an auxiliary labeling method for skin acne images, which can better utilize unlabeled data to be labeled by introducing supervised learning and semi-supervised learning, so that model accuracy is improved, and the problem of lower model prediction accuracy caused by insufficient high-quality data sets is solved. As shown in fig. 1, the labeling method specifically includes the following steps:

and step S1-1, when no data is stored in the database or no skin image sample data exists in the data stored in the database, maintaining the initial state and repeatedly refreshing the data input until the skin image sample data exists in the data stored in the database.

If the skin image sample data is stored in the database, and the skin image sample data includes skin image sample data of an original label and skin image sample data of no label, the step S1-2 is entered.

S1-2, performing supervised learning training on an infinite depth neural network model by using skin image sample data with an original label; when training, judging whether to enter the next step according to whether the specified training period is reached and whether the accuracy rate meets the requirement. The training period and the accuracy are determined according to the actual conditions, and creative labor is not required.

If the specified training period is not met or the accuracy rate is not met, continuing to conduct supervised learning training; if the specified training period is reached or the accuracy rate reaches the requirement, the step S1-3 is entered.

In the process of supervised learning training, if a better model is searched, updating the architecture of the infinite depth neural network model. The preferred model here means that as training proceeds, the lambda value is continuously updated so that the truncated poisson distribution moves to the right, the neural network depth is gradually extended to the extent that it is adaptive to the data volume, and the network weight parameters are trained simultaneously in the process; and more particularly to models with smaller loss functions.

S1-3, performing semi-supervised learning training on an infinite depth neural network model by using labeled skin image sample data and pseudo labels of unlabeled skin image sample data; the pseudo tag of the unlabeled skin image sample data can be generated and obtained by adopting the existing pseudo tag generation method by adopting the unlabeled skin image sample data.

When training, judging whether to enter the next step according to whether the specified training period is reached and whether the accuracy rate meets the requirement. The training period and the accuracy are determined according to the actual conditions, and creative labor is not required.

If the specified training period is not met or the accuracy rate is not met, continuing to perform semi-supervised learning training; if the specified training period is met or the accuracy rate meets the requirement and the unlabeled skin image sample data still exists, then the inquiry is carried out by utilizing an inquiry strategy based on uncertainty through active learning, the high-value unlabeled skin image sample data to be recommended is generated, and the step S1-4 is carried out; if the specified training period is reached or the accuracy rate reaches the requirement and there is no unlabeled skin image sample data, step S1-5 is entered.

In the semi-supervised learning training process, if a better model is searched, updating the architecture of the infinite depth neural network model.

And S1-4, labeling the recommended unlabeled skin image sample data by the labeling person.

And in the labeling process, the labeling person judges whether to enter the next step according to whether the labeling quantity reaches the budget quantity or not. The budget amount is determined according to actual conditions, and creative labor is not required.

If the quantity marked by the marker does not reach the budget quantity, repeatedly sending marking requests to the marker; if the number of labels of the label reaches the budget amount, updating the labeled data set of the labeled skin image sample data, and proceeding to step S1-2.

Step S1-5, entering a static state and waiting to store new unlabeled skin image sample data.

If no new unlabeled skin image sample data is stored, the static state is kept continuously, and the data is waited to be stored until the new unlabeled skin image sample data is stored. If new unlabeled skin image sample data is stored, the process proceeds to step S1-4.

Example 2

The embodiment also provides an auxiliary labeling system for skin acne images, which specifically comprises the following steps:

and the initial state module is used for maintaining the initial state and repeatedly refreshing data input when no data is stored in the database or skin image sample data is not stored in the data stored in the database until the skin image sample data is stored in the data stored in the database.

If the skin image sample data is stored in the database, and the skin image sample data comprises skin image sample data with original labels and skin image sample data without labels, the network updating and supervision learning module is entered.

The network updating and supervising learning module is used for supervising learning training of the infinite depth neural network model by using skin image sample data with an original label; when training, judging whether to enter the next step according to whether the specified training period is reached and whether the accuracy rate meets the requirement. The training period and the accuracy are determined according to the actual conditions, and creative labor is not required.

If the specified training period is not met or the accuracy rate is not met, continuing to conduct supervised learning training; if the specified training period is reached or the accuracy rate reaches the requirement, entering a network updating and semi-supervised learning module.

In the process of supervised learning training, if a better model is searched, updating the architecture of the infinite depth neural network model.

The network updating and semi-supervised learning module is used for performing semi-supervised learning training on the infinite depth neural network model by using the labeled skin image sample data and the pseudo label of the unlabeled skin image sample data; the pseudo tag of the unlabeled skin image sample data can be generated and obtained by adopting the existing pseudo tag generation method by adopting the unlabeled skin image sample data.

When training, judging whether to enter the next step according to whether the specified training period is reached and whether the accuracy rate meets the requirement. The training period and the accuracy are determined according to the actual conditions, and creative labor is not required.

If the specified training period is not met or the accuracy rate is not met, continuing to perform semi-supervised learning training; if the specified training period is met or the accuracy rate meets the requirement and the unlabeled skin image sample data still exists, then through active learning, inquiring by utilizing an inquiry strategy based on uncertainty and generating high-value unlabeled skin image sample data to be recommended, and entering a data request module; if the specified training period is reached or the accuracy rate reaches the requirement and no unlabeled skin image sample data exists, a static state module is entered.

In the semi-supervised learning training process, if a better model is searched, updating the architecture of the infinite depth neural network model.

And the data request module is used for labeling the recommended unlabeled skin image sample data by the labeling person.

And in the labeling process, the labeling person judges whether to enter the next step according to whether the labeling quantity reaches the budget quantity or not. The budget amount is determined according to actual conditions, and creative labor is not required.

If the quantity marked by the marker does not reach the budget quantity, repeatedly sending marking requests to the marker; if the number of labels of the labels reaches the budget, updating the labeled data set of the labeled skin image sample data, and entering a network updating and supervising learning module.

And the static state module is used for entering a static state and waiting for storing new unlabeled skin image sample data.

If no new unlabeled skin image sample data is stored, the static state is kept continuously, and the data is waited to be stored until the new unlabeled skin image sample data is stored. If new unlabeled skin image sample data is stored, a data request module is entered.

Example 3

The embodiment provides a skin acne grading method, which comprises the steps of building and training a neural network model, inputting skin images into the mature neural network model, and outputting a grading result of skin acne. As shown in fig. 2, the classification method includes the steps of:

step S1, acquiring sample data;

skin image sample data is acquired, and the skin image sample data comprises labeled skin image sample data and unlabeled skin image sample data.

To ensure feasibility of application, the sample data in this example were from the Acne skin Acne dataset provided by the national institute of medicine. The data set is from high-definition images of the face of the patient acquired by a unified device and stored in a unified format, and is classified into 8 grades according to acne conditions in the skin of the patient, and the classification process of the data set is fully participated by a first-line doctor so as to ensure the reliability of the data. The data set part attributes are shown in table 1 and fig. 6 (the picture is subjected to occlusion processing in order to avoid revealing patient privacy).

Table 1Acne dataset attributes

Data pre-processing was also performed on the basis of the original Acne skin Acne dataset. Because the original sample data size is small, and the model can be provided with richer image information by the augmentation operation, good preprocessing is also a key for improving the performance of the model. For this reason, the present embodiment uses random horizontal flip, random vertical flip, rotation, and random occlusion as data augmentation strategies, and also processes an image to 512×512 using the restore () function since the picture of the original sample data is too large.

S2, constructing an infinite depth neural network model;

An infinite depth neural network model is constructed, and each layer of the infinite depth neural network model comprises a hidden layer and an output layer.

The structure of the infinite depth neural network model is shown in fig. 3, and the initial model is only given to the initial output layer; the hidden layer of the first layer carries out convolution operation with a convolution kernel size of 3; then, the hidden layer of each of the remaining layers has two convolution operations with convolution kernel size 3.

In addition, the feature pattern size of the second layer, the fifth layer and the eighth layer of the model is halved compared with the feature pattern size of the corresponding previous layer, and the channel number of the fifth layer and the eighth layer is doubled compared with the channel number of the corresponding previous layer; the feature map size and the channel number of other layers of the model are identical to those of the nearest changing layer in the front (namely, the feature map and the channel number of the second layer, the third layer and the fourth layer are identical, and the feature map and the channel number of the fifth layer, the sixth layer and the seventh layer are identical, so on).

The output of each layer and the corresponding probability weight are summed to be used as the output of the infinite depth neural network model.

S3, training an infinite depth neural network model;

and (3) inputting the skin image sample data of the step (S1) into the infinite depth neural network model constructed in the step (S2) to train the infinite depth neural network model.

The training process is shown in fig. 4. During training, the label-free skin image sample data is subjected to auxiliary labeling by adopting the auxiliary labeling method for skin acne images, which is described in the embodiment 1.

The training process of deep al generally obtains the most "quality data" through any data obtaining manner, so as to provide model training, for example, through batch entropy information prediction, model prediction scoring or bayesian evaluation. The most classical active learning flow is also followed to adapt to the labeling update scene, a request for updating data is initiated after Model training is finished, a data index is generated and pushed to Oracle (such as a professional doctor) for labeling, a data set is updated after labeling is finished, and the data index is transferred to a labeling pool. The method comprises the following steps: the method comprises the steps of performing supervised learning by adopting labeled skin image sample data, performing semi-supervised learning after pseudo labels are given to the unlabeled skin image sample data, performing active learning, performing query by using an uncertainty-based query strategy (which can be referred to in detail by 'Burr settes 2009. Active learning literature report, technical report, university of Wisconsin-Madison Department of Computer Sciences'), recommending high-value unlabeled skin image sample data to a annotator, and finally, updating a labeled dataset of the labeled skin image sample data after the annotator annotates the recommended unlabeled skin image sample data to perform the next training.

In the traditional active learning, more emphasis is placed on the replacement of the screening method, and the method is considered to be the most important ring in the whole deep active learning process. However, it is apparent that only the selection of a different number of samples by the replacement of the method wastes all of the picture data stored in the initial stage. The introduction of semi-supervised learning is now a better method of improving model accuracy when few initial labeling samples are available, because it takes full advantage of unlabeled data, and its benefits have also proven to be "much greater than the benefits of method replacement".

As shown in fig. 5, the algorithm of the present embodiment constructs an active learning cycle framework based on the characteristic that the former has proven (efficiency improvement caused by introducing supervised learning and semi-supervised learning), and the main flow is to perform supervised learning by using already-marked data first, and after reaching a specified cycle or accuracy, perform semi-supervised learning by giving a pseudo tag to unmarked data, and at the same time, after the supervised learning and the semi-supervised learning reach the specified training cycle along with updating of network capacity, use a screening method to recommend pictures that need to be marked preferentially to the user, and the specific algorithm flow is shown in a pseudo code 1:

In semi-supervised learning, the present embodiment uses a transduction label propagation algorithm to generate pseudo labels for unlabeled skin image sample data prediction throughout the data domain before application to subsequent model training.

The embodiment adopts a method combining network depth search and parameter learning, namely a depth search strategy based on variational reasoning. The specific contents are as follows: for each round of active learning, the truncated poisson distribution is used for fitting the distribution of the depth of the neural network, the poisson distribution is used for fitting the distribution of the weight parameters of the network, and the weighted sum of the output of each layer and the corresponding probability is used as the model output, so that the confidence of the model is improved, and the probability of overfitting is reduced. In this case, the depth search problem turns to a parameter fitting problem for the network depth truncated poisson distribution and the network weight parameter normal distribution.

To solve this problem, the present embodiment proposes a variation approximate distributionFor fitting true posterior distributionThe similarity of the two is expressed by using KL divergence, and the specific expression is as follows:

the term is constant when D is determined, so it can be omitted directly when KL is optimized. Taking the expected value of the remaining terms on the data distribution D can result in an operable, suitable surrogate target, called the lower evidence bound (ELBO), taking its negative value as the loss function optimization target:

（1）

However, the above variational reasoning method, because of introducing the mean field assumption, assumesThis is independent, which will allow more opportunities for the weighting parameters of the shallow network to be trained and ultimately result in the algorithm-generated network depth being shallow.

To overcome the above problems, the present embodiment uses knotsA method for constructively and differentially reasoning. The method is based on the original conclusionNot independent, so the above expression is rewritten as:

（2）

the formula is the loss function adopted in infinite depth neural network model training。/>

wherein ,parameters representing the depth profile of the neural network, +.>Parameters representing the distribution of the weight parameters of the neural network, +.>Representing neural network depth, ++>Represents the->Layer (S)>Representing the current network depth, +.>Representing the input of the i-th layer,represents the output of the ith layer,/-)>A variation approximation distribution representing the depth of the neural network, +.>Representing a priori distribution of neural network depth, +.>Representing the approximate distribution of variation of the weight parameters of the neural network, < ->Representing a priori distribution of neural network depth, +.>The probability distribution of the output y under the neural network model f and the input x is represented.

The formula (2) is different from the original formula (1) in two points:

1. One term of KL divergence for two network weight parameter distributions, namelyWrite in about->Is not limited to the above-described embodiments.

2. The network weight parameters are no longer independent of the neural network depth.

In this embodiment, two priors are presetAnd->Thus for->On the one hand, the update of (a) is to approximate the distribution of the variation in the present embodiment->、/>Close to two a priori->、/>This appears to reduce the first two terms, namely, the two KL divergences. On the other hand, learning from the tag data is also performed, which is manifested in a reduction of the last item,. With the input of tag data from small to large, the former is mainly used as a main task in the early stage, and the latter is mainly used as a main task in the later stage.

In this embodiment, the equation is implemented by using a uniform sampling method, and for the target optimization of the shared parameter, the method may be specifically expressed as:

representing a uniform distribution over the depth selection set, the +.>Replaced byFor further calculation.

Finally, the overall optimization objective is achieved by a gradient descent method, expressed as:

wherein ,parameters representing the depth profile of the neural network, +.>Representing neural networksParameters of the weight parameter distribution, +.>Representing a labeled dataset- >Representing neural network depth, ++>Indicating the desire to obey a uniform distribution of d,is expressed as d at depth and +.>Cross entropy between the model output and the real label under the model of (a),representing a loss function.

S4, grading in real time;

and (3) acquiring skin real-time image data, and inputting the skin real-time image data into the infinite depth neural network model obtained through training in the step (S3) to obtain a grading result.

Example 4

The present embodiment provides a skin acne grading system, the grading system comprising:

the sample data acquisition module is used for acquiring skin image sample data, wherein the skin image sample data comprises labeled skin image sample data and unlabeled skin image sample data.

To ensure feasibility of application, the sample data in this example were from the Acne skin Acne dataset provided by the national institute of medicine. The data set is from high-definition images of the face of the patient acquired by a unified device and stored in a unified format, and is classified into 8 grades according to acne conditions in the skin of the patient, and the classification process of the data set is fully participated by a first-line doctor so as to ensure the reliability of the data. The data set part attribute is shown in fig. 6 (to avoid revealing patient privacy, the picture is masked).

Data pre-processing was also performed on the basis of the original Acne skin Acne dataset. Because the original sample data size is small, and the model can be provided with richer image information by the augmentation operation, good preprocessing is also a key for improving the performance of the model. For this reason, the present embodiment uses random horizontal flip, random vertical flip, rotation, and random occlusion as data augmentation strategies, and also processes an image to 512×512 using the restore () function since the picture of the original sample data is too large.

And the infinite depth neural network model building module is used for building an infinite depth neural network model, and each layer of the infinite depth neural network model comprises a hidden layer and an output layer.

Initializing a model only to an initial output layer; the hidden layer of the first layer carries out convolution operation with a convolution kernel size of 3; then, the hidden layer of each of the remaining layers has two convolution operations with convolution kernel size 3.

In addition, the feature pattern size of the second layer, the fifth layer and the eighth layer of the model is halved compared with the feature pattern size of the corresponding previous layer, and the channel number of the fifth layer and the eighth layer is doubled compared with the channel number of the corresponding previous layer; the feature map size and the channel number of other layers of the model are identical to those of the nearest changing layer in the front (namely, the feature map and the channel number of the second layer, the third layer and the fourth layer are identical, and the feature map and the channel number of the fifth layer, the sixth layer and the seventh layer are identical, so on).

The output of each layer and the corresponding probability weight are summed to be used as the output of the infinite depth neural network model.

And the infinite depth neural network model training module is used for inputting the skin image sample data of the sample data acquisition module into the infinite depth neural network model constructed by the infinite depth neural network model construction module to train the infinite depth neural network model.

During training, the label-free skin image sample data is subjected to auxiliary labeling by adopting the auxiliary labeling method for skin acne images, which is described in the embodiment 1.

The training process of deep al generally obtains the most "quality data" through any data obtaining manner, so as to provide model training, for example, through batch entropy information prediction, model prediction scoring or bayesian evaluation. The most classical active learning flow is also followed to adapt to the labeling update scene, a request for updating data is initiated after Model training is finished, a data index is generated and pushed to Oracle (such as a professional doctor) for labeling, a data set is updated after labeling is finished, and the data index is transferred to a labeling pool. The method comprises the following steps: the method comprises the steps of performing supervised learning by adopting labeled skin image sample data, performing semi-supervised learning after pseudo labels are given to the unlabeled skin image sample data, performing active learning, performing query by using an uncertainty-based query strategy (which can be referred to in detail by 'Burr settes 2009. Active learning literature report, technical report, university of Wisconsin-Madison Department of Computer Sciences'), recommending high-value unlabeled skin image sample data to a annotator, and finally, updating a labeled dataset of the labeled skin image sample data after the annotator annotates the recommended unlabeled skin image sample data to perform the next training.

In the traditional active learning, more emphasis is placed on the replacement of the screening method, and the method is considered to be the most important ring in the whole deep active learning process. However, it is apparent that only the selection of a different number of samples by the replacement of the method wastes all of the picture data stored in the initial stage. The introduction of semi-supervised learning is now a better method of improving model accuracy when few initial labeling samples are available, because it takes full advantage of unlabeled data, and its benefits have also proven to be "much greater than the benefits of method replacement".

The algorithm of the embodiment constructs an active learning period framework based on the characteristic proven by the former (efficiency improvement brought by the introduction of supervised learning and semi-supervised learning), and mainly comprises the steps of performing supervised learning by using marked data, performing semi-supervised learning by giving a false label to unmarked data after reaching a specified period or accuracy, and recommending pictures needing to be marked preferentially for a user by using a screening method after the supervised learning and the semi-supervised learning reach the specified training period along with updating of network capacity.

In semi-supervised learning, the present embodiment uses a transduction label propagation algorithm to generate pseudo labels for unlabeled skin image sample data prediction throughout the data domain before application to subsequent model training.

The embodiment adopts a method combining network depth search and parameter learning, namely a depth search strategy based on variational reasoning. The specific contents are as follows: for each round of active learning, the truncated poisson distribution is used for fitting the distribution of the depth of the neural network, the poisson distribution is used for fitting the distribution of the weight parameters of the network, and the weighted sum of the output of each layer and the corresponding probability is used as the model output, so that the confidence of the model is improved, and the probability of overfitting is reduced. In this case, the depth search problem turns to a parameter fitting problem for the network depth truncated poisson distribution and the network weight parameter normal distribution.

To solve this problem, the present embodiment proposes a variation approximate distributionFor fitting true posterior distributionThe similarity of the two is expressed by using KL divergence, and the specific expression is as follows:

the term is constant when D is determined, so it can be omitted directly when KL is optimized. Taking the expected value of the remaining terms on the data distribution D can result in an operable, suitable surrogate target, called the lower evidence bound (ELBO), taking its negative value as the loss function optimization target:

（1）/>

however, the above variational reasoning method, because of introducing the mean field assumption, assumes This is independent, which will allow more opportunities for the weighting parameters of the shallow network to be trained and ultimately result in the algorithm-generated network depth being shallow.

To overcome the above problems, the present embodiment uses a structured variational reasoning method. The method is based on the original conclusionNot independent, so the above expression is rewritten as:

（2）

the formula is the loss function adopted in infinite depth neural network model training。

wherein ,parameters representing the depth profile of the neural network, +.>Parameters representing the distribution of the weight parameters of the neural network, +.>Representing neural network depth, ++>Represents the->Layer (S)>Representing the current network depth, +.>Representing the input of the i-th layer,represents the output of the ith layer,/-)>A variation approximation distribution representing the depth of the neural network, +.>Representing a priori distribution of neural network depth, +.>Representing the approximate distribution of variation of the weight parameters of the neural network, < ->Representing a priori distribution of neural network depth, +.>The probability distribution of the output y under the neural network model f and the input x is represented.

The formula (2) is different from the original formula (1) in two points:

1. one term of KL divergence for two network weight parameter distributions, namely Write in about->Is not limited to the above-described embodiments.

2. The network weight parameters are no longer independent of the neural network depth.

In this embodiment, two priors are presetAnd->Thus for->On the one hand, the update of (a) is to approximate the distribution of the variation in the present embodiment->、/>Close to two a priori->、/>This appears to reduce the first two terms, namely, the two KL divergences. On the other hand, learning from the tag data is also performed, which is manifested in a reduction of the last item,. With the input of tag data from small to large, the former is mainly used as a main task in the early stage, and the latter is mainly used as a main task in the later stage.

In this embodiment, the equation is implemented by using a uniform sampling method, and for the target optimization of the shared parameter, the method may be specifically expressed as:

representing a uniform distribution over the depth selection set, the +.>Replaced byFor further calculation.

Finally, the overall optimization objective is achieved by a gradient descent method, expressed as:

wherein ,parameters representing the depth profile of the neural network, +.>Parameters representing the distribution of the weight parameters of the neural network, +.>Representing a labeled dataset->Representing neural network depth, ++>Indicating the desire to obey a uniform distribution of d, Is expressed as d at depth and +.>Cross entropy between the model output and the real label under the model of (a),representing a loss function.

And the real-time grading module is used for acquiring the skin real-time image data, inputting the skin real-time image data into the infinite depth neural network model obtained through training of the infinite depth neural network model training module, and obtaining grading results.

Example 5

The present embodiment provides a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of the skin acne classification method described above.

The computer equipment can be computing equipment such as a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The computer equipment can perform man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch pad or voice control equipment and the like.

The memory includes at least one type of readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or D interface display memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like. Of course, the memory may also include both internal storage units of the computer device and external storage devices. In this embodiment, the memory is often used to store an operating system and various application software installed on the computer device, such as program codes of the skin acne classification method. In addition, the memory may be used to temporarily store various types of data that have been output or are to be output.

The processor may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor is typically used to control the overall operation of the computer device. In this embodiment, the processor is configured to execute the program code stored in the memory or process data, such as the program code of the skin acne classification method.

Example 6

The present embodiment provides a computer-readable storage medium having stored therein a computer program which, when executed by a processor, causes the processor to perform the steps of the skin acne classification method described above.

Wherein the computer-readable storage medium stores an interface display program executable by at least one processor to cause the at least one processor to perform the steps of the skin acne classification method as described above.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server or a network device, etc.) to perform the skin acne classification method according to the embodiments of the present application.

Claims (8)

1. An auxiliary labeling method for skin acne images is characterized by comprising the following steps:

step S1-1, when no data is stored in the database or no skin image sample data exists in the data stored in the database, maintaining an initial state and repeating refreshing data input; if the skin image sample data is stored in the database and the skin image sample data comprises skin image sample data with original labels and skin image sample data without labels, the step S1-2 is carried out;

s1-2, performing supervised learning training on an infinite depth neural network model by using skin image sample data with an original label; if the specified training period is not met or the accuracy rate is not met, continuing to conduct supervised learning training; if the specified training period is reached or the accuracy rate reaches the requirement, the step S1-3 is entered;

in the process of supervised learning training, if a better model is searched, updating the architecture of the infinite depth neural network model;

s1-3, performing semi-supervised learning training on an infinite depth neural network model by using labeled skin image sample data and pseudo labels of unlabeled skin image sample data; if the specified training period is not met or the accuracy rate is not met, continuing to perform semi-supervised learning training; if the specified training period is met or the accuracy rate meets the requirement and the unlabeled skin image sample data still exists, then the inquiry is carried out by utilizing an inquiry strategy based on uncertainty through active learning, the high-value unlabeled skin image sample data to be recommended is generated, and the step S1-4 is carried out; if the specified training period is reached or the accuracy rate reaches the requirement and no label-free skin image sample data exists, the step S1-5 is entered;

In the semi-supervised learning training process, if a better model is searched, updating the architecture of the infinite depth neural network model;

s1-4, labeling the recommended unlabeled skin image sample data by a labeling person; if the quantity marked by the marker does not reach the budget quantity, repeatedly sending marking requests to the marker; if the number of labels of the labels reaches the budget, updating a labeled data set of the labeled skin image sample data, and entering a step S1-2;

s1-5, entering a static state, and waiting for storing new unlabeled skin image sample data; if new unlabeled skin image sample data is stored, the process proceeds to step S1-4.

2. An auxiliary labeling system for skin acne images, comprising:

an initial state module for maintaining an initial state and repeating refreshing data input when no data is stored in the database or no skin image sample data is present in the data stored in the database; if skin image sample data is stored in the database and the skin image sample data comprises skin image sample data with original labels and skin image sample data without labels, entering a network updating and supervising learning module;

The network updating and supervising learning module is used for supervising learning training of the infinite depth neural network model by using skin image sample data with an original label; if the specified training period is not met or the accuracy rate is not met, continuing to conduct supervised learning training; if the specified training period is reached or the accuracy rate reaches the requirement, entering a network updating and semi-supervised learning module;

in the process of supervised learning training, if a better model is searched, updating the architecture of the infinite depth neural network model;

the network updating and semi-supervised learning module is used for performing semi-supervised learning training on the infinite depth neural network model by using the labeled skin image sample data and the pseudo label of the unlabeled skin image sample data; if the specified training period is not met or the accuracy rate is not met, continuing to perform semi-supervised learning training; if the specified training period is met or the accuracy rate meets the requirement and the unlabeled skin image sample data still exists, then through active learning, inquiring by utilizing an inquiry strategy based on uncertainty and generating high-value unlabeled skin image sample data to be recommended, and entering a data request module; if the specified training period is reached or the accuracy rate reaches the requirement and no label-free skin image sample data exists, entering a static state module;

In the semi-supervised learning training process, if a better model is searched, updating the architecture of the infinite depth neural network model;

the data request module is used for labeling the recommended unlabeled skin image sample data by the labeling person; if the quantity marked by the marker does not reach the budget quantity, repeatedly sending marking requests to the marker; if the number of labels of the labels reaches the budget, updating the labeled data set of the labeled skin image sample data, and entering a network updating and supervising learning module;

the static state module is used for entering a static state and waiting for storing new unlabeled skin image sample data; if new unlabeled skin image sample data is stored, a data request module is entered.

3. A method for classifying acne on skin comprising the steps of:

step S1, acquiring sample data;

acquiring skin image sample data, wherein the skin image sample data comprises skin image sample data with labels and skin image sample data without labels;

s2, constructing an infinite depth neural network model;

constructing an infinite depth neural network model, wherein each layer of the infinite depth neural network model comprises a hidden layer and an output layer;

S3, training an infinite depth neural network model;

inputting the skin image sample data of the step S1 into the infinite depth neural network model constructed in the step S2, and training the infinite depth neural network model;

during training, the label-free skin image sample data is subjected to auxiliary labeling by adopting the auxiliary labeling method for skin acne images according to claim 1;

s4, grading in real time;

and (3) acquiring skin real-time image data, and inputting the skin real-time image data into the infinite depth neural network model obtained through training in the step (S3) to obtain a grading result.

4. A method of classifying skin acne according to claim 3, wherein: in the infinite depth neural network model in the step S2, the hidden layer of the first layer carries out convolution operation with a convolution kernel size of 3 once, the hidden layer of each other layer has convolution operation with a convolution kernel size of 3 twice, the feature map sizes of the second layer, the fifth layer and the eighth layer are halved compared with the feature map size of the corresponding previous layer, and the channel numbers of the fifth layer and the eighth layer are doubled compared with the channel numbers of the corresponding previous layer;

the output of each layer and the corresponding probability weight are summed to be used as the output of the infinite depth neural network model.

5. A method of classifying skin acne according to claim 3, wherein: in step S3, when each training round is performed on the infinite depth neural network model, firstly performing supervised learning by using labeled skin image sample data, then performing semi-supervised learning by giving pseudo labels to the unlabeled skin image sample data, then performing active learning, querying by using an uncertainty-based query strategy, recommending high-value unlabeled skin image sample data to a annotator, and finally, annotating the recommended unlabeled skin image sample data by the annotator, updating an annotated data set of the labeled skin image sample data, and performing the next training round.

6. A method of classifying skin acne according to claim 3, wherein: and when training the infinite depth neural network model in the step S3, fitting the distribution of the infinite depth neural network model by using truncated poisson distribution and fitting the network weight parameter distribution of the infinite depth neural network model by using poisson distribution for each round of supervised learning and semi-supervised learning.

7. A method of classifying skin acne according to claim 3, wherein: loss function during infinite depth neural network model training The method comprises the following steps:

wherein ,parameters representing the depth profile of the neural network, +.>Representing neural network weight parametersParameters of distribution->Representing neural network depth, ++>Represents the->Layer (S)>Representing the current network depth, +.>Input representing layer i,/->Represents the output of the ith layer,/-)>A variation approximation distribution representing the depth of the neural network, +.>Representing a priori distribution of neural network depth, +.>Representing the approximate distribution of variation of the weight parameters of the neural network, < ->Representing a priori distribution of neural network depth, +.>Representing a probability distribution of the output y under the neural network model f and the input x;

when training an infinite depth neural network model, adopting a uniform sampling method to perform target optimization on shared parameters, wherein the target is expressed as:

the overall optimization objective is achieved by a gradient descent method, and is expressed as:

wherein ,parameters representing the depth profile of the neural network, +.>Parameters representing the distribution of the weight parameters of the neural network, +.>Representing a labeled dataset->Representing neural network depth, ++>Representing a uniform distribution over the depth selection set, < >>Representing the desire to obey a uniform distribution of d, +.>Is expressed as d at depth and +.>Cross entropy between model output and real label under model of (a) >Representing a loss function.

8. A skin acne grading system, comprising:

the sample data acquisition module is used for acquiring skin image sample data, wherein the skin image sample data comprises labeled skin image sample data and unlabeled skin image sample data;

the infinite depth neural network model building module is used for building an infinite depth neural network model, and each layer of the infinite depth neural network model comprises a hidden layer and an output layer;

the infinite depth neural network model training module is used for inputting the skin image sample data of the sample data acquisition module into the infinite depth neural network model constructed by the infinite depth neural network model construction module to train the infinite depth neural network model;

during training, the label-free skin image sample data is subjected to auxiliary labeling by adopting the auxiliary labeling method for skin acne images according to claim 1;

and the real-time grading module is used for acquiring the skin real-time image data, inputting the skin real-time image data into the infinite depth neural network model obtained through training of the infinite depth neural network model training module, and obtaining grading results.