CN113763332B - Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium - Google Patents
Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium Download PDFInfo
- Publication number
- CN113763332B CN113763332B CN202110946389.9A CN202110946389A CN113763332B CN 113763332 B CN113763332 B CN 113763332B CN 202110946389 A CN202110946389 A CN 202110946389A CN 113763332 B CN113763332 B CN 113763332B
- Authority
- CN
- China
- Prior art keywords
- capsule network
- image
- ternary
- result
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000002775 capsule Substances 0.000 title claims abstract description 150
- 238000004458 analytical method Methods 0.000 title claims abstract description 68
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 45
- 230000002685 pulmonary effect Effects 0.000 title claims description 4
- 206010056342 Pulmonary mass Diseases 0.000 claims abstract description 63
- 238000000034 method Methods 0.000 claims abstract description 48
- 239000003795 chemical substances by application Substances 0.000 claims abstract description 23
- 238000012549 training Methods 0.000 claims abstract description 14
- 238000007405 data analysis Methods 0.000 claims abstract description 8
- 230000000875 corresponding effect Effects 0.000 claims description 35
- 230000006870 function Effects 0.000 claims description 28
- 238000003745 diagnosis Methods 0.000 claims description 26
- 230000009471 action Effects 0.000 claims description 19
- 230000008569 process Effects 0.000 claims description 13
- 210000004072 lung Anatomy 0.000 claims description 12
- 230000002787 reinforcement Effects 0.000 claims description 12
- 230000014509 gene expression Effects 0.000 claims description 11
- 239000010410 layer Substances 0.000 claims description 10
- 238000002372 labelling Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 238000010276 construction Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims description 3
- 239000002356 single layer Substances 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 description 10
- 238000002474 experimental method Methods 0.000 description 7
- 238000010191 image analysis Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000003902 lesion Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 210000001072 colon Anatomy 0.000 description 2
- 230000001186 cumulative effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000007667 floating Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 208000024891 symptom Diseases 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 206010028980 Neoplasm Diseases 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 210000000481 breast Anatomy 0.000 description 1
- 230000002308 calcification Effects 0.000 description 1
- 201000011510 cancer Diseases 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000013100 final test Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000036210 malignancy Effects 0.000 description 1
- 230000003211 malignant effect Effects 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000031068 symbiosis, encompassing mutualism through parasitism Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30061—Lung
- G06T2207/30064—Lung nodule
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Radiology & Medical Imaging (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a lung nodule analysis method, a device and a storage medium based on a ternary capsule network algorithm, wherein the method comprises the following steps: constructing a lung nodule analysis model based on a ternary capsule network algorithm; acquiring an image to be analyzed; learning a single-capsule network module, and training a capsule network intelligent agent; performing ternary capsule network module learning, and optimizing a capsule network intelligent agent based on ternary capsule network loss; based on the learned and optimized capsule network intelligent agent, analyzing the CT image to give a classification result of the lung nodule, namely obtaining the sign of the lung nodule suffered by the patient; and determining a data analysis result of the image to be analyzed according to the classification result, and outputting the data analysis result through a result determination module. Compared with the prior art, the method has the advantages of improving the analysis accuracy of the lung nodule and the like.
Description
Technical Field
The invention relates to the technical field of machine learning, in particular to a lung nodule analysis method and device based on a ternary capsule network algorithm and a storage medium.
Background
With the development of computer-aided technology, the medical field is increasingly using computer-aided equipment to lighten the working intensity of doctors. The human observation and analysis of the body images of patients by doctors are accompanied by the enhancement of the health consciousness of people, and the requirements of lung image examination and diagnosis are increased year by year. In the field of medical image analysis, the main difficulties at present include the following: the method comprises the steps of (1) labeling a sample is complex, symptoms of lesions in medical images are complex and various, the difficulty of distinguishing nodules of different categories is high, and professional knowledge and abundant experience are required for identification and labeling; this complex labeling process makes the reader tired, resulting in unstable diagnosis, and therefore each lesion requires multiple specialized tests at different time intervals, resulting in high costs. And secondly, the radiologist in the hospital is not enough, the task of reading the film every day is heavy, and the doctor cannot draw out special time to make sample marks, so that the sample in the field of medical image analysis and research is not enough. In the field of lung nodule classification and its related medical image classification, capsule network reinforcement learning has begun to be of interest. Silva uses a set of 3D geometric features in combination with reinforcement learning methods to classify lung nodules well/malignancy, resulting in an accuracy of 81%. The Ali proposes a lung nodule detection algorithm based on deep reinforcement learning under the heuristic of AlphaGo systems, wherein an agent adopts a convolutional neural network. The method was trained and tested on LIDC-IDRI database, resulting in an overall accuracy of 64.4% (sensitivity 58.9%, specificity 55.3%, PPV 54.2%, and NPV 60%). As for the ternary loss function, no related applications have been found in the field of lung nodule classification. In the field of medical images, puch proposes a small sample learning model based on a deep ternary network to classify brain MRI images. The result shows that under the condition of limited samples, the model can accurately identify images of different modes compared with the traditional convolutional neural network classifier. Medela in a research effort, it was verified that the use of triple networking techniques can migrate knowledge from well-defined source data (colon tissue images) to a more general domain consisting of colon, lung and breast tissue images using few training images.
Image analysis technology based on reinforcement learning gradually obtains trust and adoption of doctors, in particular to a detection task of lung nodules; the image analysis technology based on reinforcement learning can greatly reduce the workload of doctors. However, the imaging specialist cannot extract a large amount of time to do sample labeling work, which results in insufficient samples and low accuracy in the field of medical image analysis and research.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provide a lung nodule analysis method, a device and a storage medium based on a ternary capsule network algorithm, which are used for improving the accuracy of lung nodule analysis.
The aim of the invention can be achieved by the following technical scheme:
A lung nodule analysis method based on a ternary capsule network algorithm, the method comprising the steps of:
S1: a lung nodule analysis model based on a ternary capsule network algorithm is constructed, and the lung nodule analysis model comprises a memory pool module, a single capsule network module, a ternary capsule network module, an identification module and a result determination module.
S2: and acquiring an image to be analyzed.
S3: and (3) learning a single-capsule network module, and training the capsule network intelligent agent. Specifically, the size of a sliding window on a CT image is kept unchanged, the number of image blocks extracted by each CT is determined, the state number M of each round in the reinforcement learning process is determined, the difference between a single-capsule network diagnosis result and a calibration result is compared, and action return on each image block is correspondingly calculated.
The specific method for comparing the dissimilarity between the single-capsule network diagnosis result and the calibration result is as follows:
For an image block s t, if the single capsule network diagnosis is correct, giving a reward value, namely a return value r; otherwise, giving a prize value of 0; the judging method for judging whether the single-capsule network diagnosis result is correct or not comprises the following steps:
A: if the region of the image block s t is overlapped with a focus region of the calibration result, further checking whether the classification result of the image block is consistent with the sign type label of the calibration result, if so, judging that the single-capsule network diagnosis result is correct, otherwise, judging that the diagnosis result is wrong;
B: if the image block s t area is not overlapped with any focus area of the calibration result and the classification result of the image block is negative, judging that the single-capsule network diagnosis result is correct, otherwise, judging that the single-capsule network diagnosis result is wrong.
S4: and (3) learning a ternary capsule network module, and optimizing the capsule network intelligent agent based on ternary capsule network loss.
S5: inputting a lung CT image into the learned capsule network, and performing sliding window processing on the CT image by the model to obtain a series of states.
S6: and determining a data analysis result of the image to be analyzed according to the classification result, and outputting the data analysis result through a result determination module.
Further, in S1, the specific contents of constructing the lung nodule analysis model based on the ternary capsule network algorithm are:
11 A plurality of pictures are input, initial learning parameters are set, wherein the initial learning parameters comprise the storage amount N of a memory pool D in a memory pool module, the capacity H of a temporary memory pool D T, a return value r, an interval factor alpha, the number B of small batches, the training iteration number EPOCHS and a capsule network initial parameter omega c;
12 For a 512 x 512 pixel CT image, acquiring a 32 x 32 pixel image block on the CT image by adopting a K x K pixel sliding window, and simultaneously recording the upper left corner coordinate (x i,yi) of the image block; setting each image block as a state S t, and further forming all image blocks extracted from the CT image into a set S of all states;
13 The capsule network selects and executes the corresponding classification action a t from the action space A for the state S t in the set S and outputs the class attribute c j of the state;
14 After the capsule network acquires all the image blocks s i from a CT image and executes the corresponding action a t, the lung nodule detected by the capsule network and the sign classification result thereof are displayed back to the CT image according to the coordinate position of the image block area, so as to construct a lung nodule analysis model based on a ternary capsule network algorithm.
Further, in S3, a single capsule network adds a layer of fully connected layers to output the cumulative prize Q target value corresponding to each category after feature expression:
Qtarget＝M×r
the corresponding single step loss function is:
Where L (ω c) is the name of the loss function, R (x) is the return function, Is a capsule network intelligent agent,/>As anchor point images, a j is corresponding action, omega c is super-parameter;
And learning parameters of the single capsule network by adopting a corresponding single-step loss function of the single capsule network until the loss function is smaller than a preset value.
Further, in S4, the specific content of learning the ternary capsule network module is:
41 One time three images are input, respectively being anchor point images Positive sample image/>And negative sample imageExtracting features from the three images by using a capsule network respectively, further calculating Euclidean distance d + between the anchor point image and the positive sample image and Euclidean distance d - between the anchor point image and the negative sample image, and normalizing the Euclidean distances; and on the basis, a ternary loss function is calculated;
42 A loss function of the single-layer capsule network is converted into a ternary loss function;
43 Enabling the ternary capsule network to receive one state from the state set S, calculating rewards Q target values corresponding to different categories, selecting the action with the maximum corresponding Q target value to execute, correspondingly entering the next state, and simultaneously storing elements such as the current state, the action and the like; iteratively performing the process until the intelligent agent performs corresponding actions on all the states acquired from one CT image by using a sliding window method; the environment compares the classification result of each image block with the calibrated classification result by the ternary capsule network, correspondingly calculates a return value and stores the return value;
44 The system performs experience playback, randomly extracts a batch of data from the state, action and return combinations stored in the memory pool, calculates the loss of the capsule network and updates the network weight.
The calculation formula of the loss calculation and the network weight update of the capsule network is as follows:
Wherein L compose is the generated comprehensive loss value; b is the number of data selected by experience playback in reinforcement learning, namely, the number of experience data which is put in before is randomly selected from an experience pool; λ is the weight adjustment factor of two losses; Is a capsule network intelligent agent; alpha represents the minimum difference between the embedded expression distances of the anchor point sample and the negative sample and the anchor point sample and the positive sample; l i(d+,d-) is a ternary loss function.
Further, in S6, the specific process of determining the data analysis result of the image to be analyzed according to the classification result is:
Assuming that the samples s A to be detected and the number of C are the number of all the classes to be considered, randomly selecting N P samples from the existing labeling sample set for each class of samples, and obtaining N P multiplied by C samples in total; each sample s P is respectively combined with the sample s A to be tested to form an N P XC group sample pair I is E [0, C-1], after the distance corresponding to each group of sample pairs is calculated, the average distance corresponding to each category is calculated, and the category with the minimum average distance is used as a classification result.
A lung nodule analysis device based on a ternary capsule network algorithm, the device comprising:
a data storage module storing one or more programs;
the construction module is used for constructing a lung nodule analysis model based on a ternary capsule network algorithm;
the acquisition module is used for acquiring a lung image of a patient and taking the lung image as an image to be analyzed;
The analysis module is used for dividing the image to be analyzed into areas to form image data, inputting the image data into a pre-trained lung nodule analysis model and obtaining a classification result output by the lung nodule analysis model;
the analysis result determining module is used for determining an analysis result of the image to be analyzed according to the classification result;
the output module is used for outputting data of the analysis result and a lung nodule image;
a central processor executing one or more programs implementing the lung nodule analysis method based on a ternary capsule network algorithm as described above.
A computer-readable storage medium having stored thereon a computer program for execution by a processor to perform a lung nodule analysis method implementing a ternary capsule network algorithm based on the above.
Compared with the prior art, the lung nodule analysis method, the lung nodule analysis device and the storage medium based on the ternary capsule network algorithm provided by the invention have the following beneficial effects:
1) According to the invention, firstly, early training learning is carried out on the intelligent agent by using single-capsule network loss, after the performance improvement of the single-capsule network is gradually stopped to the best, the intelligent agent is further optimized based on ternary capsule network loss, so that the characteristic distance between classes is relatively larger, the intra-class distance is smaller, and therefore, samples of different classes can be distinguished in a finer granularity, and finally, the performance is obviously improved compared with that of the single-capsule network;
2) The method is based on a ternary loss measurement method, and can obtain higher accuracy rate for analyzing lung nodules than single-point loss and paired loss (twin network).
Drawings
Fig. 1 is a diagram of a single capsule network architecture of a lung nodule analysis method based on a ternary capsule network algorithm provided in an embodiment;
Fig. 2 is a diagram of a ternary capsule network architecture of a lung nodule analysis method based on a ternary capsule network algorithm according to an embodiment;
FIG. 3 is a flowchart of a lung nodule analysis method based on a ternary capsule network algorithm according to an embodiment;
FIG. 4 is a graph showing the comparison of the lung nodule analysis method and the average DQN classification performance based on the ternary capsule network algorithm in the example;
FIG. 5 is a graph of a confusion matrix of difference values between a lung nodule analysis method and DQN classification accuracy based on a ternary capsule network algorithm in an embodiment;
fig. 6 is a graph comparing a lung nodule analysis method based on a ternary capsule network algorithm with a sign graph generated by DQN reconstruction in an embodiment.
Detailed Description
The invention will now be described in detail with reference to the drawings and specific examples. It will be apparent that the described embodiments are some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
Examples
The invention relates to a lung nodule analysis method based on a ternary capsule network algorithm, which can be applied to the situation when lung nodules are analyzed according to lung images. The method may be performed by a lung nodule analysis apparatus, which may be implemented in software and/or hardware, e.g., which may be configured in a computer device. As shown in fig. 3, the lung nodule analysis method (hereinafter abbreviated as TriCaps-RL) based on the ternary capsule network algorithm specifically includes the following steps:
Step one, constructing a lung nodule analysis model based on a ternary capsule network algorithm, wherein the lung nodule analysis model comprises a memory pool module, a single capsule network module, a ternary capsule network module, an identification module and a result determination module. The specific contents are as follows:
Firstly inputting a large number of pictures, and encapsulating and learning image features after carrying out multistage convolution on the input pictures by a capsule network to generate an algorithm scheme; in this embodiment, as a preferred scheme, after two layers of convolution operation are performed on the adopted capsule network, features extracted by the convolution layer are sent to an initial capsule layer (PRIMARYCAPS) to be packaged, an 8-dimensional feature combination is generated, and then an 8-dimensional input space is mapped to 16 dimensions through a weight matrix of 8×16, so that a final feature expression of an image is obtained.
Setting initial learning parameters: the memory pool module has memory pool D, memory pool N, temporary memory pool D T, return value r, interval factor alpha, small lot number B, training iteration number EPOCHS, and capsule network initial parameter omega c.
For a 512 x 512 pixel CT image, the algorithm firstly uses a sliding window of K x K (K < 512) pixels to acquire different image blocks on the CT image one by one, and simultaneously records the left upper corner coordinates (x i,yi) of the image blocks; each image block is considered a state S t, and all image blocks extracted from the CT image constitute a set S of all states.
S t in the capsule network pair S selects and executes corresponding classification action a t from the action space A, and outputs a class attribute c j of the state; after the capsule network performs corresponding action a t on all image blocks s i acquired from one CT image, displaying lung nodules and sign classification results thereof detected by the capsule network back on the CT image according to the coordinate positions of the image block areas; thereby constructing a lung nodule analysis model based on a ternary capsule network algorithm.
And step two, acquiring an image to be analyzed.
Step three, learning a single-capsule network module, wherein a single-capsule network structure diagram is shown in fig. 1:
The size of a sliding window on a CT image is unchanged, so that the number of image blocks extracted by each CT image is determined, the state number M of each round in the reinforcement learning process is determined, and compared with the difference between a single-capsule network diagnosis result and a calibration result, the action return on each image block is correspondingly calculated, and the specific method comprises the following steps: for one image block s t, if the single capsule network diagnosis is correct, a bonus value r (this value is determined experimentally) is given; otherwise, giving a prize value of 0; the judging method for judging whether the single-capsule network diagnosis result is correct or not comprises the following steps:
1) If the region of the image block s t is overlapped with a focus region of the calibration result, further checking whether the classification result of the image block is consistent with the sign type label of the calibration result, if so, judging that the single-capsule network diagnosis result is correct, otherwise, judging that the diagnosis result is wrong;
2) If the region of the image block s t is not overlapped with any focus region of the calibration result and the classification result of the image block is negative, the single-capsule network diagnosis result is considered to be correct; otherwise, the single-capsule network diagnosis result is considered to be wrong.
Adding a layer of full-connection layer to output a cumulative reward Q target value corresponding to each category after feature expression by a single-capsule network:
Qtarget＝M×r (1)
the corresponding single step loss function is:
where L j(ωc) is the name of the loss function, Is capsule network intelligent agent, R (x) is return function,/>For anchor point images, representing one of the states in a round, a j represents the corresponding action, ω c represents the super-parameter.
And learning parameters of the single capsule network by adopting a loss function formula (2) of the single capsule network until the performance cannot be improved.
Step four, learning a ternary capsule network module, wherein a ternary capsule network structure diagram is shown in fig. 2:
three images are input at one time, respectively being anchor point images Positive sample image/>And negative sample image/>Features are extracted from the three images by using a capsule network respectively, so that Euclidean distance d + between the anchor point image and the positive sample image and Euclidean distance d - between the anchor point image and the negative sample image are calculated, and normalized to obtain the three images:
In the method, in the process of the invention, Capsule embedding expression representing anchor samples,/>Representing the embedded expression of positive samples,/>Representing the embedded representation of the negative sample.
The ternary loss function is calculated on the basis that:
wherein: alpha represents the minimum difference between the anchor sample and the negative sample, and the anchor sample and the positive sample are embedded in the expression distance.
The loss function of the single-layer capsule network is transited into a ternary loss function (5);
The ternary capsule network receives a state from the state set S, calculates Q target values corresponding to different categories, selects the action with the maximum value corresponding to the Q target value to execute, correspondingly enters the next state, and simultaneously stores elements such as the current state, the action and the like; iteratively performing the process until the intelligent agent performs corresponding actions on all the states acquired from one CT image by using a sliding window method respectively; the environment compares the classification result of each image block with the calibrated classification result by the ternary capsule network, and automatically calculates and stores a return value (rewarding value r).
The system executes experience playback, randomly extracts a batch from the state, action and return combination stored in the memory pool, calculates the loss of the capsule network according to the following formula (6) and updates the weight of the network:
Wherein L compose represents the integrated loss value generated by formula (6); b is the number of data selected by experience playback in reinforcement learning, namely, the number of experience data which is put in before is randomly selected from an experience pool; λ is two lost weight adjustment factors, which gradually increase with the increase of training iteration number n, and its calculation formula is:
EPOCHS in the formula (7) is a preset iteration value.
The using method of the formula (7) is as follows: when the iteration number n of the algorithm is smaller than EPOCHS, the following formula (8) is mainly adopted:
When the iteration number n is increased to be more than EPOCHS, the ternary loss shown by the formula (5) is mainly adopted, so that the classification accuracy of the intelligent agent is further fine-tuned, and classification distinction with finer granularity is realized on the sample. The effect of the unit loss stated above is to make the network converge faster.
Inputting a lung CT image into the learned capsule network, and performing sliding window processing on the CT image by the model to obtain a series of states.
Step six, the intelligent agent in the model selects the corresponding action with the largest return to the states according to the learned strategy, and outputs the analysis result, wherein the specific contents are as follows:
Assuming that the samples s A to be detected and the number of C are the number of all the classes to be considered, randomly selecting N P samples from the existing labeling sample set for each class of samples, and obtaining N P multiplied by C samples in total; combining each sample s P with the sample s A to form N P XC group sample pair I is E [0, C-1], the distance corresponding to each group of sample pairs is calculated, then the average distance corresponding to each category is calculated, and the category with the minimum average distance is used as a classification result, namely:
Wherein y * is the final output class, s A is the sample to be tested, y i is the i-th class, For the j-th sample belonging to y i in the test set, j ε [1, N P ].
According to the method, the analysis result of the image data to be analyzed is determined according to the classification result and is output, and the lung nodule analysis result of the patient is output as the sign of the lung nodule suffered by the patient.
To demonstrate the effectiveness of the method of the present invention, this example was experimentally verified. Experiments were performed on a sample set selected from the LIDC-IDRI library. The edge part of each nodule in LIDC-IDRI library is detailed sketched by a plurality of doctors, and the benign and malignant of the nodule are assessed, and the marking information is a record of the diagnosis operation of the real film reading process of the doctors. Therefore, in this embodiment, the marking data in LIDC-IDRI is used as the basis for the environment (simulated doctor) to evaluate and feedback rewards for the actions selected and performed by the agent in the method of the present invention.
For the convenience of experimental comparison, the positive CT images of the non-central calcification sign, the classification She Zheng, the burr sign and the GGO sign 4-type nodule sign are subjected to sample size increase through rotation and overturning operation according to the nodule feature classification provided by a radiologist during labeling, 12000 CT images are formed by symbiosis, and the experimental scheme is a 5-fold cross validation experiment.
For each CT image, a 32×32 pixel sliding window tile acquisition is performed, with a sliding interval of 11 pixels, and accordingly 46×46=2116 tiles can be taken from each CT image. In this experiment, if r=1 is given correctly for each agent identification, Q target corresponding to each image is 2116. The maximum capacity N of the playback memory pool D is set to 3k. 50 small batches of samples were taken from D at a time for training of the agent. For the epsilon greedy strategy, epsilon is a greedy weight, and this experiment sets epsilon to decrease linearly from 1 to 0.1 over 3000 iteration cycles. Further, according to experiments, for EPOCHS in the equation (5-8), 22000 was set.
In the stage of classification of the symptoms of the experiment, the category number C is 5. Each time, randomly selecting P n =10 individuals from each type of sample, taking P n ×c=50 samples, forming 50 sample pairs with the samples to be classified, and classifying according to a formula (9).
The change in accuracy of TriCaps-RL is relatively dramatic before 2600 steps of the learning process, which should be that the data in the empirical memory pool has not been updated all the time, the value of the loss function at this stage is unstable, rather than gradually decreasing with increasing iteration number, and after 2600 steps gradually tending to steadily increase with accumulation of experience. After 14000 iterations are performed, the accuracy of the algorithm gradually reaches a stable value, and after 22000 steps, the accuracy is slightly improved. This is because EPOCHS in equation (7) was set to 22000 in the experiment, after which the loss of the algorithm was gradually transitioned to be calculated from the ternary loss function shown in (5).
With the increasing number of training iterations, the feature embedded expressions of the same sign class samples are gradually aggregated, while the feature embedded expressions of different sign class samples are gradually separated.
A DQN (classical deep reinforcement learning algorithm) network was constructed according to the structure shown in table 1, comprising three convolutional layers, which were then trained with data from LIDC-IDRI, and the final test results were calculated and compared with TriCaps-RL for classification performance.
Table 1 CNN network architecture for comparison
The DQN differs from the TriCaps-RL method of the invention in that: a) Adopting a CNN network instead of a capsule network; 2) Only Q-learning objectives are employed, and triplet learning objectives are not considered.
The average classification performance of both methods is shown in figure 4. As can be seen from fig. 4, the classification effect of TriCaps-RL, whether in terms of sensitivity, specificity, or accuracy, is significantly better than DQN classification methods, demonstrating that the capsule network and triplet learning objectives employed herein are reasonably effective.
For finer granularity analysis, looking at the difference in performance of the two methods across different categories, this example constructs a differential confusion matrix of TriCaps-RL and DQN classification accuracy, as shown in figure 5. The floating point numbers on the diagonal represent the differences in the accuracy of the classification of the corresponding classes for TriCaps-RL and DQN, while the floating point numbers on the non-diagonal represent their differences in the classification error rate, showing the rate of misclassification into each of the other respective classes.
The values on the diagonal are TriCaps-RL and DQN classification accuracy differences; elements that are not on the diagonal are differences in classification error rates. As shown in FIG. 5, the element values on the diagonal of the difference confusion matrix are positive, and the average value of the diagonal numbers is 0.1047, which shows that the TriCaps-RL algorithm has higher accuracy on each class. Overall classification performance is better than DQN. No positive values appear in the elements on the non-diagonal, which means that the misclassification rate of TriCaps-RL is lower than DQN across all categories. Table 2 is a record of the average time and average predicted time for 20 samples for 5 trains of TriCaps-RL and DQN methods.
TABLE 2 TriCaps-RL and DQN methods training time and Classification time comparison
It can be seen from Table 2 that the TriCaps-RL method takes much longer to train than the DQN method, mainly because when training the TriCaps-RL method, the classification performance of the method is gradually shifted to policy optimization using the ternary loss metric when it is no longer improved. The ternary loss metric strategy increases the computational effort and also makes the gradient change small for each iteration, so it takes longer when the classification performance reaches a new balance. At the same time we can see that the classification stage TriCaps-RL method also takes longer than DQN, because the TriCaps-RL method takes the average after multiple sample similarity comparisons. However, the classification time is still less than 0.1 seconds, and the real-time interaction requirement can be fully met in the interaction process with doctors.
And respectively carrying out image reconstruction on the two network output characteristics so as to further compare and analyze the advantages and disadvantages of the characteristics acquired by the two networks. For the DQN network, adding a symmetrical deconvolution network on the basis of the CNN network shown in the table 1 to realize reconstruction; for the present algorithm, reconstruction is achieved by adding a reconstruction network behind the capsule network according to the scheme in document Dynamic Routing Between Capsules. Both after 30000 training, the input sign samples were regenerated with a single capsule network of TriCaps-RL and CNN in DQN, respectively. Fig. 6 shows a comparison of the two results of the generation, the first row in the figure being the original figure, and the second and third rows being the sign blocks generated by the capsule network and CNN, respectively. From the figure, the sign image block generated by the capsule network is obviously finer and more obvious than the sign image block generated by the CNN; the CNN generated images lost significantly much detail, especially for the 1 st and 9 th symbologies, or the 14 th and 18 th symbologies, with large lesion areas, where the CNN failed to represent its features, only a blank image was generated. The reconstruction of the capsule network was not very good for the 9 th sign plot, but we could find that it was also very difficult for the human eye to discern the texture of the sign for this sample.
Correspondingly, the invention also provides a lung nodule analysis device based on a ternary capsule network algorithm, which comprises:
A data storage module for storing one or more programs;
The construction module is used for constructing a lung nodule analysis model based on a ternary capsule network algorithm;
The acquisition module is used for acquiring a lung image of a patient and taking the lung image as an image to be analyzed;
the analysis module is used for dividing the image to be analyzed into areas to form image data, inputting the data into a pre-trained lung nodule analysis model, and obtaining a classification result output by the analysis model;
the analysis result determining module is used for determining an analysis result of the image to be analyzed according to the classification result;
the output module is used for outputting data of analysis results and lung nodule images;
And the central processing unit is used for executing the one or more programs, and the programs are used for realizing the analysis method for the lung nodules provided by the invention.
Further, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the lung nodule analysis method as provided above.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions may be made without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (4)
1. The lung nodule analysis method based on the ternary capsule network algorithm is characterized by comprising the following steps of:
1) Constructing a lung nodule analysis model based on a ternary capsule network algorithm, wherein the lung nodule analysis model comprises a memory pool module, a single capsule network module, a ternary capsule network module, an identification module and a result determination module;
2) Acquiring an image to be analyzed;
3) Learning a single-capsule network module, and training a capsule network intelligent agent;
4) Performing ternary capsule network module learning, and optimizing a capsule network intelligent agent based on ternary capsule network loss;
5) Inputting a lung CT image into the learned capsule network, and performing sliding window processing on the CT image by the model to obtain a series of states;
6) Determining a data analysis result of the image to be analyzed according to the classification result, and outputting the data analysis result through a result determination module;
in the step 1), the specific contents of constructing a lung nodule analysis model based on a ternary capsule network algorithm are as follows:
11 A plurality of pictures are input, initial learning parameters are set, wherein the initial learning parameters comprise the storage amount N of a memory pool D in a memory pool module, the capacity H of a temporary memory pool D T, a return value r, an interval factor alpha, the number B of small batches, the training iteration number EPOCHS and a capsule network initial parameter omega c;
12 For a 512 x 512 pixel CT image, acquiring a 32 x 32 pixel image block on the CT image by adopting a K x K pixel sliding window, and simultaneously recording the upper left corner coordinate (x i,yi) of the image block; setting each image block as a state S t, and further forming all image blocks extracted from the CT image into a set S of all states;
13 The capsule network selects and executes the corresponding classification action a t from the action space A for the state S t in the set S and outputs the class attribute c j of the state;
14 After the capsule network acquires all the image blocks s i from a CT image and executes the corresponding action a t, displaying the lung nodule detected by the capsule network and the sign classification result thereof back to the CT image according to the coordinate position of the image block area, thereby constructing a lung nodule analysis model based on a ternary capsule network algorithm;
in the step 3), the specific content of the learning of the single-capsule network module is as follows:
Keeping the size of a sliding window on a CT image unchanged, determining the number of image blocks extracted by each CT, determining the state number M of each round in the reinforcement learning process, comparing the difference between a single-capsule network diagnosis result and a calibration result, and correspondingly calculating action return on each image block;
the specific method for comparing the dissimilarity between the single-capsule network diagnosis result and the calibration result is as follows:
For an image block s t, if the single capsule network diagnosis is correct, giving a reward value, namely a return value r; otherwise, giving a prize value of 0; the judging method for judging whether the single-capsule network diagnosis result is correct or not comprises the following steps:
A: if the region of the image block s t is overlapped with a focus region of the calibration result, further checking whether the classification result of the image block is consistent with the sign type label of the calibration result, if so, judging that the single-capsule network diagnosis result is correct, otherwise, judging that the diagnosis result is wrong;
b: if the image block s t area is not overlapped with any focus area of the calibration result and the classification result of the image block is negative, judging that the single-capsule network diagnosis result is correct, otherwise, judging that the single-capsule network diagnosis result is wrong;
In the step 3), a single capsule network is added with a layer of full connection layer after feature expression to output the accumulated rewards Q target value corresponding to each category:
Qtarget＝M×r
the corresponding single step loss function is:
Where L (ω c) is the name of the loss function, R (x) is the return function, Is a network intelligent agent of the capsule,As anchor point images, a j is corresponding action, omega c is super-parameter;
Learning parameters of the single capsule network by adopting a corresponding single-step loss function of the single capsule network until the loss function is smaller than a preset value;
in the step 4), the specific content of the learning of the ternary capsule network module is as follows:
41 One time three images are input, respectively being anchor point images Positive sample image/>And negative sample image/>Extracting features from the three images by using a capsule network respectively, further calculating Euclidean distance d+ between the anchor point image and the positive sample image and Euclidean distance d - between the anchor point image and the negative sample image, and normalizing the Euclidean distance d+ and the Euclidean distance d -; and on the basis, a ternary loss function is calculated;
42 A loss function of the single-layer capsule network is converted into a ternary loss function;
43 Enabling the ternary capsule network to receive one state from the state set S, calculating rewards Q target values corresponding to different categories, selecting the action with the maximum corresponding Q target value to execute, correspondingly entering the next state, and simultaneously storing elements such as the current state, the action and the like; iteratively performing the process until the intelligent agent performs corresponding actions on all the states acquired from one CT image by using a sliding window method; the environment compares the classification result of each image block with the calibrated classification result by the ternary capsule network, correspondingly calculates a return value and stores the return value;
44 The system executes experience playback, randomly extracts a batch of data from the state, action and return combination stored in the memory pool, calculates the loss of the capsule network and updates the network weight;
The calculation formula of the loss calculation and the network weight update of the capsule network is as follows:
Wherein L compose is the generated comprehensive loss value; b is the number of data selected by experience playback in reinforcement learning, namely, the number of experience data which is put in before is randomly selected from an experience pool; λ is the weight adjustment factor of two losses; is a capsule network intelligent agent; alpha represents the minimum difference between the embedded expression distance of the anchor point sample and the negative sample; l i(d+,d-) is a ternary loss function.
2. The pulmonary nodule analysis method based on the ternary capsule network algorithm according to claim 1, wherein in step 6), the specific process of determining the data analysis result of the image to be analyzed according to the classification result is:
Assuming that the samples s A to be detected and the number of C are the number of all the classes to be considered, randomly selecting N P samples from the existing labeling sample set for each class of samples, and obtaining N P multiplied by C samples in total; each sample s P is respectively combined with the sample s A to be tested to form an N P XC group sample pair After calculating the distance corresponding to each group of sample pairs, calculating the average distance corresponding to each category, and taking the category with the minimum average distance as a classification result.
3. A pulmonary nodule analysis device based on a ternary capsule network algorithm, comprising:
a data storage module storing one or more programs;
the construction module is used for constructing a lung nodule analysis model based on a ternary capsule network algorithm;
the acquisition module is used for acquiring a lung image of a patient and taking the lung image as an image to be analyzed;
The analysis module is used for dividing the image to be analyzed into areas to form image data, inputting the image data into a pre-trained lung nodule analysis model and obtaining a classification result output by the lung nodule analysis model;
the analysis result determining module is used for determining an analysis result of the image to be analyzed according to the classification result;
the output module is used for outputting data of the analysis result and a lung nodule image;
a central processor executing one or more programs implementing the lung nodule analysis method based on a ternary capsule network algorithm according to any one of claims 1-2.
4. A computer readable storage medium having stored thereon a computer program, wherein the computer program is executed by a processor to implement the lung nodule analysis method based on a ternary capsule network algorithm according to any of claims 1-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110946389.9A CN113763332B (en) | 2021-08-18 | 2021-08-18 | Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110946389.9A CN113763332B (en) | 2021-08-18 | 2021-08-18 | Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113763332A CN113763332A (en) | 2021-12-07 |
| CN113763332B true CN113763332B (en) | 2024-05-31 |
Family
ID=78790266
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110946389.9A Active CN113763332B (en) | 2021-08-18 | 2021-08-18 | Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113763332B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117726992B (en) * | 2024-02-07 | 2024-04-16 | 吉林大学 | Nursing skill training auxiliary system and method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830826A (en) * | 2018-04-28 | 2018-11-16 | 四川大学 | A kind of system and method detecting Lung neoplasm |
| CN110570425A (en) * | 2019-10-18 | 2019-12-13 | 北京理工大学 | Lung nodule analysis method and device based on deep reinforcement learning algorithm |
| CN111325169A (en) * | 2020-02-26 | 2020-06-23 | 河南理工大学 | Deep video fingerprint algorithm based on capsule network |
| CN111767952A (en) * | 2020-06-30 | 2020-10-13 | 重庆大学 | Interpretable classification method for benign and malignant pulmonary nodules |
| CN112733701A (en) * | 2021-01-07 | 2021-04-30 | 中国电子科技集团公司信息科学研究院 | Robust scene recognition method and system based on capsule network |
| CN113208641A (en) * | 2021-05-10 | 2021-08-06 | 山东大学 | Pulmonary nodule auxiliary diagnosis method based on three-dimensional multi-resolution attention capsule network |
-
2021
- 2021-08-18 CN CN202110946389.9A patent/CN113763332B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108830826A (en) * | 2018-04-28 | 2018-11-16 | 四川大学 | A kind of system and method detecting Lung neoplasm |
| CN110570425A (en) * | 2019-10-18 | 2019-12-13 | 北京理工大学 | Lung nodule analysis method and device based on deep reinforcement learning algorithm |
| CN111325169A (en) * | 2020-02-26 | 2020-06-23 | 河南理工大学 | Deep video fingerprint algorithm based on capsule network |
| CN111767952A (en) * | 2020-06-30 | 2020-10-13 | 重庆大学 | Interpretable classification method for benign and malignant pulmonary nodules |
| CN112733701A (en) * | 2021-01-07 | 2021-04-30 | 中国电子科技集团公司信息科学研究院 | Robust scene recognition method and system based on capsule network |
| CN113208641A (en) * | 2021-05-10 | 2021-08-06 | 山东大学 | Pulmonary nodule auxiliary diagnosis method based on three-dimensional multi-resolution attention capsule network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113763332A (en) | 2021-12-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Singh et al. | Machine learning in cardiac CT: basic concepts and contemporary data | |
| Carass et al. | Longitudinal multiple sclerosis lesion segmentation: resource and challenge | |
| WO2021159742A1 (en) | Image segmentation method and apparatus, and storage medium | |
| Saikumar et al. | A novel implementation heart diagnosis system based on random forest machine learning technique. | |
| KR101874348B1 (en) | Method for facilitating dignosis of subject based on chest posteroanterior view thereof, and apparatus using the same | |
| Tian et al. | Multi-path convolutional neural network in fundus segmentation of blood vessels | |
| CN107730542B (en) | Cone beam computed tomography image correspondence and registration method | |
| Šajn et al. | Image processing and machine learning for fully automated probabilistic evaluation of medical images | |
| CN111009321A (en) | Application method of machine learning classification model in juvenile autism auxiliary diagnosis | |
| Zhang et al. | A semi-supervised learning approach for COVID-19 detection from chest CT scans | |
| CN113096137A (en) | Adaptive segmentation method and system for OCT (optical coherence tomography) retinal image field | |
| CN110570425B (en) | Pulmonary nodule analysis method and device based on deep reinforcement learning algorithm | |
| CN113763332B (en) | Pulmonary nodule analysis method and device based on ternary capsule network algorithm and storage medium | |
| Zhang et al. | Learning from multiple annotators for medical image segmentation | |
| Maffei et al. | Radiomics classifier to quantify automatic segmentation quality of cardiac sub-structures for radiotherapy treatment planning | |
| Tian et al. | Radiomics and its clinical application: artificial intelligence and medical big data | |
| Bhat et al. | Identification of intracranial hemorrhage using ResNeXt model | |
| Kakani et al. | Post-covid chest disease monitoring using self adaptive convolutional neural network | |
| Dhinagar et al. | Efficiently Training Vision Transformers on Structural MRI Scans for Alzheimer’s Disease Detection | |
| Wang et al. | Screening and diagnosis of cardiovascular disease using artificial intelligence-enabled cardiac magnetic resonance imaging | |
| Lu et al. | Data enhancement and deep learning for bone age assessment using the standards of skeletal maturity of hand and wrist for chinese | |
| Taboada-Crispi et al. | Anomaly detection in medical image analysis | |
| CN116958679A (en) | Target detection method based on weak supervision and related equipment | |
| Sameer et al. | Brain tumor segmentation and classification approach for MR images based on convolutional neural networks | |
| US20210279879A1 (en) | Similarity determination apparatus, similarity determination method, and similarity determination program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |