CN114463583A - Deep hashing method for pneumonia CT image classification - Google Patents

Abstract

The invention relates to the technical field of medical image processing, in particular to a depth hashing method for pneumonia CT image classification. The pneumonia CT image data set is established, the sizes of pneumonia CT images are unified, and the pneumonia CT images are divided into training sets T_rAnd test set T_e(ii) a Then, a deep Hash network model is constructed, and the similarity loss L is calculated according to Hash codes_SAnd the contrast loss L_clConstructing a total loss function L; second, multi-task Hash training is introducedStrategy, optimizing a loss function L by using an alternative learning algorithm, and storing a deep Hash network model; and finally, reading the test set to classify the CT images. The method has the advantages that the deep hash method can accurately find out the tiny difference among CT images of different lungs, so that the training model greatly reduces the storage space and the training time, the efficiency of classifying large-scale pneumonia CT images is effectively improved, the advantage of extracting the characteristics of fine-grained CT images by the bilinear convolutional neural network is fully exerted, and the identification accuracy and the generalization robustness are effectively improved.

Description

Deep hashing method for pneumonia CT image classification

Technical Field

The invention relates to the technical field of medical image processing, in particular to a depth hashing method for pneumonia CT image classification.

Background

The lung CT image is the gold standard for the diagnosis of pneumonia, and different pneumonia types are shown differently on the lung CT image. The bilateral thorax of the lung CT image of a normal person is symmetrical, the lung permeability and the lung texture are normal, thickening and disorder are avoided, and obvious abnormal density shadow is avoided in the field of the two lungs; the lung CT image of a patient with COVID-19 has the early change of the small spot sheet-shaped shadow of the extrapulmonary zone and the pulmonary interstitium, and further develops the frosted shadow, the infiltration shadow and the lung consolidation of two lungs, and chest and abdomen effusion, lung pathological change and lung consolidation in different degrees can also appear in serious cases, consolidation areas mainly present diffuse alveolar injury and exudative alveolitis, serous fluid, fibrin exudate and transparent film can be seen in the alveolar cavity, and the small bronchus and the bronchiole are also formed by mucus plugs; for a CT image of the lung of a patient with common pneumonia, if the common community acquired pathogenic bacteria cause lobar pneumonia, the lobar pneumonia is often represented by a spot-shaped density increase image with limited distribution of lung lobes or lung segments in an imaging way, the patient often has a clear history of respiratory tract infection, and if the pulmonary inflammation exudation of the patient with lobular pneumonia is caused, the spot-shaped exudation is often taken as the main factor.

However, in the current big data environment, tens of thousands of CT images are generated every day, and how to effectively classify the CT images is a difficult problem to be solved. Conventionally, the subjective judgment of doctors is usually needed, so that a large amount of manpower is consumed, a large amount of time is wasted, the conditions of wrong judgment and misjudgment are caused more seriously, and the life health of a patient is also influenced to a certain extent.

The rise of deep learning has driven the development of the field of medical image processing, especially in the field of lung CT image classification. At present, the deep features of images are mostly extracted by Convolutional Neural Networks (CNNs) in the classification of lung CT images, so that different types are distinguished. However, the difference between the lung CT images often appears in the lesion region, and other regions are similar, that is, the similarity of the medical images of the same part is often very large, which has a certain influence on the accuracy of classification.

Under the background, the invention provides a method for extracting fine-grained features of lung CT images by using a Bilinear Convolutional Neural Network (BCNN), finding out small differences among different lung CT images, and mapping the extracted depth features to a binary Hamming space so as to classify the lung CT images. Therefore, the classification accuracy is improved, and the classification time is saved.

Disclosure of Invention

The invention aims to solve the problems and provides a deep hash method for pneumonia CT image classification.

In order to achieve the purpose, the invention adopts the following technical scheme:

a depth hashing method for pneumonia CT image classification comprises the following steps:

s10: establishing a pneumonia CT image data set;

s20: data preprocessing, namely firstly enhancing and expanding the data of the data set, and dividing the data set into training sets T according to the proportion of 80 percent and 20 percent_r＝(x₁,x₂,...,x_n) N1, 2, N and test set T_e＝(y₁,y₂,...,y_m) M 1, 2.. multidot.m, then the size of the pneumonia CT images is uniformly adjusted to 224 × 224, the number of channels is 1, 3 types of CT images are included in the data set, namely, the lung CT image of a normal person, the lung CT image of a COVID-19 patient and the lung CT image of a general pneumonia patient,finally constructing a training set T_rOf the similarity matrix S, wherein

And S_ij∈R^N×N,i,j＝1,2,...,N；

S30: the method comprises the steps that a deep hash network model is built, wherein the model comprises a bilinear feature learning module and a hash coding learning module, when the model is trained, fine-grained features of a pneumonia CT image are extracted by using a Bilinear Convolutional Neural Network (BCNN), and then the extracted fine-grained features are input into the hash coding learning module, so that the fine-grained features of the corresponding pneumonia CT image are mapped into binary hash codes;

s40: calculating 2 losses, i.e. similarity loss L, from the hash code obtained in step S30_SAnd the contrast loss L_clAnd defining the total loss function as: l ═ L_S+αL_clWherein α ═ 0.1 is a weighting factor;

s50: introducing a multi-task Hash training strategy, repeatedly using a bilinear feature learning module, and extracting a bilinear feature vector v' (x) of the pneumonia CT image_i)∈R^262144×1,v'(x_j)∈R^262144×1I, j is 1, 2.. times, N, and i ≠ j is respectively learned through a hash code learning module comprising 4 branches, and each branch comprises 2 fully-connected layers and 1 fully-connected hash layer, so that the model can learn 12,24,32 and 48-bit hash codes at the same time;

s60: using alternate learning algorithm to target function

Performing optimization updating on the depth Hash network model parameter theta, the Hash coding matrix B, the weight matrix W and the offset vector V, and storing the model;

s70: first read test set T using a pre-trained model_ePneumonia CT image y_kK 1, 2.. times.m, which yields its hash code

C is 12,24,32,48, then

And Hash coding matrix B belongs to R^c×NC is compared with each column of 12,24,32 and 48, the first 5 with smaller Hamming distance are compared, and the number of the categories belonging to the same is larger, then y is compared_kAnd classifying the test set into the categories, and finally calculating the average accuracy of the classification of the test set.

As a preferred technical scheme of the invention: in step S30, the bilinear feature learning module mainly includes two branches a and B, and branch a and branch B are composed of two identical VGG16 models, the convolutional layer conv of each branch is divided into 5 segments, 13 convolutional layers in total, and the size of the convolutional core of each convolutional layer is 3 × 3, the step size stride and padding are both set to 1, there is a maximum pooling layer maxpool after the first 4 segments of convolutional layers, and the pooling frame size is 2 × 2, the step size stride is set to 2, taking branch a as an example, the specific steps of the designed network structure are as follows:

s31: firstly, training set T of pneumonia CT image_rRandomly partitioning into image pairs (x)_i,x_j) I, j is 1, 2.. multidot.n, and i ≠ j, reads the image pair and the similarity matrix S, and then after passing through the 1 st convolution layer conv1, conv2 with the filter number of 64, the size of the extracted pneumonia CT image feature map is 224 × 224 × 64, and then passes through the maximum pooling layer maxpool1, and the final output size of the pneumonia CT image feature map is 112 × 112 × 64;

s32: after the output of the maxpool1 passes through the 2 nd convolution layers conv3 and conv4 with the filter number of 128, the size of the extracted pneumonia CT image feature map is 112 × 112 × 128, and then the pneumonia CT image feature map passes through the maximum pooling layer maxpool2, and the final output size of the pneumonia CT image feature map is 56 × 56 × 128;

s33: after passing the output of maxpool2 through the 3 rd convolution layers conv5, conv6 and conv7 with the filter number of 256, the size of the extracted pneumonia CT image feature map is 56 × 56 × 256, and then the extracted pneumonia CT image feature map passes through the maximum pooling layer maxpool3, and the final output size of the pneumonia CT image feature map is 28 × 28 × 256;

s34: after the output of maxpool3 passes through 4 th convolution layers conv8, conv9 and conv10 with the number of filter 512, the size of the extracted pneumonia CT image feature map is 28 × 28 × 512, and then the feature map passes through the maximum pooling layer maxpool4, and the final output size of the pneumonia CT image feature map is 14 × 14 × 512;

s35: after the output of maxpool4 passes through 5 th convolution layers conv11, conv12 and conv13 with the number of filter 512, the output size of the feature map of the extracted pneumonia CT image is 14 multiplied by 512;

in this case, the feature of the CT image of pneumonia extracted from the branch A is denoted by F_A(x_i)∈R^14×14×512、F_A(x_j)∈R^{14 ×14×512}The feature of the pneumonia CT image extracted by the branch B is F_B(x_i)∈R^14×14×512、F_B(x_j)∈R^14×14×512；

S36: the outputs of branch A and branch B were passed through bilinear pooling layer bilinearcholing to obtain a CT image (x) of pneumonia_i,x_j) I, j ≠ N, 1,2, and i ≠ j is a deep feature f at position l_A(l,x_i)∈R^1×512、f_B(l,x_i)∈R^{1 ×512}And f_A(l,x_j)∈R^1×512、f_B(l,x_j)∈R^1×512The following operations were carried out:

(1) calculating pneumonia CT image pair (x)_i,x_j) I, j ≠ N, is 1,2, and i ≠ j is a bilinear feature at position l

(2) Pooling pneumonia CT image pairs by summation pooling (x)_i,x_j) I, j ≠ N, i ≠ j, which is a bilinear feature of all positions to obtain global bilinearFeature(s)

(3) Global bilinear feature matrix xi (x)_i) And xi (x)_j) Stretch into a vector to obtain v (x)_i)＝vec(ξ(x_i))∈R^262144×1、v(x_j)＝vec(ξ(x_j))∈R^262144×1Where vec (·) represents the operation of expanding the matrix into vectors;

(4) feature vector v (x) of pneumonia CT image_i) And v (x)_j) Performing normalization operation to obtain

Wherein | · | purple₂An L2 norm representing a vector;

s37: the feature vector v' (x) of the normalized pneumonia CT image is obtained_i)∈R^262144×1、v'(x_j)∈R^262144×1V "(x) was obtained after passing 2 full-junction layers FC1 and FC2 in this order_i)、v”(x_j)∈R^c×1C is 12,24,32,48, c represents the number of bits of the hash code;

s38: feature vector v ″ (x) of the pneumonia CT image after passing through the full connection layer_i)、v”(x_j) Mapping to binary hash b according to hash function h (·)_i,b_j∈R^c×1C is 12,24,32,48, finally obtaining the training set T_rIs the hash coding matrix B ∈ R^c×N，

Where h (-) represents a hash function and sign (-) represents a sign function.

As an originalThe preferred technical scheme of the invention is as follows: the step S40 obtains the hash code B e R according to S38^c×NC-12, 24,32,48 calculates 2 losses, i.e. the similarity loss L_SAnd the contrast loss L_clThe method comprises the following specific steps:

s41: for the input pneumonia CT image pair (x)_i,x_j) I, j ≠ N, 1,2, and i ≠ j, with similarity penalty L_STo optimize the distance between similar samples and to enlarge the distance between dissimilar samples, the formula is as follows:

wherein

Hash coding b_i、b_j∈R^c×1,c＝12,24,32,48；

S42: in pneumonia CT image x_nN1, 2, a deep level characteristic v (x) of N_n) And corresponding hash codes b_nSolving the contrast loss L_clThe formula is as follows:

wherein v (x)_n)＝W^TΦ(x_n(ii) a Theta) + V, theta representing all parameters of the deep hash network model, phi (x)_n；θ)∈R^262144×1Represents an input into the full connectivity layer FC1, W ∈ R^262144×cIs a weight matrix, V ∈ R^c×1Is a bias vector, | · | | non-conducting phosphor₂An L2 norm representing a vector;

s43: the overall loss function is defined as: l ═ L_S+αL_clWhere α ═ 0.1 is a weighting factor, so when optimizing the network model, the objective function should be set to minimize the loss function, which is formulated as follows:

wherein S_ijIs a matrix of the degree of similarity, and,

and is

b_jIs a pneumonia CT image x_iTransposed, pneumonia CT image x of hash coding of (a)_jHash coding of b_nRepresentative pneumonia CT image x_nN1, 2, hash coding of N, W ∈ R^262144×cIs a weight matrix, phi (x)_n；θ)∈R^262144×1Representing the input into the full connectivity layer FC1, V ∈ R^c×1Is a bias vector, | · Lixian₂Representing the L2 norm of the vector.

As a preferred technical scheme of the invention: step S50 introduces a multitask hash training strategy, and reuses the bilinear feature learning module to extract bilinear feature vector v '(x') of the CT image of pneumonia_i)∈R^262144×1,v'(x_j)∈R^262144×1I, j ≠ N, and i ≠ j passes through the hash code learning modules of 4 branches, and each branch includes 2 fully-connected hash layers and 1 fully-connected hash layer, so that the model can learn 12,24,32, and 48-bit hash codes at the same time, taking branch 1 as an example, and then c ═ 12, the specific steps are as follows:

s51: firstly, a feature vector v' (x) of a pneumonia CT image is obtained_i)∈R^262144×1,v'(x_j)∈R^262144×1V "(x) is obtained by the full junction layers FC11, FC12 of Branch 1_i)∈R^12×1、v”(x_j)∈R^12×1；

S52: then, the feature vector v ″ (x) of the pneumonia CT image is determined_i)、v”(x_j) Mapping to binary hash code b according to hash function h (·)_i、b_j∈R^12×1Finally, a training set T is formed_rIs the hash coding matrix B ∈ R^12×N，

Where h (-) represents a hash function and sign (-) represents a sign function.

Compared with other lung CT image classification methods, the lung CT image classification method has the beneficial effects that:

firstly, the convolutional neural network CNN can extract deep features of an image, but the difference between lung CT images is not too large, which results in that the accuracy of lung CT image classification is not very high. Therefore, the method effectively extracts the subtle differences among different symptoms of the lung CT image by using the bilinear convolutional neural network BCNN, fully exerts the extraction advantages of the bilinear convolutional neural network on the fine-grained image characteristics, and effectively improves the identification accuracy and the generalization robustness of the model;

secondly, if the real-value features extracted by the bilinear convolutional neural network BCNN are directly classified, the problems of overlarge storage space and overlong training time can be caused. Therefore, Hash coding learning is introduced, and real-value features extracted by the bilinear convolutional neural network BCNN are mapped to a binary Hamming space, so that the storage space is greatly reduced and the training time is shortened when a model is trained;

the bilinear feature learning module can be repeatedly used, so that the lung CT image can be simultaneously generated into Hash codes with different code lengths by the model, and the calculation time and the storage space are further saved.

Drawings

FIG. 1 is a flow chart of the present invention for data enhancement and augmentation of a collected CT image dataset of pneumonia;

FIG. 2 is a block diagram of a depth hash method for classifying CT images of pneumonia according to the present invention;

fig. 3 is a final frame diagram of a depth hashing method for classifying pneumonia CT images, which introduces a multitask hashing training strategy.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. Of course, the examples described by reference to the drawings are only for the purpose of illustrating the invention and are not to be construed as limiting the invention.

As shown in fig. 1 to 3, a depth hashing method for pneumonia CT image classification includes the following steps:

s10: establishing a pneumonia CT image data set;

s20: data preprocessing, namely firstly enhancing and expanding the data of the data set, and dividing the data set into training sets T according to the proportion of 80 percent and 20 percent_r＝(x₁,x₂,...,x_n) N1, 2, N and test set T_e＝(y₁,y₂,...,y_m) M is 1,2, 1, m, then the sizes of the pneumonia CT images are uniformly adjusted to 224 multiplied by 224, the number of channels is 1, 3 types of CT images are contained in the data set, namely, the lung CT image of a normal person, the lung CT image of a COVID-19 patient and the lung CT image of a common pneumonia patient, and finally a training set T is constructed_rOf the similarity matrix S, wherein

And S_ij∈R^N×N,i,j＝1,2,...,N；

S30: the method comprises the steps that a deep hash network model is built, wherein the model comprises a bilinear feature learning module and a hash coding learning module, when the model is trained, fine-grained features of a pneumonia CT image are extracted by using a Bilinear Convolutional Neural Network (BCNN), and then the extracted fine-grained features are input into the hash coding learning module, so that the fine-grained features of the corresponding pneumonia CT image are mapped into binary hash codes;

s40: calculating 2 losses, i.e. similarity loss L, from the hash code obtained in step S30_SAnd the contrast loss L_clAnd defining the total loss function as: l ═ L_S+αL_clWherein α ═ 0.1 is a weighting factor;

s50: introducing multitask Hash trainingRepeating the bilinear feature learning module to extract bilinear feature vector v '(x') of the pneumonia CT image_i)∈R^262144×1,v'(x_j)∈R^262144×1I, j is 1, 2.. times, N, and i ≠ j is respectively learned through a hash code learning module comprising 4 branches, and each branch comprises 2 fully-connected layers and 1 fully-connected hash layer, so that the model can learn 12,24,32 and 48-bit hash codes at the same time;

s60: using alternate learning algorithm to target function

Performing optimization updating on the depth Hash network model parameter theta, the Hash coding matrix B, the weight matrix W and the offset vector V, and storing the model;

s70: first read test set T using a pre-trained model_ePneumonia CT image y_kK 1, 2.. times.m, which yields its hash code

C is 12,24,32,48, then

And Hash coding matrix B belongs to R^c×NC is compared with each column of 12,24,32 and 48, the first 5 with smaller Hamming distance are compared, and the number of the categories belonging to the same is larger, then y is compared_kAnd classifying the test set into the categories, and finally calculating the average accuracy of the classification of the test set.

In step S30, the bilinear feature learning module mainly includes two branches a and B, and branch a and branch B are composed of two identical VGG16 models, the convolutional layer conv of each branch is divided into 5 segments, 13 convolutional layers in total, and the size of the convolutional core of each convolutional layer is 3 × 3, the step size stride and padding are both set to 1, there is a maximum pooling layer maxpool after the first 4 segments of convolutional layers, and the pooling frame size is 2 × 2, the step size stride is set to 2, taking branch a as an example, the specific steps of the designed network structure are as follows:

s31: firstly, a pneumonia CT image is obtainedTraining set T of_rStochastic partitioning into image pairs (x)_i,x_j) I, j is 1, 2.. multidot.n, and i ≠ j, reads the image pair and the similarity matrix S, and then after passing through the 1 st convolution layer conv1, conv2 with the filter number of 64, the size of the extracted pneumonia CT image feature map is 224 × 224 × 64, and then passes through the maximum pooling layer maxpool1, and the final output size of the pneumonia CT image feature map is 112 × 112 × 64;

s32: after the output of the maxpool1 passes through the 2 nd convolution layers conv3 and conv4 with the filter number of 128, the size of the extracted pneumonia CT image feature map is 112 × 112 × 128, and then the pneumonia CT image feature map passes through the maximum pooling layer maxpool2, and the final output size of the pneumonia CT image feature map is 56 × 56 × 128;

s33: after passing the output of maxpool2 through the 3 rd convolution layers conv5, conv6 and conv7 with the filter number of 256, the size of the extracted pneumonia CT image feature map is 56 × 56 × 256, and then the extracted pneumonia CT image feature map passes through the maximum pooling layer maxpool3, and the final output size of the pneumonia CT image feature map is 28 × 28 × 256;

s34: after the output of maxpool3 passes through 4 th convolution layers conv8, conv9 and conv10 with the number of filter 512, the size of the extracted pneumonia CT image feature map is 28 × 28 × 512, and then the feature map passes through the maximum pooling layer maxpool4, and the final output size of the pneumonia CT image feature map is 14 × 14 × 512;

s35: after the output of maxpool4 passes through 5 th convolution layers conv11, conv12 and conv13 with the number of filter 512, the output size of the feature map of the extracted pneumonia CT image is 14 multiplied by 512;

in this case, the feature of the CT image of pneumonia extracted from the branch A is denoted by F_A(x_i)∈R^14×14×512、F_A(x_j)∈R^{14 ×14×512}The pneumonia CT image extracted from branch B is characterized by F_B(x_i)∈R^14×14×512、F_B(x_j)∈R^14×14×512；

S36: the outputs of branch A and branch B were passed through bilinear pooling layer bilinearcholing to obtain a CT image (x) of pneumonia_i,x_j) I, j ≠ N, 1,2, and i ≠ j is a deep feature f at position l_A(l,x_i)∈R^1×512、f_B(l,x_i)∈R^{1 ×512}And f_A(l,x_j)∈R^1×512、f_B(l,x_j)∈R^1×512The following operations were carried out:

(1) calculating pneumonia CT image pair (x)_i,x_j) I, j ≠ N, is 1,2, and i ≠ j is a bilinear feature at position l

(2) Pooling pneumonia CT image pairs by summing pooled sampling_i,x_j) I, j ≠ N, i ≠ j, which is a bilinear feature at all positions to obtain a global bilinear feature

(3) Global bilinear feature matrix xi (x)_i) And xi (x)_j) Stretch into a vector to obtain v (x)_i)＝vec(ξ(x_i))∈R^262144×1、v(x_j)＝vec(ξ(x_j))∈R^262144×1Where vec (·) represents the operation of expanding the matrix into vectors;

(4) feature vector v (x) of pneumonia CT image_i) And v (x)_j) Performing normalization operation to obtain

Wherein | · | purple₂An L2 norm representing a vector;

s37: the feature vector v' (x) of the normalized pneumonia CT image is obtained_i)∈R^262144×1、v'(x_j)∈R^262144×1V "(x) was obtained after passing 2 full-junction layers FC1 and FC2 in this order_i)、v”(x_j)∈R^c×1C is 12,24,32,48, c represents the number of bits of the hash code;

s38: feature vector v ″ (x) of the pneumonia CT image after passing through the full connection layer_i)、v”(x_j) Mapping to binary hash b according to hash function h (·)_i,b_j∈R^c×1C is 12,24,32,48, finally obtaining the training set T_rIs the hash coding matrix B ∈ R^c×N，

Where h (-) represents a hash function and sign (-) represents a sign function.

The step S40 obtains the hash code B e R according to S38^c×NC-12, 24,32,48 calculates 2 losses, i.e. the similarity loss L_SAnd the contrast loss L_clThe method comprises the following specific steps:

s41: for the input pneumonia CT image pair (x)_i,x_j) I, j ≠ N, and i ≠ j, with similarity penalties L_STo optimize the distance between similar samples and to enlarge the distance between dissimilar samples, the formula is as follows:

wherein

Hash coding b_i、b_j∈R^c×1,c＝12,24,32,48；

S42: in pneumonia CT image x_nN1, 2, a deep level characteristic v (x) of N_n) And corresponding hash codes b_nSolving the contrast loss L_clThe formula is as follows:

wherein v (x)_n)＝W^TΦ(x_n(ii) a Theta) + V, theta representing all parameters of the deep hash network model, phi (x)_n；θ)∈R^262144×1Represents an input into the full connectivity layer FC1, W ∈ R^262144×cIs a weight matrix, V ∈ R^c×1Is a bias vector, | · | | non-conducting phosphor₂An L2 norm representing a vector;

s43: the overall loss function is defined as: l ═ L_S+αL_clWhere α ═ 0.1 is a weighting factor, so when optimizing the network model, the objective function should be set to minimize the loss function, which is formulated as follows:

wherein S_ijIs a matrix of the degree of similarity, and,

and is

b_jIs a pneumonia CT image x_iTransposed, pneumonia CT image x of hash coding of (a)_jHash coding of b_nRepresentative pneumonia CT image x_nN1, 2, hash coding of N, W ∈ R^262144×cIs a weight matrix, phi (x)_n；θ)∈R^262144×1Representing the input into the full connectivity layer FC1, V ∈ R^c×1Is a bias vector, | · | | non-conducting phosphor₂Representing the L2 norm of the vector.

Step S50, a multi-task Hash training strategy is introduced, a bilinear feature learning module is repeatedly used, and the extracted bilinear feature vector v' (x) of the pneumonia CT image is extracted_i)∈R^262144×1,v'(x_j)∈R^262144×1I, j ≠ N, 1,2Through the hash code learning modules of 4 branches respectively, and each branch includes 2 fully-connected hash layers and 1 fully-connected hash layer, the model can learn 12,24,32,48 bits of hash codes simultaneously, taking branch 1 as an example, at this time, c is 12, and the specific steps are as follows:

s51: firstly, a feature vector v' (x) of a pneumonia CT image is obtained_i)∈R^262144×1,v'(x_j)∈R^262144×1V "(x) is obtained through the full connection layers FC11 and FC12 of the branch 1_i)∈R^12×1、v”(x_j)∈R^12×1；

S52: then, the feature vector v ″ (x) of the pneumonia CT image is determined_i)、v”(x_j) Mapping to binary hash code b according to hash function h (·)_i、b_j∈R^12×1Finally, a training set T is formed_rIs the hash coding matrix B ∈ R^12×N，

Where h (-) represents a hash function and sign (-) represents a sign function.

Although the convolutional neural network CNN can extract deep features of an image, since the difference between the lung CT images is not too large, the accuracy of classification of the lung CT images is not very high. Therefore, the method effectively extracts the subtle differences among different symptoms of the lung CT image by using the bilinear convolutional neural network BCNN, fully exerts the extraction advantages of the bilinear convolutional neural network on the fine-grained image characteristics, and effectively improves the identification accuracy and the generalization robustness of the model; if the real-value features extracted by the bilinear convolutional neural network BCNN are directly classified, the problems of overlarge storage space and overlong training time can be caused. Therefore, Hash coding learning is introduced, and real-value features extracted by the bilinear convolutional neural network BCNN are mapped to a binary Hamming space, so that the storage space is greatly reduced and the training time is shortened when a model is trained; the invention can repeatedly use the bilinear feature learning module, so that the lung CT image can be simultaneously generated into Hash codes with different code lengths by the model, and the calculation time and the storage space are further saved.

The above-mentioned embodiments, objects, technical solutions and advantages of the present invention are further described in detail, it should be understood that the above-mentioned embodiments are only illustrative of the present invention, and are not intended to limit the scope of the present invention, and any person skilled in the art should understand that equivalent changes and modifications made without departing from the concept and principle of the present invention should fall within the protection scope of the present invention.

Claims (4)

1. A depth hash method for pneumonia CT image classification is characterized by comprising the following steps:

s10: establishing a pneumonia CT image data set;

s20: data preprocessing, namely firstly enhancing and expanding the data of the data set, and dividing the data set into training sets T according to the proportion of 80 percent and 20 percent_r＝(x₁,x₂,...,x_n) N1, 2, N and test set T_e＝(y₁,y₂,...,y_m) M is 1,2, 1, m, then the sizes of the pneumonia CT images are uniformly adjusted to 224 multiplied by 224, the number of channels is 1, 3 types of CT images are contained in the data set, namely, the lung CT image of a normal person, the lung CT image of a COVID-19 patient and the lung CT image of a common pneumonia patient, and finally a training set T is constructed_rOf the similarity matrix S, wherein

And S_ij∈R^N×N,i,j＝1,2,...,N；

S30: the method comprises the steps that a deep hash network model is built, wherein the model comprises a bilinear feature learning module and a hash coding learning module, when the model is trained, fine-grained features of a pneumonia CT image are extracted by using a Bilinear Convolutional Neural Network (BCNN), and then the extracted fine-grained features are input into the hash coding learning module, so that the fine-grained features of the corresponding pneumonia CT image are mapped into binary hash codes;

s40: calculating 2 losses, namely similarity loss L according to the hash codes obtained in the step S30_SAnd the contrast loss L_clAnd defining the total loss function as: l ═ L_S+αL_clWherein α ═ 0.1 is a weighting factor;

s50: introducing a multi-task Hash training strategy, repeatedly using a bilinear feature learning module, and extracting a bilinear feature vector v' (x) of the pneumonia CT image_i)∈R^262144×1,v'(x_j)∈R^262144×1I, j is 1, 2.. times, N, and i ≠ j is respectively learned through a hash code learning module comprising 4 branches, and each branch comprises 2 fully-connected layers and 1 fully-connected hash layer, so that the model can learn 12,24,32 and 48-bit hash codes at the same time;

s60: using alternate learning algorithm to target function

Performing optimization updating on the depth Hash network model parameter theta, the Hash coding matrix B, the weight matrix W and the offset vector V, and storing the model;

s70: first read test set T using a pre-trained model_ePneumonia CT image y_kK 1, 2.. times.m, which yields its hash code by_k∈R^c×1C is 12,24,32,48, then by_kAnd Hash coding matrix B belongs to R^c×NC is compared with each column of 12,24,32 and 48, the first 5 with smaller Hamming distance are compared, and the number of the categories belonging to the same is larger, then y is compared_kAnd classifying the test set into the categories, and finally calculating the average accuracy of the classification of the test set.

2. The deep hashing method for pneumonia CT image classification according to claim 1, wherein in step S30, the bilinear feature learning module mainly includes two branches A and B, and branch A and branch B are composed of two same VGG16 models, the convolutional layer conv of each branch is divided into 5 segments, 13 convolutional layers in total, the convolutional core size of each convolutional layer is 3 x 3, the step length stride and the padding are both set to 1, there is one maximum pooling layer maxpool after the first 4 segments of convolutional layers, and the pooling frame size is both 2 x 2, the step length stride is set to 2, and the concrete steps of the designed network structure are as follows:

s31: firstly, training set T of pneumonia CT image_rStochastic partitioning into image pairs (x)_i,x_j) I, j is 1, 2.. multidot.n, and i ≠ j, reads the image pair and the similarity matrix S, and then after passing through the 1 st convolution layer conv1, conv2 with the filter number of 64, the size of the extracted pneumonia CT image feature map is 224 × 224 × 64, and then passes through the maximum pooling layer maxpool1, and the final output size of the pneumonia CT image feature map is 112 × 112 × 64;

s32: after the output of the maxpool1 passes through the 2 nd convolution layers conv3 and conv4 with the filter number of 128, the size of the extracted pneumonia CT image feature map is 112 × 112 × 128, and then the pneumonia CT image feature map passes through the maximum pooling layer maxpool2, and the final output size of the pneumonia CT image feature map is 56 × 56 × 128;

s33: after passing the output of maxpool2 through the 3 rd convolution layers conv5, conv6 and conv7 with the filter number of 256, the size of the extracted pneumonia CT image feature map is 56 × 56 × 256, and then the extracted pneumonia CT image feature map passes through the maximum pooling layer maxpool3, and the final output size of the pneumonia CT image feature map is 28 × 28 × 256;

s34: after the output of maxpool3 passes through 4 th convolution layers conv8, conv9 and conv10 with the number of filter 512, the size of the extracted pneumonia CT image feature map is 28 × 28 × 512, and then the feature map passes through the maximum pooling layer maxpool4, and the final output size of the pneumonia CT image feature map is 14 × 14 × 512;

s35: after the output of maxpool4 passes through 5 th convolution layers conv11, conv12 and conv13 with the number of filter 512, the output size of the feature map of the extracted pneumonia CT image is 14 multiplied by 512;

in this case, the feature of the CT image of pneumonia extracted from the branch A is denoted by F_A(x_i)∈R^14×14×512、F_A(x_j)∈R^14×14×512The feature of the pneumonia CT image extracted by the branch B is F_B(x_i)∈R^14×14×512、F_B(x_j)∈R^14×14×512；

S36: the outputs of branch A and branch B were passed through bilinear pooling layer bilinearcholing to obtain a CT image (x) of pneumonia_i,x_j) I, j ≠ N, 1,2, and i ≠ j is a deep feature f at position l_A(l,x_i)∈R^1×512、f_B(l,x_i)∈R^1×512And f_A(l,x_j)∈R^1×512、f_B(l,x_j)∈R^1×512The following operations were carried out:

(1) calculating pneumonia CT image pair (x)_i,x_j) I, j ≠ N, is 1,2, and i ≠ j is a bilinear feature at position l

(2) Pooling pneumonia CT image pairs by summation pooling (x)_i,x_j) I, j ≠ N, i ≠ j, which is a bilinear feature at all positions to obtain a global bilinear feature

(3) Global bilinear feature matrix xi (x)_i) And xi (x)_j) Stretch into a vector to obtain v (x)_i)＝vec(ξ(x_i))∈R^262144×1、v(x_j)＝vec(ξ(x_j))∈R^262144×1Where vec (·) represents the operation of expanding the matrix into vectors;

(4) feature vector v (x) of pneumonia CT image_i) And v (x)_j) Performing normalization operation to obtain

Wherein | · | purple₂An L2 norm representing a vector;

s37: the feature vector v' (x) of the normalized pneumonia CT image is obtained_i)∈R^262144×1、v'(x_j)∈R^262144×1V "(x) was obtained after passing 2 full-junction layers FC1 and FC2 in this order_i)、v”(x_j)∈R^c×1C is 12,24,32,48, c represents the number of bits of the hash code;

s38: feature vector v ″ (x) of the pneumonia CT image after passing through the full connection layer_i)、v”(x_j) Mapping to binary hash b according to hash function h (·)_i,b_j∈R^c×1C is 12,24,32,48, finally obtaining the training set T_rIs the hash coding matrix B ∈ R^c×N，

Where h (-) represents a hash function and sign (-) represents a sign function.

3. The depth hashing method for pneumonia CT image classification according to claim 1, wherein said step S40 obtains Hash code B e R according to S38^c×NC-12, 24,32,48 calculates 2 losses, i.e. the similarity loss L_SAnd the contrast loss L_clThe method comprises the following specific steps:

s41: for the input pneumonia CT image pair (x)_i,x_j) I, j ≠ N, 1,2, and i ≠ j, with similarity penalty L_STo optimize the distance between similar samples and to enlarge the distance between dissimilar samples, the formula is as follows:

wherein

Hash coding b_i、b_j∈R^c×1,c＝12,24,32,48；

S42: in pneumonia CT image x_nN1, 2, a deep level characteristic v (x) of N_n) And corresponding hash codes b_nSolving the contrast loss L_clThe formula is as follows:

wherein v (x)_n)＝W^TΦ(x_n(ii) a Theta) + V, theta representing all parameters of the deep hash network model, phi (x)_n；θ)∈R^262144×1Represents an input into the full connectivity layer FC1, W ∈ R^262144×cIs a weight matrix, V ∈ R^c×1Is a bias vector, | · | | non-conducting phosphor₂An L2 norm representing a vector;

s43: the overall loss function is defined as: l ═ L_S+αL_clWhere α ═ 0.1 is the weighting factor, L_SFor loss of similarity, L_clTo compare the losses, therefore, when optimizing the network model, the objective function should be set to minimize the loss function, which is formulated as follows:

wherein S_ijIs a matrix of the degree of similarity, and,

and is

b_jIs a pneumonia CT image x_iHash ofTransposition of code, pneumonia CT image x_jHash coding of b_nRepresentative pneumonia CT image x_nN1, 2, hash coding of N, W ∈ R^262144×cIs a weight matrix, phi (x)_n；θ)∈R^262144×1Represents an input into the full connectivity layer FC1, V ∈ R^c×1Is a bias vector, | · | | non-conducting phosphor₂Representing the L2 norm of the vector.

4. The deep hashing method for classifying CT pneumonia images as claimed in claim 1, wherein said step S50 introduces a multitask Hash training strategy, and reuses a bilinear feature learning module to extract bilinear feature vector v '(x') of CT pneumonia images_i)∈R^262144×1,v'(x_j)∈R^262144×1I, j ≠ N, and i ≠ j passes through the hash code learning modules of 4 branches, and each branch includes 2 fully-connected hash layers and 1 fully-connected hash layer, so that the model can learn 12,24,32, and 48-bit hash codes at the same time, taking branch 1 as an example, and then c ═ 12, the specific steps are as follows:

s51: firstly, a feature vector v' (x) of a pneumonia CT image is obtained_i)∈R^262144×1,v'(x_j)∈R^262144×1V "(x) is obtained by the full junction layers FC11, FC12 of Branch 1_i)∈R^12×1、v”(x_j)∈R^12×1；

S52: then, the feature vector v ″ (x) of the pneumonia CT image is determined_i)、v”(x_j) Mapping to binary hash code b according to hash function h (·)_i、b_j∈R^12×1Finally, a training set T is formed_rIs the hash coding matrix B ∈ R^12×N，

Where h (-) represents a hash function and sign (-) represents a sign function.

Images