CN114724016A

CN114724016A - Image classification method, computer device, and storage medium

Info

Publication number: CN114724016A
Application number: CN202210453149.XA
Authority: CN
Inventors: 石峰; 周庆; 周翔
Original assignee: Shanghai United Imaging Intelligent Healthcare Co Ltd
Current assignee: Shanghai United Imaging Intelligent Healthcare Co Ltd
Priority date: 2022-04-24
Filing date: 2022-04-24
Publication date: 2022-07-08

Abstract

The present application relates to an image classification method, a computer device, and a storage medium. The method comprises the following steps: acquiring images of different resolutions of a region of interest of a medical image; inputting each image into a preset classification network to obtain a classification result of the region of interest; and the classification result is used for characterizing the category to which the region of interest belongs. By adopting the method, the trained model is suitable for medical images of different focus types.

Description

Image classification method, computer device, and storage medium

Technical Field

The present application relates to the field of medical imaging technology, and in particular, to an image classification method, a computer device, and a storage medium.

Background

With the development of artificial intelligence technology, more and more neural network models are applied to the auxiliary diagnosis of medical images, for example, classification networks may be used to classify medical images and classification results may be used to assist doctors in diagnosing lesions, or segmentation networks may be used to segment medical images and segmentation results may be used to assist doctors in analyzing lesion regions.

However, the current neural network model has certain limitations in application, for example, the trained model has medical images which are difficult to be applied to different lesion types.

Disclosure of Invention

In view of the above, it is necessary to provide an image classification method, a computer device and a storage medium, which can adapt the trained model to medical images of different lesion types.

In a first aspect, the present application provides an image classification method, including:

acquiring images of different resolutions of a region of interest of a medical image;

inputting each image into a preset classification network to obtain a classification result of the region of interest; and the classification result is used for characterizing the category to which the region of interest belongs.

In one embodiment, the inputting each of the images into a preset classification network to obtain a classification result of the region of interest includes:

extracting the features of the images to obtain the features of the images;

performing feature fusion on the features of the images to obtain first fusion features;

and determining the classification result according to the first fusion characteristic.

In one embodiment, the performing feature fusion on the features of each image to obtain a first fused feature includes:

performing correlation calculation on the characteristics of the images to obtain a correlation matrix among the images;

normalizing the correlation matrix among the images to obtain a normalized weight matrix;

and obtaining the first fusion characteristic according to the normalized weight matrix and the characteristic of any image.

In one embodiment, the classification results include a positive-negative classification result and a type of positive category; determining the classification result according to the first fusion feature includes:

determining a first classification result of the region of interest according to the first fusion characteristic; the first classification result is used for representing the region of interest as a positive category or a negative category;

when the first classification result is a positive classification, determining a second classification result of the region of interest according to the first fusion feature; and the second classification result is used for characterizing the type of the region of interest, wherein the type of the region of interest comprises negative types and the type of a positive type of the region of interest when the region of interest is a positive type.

In one embodiment, the method further comprises:

acquiring clinical information of the medical image;

inputting the clinical information into a preset first neural network, and acquiring clinical characteristics corresponding to the clinical information;

fusing the first fusion characteristic and the clinical characteristic by adopting a preset fusion method to obtain a second fusion characteristic;

and determining the classification result according to the second fusion characteristic.

In one embodiment, the method further comprises:

acquiring image symptom information of the medical image;

inputting the image symptom information into a preset second neural network, and acquiring image symptom characteristics corresponding to the image symptom information;

performing fusion processing on the first fusion characteristic and the image symptom characteristic by adopting a preset fusion method to obtain a third fusion characteristic;

and determining the classification result according to the third fusion characteristic.

In one embodiment, the predetermined fusion method includes a bilinear pooling method or a tandem method.

In one embodiment, the training process of the classification network includes:

acquiring sample images of different resolutions of a sample region of interest and a first label and a second label of the sample region of interest; the first label is used for characterizing the sample region of interest as a positive category or a negative category; the second label is used for characterizing the type of the sample region of interest, wherein the type of the sample region of interest comprises negative types and the type of a positive type of the sample region of interest when the sample region of interest is a positive type;

inputting each sample image into a preset initial classification network to obtain a first sample classification result and a second sample classification result of the sample interesting region; the first sample classification result is used for representing that the sample interesting region is a positive type or a negative type; the second sample classification result is used for characterizing the type of the sample interesting region, wherein the type of the sample interesting region comprises negative types and a positive type when the sample interesting region is a positive type;

weighting the second label by using the first label to obtain a weighted label corresponding to the second label, and weighting the second sample classification result by using the first sample classification result to obtain a weighted classification result corresponding to the second sample classification result; wherein the negative categories in the weighted labels are consistent with the negative categories in the first labels, and the negative categories in the weighted classification results are consistent with the negative categories in the first sample classification results;

obtaining a first loss function according to the first label and the first sample classification result, and obtaining a second loss function according to the weighting label and the weighting classification result;

and training the initial classification network according to the first loss function and the second loss function to obtain the classification network.

In a second aspect, the present application further provides an image classification apparatus, comprising:

the first acquisition module is used for acquiring images with different resolutions of an interested area of the medical image;

the classification module is used for inputting each image into a preset classification network to obtain a classification result of the region of interest; and the classification result is used for characterizing the category to which the region of interest belongs.

In a third aspect, the present application further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the following steps when executing the computer program:

In a fourth aspect, the present application further provides a computer readable storage medium having a computer program stored thereon, the computer program when executed by a processor implementing the steps of:

inputting each image into a preset classification network to obtain a classification result of the region of interest; and the classification result is used for representing the category to which the region of interest belongs.

In a fifth aspect, the present application further provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of:

According to the image classification method, the computer equipment and the storage medium, the images of the interesting region of the medical image with different resolutions are acquired, the images of the interesting region with different resolutions are input into the preset classification network, the images of the interesting region with different resolutions can be processed through the classification network, so that the classification network can be suitable for the medical images with different focus types, the images of the interesting region with different resolutions can be processed through the classification network, and the classification result representing the category to which the interesting region belongs is obtained.

Drawings

FIG. 1 is a diagram of an exemplary embodiment of an image classification method;

FIG. 2 is a flow diagram illustrating a method for image classification in one embodiment;

FIG. 3 is a flowchart illustrating an image classification method according to another embodiment;

FIG. 4 is a diagram illustrating a fusion process of image features according to an embodiment;

FIG. 5 is a flowchart illustrating an image classification method according to another embodiment;

FIG. 6 is a diagram illustrating a fusion process of image features according to an embodiment;

FIG. 7 is a flowchart illustrating an image classification method according to another embodiment;

FIG. 8 is a schematic diagram of a fusion process of image features in one embodiment;

FIG. 9 is a diagram illustrating a training process for a classification network in one embodiment;

FIG. 10 is a flowchart illustrating an image classification method according to an embodiment;

fig. 11 is a block diagram showing the configuration of an image classification device according to an embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

The image classification method provided by the embodiment of the application can be applied to computer equipment shown in fig. 1. The computer device comprises a processor and a memory connected by a system bus, wherein a computer program is stored in the memory, and the steps of the method embodiments described below can be executed when the processor executes the computer program. Optionally, the computer device may further comprise a network interface, a display screen and an input device. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a nonvolatile storage medium storing an operating system and a computer program, and an internal memory. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. Optionally, the computer device may be a server, a personal computer, a personal digital assistant, other terminal devices such as a tablet computer, a mobile phone, and the like, or a cloud or a remote server, and the specific form of the computer device is not limited in the embodiment of the present application.

Currently, it takes a long time and a great deal of effort for a radiologist to visit a medical image consisting of hundreds of slices. In recent years, with the development of artificial intelligence technology in image recognition, the connection between artificial intelligence and medicine is becoming more and more tight, and more technologies are applied to the recognition of medical images. The application provides an image classification method comprising the following steps: (1) aiming at the problem that an input image with a fixed size is difficult to be suitable for all focus targets in the model training process, a multi-scale feature combined attention module is provided, namely a bilinear fusion mode is sampled, multi-scale feature channels are interacted, the commonality among the multi-scale features is enhanced, and meanwhile, the aim of being suitable for focus feature learning of various ranges of sizes can be achieved; (2) in the task of medical image identification, the identification targets of some tasks have a certain degree of coincidence, for example, on the basis of benign and malignant tumors, the malignant tumors are further identified as atypical adenomatous hyperplasia, in-situ carcinoma, micro-infiltration and infiltration, the infiltration suggests an operation, and the atypical adenomatous hyperplasia suggests follow-up observation, which has more important clinical significance for doctors, and for the hierarchical multi-task requirement, a consistency loss function (consistency loss) is introduced for guiding a multi-task classifier module to make a consistent decision; (3) due to the requirement of deep learning on data quantity, a good and effective disease auxiliary diagnosis model usually needs a large amount of labeled data with higher quality to train, however, the data with the golden standard are unbalanced in large probability in clinic, and aiming at the problem of data unbalance, an AUC loss function is further added on the basis of a basic classification loss function, so that the network bias is inhibited; (4) in clinical application, doctors need to see medical images and analyze some clinical indexes such as blood examination indexes, gene detection indexes, medical history and the like at the same time, and the comprehensive images of the indexes have certain clinical value in clinical judgment; (5) for image features frequently adopted by some doctors, such as lesion density, coding can be carried out, a classification module is added, and a model is assisted to make judgment.

In one embodiment, as shown in fig. 2, there is provided an image classification method, which is described by taking the method as an example applied to the computer device in fig. 1, and includes the following steps:

s201, images of different resolutions of the region of interest of the medical image are acquired.

Optionally, the computer device may first segment the region of interest of the medical image from the medical image, and then perform different processing on the resolution of the region of interest of the medical image, to obtain images with different resolutions of the region of interest of the medical image, or the computer device may first perform different processing on the resolution of the region of interest of the medical image, and then segment the processed region of interest with different resolutions, to obtain images with different resolutions of the region of interest of the medical image.

Alternatively, the medical image may be a Computed Tomography (CT) image, a Magnetic Resonance Imaging (MRI) image, a Positron Emission Tomography (PET) image, an X-ray image, or the like. Optionally, the region of interest of the medical image may be a lung region in the medical image, a heart and liver region in the medical image, a lymph region in the medical image, or the like.

It can be understood that the resolution of the medical image refers to the amount of information stored in the medical image, which is how many pixels are in each inch of the medical image, and in this embodiment, the images of the region of interest with different resolutions refer to the images of the region of interest with different amounts of information stored therein.

S202, inputting each image into a preset classification network to obtain a classification result of the region of interest; and the classification result is used for representing the category of the region of interest.

The preset classification Network includes, but is not limited to, a Convolutional Neural Network (CNN), a ResNet Network, a density Net Network, and other Neural networks. Optionally, the category to which the region of interest belongs may be a negative-positive classification result, and further, when the category to which the region of interest belongs is positive, the category to which the region of interest belongs may also be a type in which the region of interest is a positive category. Optionally, the computer device may sequentially input the images of the region of interest with different resolutions into the classification network to obtain a classification result of the region of interest, or may input the fused images into the classification network after fusing the images of the region of interest with different resolutions to obtain a classification result of the region of interest. For example, the computer device may sequentially input the three images of the region of interest with different resolutions into the classification network to obtain a classification result of the region of interest, or may input the fused image into the classification network after fusing the three images of the region of interest with different resolutions to obtain a classification result of the region of interest.

In the image classification method, the images of the interesting region of the medical image with different resolutions are acquired, the images of the interesting region with different resolutions are input into the preset classification network, the images of the interesting region with different resolutions can be processed through the classification network, so that the classification network can be suitable for the medical images with different focus types, the images of the interesting region with different resolutions can be processed through the classification network, and the classification result representing the category to which the interesting region belongs is obtained.

In the above scenario in which the images of the region of interest with different resolutions are input into the preset classification network to obtain the classification result of the region of interest, in an embodiment, as shown in fig. 3, the above S202 includes:

and S301, extracting the features of the images to obtain the features of the images.

Optionally, the classification network may include a feature extraction layer, and after the images of the region of interest with different resolutions are input into the classification network, feature extraction may be performed on the images of the region of interest with different resolutions through the feature extraction layer of the classification network, so as to obtain features of the images of the region of interest with different resolutions. Optionally, the classification network may include at least one feature extraction layer, for example, the number of the feature extraction layers included in the classification network may be the same as the number of the images of the region of interest with different resolutions, and the feature extraction is performed on the images of the region of interest with different resolutions through each feature extraction layer, so as to obtain the features of each image; or, the classification network may only include one feature extraction layer, and the feature extraction layer sequentially performs feature extraction on the images of the region of interest with different resolutions to obtain features of the images.

S302, feature fusion is carried out on the features of the images to obtain first fusion features.

Optionally, the classification network may further include a feature fusion layer, and after the features of the images of the region of interest with different resolutions are obtained through the classification network, the features of the images of the region of interest with different resolutions may be feature-fused through the feature fusion layer of the classification network, so as to obtain a first fusion feature. Optionally, features of the images of the region of interest with different resolutions may be concatenated to obtain the first fusion feature. Alternatively, as an alternative embodiment, the first fusion feature may be obtained by the following steps, and a schematic diagram of the process is shown in fig. 4:

and step A, performing correlation calculation on the characteristics of the images to obtain a correlation matrix among the images.

Optionally, the computer device may perform correlation calculation on the features of the images of the region of interest with different resolutions through bilinear operation, so as to obtain a correlation matrix between the images of the region of interest with different resolutions. Taking three images with different resolutions of the region of interest as an example, extracting the features of the three images with different resolutions to obtain a feature F₁、F₂、F₃Wherein F is₁,F₂,F₃∈R^c×d×h×wC represents the number of characteristic channels, d, h and w represent the length, width and height of the characteristic, and the characteristic F of the images with three different resolutions₁、F₂、F₃Unfolding to obtain F₁',F₂',F₃'∈R^c×lD × h × w, and obtaining a correlation matrix in dimension between features by bilinear operation

Wherein, the superscript of the correlation matrix represents the feature serial numbers of different resolutions.

And step B, carrying out normalization processing on the correlation matrix among the images to obtain a normalized weight matrix.

Continuing to take the three images with different resolutions in the region of interest as an example, the obtained correlation matrix between the images is M¹²，M¹³To the correlation matrix M¹²，M¹³Normalization is carried outProcessing to obtain normalized weight matrix of

Where the index i represents the ith channel of the first feature and the index j represents the jth channel of the second feature.

And step C, obtaining a first fusion characteristic according to the normalized weight matrix and the characteristic of any image.

Optionally, the computer device may apply the normalized weight matrix to a feature of the image of interest of any resolution to obtain a first fusion feature. Continuing to take the three images with different resolutions in the region of interest as an example, the obtained normalized weight matrix is

And

optionally, the computer device may apply the normalized weight matrix to the first feature F₁The first fusion feature is obtained, that is, the computer device may obtain the first fusion feature by the following formula:

in the formula, F_W1Representing the first fused feature.

And S303, determining a classification result according to the first fusion characteristic.

Optionally, the computer device may classify the first fusion feature through a classifier, and determine a classification result of the region of interest. Or, optionally, the classification result may include a positive classification result and a negative classification result, and in one embodiment, the classification result may be determined by:

step D, determining a first classification result of the region of interest according to the first fusion characteristic; the first classification result is used for characterizing the region of interest as a positive or negative class.

Optionally, the computer device may input the first fused feature into a classifier and determine a first classification result characterizing the region of interest as a positive or negative class. Optionally, the first classification result of the region of interest may be a positive classification or a negative classification. Illustratively, taking the region of interest as a lung nodule as an example, the obtained first classification result may be that the lung nodule is positive or that the lung nodule is negative.

Step E, when the first classification result is a positive classification, determining a second classification result of the region of interest according to the first fusion characteristic; and the second classification result is used for characterizing the type of the region of interest, wherein the type of the region of interest comprises negative types and the type of a positive type which the region of interest belongs to when the region of interest is the positive type.

Optionally, when the first classification result is a positive classification, the computer device may further determine a second classification result of the region of interest according to the first fusion feature, where the second classification result of the region of interest is used to characterize a type to which the region of interest belongs, and the type to which the region of interest belongs may include types of negative and positive classifications to which the region of interest belongs when the region of interest is a positive classification. Continuing with the example where the region of interest is a lung nodule, if the first classification result is a positive classification, the computer device may further determine a second classification result of the lung nodule according to the first fusion feature, for example, the determined second classification result may be any one of atypical adenomatous hyperplasia, carcinoma in situ, micro-infiltration, and infiltration.

In this embodiment, images of different resolutions of the region of interest are input into a preset classification network, and feature extraction can be performed on the images of different resolutions of the region of interest through the classification network to obtain features of each image, so that feature fusion can be performed on the features of each image to obtain a first fusion feature, the region of interest is classified according to the first fusion feature, and a classification result of the region of interest is determined, so that the classification network can be applied to medical images of different lesion types, and a classification result of the region of interest can be obtained according to the images of different resolutions of the region of interest.

In some scenes, the region of interest can be classified by combining clinical information of the medical image to obtain a classification result of the region of interest. In one embodiment, as shown in fig. 5, the method further includes:

s401, obtaining clinical information of the medical image.

Alternatively, the clinical information of the medical image may include blood test information, genetic test information, and the like of the medical image. Optionally, the computer device may obtain clinical information corresponding to the medical image from a database storing the medical image, or the computer device may obtain clinical information of the medical image by recognizing the identification information on the medical image.

S402, inputting the clinical information into a preset first neural network, and acquiring clinical characteristics corresponding to the clinical information.

Optionally, the preset first neural network may be a multi-layer perceptron (MLP), a Long-Short Term Memory network (LSTM), or a Fully connected layer (full connected layer) in the neural network. Optionally, the computer device inputs the clinical information into a preset first neural network, and the clinical information may be encoded by the first neural network to obtain the clinical characteristics.

And S403, performing fusion processing on the first fusion characteristic and the clinical characteristic by adopting a preset fusion method to obtain a second fusion characteristic.

Optionally, the preset fusion method may include a bilinear pooling method or a tandem method, that is, the computer device may perform fusion processing on the first fusion feature and the clinical feature by using the bilinear pooling method or the tandem method to obtain the second fusion feature. Exemplarily, with F_i∈R^M×1(M is the image feature channel) is the first fusion feature, F_c∈R^S×1(S is the clinical feature channel) for example, the process of obtaining the second fusion feature may be the process illustrated in fig. 6, and as shown in fig. 6, a Bilinear pooling module (Bilinear pooling) may be used for the image feature F_iAnd clinical characteristics F_cThe fusion is carried out, and the specific operation process can be represented by the following formula:

in the formula, F_iRepresenting image features, F_cShowing clinical characteristics, F_fRepresenting the second fused feature.

S404, determining a classification result according to the second fusion characteristic.

Optionally, the computer device may classify the second fusion feature through a classifier, and determine a classification result of the region of interest. Alternatively, as an optional implementation manner, the computer device may determine the classification result of the region of interest through the descriptions in step D and step E in S303, which have similar implementation principles and technical effects, and are not described herein again.

In this embodiment, by acquiring the clinical information of the medical image, the clinical information of the medical image can be input into the preset first neural network, and the clinical features corresponding to the clinical information are acquired, so that the first fusion features and the clinical features can be subjected to fusion processing by using the preset fusion method, and the second fusion features are acquired.

In some scenes, the region of interest can be classified by combining the image symptom information of the medical image to obtain the classification result of the region of interest. In one embodiment, as shown in fig. 7, the method further includes:

s501, image symptom information of the medical image is obtained.

The image symptom information of the medical image may be density symptoms of a region of interest in the medical image, a size of the region of interest, and the like, and taking the region of interest as a lung nodule as an example, the density symptoms of the lung nodule may include solidity, ground glass, mixed ground glass, calcification, and the like. Optionally, the computer device may input the medical image into a Density network (Density Net) to obtain image symptom information of the medical image.

S502, inputting the image symptom information into a preset second neural network, and acquiring image symptom characteristics corresponding to the image symptom information.

Optionally, the preset second neural network may be a multilayer perceptron (MLP), a Long-Short Term Memory network (LSTM), or a Fully connected layer (full connected layer) in the neural network. Optionally, the computer device inputs the image symptom information of the medical image into a preset second neural network, and the image symptom information may be encoded by the second neural network to obtain an image symptom feature corresponding to the image symptom information.

And S503, performing fusion processing on the first fusion characteristic and the image characteristic by adopting a preset fusion method to obtain a third fusion characteristic.

Optionally, the preset fusion method may include a bilinear pooling method or a serial method, that is, the computer device may perform fusion processing on the first fusion feature and the image feature by using the bilinear pooling method or the serial method to obtain a third fusion feature. Optionally, the computer device may encode the image feature, encode the dimension of the image feature into the same dimension as the number of images with different resolutions of the region of interest, and perform fusion processing on the image feature and the second fusion feature to obtain a third fusion feature. Exemplarily, with F_dCharacteristic of the image, F_i∈R^M×1(M is an image feature channel) as an example of the first fusion feature, the process of obtaining the third fusion feature may be the process illustrated in fig. 8, and as shown in fig. 8, the image feature F may be obtained_dAnd a first fusion characteristic F_i∈R^M×1And performing tandem connection to obtain a third fusion characteristic.

And S504, determining a classification result according to the third fusion characteristic.

Optionally, the computer device may classify the third fusion feature through a classifier, and determine a classification result of the region of interest. Or, as an optional implementation manner, the computer device may determine the classification result of the region of interest through the descriptions in step D and step E in S303, and the implementation principle and the technical effect are similar, which are not described herein again.

In this embodiment, the image symptom information of the medical image is acquired, the image symptom information of the medical image can be input into the preset second neural network, and the image symptom features corresponding to the image symptom information are acquired, so that the first fusion features and the image symptom features can be fused by using the preset fusion method, and the third fusion features are acquired.

In the scene where the images of the region of interest with different resolutions are input into the preset classification network to obtain the classification result of the region of interest, the classification network is a pre-trained network, and in one embodiment, as shown in fig. 9, the training process of the classification network includes:

s601, obtaining sample images of different resolutions of a sample interesting region and a first label and a second label of the sample interesting region; the first label is used for representing the area of interest of the sample as a positive category or a negative category; the second label is used for characterizing the type of the sample interesting region, wherein the type of the sample interesting region comprises negative types and the type of a positive type of the sample interesting region when the sample interesting region is a positive type.

Optionally, the computer device may first segment the sample region of interest of the sample medical image from the sample medical image, and then perform different processing on the resolution of the sample region of interest of the sample medical image, to obtain sample images with different resolutions of the sample region of interest of the sample medical image, or the computer device may first perform different processing on the resolution of the region of interest of the sample medical image, and then segment the processed sample regions of interest with different resolutions, to obtain images with different resolutions of the sample region of interest of the sample medical image. Optionally, the computer device may obtain, from a database storing the sample medical images, a first tag and a second tag of a sample region of interest of the sample medical images, where the first tag of the sample region of interest is used to characterize the sample region of interest as a positive category or a negative category, the second tag of the sample region of interest is used to characterize a type to which the sample region of interest belongs, and the type to which the sample region of interest belongs includes types of negative categories and positive categories to which the sample region of interest belongs when the sample region of interest is a positive category; illustratively, taking the sample region of interest as a lung nodule, the first label of the lung nodule may be negative, or the first label of the lung nodule may also be positive; further, if the first label of the lung nodule is negative, the second label of the lung nodule is also negative; if the first signature of a lung nodule is positive, the second signature of the lung nodule may be of any type of atypical adenomatous hyperplasia, carcinoma in situ, micro-infiltration, infiltration.

S602, inputting each sample image into a preset initial classification network to obtain a first classification result and a second classification result of the sample interesting region; the first classification result is used for representing the interested area of the sample as a positive category or a negative category; and the second classification result is used for characterizing the type of the sample interesting region, wherein the type of the sample interesting region comprises negative types and the type of a positive type of the sample interesting region when the sample interesting region is a positive type.

The preset initial classification Network includes but is not limited to a Convolutional Neural Network (CNN), a ResNet Network, a sense Net Network, and other Neural networks. Optionally, the computer device may sequentially input the sample images of the sample interesting regions with different resolutions into the initial classification network to obtain a first sample classification result and a second sample classification result of the sample interesting regions, or may input the fused sample images into the initial classification network after fusing the sample images of the sample interesting regions with different resolutions to obtain the first sample classification result and the second sample classification result of the sample interesting regions. For example, the computer device may sequentially input three sample images of different resolutions of the sample region of interest into the initial classification network to obtain a first sample classification result and a second sample classification result of the sample region of interest, or may input a fused sample image into the initial classification network after fusing the three sample images of different resolutions of the sample region of interest to obtain a first sample classification result and a second sample classification result of the sample region of interest. The first sample classification result of the sample interesting region is used for representing that the sample interesting region is in a positive category or a negative category, the second sample classification result of the sample interesting region is used for representing the type of the sample interesting region, and the type of the sample interesting region comprises negative types and the type of the positive category when the sample interesting region is in the positive category. In this embodiment, if the first sample classification result is negative, the second sample classification result is also negative, and if the first sample classification result is positive, the second sample classification result is also positive, and the second sample classification result is further a type of a positive category.

S603, the first label is used for conducting weighting processing on the second label to obtain a weighted label corresponding to the second label, and the first sample classification result is used for conducting weighting processing on the second sample classification result to obtain a weighted classification result corresponding to the second sample classification result; and the negative categories in the weighted classification result are consistent with the negative categories in the first sample classification result.

It can be understood that, in order to ensure that, for the same region of interest, when the positive and negative classification is performed and the classification is performed on the type of the same region of interest, the result is consistent, that is, when the first sample classification result is negative, the second sample classification result is negative, and when the first sample classification result is positive, the second sample classification result is also the type in positive, the computer device may perform weighting processing on the second label and the second sample classification result,to ensure consistency of the two classification tasks. Illustratively, taking the sample region of interest as a lung nodule as an example, the types of positive categories to which the lung nodule belongs when the lung nodule is a positive category may include atypical adenomatous hyperplasia, carcinoma in situ, micro-infiltration, assuming that X represents the input sample set and Y represents the input sample set₁E {0,1} represents the first label, Y₂E {0,1,2,3,4} represents the second label, wherein 0 in the first label represents negative lung nodule, 1 represents positive lung nodule, and the labels 1,2,3,4 in the second label are all 1 in the first label, and definition C₁As a result of the first sample classification, C₂Classifying the result for a second sample, wherein C₂Is a five-dimensional array, the value of each dimensional array corresponds to the prediction probability of each category, and the sum of the prediction probabilities of all the categories is 1, so that the optimization target in the positive and negative classification is C₁(x)→Y₁When classifying the type of the sample interesting region, the second label Y is₂Performing weighting processing to convert the result into a weighted label, performing weighting processing to convert the result into a weighted classification result, optionally performing One-Hot coding on the second label by using the first label to obtain a weighted label corresponding to the second label, performing One-Hot coding on the result by using the first sample classification result to obtain a weighted classification result corresponding to the second sample classification result, optionally performing One-Hot coding on the classification result to obtain a coding process of the probability highest position 1 and other positions 0, for example, the second label Y can be used₂Performing One-Hot coding conversion to obtain label Y₂', wherein, Y₂'∈{[1,0,0,0,0],[0,1,0,0,0],[0,0,1,0,0],[0,0,0,1,0],[0,0,0,0,1]Further, a weighted label y ═ y' × [1-C ] may be defined₁(x),C₁(x),C₁(x),C₁(x),C₁(x)]In the formula, Y' is belonged to Y₂' the optimization objective in classifying the type to which the sample region of interest belongs is C₂(x)→Y₂”，y”∈Y₂", likewise, the computer device may classify the second sample classification result C in the same manner₂Is subjected to One-Hot coding conversion to C'₂Wherein, C'₂∈{[1,0,0,0,0],[0,1,0,0,0],[0,0,1,0,0],[0,0,0,1,0],[0,0,0,0,1]}, further, a weighted classification result C 'may be defined'₂'＝c'₂×C₂In the formula (II), c'₂∈C'₂。

S604, obtaining a first loss function according to the first label and the first sample classification result, and obtaining a second loss function according to the weighted label and the weighted classification result.

In this embodiment, please continue to refer to the above example, since the optimization goal is C in the negative-positive classification₁(x)→Y₁The optimization objective in classifying the types of the sample interesting regions is C₂(x)→Y₂", from which a first loss function can be derived

Wherein crosssensortorpyloss represents a first loss function, N is the number of samples, x represents the image of the samples, y represents a first label, C₁Representing the first sample classification result, and likewise, a second loss function can be obtained

Wherein softcrosssensorpyloss represents a second loss function, y' represents a weighting tag, C₂Representing a second sample classification result.

S605, training the initial classification network according to the first loss function and the second loss function to obtain the classification network.

Optionally, the computer device may sum the value of the first loss function and the value of the second loss function, and train the initial classification network by using the sum of the first loss function and the second loss function, so as to obtain the classification network. Further, in order to avoid the problem of data imbalance in the training process of the initial classification network, an AUC loss function can be added during the training of the initial classification network, and when the positive and negative categories of the sample region of interest are classified, it is assumed that X is^-Represents the input negative sample set, X⁺Represents the input positive sample set, and thus can be defined according to this assumption

Wherein N is⁺And N^-The number of positive and negative sample sets respectively,

when classifying the type to which the sample region of interest belongs, the AUC loss function can be split into a plurality of functions, and finally the plurality of AUC loss functions are summed up to be used as the AUC loss function when classifying the type to which the sample region of interest belongs, i.e. multi AUCloss (C)₂,X)＝AUCloss(C₂,X⁰,～X⁰)+AUCloss(C₂,X¹,～X¹)+AUCloss(C₂,X²,～X²) In the formula (I), the compound is shown in the specification,

X^krepresents the kth class sample set, k ∈ {0,1,2},. X^kRepresenting a non-k class sample set, N being the total number of samples,

and

the value of the kth channel, that is, the probability predicted as the kth class is expressed, and the computer device may further add the two AUC loss functions when training the initial classification network according to the first loss function and the second loss function, or alternatively, the computer device may train the initial classification network according to the sum of the first loss function, the second loss function, and the two AUC loss functions, so as to obtain the classification network.

In this embodiment, by inputting sample images of different resolutions of a sample region of interest into a preset initial classification network, a first sample classification result and a second sample classification result of the sample region of interest can be obtained, and performing weighting processing on a second label by using the first label, a weighting label corresponding to the second label can be obtained, and performing weighting processing on the second sample classification result by using the first sample classification result, a weighting classification result corresponding to the second sample classification result can be obtained, so that classification results of negative and positive in two classifications can be kept consistent, thereby obtaining a first loss function according to the first label and the first sample classification result, obtaining a second loss function according to the weighting label and the weighting classification result, and further training the initial classification network according to the first loss function and the second loss function, the classification network is accurately obtained, and the accuracy of the obtained classification network is improved.

To facilitate understanding of those skilled in the art, the image classification method provided in the present application is described in detail below, and with reference to fig. 10, the method may include:

s1, acquiring sample images of different resolutions of a sample interesting region and a first label and a second label of the sample interesting region; the first label is used for representing the area of interest of the sample as a positive category or a negative category; the second label is used for characterizing the type of the sample interesting region, wherein the type of the sample interesting region comprises negative types and the type of a positive type of the sample interesting region when the sample interesting region is a positive type.

S2, inputting each sample image into a preset initial classification network to obtain a first sample classification result and a second sample classification result of the sample interested region; the first sample classification result is used for representing the interested area of the sample as a positive type or a negative type; the second sample classification result is used for characterizing the type of the sample interesting region, wherein the type of the sample interesting region comprises negative types and the type of a positive type of the sample interesting region when the sample interesting region is a positive type.

S3, weighting the second label by the first label to obtain a weighted label corresponding to the second label, and weighting the second sample classification result by the first sample classification result to obtain a weighted classification result corresponding to the second sample classification result; and the negative categories in the weighted classification result are consistent with the negative categories in the first sample classification result.

And S4, obtaining a first loss function according to the first label and the first sample classification result, and obtaining a second loss function according to the weighted label and the weighted classification result.

And S5, training the initial classification network according to the first loss function and the second loss function to obtain the classification network.

And S6, acquiring images of different resolutions of the region of interest of the medical image.

And S7, inputting each image into a preset classification network, and extracting the features of each image to obtain the features of each image.

And S8, performing correlation calculation on the characteristics of the images to obtain a correlation matrix among the images.

And S9, normalizing the correlation matrix among the images to obtain a normalized weight matrix.

S10, obtaining a first fusion feature according to the normalized weight matrix and the feature of any image, and executing S11-S12 or executing S13-S14.

S11, acquiring clinical information of the medical image; inputting clinical information into a preset first neural network, and acquiring clinical characteristics corresponding to the clinical information; and performing fusion processing on the first fusion characteristic and the clinical characteristic by adopting a preset fusion method to obtain a second fusion characteristic.

S12, determining a first classification result of the region of interest according to the second fusion characteristic; the first classification result is used for representing the region of interest as a positive type or a negative type; when the first classification result is a positive classification, determining a second classification result of the region of interest according to the second fusion characteristic; and the second classification result is used for characterizing the type of the region of interest, wherein the type of the region of interest comprises negative types and the type of a positive type of the region of interest when the region of interest is a positive type.

S13, acquiring image symptom information of the medical image; inputting the image symptom information into a preset second neural network, and acquiring image symptom characteristics corresponding to the image symptom information; and performing fusion processing on the first fusion characteristic and the image symptom characteristic by adopting a preset fusion method to obtain a third fusion characteristic.

S14, determining a first classification result of the region of interest according to the third fusion characteristic; the first classification result is used for representing the region of interest as a positive type or a negative type; when the first classification result is a positive classification, determining a second classification result of the region of interest according to the third fusion characteristic; and the second classification result is used for characterizing the type of the region of interest, wherein the type of the region of interest comprises negative types and the type of a positive type of the region of interest when the region of interest is a positive type.

It should be noted that, for the descriptions in S1-S14, reference may be made to the descriptions related to the above embodiments, and the effects are similar, and the description of this embodiment is not repeated herein.

It should be understood that, although the steps in the flowcharts related to the embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the above embodiments may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the application also provides an image classification device for realizing the image classification method. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so specific limitations in one or more embodiments of the image classification apparatus provided below can be referred to as limitations on the image classification method in the foregoing, and details are not described herein again.

In one embodiment, as shown in fig. 11, there is provided an image classification apparatus including: a first obtaining module and a classifying module, wherein:

the first acquisition module is used for acquiring images with different resolutions of an interested area of the medical image.

The classification module is used for inputting each image into a preset classification network to obtain a classification result of the region of interest; and the classification result is used for representing the category of the region of interest.

The image classification apparatus provided in this embodiment may implement the method embodiments described above, and the implementation principle and the technical effect are similar, which are not described herein again.

On the basis of the foregoing embodiment, optionally, the classification module includes: an extraction unit, a fusion unit and a determination unit, wherein:

and the extraction unit is used for extracting the features of the images to obtain the features of the images.

And the fusion unit is used for performing feature fusion on the features of the images to obtain a first fusion feature.

And the determining unit is used for determining a classification result according to the first fusion characteristic.

On the basis of the above embodiment, optionally, the fusion unit is configured to perform correlation calculation on the features of the images to obtain a correlation matrix between the images; normalizing the correlation matrix among the images to obtain a normalized weight matrix; and obtaining a first fusion characteristic according to the normalized weight matrix and the characteristic of any image.

On the basis of the above embodiment, optionally, the classification result includes a positive-negative classification result and a type of a positive category; the determining unit is configured to determine a first classification result of the region of interest according to the first fusion feature; the first classification result is used for representing the region of interest as a positive classification or a negative classification; when the first classification result is a positive classification, determining a second classification result of the region of interest according to the first fusion characteristic; and the second classification result is used for characterizing the type of the region of interest, wherein the type of the region of interest comprises negative types and the type of a positive type of the region of interest when the region of interest is a positive type.

On the basis of the foregoing embodiment, optionally, the apparatus further includes: the device comprises a second acquisition module, a third acquisition module, a first fusion module and a first determination module, wherein:

and the second acquisition module is used for acquiring clinical information of the medical image.

And the third acquisition module is used for inputting the clinical information into a preset first neural network and acquiring clinical characteristics corresponding to the clinical information.

And the first fusion module is used for performing fusion processing on the first fusion characteristic and the clinical characteristic by adopting a preset fusion method to obtain a second fusion characteristic.

And the first determining module is used for determining a classification result according to the second fusion characteristic.

Optionally, the predetermined fusion method includes a bilinear pooling method or a tandem method.

On the basis of the foregoing embodiment, optionally, the apparatus further includes: a fourth obtaining module, a fifth obtaining module, a second fusion module and a second determining module, wherein:

and the fourth acquisition module is used for acquiring image symptom information of the medical image.

And the fifth acquisition module is used for inputting the image symptom information into a preset second neural network and acquiring the image symptom characteristics corresponding to the image symptom information.

And the second fusion module is used for performing fusion processing on the first fusion characteristic and the image symptom characteristic by adopting a preset fusion method to obtain a third fusion characteristic.

And the second determining module is used for determining a classification result according to the third fusion characteristic.

On the basis of the foregoing embodiment, optionally, the apparatus further includes: the sixth obtaining module, the seventh obtaining module, the eighth obtaining module, the ninth obtaining module and the training module, wherein:

the sixth acquisition module is used for acquiring sample images with different resolutions of the sample interesting region and the first label and the second label of the sample interesting region; the first label is used for representing the area of interest of the sample as a positive category or a negative category; the second label is used for characterizing the type of the sample interesting region, wherein the type of the sample interesting region comprises negative types and the type of a positive type of the sample interesting region when the sample interesting region is a positive type.

The seventh obtaining module is used for inputting each sample image into a preset initial classification network to obtain a first sample classification result and a second sample classification result of the sample interesting region; the first sample classification result is used for representing the interested area of the sample as a positive type or a negative type; the second sample classification result is used for characterizing the type of the sample interesting region, and the type of the sample interesting region comprises negative types and a positive type when the sample interesting region is a positive type.

The eighth obtaining module is configured to perform weighting processing on the second label by using the first label to obtain a weighted label corresponding to the second label, and perform weighting processing on the second sample classification result by using the first sample classification result to obtain a weighted classification result corresponding to the second sample classification result; and the negative categories in the weighted classification result are consistent with the negative categories in the first sample classification result.

And the ninth obtaining module is used for obtaining a first loss function according to the first label and the first sample classification result, and obtaining a second loss function according to the weighting label and the weighting classification result.

And the training module is used for training the initial classification network according to the first loss function and the second loss function to obtain the classification network.

The modules in the image classification device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.

In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:

In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:

In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the steps of:

It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high-density embedded nonvolatile Memory, resistive Random Access Memory (ReRAM), Magnetic Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (PCM), graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases involved in the embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the various embodiments provided herein may be, without limitation, general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, or the like.

The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method of image classification, the method comprising:

2. The method according to claim 1, wherein the inputting each image into a preset classification network to obtain the classification result of the region of interest includes:

extracting the features of the images to obtain the features of the images;

3. The method according to claim 2, wherein the performing feature fusion on the features of each image to obtain a first fused feature comprises:

4. The method of claim 3, wherein the classification results include a positive-negative classification result and a type of positive category; determining the classification result according to the first fusion feature includes:

5. The method of claim 2, further comprising:

acquiring clinical information of the medical image;

6. The method of claim 2, further comprising:

acquiring image symptom information of the medical image;

7. The method according to claim 5 or 6, wherein the predetermined fusion method comprises a bilinear pooling method or a tandem method.

8. The method of claim 1, wherein the training process of the classification network comprises:

inputting each sample image into a preset initial classification network to obtain a first sample classification result and a second sample classification result of the sample interesting region; the first sample classification result is used for representing that the sample interesting region is a positive type or a negative type; the second sample classification result is used for characterizing the type of the sample interesting region, wherein the type of the sample interesting region comprises negative types and the type of a positive type of the sample interesting region when the sample interesting region is a positive type;

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 8.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 8.