CN115082718A - Glioma grading method, device, equipment and medium based on histopathology image - Google Patents
Glioma grading method, device, equipment and medium based on histopathology image Download PDFInfo
- Publication number
- CN115082718A CN115082718A CN202210488276.3A CN202210488276A CN115082718A CN 115082718 A CN115082718 A CN 115082718A CN 202210488276 A CN202210488276 A CN 202210488276A CN 115082718 A CN115082718 A CN 115082718A
- Authority
- CN
- China
- Prior art keywords
- glioma
- image
- detected
- wsi
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 208000032612 Glial tumor Diseases 0.000 title claims abstract description 141
- 206010018338 Glioma Diseases 0.000 title claims abstract description 141
- 238000000034 method Methods 0.000 title claims abstract description 41
- 206010028980 Neoplasm Diseases 0.000 claims abstract description 63
- 201000011510 cancer Diseases 0.000 claims abstract description 62
- 230000036210 malignancy Effects 0.000 claims abstract description 62
- 238000012549 training Methods 0.000 claims abstract description 42
- 239000013598 vector Substances 0.000 claims description 123
- 230000004927 fusion Effects 0.000 claims description 22
- 230000002776 aggregation Effects 0.000 claims description 17
- 238000004220 aggregation Methods 0.000 claims description 17
- 230000009466 transformation Effects 0.000 claims description 17
- 238000000605 extraction Methods 0.000 claims description 11
- 238000001514 detection method Methods 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 8
- 230000004931 aggregating effect Effects 0.000 claims description 7
- 230000003211 malignant effect Effects 0.000 claims description 7
- 238000007781 pre-processing Methods 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000003860 storage Methods 0.000 claims description 6
- 238000002546 full scan Methods 0.000 claims description 2
- 238000003745 diagnosis Methods 0.000 abstract description 32
- 238000010586 diagram Methods 0.000 description 9
- 238000004590 computer program Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 230000008520 organization Effects 0.000 description 4
- 238000010827 pathological analysis Methods 0.000 description 3
- 230000004663 cell proliferation Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000001575 pathological effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 206010003571 Astrocytoma Diseases 0.000 description 1
- 208000003174 Brain Neoplasms Diseases 0.000 description 1
- 208000006274 Brain Stem Neoplasms Diseases 0.000 description 1
- 206010014967 Ependymoma Diseases 0.000 description 1
- 206010058467 Lung neoplasm malignant Diseases 0.000 description 1
- 201000010133 Oligodendroglioma Diseases 0.000 description 1
- 206010061902 Pancreatic neoplasm Diseases 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 201000007455 central nervous system cancer Diseases 0.000 description 1
- 208000025997 central nervous system neoplasm Diseases 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000003759 clinical diagnosis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 208000005017 glioblastoma Diseases 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 238000003064 k means clustering Methods 0.000 description 1
- 208000013409 limited attention Diseases 0.000 description 1
- 201000005202 lung cancer Diseases 0.000 description 1
- 208000020816 lung neoplasm Diseases 0.000 description 1
- 208000015486 malignant pancreatic neoplasm Diseases 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000011278 mitosis Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 201000002528 pancreatic cancer Diseases 0.000 description 1
- 208000008443 pancreatic carcinoma Diseases 0.000 description 1
- 230000007170 pathology Effects 0.000 description 1
- 238000004393 prognosis Methods 0.000 description 1
- 230000035755 proliferation Effects 0.000 description 1
- 230000004083 survival effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
According to the glioma grading method, device, equipment and medium based on the histopathology image, the acquired WSI image to be detected is input into a glioma grading model obtained through pre-training, the glioma malignancy grade corresponding to the WSI image to be detected is obtained, and then the glioma malignancy grade is stored, so that intelligent diagnosis is achieved, a doctor is assisted in diagnosing, the objectivity of glioma diagnosis results is enhanced, and the diagnosis accuracy is improved.
Description
Technical Field
The present application relates to the field of medical technology, and in particular, to a glioma grading method, device, apparatus, and medium based on histopathological images.
Background
Glioma accounts for about 30% of all brain and central nervous system tumors and 80% of all malignant brain tumors, the annual incidence rate of glioma in China is 5-10 ten thousand, and the 5-year mortality rate is third in all tumors, and is second to pancreatic cancer and lung cancer. Clinically, based on pathological diagnosis, different grades of glioma exhibit different characteristics in histopathological sections, for example, the appearance of mitosis indicates that the glioma is at least grade II or above, while microvascular proliferation and increased cell density correspond to grade III characteristics.
In the prior art, for the diagnosis of glioma, a pathologist usually preliminarily judges the malignancy degree by observing histopathological sections, and for some section images which are difficult to judge, molecular characteristics such as a cell proliferation fraction Ki-67 and the like need to be combined for judgment so as to determine the grade of the malignancy degree of glioma.
However, in the above diagnosis method, the diagnosis result of the pathologist has strong subjectivity, and the accuracy of the diagnosis result of glioma is not high.
Disclosure of Invention
The application provides a glioma grading method, a glioma grading device, glioma grading equipment and a glioma grading medium based on histopathology images, which are used for solving the problems that in the prior art, the diagnosis result of a pathologist has strong subjectivity, and the accuracy of the glioma diagnosis result is not high.
In a first aspect, the present application provides a glioma grading method based on histopathological images, comprising:
acquiring a histopathology full-scanning WSI image to be detected;
inputting the WSI image to be detected into a glioma grading model obtained by pre-training to obtain the glioma malignancy grade of the WSI image to be detected;
and storing the glioma malignancy grade of the WSI image to be detected.
Optionally, the method further includes:
acquiring training data, wherein the training data comprises a WSI image and a grade label, and the grade label is used for representing the malignant grade of glioma;
and training a preset model through the training data to obtain the glioma grading model.
Optionally, the inputting the WSI image to be detected into a glioma grading model obtained by pre-training to obtain a glioma malignancy grade of the WSI image to be detected includes:
performing data preprocessing on the WSI image to be detected to obtain image blocks under different magnification factors, wherein the image blocks are sub-images of the WSI image to be detected;
inputting the image blocks under each magnification factor into a feature extraction network for feature extraction to obtain feature vectors corresponding to the image blocks;
clustering the feature vectors corresponding to the image blocks respectively to obtain the classification of the feature vectors under each magnification, and obtaining a first preset number of ROI sub-packets according to the classification of the feature vectors under each magnification, wherein the ROI sub-packets comprise a second preset number of feature vectors, and the ROI sub-packets comprise feature vectors under different magnifications;
aggregating the feature vectors in the ROI sub-packet through a multi-example feature aggregation operator to generate feature representation of the ROI sub-packet;
classifying the characteristic representation of the ROI sub-packet through an MLP module, and predicting the malignant grade of glioma of the ROI sub-packet;
and determining the glioma malignancy level of the WSI image to be detected according to the glioma malignancy level of each ROI subcontractor.
Optionally, the performing data preprocessing on the WSI image to be detected to obtain image blocks under different magnification factors includes:
dividing the WSI image to be detected into a plurality of image blocks with a first preset size in a sliding window mode;
calculating the tissue proportion of each image block, and determining a target image block according to the tissue proportion, wherein the target image block is an image block with the tissue proportion larger than a preset threshold;
clipping the target image block to obtain an image block with a second preset size and an image block with a third preset size;
and scaling the image blocks with the first preset size, the image blocks with the second preset size and the image blocks with the third preset size to fixed sizes to obtain image blocks under different magnification factors.
Optionally, the aggregating the feature vectors in the ROI sub-packet by using a multi-instance feature aggregation operator to generate a feature representation of the ROI sub-packet includes:
inputting n feature vectors and an initialization vector of the ROI sub-packet into the multi-example feature aggregation operator, performing information fusion between the n feature vectors and the initialization vector through an information fusion formula to obtain a fusion result corresponding to the initialization vector, generating a new feature vector according to the fusion result, and taking the new feature vector as the feature vector of the ROI sub-packet;
the information fusion formula is as follows:
wherein Q is the vector of the initialization vector after the first linear transformation, K and V are the vectors of the target feature vector in the ROI sub-packet after the second linear transformation and the third linear transformation, U is a hyper-parameter threshold, S is the similarity of the initialization vector and the target feature vector, QK T Is a matrix of the attention of the user,
is a real number used to normalize the attention matrix.
Optionally, the determining the glioma malignancy level corresponding to the WSI image to be detected according to the glioma malignancy level of the ROI sub-packet includes:
and determining the glioma malignancy grade with the maximum quantity as the glioma malignancy grade corresponding to the WSI image to be detected through a voting mechanism.
In a second aspect, the present application provides a glioma grading device based on histopathological images, comprising:
the first acquisition module is used for acquiring a WSI image to be detected for full-scanning of the histopathology;
the detection module is used for inputting the WSI image to be detected into a glioma grading model obtained by pre-training to obtain the glioma malignancy grade of the WSI image to be detected;
and the storage module is used for storing the glioma malignancy grade of the WSI image to be detected.
Optionally, the apparatus further comprises:
a second obtaining module, configured to obtain training data, where the training data includes a WSI image and a grade label, and the grade label is used to indicate a malignancy grade of a glioma;
and the training module is used for training a preset model through the training data to obtain the glioma grading model.
In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes the computer-executable instructions stored in the memory to implement a method for glioma grading based on histopathological images as described in the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon computer-executable instructions for implementing a method for glioma grading based on histopathological images as described in the first aspect when the computer-executable instructions are executed by a processor.
In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, is adapted to perform a method for glioma grading based on images of histopathology according to the first aspect.
According to the glioma grading method, device, equipment and medium based on the histopathology image, the acquired WSI image to be detected is input into a glioma grading model obtained through pre-training, the glioma malignancy grade corresponding to the WSI image to be detected is obtained, and then the glioma malignancy grade is stored, so that intelligent diagnosis is achieved, a doctor is assisted in diagnosing, the objectivity of glioma diagnosis results is enhanced, and the diagnosis accuracy is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic flowchart of a glioma grading method based on histopathological images according to an embodiment of the present disclosure;
fig. 2 is a schematic flowchart of a glioma grading method based on histopathological images according to the second embodiment of the present application;
fig. 3 is a schematic view of a sliding window according to a second embodiment of the present application;
FIG. 4 is a schematic diagram of image cropping according to the second embodiment of the present application;
FIG. 5 is a diagram of image blocks in an ROI subpacket according to a second embodiment of the present application;
FIG. 6 is a diagram of a multi-example feature aggregation operator provided in embodiment two of the present application;
fig. 7 is a schematic processing procedure diagram of a glioma grading method based on histopathological images according to the second embodiment of the present application;
fig. 8 is a schematic structural diagram of a glioma grading device based on histopathological images according to a third embodiment of the present application;
fig. 9 is a schematic structural diagram of a glioma grading device based on a histopathological image according to a fourth embodiment of the present invention.
With the above figures, there are shown specific embodiments of the present application, which will be described in more detail below. These drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the inventive concepts to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
Clinical grading of glioma is based on pathological diagnosis, pathological diagnosis is final diagnosis and is also known as 'gold standard' in clinical diagnosis, a WSI image contains rich phenotype information, the WSI is a technology for scanning traditional pathological sections by using a digital scanner, acquiring digital images with high resolution, and seamlessly splicing and integrating obtained fragmented images by using a computer to manufacture visual digital images, and the WSI image can be used for monitoring potential mechanisms causing disease development and patient survival results.
The grading of glioma has important significance for the clinical treatment and prognosis of patients, and the world health organization is classified into I-IV grades according to the malignancy degree. In the prior art, for the diagnosis of glioma, a pathologist generally preliminarily judges the malignancy degree by observing histopathological sections, and for some section images which are difficult to judge, molecular characteristics such as a cell proliferation fraction Ki-67 and the like are combined for judgment so as to determine the grade of the malignancy degree of glioma.
However, in the above diagnosis method, the diagnosis result of the pathologist has strong subjectivity, and the accuracy of the diagnosis result of glioma is not high.
Therefore, the application provides a glioma grading method, device, equipment and medium based on histopathological images, a WSI image to be detected is input into a glioma grading model obtained through pre-training to obtain a diagnosis result of the WSI image, namely a glioma malignancy grade, so that intelligent diagnosis is realized, a doctor is assisted to diagnose, the objectivity of the glioma diagnosis result is enhanced, and the diagnosis accuracy is improved.
The application scenario of the present application may be diagnosis of oligodendroglioma, or may be diagnosis of glioblastoma (astrocytoma) or ependymoma. It is understood that the glioma grading method based on the histopathological image provided by the application includes, but is not limited to, the above application scenarios.
The following describes the technical solution of the present application and how to solve the above technical problems in detail by specific embodiments. The following embodiments may exist independently or in combination, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Referring to fig. 1, fig. 1 is a schematic flowchart of a method for glioma grading based on histopathology images according to an embodiment of the present application, where the method may be performed by a glioma grading device based on histopathology images, the device may be a server, and the method is described below by taking the server as an example, and includes the following steps.
S101, acquiring a WSI image to be detected.
The method comprises the steps that a server acquires a WSI image to be detected, specifically, the WSI image is obtained by scanning a traditional pathological section by a digital scanner, acquiring a digital image with high resolution, and seamlessly splicing and integrating the obtained fragmented images through the server.
S102, inputting the WSI image to be detected into a glioma grading model obtained through pre-training to obtain the glioma malignancy grade of the WSI image to be detected.
The glioma grading model obtained through pre-training is obtained by a server through training a preset model through obtained training data, the training data comprises WSI images and grade labels, illustratively, the grade labels can comprise I grade, II grade, III grade and IV grade, and the grade labels are used for distinguishing glioma malignancy grade.
After acquiring a WSI image to be detected, the server inputs the image into a glioma grading model obtained through pre-training to obtain a glioma malignancy grade of the image, so as to implement intelligent diagnosis.
S103, storing the glioma malignancy grade of the WSI image to be detected.
After the server determines the glioma malignancy grade of the WSI image to be detected, the glioma malignancy grade is stored, so that a pathologist can obtain the diagnosis result and perform targeted treatment on a patient.
In this embodiment, the acquired WSI image to be detected is input into a glioma grading model obtained through pre-training, so as to obtain a glioma malignancy grade corresponding to the WSI image to be detected, and then the glioma malignancy grade is stored, so as to realize intelligent diagnosis, assist a doctor in diagnosing, enhance the objectivity of a glioma diagnosis result, and improve the diagnosis accuracy.
In the following, two pairs of WSI images to be detected are input into a glioma grading model obtained by pre-training, and prediction of the grade of malignancy of glioma is performed.
Referring to fig. 2, fig. 2 is a schematic flowchart of a glioma grading method based on a histopathology image according to a second embodiment of the present application, where the glioma grading method based on a histopathology image may be executed by a glioma grading device based on a histopathology image, where the glioma grading device may be a server, and the method is described below by taking the server as an example, and includes the following steps.
S201, carrying out data preprocessing on the WSI image to be detected to obtain image blocks under different magnification factors, wherein the image blocks are sub-images of the WSI image to be detected.
Firstly, the server divides the WSI image to be detected into a plurality of image blocks of a first preset size in a sliding window manner, where the sliding window size is the first preset size, and exemplarily, the first preset size is 2048 × 2048 pixel size, and the sliding window refers to fig. 3.
Then, the organization ratio of each image block, that is, the ratio of the organization region is calculated, specifically, the organization ratio is obtained by calculating the ratio of the number of the pixels in the foreground region to the total number of the pixels, and taking the size of the sliding window example as an example, the total number of the pixels is 2048 2 . And determining a target image block according to the tissue ratio, wherein the target image block is an image block with the tissue area ratio larger than a preset threshold, and the preset threshold can be set to be 0.5. By discarding image blocks with excessively high background area ratio, the accuracy of the detection result can be improved.
Then, the target image block is cropped to obtain an image block of a second preset size and an image block of a third preset size, for example, for a target image block of 2048 × 2048, the target image block is cropped to obtain 4 image blocks of 1024 × 1024, and then the 1024 × 1024 image blocks are cropped again to obtain 4 image blocks of 512 × 512, referring to fig. 4.
Then scaling the first, second, and third preset image blocks to fixed sizes may obtain image blocks under different magnifications, for example, the fixed size may be 256 × 256, and the magnifications may be 5 times, 10 times, and 20 times. Of course, a magnification factor of 40 times, etc. may be included, and the present application is not limited thereto, and different magnification factors may be determined according to the size of the WSI image, and when the size of the WSI image reaches 40000 × 40000 and above, the magnification factor may be 5 times, 10 times, 20 times, 40 times, etc.
Through the data preprocessing, the WSI image to be detected is divided into image blocks with different sizes, and clustering can be performed according to the image blocks with different scales in the subsequent step S203, so that multi-scale features are effectively combined, and the model has better accuracy on a grading task.
S202, inputting the image blocks under each magnification factor into a feature extraction network for feature extraction to obtain feature vectors corresponding to the image blocks.
The extractable network can be a ResNet50 network, and can also be other neural networks for feature extraction.
S203, clustering the feature vectors corresponding to the image blocks respectively to obtain the classification of the feature vectors under each magnification, and obtaining a first preset number of ROI sub-packets according to the classification of the feature vectors under each magnification.
Specifically, the following examples are given:
the server clusters the feature vectors corresponding to the image blocks with 5 times magnification (2048 × 2048) through a clustering module, and the number of categories can be set to 4. The clustering module can be a K-means clustering algorithm Kmeans, and similarly, the feature vectors of image blocks with 10 times of magnification (1024 × 1024) and the feature vectors of image blocks with 20 times of magnification (512 × 512) are equally divided into 4 classes.
Then m feature vectors are selected from the four classes of feature vectors corresponding to the image blocks with different magnifications, the setting of m is 1, 4, 4 in sequence for the feature vectors corresponding to the image blocks with different magnifications, that is, 1 feature vector is selected from the four classes of feature vectors corresponding to the image blocks with 5 magnifications, 4 feature vectors are selected from the four classes of feature vectors corresponding to the image blocks with 10 magnifications, 4 feature vectors are selected from the four classes of feature vectors corresponding to the image blocks with 20 magnifications, respectively, an ROI sub-packet is formed, the sub-packet comprises the feature vectors corresponding to the image blocks with 4 magnifications of 5, the feature vectors corresponding to the image blocks with 16 magnifications of 10 and the feature vectors corresponding to the image blocks with 16 magnifications of 20, namely 36 feature vectors form an ROI sub-packet, 36 is the first predetermined number. Illustratively, a schematic diagram of image blocks in an ROI subpacket is shown in fig. 5.
The server generates a second preset number of ROI sub-packets, where the second preset number may be 20, and then the generation of the ROI sub-packets 20 times is described, it needs to be described that feature vectors included in each generated ROI sub-packet are different, that is, each ROI sub-packet does not have a repeated feature vector, so that multi-scale features are effectively combined, and the model obtains better accuracy in the hierarchical prediction.
S204, aggregating the feature vectors in the ROI sub-packet through a multi-example feature aggregation operator to generate feature representation of the ROI sub-packet.
The multi-example feature aggregation operator can be called as an encoder, because the feature vector Embedding in the ROI sub-packet is a highly concentrated high-level semantic feature, the contained information cannot be changed through subsequent back propagation, the essential function of the multi-example feature aggregation operator is that the feature vectors in the ROI sub-packet are effectively combined, and after too many layers of encoders are added, a result which cannot be converged is caused, because the data volume of the ROI sub-packet does not reach millions of levels, a layer is built on the selection of the encoder in the embodiment of the application. Experiments prove that the effect of the hierarchical model is greatly reduced along with the increase of the number of layers of the encoder. For an exemplary structure of the encoder, reference may be made to fig. 6.
Specifically, the process of aggregating the feature vectors in the ROI subpacket by the multi-instance feature aggregation operator is as follows:
the server inputs n eigenvectors and an initialization vector of the ROI sub-packet into the multi-instance feature aggregation operator, firstly regularizes the n +1 eigenvectors respectively to obtain regularized eigenvectors, then carries out first linear transformation, second linear transformation and third linear transformation on the n +1 eigenvectors respectively to obtain Query, Key and Value vectors which are Q, K and V for short.
The linear transformation is to multiply the eigenvector by three different linear transformation matrices to generate linearly transformed vectors Q, K and V.
Then, performing information fusion between the n feature vectors and the initialization vector through an information fusion formula (Attention mechanism Multi-head Limited Attention) to obtain a fusion result corresponding to the initialization vector, wherein the information fusion formula is specifically as follows:
q is the vector of the initialized vector after the first linear transformation, K and V are the vectors of the target characteristic vector in the ROI sub-packet after the second linear transformation and the third linear transformation, U is a super-parameter threshold value which can be 0.8, S is the similarity of the initialized vector and the target characteristic vector, and QK is the similarity of the initialized vector and the target characteristic vector T Is a matrix of the attention of the user,
is a real number, where d k Which may be 64, for normalizing the attention moment array. The target feature vector is any one of the feature vectors in the ROI sub-packet.
After the calculation, the server obtains a fusion result corresponding to the initialization vector, the fusion result is a feature vector, residual calculation is performed on the fusion result and the initialization vector, the result of the residual calculation is regularized, the regularization result is processed through a multi-layer Perceptron (MLP) to obtain an optimized feature vector, residual calculation is performed on the optimized feature vector and the residual calculation result to generate a new feature vector, the new feature vector is used as the feature representation of the ROI sub-packet, so that the model obtains a target region needing important attention, an attention focus is obtained, and then more attention is given to the region to obtain more detailed information of the target needing attention, thereby inhibiting other useless information and improving the accuracy of the model.
It should be noted that, in the n +1 feature vectors, every two feature vectors need to be calculated according to the formula (1), the number of the obtained new feature vectors is n +1, and since the new feature vectors corresponding to the initialization vector only need to be used as the feature representation of the ROI sub-packet, the calculation between every two feature vectors in the n feature vectors in the ROI sub-packet is not described in the above description.
S205, classifying the characteristic representation of the ROI sub-package through an MLP module, and predicting the malignant grade of the glioma of the ROI sub-package.
And after obtaining the characteristic representation of the ROI sub-package, the server classifies the characteristic representation of the ROI sub-package through an MLP module and predicts the malignant grade of the glioma of the ROI sub-package.
S206, determining the glioma malignancy level of the WSI image to be detected according to the glioma malignancy level of each ROI subcontractor.
Through the operation, the server can determine the glioma malignancy grade of each ROI sub-packet, and then can determine the glioma malignancy grade of the WSI image to be detected through a voting mechanism according to the glioma malignancy grade of each ROI sub-packet, and specifically, the server determines the glioma malignancy grade with the largest number as the glioma malignancy grade corresponding to the WSI image to be detected. Illustratively, if the grade of malignancy of glioma in 15 ROI subpackets of 20 ROI subpackets is grade iii, the grade of malignancy of glioma in the WSI image to be detected is determined to be grade iii.
The glioma malignancy grade of the WSI image to be detected is determined through the voting mechanism, so that the model prediction accuracy is high.
For example, the above processing may refer to fig. 7.
In this embodiment, the server inputs image blocks of the WSI image to be detected under different magnification factors into the feature extraction network for feature extraction, so as to obtain feature vectors corresponding to the image blocks. Then the server clusters the feature vectors under each magnification factor respectively, divides the feature vectors under each magnification factor into four types, obtains a first preset number of ROI sub-packets according to the four types of feature vectors under each magnification factor, and effectively combines multi-scale features, so that the model obtains better accuracy in hierarchical prediction. And then, the server aggregates the feature vectors in the ROI sub-packets through a multi-example feature aggregation operator, classifies the feature representations of the ROI sub-packets obtained after aggregation through an MLP (Multi-instance Markov model) module, predicts the glioma malignancy level of each ROI sub-packet, and determines the glioma malignancy level of the WSI image to be detected through a voting mechanism. The intelligent diagnosis is realized, the diagnosis is assisted by doctors, the objectivity of glioma diagnosis results is enhanced, and the diagnosis accuracy is improved.
It should be noted that, the server trains the preset model through the training data to obtain the glioma grading model, and the operations in steps S501-S504 in the second embodiment also need to be performed, except that in step S503, after the ROI sub-packet is obtained, the same level labels as those of the WSI image need to be allocated to the ROI sub-packet, and then the multi-instance feature aggregation operator and the MLP module in the preset model are trained through each ROI sub-packet and the corresponding level labels to obtain the glioma grading model.
Referring to fig. 8, fig. 8 is a schematic structural diagram of a glioma grading device based on histopathological images according to a third embodiment of the present application. The apparatus 80 comprises: a first acquisition module 801, a detection module 802 and a saving module 803.
The first acquisition module 801 is configured to acquire a full-scan WSI image of a tissue and a pathology to be detected.
The detection module 802 is configured to input the WSI image to be detected into a glioma classification model obtained through pre-training, so as to obtain a glioma malignancy grade of the WSI image to be detected.
A saving module 803, configured to save the glioma malignancy grade of the WSI image to be detected.
Optionally, the apparatus 80 further comprises:
and a second acquisition module for acquiring training data, wherein the training data comprises a WSI image and a grade label, and the grade label is used for representing the malignant grade of the glioma.
And the training module is used for training the preset model through the training data to obtain the glioma grading model.
Optionally, the detection module 802 is specifically configured to:
and performing data preprocessing on the WSI image to be detected to obtain image blocks under different magnification factors, wherein the image blocks are sub-images of the WSI image to be detected.
And inputting the image blocks under each magnification factor into a feature extraction network for feature extraction to obtain feature vectors corresponding to the image blocks.
And clustering the feature vectors corresponding to the image blocks respectively to obtain the classification of the feature vectors under each magnification, and obtaining a first preset number of ROI sub-packets according to the classification of the feature vectors under each magnification, wherein the ROI sub-packets comprise a second preset number of feature vectors and comprise feature vectors under different magnifications.
And aggregating the feature vectors in the ROI sub-packet through a multi-example feature aggregation operator to generate the feature representation of the ROI sub-packet.
And classifying the characteristic representation of the ROI sub-packet through an MLP module, and predicting the glioma malignancy level of the ROI sub-packet.
And determining the glioma malignancy level of the WSI image to be detected according to the glioma malignancy level of each ROI subcontractor.
Optionally, the detection module 802 is specifically configured to:
the WSI image to be detected is divided into a plurality of image blocks with a first preset size in a sliding window mode.
Calculating the tissue ratio of each image block, and determining a target image block according to the tissue ratio, wherein the target image block is an image block with the tissue ratio larger than a preset threshold.
And clipping the target image block to obtain an image block with a second preset size and an image block with a third preset size.
And scaling the image blocks with the first preset size, the second preset size and the third preset size to fixed sizes to obtain image blocks under different magnification factors.
Optionally, the detection module 802 is specifically configured to:
inputting n feature vectors and an initialization vector of the ROI sub-packet into a multi-example feature aggregation operator, performing information fusion between the n feature vectors and the initialization vector through an information fusion formula to obtain a fusion result corresponding to the initialization vector, generating a new feature vector according to the fusion result, and using the new feature vector as the feature representation of the ROI sub-packet.
The information fusion formula is as follows:
wherein Q is a vector of the initialization vector after the first linear transformation, K and V are vectors of the target feature vector in the ROI sub-packet after the second linear transformation and the third linear transformation, U is a hyper-parameter threshold, S is the similarity of the initialization vector and the target feature vector, QK T Is a matrix of the attention of the user,
is a real number used to normalize the attention moment array.
Optionally, the detection module 802 is specifically configured to:
and determining the glioma malignancy grade with the maximum quantity as the glioma malignancy grade corresponding to the WSI image to be detected through a voting mechanism.
The apparatus of this embodiment may be used to perform the step of the glioma grading method based on the histopathological image in the first embodiment or the second embodiment, and the specific implementation and the technical effect are similar, and are not described herein again.
Fig. 9 is a schematic structural diagram of a glioma grading device based on histopathological images according to a fourth embodiment of the present invention, as shown in fig. 9, the device 90 includes: the processor 901, the memory 902, the transceiver 903, the memory 902 is configured to store instructions, the transceiver 903 is configured to communicate with other devices, and the processor 901 is configured to execute the instructions stored in the memory, so that the apparatus 90 executes the steps of the glioma grading method based on the histopathology image according to the first embodiment or the second embodiment.
A fifth embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program is used for implementing the steps of the glioma grading method based on histopathological images in the first embodiment or the second embodiment when being executed by a processor, where specific implementation manners and technical effects are similar, and are not described herein again.
A sixth embodiment of the present invention provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the glioma grading method based on histopathology images according to the first embodiment or the second embodiment are implemented, and the specific implementation manner and the technical effect are similar, and are not described herein again.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (10)
1. A method for glioma grading based on histopathological images, characterized in that it comprises:
acquiring a histopathology full-scanning WSI image to be detected;
inputting the WSI image to be detected into a glioma grading model obtained by pre-training to obtain the glioma malignancy grade of the WSI image to be detected;
and storing the glioma malignancy grade of the WSI image to be detected.
2. The method of claim 1, further comprising:
acquiring training data, wherein the training data comprises a WSI image and a grade label, and the grade label is used for representing the malignant grade of glioma;
and training a preset model through the training data to obtain the glioma grading model.
3. The method according to claim 1 or 2, wherein the step of inputting the WSI image to be detected into a glioma grading model obtained by pre-training to obtain the glioma malignancy grade of the WSI image to be detected comprises the following steps:
performing data preprocessing on the WSI image to be detected to obtain image blocks under different magnification factors, wherein the image blocks are sub-images of the WSI image to be detected;
inputting the image blocks under each magnification factor into a feature extraction network for feature extraction to obtain feature vectors corresponding to the image blocks;
clustering the feature vectors corresponding to the image blocks respectively to obtain the classification of the feature vectors under each magnification, and obtaining a first preset number of ROI sub-packets according to the classification of the feature vectors under each magnification, wherein the ROI sub-packets comprise a second preset number of feature vectors, and the ROI sub-packets comprise feature vectors under different magnifications;
aggregating the feature vectors in the ROI sub-packet through a multi-example feature aggregation operator to generate feature representation of the ROI sub-packet;
classifying the characteristic representation of the ROI sub-packet through an MLP module, and predicting the malignant grade of glioma of the ROI sub-packet;
and determining the glioma malignancy level of the WSI image to be detected according to the glioma malignancy level of each ROI subcontractor.
4. The method according to claim 3, wherein the pre-processing the data of the WSI image to be detected to obtain image blocks under different magnifications comprises:
dividing the WSI image to be detected into a plurality of image blocks with a first preset size in a sliding window mode;
calculating the tissue proportion of each image block, and determining a target image block according to the tissue proportion, wherein the target image block is an image block with the tissue proportion larger than a preset threshold;
clipping the target image block to obtain an image block with a second preset size and an image block with a third preset size;
and scaling the image blocks with the first preset size, the image blocks with the second preset size and the image blocks with the third preset size to fixed sizes to obtain image blocks under different magnification factors.
5. The method of claim 4, wherein the generating the feature representation of the ROI subpacket by aggregating feature vectors within the ROI subpacket through a multi-instance feature aggregation operator comprises:
inputting n feature vectors and an initialization vector of the ROI sub-packet into the multi-example feature aggregation operator, performing information fusion between the n feature vectors and the initialization vector through an information fusion formula to obtain a fusion result corresponding to the initialization vector, generating a new feature vector according to the fusion result, and using the new feature vector as feature representation of the ROI sub-packet;
the information fusion formula is as follows:
wherein Q is the initialTransforming the vector after the first linear transformation, K and V are the vectors after the second linear transformation and the third linear transformation of the target feature vector in the ROI sub-packet, U is a hyper-parameter threshold, S is the similarity between the initialization vector and the target feature vector, QK T Is a matrix of the attention of the user,
is a real number used to normalize the attention matrix.
6. The method of claim 5, wherein the determining the glioma malignancy level corresponding to the WSI image to be detected according to the glioma malignancy level of the ROI sub-packet comprises:
and determining the glioma malignancy grade with the maximum quantity as the glioma malignancy grade corresponding to the WSI image to be detected through a voting mechanism.
7. A glioma grading device based on histopathological images, characterized in that it comprises:
the first acquisition module is used for acquiring a histopathology full-scan WSI image to be detected;
the detection module is used for inputting the WSI image to be detected into a glioma grading model obtained by pre-training to obtain the glioma malignancy grade of the WSI image to be detected;
and the storage module is used for storing the glioma malignancy grade of the WSI image to be detected.
8. The apparatus of claim 7, further comprising:
a second obtaining module, configured to obtain training data, where the training data includes a WSI image and a grade label, and the grade label is used to indicate a malignancy grade of a glioma;
and the training module is used for training a preset model through the training data to obtain the glioma grading model.
9. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes the computer-executable instructions stored in the memory to implement the method for glioma grading based on histopathological images according to any one of claims 1-6.
10. A computer-readable storage medium having stored thereon computer-executable instructions for implementing the method for glioma grading based on histopathological images according to any one of claims 1 to 6 when being executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210488276.3A CN115082718A (en) | 2022-05-06 | 2022-05-06 | Glioma grading method, device, equipment and medium based on histopathology image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210488276.3A CN115082718A (en) | 2022-05-06 | 2022-05-06 | Glioma grading method, device, equipment and medium based on histopathology image |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115082718A true CN115082718A (en) | 2022-09-20 |
Family
ID=83248022
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210488276.3A Pending CN115082718A (en) | 2022-05-06 | 2022-05-06 | Glioma grading method, device, equipment and medium based on histopathology image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115082718A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109271969A (en) * | 2018-10-29 | 2019-01-25 | 北京青燕祥云科技有限公司 | A kind of Histopathologic Grade of Cerebral Gliomas evaluation method and its device |
| CN109754879A (en) * | 2019-01-04 | 2019-05-14 | 湖南兰茜生物科技有限公司 | A kind of lung cancer computer aided detection method and system based on deep learning |
| US20190156159A1 (en) * | 2017-11-20 | 2019-05-23 | Kavya Venkata Kota Sai KOPPARAPU | System and method for automatic assessment of cancer |
| CN110427970A (en) * | 2019-07-05 | 2019-11-08 | 平安科技（深圳）有限公司 | Image classification method, device, computer equipment and storage medium |
| CN112884724A (en) * | 2021-02-02 | 2021-06-01 | 广州智睿医疗科技有限公司 | Intelligent judgment method and system for lung cancer histopathological typing |
| CN113366530A (en) * | 2019-05-29 | 2021-09-07 | 徕卡生物***成像股份有限公司 | Tumor review and post-operative tumor margin assessment in computer-supported histological images |
-
2022
- 2022-05-06 CN CN202210488276.3A patent/CN115082718A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190156159A1 (en) * | 2017-11-20 | 2019-05-23 | Kavya Venkata Kota Sai KOPPARAPU | System and method for automatic assessment of cancer |
| CN109271969A (en) * | 2018-10-29 | 2019-01-25 | 北京青燕祥云科技有限公司 | A kind of Histopathologic Grade of Cerebral Gliomas evaluation method and its device |
| CN109754879A (en) * | 2019-01-04 | 2019-05-14 | 湖南兰茜生物科技有限公司 | A kind of lung cancer computer aided detection method and system based on deep learning |
| CN113366530A (en) * | 2019-05-29 | 2021-09-07 | 徕卡生物***成像股份有限公司 | Tumor review and post-operative tumor margin assessment in computer-supported histological images |
| CN110427970A (en) * | 2019-07-05 | 2019-11-08 | 平安科技（深圳）有限公司 | Image classification method, device, computer equipment and storage medium |
| CN112884724A (en) * | 2021-02-02 | 2021-06-01 | 广州智睿医疗科技有限公司 | Intelligent judgment method and system for lung cancer histopathological typing |
Non-Patent Citations (3)
| Title |
|---|
| PENGSHUAI YANG等: "Self-supervised Visual Representation Learning for Histopathological Images", MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION-MICCAI 2021, 21 September 2021 (2021-09-21) * |
| XI WANG等: "Weakly Supervised Deep Learning for Whole Slide Lung Cancer Image Analysis", IEEE TRANSACTIONS ON CYBERNETICS, 2 September 2019 (2019-09-02) * |
| 颜锐等: "基于深度学习和染色病理图像的肿瘤相关指标预测研究综述", 计算机科学, 28 February 2022 (2022-02-28) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Abdelrahman et al. | Convolutional neural networks for breast cancer detection in mammography: A survey | |
| Priego-Torres et al. | Automatic segmentation of whole-slide H&E stained breast histopathology images using a deep convolutional neural network architecture | |
| Vyas et al. | Generative Adversarial Networks for Anomaly Detection in Medical Images | |
| KR20230059799A (en) | A Connected Machine Learning Model Using Collaborative Training for Lesion Detection | |
| CN109389129A (en) | A kind of image processing method, electronic equipment and storage medium | |
| CN110046627B (en) | Method and device for identifying mammary gland image | |
| CN114565761A (en) | Deep learning-based method for segmenting tumor region of renal clear cell carcinoma pathological image | |
| Chen et al. | Automatic whole slide pathology image diagnosis framework via unit stochastic selection and attention fusion | |
| Crasta et al. | A novel Deep Learning architecture for lung cancer detection and diagnosis from Computed Tomography image analysis | |
| Shen et al. | Multicontext multitask learning networks for mass detection in mammogram | |
| Jain et al. | Brain Tumor Detection using MLops and Hybrid Multi-Cloud | |
| Üzen | Convmixer-based encoder and classification-based decoder architecture for breast lesion segmentation in ultrasound images | |
| CN116416221A (en) | Ultrasonic image analysis method | |
| Chaudhury et al. | Classification of Breast Masses Using Ultrasound Images by Approaching GAN, Transfer Learning, and Deep Learning Techniques | |
| CN115082718A (en) | Glioma grading method, device, equipment and medium based on histopathology image | |
| Zhou et al. | Pcrlv2: A unified visual information preservation framework for self-supervised pre-training in medical image analysis | |
| Gargouri et al. | An automatic breast computer‐aided diagnosis scheme based on a weighted fusion of relevant features and a deep CNN classifier | |
| Nida et al. | A Novel Region‐Extreme Convolutional Neural Network for Melanoma Malignancy Recognition | |
| Hwang et al. | RBCA-Net: Reverse boundary channel attention network for kidney tumor segmentation in CT images | |
| Princy Magdaline et al. | Detection of lung cancer using novel attention gate residual U-Net model and KNN classifier from computer tomography images | |
| Hsu et al. | Spatial-slice feature learning using visual transformer and essential slices selection module for covid-19 detection of ct scans in the wild | |
| Soundarya et al. | Brain Tumor Detection Using Image Processing | |
| Sreelekshmi et al. | SwinCNN: An Integrated Swin Trasformer and CNN for Improved Breast Cancer Grade Classification | |
| Al Khalil et al. | Multi-modal brain tumor segmentation via conditional synthesis with Fourier domain adaptation | |
| CN116486184B (en) | Mammary gland pathology image identification and classification method, system, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |