CN117437207A - Multi-expert fusion chest X-ray image auxiliary diagnosis system and method - Google Patents

Multi-expert fusion chest X-ray image auxiliary diagnosis system and method Download PDF

Info

Publication number
CN117437207A
CN117437207A CN202311494684.0A CN202311494684A CN117437207A CN 117437207 A CN117437207 A CN 117437207A CN 202311494684 A CN202311494684 A CN 202311494684A CN 117437207 A CN117437207 A CN 117437207A
Authority
CN
China
Prior art keywords
chest
image
ray
disease
focus
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311494684.0A
Other languages
Chinese (zh)
Inventor
唐智威
杨有
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing Normal University
Original Assignee
Chongqing Normal University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing Normal University filed Critical Chongqing Normal University
Priority to CN202311494684.0A priority Critical patent/CN117437207A/en
Publication of CN117437207A publication Critical patent/CN117437207A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/803—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of input or preprocessed data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/809—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data
    • G06V10/811—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data the classifiers operating on different input data, e.g. multi-modal recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10116—X-ray image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Quality & Reliability (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

The invention relates to the technical field of medical information, in particular to a multi-expert fusion chest X-ray image auxiliary diagnosis system and method.

Description

Multi-expert fusion chest X-ray image auxiliary diagnosis system and method
Technical Field
The invention relates to the technical field of medical information, in particular to a multi-expert fusion chest X-ray image auxiliary diagnosis system and method.
Background
Disease classification and lesion localization are important research topics in automated chest radiography disease diagnosis techniques that are expected to improve disease diagnosis accuracy and reduce the workload of radiologists. Such techniques are largely dependent on a large number of image level and lesion level labels.
And because the disclosed data set has the problem of unbalanced category, the model trained by using the disclosed data set often has the phenomenon of eccentric diagnosis, and the diagnosis accuracy is affected.
Disclosure of Invention
The invention aims to provide a multi-expert fusion chest X-ray image auxiliary diagnosis system and a multi-expert fusion chest X-ray image auxiliary diagnosis method, and aims to solve the problem that an existing diagnosis model is low in accuracy.
In order to achieve the above object, in a first aspect, the present invention provides a multi-expert fusion chest X-ray image auxiliary diagnosis system, which includes an input module, an extraction module, a first chest X-ray multi-head network module, a second chest X-ray multi-head network module, a focus positioning fusion module and a disease classification fusion module;
the input module is used for inputting chest X-ray images;
the extraction module is used for extracting the chest X-ray image to obtain a chest region-of-interest image and an image cardiopulmonary mask;
the first chest X-ray multi-head network module is used for calculating the chest region-of-interest image to obtain a first focus candidate region and a first disease positive probability;
the second chest X-ray multi-head network module is used for calculating the chest region-of-interest image and the image heart-lung mask to obtain a second focus candidate region and a second disease positive probability;
the focus positioning fusion module outputs a corresponding focus area based on the first focus candidate area and the second focus candidate area;
and the disease classification fusion module outputs disease data based on the second disease positive probability and the second disease positive probability.
The extraction module comprises an area extraction unit and a mask extraction unit;
the region extraction unit is used for extracting the chest X-ray image to obtain a chest region of interest image;
the mask extraction unit is used for extracting the chest X-ray image to obtain an image heart-lung mask.
Wherein the first chest X-ray multi-head network module and the second chest X-ray multi-head network module each comprise a plurality of lesion localization heads and a disease classification head;
the focus positioning head is used for outputting the first focus candidate zone and the second focus candidate zone by a focus positioning expert;
the disease classification head is used for outputting the first disease positive probability and the second disease positive probability by a disease classification expert.
The focus positioning heads are respectively a GradCAM positioning head, an XGradCAM positioning head, a HiResCAM positioning head and a LayerCAM positioning head.
In a second aspect, a multi-expert fusion chest X-ray image aided diagnosis method is applied to the multi-expert fusion chest X-ray image aided diagnosis system of the first aspect, and includes the following steps:
inputting chest X-ray images;
extracting chest X-ray images to obtain chest region-of-interest images and image cardiopulmonary masks;
calculating the chest region of interest image to obtain a first focus candidate region and a first disease positive probability;
calculating the chest region of interest image and the image heart-lung mask to obtain a second focus candidate region and a second disease positive probability;
outputting a corresponding lesion area based on the first lesion candidate zone and the second lesion candidate zone;
disease data is output based on the second disease positive probability and the second disease positive probability.
The invention relates to a multi-expert fusion chest X-ray image auxiliary diagnosis system and a method, which are used for inputting chest X-ray images; extracting chest X-ray images to obtain chest region-of-interest images and image cardiopulmonary masks; calculating the chest region of interest image to obtain a first focus candidate region and a first disease positive probability, and calculating the chest region of interest image and an image heart-lung mask to obtain a second focus candidate region and a second disease positive probability; outputting a corresponding lesion area based on the first lesion candidate zone and the second lesion candidate zone; based on the second disease positive probability and the second disease positive probability, a disease diagnosis model named as a multi-expert fusion chest X-ray image auxiliary diagnosis model is provided for detecting diseases and focus areas in X-ray chest X-ray images, the model adopts a multi-expert fusion method, consists of two chest X-ray multi-head networks, comprises 8 disease positioning experts and 28 disease classification experts, trains the two models by using different training strategies so as to enable each expert to obtain different diagnosis 'features', and then carries out weighted summation on diagnosis conclusions of each expert to obtain a more accurate result, thereby solving the problem of lower accuracy of the existing diagnosis model.
Drawings
The invention may be further illustrated by means of non-limiting examples given in the accompanying drawings.
FIG. 1 is a diagram of the overall architecture of a multi-expert fusion chest X-ray image aided diagnosis model.
Fig. 2 is a architecture of a chest X-ray multi-head network.
Fig. 3 is a diagram showing 12 lesion localization.
Fig. 4 is a lesion localization map of 8 CAM Haad outputs.
Fig. 5 is a schematic structural diagram of a multi-expert fusion chest X-ray image aided diagnosis system according to the present invention.
Fig. 6 is a flow chart of a multi-expert fusion chest X-ray image aided diagnosis model.
FIG. 7 is a flow chart of a multi-expert fusion chest X-ray image assisted diagnostic system of the present invention.
1-input module, 2-extraction module, 3-first chest X-ray multi-head network module, 4-second chest X-ray multi-head network module, 5-focus location fusion module, 6-disease classification fusion module.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative and intended to explain the present invention and should not be construed as limiting the invention.
Referring to fig. 1 to 5, in a first aspect, the present invention provides a multi-expert fusion chest X-ray image auxiliary diagnosis system, which includes an input module 21, an extraction module, a first chest X-ray multi-head network module 3, a second chest X-ray multi-head network module 4, a focus positioning fusion module 5 and a disease classification fusion module 6;
the input module 21 is used for inputting chest X-ray images;
the extraction module is used for extracting the chest X-ray image to obtain a chest region-of-interest image and an image cardiopulmonary mask;
the first chest X-ray multi-head network module 3 is used for calculating the chest region-of-interest image to obtain a first focus candidate region and a first disease positive probability;
the second chest X-ray multi-head network module 4 is used for calculating the chest region-of-interest image and the image heart-lung mask to obtain a second focus candidate region and a second disease positive probability;
the focus positioning and fusion module 5 outputs a corresponding focus area based on the first focus candidate area and the second focus candidate area;
and the disease classification fusion module 6 outputs disease data based on the second disease positive probability and the second disease positive probability.
In the present embodiment, the focus positioning fusion device is specially used for constructing a plurality of focus positioning obtained by focus positioning expert groupThe weighted summation is used to generate more accurate lesion localization results. The focus positioning fusion device adopts a weighted boundary box fusion algorithm to fuse a plurality of positioning suggestion boxes, and the weight of each positioning suggestion box is obtained by a Discrete Particle Swarm Optimization (DPSO). Here, 4 coordinates of each localization advice rectangular frame are synthesized into an 8-dimensional vector and then a focus localization advice matrix P output by 8 focus localization experts is obtained n ∈R 8×8 In addition, under the nth disease, the weight matrix of the whole focus positioning expert group is set asPn after weighted fusion is set to +.>Then->The calculation formula of (2) is as follows:
where x represents matrix multiplication and sum represents summing the matrix elements. Then constructing a fitness function Fit of the discrete particle swarm optimizer as follows:
wherein IoR () represents the computationAnd true focus marking frame->The IoR score in between. Finally substituting the formula (1) into the formula (2) and taking all K' diseases into consideration to establish an optimization target T:
the discrete particle swarm optimization algorithm can search a group of optimal parameters in the parameter spaceSo that the value of the fitness function Fit reaches a minimum, i.e. maximally draws +.>And->Distance between them. The final experimental results are obtained from table 3:
the focus positioning IoR score of the multi-expert fusion chest X-ray image auxiliary diagnosis model after the focus positioning fusion device is adopted exceeds the highest score of a single chest X-ray multi-head network in all K' diseases;
the class fusion is essentially a selector that selects the one of the two disease classification specialists with the highest classification confidence as the final prediction result of the disease. Here, the probability that the first group of disease classification experts predicts that the nth disease is positive is set to be The probability of the second group is likewise set to +.>Final prediction result after fusion by category fusion device +.>The calculation formula of (2) is as follows:
where max () returns the parameter of the two parameters with the largest value. The final experimental results are obtained from table 3, and the classification AUC score of the multi-expert fusion chest X-ray image auxiliary diagnosis model after the disease classification fusion device is adopted exceeds the highest score of a single chest X-ray multi-head network in all K diseases.
The first chest X-ray multi-head network module 3 further comprises a global attention alignment unit (GAA); GAA is responsible for A n Is limited to the cardiopulmonary area of the image. The input to GAA is significance map a n And input image cardiopulmonary area mask B, GAA can mask A n Limiting the model to the cardiopulmonary region of the image can make the model pay more attention to the cardiopulmonary region of the input image when the diagnosis result is obtained, so that the model can obtain performance improvement on the diagnosis of certain diseases, namely, the model can obtain different 'features'. To A is to n Limiting to the cardiopulmonary region of the image requires a loss to measure a n How much attention component is contained in the input image cardiopulmonary area mask B. The loss should meet the following two conditions: first, A n The more intersections with B, the lower the loss, when A n The loss drops to 0 when fully contained in B. Second, unlike IOU metrics, the penalty does not require A n Similar to the shape of B or the same area. Let the sign of the loss function be L c1 Named convergence loss, the calculation mode is called convergence calculation, and the calculation symbol is θ, L cl The calculation formula of (2) is as follows.
L cl ＝AθB， (8)
AθB＝S A S A∩B ， (9)
Wherein S is A And S is A∩B The calculation formula of (2) is as follows:
S A represents the result of summing A, S A∩B Representing the result of summing the intersection of a and B,representing the result of normalizing A, the matrix is bit-wise multiplied by the result of the normalization. To obtain the convergence loss corresponding to the nth diseaseReplacing A in the formulas (8) to (11) with A corresponding to the nth disease n And (3) obtaining the product. The GAA module is used for applying convergence loss in the model, and the network structure of the model is not changed by the GAA;
inputting chest X-ray images; extracting chest X-ray images to obtain chest region-of-interest images and image cardiopulmonary masks; calculating the chest region of interest image to obtain a first focus candidate region and a first disease positive probability; calculating the chest region of interest image and the image heart-lung mask to obtain a second focus candidate region and a second disease positive probability; outputting a corresponding lesion area based on the first lesion candidate zone and the second lesion candidate zone; based on the second disease positive probability and the second disease positive probability, a disease diagnosis model named as a multi-expert fusion chest X-ray image auxiliary diagnosis model is provided for detecting diseases and focus areas in X-ray chest X-ray images, the model adopts a multi-expert fusion method, consists of two chest X-ray multi-head networks, comprises 8 disease positioning experts and 28 disease classification experts, trains the two models by using different training strategies so as to enable each expert to obtain different diagnosis 'features', and then carries out weighted summation on diagnosis conclusions of each expert to obtain a more accurate result, thereby solving the problem of lower accuracy of the existing diagnosis model.
Further, the extraction module comprises a region extraction unit and a mask extraction unit;
the region extraction unit is used for extracting the chest X-ray image to obtain a chest region of interest image;
the mask extraction unit is used for extracting the chest X-ray image to obtain an image heart-lung mask.
In this embodiment, please refer to fig. 2, wherein X' represents the X-ray image cut by the chest ROI extractor, M represents the feature map of the last convolution layer output, a n Represents the saliency map of the nth disease, the white part is a saliency area, B represents a mask image of a heart lung area corresponding to the input image, K represents the total number of disease types, K' represents the total number of disease types with focus level labels,for convergence calculation, ++>Is Sigmoid, f n The full-connection layer global classification head representing the nth disease is mainly used for predicting disease classification results; the global attention alignment module (GAA) is mainly used for generating a saliency region and limiting the saliency region to a heart-lung region of an input image; the CAM heads are used for providing focus area suggestion, and four different CAM heads are arranged in a chest X-ray multi-head network; the fusion block is a feature fusion module in the chest X-ray multi-head network and is used for fusing a plurality of features with different scales. The X-ray image X ' and the mask image B of the heart and lung area corresponding to the input image after the cutting by the extractor are output as positive probability of K diseases and 4K ' focus candidate frames of the K ' diseases, and the chest X-ray multi-head network is composed of four modules: the feature extractor similar to UNet++ is used for obtaining a high-resolution feature map M of the input image X; all-aroundThe office classification head module generates a saliency map A according to the feature map M n And classification result Y n ，n∈[1，K]The method comprises the steps of carrying out a first treatment on the surface of the The global attention alignment module outputs the salient region A of the global classification head n Limiting to the heart and lung area B of the input image; the four CAM modules acquire the Weight of the last convolution layer in the feature extractor by using the Weight Hook technology, and four different focus area suggestions are provided according to the Weight>Each global classification head instance in the chest X-ray multi-head network is responsible for the prediction of a disease, and each GAA module instance is responsible for generating a region of disease generation saliency and confining the saliency region to the input image cardiopulmonary region.
Further, the first chest X-ray multi-head network module 3 and the second chest X-ray multi-head network module 4 each comprise a plurality of lesion localization heads and a disease classification head;
the focus positioning head is used for outputting the first focus candidate zone and the second focus candidate zone by a focus positioning expert;
the disease classification head is used for outputting the first disease positive probability and the second disease positive probability by a disease classification expert.
In this embodiment, the disease classification head is used to generate a saliency map and a predictive classification result. Output prediction class Y of GCH n The calculation formula of (2) is as follows:
Y n ＝σ(f n (g(M)))， (5)
wherein Y is n Represents the probability that the nth disease is positive, n.epsilon.0, K). f (f) n Is the full connection layer of the nth disease, g is the maximum pooling layer, and sigma is Sigmoid. To obtain significance map A of nth disease n Classical CAM is to multiply the weight parameters of the last fully connected layer with the feature map output by the convolutional layer of the last layer of the feature extraction network and then sum. The scheme is slightly different, wherein M is the last layer output of the feature extraction network, f n Full ligation for nth diseaseLayer, f in equation (1) and equation (2) n Sharing parameters. f (f) n Is defined as:
wherein M is E R C×W×H ，A n ∈R W×H ，B∈R W×H W, H are the height and width of the feature map, respectively, and C is the number of channels of the feature map.Is f n Transposed weight matrix of b) n ∈R 1×1 Is f n Is included in the bias matrix of (a). Final output A n Is defined as:
A n ＝σ(f n (M))， (7)
wherein sigma (·) is a Sigmoid function, the only difference being the offset b in the fully connected layer n Also add to A n In the calculation of (2), the results obtained by the two are similar, but the application method of the scheme is simpler and more convenient to realize.
The plurality of focus positioning heads are respectively a GradCAM positioning head, an XGradCAM positioning head, a HiResCAM positioning head and a LayerCAM positioning head.
Because each CAM head provides a focus positioning suggestion for K' diseases of an input X-ray chest image, the focus positioning suggestion is called a focus positioning expert, and a plurality of focus positioning experts form a focus positioning expert group. The multi-expert fusion chest X-ray image auxiliary diagnosis model comprises two chest X-ray multi-head networks, so eight focus positioning experts are formed in total, and 4*8 focus positioning suggestions can be provided for one disease of which the X-ray chest image is input. Lesion localization accuracy on pulmonary atelectasis disease using XGradCAM was 5% higher than LayerCAM, whereas lesion localization accuracy on nodular disease using GradCAM was 3% higher than hirascam. Meanwhile, the focus positioning effect can be influenced by using different chest X-ray multi-head networks. For example, chest X-ray multi-headed networks trained using GAA strategy have been shown to be superior to those that were not trained using GAA strategy in terms of localization of pulmonary insufficiency, penetration, nodules, pneumonia, and pneumothorax diseases, but chest X-ray multi-headed networks trained using GAA strategy were significantly inferior to the latter in terms of cardiac hypertrophy, effusion, and mass. Therefore, in the multi-expert fusion chest X-ray image aided diagnosis model, the 8 focus positioning experts with different 'features' are combined into a focus positioning expert group, and the 'features' of each focus positioning expert can influence the weight of the focus positioning expert group in focus positioning results of target diseases. In this way, when the focus of the target disease is located, the results of the 8 focus locating experts are weighted and summed, so that a more accurate focus locating result can be obtained.
Each global classification head is capable of predicting the positive probability of a disease, and is therefore called a disease classification expert, and a plurality of disease classification experts form a disease classification expert group. The multi-expert fusion chest X-ray image auxiliary diagnosis model comprises two chest X-ray multi-head networks, so 28 disease classification experts are formed in total, and 2X 14 disease diagnosis suggestions can be provided for one input X-ray chest image.
The same training strategy will also lead to different results of the disease classification of the chest X-ray multi-head network, similar to the one mentioned above, which also has different "features" in the disease classification,
as shown in table 2. Therefore, in a multi-expert fusion chest radiography auxiliary diagnostic model, the two groups of disease classification experts with different 'features' are combined into one disease classification expert group, and the 'features' of each disease classification expert can influence the weight of each disease classification expert in the result of target disease positive prediction. In this way, when the target diseases are classified, the conclusions of the two groups of disease classification experts are weighted and summed, so that more accurate disease classification results can be obtained.
As shown in fig. 3, 12 cases of focus positioning generated by the multi-expert fusion chest X-ray image auxiliary diagnosis model are visualized, the images are divided into an upper group and a lower group, each group has six images, a yellow frame in each image is focus positioning suggestion output by the multi-expert fusion chest X-ray image auxiliary diagnosis model, and a red frame is focus positioning mark. The optimization iteration number of a Discrete Particle Swarm Optimization (DPSO) algorithm of the upper six images is set to be 1, and then a focus positioning suggestion frame obtained by fusing the chest X-ray image auxiliary diagnosis model by multiple experts tends to fill the whole focus area. The optimization iteration number of the discrete particle swarm optimization algorithm of the lower six images is set to be 100, and then a focus positioning suggestion frame obtained by fusing the chest X-ray image auxiliary diagnosis model by multiple experts tends to be converged into one point to appear in a focus area. This is caused by directly applying the IoR index to the objective function of the ion particle swarm optimization algorithm, because the IoR index only considers the ratio of the intersection area between the predicted frame and the real frame to the area of the predicted frame, and ignores the size of the predicted frame. Therefore, when the optimization iteration number of the discrete particle swarm optimization algorithm is set to be sufficiently large, the lesion localization suggestion box gradually shrinks to a point.
Fig. 3 is a diagram showing 12 cases of lesion localization expressions of the multi-expert fusion chest X-ray image aided diagnosis model, wherein a yellow frame is a lesion localization suggestion output by the multi-expert fusion chest X-ray image aided diagnosis model, and a red frame is a lesion localization label.
As shown in fig. 4, the lesion localization suggestions (right) output by 8 CAM heads in the multi-expert fusion chest X-ray image aided diagnosis model and the lesion localization suggestions (left) after final fusion are visualized. The right images are divided into upper and lower groups of four images each. The upper four images are generated by the chest X-ray multi-head network without using GAA, the lower four images are generated by the chest X-ray multi-head network with using GAA, and the focus positioning advice generated by the chest X-ray multi-head network with using GAA can be found to be closer to the heart lung region of the image by observing the upper image and the lower image, and the focus positioning advice generated by the chest X-ray multi-head network without using GAA is closer to the edge of the image, so that the GAA can be known to enable the model to learn the heart lung region of the image of interest. Furthermore, lesion localization suggestions derived from GradCAM and XGradCAM are more prone to cover the entire lesion area, while lesion localization suggestions derived from LayerCAM and HiResCAM are more prone to fine indication of the location of the lesion. Actually, the multi-expert fusion chest X-ray image auxiliary diagnosis model uses GradCAM and XGradCAM to estimate the area and rough position of the focus area, and then uses LayerCAM and HiResCAM to fine tune the previous positioning result, so that a more reasonable and accurate focus area positioning suggestion is obtained.
Fig. 4 shows lesion localization suggestions (right) and final fusion results (left) output by 8 CAM hades in a multi-expert fusion chest radiography aided diagnosis model. The yellow frame is a focus positioning suggestion after fusion output by the multi-expert fusion chest X-ray image auxiliary diagnosis model, the black frame is a focus positioning suggestion frame output by the CAM head, and the red frame is focus positioning mark. In addition, gradCAM and XGradCAM, layerCAM are visually similar to the HiResCAM derived lesion localization advice frame.
Referring to fig. 6-7, in a second aspect, a multi-expert fusion chest X-ray image aided diagnosis method is applied to the multi-expert fusion chest X-ray image aided diagnosis system of the first aspect, and includes the following steps:
s1, inputting chest X-ray images;
specifically, an X-chest X-ray image is input.
S2, extracting the chest X-ray image to obtain a chest region-of-interest image and an image cardiopulmonary mask;
specifically, the chest X-ray image is extracted through the extraction module, and the chest region-of-interest image and the image cardiopulmonary mask are obtained.
S3, calculating the chest region-of-interest image to obtain a first focus candidate region and a first disease positive probability;
specifically, the first chest X-ray multi-head network module 3 calculates the chest region of interest image to obtain a first focus candidate region and a first disease positive probability;
s4, calculating the chest region of interest image and the image heart-lung mask to obtain a second focus candidate region and a second disease positive probability;
specifically, the second chest X-ray multi-head network module 4 calculates the chest region of interest image and the image cardiopulmonary mask to obtain a second focus candidate region and a second disease positive probability.
S5, outputting a corresponding focus area based on the first focus candidate area and the second focus candidate area;
specifically, the focus positioning fusion module 5 outputs a corresponding focus area based on the first focus candidate area and the second focus candidate area.
S6, outputting disease data based on the second disease positive probability and the second disease positive probability.
Specifically, the disease classification fusion module 6 outputs disease data based on the second disease positive probability and the second disease positive probability.
The above disclosure is illustrative of the preferred embodiments of the present invention and is not to be construed as limiting the scope of the invention, but rather as providing for the full or partial flow of the solution to the above-described embodiments, and equivalent variations according to the appended claims, will be understood by those skilled in the art.

Claims (5)

1. A multi-expert fusion chest X-ray image auxiliary diagnosis system is characterized in that,
the system comprises an input module, an extraction module, a first chest X-ray multi-head network module, a second chest X-ray multi-head network module, a focus positioning fusion module and a disease classification fusion module;
the input module is used for inputting chest X-ray images;
the extraction module is used for extracting the chest X-ray image to obtain a chest region-of-interest image and an image cardiopulmonary mask;
the first chest X-ray multi-head network module is used for calculating the chest region-of-interest image to obtain a first focus candidate region and a first disease positive probability;
the second chest X-ray multi-head network module is used for calculating the chest region-of-interest image and the image heart-lung mask to obtain a second focus candidate region and a second disease positive probability;
the focus positioning fusion module outputs a corresponding focus area based on the first focus candidate area and the second focus candidate area;
and the disease classification fusion module outputs disease data based on the second disease positive probability and the second disease positive probability.
2. A multi-expert fusion chest radiography auxiliary diagnostic system as defined in claim 1 wherein,
the extraction module comprises a region extraction unit and a mask extraction unit;
the region extraction unit is used for extracting the chest X-ray image to obtain a chest region of interest image;
the mask extraction unit is used for extracting the chest X-ray image to obtain an image heart-lung mask.
3. A multi-expert fusion chest radiography auxiliary diagnostic system as defined in claim 2 wherein,
the first chest X-ray multi-head network module and the second chest X-ray multi-head network module each include a plurality of lesion localization heads and a disease classification head;
the focus positioning head is used for outputting the first focus candidate zone and the second focus candidate zone by a focus positioning expert;
the disease classification head is used for outputting the first disease positive probability and the second disease positive probability by a disease classification expert.
4. A multi-expert fusion chest radiography auxiliary diagnostic system as described in claim 3, wherein,
the plurality of focus positioning heads are respectively a GradCAM positioning head, an XGradCAM positioning head, a HiResCAM positioning head and a LayerCAM positioning head.
5. A multi-expert fusion chest X-ray image aided diagnosis method applied to the multi-expert fusion chest X-ray image aided diagnosis system of any one of claims 1-4, characterized by comprising the following steps:
inputting chest X-ray images;
extracting chest X-ray images to obtain chest region-of-interest images and image cardiopulmonary masks;
calculating the chest region of interest image to obtain a first focus candidate region and a first disease positive probability;
calculating the chest region of interest image and the image heart-lung mask to obtain a second focus candidate region and a second disease positive probability;
outputting a corresponding lesion area based on the first lesion candidate zone and the second lesion candidate zone;
disease data is output based on the second disease positive probability and the second disease positive probability.
CN202311494684.0A 2023-11-09 2023-11-09 Multi-expert fusion chest X-ray image auxiliary diagnosis system and method Pending CN117437207A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311494684.0A CN117437207A (en) 2023-11-09 2023-11-09 Multi-expert fusion chest X-ray image auxiliary diagnosis system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311494684.0A CN117437207A (en) 2023-11-09 2023-11-09 Multi-expert fusion chest X-ray image auxiliary diagnosis system and method

Publications (1)

Publication Number Publication Date
CN117437207A true CN117437207A (en) 2024-01-23

Family

ID=89547863

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311494684.0A Pending CN117437207A (en) 2023-11-09 2023-11-09 Multi-expert fusion chest X-ray image auxiliary diagnosis system and method

Country Status (1)

Country Link
CN (1) CN117437207A (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111429473A (en) * 2020-02-27 2020-07-17 西北大学 Chest film lung field segmentation model establishment and segmentation method based on multi-scale feature fusion
CN112784856A (en) * 2021-01-29 2021-05-11 长沙理工大学 Channel attention feature extraction method and identification method of chest X-ray image
CN113555089A (en) * 2021-07-14 2021-10-26 江苏宏创信息科技有限公司 Artificial intelligence medical image quality control method applied to clinical image
CN113643261A (en) * 2021-08-13 2021-11-12 江南大学 Lung disease diagnosis method based on frequency attention network
US20220036564A1 (en) * 2020-08-03 2022-02-03 Korea Advanced Institute Of Science And Technology Method of classifying lesion of chest x-ray radiograph based on data normalization and local patch and apparatus thereof
CN114242232A (en) * 2021-11-09 2022-03-25 北京深睿博联科技有限责任公司 Disease diagnosis device and method based on medical image
CN115187819A (en) * 2022-08-23 2022-10-14 北京医准智能科技有限公司 Training method and device for image classification model, electronic equipment and storage medium
CN115482230A (en) * 2022-09-27 2022-12-16 西北师范大学 Pulmonary tuberculosis assistant decision-making system based on deep convolution pulse neural network

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111429473A (en) * 2020-02-27 2020-07-17 西北大学 Chest film lung field segmentation model establishment and segmentation method based on multi-scale feature fusion
US20220036564A1 (en) * 2020-08-03 2022-02-03 Korea Advanced Institute Of Science And Technology Method of classifying lesion of chest x-ray radiograph based on data normalization and local patch and apparatus thereof
CN112784856A (en) * 2021-01-29 2021-05-11 长沙理工大学 Channel attention feature extraction method and identification method of chest X-ray image
CN113555089A (en) * 2021-07-14 2021-10-26 江苏宏创信息科技有限公司 Artificial intelligence medical image quality control method applied to clinical image
CN113643261A (en) * 2021-08-13 2021-11-12 江南大学 Lung disease diagnosis method based on frequency attention network
CN114242232A (en) * 2021-11-09 2022-03-25 北京深睿博联科技有限责任公司 Disease diagnosis device and method based on medical image
CN115187819A (en) * 2022-08-23 2022-10-14 北京医准智能科技有限公司 Training method and device for image classification model, electronic equipment and storage medium
CN115482230A (en) * 2022-09-27 2022-12-16 西北师范大学 Pulmonary tuberculosis assistant decision-making system based on deep convolution pulse neural network

Similar Documents

Publication Publication Date Title
Zhao et al. Rethinking dice loss for medical image segmentation
CN112150478B (en) Method and system for constructing semi-supervised image segmentation framework
CN110363138A (en) Model training method, image processing method, device, terminal and storage medium
CN111223553A (en) Two-stage deep migration learning traditional Chinese medicine tongue diagnosis model
Glaunes et al. Template estimation form unlabeled point set data and surfaces for computational anatomy
CN110895809A (en) Method for accurately extracting key points in hip joint image
CN104715251B (en) A kind of well-marked target detection method based on histogram linear fit
CN114897914B (en) Semi-supervised CT image segmentation method based on countermeasure training
CN111325750B (en) Medical image segmentation method based on multi-scale fusion U-shaped chain neural network
CN111429407A (en) Chest X-ray disease detection device and method based on two-channel separation network
CN108241854A (en) A kind of deep video conspicuousness detection method based on movement and recall info
CN105976364B (en) Based on the statistical average model building method for simplifying weighted-graph
CN109859184A (en) A kind of real-time detection of continuous scanning breast ultrasound image and Decision fusion method
CN113693563A (en) Brain function network classification method based on hypergraph attention network
Khakzar et al. Learning interpretable features via adversarially robust optimization
CN115345938A (en) Global-to-local-based head shadow mark point positioning method, equipment and medium
CN116934807A (en) Twin network single target tracking method and device
CN116468734A (en) Deep learning-based method for segmenting three categories of echocardiography
CN116091979A (en) Target tracking method based on feature fusion and channel attention
Yong et al. Automatic ventricular nuclear magnetic resonance image processing with deep learning
Ishkov et al. Processing and visualization of tomography signal with AI: Decision making on COVID-19
CN117437207A (en) Multi-expert fusion chest X-ray image auxiliary diagnosis system and method
CN117557804A (en) Multi-label classification method combining target structure embedding and multi-level feature fusion
CN111898756A (en) Multi-target information associated neural network loss function calculation method and device
Chen et al. Edge Enhanced GCIFFNet: A Multiclass Semantic Segmentation Network Based on Edge Enhancement and Multiscale Attention Mechanism

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination