CN113367644B - Digestive tract disease recognition system with device migration capability - Google Patents
Digestive tract disease recognition system with device migration capability Download PDFInfo
- Publication number
- CN113367644B CN113367644B CN202110917028.1A CN202110917028A CN113367644B CN 113367644 B CN113367644 B CN 113367644B CN 202110917028 A CN202110917028 A CN 202110917028A CN 113367644 B CN113367644 B CN 113367644B
- Authority
- CN
- China
- Prior art keywords
- digestive tract
- model
- data
- fusion
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 210000001035 gastrointestinal tract Anatomy 0.000 title claims abstract description 103
- 201000010099 disease Diseases 0.000 title claims abstract description 64
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 title claims abstract description 64
- 208000036829 Device dislocation Diseases 0.000 title claims abstract description 9
- 230000004927 fusion Effects 0.000 claims abstract description 74
- 238000001514 detection method Methods 0.000 claims abstract description 23
- 238000000034 method Methods 0.000 claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 11
- 238000006243 chemical reaction Methods 0.000 claims abstract description 6
- 238000012549 training Methods 0.000 claims description 46
- 239000013598 vector Substances 0.000 claims description 41
- 230000006870 function Effects 0.000 claims description 19
- 238000004364 calculation method Methods 0.000 claims description 8
- 238000011161 development Methods 0.000 claims description 6
- 238000013145 classification model Methods 0.000 claims description 3
- 238000013135 deep learning Methods 0.000 claims description 3
- 239000000463 material Substances 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 7
- 238000002372 labelling Methods 0.000 description 6
- 239000000523 sample Substances 0.000 description 6
- 206010028980 Neoplasm Diseases 0.000 description 5
- 201000011510 cancer Diseases 0.000 description 4
- 238000007499 fusion processing Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 230000002496 gastric effect Effects 0.000 description 3
- 201000005917 gastric ulcer Diseases 0.000 description 3
- 230000014509 gene expression Effects 0.000 description 3
- 230000003902 lesion Effects 0.000 description 3
- 238000003062 neural network model Methods 0.000 description 3
- 208000000624 Esophageal and Gastric Varices Diseases 0.000 description 2
- 208000007107 Stomach Ulcer Diseases 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 239000013604 expression vector Substances 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003211 malignant effect Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 208000023514 Barrett esophagus Diseases 0.000 description 1
- 208000023665 Barrett oesophagus Diseases 0.000 description 1
- 206010013833 Duodenal polyp Diseases 0.000 description 1
- 208000000461 Esophageal Neoplasms Diseases 0.000 description 1
- 206010061164 Gastric mucosal lesion Diseases 0.000 description 1
- 206010051012 Gastric varices Diseases 0.000 description 1
- 206010067551 Gastric xanthoma Diseases 0.000 description 1
- 208000007882 Gastritis Diseases 0.000 description 1
- 208000034991 Hiatal Hernia Diseases 0.000 description 1
- 206010020028 Hiatus hernia Diseases 0.000 description 1
- 206010030155 Oesophageal carcinoma Diseases 0.000 description 1
- 206010071192 Oesophageal papilloma Diseases 0.000 description 1
- 206010051935 Oesophageal polyp Diseases 0.000 description 1
- 206010030216 Oesophagitis Diseases 0.000 description 1
- 208000037062 Polyps Diseases 0.000 description 1
- 208000005718 Stomach Neoplasms Diseases 0.000 description 1
- 208000025865 Ulcer Diseases 0.000 description 1
- 206010056091 Varices oesophageal Diseases 0.000 description 1
- 235000010724 Wisteria floribunda Nutrition 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000001574 biopsy Methods 0.000 description 1
- 230000000747 cardiac effect Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000002405 diagnostic procedure Methods 0.000 description 1
- 230000002183 duodenal effect Effects 0.000 description 1
- 208000000718 duodenal ulcer Diseases 0.000 description 1
- 206010013864 duodenitis Diseases 0.000 description 1
- 238000013399 early diagnosis Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 201000004101 esophageal cancer Diseases 0.000 description 1
- 208000024170 esophageal varices Diseases 0.000 description 1
- 201000010120 esophageal varix Diseases 0.000 description 1
- 208000006881 esophagitis Diseases 0.000 description 1
- 210000003238 esophagus Anatomy 0.000 description 1
- 206010017758 gastric cancer Diseases 0.000 description 1
- 208000021302 gastroesophageal reflux disease Diseases 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 208000000689 peptic esophagitis Diseases 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 201000011549 stomach cancer Diseases 0.000 description 1
- 208000023984 stomach polyp Diseases 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 231100000397 ulcer Toxicity 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/273—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for the upper alimentary canal, e.g. oesophagoscopes, gastroscopes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10068—Endoscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30092—Stomach; Gastric
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Radiology & Medical Imaging (AREA)
- Surgery (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Medical Informatics (AREA)
- General Physics & Mathematics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Public Health (AREA)
- Molecular Biology (AREA)
- Heart & Thoracic Surgery (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Veterinary Medicine (AREA)
- Animal Behavior & Ethology (AREA)
- Pathology (AREA)
- Optics & Photonics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Gastroenterology & Hepatology (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Image Analysis (AREA)
- Endoscopes (AREA)
Abstract
The present application relates to a digestive tract disease recognition system with device migration capability, comprising: a client and a server; the client comprises a data conversion module, an equipment detection module, a basic identification module and a data display module; the server side comprises an equipment information management module, a data processing module and a fusion identification module; and the fusion identification module identifies digestive tract diseases of the frame picture data uploaded by the client by a model fusion identification method. The method and the system can enable the new digestive tract endoscope equipment to be rapidly migrated, and enable the system to have better prediction performance based on data acquired by the new equipment through the fusion recognition model.
Description
Technical Field
The application relates to a digestive tract disease recognition system with equipment migration capability, and belongs to the technical field of intelligent medical image processing.
Background
According to global cancer statistics, it was shown that 4 of the top 10 tumors in incidence rank from the digestive tract. Digestive tract diseases including benign, precancerous and malignant diseases of the digestive tract are seriously threatening the quality of life and life safety of patients, and causing huge health burden. The incidence of digestive tract tumors is also at the top in various malignant tumors in China. Since the endoscope for digestive tract is remarkably effective in diagnosing digestive tract cancer, it has been proposed as a main diagnostic method for digestive tract cancer. In particular, the digestive tract endoscope can directly probe the lesion tissue area in the digestive tract for medical personnel to make corresponding diagnosis, and tissue biopsy can be made under the digestive tract endoscope, so that the digestive tract endoscope has important functions on early diagnosis of the precancerous disease or precancerous lesion of the digestive tract and identification of benign and malignant ulcer.
Although the technology of recognizing the digestive tract diseases by using artificial intelligence image recognition is available at present, which can assist doctors to work and improve the diagnosis efficiency of doctors, the existing technology of recognizing the digestive tract diseases is usually based on a single network model, that is, a neural network model is trained by collecting some training data (large-scale picture data are generally collected from patient medical records in hospitals), and then the neural network model is applied to different digestive tract detection devices. However, the digestive tract detection equipment in the market is not uniform, and different digestive tract detection equipment has differences in display and detection due to differences of manufacturers and technical specifications. For example, there may be some differences in the resolution, color, etc. of videos from the olympus brand gastroscope system and the fuji brand gastroscope system. As shown in fig. 1, (a) (B) are two digestive tract pictures from different digestive tract detection devices, i.e., device a and device B, respectively, and the shapes and resolutions of the borders have obvious differences. For example: the picture of device a is square and the picture of device B is rectangular. Therefore, the existing digestive tract disease identification models have equipment dependence problems. In other words, when the existing model is applied to different digestive tract detection devices, the recognition accuracy of the model is lost due to the difference of the detection devices.
People expect to obtain a system, which can utilize an artificial intelligence image recognition technology to recognize diseases of the digestive tract, can be quickly transferred to different digestive tract detection equipment, and has good adaptability and higher prediction accuracy.
Disclosure of Invention
In order to solve the above technical problem, the present application provides a digestive tract disease recognition system with device migration capability, comprising: a client and a server;
the client comprises a data conversion module, an equipment detection module, a basic identification module and a data display module;
the data conversion module receives and converts video data output by the digestive tract endoscope equipment into a picture format to obtain frame picture data;
the device detection module detects whether the digestive tract endoscope device is a new digestive tract endoscope device, transmits the frame picture data to a server end when the digestive tract endoscope device is the new device, and transmits the frame picture data to a basic identification module when the digestive tract endoscope device is an existing device;
the basic identification module identifies the frame picture data transmitted by the equipment detection module to obtain a digestive tract disease identification result;
the data display module receives the digestive tract disease recognition result and displays the digestive tract disease recognition result through display equipment;
the server side comprises an equipment information management module, a data processing module and a fusion identification module;
the device information management module manages information of the digestive tract endoscope device;
the data processing module receives, marks and stores frame picture data uploaded by a client;
and the fusion identification module identifies the frame picture data uploaded by the client by a model fusion identification method to obtain a digestive tract disease identification result and transmits the digestive tract disease identification result to the client.
Optionally, the device information management module stores a device number sequence of an existing endoscopic device for the digestive tract, and adds a device number of a new endoscopic device for the digestive tract to the device number sequence after the frame picture data of the new endoscopic device for the digestive tract is labeled.
Optionally, the data processing module stores basic training data and new device training data, the basic training data is the labeled frame picture data of the existing multiple gastrointestinal endoscope devices, and the new device training data is the labeled frame picture data of the new gastrointestinal endoscope device uploaded by the client.
Optionally, the model fusion identification method includes:
constructing a basic model, a new equipment model and a tensor-based hierarchical fusion network model, wherein the basic model and the new equipment model are deep learning image classification models, and the tensor-based hierarchical fusion network model consists of a first layer intermediate representation tensor fusion network layer, a first layer result representation tensor fusion network layer, a second layer tensor fusion network layer connected with the first layer intermediate representation tensor fusion network layer and the first layer result representation tensor fusion network layer, a full-connection network connected with the second layer tensor fusion network layer and a classification network;
training a basic model by using the basic training data, and training a new equipment model by using the new equipment training data to obtain a trained basic model and a trained new equipment model;
inputting the base training data into the trained base model to obtain an intermediate representation vector denoted B1 and a resultant representation vector denoted B2, and simultaneously inputting the new plant training data into the trained new plant model to obtain an intermediate representation vector denoted E1 and a resultant representation vector denoted E2;
and inputting the intermediate representation vectors B1 and E1 and the result representation vectors B2 and E2 into the tensor-based hierarchical fusion network model and training by using a preset loss function to obtain a trained fusion recognition model, and performing digestive tract disease recognition on input frame picture data by using the trained fusion recognition model to obtain a recognition result.
Optionally, wherein the output of the fused recognition model isWhereinIn the general category of diseases,is as followsAnd (4) outputting the prediction of each category.
Optionally, wherein the first layer intermediate representation tensor fusion network layer fuses two intermediate representation vectors and obtains the dimension ofNIs/are as followsM1 vector:
Wherein,as a function of the fully-connected network,for outer product calculations, B1 and E1 are intermediate representation vectors.
Optionally, the first layer of resultant expression tensors fuses the two resultant expression vectors, and obtains the dimensionality ofNIs/are as followsM2Vector quantity:
wherein,as a function of the fully-connected network,for outer product calculations, B2 and E2 represent vectors for the results.
Optionally, wherein the second layer tensor fusion network layer fuses the results of the first layer outputM1AndM2and through a fully connected network, obtaining a dimension ofNIs/are as followsRVector quantity:
wherein,as a function of the fully-connected network,for the outer product calculation, M1 is the output of the first layer intermediate representation tensor fusion network, and M2 is the output of the first layer result representation tensor fusion network.
Optionally, wherein the classification network issoftmaxA layer of a material selected from the group consisting of,
wherein,is as followsThe probability values of the individual categories are,in the general category of diseases,is as followsThe predicted output of the individual categories is,for the predicted output of the c-th class,is the base of the natural logarithmic function.
Optionally, a gradient descent algorithm may be used in training the tensor-based hierarchical fusion network model.
Optionally, wherein the preset loss function is:
wherein,Nfor the number of development set samples that are input,the number of the total disease categories is,for the prediction output of all the input samples,is as followsnA first sample ofcThe value of the tag for each of the categories,is as followsnA first sample ofcAnd (4) outputting the prediction of each category.
Optionally, the identification model used by the basic identification module in the client side is consistent with the basic model in the server side.
The beneficial effects of this application include at least: the system comprises a client and a server; the client side package data conversion module, the equipment detection module, the data uploading module, the basic identification module and the data display module are used for data acquisition and transmission of new equipment and display of digestive tract disease identification results; the server side comprises an equipment information management module, a data processing module and a fusion recognition module, and is used for training a basic model, a new equipment model and a fusion recognition model of equipment information management, data annotation and digestive tract disease recognition, and performing digestive tract disease recognition through the fusion recognition model. The method and the system can enable new digestive tract endoscope equipment to be rapidly migrated to the recognition system, and enable the system to have better prediction performance based on data collected by the new equipment through fusion of recognition models.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention.
Drawings
The present application may be better understood by describing exemplary embodiments thereof in conjunction with the following drawings, wherein:
FIG. 1 is a schematic diagram of the differences between different digestive tract detection devices;
FIG. 2 is a system architecture diagram provided in accordance with one embodiment of the present application;
FIG. 3 is a schematic illustration of disease information annotation provided in accordance with an embodiment of the present application;
FIG. 4 is a schematic diagram of a base model provided by an embodiment of the present application;
FIG. 5 is a schematic view of a fusion model provided in an embodiment of the present application;
FIG. 6 is a schematic illustration of digestive tract disease identification provided by an embodiment of the present application;
FIG. 7 is a graph showing recognition results provided by one embodiment of the present application;
fig. 8 is a flowchart of a fusion recognition method according to an embodiment of the present application.
Detailed Description
The following detailed description of the embodiments of the present application, taken in conjunction with the accompanying drawings and examples, will enable those skilled in the art to practice the embodiments of the present application with reference to the description.
It is noted that in the detailed description of these embodiments, in order to provide a concise description, all features of an actual implementation may not be described in detail. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions are made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
As shown in fig. 2, the digestive tract disease identification system with device migration capability provided by the present invention includes a client and a server.
The client comprises:
and the data acquisition module is used for receiving the video data from the digestive tract endoscope equipment, processing and acquiring the video data and acquiring frame picture data. Specifically, since the identification model cannot directly input the original video, the video data is further extracted by using a certain time interval, and the specific time interval is not limited in this embodiment. For example, a segment of 5-minute raw inspection video is extracted at 200ms intervals, and a total of 1500 video frame pictures are obtained.
And the new equipment detection module is used for detecting whether the current digestive tract endoscope equipment is new equipment or not, and specifically, the new equipment detection module judges whether the current digestive tract endoscope equipment is new equipment or not according to the equipment serial number and the frame picture characteristics. And when the digestive tract endoscope equipment is new equipment, uploading the frame picture data to a server end, and when the digestive tract endoscope equipment is existing equipment, transmitting the frame picture to the basic identification module.
And the basic identification module is used for identifying the frame picture data transmitted by the equipment detection module. And if the current equipment is not the new equipment, directly calling the model to identify the video frame picture and returning an identification result. Optionally, the identification model used by the basic identification module in the client is consistent with the basic model in the server.
And the data display module is used for displaying the server-side digestive tract disease identification result and the client-side digestive tract disease identification result to a user and comprises a data organization unit and a display unit. The data organizing unit generates a recognition result map as shown in fig. 7, in which the left part is a currently recognized endoscopic picture of the digestive tract and the right part is a recognition result, i.e., a probability value of each disease category. The display unit is responsible for rendering the display result generated by the data organization unit, assists the work of a doctor and improves the diagnosis efficiency of the doctor.
The server side includes:
and the equipment information management module is used for managing the equipment information and the data uploading log information.
A digestive tract disease recognition system having a device migration capability needs to cope with a variety of scenes of different digestive tract detection devices, and thus needs to efficiently manage different device information. In this embodiment, the device information management module includes a device information management unit and a new device access log unit. The device information management unit stores the device number sequence of the existing data source, and adds the new device number to the managed device number sequence when the new device data is labeled. The new device access log unit is responsible for recording access information of the devices, including access time and access times.
And the data processing module receives the new digestive tract endoscope equipment data, the labeling data and the storage data from the client.
Digestive tract disease identification systems with device migration capabilities require storage of large amounts of application data, including new device data, annotation data, and save data. Specifically, the new device data is frame picture data after the detection device is preprocessed, and the labeled data is frame picture data subjected to disease category information labeling and used for model training.
The marked data are of two types, wherein one type is marked data pictures of a plurality of existing offline devices and is used for training a basic model for identifying the digestive tract diseases, and the other type is marked frame pictures acquired by a new device and is used for training a new device model for identifying the digestive tract diseases.
Wherein the disease category includes but is not limited to at least one of the following 22: gastric ulcer, gastric polyp, gastritis, gastric cancer, gastric submucosal mass, gastric varices, acute gastric mucosal lesions, gastric xanthoma, cardiac polyp, pyloric ulcer, duodenal ulcer, duodenitis, duodenal polyp, duodenal submucosal mass, reflux esophagitis, Barrett's esophagus, mycotic esophagitis, benign protrusion type lesions of esophagus (esophageal polyp, esophageal papilloma), esophageal varices, hiatal hernia, esophageal cancer, esophageal submucosal mass. For example, referring to fig. 3, a schematic view of disease information labeling provided by an embodiment of the present application, the disease information in this picture is labeled as gastric ulcer.
The method comprises the following steps that original frame picture data acquired by different new equipment and pictures in the label data of a plurality of existing offline equipment have differences, including pixel point differences and shape inconsistencies, and in order to obtain a better training effect, the preprocessing operation comprises the steps of enabling the new equipment frame picture and the pictures in the label data of the plurality of existing offline equipment to be consistent in pixel through image zooming operation; and through image cutting operation, the shape of the frame picture of the new equipment is consistent with the shape of the picture in the existing offline labeling data of a plurality of pieces of equipment.
Alternatively, the data access speed can be increased by adding a storage and calculation unit of the data processing module, and the storage space is expanded, so that the digestive tract disease recognition system with the device migration capability can cope with a large number of different digestive tract detection devices.
And the fusion identification module is used for identifying digestive tract diseases of the frame picture data uploaded by the client by a model fusion identification method to obtain an identification result.
Referring to fig. 8, the fusion recognition method in this embodiment at least includes the following steps:
and 810, constructing a basic model, a new equipment model and a tensor hierarchy-based fusion network model, solving the problem that a single identification model cannot be well transferred to different digestive tract endoscope equipment by using the models, and obtaining better prediction performance on new equipment.
The basic model and the new equipment model adopt a deep learning image classification model. For example, fig. 4 is a schematic diagram of a basic model provided in an embodiment of the present application. The network structure is composed of 3 parts, namely a feature extraction network, a capture relation network and a category prediction network. The original digestive tract picture is firstly input into a feature extraction network to extract picture features, wherein the feature extraction network is composed of 12 layers of transform structures, and a 768-dimensional picture feature vector is finally obtained. Then, the picture features are copied intoCParts of, whereinCAs the number of categories, willCThe picture features are input into a Transformer structure, and an attention mechanism inside the Transformer can automatically learn the dependency relationship among different categories and output the category features (intermediate representation) of each category. Category predictive network sharingCEach disease category corresponds to a dedicated prediction network and is responsible for decoding corresponding category characteristics (intermediate representation), and the category prediction network receives the category characteristics (intermediate representation) as input and outputs probability values (result representation) belonging to the corresponding disease categories through decoding.
The fusion recognition model adopts a neural network model based on tensor hierarchical fusion, and is a schematic diagram of the fusion recognition model provided by an embodiment of the application with reference to fig. 5.
And step 820, training a basic model by using the basic training data and training a new equipment model by using the new equipment training data to obtain a trained basic model and a trained new equipment model.
Training data used by a basic model for identifying digestive tract diseases are the existing offline labeling data of a plurality of devices; the training data used by the new equipment model for digestive tract disease recognition is the labeling data of the new digestive tract endoscope equipment uploaded by the client.
Alternatively, a gradient descent algorithm may be used to train the model, using the cross entropy as a loss function during the model training process, and the formula is as follows:
wherein,Nfor the number of development set samples that are input,the number of the total disease categories is,for the prediction output of all the input samples,is as followsnA first sample ofcThe value of the tag for each of the categories,is as followsnA first sample ofcAnd (4) outputting the prediction of each category.
Optionally, in the training process, the batch size is set to 8, the initial learning rate is set to 0.00001, and an Adam optimizer is adopted, in other embodiments, corresponding hyper-parameters may also be different during model training, and the batch size and the initial learning rate may also be other values, which does not limit values of each parameter in the training process. After training, the basic model has good recognition accuracy and can be used for carrying out disease classification on the input digestive tract picture.
Training data used by the fusion recognition model for digestive tract disease recognition is a small development set in new equipment marking data uploaded by a client. The input of the fusion recognition model is intermediate representation of the sample in the basic model and the new equipment model respectivelyB1AndE1and a result representation vector (i.e., a probability vector belonging to each class), respectively represented asB2AndE2。
and 840, inputting the intermediate representation vectors B1 and E1 and the result representation vectors B2 and E2 into a tensor-level-based fusion network model and training by using a preset loss function to obtain a trained fusion recognition model, and performing digestive tract disease recognition on the input frame picture data by using the trained fusion recognition model to obtain a recognition result.
Referring to fig. 5, which is a schematic diagram of a fusion recognition model provided in an embodiment of the present application, a first layer intermediate representation tensor fusion network layer fuses two intermediate representation vectorsB1AndE1and get the dimension of through the fully connected networkNIs/are as followsM1And (5) vector quantity. First layer result expression tensor fusion network layer fusion two result expression vectorsB2AndE2and get the dimension of through the fully connected networkNIs/are as followsM2And (5) vector quantity. Then, the second layer tensor fusion network layer fuses the result output by the first layerM1AndM2and through a fully connected network, obtaining a dimension ofNIs/are as followsRAnd (5) vector quantity. Finally, the network is fully connected andsoftmaxclassified network receptionRThe vector is used as input, and the final prediction result is output.
Wherein,is as followsThe probability values of the individual categories are,in the general category of diseases,is as followsThe predicted output of the individual categories is,for the predicted output of the c-th class,is the base of the natural logarithmic function.
Wherein, the specific fusion process of the tensor fusion network layer can be expressed by the following formula,
wherein, the formula (1) corresponds to the middle of the first layer to represent the fusion process of the tensor fusion network layer, the formula (2) corresponds to the result of the first layer to represent the fusion process of the tensor fusion network layer, the formula (3) corresponds to the fusion process of the tensor fusion network layer of the second layer,as a function of the fully-connected network,the outer product calculation is represented as a function of,B1ande1 isThe intermediate representation vector, B2 and E2 are the resulting representation vectors, M1andM2and outputting the result of the first layer network.
Alternatively, the fused recognition model may employ the same training mode and loss function as the base model.
Referring to fig. 6, a schematic diagram of digestive tract disease recognition according to an embodiment of the present application is provided, first, a digestive tract image is respectively input into a basic model and an equipment model, and an intermediate representation of the basic model is obtainedB1And the results showB2And intermediate representation of the plant modelE1And the results showE2. Then, willB1、B2、E1、E2And inputting a fusion recognition model, wherein the fusion recognition model can fully combine the potential characteristics of different expressions to give a recognition result of the digestive tract diseases.
Optionally, the fused recognition model may be tested using test data, including: using the prediction results, MAP index and F1 index are calculated, MAP is Average Precision Mean (Mean Average Precision), F1 is equal weight and Average of Precision and recall (F1-Score) for evaluating the recognition Precision of the model.
The results of comparison of the basic digestive tract disease recognition model, the digestive tract recognition model of the new device and the fused digestive tract disease recognition model at the two indexes of MAP and F1 are referred to as the table one below. As can be seen from the table I, the fused recognition model has further improved accuracy (the improvement range is about 5%) compared with the basic recognition model. Therefore, the problem that a single recognition model is difficult to apply to different digestive tract endoscope devices is effectively solved, and different digestive tract endoscope devices can be better adapted by fusing the recognition models of new devices.
Table one:
the basic principles of the present application have been described in connection with specific embodiments, but it should be noted that, for those skilled in the art, it can be understood that all or any of the steps or components of the method and apparatus of the present application can be implemented in hardware, firmware, software or their combination in any computing device (including processors, storage media, etc.) or network of computing devices, which can be implemented by those skilled in the art using their basic programming skills after reading the description of the present application.
The object of the present application can thus also be achieved by running a program or a set of programs on any computing device. The computing device may be a general purpose device as is well known. The object of the application can thus also be achieved merely by providing a program product comprising program code for implementing the method or the apparatus. That is, such a program product also constitutes the present application, and a storage medium storing such a program product also constitutes the present application. It is to be understood that the storage medium may be any known storage medium or any storage medium developed in the future.
It is further noted that in the apparatus and method of the present application, it is apparent that the components or steps may be disassembled and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application. Also, the steps of executing the series of processes described above may naturally be executed chronologically in the order described, but need not necessarily be executed chronologically. Some steps may be performed in parallel or independently of each other.
Unless otherwise defined, technical or scientific terms used in the claims and the specification should have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. The use of "first," "second," and similar terms in the description and claims of this patent application do not denote any order, quantity, or importance, but rather the terms are used to distinguish one element from another. The terms "a" or "an," and the like, do not denote a limitation of quantity, but rather denote the presence of at least one. The word "comprise" or "comprises", and the like, means that the element or item listed before "comprises" or "comprising" covers the element or item listed after "comprising" or "comprises" and its equivalent, and does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, nor are they restricted to direct or indirect connections.
The above-described embodiments should not be construed as limiting the scope of the present application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A digestive tract disease identification system with device migration capabilities, comprising: a client and a server;
the client comprises a data conversion module, an equipment detection module, a basic identification module and a data display module;
the data conversion module receives and converts video data output by the digestive tract endoscope equipment into a picture format to obtain frame picture data;
the device detection module detects whether the digestive tract endoscope device is a new digestive tract endoscope device, transmits the frame picture data to a server end when the digestive tract endoscope device is the new device, and transmits the frame picture data to a basic identification module when the digestive tract endoscope device is an existing device;
the basic identification module identifies the frame picture data transmitted by the equipment detection module to obtain a digestive tract disease identification result;
the data display module receives the digestive tract disease recognition result and displays the digestive tract disease recognition result through display equipment;
the server side comprises an equipment information management module, a data processing module and a fusion identification module;
the device information management module manages information of the digestive tract endoscope device;
the data processing module receives, marks and stores frame picture data uploaded by a client, and stores basic training data and new equipment training data, wherein the basic training data are the marked frame picture data of a plurality of existing digestive tract endoscope equipment, and the new equipment training data are the marked frame picture data of the new digestive tract endoscope equipment uploaded by the client;
the fusion recognition module recognizes the frame picture data uploaded by the client by a model fusion recognition method to obtain a digestive tract disease recognition result and transmits the digestive tract disease recognition result to the client,
the model fusion identification method comprises the following steps: constructing a basic model, a new equipment model and a tensor-based hierarchical fusion network model, wherein the basic model and the new equipment model are deep learning image classification models, and the tensor-based hierarchical fusion network model consists of a first layer intermediate representation tensor fusion network layer, a first layer result representation tensor fusion network layer, a second layer tensor fusion network layer connected with the first layer intermediate representation tensor fusion network layer and the first layer result representation tensor fusion network layer, a full-connection network connected with the second layer tensor fusion network layer and a classification network; training a basic model by using the basic training data, and training a new equipment model by using the new equipment training data to obtain a trained basic model and a trained new equipment model;
inputting the base training data into the trained base model to obtain an intermediate representation vector denoted B1 and a resultant representation vector denoted B2, and simultaneously inputting the new plant training data into the trained new plant model to obtain an intermediate representation vector denoted E1 and a resultant representation vector denoted E2; and inputting the intermediate representation vectors B1 and E1 and the result representation vectors B2 and E2 into the tensor-based hierarchical fusion network model and training by using a preset loss function to obtain a trained fusion recognition model, and performing digestive tract disease recognition on input frame picture data by using the trained fusion recognition model to obtain a recognition result.
2. The system according to claim 1, wherein the device information management module stores a device number sequence of an existing endoscopic device for the digestive tract, and adds a device number of a new endoscopic device for the digestive tract to the device number sequence when tagging of frame picture data of the new endoscopic device for the digestive tract is completed.
4. The system of claim 1, wherein the first layer intermediate representation tensor fusion meshThe network layer fuses the two intermediate representation vectors and obtains the dimension ofNIs/are as followsM1 vector:
5. The system of claim 1, wherein the first layer resultant tensor fusion network layer fuses two resultant representation vectors and obtains a dimension of through a fully connected network ofNIs/are as followsM2Vector quantity:
6. The system of claim 1, wherein the second layer tensor fusion network layer fuses the results of the first layer outputM1AndM2and through a fully connected network, obtaining a dimension ofNIs/are as followsRVector quantity:
7. The system of claim 1, wherein the classification network issoftmaxA layer of a material selected from the group consisting of,
8. The system of claim 1, wherein a gradient descent algorithm may be employed in training the tensor-based hierarchical fusion network model.
9. The system of claim 1, wherein the preset loss function is:
wherein,Nfor the number of development set samples that are input,the number of the total disease categories is,for the prediction output of all the input samples,is as followsnA first sample ofcThe value of the tag for each of the categories,is as followsnA first sample ofcAnd (4) outputting the prediction of each category.
10. The system of claim 1, wherein the base identification module in the client uses an identification model that is consistent with the base model at the server.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110917028.1A CN113367644B (en) | 2021-08-11 | 2021-08-11 | Digestive tract disease recognition system with device migration capability |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110917028.1A CN113367644B (en) | 2021-08-11 | 2021-08-11 | Digestive tract disease recognition system with device migration capability |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113367644A CN113367644A (en) | 2021-09-10 |
CN113367644B true CN113367644B (en) | 2022-04-15 |
Family
ID=77576713
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110917028.1A Active CN113367644B (en) | 2021-08-11 | 2021-08-11 | Digestive tract disease recognition system with device migration capability |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113367644B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113822389B (en) * | 2021-11-24 | 2022-02-22 | 紫东信息科技(苏州)有限公司 | Digestive tract disease classification system based on endoscope picture |
CN116612340B (en) * | 2023-07-21 | 2023-10-13 | 紫东信息科技(苏州)有限公司 | Peptic ulcer part classification method, device and readable storage medium |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1642467A (en) * | 2002-03-14 | 2005-07-20 | 奥林巴斯株式会社 | Endoscope image processing apparatus |
CN107622485A (en) * | 2017-08-15 | 2018-01-23 | 中国科学院深圳先进技术研究院 | A kind of medical image data analysis method and system for merging depth tensor neutral net |
CN108366788A (en) * | 2015-11-30 | 2018-08-03 | 任旭彬 | Diagnostic system and diagnosis management method whether abnormal using the cell of DNN study |
CN109614487A (en) * | 2018-10-29 | 2019-04-12 | 山东大学 | A method of the emotional semantic classification based on tensor amalgamation mode |
CN112965381A (en) * | 2021-02-09 | 2021-06-15 | 重庆高开清芯智联网络科技有限公司 | Method for establishing cooperative intelligent self-adaptive decision model |
CN113139937A (en) * | 2021-04-13 | 2021-07-20 | 上海市第一人民医院 | Digestive tract endoscope video image identification method based on deep learning |
-
2021
- 2021-08-11 CN CN202110917028.1A patent/CN113367644B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN1642467A (en) * | 2002-03-14 | 2005-07-20 | 奥林巴斯株式会社 | Endoscope image processing apparatus |
CN108366788A (en) * | 2015-11-30 | 2018-08-03 | 任旭彬 | Diagnostic system and diagnosis management method whether abnormal using the cell of DNN study |
CN107622485A (en) * | 2017-08-15 | 2018-01-23 | 中国科学院深圳先进技术研究院 | A kind of medical image data analysis method and system for merging depth tensor neutral net |
CN109614487A (en) * | 2018-10-29 | 2019-04-12 | 山东大学 | A method of the emotional semantic classification based on tensor amalgamation mode |
CN112965381A (en) * | 2021-02-09 | 2021-06-15 | 重庆高开清芯智联网络科技有限公司 | Method for establishing cooperative intelligent self-adaptive decision model |
CN113139937A (en) * | 2021-04-13 | 2021-07-20 | 上海市第一人民医院 | Digestive tract endoscope video image identification method based on deep learning |
Also Published As
Publication number | Publication date |
---|---|
CN113367644A (en) | 2021-09-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113367644B (en) | Digestive tract disease recognition system with device migration capability | |
Mehmood et al. | Video summarization based tele-endoscopy: a service to efficiently manage visual data generated during wireless capsule endoscopy procedure | |
CN113496489B (en) | Training method of endoscope image classification model, image classification method and device | |
CN109997147A (en) | System and method for diagnosing the otoscope image analysis of ear's pathology | |
Ghosh et al. | Effective deep learning for semantic segmentation based bleeding zone detection in capsule endoscopy images | |
Ghosh et al. | An automatic bleeding detection scheme in wireless capsule endoscopy based on histogram of an RGB-indexed image | |
CN114782760B (en) | Stomach disease picture classification system based on multitask learning | |
Biniaz et al. | Automatic reduction of wireless capsule endoscopy reviewing time based on factorization analysis | |
Sadasivan et al. | High accuracy patch-level classification of wireless capsule endoscopy images using a convolutional neural network | |
Lazăr et al. | The impact of artificial intelligence in the endoscopic assessment of premalignant and malignant esophageal lesions: present and future | |
CN114399634B (en) | Three-dimensional image classification method, system, equipment and medium based on weak supervision learning | |
CN114399465A (en) | Benign and malignant ulcer identification method and system | |
Du et al. | Improving the classification performance of esophageal disease on small dataset by semi-supervised efficient contrastive learning | |
CN116664929A (en) | Laryngoscope image multi-attribute classification method based on multi-modal information fusion | |
Bhosale et al. | Graph and capsule convolutional neural network based classification of lung Cancer, pneumonia, COVID-19 using lung CT and ultrasound radiography imaging | |
Kim et al. | Fostering transparent medical image AI via an image-text foundation model grounded in medical literature | |
Henry | Machine learning approaches for early diagnosis of thyroid cancer | |
Kim et al. | Transparent medical image AI via an image–text foundation model grounded in medical literature | |
Nien et al. | Criss-cross attention based multi-level fusion network for gastric intestinal metaplasia segmentation | |
Ye et al. | See what you see: Self-supervised cross-modal retrieval of visual stimuli from brain activity | |
Rani et al. | Classification of wireless capsule endoscopy images for bleeding using deep features fusion | |
Yang et al. | Identification of upper GI diseases during screening gastroscopy using a deep convolutional neural network algorithm | |
CN113313177A (en) | Digestive tract endoscope picture multi-label classification system | |
Moalla et al. | Exploiting Pre-trained Architectures for Dual-Stream Classification of LCA-RCA in a Private AngioData | |
Fisher et al. | Colour image analysis of wireless capsule endoscopy video: A review |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |