CN113367644B - Digestive tract disease recognition system with device migration capability - Google Patents

Digestive tract disease recognition system with device migration capability Download PDF

Info

Publication number
CN113367644B
CN113367644B CN202110917028.1A CN202110917028A CN113367644B CN 113367644 B CN113367644 B CN 113367644B CN 202110917028 A CN202110917028 A CN 202110917028A CN 113367644 B CN113367644 B CN 113367644B
Authority
CN
China
Prior art keywords
digestive tract
model
data
fusion
layer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110917028.1A
Other languages
Chinese (zh)
Other versions
CN113367644A (en
Inventor
戴捷
李亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zidong Information Technology Suzhou Co ltd
Original Assignee
Zidong Information Technology Suzhou Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zidong Information Technology Suzhou Co ltd filed Critical Zidong Information Technology Suzhou Co ltd
Priority to CN202110917028.1A priority Critical patent/CN113367644B/en
Publication of CN113367644A publication Critical patent/CN113367644A/en
Application granted granted Critical
Publication of CN113367644B publication Critical patent/CN113367644B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/273—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for the upper alimentary canal, e.g. oesophagoscopes, gastroscopes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30092—Stomach; Gastric

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Radiology & Medical Imaging (AREA)
  • Surgery (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Medical Informatics (AREA)
  • General Physics & Mathematics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Public Health (AREA)
  • Molecular Biology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Veterinary Medicine (AREA)
  • Animal Behavior & Ethology (AREA)
  • Pathology (AREA)
  • Optics & Photonics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Quality & Reliability (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Gastroenterology & Hepatology (AREA)
  • Epidemiology (AREA)
  • Primary Health Care (AREA)
  • Image Analysis (AREA)
  • Endoscopes (AREA)

Abstract

The present application relates to a digestive tract disease recognition system with device migration capability, comprising: a client and a server; the client comprises a data conversion module, an equipment detection module, a basic identification module and a data display module; the server side comprises an equipment information management module, a data processing module and a fusion identification module; and the fusion identification module identifies digestive tract diseases of the frame picture data uploaded by the client by a model fusion identification method. The method and the system can enable the new digestive tract endoscope equipment to be rapidly migrated, and enable the system to have better prediction performance based on data acquired by the new equipment through the fusion recognition model.

Description

Digestive tract disease recognition system with device migration capability
Technical Field
The application relates to a digestive tract disease recognition system with equipment migration capability, and belongs to the technical field of intelligent medical image processing.
Background
According to global cancer statistics, it was shown that 4 of the top 10 tumors in incidence rank from the digestive tract. Digestive tract diseases including benign, precancerous and malignant diseases of the digestive tract are seriously threatening the quality of life and life safety of patients, and causing huge health burden. The incidence of digestive tract tumors is also at the top in various malignant tumors in China. Since the endoscope for digestive tract is remarkably effective in diagnosing digestive tract cancer, it has been proposed as a main diagnostic method for digestive tract cancer. In particular, the digestive tract endoscope can directly probe the lesion tissue area in the digestive tract for medical personnel to make corresponding diagnosis, and tissue biopsy can be made under the digestive tract endoscope, so that the digestive tract endoscope has important functions on early diagnosis of the precancerous disease or precancerous lesion of the digestive tract and identification of benign and malignant ulcer.
Although the technology of recognizing the digestive tract diseases by using artificial intelligence image recognition is available at present, which can assist doctors to work and improve the diagnosis efficiency of doctors, the existing technology of recognizing the digestive tract diseases is usually based on a single network model, that is, a neural network model is trained by collecting some training data (large-scale picture data are generally collected from patient medical records in hospitals), and then the neural network model is applied to different digestive tract detection devices. However, the digestive tract detection equipment in the market is not uniform, and different digestive tract detection equipment has differences in display and detection due to differences of manufacturers and technical specifications. For example, there may be some differences in the resolution, color, etc. of videos from the olympus brand gastroscope system and the fuji brand gastroscope system. As shown in fig. 1, (a) (B) are two digestive tract pictures from different digestive tract detection devices, i.e., device a and device B, respectively, and the shapes and resolutions of the borders have obvious differences. For example: the picture of device a is square and the picture of device B is rectangular. Therefore, the existing digestive tract disease identification models have equipment dependence problems. In other words, when the existing model is applied to different digestive tract detection devices, the recognition accuracy of the model is lost due to the difference of the detection devices.
People expect to obtain a system, which can utilize an artificial intelligence image recognition technology to recognize diseases of the digestive tract, can be quickly transferred to different digestive tract detection equipment, and has good adaptability and higher prediction accuracy.
Disclosure of Invention
In order to solve the above technical problem, the present application provides a digestive tract disease recognition system with device migration capability, comprising: a client and a server;
the client comprises a data conversion module, an equipment detection module, a basic identification module and a data display module;
the data conversion module receives and converts video data output by the digestive tract endoscope equipment into a picture format to obtain frame picture data;
the device detection module detects whether the digestive tract endoscope device is a new digestive tract endoscope device, transmits the frame picture data to a server end when the digestive tract endoscope device is the new device, and transmits the frame picture data to a basic identification module when the digestive tract endoscope device is an existing device;
the basic identification module identifies the frame picture data transmitted by the equipment detection module to obtain a digestive tract disease identification result;
the data display module receives the digestive tract disease recognition result and displays the digestive tract disease recognition result through display equipment;
the server side comprises an equipment information management module, a data processing module and a fusion identification module;
the device information management module manages information of the digestive tract endoscope device;
the data processing module receives, marks and stores frame picture data uploaded by a client;
and the fusion identification module identifies the frame picture data uploaded by the client by a model fusion identification method to obtain a digestive tract disease identification result and transmits the digestive tract disease identification result to the client.
Optionally, the device information management module stores a device number sequence of an existing endoscopic device for the digestive tract, and adds a device number of a new endoscopic device for the digestive tract to the device number sequence after the frame picture data of the new endoscopic device for the digestive tract is labeled.
Optionally, the data processing module stores basic training data and new device training data, the basic training data is the labeled frame picture data of the existing multiple gastrointestinal endoscope devices, and the new device training data is the labeled frame picture data of the new gastrointestinal endoscope device uploaded by the client.
Optionally, the model fusion identification method includes:
constructing a basic model, a new equipment model and a tensor-based hierarchical fusion network model, wherein the basic model and the new equipment model are deep learning image classification models, and the tensor-based hierarchical fusion network model consists of a first layer intermediate representation tensor fusion network layer, a first layer result representation tensor fusion network layer, a second layer tensor fusion network layer connected with the first layer intermediate representation tensor fusion network layer and the first layer result representation tensor fusion network layer, a full-connection network connected with the second layer tensor fusion network layer and a classification network;
training a basic model by using the basic training data, and training a new equipment model by using the new equipment training data to obtain a trained basic model and a trained new equipment model;
inputting the base training data into the trained base model to obtain an intermediate representation vector denoted B1 and a resultant representation vector denoted B2, and simultaneously inputting the new plant training data into the trained new plant model to obtain an intermediate representation vector denoted E1 and a resultant representation vector denoted E2;
and inputting the intermediate representation vectors B1 and E1 and the result representation vectors B2 and E2 into the tensor-based hierarchical fusion network model and training by using a preset loss function to obtain a trained fusion recognition model, and performing digestive tract disease recognition on input frame picture data by using the trained fusion recognition model to obtain a recognition result.
Optionally, wherein the output of the fused recognition model is
Figure 478175DEST_PATH_IMAGE001
Wherein
Figure 957698DEST_PATH_IMAGE002
In the general category of diseases,
Figure 660075DEST_PATH_IMAGE003
is as follows
Figure 225048DEST_PATH_IMAGE004
And (4) outputting the prediction of each category.
Optionally, wherein the first layer intermediate representation tensor fusion network layer fuses two intermediate representation vectors and obtains the dimension ofNIs/are as followsM1 vector：
Figure 671073DEST_PATH_IMAGE005
Wherein,
Figure 270682DEST_PATH_IMAGE006
as a function of the fully-connected network,
Figure 878381DEST_PATH_IMAGE007
for outer product calculations, B1 and E1 are intermediate representation vectors.
Optionally, the first layer of resultant expression tensors fuses the two resultant expression vectors, and obtains the dimensionality ofNIs/are as followsM2Vector quantity:
Figure 196230DEST_PATH_IMAGE008
wherein,
Figure 180366DEST_PATH_IMAGE006
as a function of the fully-connected network,
Figure 634481DEST_PATH_IMAGE007
for outer product calculations, B2 and E2 represent vectors for the results.
Optionally, wherein the second layer tensor fusion network layer fuses the results of the first layer outputM1AndM2and through a fully connected network, obtaining a dimension ofNIs/are as followsRVector quantity:
Figure 180125DEST_PATH_IMAGE009
wherein,
Figure 719691DEST_PATH_IMAGE006
as a function of the fully-connected network,
Figure 507519DEST_PATH_IMAGE007
for the outer product calculation, M1 is the output of the first layer intermediate representation tensor fusion network, and M2 is the output of the first layer result representation tensor fusion network.
Optionally, wherein the classification network issoftmaxA layer of a material selected from the group consisting of,
Figure 81719DEST_PATH_IMAGE010
wherein,
Figure 296800DEST_PATH_IMAGE011
is as follows
Figure 58083DEST_PATH_IMAGE004
The probability values of the individual categories are,
Figure 649601DEST_PATH_IMAGE002
in the general category of diseases,
Figure 78308DEST_PATH_IMAGE003
is as follows
Figure 198711DEST_PATH_IMAGE004
The predicted output of the individual categories is,
Figure 978448DEST_PATH_IMAGE012
for the predicted output of the c-th class,
Figure 842499DEST_PATH_IMAGE013
is the base of the natural logarithmic function.
Optionally, a gradient descent algorithm may be used in training the tensor-based hierarchical fusion network model.
Optionally, wherein the preset loss function is:
Figure 125713DEST_PATH_IMAGE015
wherein,Nfor the number of development set samples that are input,
Figure 446710DEST_PATH_IMAGE002
the number of the total disease categories is,
Figure 448165DEST_PATH_IMAGE016
for the prediction output of all the input samples,
Figure 115906DEST_PATH_IMAGE017
is as followsnA first sample ofcThe value of the tag for each of the categories,
Figure 253627DEST_PATH_IMAGE018
is as followsnA first sample ofcAnd (4) outputting the prediction of each category.
Optionally, the identification model used by the basic identification module in the client side is consistent with the basic model in the server side.
The beneficial effects of this application include at least: the system comprises a client and a server; the client side package data conversion module, the equipment detection module, the data uploading module, the basic identification module and the data display module are used for data acquisition and transmission of new equipment and display of digestive tract disease identification results; the server side comprises an equipment information management module, a data processing module and a fusion recognition module, and is used for training a basic model, a new equipment model and a fusion recognition model of equipment information management, data annotation and digestive tract disease recognition, and performing digestive tract disease recognition through the fusion recognition model. The method and the system can enable new digestive tract endoscope equipment to be rapidly migrated to the recognition system, and enable the system to have better prediction performance based on data collected by the new equipment through fusion of recognition models.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention.
Drawings
The present application may be better understood by describing exemplary embodiments thereof in conjunction with the following drawings, wherein:
FIG. 1 is a schematic diagram of the differences between different digestive tract detection devices;
FIG. 2 is a system architecture diagram provided in accordance with one embodiment of the present application;
FIG. 3 is a schematic illustration of disease information annotation provided in accordance with an embodiment of the present application;
FIG. 4 is a schematic diagram of a base model provided by an embodiment of the present application;
FIG. 5 is a schematic view of a fusion model provided in an embodiment of the present application;
FIG. 6 is a schematic illustration of digestive tract disease identification provided by an embodiment of the present application;
FIG. 7 is a graph showing recognition results provided by one embodiment of the present application;
fig. 8 is a flowchart of a fusion recognition method according to an embodiment of the present application.
Detailed Description
The following detailed description of the embodiments of the present application, taken in conjunction with the accompanying drawings and examples, will enable those skilled in the art to practice the embodiments of the present application with reference to the description.
It is noted that in the detailed description of these embodiments, in order to provide a concise description, all features of an actual implementation may not be described in detail. It should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions are made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
As shown in fig. 2, the digestive tract disease identification system with device migration capability provided by the present invention includes a client and a server.
The client comprises:
and the data acquisition module is used for receiving the video data from the digestive tract endoscope equipment, processing and acquiring the video data and acquiring frame picture data. Specifically, since the identification model cannot directly input the original video, the video data is further extracted by using a certain time interval, and the specific time interval is not limited in this embodiment. For example, a segment of 5-minute raw inspection video is extracted at 200ms intervals, and a total of 1500 video frame pictures are obtained.
And the new equipment detection module is used for detecting whether the current digestive tract endoscope equipment is new equipment or not, and specifically, the new equipment detection module judges whether the current digestive tract endoscope equipment is new equipment or not according to the equipment serial number and the frame picture characteristics. And when the digestive tract endoscope equipment is new equipment, uploading the frame picture data to a server end, and when the digestive tract endoscope equipment is existing equipment, transmitting the frame picture to the basic identification module.
And the basic identification module is used for identifying the frame picture data transmitted by the equipment detection module. And if the current equipment is not the new equipment, directly calling the model to identify the video frame picture and returning an identification result. Optionally, the identification model used by the basic identification module in the client is consistent with the basic model in the server.
And the data display module is used for displaying the server-side digestive tract disease identification result and the client-side digestive tract disease identification result to a user and comprises a data organization unit and a display unit. The data organizing unit generates a recognition result map as shown in fig. 7, in which the left part is a currently recognized endoscopic picture of the digestive tract and the right part is a recognition result, i.e., a probability value of each disease category. The display unit is responsible for rendering the display result generated by the data organization unit, assists the work of a doctor and improves the diagnosis efficiency of the doctor.
The server side includes:
and the equipment information management module is used for managing the equipment information and the data uploading log information.
A digestive tract disease recognition system having a device migration capability needs to cope with a variety of scenes of different digestive tract detection devices, and thus needs to efficiently manage different device information. In this embodiment, the device information management module includes a device information management unit and a new device access log unit. The device information management unit stores the device number sequence of the existing data source, and adds the new device number to the managed device number sequence when the new device data is labeled. The new device access log unit is responsible for recording access information of the devices, including access time and access times.
And the data processing module receives the new digestive tract endoscope equipment data, the labeling data and the storage data from the client.
Digestive tract disease identification systems with device migration capabilities require storage of large amounts of application data, including new device data, annotation data, and save data. Specifically, the new device data is frame picture data after the detection device is preprocessed, and the labeled data is frame picture data subjected to disease category information labeling and used for model training.
The marked data are of two types, wherein one type is marked data pictures of a plurality of existing offline devices and is used for training a basic model for identifying the digestive tract diseases, and the other type is marked frame pictures acquired by a new device and is used for training a new device model for identifying the digestive tract diseases.
Wherein the disease category includes but is not limited to at least one of the following 22: gastric ulcer, gastric polyp, gastritis, gastric cancer, gastric submucosal mass, gastric varices, acute gastric mucosal lesions, gastric xanthoma, cardiac polyp, pyloric ulcer, duodenal ulcer, duodenitis, duodenal polyp, duodenal submucosal mass, reflux esophagitis, Barrett's esophagus, mycotic esophagitis, benign protrusion type lesions of esophagus (esophageal polyp, esophageal papilloma), esophageal varices, hiatal hernia, esophageal cancer, esophageal submucosal mass. For example, referring to fig. 3, a schematic view of disease information labeling provided by an embodiment of the present application, the disease information in this picture is labeled as gastric ulcer.
The method comprises the following steps that original frame picture data acquired by different new equipment and pictures in the label data of a plurality of existing offline equipment have differences, including pixel point differences and shape inconsistencies, and in order to obtain a better training effect, the preprocessing operation comprises the steps of enabling the new equipment frame picture and the pictures in the label data of the plurality of existing offline equipment to be consistent in pixel through image zooming operation; and through image cutting operation, the shape of the frame picture of the new equipment is consistent with the shape of the picture in the existing offline labeling data of a plurality of pieces of equipment.
Alternatively, the data access speed can be increased by adding a storage and calculation unit of the data processing module, and the storage space is expanded, so that the digestive tract disease recognition system with the device migration capability can cope with a large number of different digestive tract detection devices.
And the fusion identification module is used for identifying digestive tract diseases of the frame picture data uploaded by the client by a model fusion identification method to obtain an identification result.
Referring to fig. 8, the fusion recognition method in this embodiment at least includes the following steps:
and 810, constructing a basic model, a new equipment model and a tensor hierarchy-based fusion network model, solving the problem that a single identification model cannot be well transferred to different digestive tract endoscope equipment by using the models, and obtaining better prediction performance on new equipment.
The basic model and the new equipment model adopt a deep learning image classification model. For example, fig. 4 is a schematic diagram of a basic model provided in an embodiment of the present application. The network structure is composed of 3 parts, namely a feature extraction network, a capture relation network and a category prediction network. The original digestive tract picture is firstly input into a feature extraction network to extract picture features, wherein the feature extraction network is composed of 12 layers of transform structures, and a 768-dimensional picture feature vector is finally obtained. Then, the picture features are copied intoCParts of, whereinCAs the number of categories, willCThe picture features are input into a Transformer structure, and an attention mechanism inside the Transformer can automatically learn the dependency relationship among different categories and output the category features (intermediate representation) of each category. Category predictive network sharingCEach disease category corresponds to a dedicated prediction network and is responsible for decoding corresponding category characteristics (intermediate representation), and the category prediction network receives the category characteristics (intermediate representation) as input and outputs probability values (result representation) belonging to the corresponding disease categories through decoding.
The fusion recognition model adopts a neural network model based on tensor hierarchical fusion, and is a schematic diagram of the fusion recognition model provided by an embodiment of the application with reference to fig. 5.
And step 820, training a basic model by using the basic training data and training a new equipment model by using the new equipment training data to obtain a trained basic model and a trained new equipment model.
Training data used by a basic model for identifying digestive tract diseases are the existing offline labeling data of a plurality of devices; the training data used by the new equipment model for digestive tract disease recognition is the labeling data of the new digestive tract endoscope equipment uploaded by the client.
Alternatively, a gradient descent algorithm may be used to train the model, using the cross entropy as a loss function during the model training process, and the formula is as follows:
Figure 981411DEST_PATH_IMAGE015
wherein,Nfor the number of development set samples that are input,
Figure 470161DEST_PATH_IMAGE002
the number of the total disease categories is,
Figure 941594DEST_PATH_IMAGE016
for the prediction output of all the input samples,
Figure 933821DEST_PATH_IMAGE017
is as followsnA first sample ofcThe value of the tag for each of the categories,
Figure 98086DEST_PATH_IMAGE018
is as followsnA first sample ofcAnd (4) outputting the prediction of each category.
Optionally, in the training process, the batch size is set to 8, the initial learning rate is set to 0.00001, and an Adam optimizer is adopted, in other embodiments, corresponding hyper-parameters may also be different during model training, and the batch size and the initial learning rate may also be other values, which does not limit values of each parameter in the training process. After training, the basic model has good recognition accuracy and can be used for carrying out disease classification on the input digestive tract picture.
Step 830, inputting the basic training data into the trained basic model to obtain an intermediate representation vector denoted as B1 and a result representation vector denoted as B2, and simultaneously inputting the new device training data into the trained new device model to obtain an intermediate representation vector denoted as E1 and a result representation vector denoted as E2.
Training data used by the fusion recognition model for digestive tract disease recognition is a small development set in new equipment marking data uploaded by a client. The input of the fusion recognition model is intermediate representation of the sample in the basic model and the new equipment model respectivelyB1AndE1and a result representation vector (i.e., a probability vector belonging to each class), respectively represented asB2AndE2。
and 840, inputting the intermediate representation vectors B1 and E1 and the result representation vectors B2 and E2 into a tensor-level-based fusion network model and training by using a preset loss function to obtain a trained fusion recognition model, and performing digestive tract disease recognition on the input frame picture data by using the trained fusion recognition model to obtain a recognition result.
Referring to fig. 5, which is a schematic diagram of a fusion recognition model provided in an embodiment of the present application, a first layer intermediate representation tensor fusion network layer fuses two intermediate representation vectorsB1AndE1and get the dimension of through the fully connected networkNIs/are as followsM1And (5) vector quantity. First layer result expression tensor fusion network layer fusion two result expression vectorsB2AndE2and get the dimension of through the fully connected networkNIs/are as followsM2And (5) vector quantity. Then, the second layer tensor fusion network layer fuses the result output by the first layerM1AndM2and through a fully connected network, obtaining a dimension ofNIs/are as followsRAnd (5) vector quantity. Finally, the network is fully connected andsoftmaxclassified network receptionRThe vector is used as input, and the final prediction result is output.
Figure 542974DEST_PATH_IMAGE010
Wherein,
Figure 818097DEST_PATH_IMAGE011
is as follows
Figure 195989DEST_PATH_IMAGE004
The probability values of the individual categories are,
Figure 495602DEST_PATH_IMAGE002
in the general category of diseases,
Figure 958945DEST_PATH_IMAGE003
is as follows
Figure 506601DEST_PATH_IMAGE004
The predicted output of the individual categories is,
Figure 738999DEST_PATH_IMAGE012
for the predicted output of the c-th class,
Figure 979487DEST_PATH_IMAGE013
is the base of the natural logarithmic function.
Wherein, the specific fusion process of the tensor fusion network layer can be expressed by the following formula,
Figure DEST_PATH_IMAGE020
wherein, the formula (1) corresponds to the middle of the first layer to represent the fusion process of the tensor fusion network layer, the formula (2) corresponds to the result of the first layer to represent the fusion process of the tensor fusion network layer, the formula (3) corresponds to the fusion process of the tensor fusion network layer of the second layer,
Figure 398967DEST_PATH_IMAGE006
as a function of the fully-connected network,
Figure 15893DEST_PATH_IMAGE007
the outer product calculation is represented as a function of,B1ande1 isThe intermediate representation vector, B2 and E2 are the resulting representation vectors, M1andM2and outputting the result of the first layer network.
Alternatively, the fused recognition model may employ the same training mode and loss function as the base model.
Referring to fig. 6, a schematic diagram of digestive tract disease recognition according to an embodiment of the present application is provided, first, a digestive tract image is respectively input into a basic model and an equipment model, and an intermediate representation of the basic model is obtainedB1And the results showB2And intermediate representation of the plant modelE1And the results showE2. Then, willB1、B2、E1、E2And inputting a fusion recognition model, wherein the fusion recognition model can fully combine the potential characteristics of different expressions to give a recognition result of the digestive tract diseases.
Optionally, the fused recognition model may be tested using test data, including: using the prediction results, MAP index and F1 index are calculated, MAP is Average Precision Mean (Mean Average Precision), F1 is equal weight and Average of Precision and recall (F1-Score) for evaluating the recognition Precision of the model.
The results of comparison of the basic digestive tract disease recognition model, the digestive tract recognition model of the new device and the fused digestive tract disease recognition model at the two indexes of MAP and F1 are referred to as the table one below. As can be seen from the table I, the fused recognition model has further improved accuracy (the improvement range is about 5%) compared with the basic recognition model. Therefore, the problem that a single recognition model is difficult to apply to different digestive tract endoscope devices is effectively solved, and different digestive tract endoscope devices can be better adapted by fusing the recognition models of new devices.
Table one:
Figure DEST_PATH_IMAGE021
the basic principles of the present application have been described in connection with specific embodiments, but it should be noted that, for those skilled in the art, it can be understood that all or any of the steps or components of the method and apparatus of the present application can be implemented in hardware, firmware, software or their combination in any computing device (including processors, storage media, etc.) or network of computing devices, which can be implemented by those skilled in the art using their basic programming skills after reading the description of the present application.
The object of the present application can thus also be achieved by running a program or a set of programs on any computing device. The computing device may be a general purpose device as is well known. The object of the application can thus also be achieved merely by providing a program product comprising program code for implementing the method or the apparatus. That is, such a program product also constitutes the present application, and a storage medium storing such a program product also constitutes the present application. It is to be understood that the storage medium may be any known storage medium or any storage medium developed in the future.
It is further noted that in the apparatus and method of the present application, it is apparent that the components or steps may be disassembled and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application. Also, the steps of executing the series of processes described above may naturally be executed chronologically in the order described, but need not necessarily be executed chronologically. Some steps may be performed in parallel or independently of each other.
Unless otherwise defined, technical or scientific terms used in the claims and the specification should have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. The use of "first," "second," and similar terms in the description and claims of this patent application do not denote any order, quantity, or importance, but rather the terms are used to distinguish one element from another. The terms "a" or "an," and the like, do not denote a limitation of quantity, but rather denote the presence of at least one. The word "comprise" or "comprises", and the like, means that the element or item listed before "comprises" or "comprising" covers the element or item listed after "comprising" or "comprises" and its equivalent, and does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, nor are they restricted to direct or indirect connections.
The above-described embodiments should not be construed as limiting the scope of the present application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims (10)

1. A digestive tract disease identification system with device migration capabilities, comprising: a client and a server;
the client comprises a data conversion module, an equipment detection module, a basic identification module and a data display module;
the data conversion module receives and converts video data output by the digestive tract endoscope equipment into a picture format to obtain frame picture data;
the device detection module detects whether the digestive tract endoscope device is a new digestive tract endoscope device, transmits the frame picture data to a server end when the digestive tract endoscope device is the new device, and transmits the frame picture data to a basic identification module when the digestive tract endoscope device is an existing device;
the basic identification module identifies the frame picture data transmitted by the equipment detection module to obtain a digestive tract disease identification result;
the data display module receives the digestive tract disease recognition result and displays the digestive tract disease recognition result through display equipment;
the server side comprises an equipment information management module, a data processing module and a fusion identification module;
the device information management module manages information of the digestive tract endoscope device;
the data processing module receives, marks and stores frame picture data uploaded by a client, and stores basic training data and new equipment training data, wherein the basic training data are the marked frame picture data of a plurality of existing digestive tract endoscope equipment, and the new equipment training data are the marked frame picture data of the new digestive tract endoscope equipment uploaded by the client;
the fusion recognition module recognizes the frame picture data uploaded by the client by a model fusion recognition method to obtain a digestive tract disease recognition result and transmits the digestive tract disease recognition result to the client,
the model fusion identification method comprises the following steps: constructing a basic model, a new equipment model and a tensor-based hierarchical fusion network model, wherein the basic model and the new equipment model are deep learning image classification models, and the tensor-based hierarchical fusion network model consists of a first layer intermediate representation tensor fusion network layer, a first layer result representation tensor fusion network layer, a second layer tensor fusion network layer connected with the first layer intermediate representation tensor fusion network layer and the first layer result representation tensor fusion network layer, a full-connection network connected with the second layer tensor fusion network layer and a classification network; training a basic model by using the basic training data, and training a new equipment model by using the new equipment training data to obtain a trained basic model and a trained new equipment model;
inputting the base training data into the trained base model to obtain an intermediate representation vector denoted B1 and a resultant representation vector denoted B2, and simultaneously inputting the new plant training data into the trained new plant model to obtain an intermediate representation vector denoted E1 and a resultant representation vector denoted E2; and inputting the intermediate representation vectors B1 and E1 and the result representation vectors B2 and E2 into the tensor-based hierarchical fusion network model and training by using a preset loss function to obtain a trained fusion recognition model, and performing digestive tract disease recognition on input frame picture data by using the trained fusion recognition model to obtain a recognition result.
2. The system according to claim 1, wherein the device information management module stores a device number sequence of an existing endoscopic device for the digestive tract, and adds a device number of a new endoscopic device for the digestive tract to the device number sequence when tagging of frame picture data of the new endoscopic device for the digestive tract is completed.
3. The system of claim 1, wherein the output of the fused recognition model is
Figure 802059DEST_PATH_IMAGE001
Wherein
Figure 171730DEST_PATH_IMAGE002
In the general category of diseases,
Figure 217046DEST_PATH_IMAGE003
is as follows
Figure 815518DEST_PATH_IMAGE004
And (4) outputting the prediction of each category.
4. The system of claim 1, wherein the first layer intermediate representation tensor fusion meshThe network layer fuses the two intermediate representation vectors and obtains the dimension ofNIs/are as followsM1 vector：
Figure 629890DEST_PATH_IMAGE005
Wherein,
Figure 717932DEST_PATH_IMAGE006
as a function of the fully-connected network,
Figure 188227DEST_PATH_IMAGE007
for outer product calculations, B1 and E1 are intermediate representation vectors.
5. The system of claim 1, wherein the first layer resultant tensor fusion network layer fuses two resultant representation vectors and obtains a dimension of through a fully connected network ofNIs/are as followsM2Vector quantity:
Figure 652707DEST_PATH_IMAGE008
wherein,
Figure 524848DEST_PATH_IMAGE006
as a function of the fully-connected network,
Figure 518211DEST_PATH_IMAGE007
for outer product calculations, B2 and E2 represent vectors for the results.
6. The system of claim 1, wherein the second layer tensor fusion network layer fuses the results of the first layer outputM1AndM2and through a fully connected network, obtaining a dimension ofNIs/are as followsRVector quantity:
Figure 538120DEST_PATH_IMAGE009
wherein,
Figure 727662DEST_PATH_IMAGE006
as a function of the fully-connected network,
Figure 516626DEST_PATH_IMAGE007
for the outer product calculation, M1 is the first layer intermediate representation tensor fusion network output, and M2 is the first layer result representation tensor fusion network output.
7. The system of claim 1, wherein the classification network issoftmaxA layer of a material selected from the group consisting of,
Figure 149733DEST_PATH_IMAGE010
wherein,
Figure 391358DEST_PATH_IMAGE011
is as follows
Figure 932061DEST_PATH_IMAGE004
The probability values of the individual categories are,
Figure 778794DEST_PATH_IMAGE002
the number of the total disease categories is,
Figure 379540DEST_PATH_IMAGE003
is as follows
Figure 311724DEST_PATH_IMAGE004
The predicted output of the individual categories is,
Figure 390538DEST_PATH_IMAGE012
for the predicted output of the c-th class,
Figure 154095DEST_PATH_IMAGE013
is the base of the natural logarithmic function.
8. The system of claim 1, wherein a gradient descent algorithm may be employed in training the tensor-based hierarchical fusion network model.
9. The system of claim 1, wherein the preset loss function is:
Figure 378272DEST_PATH_IMAGE014
wherein,Nfor the number of development set samples that are input,
Figure 594490DEST_PATH_IMAGE002
the number of the total disease categories is,
Figure 680257DEST_PATH_IMAGE015
for the prediction output of all the input samples,
Figure 32741DEST_PATH_IMAGE016
is as followsnA first sample ofcThe value of the tag for each of the categories,
Figure 240869DEST_PATH_IMAGE017
is as followsnA first sample ofcAnd (4) outputting the prediction of each category.
10. The system of claim 1, wherein the base identification module in the client uses an identification model that is consistent with the base model at the server.
CN202110917028.1A 2021-08-11 2021-08-11 Digestive tract disease recognition system with device migration capability Active CN113367644B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110917028.1A CN113367644B (en) 2021-08-11 2021-08-11 Digestive tract disease recognition system with device migration capability

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110917028.1A CN113367644B (en) 2021-08-11 2021-08-11 Digestive tract disease recognition system with device migration capability

Publications (2)

Publication Number Publication Date
CN113367644A CN113367644A (en) 2021-09-10
CN113367644B true CN113367644B (en) 2022-04-15

Family

ID=77576713

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110917028.1A Active CN113367644B (en) 2021-08-11 2021-08-11 Digestive tract disease recognition system with device migration capability

Country Status (1)

Country Link
CN (1) CN113367644B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113822389B (en) * 2021-11-24 2022-02-22 紫东信息科技(苏州)有限公司 Digestive tract disease classification system based on endoscope picture
CN116612340B (en) * 2023-07-21 2023-10-13 紫东信息科技(苏州)有限公司 Peptic ulcer part classification method, device and readable storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1642467A (en) * 2002-03-14 2005-07-20 奥林巴斯株式会社 Endoscope image processing apparatus
CN107622485A (en) * 2017-08-15 2018-01-23 中国科学院深圳先进技术研究院 A kind of medical image data analysis method and system for merging depth tensor neutral net
CN108366788A (en) * 2015-11-30 2018-08-03 任旭彬 Diagnostic system and diagnosis management method whether abnormal using the cell of DNN study
CN109614487A (en) * 2018-10-29 2019-04-12 山东大学 A method of the emotional semantic classification based on tensor amalgamation mode
CN112965381A (en) * 2021-02-09 2021-06-15 重庆高开清芯智联网络科技有限公司 Method for establishing cooperative intelligent self-adaptive decision model
CN113139937A (en) * 2021-04-13 2021-07-20 上海市第一人民医院 Digestive tract endoscope video image identification method based on deep learning

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1642467A (en) * 2002-03-14 2005-07-20 奥林巴斯株式会社 Endoscope image processing apparatus
CN108366788A (en) * 2015-11-30 2018-08-03 任旭彬 Diagnostic system and diagnosis management method whether abnormal using the cell of DNN study
CN107622485A (en) * 2017-08-15 2018-01-23 中国科学院深圳先进技术研究院 A kind of medical image data analysis method and system for merging depth tensor neutral net
CN109614487A (en) * 2018-10-29 2019-04-12 山东大学 A method of the emotional semantic classification based on tensor amalgamation mode
CN112965381A (en) * 2021-02-09 2021-06-15 重庆高开清芯智联网络科技有限公司 Method for establishing cooperative intelligent self-adaptive decision model
CN113139937A (en) * 2021-04-13 2021-07-20 上海市第一人民医院 Digestive tract endoscope video image identification method based on deep learning

Also Published As

Publication number Publication date
CN113367644A (en) 2021-09-10

Similar Documents

Publication Publication Date Title
CN113367644B (en) Digestive tract disease recognition system with device migration capability
Mehmood et al. Video summarization based tele-endoscopy: a service to efficiently manage visual data generated during wireless capsule endoscopy procedure
CN113496489B (en) Training method of endoscope image classification model, image classification method and device
CN109997147A (en) System and method for diagnosing the otoscope image analysis of ear's pathology
Ghosh et al. Effective deep learning for semantic segmentation based bleeding zone detection in capsule endoscopy images
Ghosh et al. An automatic bleeding detection scheme in wireless capsule endoscopy based on histogram of an RGB-indexed image
CN114782760B (en) Stomach disease picture classification system based on multitask learning
Biniaz et al. Automatic reduction of wireless capsule endoscopy reviewing time based on factorization analysis
Sadasivan et al. High accuracy patch-level classification of wireless capsule endoscopy images using a convolutional neural network
Lazăr et al. The impact of artificial intelligence in the endoscopic assessment of premalignant and malignant esophageal lesions: present and future
CN114399634B (en) Three-dimensional image classification method, system, equipment and medium based on weak supervision learning
CN114399465A (en) Benign and malignant ulcer identification method and system
Du et al. Improving the classification performance of esophageal disease on small dataset by semi-supervised efficient contrastive learning
CN116664929A (en) Laryngoscope image multi-attribute classification method based on multi-modal information fusion
Bhosale et al. Graph and capsule convolutional neural network based classification of lung Cancer, pneumonia, COVID-19 using lung CT and ultrasound radiography imaging
Kim et al. Fostering transparent medical image AI via an image-text foundation model grounded in medical literature
Henry Machine learning approaches for early diagnosis of thyroid cancer
Kim et al. Transparent medical image AI via an image–text foundation model grounded in medical literature
Nien et al. Criss-cross attention based multi-level fusion network for gastric intestinal metaplasia segmentation
Ye et al. See what you see: Self-supervised cross-modal retrieval of visual stimuli from brain activity
Rani et al. Classification of wireless capsule endoscopy images for bleeding using deep features fusion
Yang et al. Identification of upper GI diseases during screening gastroscopy using a deep convolutional neural network algorithm
CN113313177A (en) Digestive tract endoscope picture multi-label classification system
Moalla et al. Exploiting Pre-trained Architectures for Dual-Stream Classification of LCA-RCA in a Private AngioData
Fisher et al. Colour image analysis of wireless capsule endoscopy video: A review

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant