EP4190271A1 - Verarbeitungsvorrichtung für endoskopbilder - Google Patents

Verarbeitungsvorrichtung für endoskopbilder Download PDF

Info

Publication number
EP4190271A1
EP4190271A1 EP21212304.6A EP21212304A EP4190271A1 EP 4190271 A1 EP4190271 A1 EP 4190271A1 EP 21212304 A EP21212304 A EP 21212304A EP 4190271 A1 EP4190271 A1 EP 4190271A1
Authority
EP
European Patent Office
Prior art keywords
endoscope
images
image
stream
lumen
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP21212304.6A
Other languages
English (en)
French (fr)
Inventor
Alejandro ALONSO DÍAZ
Josefine Dam GADE
Andreas Härstedt JØRGENSEN
Lee Herluf Lund LASSEN
Dana Marie YU
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ambu AS
Original Assignee
Ambu AS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ambu AS filed Critical Ambu AS
Priority to EP21212304.6A priority Critical patent/EP4190271A1/de
Priority to US18/074,436 priority patent/US20230172428A1/en
Publication of EP4190271A1 publication Critical patent/EP4190271A1/de
Priority to US18/206,935 priority patent/US20240021103A1/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/361—Image-producing devices, e.g. surgical cameras
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/31—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for the rectum, e.g. proctoscopes, sigmoidoscopes, colonoscopes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2048—Tracking techniques using an accelerometer or inertia sensor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2059—Mechanical position encoders
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/30—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
    • A61B2090/306—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure using optical fibres
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/30—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
    • A61B2090/309—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure using white LEDs
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/372—Details of monitor hardware
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20021—Dividing image into blocks, subimages or windows
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30028—Colon; Small intestine

Definitions

  • the present disclosure relates to an image processing device, a display unit, an endoscope system, a method for estimating a quality measure of an endoscopy procedure, and a computer program product.
  • Endoscopes are widely used in hospitals for visually examining body cavities and obtaining samples of tissue identified as potentially pathological.
  • An endoscope typically comprises an image capturing device arranged at the distal end of the endoscope either looking forward or to the side.
  • An endoscope is further typically provided with a working channel allowing a medical device such as a gripping device, a suction device, or a catheter to be introduced.
  • a colonoscopy is an example of a complex endoscopic procedure.
  • a medical professional may calculate his / hers Polyp detection rate and / or Ad-enoma detection rate and compare those with expected values. This may provide the medical professional with a general idea of their performance over time. However, an evaluation of the quality of a specific colonoscopy is not possible.
  • the present disclosure relates to an image processing device for estimating a quality measure of an endoscopic procedure performed using an endoscope, the endoscope comprising an image capturing device, the image processing device comprising a processing unit operationally connectable to the image capturing device, wherein the processing unit is configured to:
  • the processing unit of the image processing device may be any processing unit, such as a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller unit (MCU), a field-programmable gate array (FPGA), or any combination thereof.
  • the processing unit may comprise one or more physical processors and/or may be combined by a plurality of individual processing units.
  • the image processing device may comprise a display, i.e. the image processing device may be integrated in a display unit. Alternatively / additionally, the image processing device may be operationally connectable to a separate display unit.
  • the processing unit may be configured to detect the lumen using an adaptive method such as a machine learning data architecture.
  • the processing unit may be configured to detect the lumen using a non-adaptive method relying on static rules.
  • the processing unit may be configured to detect a lumen by using a method comprising: finding a set of connected pixels having an intensity value below a first threshold.
  • the first threshold may be an absolute threshold or a threshold determined based on the average intensity in an image.
  • a group of connected pixels may be a set of pixels, where each pixel shares an edge or a corner with at least one other pixel in the set of pixels.
  • a lumen may be detected if a set of connected pixels is found having a size above a second threshold.
  • the quality measure may be presented to the user during the procedure e.g. on a display. Alternatively, the quality measure may be presented to the user after the procedure has been completed and / or stored in a memory unit connected to the processing unit of the image processing device.
  • the processing unit is configured to estimate the location of the lumen by providing the stream of image to a machine learning data architecture trained to identify the location of lumens in endoscope images.
  • the machine learning data architecture may be a supervised machine learning architecture.
  • the machine learning data architecture may be trained by obtaining a training data set comprising endoscope images of different parts of different body cavities and for each endoscope image having a human operator identify a lumen (if present) e.g. a human operator may for each image firstly specify if a lumen can be seen in the image (if the image capturing device is pointing towards a wall of the body cavity no lumen is visible), and secondly the location of the lumen.
  • a quality measure for a particular type of endoscopic procedure is determined, then the training data set may comprise endoscope images from that particular type of endoscopic procedure.
  • the quality measure is for a colonoscopy procedure
  • the training data set may comprise endoscope images from different colonoscopy procedures on different patients.
  • the location of the lumen may be a center of the lumen or a circumference of the lumen.
  • the circumference of the lumen may be a circle e.g. the smallest circle that encompasses the lumen or the largest circle that can be fully contained within the lumen.
  • the circumference may be a curve substantially arranged above the border of the lumen.
  • the processing unit may implement the machine learning data architecture, i.e. the processing unit may provide the image to the machine learning data architecture by processing the image using the machine learning data architecture.
  • the processing unit may provide the image, a parameterized version of the image, or a dimensionally reduced version of the image to another processing unit e.g. outside of the image processing device, where said another processing unit implements the machine learning data architecture.
  • the machine learning data architecture is an artificial neural network such as a deep structured learning architecture.
  • the machine learning data architectures may be U-Net (reference: Ronneberger et el. "U-Net: Convolutional Networks for Biomedical Image Segmentation” (2015), arXiv:1505.04597 [cs.CV] or Mask R-CNN (reference: He et al. "Mask R-CNN” (2017), arXiv:1703.06870 [cs.CV].
  • the endoscopic procedure is a colonoscopy.
  • the location of the lumen in the image may indicate what area of the circumference of the colon is being investigated. Consequently, by using the estimated location of the lumen in the image for determining a quality measure, the quality measure may indicate if all parts of the colon have been sufficiently investigated.
  • the processing unit is further configured to divide the circumference of the colon into a plurality of areas, based on the estimated location of the lumen in the stream of images estimate which area of the plurality of areas is being investigated, and for each area of the plurality of areas determine a quality measure.
  • the medical professional may be provided with information not only specifying the overall quality of the colonoscopy, but detailed information specifying the quality of different areas of the colon. This may both be useful information for the medical professional during the procedure and after the procedure.
  • the circumference of the colon may be divided into equally large areas e.g. four equally large areas or 8 equally large areas.
  • the areas may be upper left, upper right, lower left, and lower right.
  • each area of the plurality of areas corresponds to an image zone of the stream of images and wherein the processing unit is configured to estimate that a particular area of the plurality of areas is being investigated if the estimated location of the lumen is arranged within the image zone of the particular area.
  • the image zones may have an equal size.
  • the image zones may be centered around the center of image.
  • the image zone may be arranged rotationally symmetric around the center of the image.
  • Each image zone may have an opposing image zone with the same shape but only rotated 180 degrees around the center of the image.
  • the lumen In the event the lumen is located in more than one image zone, it may be decided that the lumen is present in the image zone where most of the lumen is present. However, it may also be determined that the lumen is only present in an image zone if the entire lumen is present in the image zone or at least a percentage of the lumen above a threshold.
  • the processing unit is configured to, for one or more images of the stream of images where a lumen is not found, estimate which area of the plurality of areas is being investigated based on a previous image in the stream of images where a lumen if found and / or a subsequent image where a lumen is found.
  • the processing unit is further configured to divide the colonoscopy procedure into a plurality of parts and for each part estimate a quality measure.
  • each part corresponds to a section of the colon having a predetermined length
  • the processing unit is configured to process the stream of images to estimate in which section the endoscope is located.
  • each section may be 5cm, 10cm or 15cm.
  • Each section may correspond to specific anatomical positions or be defined with respect to a specific anatomic position. Examples of specific anatomical positions are: the cecum, the anus, and the entrance to the appendix.
  • each section may also be defined relative to a selected point e.g. the initial position of the colonoscope.
  • the processing unit is configured to process the stream of images to estimate the withdrawal speed of the endoscope and use the estimated withdrawal speed to estimate in which section the endoscope is located.
  • each section is defined relative to a selected point e.g. the initial position of the colonoscope.
  • the processing unit is configured to process the stream of images to determine a parameter value related to the sharpness of the images and estimate the withdrawal speed based on the parameter value.
  • the sharpness of an image is dependent on the speed of the endoscope. Specifically, if the endoscope is moving fast the resulting images will become less sharp than if the endoscope is moving slow.
  • the stream of images may be processed in the spatial domain and / or frequency domain to determine a parameter value related to the sharpness of the images.
  • a value may be determined specifying the percentage of energy in the image above a particular set of frequencies.
  • a high value will be indicative of a sharp image and a low value will be indicative of a unsharp / blurry image.
  • a set of frequencies may be found containing a particular amount of the energy in the image. Again, a high value will be indicative of a sharp image and a low value will be indicative of a unsharp / blurry image.
  • the average change in intensity between neighboring pixels may be determined.
  • a high value will be indicative of a sharp image and a low value will be indicative of a unsharp / blurry image.
  • the relationship between the parameter value and the withdrawal speed is dependent on a number of factors including the lightning, the optical system, the tissue type etc. In practice, it may be experimentally determined. A function may be found describing the relationship. Alternatively, a look-up table may be stored specifying the relationship.
  • the processing unit is configured to estimate the withdrawal speed of the endoscope by providing the stream of images to a machine learning data architecture trained to estimate the withdrawal speed of an endoscope during a colonoscopy procedure based on endoscope images.
  • the machine learning data architecture may be a supervised machine learning architecture.
  • the machine learning data architecture may be trained by obtaining a training data set comprising a plurality of streams of endoscope images obtained from different colonoscopies, where for each stream of endoscope image the withdrawal speed is provided.
  • the withdrawal speed may be measured using an extra measurement system.
  • the endoscope may be provided with accelerometers enabling the withdrawal speed to be estimated.
  • the withdrawal speed may be estimated by providing the insertion tube of the endoscope with markings e.g. a marking every 5 cm. The withdrawal speed may then be found by having another imaging system monitoring the anus of the patient and measuring the time it takes for a new marking to become visible.
  • the withdrawal speed is estimated to be 0.25cm/second.
  • the extra measurement system is only needed for training, i.e. the extra measurement system is not needed when using the machine learning data structure to estimate the withdrawal speed.
  • the machine learning data architecture is an artificial neural network such as a deep structured learning architecture.
  • the machine learning data architectures may be U-Net (reference: Ronneberger et el. "U-Net: Convolutional Networks for Biomedical Image Segmentation” (2015), arXiv:1505.04597 [cs.CV] or Mask R-CNN (reference: He et al. "Mask R-CNN” (2017), arXiv:1703.06870 [cs.CV].
  • the present disclosure relates to a display unit for displaying images obtained by an image capturing device of an endoscope, wherein the display unit comprises an image processing device as disclosed in relation to the first asepct.
  • the present disclosure relates to an endoscope system comprising an endoscope and an image processing device as disclosed in relation to the first aspect of the present disclosure, wherein the endoscope has an image capturing device and the processing unit of the image processing device is operationally connectable to the image capturing device of the endoscope.
  • the image processing device forms part of a display unit as disclosed in relation to the second aspect of the disclosure.
  • the present disclosure relates to a method for estimating a quality measure of a an endoscopic procedure performed using an endoscope, the endoscope comprising an image capturing device, wherein the method comprises:
  • the present disclosure relates to a computer program product comprising program code means adapted to cause a data processing system to perform the steps of the method as disclosed in relation to the fourth aspect of the present disclosure, when said program code means are executed on the data processing system.
  • said computer program product comprises a non-transitory computer-readable medium having stored thereon the program code means.
  • the different aspects of the present disclosure can be implemented in different ways including image processing devices, display units, endoscope systems, methods for estimating a quality measure, and computer program products described above and in the following, each yielding one or more of the benefits and advantages described in connection with at least one of the aspects described above, and each having one or more preferred embodiments corresponding to the preferred embodiments described in connection with at least one of the aspects described above and/or disclosed in the dependent claims. Furthermore, it will be appreciated that embodiments described in connection with one of the aspects described herein may equally be applied to the other aspects.
  • Fig. 1 illustrates an example of an endoscope 100.
  • This endoscope may be adapted for single-use.
  • the endoscope 100 is provided with a handle 102 attached to an insertion tube 104 provided with a bending section 106.
  • the endoscope 100 may also be designed without a bending section 106.
  • the illustrated endoscope 100 is a flexible endoscope with a flexible insertion tube 104, but the principle of the disclosure can also be used with any type of endoscope.
  • Some embodiments of the disclosure relate to colonoscopy procedures. Such procedures are typically performed using a colonoscope.
  • the insertion tube 104 as well as the bending section 106 may be provided with one or several working channels such that instruments, such as a gripping device or a catheter, may extend from the tip and be inserted into a human body via the endoscope.
  • One or several exit holes of the one or several channels may be provided in a tip part 108 of the endoscope 100.
  • a camera sensor such as a CMOS sensor or any other image capturing device, as well as one or several light sources, such as light emitting diodes (LEDs), fiber, or any other light emitting devices, may be placed in the tip part 108.
  • LEDs light emitting diodes
  • the display unit comprises a display 201, a processing unit 220 (only schematically shown), and a connection port 202 for operationally connecting the endoscope to the processing unit 220.
  • the connection port 202 may further be used to provide power to the endoscope.
  • the endoscope has a bending section 106 that can be bent in different directions with respect to the insertion tube 104.
  • the bending section 106 may be controlled by the operator by using a knob 110 placed on the handle 102.
  • the handle 102 illustrated in Fig. 1 is designed such that the knob 110 is controlled by a thumb of the operator, but other designs are also possible.
  • a push button 112 may be used.
  • the handle 102 illustrated in Fig. 1 is designed such that a index finger of the operator is used for controlling the gripping device, but other designs are also possible.
  • Fig. 3 show a schematic drawing of an image processing device 300 for estimating a quality measure of an endoscopic procedure performed using an endoscope (not shown) according to an embodiment of the disclosure.
  • the endoscope comprises an image capturing device.
  • the image processing device 300 comprises a processing unit 301 operationally connectable 302 to the image capturing device.
  • the processing unit 301 is configured to obtain a stream of images captured by the image capturing device of the endoscope.
  • the processing unit may be connectable to the image capturing device, e.g. by wire.
  • the image processing device may comprise a connection port for operationally connecting the endoscope to the processing unit 301.
  • the connection port may further be used to provide power to the endoscope.
  • the processing unit 301 may be wireless connectable to the image capturing device e.g. the image processing device 300 may comprise a wireless communication unit (not shown) configured to receive images from the image capturing device.
  • the processing unit 301 is further configured to process the stream of images to estimate the location of a lumen in the stream of images, and determine a quality measure 304 of the endoscopic procedure based on the estimated location of the lumen.
  • the processing unit may be configured to estimate the location of the lumen by processing the stream of images using a machine learning data architecture 305 trained to identify the location of lumens in endoscope images.
  • the machine learning data architecture may be stored in a memory unit 303 of the image processing device.
  • the quality measure 304 may be stored in the memory unit 303.
  • the quality measure may be provided 306 to a display (not) shown to be displayed to an operator, possibly together with endoscope images.
  • Fig. 4 shows a flowchart of a method 400 for estimating a quality measure of an endoscopic procedure performed using an endoscope, the endoscope comprising an image capturing device.
  • a stream of images captured by the image capturing device of the endoscope is obtained.
  • the stream of images is processed to estimate locations of a lumen in the stream of images.
  • a quality measure is determined of the endoscopic procedure based on the estimated locations of the lumen.
  • Fig. 5a-e show schematically images captured by an image capturing device of an endoscope during a colonoscopy and provided to the processing unit 301 of the image processing device 300 shown in Fig. 3 according to an embodiment of the present disclosure.
  • Fig. 5a is the lumen 510 located centrally. This is how an image may look when the endoscope is being withdrawn without investigating any particular area of the circumference of the colon.
  • Fig. 5b is the lumen 510 located in the upper right corner of the image. This is how an image may look when the lower left part of the circumference of the colon is being investigated.
  • Fig. 5c is the lumen 510 located in the lower right corner of the image.
  • Fig. 5d is the lumen 510 located in the lower left corner of the image. This is how an image may look when the upper right part of the circumference of the colon is being investigated.
  • Fig. 5e is the lumen 510 located in the upper left corner of the image. This is how an image may look when the lower right part of the circumference of the colon is being investigated.
  • the image processing device 300 of Fig. 3 may be configured to determine a quality measure of a colonoscopy.
  • the processing unit 301 may be further configured to divide the circumference of the colon into a plurality of areas, and based on the estimated location of the lumen in the stream of images estimate which area of the plurality of areas is being investigated, and for each area of the plurality of areas determine a quality measure.
  • the circumference of the colon may be divided into equally large areas e.g. four equally large areas or 8 equally large areas. As an example, the areas may be upper left, upper right, lower left, and lower right.
  • Each area of the plurality of areas may corresponds to an image zone of the stream of images and wherein the processing unit is configured to estimate that a particular area of the plurality of areas is being investigated if the estimated location of the lumen is arranged within the image zone of the particular area.
  • the circumference of the colon is divided into 4 equally large areas.
  • the areas are upper left, upper right, lower left, and lower right.
  • Each area of the plurality of areas corresponds to an image zone 501 502 503 504 of the stream of images.
  • the area upper left of the circumference corresponds to the image zone 504
  • the area upper right of the circumference corresponds to the image zone 503
  • the area lower left of the circumference corresponds to the image zone 502
  • the area lower right of the circumference corresponds to the image zone 501.
  • the processing unit 301 may be configured to estimate that a particular area of the plurality of areas is being investigated if the estimated location of the lumen is arranged within the image zone of the particular area.
  • the lumen 510 is located in the image zone 501 (see Fig. 5e ) then it will be determined that lower right area is being investigated, if the lumen 510 is located in the image zone 502 (see Fig. 5b ) then it will be determined that lower left area is being investigated, if the lumen 510 is located in the image zone 503 (see Fig. 5d ) then it will be determined that upper right area is being investigated, and if the lumen 510 is located in the image zone 504 (see Fig. 5c ) then it will be determined that the upper left area is being investigated.
  • the quality measure for each area may be based on the amount of time spend on investigating the area e.g. the quality measure may be based on one or more thresholds.
  • a first threshold may be used to determine the quality measure. Thus, if the medical professional uses more time than the first threshold on investigating an area, then a high quality measure may result. Additionally, a second first threshold may be used to determine the quality measure. Thus, if the medical professional uses more time than the second threshold but less than the first threshold on investigating an area, then a mediocre quality measure may result, and if the medical professional uses less time than the second threshold then a low quality measure may results.
  • Fig. 6 illustrates how a machine learning data architecture may be trained to estimate locations of a lumen in endoscope images. Shown is a single image 601 of a training data set. A typical training data set comprises endoscope images of different parts of the colon from the different procedures, i.e. endoscope images from a significant number of endoscope examinations. To train the machine learning data architecture a human operator indicates the location of the lumen 603. The location of the lumens is indicated by drawing up the circumference of the lumen 603.
  • Fig. 7 illustrates how a lumen may be identified in endoscope images according to an embodiment of the disclosure.
  • the lumen is identified using a machine learning data architecture trained to estimate the location of a lumen in endoscope images as disclosed in relation to Fig. 6 .
  • the input image 701 is shown to the right and the output from the machine learning data architecture 702 is shown to the left.
  • the machine learning data architecture has been trained to estimate the circumference of lumen, not only the location of the lumen 703 is estimated but also the circumference of the lumen 703.
  • Fig. 8 shows a flowchart of a method 800 for estimating quality measures of a coloscopy procedure performed using an endoscope, the endoscope comprising an image capturing device.
  • step 801 an image captured by the image capturing device of the endoscope is obtained.
  • step 802 the obtained image is processed to estimate the withdrawal speed of the endoscope.
  • step 803 the estimated withdrawal speed is used to estimate in which section of a plurality of section of the colon the endoscope is located.
  • Each section of the plurality of sections may have a predetermined length e.g. each section may have a length of 5cm, 10cm or 15cm.
  • step 804 the image is processed to estimate the location of a lumen.
  • step 805 it is estimated what area of a plurality of areas of the circumference of the colon is being investigated based on the estimated location of the lumen e.g. in the same way as disclosed in relation to Figs. 5a-e .
  • the quality measure for the estimated area of the circumference of the colon for the estimated section is updated in step 806.
  • the quality measure for each area is based on the amount of time spend on investigating the area, then the quality measure for the estimated area of the circumference of the colon for the estimated section may be updated based on how often a new image is obtained e.g. if a new image is obtained with a frequency of 60Hz then the amount of time spend may be increased with 1/60 seconds.
  • the frequency with which a new image is obtained may be the same as the frequency of the image capturing device but it may also be lower e.g. only every second or every fourth captured image may be processed to estimate a quality measure.
  • the final output of the method after the colonoscopy has been completed is a quality measure for each area of the circumference of the colon for each section of the colon. Thus, if the colon is divided into 10 sections and the circumference of the colon is divided into 4 areas, then a total of 40 quality measures will be estimated. This will allow the medical professional not only to obtain an overall evaluation of the quality of the procedure, but a detailed map of the quality of the different parts of the procedure.
  • steps 804 and 805 may be performed before steps 802 and 803.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • Pathology (AREA)
  • General Health & Medical Sciences (AREA)
  • Veterinary Medicine (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Public Health (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biophysics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Optics & Photonics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Endoscopes (AREA)
EP21212304.6A 2021-12-03 2021-12-03 Verarbeitungsvorrichtung für endoskopbilder Withdrawn EP4190271A1 (de)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP21212304.6A EP4190271A1 (de) 2021-12-03 2021-12-03 Verarbeitungsvorrichtung für endoskopbilder
US18/074,436 US20230172428A1 (en) 2021-12-03 2022-12-02 Endoscope image processing device
US18/206,935 US20240021103A1 (en) 2021-12-03 2023-06-29 Endoscopic training system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP21212304.6A EP4190271A1 (de) 2021-12-03 2021-12-03 Verarbeitungsvorrichtung für endoskopbilder

Publications (1)

Publication Number Publication Date
EP4190271A1 true EP4190271A1 (de) 2023-06-07

Family

ID=78822002

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21212304.6A Withdrawn EP4190271A1 (de) 2021-12-03 2021-12-03 Verarbeitungsvorrichtung für endoskopbilder

Country Status (1)

Country Link
EP (1) EP4190271A1 (de)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2006138504A2 (en) * 2005-06-17 2006-12-28 Mayo Foundation For Medical Education And Research Colonoscopy video processing for quality metrics determination
US20110301447A1 (en) * 2010-06-07 2011-12-08 Sti Medical Systems, Llc Versatile video interpretation, visualization, and management system
US20180296281A1 (en) * 2017-04-12 2018-10-18 Bio-Medical Engineering (HK) Limited Automated steering systems and methods for a robotic endoscope
WO2020160567A1 (en) * 2019-04-05 2020-08-06 Carnegie Mellon University Real-time measurement of visible surface area from colonoscopy video
US20200279373A1 (en) * 2019-02-28 2020-09-03 EndoSoft LLC Ai systems for detecting and sizing lesions

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2006138504A2 (en) * 2005-06-17 2006-12-28 Mayo Foundation For Medical Education And Research Colonoscopy video processing for quality metrics determination
US20110301447A1 (en) * 2010-06-07 2011-12-08 Sti Medical Systems, Llc Versatile video interpretation, visualization, and management system
US20180296281A1 (en) * 2017-04-12 2018-10-18 Bio-Medical Engineering (HK) Limited Automated steering systems and methods for a robotic endoscope
US20200279373A1 (en) * 2019-02-28 2020-09-03 EndoSoft LLC Ai systems for detecting and sizing lesions
WO2020160567A1 (en) * 2019-04-05 2020-08-06 Carnegie Mellon University Real-time measurement of visible surface area from colonoscopy video

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
HE ET AL.: "Mask R-CNN", ARXIV:1703.06870, 2017
RONNEBERGER ET AL.: "U-Net: Convolutional Networks for Biomedical Image Segmentation'' (2015), arXiv:1505.04597 [cs.CV] or Mask R-CNN (reference: He et al. ''Mask R-CNN", ARXIV:1703.06870, 2017
RONNEBERGER: "U-Net: Convolutional Networks for Biomedical Image Segmentation", ARXIV:1505.04597
SAE HWANG ET AL: "Automatic measurement of quality metrics for colonoscopy videos", 13TH ANNUAL ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA - 6-11 NOV. 2005 - SINGAPORE, ACM, NEW YORK, NY, USA, 6 November 2005 (2005-11-06), pages 912 - 921, XP058250160, ISBN: 978-1-59593-044-6, DOI: 10.1145/1101149.1101343 *

Similar Documents

Publication Publication Date Title
US11074690B2 (en) Method for enhancing the visibility of blood vessels in color images and visualization systems implementing the method
US20150313445A1 (en) System and Method of Scanning a Body Cavity Using a Multiple Viewing Elements Endoscope
EP3357406A1 (de) Bildverarbeitungsvorrichtung, endoskopsystem und bildverarbeitungsverfahren
CN110913746B (zh) 诊断辅助装置、诊断辅助方法及存储介质
KR20170055526A (ko) 방광의 진단 맵핑을 위한 방법 및 시스템
JPWO2014168128A1 (ja) 内視鏡システム及び内視鏡システムの作動方法
JP7125479B2 (ja) 医療画像処理装置、医療画像処理装置の作動方法及び内視鏡システム
CN106473807A (zh) 使用回溯迷宫问题解决方案的自动ent外科手术预先计划
CN111105873B (zh) 经内镜逆行胰胆管造影术中辅助诊断和测量方法及***
US20200237200A1 (en) A digital device facilitating body cavity screening and diagnosis
CN114980793A (zh) 内窥镜检查辅助装置、内窥镜检查辅助装置的工作方法以及程序
CN113271839A (zh) 内窥镜装置和内窥镜***
JP7441934B2 (ja) 処理装置、内視鏡システム及び処理装置の作動方法
EP4190271A1 (de) Verarbeitungsvorrichtung für endoskopbilder
US20230255461A1 (en) Endoscope image processing device
EP4190270A1 (de) Verarbeitungsvorrichtung für endoskopbilder
CN114730478A (zh) 用于处理电子医学图像以确定增强的电子医学图像的***和方法
EP4191598A1 (de) Verarbeitungsvorrichtung für endoskopbilder
JP2021194268A (ja) 血管観察システムおよび血管観察方法
CN112672677A (zh) 有助于体腔检查和诊断的数字设备
WO2023175732A1 (ja) 内視鏡検査支援システム、内視鏡検査支援方法および記憶媒体
US20200100842A1 (en) Systems and methods for image-guided medical procedures
EP4191527A1 (de) Vorrichtung zur verarbeitung von endoskopbildern
Sánchez et al. Tracheal ring detection in bronchoscopy

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20231208