US20230218145A1 - Endoscopic system and method for displaying an adaptive overlay - Google Patents
Endoscopic system and method for displaying an adaptive overlay Download PDFInfo
- Publication number
- US20230218145A1 US20230218145A1 US18/183,683 US202318183683A US2023218145A1 US 20230218145 A1 US20230218145 A1 US 20230218145A1 US 202318183683 A US202318183683 A US 202318183683A US 2023218145 A1 US2023218145 A1 US 2023218145A1
- Authority
- US
- United States
- Prior art keywords
- image
- video
- video image
- predetermined area
- endoscopic system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims description 21
- 230000003044 adaptive effect Effects 0.000 title description 2
- 230000005284 excitation Effects 0.000 claims description 13
- 238000001429 visible spectrum Methods 0.000 claims description 8
- 230000005670 electromagnetic radiation Effects 0.000 claims description 6
- 238000005286 illumination Methods 0.000 claims description 5
- 230000002093 peripheral effect Effects 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000005855 radiation Effects 0.000 claims 8
- 239000002131 composite material Substances 0.000 claims 3
- 238000002329 infrared spectrum Methods 0.000 claims 2
- 238000001228 spectrum Methods 0.000 claims 1
- MOFVSTNWEDAEEK-UHFFFAOYSA-M indocyanine green Chemical compound [Na+].[O-]S(=O)(=O)CCCCN1C2=CC=C3C=CC=CC3=C2C(C)(C)C1=CC=CC=CC=CC1=[N+](CCCCS([O-])(=O)=O)C2=CC=C(C=CC=C3)C3=C2C1(C)C MOFVSTNWEDAEEK-UHFFFAOYSA-M 0.000 description 10
- 229960004657 indocyanine green Drugs 0.000 description 10
- 238000001356 surgical procedure Methods 0.000 description 6
- 206010028980 Neoplasm Diseases 0.000 description 4
- 230000017531 blood circulation Effects 0.000 description 4
- 208000035250 cutaneous malignant susceptibility to 1 melanoma Diseases 0.000 description 4
- 201000001441 melanoma Diseases 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000003247 decreasing effect Effects 0.000 description 3
- 239000000975 dye Substances 0.000 description 3
- 210000003484 anatomy Anatomy 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000010412 perfusion Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004040 coloring Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- GNBHRKFJIUUOQI-UHFFFAOYSA-N fluorescein Chemical compound O1C(=O)C2=CC=CC=C2C21C1=CC=C(O)C=C1OC1=CC(O)=CC=C21 GNBHRKFJIUUOQI-UHFFFAOYSA-N 0.000 description 1
- 238000012632 fluorescent imaging Methods 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 102000004169 proteins and genes Human genes 0.000 description 1
- 108090000623 proteins and genes Proteins 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00043—Operational features of endoscopes provided with output arrangements
- A61B1/00045—Display arrangement
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
- A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00039—Operational features of endoscopes provided with input arrangements for the user
- A61B1/0004—Operational features of endoscopes provided with input arrangements for the user for electronic operation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00043—Operational features of endoscopes provided with output arrangements
- A61B1/00045—Display arrangement
- A61B1/0005—Display arrangement combining images e.g. side-by-side, superimposed or tiled
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
- A61B1/043—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances for fluorescence imaging
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
- A61B1/0638—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements providing two or more wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/555—Constructional details for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
Definitions
- the video image may be partitioned into four displays.
- the first display also referenced as the main display as the first display is larger than the other displays, is shown on the right side of the video image depicting an enlarged overlay image.
- the surgical site is displayed using both reflected white light and emitted fluorescent light that has been colored by image processing, which helps features such as blood flow, tumors, malignant melanoma and the like become more visible.
- On the left side are three different images.
- the top image shows a white light image of the surgical site.
- the middle image shows a grayscale image of the fluorescent light of the surgical site.
- the grayscale image is due to the fluorescence occurring in a wavelength outside the visible spectrum yet detected as luminance by the image sensor.
- the video display system includes a first video image displaying a white light video of a surgical site.
- the first video image has a first predetermined area.
- a second video image is overlaid on the first video image.
- the second video image is displayed as a fluorescent light video depicting, for example, ICG fluorescence.
- the second video image is centered on the surgical site and includes a second predetermined area.
- the second predetermined area is smaller than the first predetermined area to define a boundary of white light video.
- FIG. 6 shows the tool 26 being introduced into an upper right hand corner of the video image 14 .
- the tool 26 is detected by the image processor 113 and the second video image 20 is moved over to the upper right hand corner of the video image 14 so as to be off-center with respect to the first video image 16 .
- the white light boundary 24 is overshadowed by the second video image 20 with respect to the upper right hand corner of the video display. Alternately, the white light boundary 24 may overshadow the second video image 20 by including a minimum boundary size. Left of the second video image 20 and below the second video image 20 , additional portions of the first video image 16 may be revealed.
- the second video image 20 may be moved to any location within the field of view of the image sensor 108 so as to provide the surgeon with details regarding tissue perfusion and the like in any area of the field of view of the endoscope 102 .
- Any input 28 currently known or later developed may be modified for use herein, illustratively including a mouse, voice command, eyeglasses, keyboard or the like.
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Biophysics (AREA)
- Physics & Mathematics (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Optics & Photonics (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Endoscopes (AREA)
Abstract
The present disclosure is directed towards a video display system and an endoscopic system configured to provide the surgeon with an optimized video image based upon user preference. The video display system includes a first video image displaying a white light video of a surgical site. The first video image has a first predetermined area. A second video image is overlaid on the first video image. The second video image is displayed in a fluorescent light video. The second video image is centered on the surgical site and includes a second predetermined area. The second predetermined area is smaller than the first predetermined area so as to define a boundary of white light video. Thus, the video display system provides the surgeon with the ability to reference the location of the fluorescent light video with respect to anatomical features of the surgical site.
Description
- This application is a continuation of U.S. patent application Ser. No. 16/400,255, filed on May 1, 2019, and entitled “Video Display System Having an Adaptive Overlay,” the content of which is incorporated herein by reference in its entirety.
- The disclosure relates to a video display system and method having a first video image and a second video image overlaid on the first video image to facilitate a surgical procedure.
- Endoscopes are commonly used to provide access to body cavities while decreasing the invasiveness of a surgical procedure. Certain endoscopes include a video display for displaying the surgical site. It is known to have the video display provide a first video image which displays the surgical site using reflected white light and a second video image which displays the surgical site using emitted fluorescent light, for example when using a fluorescing dye such as indocyanine green (“ICG”) with an excitation light such as near-infrared light (NIR).
-
FIG. 1 shows an embodiment of a video display system. The video display system displays video captured by an endoscope. The endoscope is configured to capture the surgical site in both reflected white light due to white illumination light and emitted fluorescent light due to, for example, NIR excitation light. The fluorescent light is illustratively depicted as fluorescence due to excitation of ICG. The video display system includes a display device, which outputs a video image. - The video image may be partitioned into four displays. The first display, also referenced as the main display as the first display is larger than the other displays, is shown on the right side of the video image depicting an enlarged overlay image. Here, the surgical site is displayed using both reflected white light and emitted fluorescent light that has been colored by image processing, which helps features such as blood flow, tumors, malignant melanoma and the like become more visible. On the left side are three different images. The top image shows a white light image of the surgical site. The middle image shows a grayscale image of the fluorescent light of the surgical site. The grayscale image is due to the fluorescence occurring in a wavelength outside the visible spectrum yet detected as luminance by the image sensor. The middle image may show another image corresponding to a wavelength range outside or inside the visible spectrum as known in procedures involving photodynamic diagnosis (PDD), autofluorescence, fluorescein, and the like. In any of these, the middle image may be pseudo-colored by assigning colors to gray values based on specific criterion. The bottom image shows the overlay image that appears enlarged on the right. The overlay image shows how overlaying the a pseudo-colored fluorescent light image on top of the white light image may highlight areas of ICG concentration but may also obscure anatomical features of the white light image where excess ICG concentration occurs.
- The system allows the user to select between one of the three images shown on the main display. As shown, such systems require the surgeon to view any one of the three types of display in the main display. For instance, the user may select the white light video, the grayscale video, or the overlay video. In some instances, it is desirable to view the white light video in the main display and reference the overlay video on one of the partitioned displays, or vice-versa as the overlay video obscures certain anatomical features clearly shown in the white light video.
- Accordingly, it remains desirable to have a system that optimizes the camera video images by allowing the surgeon to manipulate the overlay to provide a video image, which is customized to the surgeon's preference.
- A video display system for use in a medical procedure and an endoscopic system is provided. The video display system includes a first video image displaying a white light video of the surgical site. The first video image has a first predetermined area. A second video image is overlaid on top of the first video image. The second video image displays a fluorescent light video, which is centered on the surgical site. The second video image has a second predetermined area, which is smaller than the first predetermined area to define a boundary of white light video, which surrounds the second video image. As such, the surgeon is able to view the surgical site in both a fluorescent light video image and a white light video image wherein the white light video image is able to point out anatomical features so as to allow the surgeon to quickly reference where the fluorescent light video image is with respect to the anatomical features of the surgical site.
- In one aspect of the video display system and endoscopic system the boundary is a pixelated line what surrounds an entire periphery of the second video image wherein an input may be provided to adjust the thickness of the boundary.
- In another aspect of the disclosure, the size of the second video image may be adjusted to increase or decrease the area of the first video image. In another aspect of the disclosure, the transparency of the second video image may be adjusted.
- In another aspect of the disclosure, the video display system may include an image processor configured to detect a tool. In particular, the image processor may be configured to detect a cutting tool, wherein the image processor detects the presence of a cutting tool as the cutting tool enters the video display. In yet another aspect of the disclosure, the video display system may be configured to automatically center the second video image on the tool to move the second display with respect to the movement of the tool within the surgical site.
- The embodiments set forth in the drawings are illustrative and exemplary in nature and not intended to limit the subject matter defined by the claims. The following description of the illustrative embodiments can be understood when read in conjunction with the following drawings, where like structure is indicated with like reference numerals and in which:
-
FIG. 1 is a video display system of the prior art. -
FIG. 2 is a block diagram of an endoscopic system according to one or more embodiments described herein. -
FIG. 3 is a video display showing the first and second video images. -
FIG. 4 is an illustrative view of the video display ofFIG. 3 showing the thickness of the boundary being modified relative toFIG. 3 . -
FIG. 5 is an illustrative view of the video display ofFIG. 3 showing dimensions of the second video image enlarged relative toFIG. 3 . -
FIG. 6 shows an embodiment of the video display wherein a tool is introduced into the video image. -
FIG. 7 illustratively depicts the second video image tracking the movement of the tool relative toFIG. 6 . - Referring generally to the figures, embodiments of the present disclosure directed towards a video display system and an endoscopic system configured to generate a video image of a surgical site is provided. The video display system includes a first video image displaying a white light video of a surgical site. The first video image has a first predetermined area. A second video image is overlaid on the first video image. The second video image is displayed as a fluorescent light video depicting, for example, ICG fluorescence. The second video image is centered on the surgical site and includes a second predetermined area. The second predetermined area is smaller than the first predetermined area to define a boundary of white light video. Thus, the video display system provides the surgeon with the ability to reference the location of the fluorescent light video with respect to anatomical features of the surgical site.
- With reference first to
FIG. 2 , thevideo display system 10 is illustratively shown. For illustrative purposes, a description ofvideo display system 10 is provided within the context of an endoscopic system 100. However, it should be appreciated that thevideo display system 10 may be utilized in other applications, illustratively including an exoscope, borescopes and other systems having two or more illumination-types and one or more image sensors. - Furthermore, although the
video display system 10 is described with respect to medical applications using fluorescing dye, it should be understood that industrial applications using other combinations of illumination or excitation light types such as white light, nonvisible light (ultraviolet, infrared, or other), or light within narrow wavelength ranges (visible or nonvisible) may benefit from the same principles. - The endoscopic system 100 includes an endoscope 102 having a camera head unit 104 coupled to the endoscope 102 by optics or optical fiber. The camera head unit 104 is in communication with a camera controller unit 106, through either cable or a wireless connection.
- The camera control unit 106 controls various processing functions of the camera head unit 104. The camera controller unit 106 may provide a timing signal to the camera head unit 104 to control the actuation of an
image sensor 108 and alight source 110 and process the image data from theimage sensor 108 to generate a video stream. Alternatively, thelight source 110 may generate a timing signal independently of the camera controller unit 106 such that the camera controller unit 106 and thelight source 110 operate independently. - The
light source 110 is configured to illuminate the surgical site of the endoscope 102 and theimage sensor 108 is configured to collect image data from thelight source 110. Thelight source 110 may be formed of one or more light-emitting diodes. Thelight source 110 may be configured to generate electromagnetic radiation in the visible spectrum commonly called white light and electromagnetic radiation outside the visible spectrum such as near infrared (NIR) excitation light so as to provide for fluorescent imaging such as an indocyanine green (ICG). Other light-emitting diodes may be used to produce various other excitation lights at wavelengths associated with other dyes and auto-fluorescing tissues, proteins, and chemicals of differing procedures. - In some examples, the
light source 110 is configured to switch between illuminating with white light and excitation light. Theimage sensor 108 is actuated to collect image data for both the white light and excitation light sequences. In other examples, thelight source 110 is configured to illuminate with white light and excitation light simultaneously. Theimage sensor 108 is configured to provide an image of the body cavity thereby allowing the surgeon to locate the treatment area and monitor the surgical procedure. - The
image sensor 108 can be a complementary metal oxide semiconductor “CMOS” or a charged coupled device “CCD”. It should be appreciated that anypixelated image sensor 108 currently known or later developed may be modified and adopted for use herein. In one embodiment, theimage sensor 108 is configured to receive electromagnetic radiation in the visible spectrum and in an infrared range between about 800 nanometers to 1200 nanometers associated with a particular field of view. In another aspect, the endoscopic system 100 may include a pair ofimage sensors 108, wherein one of theimage sensors 108 is configured to receive electromagnetic radiation in the visible spectrum with a particular field of view and the other of theimage sensors 108 is configured to receive electromagnetic radiation in an infrared range between about 800 nanometers to 1200 nanometers associated with the same particular field of view. One skilled in the art would recognize that various systems using combinations of one ormore image sensors 108 may benefit from the principles of the present disclosure. - The
video display system 10 includes adisplay unit 12. Thedisplay unit 12 is coupled to thedisplay controller unit 112. Generally speaking, theimage sensor 108 delivers image data to the camera controller unit 106, which processes the image data, and outputs processed image data to thedisplay controller unit 112. Thedisplay controller unit 112 may arrange videos and images based on the processed image data on themonitor 12 in various ways. In particular, the camera controller unit 106 may provide the image data to the display controller, which includes animage processor 113. Theimage processor 113 further configures the processed image data collected by theimage sensor 108 from both reflected white light and emitted fluorescent light to generate avideo image 14, in particular, a white light video image and a fluorescent light video image. As thelight source 110 switches between white light and excitation light in synchronization with the exposure and operation of theimage sensor 108, the white light and fluorescent light videos are simultaneously displayed to the display unit. - With reference now to
FIG. 3 , the video image displayed by thedisplay unit 12 is provided.FIG. 3 shows thedisplay unit 12 displaying thevideo image 14. The video image includes afirst video image 16 displaying a white light video having a predetermined field of view of a surgical site, the field of view may be determined by the size of theimage sensor 108 and the function of theimage processor 113 as is known by those skilled in the art. For instance, the field of view of theimage sensor 108 may be purposefully narrowed by digital cropping performed by theimage processor 113. Thefirst video image 16 includes a firstpredetermined area 18, which defines a first width “W1”, and a first height “H1” of thevideo image 14. - The
video image 14 further includes asecond video image 20. Thesecond video image 20 is overlaid on thefirst video image 16 wherein the images of thesecond video image 20 are aligned to the images of thefirst video image 16. Thesecond video image 20 displays a fluorescent light video, which is centered on the surgical site and aligned with thefirst video image 16. Thesecond video image 20 has a secondpredetermined area 22, which defines a second width “W2” and a second height “H2” of thevideo image 14. The secondpredetermined area 22 is smaller than the firstpredetermined area 18 so as to define aboundary 24 of white light video surrounding thesecond video image 20. Thus, the surgeon can readily identify anatomical features, shown in white light, surrounding the surgical site with respect to the anatomical features such as blood flow, tumors, malignant melanoma and the like which are better shown in fluorescent light video. - For instance, it is known that ICG distribution within the tissue enables intraoperative evaluation of tissue perfusion and vacuolization, identification of critical neurovascular structures and differentiation of tissue plains between lesions and adjacent structures. By maintaining the
white light boundary 24 of white light video, critical anatomical features are readily identifiable which ordinarily may be obscured by the overlay of the fluorescent image or require the surgeon to switch to a white light only mode. Such visualization may obscure anatomical features, which are necessary for the surgeon to identify so as to ensure a tool is placed in the proper position. As such,FIG. 3 depicts avideo display system 10, which allows the surgeon to manipulate a cutting tool 26 (not shown) within the surgical site and providing anatomical references to help the surgeon place thetool 26 in the proper location. - With reference now to
FIG. 4 an aspect of thevideo display system 10 is provided wherein thewhite light boundary 24 surrounds the peripheral edges of thesecond video image 20.FIG. 4 depicts an aspect of thevideo display system 10 wherein aninput 28 may be used to differentiate theboundary 24 of white light video from fluorescent light video.FIG. 4 shows theinput 28 being configured to increase the thickness of a pixelated boundary, which defines aframe 30. - In particular, the
frame 30 delineating thewhite light boundary 24 from thesecond video image 20 is thickened relative to what is shown inFIG. 3 . Theframe 30 is shown in a color different from the pseudo-coloring used with the fluorescent light video shown in thesecond video image 20. Theinput 28 may be further configured to adjust the transparency of theframe 30, change a color of theframe 30, or remove theframe 30 in its entirety. Thus, thevideo image 14 may be optimized to the surgeon's preference to help the surgeon perform the surgical procedure. -
FIG. 5 depicts another aspect of thevideo display system 10 wherein thesecond video image 20 is enlarged with respect to thevideo image 14 shown inFIGS. 3 and 4 .FIG. 5 shows thesecond video image 20 occupying a greater area relative to thesecond video image 20 shown inFIG. 3 . The adjustment of thesecond video image 20 may be done by theimage processor 113. The size of the secondpredetermined area 22 may be programmed. Thevideo display system 10 may be configured to adjust the size of the programmed second predeterminedarea 22 to accommodate the surgeon's preference. The transparency of thesecond video image 20 may also be adjusted in addition to or irrespective of the size of the secondpredetermined area 22. - In one aspect, the
video display system 10 includes a first image sensor 108 a and a second image sensor 108 b. The first image sensor 108 a has a predetermined pixel array configured to have a predetermined field of view. The first image sensor 108 a is configured to process white light image. The second image sensor 108 b has a pixel array of the same dimension as the first image sensor 108 a and is configured to have the same field of view as the first image sensor 108 a. The second image sensor 108 b is configured to gather near infrared image data such as for ICG applications. - In such an embodiment, the
image processor 113 crops the image generated by the second image sensor 108 b so as to produce thesecond video image 20 which is smaller in size relative to thefirst video image 16. As the first and second image sensors 108 a, 108 b are of the same dimension and have the same field of view, cropping the boundaries of thesecond video image 20 reduces the size. Theimage processor 113 then centers thesecond video image 20 with respect to thefirst video image 16. - In another embodiment, a
single image sensor 108 may be used. In such an embodiment, the size of thesecond video image 20 is again cropped by theimage processor 113. Thus, theinput 28 may be configured to provide options for enlarging or decreasing thesecond video image 20 relative to thefirst video image 16. As asingle image sensor 108 is used in this embodiment, thesecond video image 20 is always centered with respect to thefirst video image 16. - In another aspect of the video system, the first image sensor 108 a is configured to capture white light image data so as to generate the
first video image 16 as a white light video. The first image sensor 108 a may generate thefirst video image 16 with a first predetermined pixel matrix having a first resolution. The second image sensor 108 b is configured to capture near infrared image data so as to generate thesecond video image 20, which is a fluorescent light video. The second image sensor 108 b has a second predetermined pixel matrix having a second resolution less than the first resolution. The second image sensor 108 b may be positioned relative to various optical features (prisms, lenses, and beam splitters) such that thesecond video image 20 captures light at the same focal distance. The first image sensor 108 a and the second image sensor 108 b are arranged so as to be centered on the same point, thus the center of each of the first andsecond video images second video image 20. Accordingly, it should be appreciated that any methods currently known or later developed may be used to produce asecond video image 20 of colored light which is smaller than thefirst video image 16 of white light and the methods and systems described herein are provided for illustrative purposes and are not limiting to the scope of the appended claims. - With reference now to
FIGS. 6 and 7 , an aspect of thevideo display system 10 is provided wherein thevideo display system 10 may be configured to move thesecond video image 20 with respect to asurgical tool 26. For illustrative purposes, thetool 26 is shown as acutting tool 26 configured to remove tissue. However, it should be appreciated that any other tool used in a surgical procedure may be detected. In such an embodiment, theimage processor 113 may be configured to detect thetool 26. Thevideo display system 10 may include adatabase 114 storing a plurality ofdifferent tools 26, wherein through known processes such as edge detection, theimage processor 113 may detect the presence of thetool 26 within thevideo image 14. In such an embodiment, thevideo display system 10 is configured to move and adjust thevideo image 14 with respect to the movement of thetool 26 as described in greater detail below. -
FIG. 6 shows thetool 26 being introduced into an upper right hand corner of thevideo image 14. Thetool 26 is detected by theimage processor 113 and thesecond video image 20 is moved over to the upper right hand corner of thevideo image 14 so as to be off-center with respect to thefirst video image 16. Thewhite light boundary 24 is overshadowed by thesecond video image 20 with respect to the upper right hand corner of the video display. Alternately, thewhite light boundary 24 may overshadow thesecond video image 20 by including a minimum boundary size. Left of thesecond video image 20 and below thesecond video image 20, additional portions of thefirst video image 16 may be revealed. For example, thesecond video image 20 may be a fixed width “W2” and a fixed height “H2” that shifts with movement of thetool 26. However, a minimumwhite light boundary 24 may remain, regardless of the movement of thesecond video image 20. In addition, thewhite light boundary 24 may increase in areas of thevideo image 14 further away from thetool 26. -
FIG. 7 shows thesecond video image 20 being moved to the center of thevideo image 14 in correlation to the movement of thecutting tool 26. It should be appreciated that thesecond video image 20 may move to other areas within thevideo image 14 upon movement of thecutting tool 26. - The
video display system 10 may include aninput 28 which allows for the user to choose one of two modes of operation wherein in a first mode of operation thesecond video image 20 detects and automatically centers itself around thetool 26 upon the detection of thetool 26 and in a second mode of operation, thesecond video image 20 remains stationary with respect to thefirst video image 16 and stays centered with respect to thefirst video image 16. - In yet another aspect, the
second video image 20 may be moved to any location within the field of view of theimage sensor 108 so as to provide the surgeon with details regarding tissue perfusion and the like in any area of the field of view of the endoscope 102. Anyinput 28 currently known or later developed may be modified for use herein, illustratively including a mouse, voice command, eyeglasses, keyboard or the like. - Accordingly, a
video display system 10 and an endoscopic system 100 is provided which helps the surgeon conduct a surgical procedure by providing the surgeon with a visible reference of the surgical area. In particular, thevideo image 14 includes afirst video image 16 having a firstpredetermined area 18 and asecond video image 20 overlaid on top of thefirst video image 16. Thesecond video image 20 has a secondpredetermined area 22, which is smaller than the firstpredetermined area 18 so as to form aboundary 24 of white light video image. Theboundary 24 of white light video image provides a clearer image of the anatomy of the surgical site relative to the fluorescent light video image of thesecond video image 20. However, thesecond video image 20 provides details as to features relating to the treatment of the surgical site such as blood flow, tumors, malignant melanoma and the like. Thus, thevideo display system 10 and the endoscopic system 100 location of thetool 26 with respect to the treatment site may be easily done by reference to theboundary 24 of the anatomy shown in a white light image. - Further, the
video display system 10 and the endoscopic system 100 disclosed herein is customizable based upon the user preference, wherein the dimension of thesecond video image 20 may be increased or decreased to suit the surgeon's preference. For instance, thesecond video image 20 may be reduced to show only the area surrounding the surgical area, e.g. around thetool 26, alternatively thesecond video image 20 may be enlarged so as to cover the entirety offirst video image 16. Thesecond video image 20 may also be adjusted to increase or decrease the transparency of thesecond video image 20. For instance, it may be preferable to reduce the size of thesecond video image 20 but increase the intensity of thesecond video image 20 so as to better show features such as blood flow, tumors, malignant melanoma and the like. - Further, the
video display system 10 and the endoscopic system 100 disclosed herein may be configured to detect atool 26. In such an aspect, thesecond video image 20 is moved to the position of thetool 26. In another aspect, thesecond video image 20 may be manually moved by the surgeon relative to thefirst video image 16, such as what is illustratively shown inFIG. 6 . Although,FIG. 6 is provided to show the automatic movement of thesecond video image 20 is response to the detection of atool 26, it should be appreciated that the teachings ofFIG. 6 is not limited to the feature of detecting atool 26. - While particular embodiments have been illustrated and described herein, it should be understood that various other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter. Moreover, although various aspects of the claimed subject matter have been described herein, such aspects need not be utilized in combination. It is therefore intended that the appended claims cover all such changes and modifications that are within the scope of the claimed subject matter.
Claims (20)
1. An endoscopic system configured to generate a video image of a surgical site, the endoscopic system comprising:
a light source configured to transmit electromagnetic radiation in a first visible spectrum and in a second spectrum selected to include excitation wavelengths for exciting a fluorophore;
an imager configured to acquire an image data, the imager comprising a first image sensor, a beam splitter, and a second image sensor, where the first and second image sensors are configured to capture light at the same focal distance;
an image processor configured generate a white light image of the surgical site from the first image sensor, the white light image having a first predetermined area, and to generate an emission radiation image of the surgical site from the second image sensor, the emission radiation image data having a second predetermined area, and to generate from the white light image and the emission radiation image, a display image comprising a false colored emission radiation image overlaid on the white light image, where the second predetermined area is smaller than the first predetermined area; and
a display for displaying the display image.
2. The endoscopic system of claim 1 , wherein the second image sensor is configured to capture image data in the near infrared spectrum.
3. The endoscopic system of claim 1 , wherein a field of view of the second image sensor is smaller than a field of view of the first image sensor.
4. The endoscopic system of claim 1 , wherein a field of view of the first image sensor is the same as the field of view of the second image sensor, and wherein the second predetermined area is made smaller than the first predetermined area by the image processor cropping the emission radiation image.
5. The endoscopic system of claim 1 , wherein the first image sensor has a higher resolution than the second image sensor.
6. The endoscopic system of claim 1 , wherein a boundary comprising portions of the white light image surrounds second predetermined area, creating thereby a peripheral region consisting only of the white light image.
7. The endoscopic system of claim 6 , wherein the display image includes a frame bounding the peripheral region, visually separating it, thereby from the second predetermined area.
8. The endoscopic system of claim 1 , further including an input, the input configured to adjust a transparency of the false color emission radiation image.
9. The endoscopic system of claim 1 , further including an image processor configured to detect a tool, wherein the image processor is further configured to center the second predetermined area.
10. The endoscopic system of claim 1 , further including an input configured to adjust a size of the second predetermined area.
11. The endoscopic system of claim 1 , further including an input configured to move the second video image with respect to the first video image.
12. A method for displaying video imagery collected during a medical procedure, comprising the steps of:
collecting a first video image comprising a white light image of a surgical site;
collecting a second video image consisting of an emission radiation image of the surgical scene;
processing the second video image such that the emission radiation image is mapped to a color in the visible spectrum, creating a pseudo-color representation of the second video image;
displaying a composite image on an image display, wherein the composite image comprises the first video image registered to and overlaid with the processed second video image,
wherein, the processed second video image is contained within a second predetermined area of the display which is smaller than a first predetermined area that contains the first video image.
13. The method of claim 12 , wherein the step of collecting a first video image comprises the step of illuminating the surgical site with a broad spectrum visible light illumination.
14. The method of claim 12 , wherein the step of collecting a second video image comprises the step of illuminating the surgical site with an excitation illumination.
15. The method of claim 14 , wherein the excitation illumination comprises at least a portion of the near infrared spectrum.
16. The method of claim 12 , comprising the further step of providing a visible border between the first predetermined area and the second predetermined area.
17. The method of claim 12 , comprising the further step of identifying, by means of an image processor, if there is a tool present in the first video image.
18. The method of claim 17 , comprising the further step of positioning the second video image such that it contains at least a portion of the identified tool.
19. The method of claim 18 , comprising the further step of detecting movement of identified tool, and repositioning the second video image such that it continues to contain at least a portion of the identified tool.
20. The method of claim 12 , comprising the further step of adjusting a transparency value of the second video image prior to the step of displaying the composite image.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/183,683 US20230218145A1 (en) | 2019-05-01 | 2023-03-14 | Endoscopic system and method for displaying an adaptive overlay |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/400,255 US11730340B2 (en) | 2019-05-01 | 2019-05-01 | Video display system having an adaptive overlay |
US18/183,683 US20230218145A1 (en) | 2019-05-01 | 2023-03-14 | Endoscopic system and method for displaying an adaptive overlay |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/400,255 Continuation US11730340B2 (en) | 2019-05-01 | 2019-05-01 | Video display system having an adaptive overlay |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230218145A1 true US20230218145A1 (en) | 2023-07-13 |
Family
ID=70470903
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/400,255 Active US11730340B2 (en) | 2019-05-01 | 2019-05-01 | Video display system having an adaptive overlay |
US18/183,683 Pending US20230218145A1 (en) | 2019-05-01 | 2023-03-14 | Endoscopic system and method for displaying an adaptive overlay |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/400,255 Active US11730340B2 (en) | 2019-05-01 | 2019-05-01 | Video display system having an adaptive overlay |
Country Status (2)
Country | Link |
---|---|
US (2) | US11730340B2 (en) |
EP (1) | EP3733042A1 (en) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160262602A1 (en) * | 2015-03-09 | 2016-09-15 | Steven Sounyoung Yu | Laparoscopic Cholecystectomy With Fluorescence Cholangiography |
US20170020627A1 (en) * | 2015-03-25 | 2017-01-26 | Camplex, Inc. | Surgical visualization systems and displays |
US20200184640A1 (en) * | 2018-12-05 | 2020-06-11 | Stryker Corporation | Systems and methods for displaying medical imaging data |
US20200337542A1 (en) * | 2019-04-26 | 2020-10-29 | Chemimage Corporation | Hybrid imaging product and hybrid endoscopic system |
Family Cites Families (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6537211B1 (en) * | 1998-01-26 | 2003-03-25 | Massachusetts Institute Of Technology | Flourescence imaging endoscope |
JP2001299676A (en) | 2000-04-25 | 2001-10-30 | Fuji Photo Film Co Ltd | Method and system for detecting sentinel lymph node |
AU2003218116A1 (en) * | 2002-03-12 | 2003-09-29 | Beth Israel Deaconess Medical Center | Medical imaging systems |
US7794396B2 (en) * | 2006-11-03 | 2010-09-14 | Stryker Corporation | System and method for the automated zooming of a surgical camera |
US20090192390A1 (en) * | 2008-01-24 | 2009-07-30 | Lifeguard Surgical Systems | Common bile duct surgical imaging system |
US8155479B2 (en) * | 2008-03-28 | 2012-04-10 | Intuitive Surgical Operations Inc. | Automated panning and digital zooming for robotic surgical systems |
US10507065B2 (en) * | 2009-05-29 | 2019-12-17 | Jack Wade | System and method for enhanced data analysis with video enabled software tools for medical environments |
CN102781305B (en) * | 2010-03-09 | 2015-06-03 | 奥林巴斯株式会社 | Fluorescent endoscope device |
JP2011229850A (en) * | 2010-04-30 | 2011-11-17 | Fujifilm Corp | Endoscopic system, method and program |
CN103068294B (en) * | 2011-01-24 | 2015-06-24 | 奥林巴斯医疗株式会社 | Medical device |
US20120197086A1 (en) * | 2011-01-31 | 2012-08-02 | Nellcor Puritan Bennett Llc | Medical visualization technique and apparatus |
JPWO2013015120A1 (en) | 2011-07-22 | 2015-02-23 | オリンパス株式会社 | Fluorescence endoscope device |
JP6005918B2 (en) * | 2011-09-08 | 2016-10-12 | オリンパス株式会社 | Fluorescence observation equipment |
JP6103824B2 (en) * | 2012-06-04 | 2017-03-29 | オリンパス株式会社 | Fluorescence endoscope device |
US10022041B2 (en) * | 2012-06-27 | 2018-07-17 | Camplex, Inc. | Hydraulic system for surgical applications |
CN104853666B (en) * | 2012-12-13 | 2016-11-02 | 奥林巴斯株式会社 | Fluorescence monitoring apparatus |
JP6533358B2 (en) | 2013-08-06 | 2019-06-19 | 三菱電機エンジニアリング株式会社 | Imaging device |
KR102245193B1 (en) * | 2014-03-13 | 2021-04-28 | 삼성메디슨 주식회사 | Medical diagnostic apparatus and operating method thereof |
KR101751140B1 (en) * | 2015-12-24 | 2017-06-26 | 삼성전기주식회사 | Image sensor and camera module |
US10708478B2 (en) * | 2016-03-23 | 2020-07-07 | Karl Storz Imaging, Inc. | Image transformation and display for fluorescent and visible imaging |
WO2018217951A1 (en) * | 2017-05-24 | 2018-11-29 | Camplex, Inc. | Surgical visualization systems and displays |
CN113164013B (en) * | 2018-12-05 | 2024-06-21 | 直观外科手术操作公司 | Combination of near infrared information and color images in image-guided surgery |
-
2019
- 2019-05-01 US US16/400,255 patent/US11730340B2/en active Active
-
2020
- 2020-04-27 EP EP20171598.4A patent/EP3733042A1/en active Pending
-
2023
- 2023-03-14 US US18/183,683 patent/US20230218145A1/en active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160262602A1 (en) * | 2015-03-09 | 2016-09-15 | Steven Sounyoung Yu | Laparoscopic Cholecystectomy With Fluorescence Cholangiography |
US20170020627A1 (en) * | 2015-03-25 | 2017-01-26 | Camplex, Inc. | Surgical visualization systems and displays |
US20200184640A1 (en) * | 2018-12-05 | 2020-06-11 | Stryker Corporation | Systems and methods for displaying medical imaging data |
US20200337542A1 (en) * | 2019-04-26 | 2020-10-29 | Chemimage Corporation | Hybrid imaging product and hybrid endoscopic system |
Also Published As
Publication number | Publication date |
---|---|
US11730340B2 (en) | 2023-08-22 |
EP3733042A1 (en) | 2020-11-04 |
US20200345204A1 (en) | 2020-11-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7086121B2 (en) | Ureter detection using imaging by wavelength band selection | |
US9906739B2 (en) | Image pickup device and image pickup method | |
CN110325100B (en) | Endoscope system and method of operating the same | |
US20190357773A1 (en) | System and Method for Laser Imaging and Ablation of Cancer Cells Using Fluorescence | |
EP3332687B1 (en) | Augmented stereoscopic visualization for a surgical robot | |
US20170064257A1 (en) | Imaging systems and methods for displaying fluorescence and visible images | |
JP7135082B2 (en) | Endoscope device, method of operating endoscope device, and program | |
EP1535568A1 (en) | Method and apparatus for obtaining fluorescence images, and computer executable program therefor | |
JP6234375B2 (en) | Observation device | |
JP6779089B2 (en) | Endoscope system and how to drive the endoscope system | |
JP4394356B2 (en) | Electronic endoscope device | |
US20180033142A1 (en) | Image-processing apparatus, biological observation apparatus, and image-processing method | |
JP2012065698A (en) | Operation support system, and operation support method using the same | |
JP4298151B2 (en) | Fluorescent image display method and apparatus | |
JP6827516B2 (en) | Endoscope system and how to drive the endoscope system | |
JP2022183277A (en) | Endoscope device, endoscope processor, and endoscope device operation method | |
US11689689B2 (en) | Infrared imaging system having structural data enhancement | |
KR102148685B1 (en) | Surgical video creation system | |
US11684238B2 (en) | Control device and medical observation system | |
JP2002345739A (en) | Image display device | |
US20230218145A1 (en) | Endoscopic system and method for displaying an adaptive overlay | |
US12052526B2 (en) | Imaging system having structural data enhancement for non-visible spectra | |
JP5160958B2 (en) | Fundus photographing apparatus and fundus image processing apparatus | |
US20230397801A1 (en) | Medical imaging system, medical imaging device, and operation method | |
JP2001128926A (en) | Method and device for fluorescent character display |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: KARL STORZ IMAGING, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GRANNEMAN, RUSSELL;REEL/FRAME:064319/0109 Effective date: 20190425 |