US20210106397A1 - Surgical optical system with heads up display - Google Patents
Surgical optical system with heads up display Download PDFInfo
- Publication number
- US20210106397A1 US20210106397A1 US17/034,630 US202017034630A US2021106397A1 US 20210106397 A1 US20210106397 A1 US 20210106397A1 US 202017034630 A US202017034630 A US 202017034630A US 2021106397 A1 US2021106397 A1 US 2021106397A1
- Authority
- US
- United States
- Prior art keywords
- image
- display
- display device
- surgical
- image sensor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 230000003287 optical effect Effects 0.000 title claims abstract description 80
- 238000012545 processing Methods 0.000 claims description 24
- 230000000007 visual effect Effects 0.000 claims description 11
- 238000004891 communication Methods 0.000 claims description 9
- 238000012544 monitoring process Methods 0.000 claims description 9
- 238000003325 tomography Methods 0.000 claims description 9
- 230000036772 blood pressure Effects 0.000 claims description 2
- 230000036760 body temperature Effects 0.000 claims description 2
- 229940079593 drug Drugs 0.000 claims description 2
- 239000003814 drug Substances 0.000 claims description 2
- 239000011521 glass Substances 0.000 claims description 2
- 238000000034 method Methods 0.000 abstract description 26
- 238000010586 diagram Methods 0.000 description 7
- 238000007408 cone-beam computed tomography Methods 0.000 description 5
- 239000004065 semiconductor Substances 0.000 description 4
- 206010002091 Anaesthesia Diseases 0.000 description 3
- 230000037005 anaesthesia Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 208000027418 Wounds and injury Diseases 0.000 description 2
- 230000003444 anaesthetic effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 150000001875 compounds Chemical class 0.000 description 2
- 230000006378 damage Effects 0.000 description 2
- 230000003090 exacerbative effect Effects 0.000 description 2
- 208000014674 injury Diseases 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 230000001575 pathological effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 208000003464 asthenopia Diseases 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000007654 immersion Methods 0.000 description 1
- 230000001939 inductive effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/361—Optical details, e.g. image relay to the camera or image sensor
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/24—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for the mouth, i.e. stomatoscopes, e.g. with tongue depressors; Instruments for opening or keeping open the mouth
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/01—Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/021—Measuring pressure in heart or blood vessels
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/02—Arrangements for diagnosis sequentially in different planes; Stereoscopic radiation diagnosis
- A61B6/03—Computed tomography [CT]
- A61B6/032—Transmission computed tomography [CT]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/52—Devices using data or image processing specially adapted for radiation diagnosis
- A61B6/5294—Devices using data or image processing specially adapted for radiation diagnosis involving using additional data, e.g. patient information, image labeling, acquisition parameters
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/20—Surgical microscopes characterised by non-optical aspects
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/361—Image-producing devices, e.g. surgical cameras
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/0004—Microscopes specially adapted for specific applications
- G02B21/0012—Surgical microscopes
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/18—Arrangements with more than one light path, e.g. for comparing two specimens
- G02B21/20—Binocular arrangements
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/364—Projection microscopes
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B21/00—Microscopes
- G02B21/36—Microscopes arranged for photographic purposes or projection purposes or digital imaging or video purposes including associated control and data processing arrangements
- G02B21/365—Control or image processing arrangements for digital or video microscopes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/372—Details of monitor hardware
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/373—Surgical systems with images on a monitor during operation using light, e.g. by using optical scanners
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/376—Surgical systems with images on a monitor during operation using X-rays, e.g. fluoroscopy
- A61B2090/3762—Surgical systems with images on a monitor during operation using X-rays, e.g. fluoroscopy using computed tomography systems [CT]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/74—Details of notification to user or communication with user or patient ; user input means
- A61B5/742—Details of notification to user or communication with user or patient ; user input means using visual displays
- A61B5/7425—Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10072—Tomographic images
- G06T2207/10081—Computed x-ray tomography [CT]
Definitions
- the disclosure relates generally to microscope systems and more particularly to dental microscopes having a heads up display.
- microscopes such as dental, surgical, clinical, pathological, or laboratory microscopes include only a traditional eyepiece, requiring the user to assume an ergonomically undesirable hunched position while using the microscope and inducing eyestrain.
- dental practitioners experience fatigue and high rates of ergonomic injury.
- the user must look away from the eyepiece in order to check important procedure-related information such as instrument status, patient condition, radiographs, clinical or pathological data, procedural instructions, etc., thereby increasing procedure time, thereby exacerbating the ergonomic issues described above, creating lower patient or sample throughput, and, in dental/surgical applications, potentially increasing the amount of anesthesia and/or anesthetic required during the procedure.
- FIG. 1 is a side view of an exemplary microscope system in accordance with various embodiments.
- FIG. 2A is an isometric view of an exemplary beam splitting device in accordance with various embodiments.
- FIG. 2B is an isometric view of an interior of an exemplary beam splitting device in accordance with various embodiments.
- FIG. 2C is a bottom view of an exemplary beam splitting device in accordance with various embodiments.
- FIG. 2D is a first side view of an exemplary beam splitting device in accordance with various embodiments.
- FIG. 2E is a rear view of an exemplary beam splitting device in accordance with various embodiments.
- FIG. 2F is a second side view of an exemplary beam splitting device in accordance with various embodiments.
- FIG. 2G is a third side view of an exemplary beam splitting device in accordance with various embodiments.
- FIG. 3 is a block diagram representing an exemplary surgical optical system in accordance with various embodiments.
- FIG. 4 is a flow diagram illustrating an exemplary method for operating a surgical optical system in accordance with various embodiments.
- FIG. 5 is a flow diagram illustrating an exemplary method for converting a microscope to a surgical system in accordance with various embodiments.
- FIG. 6 is an exemplary computational device block diagram depicting various components that can be used to implement various of the disclosed embodiments.
- FIG. 7 is an exemplary computational device block diagram depicting various components that can be used to implement various of the disclosed embodiments in a distributed system.
- conventional dental and surgical microscopes include only a traditional eyepiece, requiring the practitioner to remain in an ergonomically undesirable hunched position while performing a procedure.
- dental practitioners experience fatigue and high rates of ergonomic injury.
- the practitioner must look away from the eyepiece in order to check important procedure-related information such as instrument status, patient condition, radiographs, etc., thereby increasing procedure time, thereby exacerbating the ergonomic issues described above, creating lower patient throughput, and potentially increasing the amount of anesthesia and/or anesthetic required during the procedure.
- a surgical optical system having a heads up display including, in accordance with various embodiments, an optical device, an image sensor optically coupled to the optical device to acquire image data from a field of view of the optical device, and a display device configured to display an acquired image representing the image data acquired by the image sensor.
- an exemplary retrofitted or converted surgical microscope 100 includes an objective 101 having a field of view, a beam splitter 107 (e.g., within a beam splitting apparatus 200 as shown) for splitting an optical input received from the objective 101 , an eyepiece 109 optically coupled to a first split beam path (e.g., first optical output 203 a as shown in FIGS. 2A-G ) of the beam splitter 107 (or, for a bifocal eyepiece as shown, the eyepiece can be optically coupled to each of two distinct split beam paths (e.g., first optical output 203 a and third optical output path 203 c as shown in FIGS.
- a beam splitter 107 e.g., within a beam splitting apparatus 200 as shown
- an eyepiece 109 optically coupled to a first split beam path (e.g., first optical output 203 a as shown in FIGS. 2A-G ) of the beam splitter 107 (or, for a bifocal eyepiece as shown, the
- an image sensor 103 optically coupled to a second split beam path (e.g., second optical output path 203 b as shown in FIGS. 2A-G ) of the beam splitter 107 to acquire image data
- a display device 105 in electrical communication with the image sensor for displaying an acquired image representing the image data received from the image sensor 103 .
- Objective 101 can include but is not limited to, any suitable objective configuration, including, for example but not limited to, a single lens, a multi-element compound lens, or an oil immersion lens.
- the objective 101 in accordance with various embodiments, can be designed for any suitable level of magnification in combination with any suitable numerical aperture.
- Image sensor 103 can include but is not limited to, video sensors, any focal plane array/staring array, charge coupled device (CCD) sensors, complimentary metal-oxide-semiconductor (CMOS) sensors, back-side illuminated CMOS sensors, hybrid CCD/CMOS (sCMOS) sensors, N-type metal-oxide-semiconductor (NMOS) sensors, specialty sensors, curved sensors, or any other suitable image sensing device.
- CCD charge coupled device
- CMOS complimentary metal-oxide-semiconductor
- sCMOS hybrid CCD/CMOS
- NMOS N-type metal-oxide-semiconductor
- specialty sensors curved sensors, or any other suitable image sensing device.
- a camera adapter (not shown) can be interposed between the objective 101 and the image sensor 103 for providing focus, iris, zoom, and aperture control functionality.
- Display device 105 can include but is not limited to, a computer monitor, a television, a touchscreen monitor, a screen of a mobile device (such as but not limited to, a smartphone, a tablet, or an electronic book), and/or any other display device that can be used for displaying an image and/or video representing image data.
- a mobile device such as but not limited to, a smartphone, a tablet, or an electronic book
- the display device 105 can also double as an input device for receiving user input.
- the display device 105 can be configured to directly receive image data or other input via an integral processor.
- the display device can be configured to receive image data or other input via one or more intermediary processing devices such as a computer or tablet, a surgical instrument monitor, a patient data store.
- display device 105 can be mounted via a movable mount (e.g., a gimbal, an articulating arm, a hinge, or any other suitable movable mount) for repositioning the display relative to the objective 101 .
- a movable mount e.g., a gimbal, an articulating arm, a hinge, or any other suitable movable mount
- Such repositioning advantageously allows a user to achieve better ergonomic position during use and/or to enable shared viewing with other members of the surgical team.
- beam splitter 107 is positioned within a beam splitting apparatus 200 .
- the beam splitting apparatus 200 includes a housing 207 defining an interior volume, one or more beam splitters 107 (not shown) being positioned within the interior volume.
- the housing 207 includes a first surface 207 a , a second surface 207 b , and a third surface 207 c .
- the first surface 207 a of the beam splitting apparatus 200 in accordance with various embodiments, defines at least one optical input path 201 a , 201 b for receiving an optical input from the objective lens 101 . For example, as shown in FIG.
- optical input paths 201 a , 201 b are defined to provide a binocular optical input.
- the input is then split within the beam splitting apparatus by the beam splitter(s) 107 into at least first and second optical outputs, although it will be apparent in view of this disclosure that splitting the optical input into third, fourth, fifth, or any number of additional optical outputs is contemplated in accordance with various embodiments.
- the beam splitting apparatus 200 includes five optical outputs 203 a , 203 b , 203 c , 203 d , 203 e .
- the second surface 207 b of the beam splitting apparatus 200 defines at least a first optical output path 203 a , 203 c for emitting the first optical output and the third surface 207 c of the beam splitting apparatus 200 defines at least a second optical output path 203 b , 203 d , 203 e for emitting the second optical output.
- first 203 a and third 203 c optical outputs are defined in the second surface 207 b for providing a binocular output to a binocular eyepiece (e.g., eyepiece 109 as shown in FIG.
- second 203 b , fourth 203 d , and fifth 203 e optical outputs are defined in the third surface 207 c for providing optical output to one or more image sensors, additional eyepieces, or other devices.
- a single image sensor 103 is shown attached to the second optical output 203 b in FIG. 1 , it will be apparent in view of this disclosure that one or more additional image sensors may be desirable in accordance with various embodiments.
- an additional image sensor (not shown) can be optically coupled to the fourth or fifth optical output to provide acquired image data to an additional display (not shown) for use by a surgical assistant or other interested party.
- an additional image sensor (not shown) can be optically coupled to the fourth or fifth optical output to provide acquired image data to a recording device for providing raw footage of the surgery (e.g., for training purposes).
- the beam splitting apparatus 200 also includes one or more mounting attachments 205 a , 205 b for mounting the beam splitting apparatus 200 to an optical device (e.g., a surgical microscope as shown in FIG. 1 ).
- Eyepiece 109 can include but is not limited to, one or more of a monocular eyepiece, a binocular eyepiece, a single lens eyepiece, a compound lens eyepiece, a Galilean lens, a convex lens, a Huygens eyepiece, a Ramsden eyepiece, a Kellner eyepiece, a symmetrical eyepiece, an orthoscopic eyepiece, a monocentric eyepiece, an Erfle eyepiece, a Konig eyepiece, an RKE eyepiece, or a Nagler eyepiece.
- a surgical optical system 300 can include a visual monitoring component 301 including an optical device 302 having a field of view and an image sensor 303 optically coupled to the optical device 302 for acquiring image data from the field of view of the optical device 302 .
- the surgical optical system 300 also includes a display 305 for displaying acquired images such as, for example, still photographs, video, etc., from the image sensor 303 .
- the system 300 in accordance with various embodiments, can include one or more processing devices 312 , 313 (e.g., an integral processor of the display or an external processing or computing device).
- the processing device 312 , 313 can be configured to receive surgical instrument data from one or more surgical instruments 307 and to instruct the display 305 or an additional display (not shown) to render the received surgical instrument data.
- the processing device 312 , 313 in accordance with various embodiments, can be configured to receive patient data from a patient monitoring system 309 or a patient data store 311 and to instruct the display 305 or an additional display (not shown) to render the received patient data.
- the visual monitoring component can also include one or more additional optical devices, image sensors, displays, or data inputs.
- Optical devices 302 can be any suitable optical lens or set of lenses arranged to have a field of view (FOV) including, for example but not limited to, a microscope, a dental microscope, a surgical microscope, an endoscope, an exoscope, a loupe, a magnifying glass, an eyepiece, an objective, a binocular, a telescope, a camera, and any combinations thereof.
- FOV field of view
- Optical sensors 303 can include, for example but not limited to, video sensors, any focal plane array/staring array, charge coupled device (CCD) sensors, complimentary metal-oxide-semiconductor (CMOS) sensors, back-side illuminated CMOS sensors, hybrid CCD/CMOS (sCMOS) sensors, N-type metal-oxide-semiconductor (NMOS) sensors, specialty sensors, curved sensors, or any other suitable image sensing device.
- One or more image sensors 303 can be coupled to the optical device 302 for acquiring image data from the FOV of the optical device 302 .
- an image sensor 303 can be positioned directly within an optical path of the optical device 302 .
- a beam splitter (e.g., 107 , 200 as shown in FIGS. 1-2G ) can be interposed between the image sensor 303 and the optical path of the optical device 302 for providing optical image data to the image sensor 303 and at least one other device (e.g., an eyepiece or additional image sensor).
- optical devices 302 can also include a camera adapter or similar functionality interposed between the FOV and the image sensor 303 for providing focus, iris, zoom, and aperture control functionality.
- Display 305 is configured to receive and display the optical image data acquired by the one or more image sensors 303 .
- Display 305 can include, for example but not limited to, a computer monitor, a television, a touchscreen monitor, a screen of a mobile device (such as but not limited to, a smartphone, a tablet, or an electronic book), and/or any other display device that can be used for displaying an image and/or video representing image data.
- the display 305 can receive the image data directly from the image sensor 303 or can receive the image data via one or more internal processing devices 312 or external processing devices 313 .
- the processing devices 312 , 313 can include, for example but not limited to general or special purpose processors, microprocessors, field programmable gate arrays (FPGA), multi-core processors, or any other suitable processing device.
- display 305 receives the optical image data via one or more electronic signal splitting devices (not shown).
- Electronic signal splitting devices can include, for example but not limited to, coaxial splitters, HDMI splitters, digital signal processors, combinations thereof, or any other suitable electronic signal splitting device.
- the display 305 or the processing device(s) can be configured to receive and display other external data alongside or overlaid on the rendered image data.
- the display can be configured to receive surgical instrument data from one or more surgical instruments 307 .
- Surgical instrument data can include, for example, any relevant data, including for example, torque, revolutions per minute, pneumatic pressure, temperature, attachment wear, quantity of medication/anesthesia delivered, or combinations thereof.
- the display can be configured to receive patient data from one or more patient monitoring systems 309 or patient data stores 311 .
- Patient data received from a patient monitoring system can include, for example, heart rate, body temperature, or blood pressure.
- Patient data received from a patient data store 311 can include, for example, radiographs, computed cone beam computed tomography images, photographs, medical (including dental) history, or personal data (e.g., name, address, insurance information, date of birth).
- Such external data can, in accordance with various embodiments, be displayed contemporaneously on a segregated portion of the display 305 , can temporarily replace the rendered image on the display 305 , can be overlaid on the rendered image on the display 305 , be displayed on one or more additional displays, or be presented in any other suitable configuration.
- the display 305 or processor(s) 312 , 313 can also be configured to receive and display appointment or other non-patient data (e.g., an appointment schedule calendar).
- the surgical optical system 300 can be configured to, by the processing device(s) 312 , 313 , align the CBCT image with the acquired image from the image sensor 303 and instruct the display 305 to overlay the displayed acquired image with the CBCT image.
- the surgical optical system 300 can be configured to identify, by the processing device(s) 312 , 313 , areas of interest within the overlaid image and instruct the display 305 to render visual indicators at the locations of interest within the displayed acquired image.
- the surgical optical system 300 can be configured to identify, by the processing device(s) 312 , 313 , one or more sectioning planes based on the computed cone beam tomography image and instruct the display 305 to render visual representations of the sectioning planes within the displayed acquired image.
- the display 305 can be interactive (e.g., via a touchscreen input, keyboard/mouse control, and/or any other suitable means) and thereby allow the user to, for example, zoom in and out, adjust focus, adjust aperture, query displayed data, freeze, rewind, or otherwise control video, etc.
- the display 305 or the processing device(s) 312 , 313 can be configured to receive audio data (e.g., via a microphone) and/or produce sound (e.g., via built in or external speakers).
- audio capability can be used to produce a warning signal (e.g., if a surgical instrument fails, needs to be replaced, or exceeds an operational threshold or if monitored patient data exceeds a preset limit).
- audio capability can be used to allow the user to make audio, audio/video, or “intercom” calls to consult with colleagues or staff.
- audio capability can be used to play music or other media content during use.
- the audio capability can provide voice command capability such as, for example, search functionality (e.g., of patient data, patient electronic charts, instrument data, procedural guidelines, a secure cloud account, the internet, or any other searchable electronic information), display control functionality (e.g., content and configuration of the display), optical control functionality (e.g., zoom, focus, aperture, magnification level, or other optical controls), and communication functionality (e.g., file transfers, uploads, downloads, email, instant message, voice calls, video calls, or any other form of electronic communication).
- search functionality e.g., of patient data, patient electronic charts, instrument data, procedural guidelines, a secure cloud account, the internet, or any other searchable electronic information
- display control functionality e.g., content and configuration of the display
- optical control functionality e.g., zoom, focus,
- audio capability can be used to allow the user to create and store audio or audio/video files (e.g., narrated notes or a recording of the displayed image data).
- the audio or video files can be associated with a particular record or file (e.g., a patient record, a patient radiograph, a video of the procedure, or any other record or file).
- the audio capability can allow the user to play audio, audio/video, or other audible media content accessed or created as described above.
- the method includes the step of providing a visual monitoring component including an optical device having a field of view and an image sensor optically coupled to the optical device 401 .
- the method also includes the step of acquiring, by the image sensor, image data from the field of view of the optical device 403 .
- the method also includes displaying, on a display device, an acquired image representing the image data acquired by the image sensor 405 .
- the step of providing a visual monitoring component including an optical device having a field of view and an image sensor optically coupled to the optical device 401 can be performed, for example but not limited to, by providing an optical device 302 and an image sensor 303 as described above with reference to FIG. 3 or a converted surgical microscope 100 including an objective 101 , a beam splitter 107 , and an image sensor 103 as described above with reference to FIG. 1 .
- the step of acquiring, by the image sensor, image data from the field of view of the optical device 403 can be performed, for example but not limited to, using an image sensor 103 , 303 as described above with reference to FIGS. 1 and 3 .
- the step of displaying, on a display device, an acquired image representing the image data acquired by the image sensor 405 can be performed, for example but not limited to, using display 105 , 305 , image sensor 103 , 303 , and/or processor 312 , 313 as described above with reference to FIGS. 1 and 3 .
- the method includes the step of providing a microscope having an objective lens and an eyepiece 501 .
- the method also includes the step of installing a beam splitter between the objective lens and the eyepiece, the beam splitter optically coupled to the objective lens and the eyepiece optically coupled to a first split beam path of the beam splitter 503 .
- the method also includes optically coupling an image sensor to a second split beam path of the beam splitter; the image sensor configured to acquire image data from the field of view of the optical device and transmit an electronic image data signal 505 .
- the method also includes providing a display device in electrical communication with the image sensor, the display device configured to receive the electronic image data signal transmitted by the image sensor and to display an acquired image representing the image data acquired by the image sensor 507 .
- the step of providing a microscope having an objective lens and an eyepiece 501 can be performed, for example but not limited to, by providing a microscope having an objective 101 and an eyepiece 109 as described above with reference to FIG. 1 .
- the step of installing a beam splitter between the objective lens and the eyepiece, the beam splitter optically coupled to the objective lens and the eyepiece optically coupled to a first split beam path of the beam splitter 503 can be performed, for example but not limited to, by installing a beam splitter 107 or beam splitting apparatus 200 as described above with reference to FIGS. 1 and 2 .
- the step of optically coupling an image sensor to a second split beam path of the beam splitter; the image sensor configured to acquire image data from the field of view of the optical device and transmit an electronic image data signal 505 can be performed, for example but not limited to, optically coupling image sensor 103 to beam splitter 107 as described above with reference to FIG. 1 .
- the step of providing a display device in electrical communication with the image sensor, the display device configured to receive the electronic image data signal transmitted by the image sensor and to display an acquired image representing the image data acquired by the image sensor 507 can be performed, for example but not limited to, using image sensor 103 , 303 and display device 105 , 305 as described above with reference to FIGS. 1 and 3 .
- the display device can be mounted via a movable mount (e.g., a gimbal, an articulating arm, a hinge, or any other suitable movable mount) for repositioning the display relative to other components of the microscope. As discussed above, such repositioning advantageously allows a user to achieve better ergonomic position during use and/or to enable shared viewing with other members of the surgical team.
- FIG. 6 is a block diagram of an exemplary computing device 1010 such as can be used, or portions thereof, in accordance with various embodiments as described above with reference to FIGS. 1-5 .
- the computing device 1010 includes one or more non-transitory computer-readable media for storing one or more computer-executable instructions or software for implementing exemplary embodiments.
- the non-transitory computer-readable media can include, but are not limited to, one or more types of hardware memory, non-transitory tangible media (for example, one or more magnetic storage disks, one or more optical disks, one or more flash drives), and the like.
- memory 1016 included in the computing device 1010 can store computer-readable and computer-executable instructions or software for performing the operations disclosed herein.
- the memory can store software application 1040 which is programmed to perform various of the disclosed operations as discussed with respect to FIGS. 1-5 .
- the computing device 1010 can also include configurable and/or programmable processor 1012 and associated core 1014 , and optionally, one or more additional configurable and/or programmable processing devices, e.g., processor(s) 1012 ′ and associated core(s) 1014 ′ (for example, in the case of computational devices having multiple processors/cores), for executing computer-readable and computer-executable instructions or software stored in the memory 1016 and other programs for controlling system hardware.
- Processor 1012 and processor(s) 1012 ′ can each be a single core processor or multiple core ( 1014 and 1014 ′) processor.
- Virtualization can be employed in the computing device 1010 so that infrastructure and resources in the computing device can be shared dynamically.
- a virtual machine 1024 can be provided to handle a process running on multiple processors so that the process appears to be using only one computing resource rather than multiple computing resources. Multiple virtual machines can also be used with one processor.
- Memory 1016 can include a computational device memory or random access memory, such as but not limited to DRAM, SRAM, EDO RAM, and the like. Memory 1016 can include other types of memory as well, or combinations thereof.
- a user can interact with the computing device 1010 through a visual display device 101 , 111 A-D, such as a computer monitor, which can display one or more user interfaces 102 that can be provided in accordance with exemplary embodiments.
- the computing device 1010 can include other I/O devices for receiving input from a user, for example, a keyboard or any suitable multi-point touch interface 1018 , a pointing device 1020 (e.g., a mouse).
- the keyboard 1018 and the pointing device 1020 can be coupled to the visual display device 101 , 111 A-D.
- the computing device 1010 can include other suitable conventional I/O peripherals.
- the computing device 1010 can also include one or more storage devices 1034 , such as but not limited to a hard-drive, CD-ROM, or other computer readable media, for storing data and computer-readable instructions and/or software that perform operations disclosed herein.
- Exemplary storage device 1034 can also store one or more databases for storing any suitable information required to implement exemplary embodiments. The databases can be updated manually or automatically at any suitable time to add, delete, and/or update one or more items in the databases.
- the computing device 1010 can include a network interface 1022 configured to interface via one or more network devices 1032 with one or more networks, for example, Local Area Network (LAN), Wide Area Network (WAN) or the Internet through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (for example, 802.11, T1, T3, 56 kb, X.25), broadband connections (for example, ISDN, Frame Relay, ATM), wireless connections, controller area network (CAN), or some combination of any or all of the above.
- LAN Local Area Network
- WAN Wide Area Network
- the computing device 1010 can include a network interface 1022 configured to interface via one or more network devices 1032 with one or more networks, for example, Local Area Network (LAN), Wide Area Network (WAN) or the Internet through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (for example, 802.11, T1, T3, 56 kb, X.25), broadband connections (for example, ISDN, Frame Relay, ATM), wireless connections,
- the network interface 1022 can include a built-in network adapter, network interface card, PCMCIA network card, card bus network adapter, wireless network adapter, USB network adapter, modem or any other device suitable for interfacing the computing device 1010 to any type of network capable of communication and performing the operations described herein.
- the computing device 1010 can be any computational device, such as a workstation, desktop computer, server, laptop, handheld computer, tablet computer, or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein.
- the computing device 1010 can run any operating system 1026 , such as any of the versions of the Microsoft® Windows® operating systems (Microsoft, Redmond, Wash.), the different releases of the Unix and Linux operating systems, any version of the MAC OS® (Apple, Inc., Cupertino, Calif.) operating system for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, or any other operating system capable of running on the computing device and performing the operations described herein.
- the operating system 1026 can be run in native mode or emulated mode.
- the operating system 1026 can be run on one or more cloud machine instances.
- FIG. 7 is an example computational device block diagram of certain distributed embodiments.
- FIGS. 1 and 6 make reference to a centralized multi-display system 100 operating on a single computing device, one will recognize that various of the modules within the multi-display system 100 may instead be distributed across a network 1105 in separate server systems 1101 a - d and possibly in user systems, such as kiosk, desktop computer device 1102 , or mobile computer device 1103 .
- users may download an application to their desktop computer device or mobile computer device, which is configured to show an interactive digital rendering of the meeting room and perform the operations of the display device 101 , graphical user interface 102 , output modules 109 A-D, and/or output displays 111 A-D.
- the modules of the multi-display system 100 can be separately located on server systems 1101 a - d and can be in communication with one another across the network 1105 .
- Exemplary flowcharts are provided herein for illustrative purposes and are non-limiting examples of methods.
- One of ordinary skill in the art will recognize that exemplary methods may include more or fewer steps than those illustrated in the exemplary flowcharts, and that the steps in the exemplary flowcharts may be performed in a different order than the order shown in the illustrative flowcharts.
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Surgery (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Pathology (AREA)
- Animal Behavior & Ethology (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Molecular Biology (AREA)
- Heart & Thoracic Surgery (AREA)
- Biomedical Technology (AREA)
- Optics & Photonics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- General Physics & Mathematics (AREA)
- Analytical Chemistry (AREA)
- Chemical & Material Sciences (AREA)
- Biophysics (AREA)
- Multimedia (AREA)
- Radiology & Medical Imaging (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- High Energy & Nuclear Physics (AREA)
- Cardiology (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Gynecology & Obstetrics (AREA)
- Pulmonology (AREA)
- Human Computer Interaction (AREA)
- Vascular Medicine (AREA)
- Physiology (AREA)
- Dentistry (AREA)
- Microscoopes, Condenser (AREA)
Abstract
Description
- This application is a divisional of U.S. patent application Ser. No. 15/870,424, filed Jan. 12, 2018, which is a continuation of International Application No. PCT/US2016/042252, filed Jul. 14, 2016, which claims the benefit of U.S. Provisional Application No. 62/192,351, filed Jul. 14, 2015. The entire contents of each of the above documents is herein incorporated by reference in their entirety.
- The disclosure relates generally to microscope systems and more particularly to dental microscopes having a heads up display.
- Conventionally, microscopes such as dental, surgical, clinical, pathological, or laboratory microscopes include only a traditional eyepiece, requiring the user to assume an ergonomically undesirable hunched position while using the microscope and inducing eyestrain. As a result, for example, dental practitioners experience fatigue and high rates of ergonomic injury. Furthermore, the user must look away from the eyepiece in order to check important procedure-related information such as instrument status, patient condition, radiographs, clinical or pathological data, procedural instructions, etc., thereby increasing procedure time, thereby exacerbating the ergonomic issues described above, creating lower patient or sample throughput, and, in dental/surgical applications, potentially increasing the amount of anesthesia and/or anesthetic required during the procedure.
- The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:
-
FIG. 1 is a side view of an exemplary microscope system in accordance with various embodiments. -
FIG. 2A is an isometric view of an exemplary beam splitting device in accordance with various embodiments. -
FIG. 2B is an isometric view of an interior of an exemplary beam splitting device in accordance with various embodiments. -
FIG. 2C is a bottom view of an exemplary beam splitting device in accordance with various embodiments. -
FIG. 2D is a first side view of an exemplary beam splitting device in accordance with various embodiments. -
FIG. 2E is a rear view of an exemplary beam splitting device in accordance with various embodiments. -
FIG. 2F is a second side view of an exemplary beam splitting device in accordance with various embodiments. -
FIG. 2G is a third side view of an exemplary beam splitting device in accordance with various embodiments. -
FIG. 3 is a block diagram representing an exemplary surgical optical system in accordance with various embodiments. -
FIG. 4 is a flow diagram illustrating an exemplary method for operating a surgical optical system in accordance with various embodiments. -
FIG. 5 is a flow diagram illustrating an exemplary method for converting a microscope to a surgical system in accordance with various embodiments. -
FIG. 6 is an exemplary computational device block diagram depicting various components that can be used to implement various of the disclosed embodiments. -
FIG. 7 is an exemplary computational device block diagram depicting various components that can be used to implement various of the disclosed embodiments in a distributed system. - As discussed above, conventional dental and surgical microscopes include only a traditional eyepiece, requiring the practitioner to remain in an ergonomically undesirable hunched position while performing a procedure. As a result, dental practitioners experience fatigue and high rates of ergonomic injury. Furthermore, the practitioner must look away from the eyepiece in order to check important procedure-related information such as instrument status, patient condition, radiographs, etc., thereby increasing procedure time, thereby exacerbating the ergonomic issues described above, creating lower patient throughput, and potentially increasing the amount of anesthesia and/or anesthetic required during the procedure.
- Methods and systems are provided herein for a surgical optical system having a heads up display including, in accordance with various embodiments, an optical device, an image sensor optically coupled to the optical device to acquire image data from a field of view of the optical device, and a display device configured to display an acquired image representing the image data acquired by the image sensor.
- Referring now to
FIGS. 1 and 2A -G, an exemplary retrofitted or convertedsurgical microscope 100 includes an objective 101 having a field of view, a beam splitter 107 (e.g., within abeam splitting apparatus 200 as shown) for splitting an optical input received from the objective 101, aneyepiece 109 optically coupled to a first split beam path (e.g., firstoptical output 203 a as shown inFIGS. 2A-G ) of the beam splitter 107 (or, for a bifocal eyepiece as shown, the eyepiece can be optically coupled to each of two distinct split beam paths (e.g., firstoptical output 203 a and thirdoptical output path 203 c as shown inFIGS. 2A-G ) of the beam splitter 107), animage sensor 103 optically coupled to a second split beam path (e.g., secondoptical output path 203 b as shown inFIGS. 2A-G ) of thebeam splitter 107 to acquire image data, and adisplay device 105 in electrical communication with the image sensor for displaying an acquired image representing the image data received from theimage sensor 103. -
Objective 101, in accordance with various embodiments, can include but is not limited to, any suitable objective configuration, including, for example but not limited to, a single lens, a multi-element compound lens, or an oil immersion lens. The objective 101, in accordance with various embodiments, can be designed for any suitable level of magnification in combination with any suitable numerical aperture. -
Image sensor 103, in accordance with various embodiments, can include but is not limited to, video sensors, any focal plane array/staring array, charge coupled device (CCD) sensors, complimentary metal-oxide-semiconductor (CMOS) sensors, back-side illuminated CMOS sensors, hybrid CCD/CMOS (sCMOS) sensors, N-type metal-oxide-semiconductor (NMOS) sensors, specialty sensors, curved sensors, or any other suitable image sensing device. In accordance with various embodiments, a camera adapter (not shown) can be interposed between the objective 101 and theimage sensor 103 for providing focus, iris, zoom, and aperture control functionality. -
Display device 105, in accordance with various embodiments, can include but is not limited to, a computer monitor, a television, a touchscreen monitor, a screen of a mobile device (such as but not limited to, a smartphone, a tablet, or an electronic book), and/or any other display device that can be used for displaying an image and/or video representing image data. In accordance with various embodiments, such as but not limited to where thedisplay device 105 is a touchscreen or a display of a mobile device, thedisplay device 105 can also double as an input device for receiving user input. In accordance with various embodiments, thedisplay device 105 can be configured to directly receive image data or other input via an integral processor. In accordance with various embodiments, the display device can be configured to receive image data or other input via one or more intermediary processing devices such as a computer or tablet, a surgical instrument monitor, a patient data store. In some embodiments,display device 105 can be mounted via a movable mount (e.g., a gimbal, an articulating arm, a hinge, or any other suitable movable mount) for repositioning the display relative to the objective 101. Such repositioning advantageously allows a user to achieve better ergonomic position during use and/or to enable shared viewing with other members of the surgical team. -
Beam splitter 107, in accordance with various embodiments, can include but is not limited to, one or more of a polarizing beam splitter, non-polarizing beam splitter, Wollaston prism, pellicle beam splitter, dichroic beam splitter, mirror-type beam splitter, plate beam splitter, cube beam splitter, polka dot beam splitter, Brewster window, variable beam splitter, or a wedged beam splitter. Thebeam splitter 107 can divide a light beam of the objective 101 in any suitable ratio including, for example but not limited to, 50:50, 80:20, 70:30, 95:5, or any other desired ratio. - In accordance with various embodiments,
beam splitter 107 is positioned within abeam splitting apparatus 200. Referring now toFIGS. 2A-G , thebeam splitting apparatus 200, in accordance with various embodiments, includes ahousing 207 defining an interior volume, one or more beam splitters 107 (not shown) being positioned within the interior volume. Thehousing 207 includes afirst surface 207 a, asecond surface 207 b, and athird surface 207 c. Thefirst surface 207 a of thebeam splitting apparatus 200, in accordance with various embodiments, defines at least oneoptical input path objective lens 101. For example, as shown inFIG. 2B , tooptical input paths optical input path FIGS. 2A-G , thebeam splitting apparatus 200 includes fiveoptical outputs second surface 207 b of thebeam splitting apparatus 200 defines at least a firstoptical output path third surface 207 c of thebeam splitting apparatus 200 defines at least a secondoptical output path FIGS. 1 and 2A -G, first 203 a and third 203 c optical outputs are defined in thesecond surface 207 b for providing a binocular output to a binocular eyepiece (e.g.,eyepiece 109 as shown inFIG. 1 ) and second 203 b, fourth 203 d, and fifth 203 e optical outputs are defined in thethird surface 207 c for providing optical output to one or more image sensors, additional eyepieces, or other devices. Although asingle image sensor 103 is shown attached to the secondoptical output 203 b inFIG. 1 , it will be apparent in view of this disclosure that one or more additional image sensors may be desirable in accordance with various embodiments. For example, in some embodiments, an additional image sensor (not shown) can be optically coupled to the fourth or fifth optical output to provide acquired image data to an additional display (not shown) for use by a surgical assistant or other interested party. In some embodiments, an additional image sensor (not shown) can be optically coupled to the fourth or fifth optical output to provide acquired image data to a recording device for providing raw footage of the surgery (e.g., for training purposes). In accordance with various embodiments, thebeam splitting apparatus 200 also includes one or more mountingattachments beam splitting apparatus 200 to an optical device (e.g., a surgical microscope as shown inFIG. 1 ). -
Eyepiece 109, in accordance with various embodiments, can include but is not limited to, one or more of a monocular eyepiece, a binocular eyepiece, a single lens eyepiece, a compound lens eyepiece, a Galilean lens, a convex lens, a Huygens eyepiece, a Ramsden eyepiece, a Kellner eyepiece, a symmetrical eyepiece, an orthoscopic eyepiece, a monocentric eyepiece, an Erfle eyepiece, a Konig eyepiece, an RKE eyepiece, or a Nagler eyepiece. - Referring now to
FIG. 3 , a surgicaloptical system 300 can include avisual monitoring component 301 including anoptical device 302 having a field of view and animage sensor 303 optically coupled to theoptical device 302 for acquiring image data from the field of view of theoptical device 302. The surgicaloptical system 300 also includes adisplay 305 for displaying acquired images such as, for example, still photographs, video, etc., from theimage sensor 303. Thesystem 300, in accordance with various embodiments, can include one ormore processing devices 312, 313 (e.g., an integral processor of the display or an external processing or computing device). Theprocessing device surgical instruments 307 and to instruct thedisplay 305 or an additional display (not shown) to render the received surgical instrument data. Theprocessing device patient monitoring system 309 or apatient data store 311 and to instruct thedisplay 305 or an additional display (not shown) to render the received patient data. Although shown herein as including oneimage sensor 303, it will be apparent in view of this disclosure that, in accordance with various embodiments, the visual monitoring component can also include one or more additional optical devices, image sensors, displays, or data inputs. -
Optical devices 302 can be any suitable optical lens or set of lenses arranged to have a field of view (FOV) including, for example but not limited to, a microscope, a dental microscope, a surgical microscope, an endoscope, an exoscope, a loupe, a magnifying glass, an eyepiece, an objective, a binocular, a telescope, a camera, and any combinations thereof.Optical sensors 303 can include, for example but not limited to, video sensors, any focal plane array/staring array, charge coupled device (CCD) sensors, complimentary metal-oxide-semiconductor (CMOS) sensors, back-side illuminated CMOS sensors, hybrid CCD/CMOS (sCMOS) sensors, N-type metal-oxide-semiconductor (NMOS) sensors, specialty sensors, curved sensors, or any other suitable image sensing device. One ormore image sensors 303 can be coupled to theoptical device 302 for acquiring image data from the FOV of theoptical device 302. In some embodiments, animage sensor 303 can be positioned directly within an optical path of theoptical device 302. In some embodiments, a beam splitter (e.g., 107, 200 as shown inFIGS. 1-2G ) can be interposed between theimage sensor 303 and the optical path of theoptical device 302 for providing optical image data to theimage sensor 303 and at least one other device (e.g., an eyepiece or additional image sensor). In some embodiments,optical devices 302 can also include a camera adapter or similar functionality interposed between the FOV and theimage sensor 303 for providing focus, iris, zoom, and aperture control functionality. -
Display 305 is configured to receive and display the optical image data acquired by the one ormore image sensors 303.Display 305 can include, for example but not limited to, a computer monitor, a television, a touchscreen monitor, a screen of a mobile device (such as but not limited to, a smartphone, a tablet, or an electronic book), and/or any other display device that can be used for displaying an image and/or video representing image data. Thedisplay 305 can receive the image data directly from theimage sensor 303 or can receive the image data via one or moreinternal processing devices 312 orexternal processing devices 313. Theprocessing devices display 305 receives the optical image data via one or more electronic signal splitting devices (not shown). Electronic signal splitting devices can include, for example but not limited to, coaxial splitters, HDMI splitters, digital signal processors, combinations thereof, or any other suitable electronic signal splitting device. - In accordance with various embodiments, the
display 305 or the processing device(s) can be configured to receive and display other external data alongside or overlaid on the rendered image data. For example, in some embodiments, the display can be configured to receive surgical instrument data from one or moresurgical instruments 307. Surgical instrument data can include, for example, any relevant data, including for example, torque, revolutions per minute, pneumatic pressure, temperature, attachment wear, quantity of medication/anesthesia delivered, or combinations thereof. In some embodiments, the display can be configured to receive patient data from one or morepatient monitoring systems 309 orpatient data stores 311. Patient data received from a patient monitoring system can include, for example, heart rate, body temperature, or blood pressure. Patient data received from apatient data store 311 can include, for example, radiographs, computed cone beam computed tomography images, photographs, medical (including dental) history, or personal data (e.g., name, address, insurance information, date of birth). Such external data can, in accordance with various embodiments, be displayed contemporaneously on a segregated portion of thedisplay 305, can temporarily replace the rendered image on thedisplay 305, can be overlaid on the rendered image on thedisplay 305, be displayed on one or more additional displays, or be presented in any other suitable configuration. In some embodiments, thedisplay 305 or processor(s) 312, 313 can also be configured to receive and display appointment or other non-patient data (e.g., an appointment schedule calendar). - Where the patient data includes cone beam computed tomography (CBCT) images, the surgical
optical system 300 can be configured to, by the processing device(s) 312, 313, align the CBCT image with the acquired image from theimage sensor 303 and instruct thedisplay 305 to overlay the displayed acquired image with the CBCT image. In some embodiments, the surgicaloptical system 300 can be configured to identify, by the processing device(s) 312, 313, areas of interest within the overlaid image and instruct thedisplay 305 to render visual indicators at the locations of interest within the displayed acquired image. In some embodiments, the surgicaloptical system 300 can be configured to identify, by the processing device(s) 312, 313, one or more sectioning planes based on the computed cone beam tomography image and instruct thedisplay 305 to render visual representations of the sectioning planes within the displayed acquired image. - It will be apparent in view of this disclosure that the
display 305 can be interactive (e.g., via a touchscreen input, keyboard/mouse control, and/or any other suitable means) and thereby allow the user to, for example, zoom in and out, adjust focus, adjust aperture, query displayed data, freeze, rewind, or otherwise control video, etc. In accordance with various embodiments, thedisplay 305 or the processing device(s) 312, 313 can be configured to receive audio data (e.g., via a microphone) and/or produce sound (e.g., via built in or external speakers). In some embodiments, audio capability can be used to produce a warning signal (e.g., if a surgical instrument fails, needs to be replaced, or exceeds an operational threshold or if monitored patient data exceeds a preset limit). - In some embodiments, audio capability can be used to allow the user to make audio, audio/video, or “intercom” calls to consult with colleagues or staff. In some embodiments, audio capability can be used to play music or other media content during use. In some embodiments, the audio capability can provide voice command capability such as, for example, search functionality (e.g., of patient data, patient electronic charts, instrument data, procedural guidelines, a secure cloud account, the internet, or any other searchable electronic information), display control functionality (e.g., content and configuration of the display), optical control functionality (e.g., zoom, focus, aperture, magnification level, or other optical controls), and communication functionality (e.g., file transfers, uploads, downloads, email, instant message, voice calls, video calls, or any other form of electronic communication). In some embodiments, audio capability can be used to allow the user to create and store audio or audio/video files (e.g., narrated notes or a recording of the displayed image data). In some embodiments, the audio or video files can be associated with a particular record or file (e.g., a patient record, a patient radiograph, a video of the procedure, or any other record or file). Similarly, the audio capability can allow the user to play audio, audio/video, or other audible media content accessed or created as described above.
- Referring now to
FIG. 4 , a method is provided for operating a surgical system. The method includes the step of providing a visual monitoring component including an optical device having a field of view and an image sensor optically coupled to theoptical device 401. The method also includes the step of acquiring, by the image sensor, image data from the field of view of theoptical device 403. The method also includes displaying, on a display device, an acquired image representing the image data acquired by theimage sensor 405. - The step of providing a visual monitoring component including an optical device having a field of view and an image sensor optically coupled to the
optical device 401 can be performed, for example but not limited to, by providing anoptical device 302 and animage sensor 303 as described above with reference toFIG. 3 or a convertedsurgical microscope 100 including an objective 101, abeam splitter 107, and animage sensor 103 as described above with reference toFIG. 1 . - The step of acquiring, by the image sensor, image data from the field of view of the
optical device 403 can be performed, for example but not limited to, using animage sensor FIGS. 1 and 3 . - The step of displaying, on a display device, an acquired image representing the image data acquired by the
image sensor 405 can be performed, for example but not limited to, usingdisplay image sensor processor FIGS. 1 and 3 . - Referring now to
FIG. 5 , a method is provided for converting a microscope to a surgical system. The method includes the step of providing a microscope having an objective lens and aneyepiece 501. The method also includes the step of installing a beam splitter between the objective lens and the eyepiece, the beam splitter optically coupled to the objective lens and the eyepiece optically coupled to a first split beam path of thebeam splitter 503. The method also includes optically coupling an image sensor to a second split beam path of the beam splitter; the image sensor configured to acquire image data from the field of view of the optical device and transmit an electronic image data signal 505. The method also includes providing a display device in electrical communication with the image sensor, the display device configured to receive the electronic image data signal transmitted by the image sensor and to display an acquired image representing the image data acquired by theimage sensor 507. - The step of providing a microscope having an objective lens and an
eyepiece 501 can be performed, for example but not limited to, by providing a microscope having an objective 101 and aneyepiece 109 as described above with reference toFIG. 1 . - The step of installing a beam splitter between the objective lens and the eyepiece, the beam splitter optically coupled to the objective lens and the eyepiece optically coupled to a first split beam path of the
beam splitter 503 can be performed, for example but not limited to, by installing abeam splitter 107 orbeam splitting apparatus 200 as described above with reference toFIGS. 1 and 2 . - The step of optically coupling an image sensor to a second split beam path of the beam splitter; the image sensor configured to acquire image data from the field of view of the optical device and transmit an electronic image data signal 505 can be performed, for example but not limited to, optically coupling
image sensor 103 tobeam splitter 107 as described above with reference toFIG. 1 . - The step of providing a display device in electrical communication with the image sensor, the display device configured to receive the electronic image data signal transmitted by the image sensor and to display an acquired image representing the image data acquired by the
image sensor 507 can be performed, for example but not limited to, usingimage sensor display device FIGS. 1 and 3 . In some embodiments, the display device can be mounted via a movable mount (e.g., a gimbal, an articulating arm, a hinge, or any other suitable movable mount) for repositioning the display relative to other components of the microscope. As discussed above, such repositioning advantageously allows a user to achieve better ergonomic position during use and/or to enable shared viewing with other members of the surgical team. - Non-Limiting Example Computing Devices
-
FIG. 6 is a block diagram of an exemplary computing device 1010 such as can be used, or portions thereof, in accordance with various embodiments as described above with reference toFIGS. 1-5 . The computing device 1010 includes one or more non-transitory computer-readable media for storing one or more computer-executable instructions or software for implementing exemplary embodiments. The non-transitory computer-readable media can include, but are not limited to, one or more types of hardware memory, non-transitory tangible media (for example, one or more magnetic storage disks, one or more optical disks, one or more flash drives), and the like. For example,memory 1016 included in the computing device 1010 can store computer-readable and computer-executable instructions or software for performing the operations disclosed herein. For example, the memory can storesoftware application 1040 which is programmed to perform various of the disclosed operations as discussed with respect toFIGS. 1-5 . The computing device 1010 can also include configurable and/orprogrammable processor 1012 and associatedcore 1014, and optionally, one or more additional configurable and/or programmable processing devices, e.g., processor(s) 1012′ and associated core(s) 1014′ (for example, in the case of computational devices having multiple processors/cores), for executing computer-readable and computer-executable instructions or software stored in thememory 1016 and other programs for controlling system hardware.Processor 1012 and processor(s) 1012′ can each be a single core processor or multiple core (1014 and 1014′) processor. - Virtualization can be employed in the computing device 1010 so that infrastructure and resources in the computing device can be shared dynamically. A
virtual machine 1024 can be provided to handle a process running on multiple processors so that the process appears to be using only one computing resource rather than multiple computing resources. Multiple virtual machines can also be used with one processor. -
Memory 1016 can include a computational device memory or random access memory, such as but not limited to DRAM, SRAM, EDO RAM, and the like.Memory 1016 can include other types of memory as well, or combinations thereof. - A user can interact with the computing device 1010 through a
visual display device 101, 111A-D, such as a computer monitor, which can display one or more user interfaces 102 that can be provided in accordance with exemplary embodiments. The computing device 1010 can include other I/O devices for receiving input from a user, for example, a keyboard or any suitablemulti-point touch interface 1018, a pointing device 1020 (e.g., a mouse). Thekeyboard 1018 and thepointing device 1020 can be coupled to thevisual display device 101, 111A-D. The computing device 1010 can include other suitable conventional I/O peripherals. - The computing device 1010 can also include one or
more storage devices 1034, such as but not limited to a hard-drive, CD-ROM, or other computer readable media, for storing data and computer-readable instructions and/or software that perform operations disclosed herein.Exemplary storage device 1034 can also store one or more databases for storing any suitable information required to implement exemplary embodiments. The databases can be updated manually or automatically at any suitable time to add, delete, and/or update one or more items in the databases. - The computing device 1010 can include a
network interface 1022 configured to interface via one ormore network devices 1032 with one or more networks, for example, Local Area Network (LAN), Wide Area Network (WAN) or the Internet through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (for example, 802.11, T1, T3, 56 kb, X.25), broadband connections (for example, ISDN, Frame Relay, ATM), wireless connections, controller area network (CAN), or some combination of any or all of the above. Thenetwork interface 1022 can include a built-in network adapter, network interface card, PCMCIA network card, card bus network adapter, wireless network adapter, USB network adapter, modem or any other device suitable for interfacing the computing device 1010 to any type of network capable of communication and performing the operations described herein. Moreover, the computing device 1010 can be any computational device, such as a workstation, desktop computer, server, laptop, handheld computer, tablet computer, or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein. - The computing device 1010 can run any
operating system 1026, such as any of the versions of the Microsoft® Windows® operating systems (Microsoft, Redmond, Wash.), the different releases of the Unix and Linux operating systems, any version of the MAC OS® (Apple, Inc., Cupertino, Calif.) operating system for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, or any other operating system capable of running on the computing device and performing the operations described herein. In exemplary embodiments, theoperating system 1026 can be run in native mode or emulated mode. In an exemplary embodiment, theoperating system 1026 can be run on one or more cloud machine instances. -
FIG. 7 is an example computational device block diagram of certain distributed embodiments. AlthoughFIGS. 1 and 6 , and portions of the exemplary discussion above, make reference to acentralized multi-display system 100 operating on a single computing device, one will recognize that various of the modules within themulti-display system 100 may instead be distributed across anetwork 1105 in separate server systems 1101 a-d and possibly in user systems, such as kiosk,desktop computer device 1102, ormobile computer device 1103. For example, users may download an application to their desktop computer device or mobile computer device, which is configured to show an interactive digital rendering of the meeting room and perform the operations of thedisplay device 101, graphical user interface 102, output modules 109A-D, and/or output displays 111A-D. In some distributed systems, the modules of themulti-display system 100 can be separately located on server systems 1101 a-d and can be in communication with one another across thenetwork 1105. - In describing exemplary embodiments, specific terminology is used for the sake of clarity. For purposes of description, each specific term is intended to at least include all technical and functional equivalents that operate in a similar manner to accomplish a similar purpose. Additionally, in some instances where a particular exemplary embodiment includes a plurality of system elements, device components or method steps, those elements, components or steps may be replaced with a single element, component or step. Likewise, a single element, component or step may be replaced with a plurality of elements, components or steps that serve the same purpose. Moreover, while exemplary embodiments have been shown and described with references to particular embodiments thereof, those of ordinary skill in the art will understand that various substitutions and alterations in form and detail may be made therein without departing from the scope of the invention. Further still, other aspects, functions and advantages are also within the scope of the invention.
- Exemplary flowcharts are provided herein for illustrative purposes and are non-limiting examples of methods. One of ordinary skill in the art will recognize that exemplary methods may include more or fewer steps than those illustrated in the exemplary flowcharts, and that the steps in the exemplary flowcharts may be performed in a different order than the order shown in the illustrative flowcharts.
Claims (18)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/034,630 US20210106397A1 (en) | 2015-07-14 | 2020-09-28 | Surgical optical system with heads up display |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562192351P | 2015-07-14 | 2015-07-14 | |
PCT/US2016/042252 WO2017011643A1 (en) | 2015-07-14 | 2016-07-14 | Surgical optical system with heads up display |
US15/870,424 US10820962B2 (en) | 2015-07-14 | 2018-01-12 | Surgical optical system with heads up display |
US17/034,630 US20210106397A1 (en) | 2015-07-14 | 2020-09-28 | Surgical optical system with heads up display |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/870,424 Division US10820962B2 (en) | 2015-07-14 | 2018-01-12 | Surgical optical system with heads up display |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210106397A1 true US20210106397A1 (en) | 2021-04-15 |
Family
ID=57757724
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/870,424 Active 2036-10-04 US10820962B2 (en) | 2015-07-14 | 2018-01-12 | Surgical optical system with heads up display |
US17/034,630 Abandoned US20210106397A1 (en) | 2015-07-14 | 2020-09-28 | Surgical optical system with heads up display |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/870,424 Active 2036-10-04 US10820962B2 (en) | 2015-07-14 | 2018-01-12 | Surgical optical system with heads up display |
Country Status (4)
Country | Link |
---|---|
US (2) | US10820962B2 (en) |
EP (1) | EP3322329A1 (en) |
CA (1) | CA2992454A1 (en) |
WO (1) | WO2017011643A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024062326A1 (en) * | 2022-09-21 | 2024-03-28 | Alcon Inc. | Low profile optical systems for surgical procedures |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11006093B1 (en) | 2020-01-22 | 2021-05-11 | Photonic Medical Inc. | Open view, multi-modal, calibrated digital loupe with depth sensing |
KR20240028675A (en) * | 2022-08-25 | 2024-03-05 | 최성백 | Digital Microscope for Medical procedure |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5235457A (en) * | 1987-09-24 | 1993-08-10 | Washington University | Kit for converting a standard microscope into a single aperture confocal scanning epi-illumination microscope |
US5662111A (en) * | 1991-01-28 | 1997-09-02 | Cosman; Eric R. | Process of stereotactic optical navigation |
US5239984A (en) * | 1991-11-25 | 1993-08-31 | Cane Richard M | Hand-held opto-diagnostic instrument system |
US5608210A (en) * | 1994-09-29 | 1997-03-04 | Esparza; Joel | Infrared aided method and apparatus for venous examination |
JP2003522576A (en) * | 2000-02-18 | 2003-07-29 | ウィリアム・ボーモント・ホスピタル | Cone beam computed tomography apparatus with flat panel imaging device |
US20040224279A1 (en) * | 2003-05-06 | 2004-11-11 | Ormco Corporation | Dental display system |
US8688588B2 (en) * | 2007-06-08 | 2014-04-01 | Sandisk Technologies Inc. | Method for improving accuracy of a time estimate used in digital rights management (DRM) license validation |
DE102008062650B9 (en) * | 2008-12-17 | 2021-10-28 | Carl Zeiss Meditec Ag | Surgical microscope for observing infrared fluorescence and methods for this |
JP2011000258A (en) | 2009-06-18 | 2011-01-06 | Fujifilm Corp | Endoscope system, endoscope, and distance-irradiation angle measuring method |
FR2975804B1 (en) * | 2011-05-27 | 2022-06-17 | Lvmh Rech | METHOD FOR CHARACTERIZING THE TINT OF THE SKIN OR DYES |
US9438773B2 (en) * | 2011-10-21 | 2016-09-06 | Transamerican Technologies International | Universal dual camera adapter |
US9788730B2 (en) * | 2012-03-08 | 2017-10-17 | Dermasensor, Inc. | Optical process and apparatus for non-invasive detection of melanoma |
DE102012019472B4 (en) * | 2012-09-28 | 2023-05-04 | Carl Zeiss Microscopy Gmbh | Optical filter device, in particular for microscopes, and microscope |
EP2951779B1 (en) | 2013-01-29 | 2018-09-05 | Brainlab AG | Three-dimensional image segmentation based on a two-dimensional image information |
US9989762B2 (en) * | 2014-03-19 | 2018-06-05 | Perceptoscope | Optically composited augmented reality pedestal viewer |
-
2016
- 2016-07-14 CA CA2992454A patent/CA2992454A1/en not_active Abandoned
- 2016-07-14 EP EP16825165.0A patent/EP3322329A1/en not_active Withdrawn
- 2016-07-14 WO PCT/US2016/042252 patent/WO2017011643A1/en active Application Filing
-
2018
- 2018-01-12 US US15/870,424 patent/US10820962B2/en active Active
-
2020
- 2020-09-28 US US17/034,630 patent/US20210106397A1/en not_active Abandoned
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024062326A1 (en) * | 2022-09-21 | 2024-03-28 | Alcon Inc. | Low profile optical systems for surgical procedures |
Also Published As
Publication number | Publication date |
---|---|
US10820962B2 (en) | 2020-11-03 |
EP3322329A1 (en) | 2018-05-23 |
CA2992454A1 (en) | 2017-01-19 |
US20180235723A1 (en) | 2018-08-23 |
WO2017011643A1 (en) | 2017-01-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210106397A1 (en) | Surgical optical system with heads up display | |
Hanna et al. | Augmented reality technology using Microsoft HoloLens in anatomic pathology | |
US9916435B2 (en) | System and method for the display of restricted information on private displays | |
Chhablani et al. | Smartphones in ophthalmology | |
Pantanowitz et al. | Experience with multimodality telepathology at the University of Pittsburgh Medical Center | |
JP5172085B2 (en) | How to display medical information based on gaze detection | |
US7613478B2 (en) | Method and system for portability of clinical images using a high-quality display and portable device | |
Romero Lauro et al. | Digital pathology consultations—a new era in digital imaging, challenges and practical applications | |
US11830614B2 (en) | Method and system for optimizing healthcare delivery | |
JP2021500690A (en) | Self-expanding augmented reality-based service instruction library | |
US20090182577A1 (en) | Automated information management process | |
US8601385B2 (en) | Zero pixel travel systems and methods of use | |
KR20140070376A (en) | Wireless surgical loupe | |
US20100131873A1 (en) | Clinical focus tool systems and methods of use | |
von Wangenheim et al. | Creating a web infrastructure for the support of clinical protocols and clinical management: an example in teledermatology | |
Dietz et al. | Review of the use of telepathology for intraoperative consultation | |
US11145421B2 (en) | System and method for remote medical information exchange | |
Widmer et al. | Facilitating medical information search using Google Glass connected to a content-based medical image retrieval system | |
Bae et al. | Smartphone-based endoscope system for advanced point-of-care diagnostics: feasibility study | |
Armstrong et al. | Anterior segment imaging devices in ophthalmic telemedicine | |
Dutt et al. | A novel approach to anterior segment imaging with smartphones in the COVID-19 era | |
Mistry et al. | Endoscope-i: an innovation in mobile endoscopic technology transforming the delivery of patient care in otolaryngology | |
US20180247701A1 (en) | System and Method to Combine, View, and Distribute Patient Health Records from Multiple Sources | |
US20120131436A1 (en) | Automated report generation with links | |
Feroz et al. | Telepathology: an update on applications, latest advances, and current status in Indian scenario |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: TSME, LLC, NEW HAMPSHIRE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PAGONIS, TOM C.;BEYLIN, SIMON;KANG, MICHAEL J.;AND OTHERS;SIGNING DATES FROM 20200901 TO 20200911;REEL/FRAME:054542/0353 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |