EP3554411B1 - Robotic surgical system with virtual control panel for tool actuation - Google Patents

Robotic surgical system with virtual control panel for tool actuation Download PDF

Info

Publication number
EP3554411B1
EP3554411B1 EP17832119.6A EP17832119A EP3554411B1 EP 3554411 B1 EP3554411 B1 EP 3554411B1 EP 17832119 A EP17832119 A EP 17832119A EP 3554411 B1 EP3554411 B1 EP 3554411B1
Authority
EP
European Patent Office
Prior art keywords
surgical instrument
control panel
surgical
processor
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP17832119.6A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3554411C0 (en
EP3554411A1 (en
Inventor
Jeffrey S. Swayze
Joshua Young
Andrew BECKMAN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ethicon LLC
Original Assignee
Ethicon LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ethicon LLC filed Critical Ethicon LLC
Publication of EP3554411A1 publication Critical patent/EP3554411A1/en
Application granted granted Critical
Publication of EP3554411C0 publication Critical patent/EP3554411C0/en
Publication of EP3554411B1 publication Critical patent/EP3554411B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/313—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for introducing through surgical openings, e.g. laparoscopes
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0033—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
    • A61B5/0036—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room including treatment, e.g., using an implantable medical device, ablating, ventilating
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0082—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence adapted for particular medical purposes
    • A61B5/0084—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence adapted for particular medical purposes for introduction into the body, e.g. by catheters
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/06—Devices, other than using radiation, for detecting or locating foreign bodies ; determining position of probes within or on the body of the patient
    • A61B5/061—Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body
    • A61B5/064—Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body using markers
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/74—Details of notification to user or communication with user or patient ; user input means
    • A61B5/742—Details of notification to user or communication with user or patient ; user input means using visual displays
    • A61B5/743—Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/361—Image-producing devices, e.g. surgical cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/147—Details of sensors, e.g. sensor lenses
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
    • A61B17/28—Surgical forceps
    • A61B17/29—Forceps for use in minimally invasive surgery
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2065—Tracking using image or pattern recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2068—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis using pointers, e.g. pointers having reference marks for determining coordinates of body points
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/25—User interfaces for surgical systems
    • A61B2034/256—User interfaces for surgical systems having a database of accessory information, e.g. including context sensitive help or scientific articles
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/36—Image-producing devices or illumination devices not otherwise provided for
    • A61B90/37—Surgical systems with images on a monitor during operation
    • A61B2090/373—Surgical systems with images on a monitor during operation using light, e.g. by using optical scanners
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
    • A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
    • A61B2090/3937—Visible markers
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/25—User interfaces for surgical systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/30—Surgical robots
    • A61B34/37—Master-slave robots
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images
    • G06V2201/034—Recognition of patterns in medical or anatomical images of medical instruments

Definitions

  • Methods and devices are provided for minimally invasive surgery, and in particular for providing virtual control panel of a surgical instrument.
  • MIS Minimally invasive surgical
  • Laparoscopic surgery is one type of MIS procedure in which one or more small incisions are formed in the abdomen and a trocar is inserted through the incision to form a pathway that provides access to the abdominal cavity.
  • the trocar is used to introduce various instruments and tools into the abdominal cavity, as well as to provide insufflation to elevate the abdominal wall above the organs.
  • the instruments and tools can be used to engage and/or treat tissue in a number of ways to achieve a diagnostic or therapeutic effect.
  • Endoscopic surgery is another type of MIS procedure in which elongate flexible shafts are introduced into the body through a natural orifice.
  • Telesurgery is a general term for surgical operations using systems where the surgeon uses some form of remote control, e.g., a servomechanism, or the like, to manipulate surgical instrument movements, rather than directly holding and moving the tools by hand.
  • the surgeon is typically provided with an image of the surgical environment on a visual display at a location remote from the patient. The surgeon can typically perform the surgical procedure at the location remote from the patient whilst viewing the end effector movement on the visual display during the surgical procedure.
  • An example of an operating room with surgical site awareness can be found in WO2016133644A1 .
  • An example of interactive user interfaces for robotic minimally invasive surgical systems can be found in US2009036902A1 .
  • Another example of an interactive user interface for minimally invasive surgery can be found in JP2016530950 and corresponding US patent application publication 2017/0196560A1 .
  • Methods, devices, and systems are provided for displaying a modified image of a surgical environment wherein the modified image includes at least a portion of a surgical instrument and a control panel associated with the surgical instrument.
  • Example methods disclosed herein are not explicitly recited by the wording of the claims (but are considered as useful for understanding the invention). For example, the example methods can be used with the devices of the present invention.
  • the control panel displayed in the modified image is active in that functions associated with the surgical instrument can be actuated by interacting with various specific elements of the control panel displayed in the modified image.
  • the invention concerns a surgical system as defined by appended independent claim 1.
  • a surgical system that includes a detector comprising an array of pixels configured to detect light reflected by a surgical instrument and generate a first signal comprising a first dataset representative of a visible image of the surgical instrument.
  • the surgical system also includes a processor configured to receive the first signal, generate a modified image of the surgical instrument that includes a control panel that includes one or more control elements representative of one or more operating parameters of at least one of the surgical system and the surgical instrument.
  • the processor is further configured to receive an input to the control panel from a user, the input being effective to change one of the operating parameters.
  • the processor is also configured to generate a command signal based on the input to the control panel to change one of the operating parameters.
  • the surgical system can vary in a number of ways.
  • the modified image is displayed on a display device.
  • the processor identifies from the first dataset data representative of the surgical instrument based on position of the detector with respect to the surgical instrument.
  • At least one of the position and shape of the control panel in the modified image is adjusted based on the orientation of the surgical instrument. In another embodiment at least one of the position and shape of the control panel with respect to the surgical instrument remains fixed.
  • the surgical system includes an input device to receive the input from the user, and in one embodiment the input device is a user interface.
  • the processor generates a pointer that moves relative to the modified image based on the input by the user.
  • the command signal is generated when the user actuates one of the one or more control elements using the user interface device.
  • the processor is also configured to receive a second signal representative of information related to the surgical instrument.
  • the control panel is generated based on the second signal and data related to the surgical instrument that is stored in a database.
  • the surgical instrument can be any type of instrument that is used in a surgical procedure.
  • the surgical instrument is one that includes functionalities including at least one of tissue cutting, tissue stapling, and delivery of energy to tissue.
  • a robotic surgical system in another aspect, includes a robotic arm, and a tool assembly removably coupled to the robotic arm and including a shaft extending distally from a housing and an end effector coupled to a distal end of the shaft, the end effector being configured to treat tissue.
  • the robotic surgical system also includes a camera system configured to capture light reflected by at least one of the end effector and the shaft and generate a signal representative of a visible image of the at least one of the end effector and shaft.
  • the robotic surgical system further includes a processor configured to receive the signal and generate a modified image of the at least one of the end effector and shaft that includes a control panel.
  • the control panel in the modified image comprises one or more control elements representative of one or more operating parameters of the robotic surgical system.
  • the processor is configured to receive an input to the control panel from a user, the input being effective to change one of the operating parameters, and generate a command signal based on the input to change the one of the operating parameters.
  • the robotic surgical system includes a display device, which can be wall or table mounted or on an accessory (e.g., head set, glasses, etc.), configured to display a modified image of the end effector and the shaft.
  • the robotic surgery system also includes an input device to receive the input from the user.
  • the input device is a user input device.
  • the processor generates a pointer that moves relative to the modified image based on the input by the user.
  • An example surgical method includes receiving a first signal representative of an image of a surgical instrument and generating a modified image of the surgical instrument that includes a control panel.
  • the control panel comprises one or more control elements representative of one or more operating parameters of at least one of the surgical system and the surgical instrument.
  • the surgical method also includes receiving an input to the control panel from a user, the input being effective to change one of the operating parameters.
  • the surgical method further includes generating a command signal based on the input to change the one of the operating parameters.
  • like-named components of the embodiments generally have similar features, and thus within a particular embodiment each feature of each like-named component is not necessarily fully elaborated upon.
  • linear or circular dimensions are used in the description of the disclosed systems, devices, and methods, such dimensions are not intended to limit the types of shapes that can be used in conjunction with such systems, devices, and methods.
  • a person skilled in the art will recognize that an equivalent to such linear and circular dimensions can easily be determined for any geometric shape. Sizes and shapes of the systems and devices, and the components thereof, can depend at least on the anatomy of the subject in which the systems and devices will be used, the size and shape of components with which the systems and devices will be used, and the methods and procedures in which the systems and devices will be used.
  • a surgical system that allows a user (e.g., a surgeon) to control the operation of surgical instruments.
  • the surgical instruments controlled by the described system are advanced surgical tools, for example, endocutters, and/or tools that have functionalities including at least one of cutting tissue, stapling tissue, and delivering energy to tissue.
  • the controls of advanced tools may not be easily mapped to those of standard robotic tools (e.g., graspers, needle drivers etc.). Therefore, there is a need to develop control systems that allow a user to seamlessly control advanced surgical tools during surgery.
  • the systems and methods described herein, however, are not limited to advanced surgical tools, and can be applied to virtually any surgical instrument used in a surgical procedure, including robotic tools.
  • an image of the surgical environment as well as relevant portions of the surgical instrument is typically generated and displayed on a display to the surgeon, such as on a video monitor. Such an image is typically displayed in real-time.
  • the image of the surgical instrument is modified to include a virtual control panel.
  • the surgeon (or any other user) can interact with the control panel through a user interface device (e.g., one that is able to move in 3D space)
  • the control panel can be pinned to a fixed location in the display, or can be moved to various locations in the modified image.
  • the location of the control panel can be fixed relative to a portion of the surgical instrument (e.g., shaft of a surgical instrument), and the control panel can follow the surgical instrument as it moves in the modified image.
  • the control panel can change in size, shape, aspect ratio, and perspective to match the 2D or 3D orientation of the surgical instrument and/or the surrounding environment.
  • the control panel can appear in two dimensions (if viewed on a 2D display) or in three dimensions ((if viewed stereoscopically via headset, goggles, or glasses).
  • the control panel can be configured to eliminate or minimize any obstruction to the field of view of the surgical environment.
  • portions of the control panel can be transparent or semi-transparent, and the degree of transparency can be adjustable.
  • the surgeon can change the location of the control panel in the modified image, for example, by dragging and dropping the control panel to a desired location.
  • a control panel can be customized for the surgical system and/or the instrument and/or for a particular procedure or use of the surgical system or instrument.
  • the operating parameters of an endocutter are different from those of a needle driver or an energy-delivering tool, and therefore, the control panel of an endocutter may be different from that of needle driver or an energy-delivering tool.
  • the relevant operating parameters of a surgical instrument may change based on the needs of a surgical procedure at any given time. For example, once an endocutter has been positioned in a desired location and clamped on a target tissue, the relevant operating parameters may include articulation angle, knife position, timing of jaw opening, and firing etc.
  • the image of the control panel that is displayed after the endocutter has been clamped onto tissue will include the control for the operating parameters mentioned above.
  • FIG. 1 is a perspective view of one embodiment of a surgical robotic system 100 that can be used in telesurgery.
  • the system 100 includes a patient-side portion 110 that is positioned adjacent to a patient 112, and a user-side portion 111 that is located a distance from the patient, either in the same room and/or in a remote location.
  • the patient-side portion 110 generally includes one or more robotic arms 120 and one or more tool assemblies 130 that are configured to releasably couple to a robotic arm 120.
  • the user-side portion 111 generally includes a vision system 113 for viewing the patient 112 and/or surgical environment, and a control system 115 for controlling the movement of the robotic arms 120 and each tool assembly 130 during a surgical procedure.
  • the control system 115 can have a variety of configurations and it can be located adjacent to the patient, e.g., in the operating room, remote from the patient, e.g., in a separate control room, or it can be distributed at two or more locations.
  • a dedicated system a dedicated system control console can be located in the operating room, and a separate console can be located at a remote location.
  • the control system 115 can include components that enable a user to view a surgical environment of a patient 112 being operated on by the patient-side portion 110 and/or to control one or more parts of the patient-side portion 110 (e.g., to perform a surgical procedure at the surgical environment 112).
  • control system 115 can also include one or more manually-operated input devices, such as a joystick, exoskeletal glove, a powered and gravity-compensated manipulator, or the like. These input devices can control teleoperated motors which, in turn, control the movement of the surgical system, including the robotic arms 120 and tool assemblies 130.
  • manually-operated input devices such as a joystick, exoskeletal glove, a powered and gravity-compensated manipulator, or the like.
  • These input devices can control teleoperated motors which, in turn, control the movement of the surgical system, including the robotic arms 120 and tool assemblies 130.
  • the patient-side portion can also have a variety of configurations. As depicted in FIG. 1 , the patient-side portion 110 can couple to an operating table 114. However, in other embodiments, the patient-side portion 110 can be mounted to a wall, to the ceiling, to the floor, or to other operating room equipment. Further, while the patient-side portion 110 is shown as including two robotic arms 120, more or fewer robotic arms 120 can be included. Furthermore, the patient-side portion 110 can include separate robotic arms 120 mounted in various positions, such as relative to the surgical table 114. Alternatively, the patient-side portion 110 can include a single assembly that includes one or more robotic arms 120 extending therefrom.
  • FIG. 2 is a schematic view of an example of surgical system 200 configured to generate modified images of a surgical environment (e.g., surgical instrument 204, target tissues, tissues surrounding target tissues, etc.) in real-time.
  • the surgical system 200 includes a surgical instrument 204, a controller 202 to control the operation of the surgical instrument 204, a camera module 210 configured to capture visible-light images of the surgical instrument 204, and relay one or more signals related to the captured image to a processor 216.
  • the processor 216 can also communicate with the controller 202.
  • the processor can receive operating parameters of the surgical instrument 204 from the controller 202, and transmit to the controller 202 control signals that can change the operating parameters.
  • the processor 216 can communicate with a database 218 which can store information related to the surgical instrument 204 (e.g., control panel information).
  • the processor 216 can generate a modified image that includes the visible-light image from the camera 210, and a virtual control panel that allows a user to control the operating parameters of the surgical instrument 204.
  • the modified image can be displayed on a display 222, which can be wall or table mounted or on an accessory worn by the surgeon.
  • the surgical system 200 also includes an input device 224 which can communicate with the processor 216.
  • a user e.g., surgeon
  • can interact with the modified image e.g., change operating parameters in the control panel
  • the input device 224 can communicate with the input device 224.
  • Signals in the surgical system 200 can be communicated wirelessly (Bluetooth, WiFi, etc.) or through a data cable (e.g., optical fiber, coaxial cable, etc.).
  • a light source (not shown) can generate visible light which is reflected by the surgical environment.
  • the reflected light can be visible light (e.g., having a wavelength of about 400 nm to 800 nm) or light of a wavelength that is outside of the visible spectrum (e.g., infrared and ultraviolet light).
  • a portion of the reflected visible-light (e.g., having a wavelength of about 400 nm to 800 nm) is captured by the camera module 210.
  • the camera module 210 comprises a lens 206 configured to focus visible-light onto a detector 212. The quality of the visible-light image can be improved, for example, by placing detector 212 in the focal plane of the lens 206.
  • the detector 212 is able to detect light reflected by the surgical instrument.
  • an exemplary detector 212 comprises an array of photosensitive sites (e.g., 212a-c, etc.), which can absorb electromagnetic radiation impinging on the site, and generate an electrical signal (e.g., voltage signal, current signal, etc.) that is representative of the impinged radiation.
  • an electrical signal e.g., voltage signal, current signal, etc.
  • the strength of the electrical signal can be proportional to the intensity of the impinged electromagnetic radiation.
  • Photosensitive sites typically have a spectral range which determines the range of frequencies that can be efficiently detected by the site.
  • a silicon (Si) photosensitive site can detect visible to near infrared radiation (spectral range 400-1000 nm), and a germanium (Ge) or indium gallium arsenide (InGaAs) photosensitive site can detect near infrared radiation (spectral range 800-2600 nm).
  • Si silicon
  • Ge germanium
  • InGaAs indium gallium arsenide
  • a suitable type of photosensitive site that is appropriate for the spectral range of the electromagnetic radiation that one wants to detect can be selected by a person skilled in the art.
  • a photosensitive site can be configured to detect a desired wavelength (or a narrow range of wavelength around the desired wavelength) of electromagnetic radiation that lies within its spectral range by using an optical filter.
  • the optical filter which is placed in the path of the electromagnetic radiation directed towards the photosensitive site, filters out all radiation except for that corresponding to the desired wavelength.
  • a Si photosensitive site e.g., 212a
  • a green color filter will primarily detect green light (approximately 500 nm).
  • a detector detects an image of the surgical environment by combining the images of different regions of the object captured by various photosensitive sites in the detector.
  • a photosensitive site therein e.g., 212a, 212b, 212c, etc.
  • ADC analog-to-digital converter
  • the digital signal can also include information related to the frequency (color) of the detected radiation.
  • the values of the digital signals from the various photosensitive sites are representative of the image of the surgical instrument. There can be a one-to-one relationship between a digital signal value stored in the image dataset, and the photosensitive site that has produced the digital signal value (e.g., the digital signal value can include information that identifies photosensitive site that has generated the digital signal). Therefore, by identifying a digital signal value in the image dataset the photosensitive site that generated the digital value can be identified (or vice-versa).
  • the processor then generates the image of the surgical environment from the image dataset that can be displayed on a display device 222 (e.g., a monitor).
  • a display device 222 e.g., a monitor.
  • Each pixel in the display device can represent one digital signal value in the image dataset. In other words, each pixel in the display device can represent the radiation detected by a unique photosensitive site in the detector 212.
  • a colored image of a surgical environment can be generated by placing optical filters (or an array of optical filters) in the path of the electromagnetic radiation directed towards a detector.
  • optical filters or an array of optical filters
  • an array of color filters e.g., Bayer filter, RGBE filter, CYYM filter, CYGM filter, etc.
  • each photosensitive site receives electromagnetic radiation of a particular wavelength (or color).
  • each photosensitive site detects one of red, blue or green color.
  • the processor can use a demosaicing algorithm to process an image dataset obtained using a Bayer filter to generate a "full-color" image (i.e., an image with multiple colors).
  • the light image dataset can include information related to the intensity and wavelength (color) of the detected light for each photosensitive site.
  • the controller 202 is configured to control the operation of the surgical instrument 204. For example, if the surgical instrument 204 is an endocutter, the controller can change its operating parameters (e.g., articulation angle, knife position, timing of opening and firing, etc.). The controller 202 can also control the energy delivered by the surgical instrument 204 to a target tissue. For example, the controller 202 may drive an ultrasonic transducer in the surgical instrument and/or generate radio frequency energy which the surgical instrument can deliver to a tissue through an energy-delivering end effector. The controller 202 is configured to receive control signals from the processor 216 and such control signals can change the operating parameters of the surgical instrument 204.
  • the controller 202 can change its operating parameters (e.g., articulation angle, knife position, timing of opening and firing, etc.).
  • the controller 202 can also control the energy delivered by the surgical instrument 204 to a target tissue.
  • the controller 202 may drive an ultrasonic transducer in the surgical instrument and/or generate radio frequency energy which the surgical instrument can deliver to
  • the controller 202 can transmit a signal to the processor 216 that includes information related to the operating parameters and identity of the surgical instrument 204.
  • the signal may be transmitted when the controller 202 receives a request signal from the processor 216 requesting information related to the surgical instrument 204.
  • the controller 202 can include a memory device that has a log file which keeps track of the operating parameters of the surgical instrument 202.
  • the log file (or a part thereof) can be transmitted to the processor 216.
  • the processor 216 and the controller 202 may not communicate directly. For example, their communication may be routed through one or more devices (e.g., processors, routers etc.).
  • the processor 216 can identify the surgical instrument 204 and its operating parameters, for example, based on the signal from the controller 202.
  • Processor 216 can access information associated with the surgical instrument 204 that may be stored in the database 218 or that may be communicated to the processor from the surgical instrument itself.
  • the processor may search the database 21 8 for a template of a virtual control panel related to surgical instrument 204, or it can receive such a template from the surgical instrument.
  • the database may also contain a control panel template that has been customized by the user based on their specific requirements. The choice of the template may also be based on the operating parameters of the surgical instrument 204.
  • the processor 216 can modify the light image of the surgical environment obtained from the camera module 210 by superimposing the control panel 232 onto the image. The modified image can be displayed on the display 222.
  • the modified image includes the image of the surgical environment 230 and the control panel 232.
  • the control panel can be located at a fixed location on the display 222 (e.g., top left, top right, bottom left, bottom right, etc.) or the location of the control panel can change, such as based on user commands.
  • a user can interact with the control panel 232 through the input device 224.
  • the input device 224 is a user interface device
  • processor 216 can generate a pointer in the modified image that can track the motion of the user interface device.
  • the user can use the user interface device to scroll, drag, actuate (e.g., click, double click, etc.) on various control elements in the control panel.
  • the input device 224 may appear to the user as a 3D pointer that moves within the 3D space surrounding the device and/or control panel. Based on this input, the processor 216 can generate a control signal to change one or more operating parameters, and such a control signal is transmitted to the controller 202.
  • FIG. 4 illustrates an example of the control panel 232 when the surgical instrument 204 is an endocutter.
  • the control panel 232 comprises various control elements or buttons (402-414).
  • the user may open or close the jaws of the endocutter by actuating virtual buttons 402, 404, respectively.
  • the user may change the articulation angle to the left or right by actuating virtual buttons 410a and 410b, respectively, and button 408 can be actuated to return the tool to its home position.
  • Window 412 within control panel 232 indicates the degree of articulation associated with the surgical instrument.
  • the user can control the knife action by actuating fire button 406 to cause the knife to fire.
  • Part of the virtual control panel also includes a knife position tab 414.
  • the control elements in windows 412, 414 can be used to change the articulation angle and the knife positon, respectively, although it is understood that windows 412, 414 can be configured only to display information and not to enable control of the instrument.
  • a user can vary the degree of transparency of the control panel 232. For example, if the transparency of the control panel is lowered, the visibility of the surgical environment behind the control panel decreases and vice versa.
  • the user may want to adjust the transparency of the control panel, for example, when the control panel is overlaid on an area of the surgical environment that needs to be monitored (e.g., target tissue).
  • the user may move the control panel in the modified image (e.g., by dragging and dropping using the input device 224).
  • the control panel 232 can follow the motion of the surgical instrument 204 in the modified image.
  • the processor 216 can be configured to identify the image of the surgical instrument 204 from the image of the surgical environment (obtained from the camera module 210), and superimpose the control panel 232 at a fixed position relative to the image of the image of the surgical instrument.
  • FIG. 5 illustrates an example of a modified image 500 of an endocutter 502 performing a surgical procedure on a target tissue 506.
  • the control panel 504 is located near the distal end of the endocutter so that it is most easily viewed by a surgeon.
  • the control panel 506 can be configured to appear after the endocutter is locked to the location, for example, by the release of a clutch pedal by the user.
  • the control panel 232 can be located on or near the shaft of the endocutter.
  • the processor can identify data related to the image of the surgical instrument 204 (e.g., endocutter) from the reflected light image dataset generated by the camera module 210. In one embodiment, the data is identified based on a predetermined relative position between the first detector 212 and the surgical instrument 204.
  • the camera module 210 is attached to the surgical instrument 204 such that the relative position of the surgical instrument with respect to the camera module 210 remains fixed. This can be done, for example, by having a mounting feature on the surgical instrument to which the camera module 210 can be removably attached. Devices (detector 212, lens 206, etc.) inside the camera module 210 can be positioned in a predetermined configuration.
  • the devices can be attached to piezoelectric actuators that allow them to move relative to one another. This can allow the detector to detect a sharp image of the surgical instrument. For example, it can be desirable to place the detector 212 in the focal plane of the lens 206. Mechanical movements and thermal expansion of the camera module 210 and the devices therein can move the detectors out of the focal plane of the lens. The detectors can be adjusted back into the focal plane by the piezoelectric actuators that can be controlled by the processor 216, or by an input from a user. The surgical instrument 204 and the camera module 210 (and the devices inside the camera module) can be adjusted to a desired predetermined position prior to the insertion of camera module 210 and surgical instrument 204 in the surgical environment.
  • the photosensitive sites in detectors 212 that capture the image of the surgical instrument 204 can be identified based on the predetermined orientation of the detector 212 and the surgical instrument 204. Information related to the location of the aforementioned photosensitive sites can be stored in the database 218.
  • the processor 216 can identify surgical instrument image data in the reflected light image dataset. This can be done, for example, by arranging the image data, captured by the photosensitive sites, in a predetermined pattern in the light image. For example, the image data captured by the photosensitive site 212a can be placed at a predetermined location in the visible-light dataset. Information about this relationship can be stored in an index data file in the database 218. Based on the index data file, the processor 218 can identify the image data (from the reflected image dataset) corresponding to the image detected by the photosensitive site 212a. Alternately, the image data can include information that identifies the photosensitive site that generated it.
  • the surgical instrument is identified in the light image based on one or more markers on the surgical instrument 204, and multiple cameras 610, 612, 614, 616 are used to image the surgical environment.
  • the surgical instrument 204 includes a number of markers 602, 604, 606, 608 located on its surface.
  • the markers are regions on the surgical instrument 204 that reflect electromagnetic radiation of a given frequency.
  • the markers can be configured to reflect light of a certain color. The color of the markers can be selected such that the color is not naturally present in the surgical environment (e.g., green, blue, etc.).
  • the processor 216 thus identifies photosensitive sites that detect the image of the markers based on marker color.
  • the reflected light image dataset can include, for each photosensitive site, information related to the color of the detected light.
  • the processor can search in the reflected light image dataset for data representative of the color of the marker.
  • the processor 216 identifies the markers 602, 604, 606, 608 (and therefore the relative positions of the markers) in the reflected light image and compares this information with data from a database of surgical instruments stored in the database 218.
  • the database includes information related to marker color and marker position for various surgical instruments. Additionally, for a given surgical instrument, the database may include information related to the relative position of the makers in an image of the surgical instruments from multiple viewpoints. For example, the relative positions of the markers 602, 604, 606, 608 in the image of the surgical instrument 204 in the embodiment of FIG. 6 will depend on the relative position of the camera (e.g., 610, 612, 614, 616) that captures the image.
  • the processor can use an image recognition algorithm to identify the data in the visible-light dataset that represents the image of the surgical instrument.
  • the image recognition algorithm receives input information related to the position of the markers in the captured image, and information related to various surgical instruments stored in the database 218.
  • the processor compares the relative positions of markers in the captured image with the orientation information of markers of the devices in the database 218 by using various pattern recognition techniques. Based on this comparison, the processor can identify data representative of image of the surgical instrument in the visible-image dataset.
  • the processor can use a pattern recognition algorithm and/or device database to work with devices that are not readily known to or controlled by the robotic surgical system (i.e., a hand-operated stapler, retractors, etc.)
  • markers may comprise a specific geometric shape and/or color, either as something that is applied to the device (e.g., green dots) or the color can be inherent of typical devices (e.g., silver straight jaws and blackened shaft).
  • the processor 216 After the processor 216 identifies the image of the surgical instrument 204, it can provide the user of the surgical system 200 with an option to choose the location of the control panel 232.
  • the user may drag the control panel 232 to a desired orientation relative to the image of the surgical instrument in the modified image (e.g., at the proximal or distal ends of the surgical instrument, shaft of the surgical instrument, etc.) and lock the position of the control panel 232.
  • the orientation of the control panel 232 is accordingly adjusted to maintain the locked relative orientation.
  • One or more aspects or features of the subject matter described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and/or combinations thereof.
  • ASICs application specific integrated circuits
  • FPGAs field programmable gate arrays
  • These various aspects or features can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
  • the programmable system or computer system may include clients and servers.
  • a client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
  • the computer programs which can also be referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logical programming language, and/or in assembly/machine language.
  • machine-readable medium refers to any computer program product, apparatus and/or device, such as for example magnetic discs, optical disks, memory, and Programmable Logic Devices (PLDs), used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.
  • machine-readable signal refers to any signal used to provide machine instructions and/or data to a programmable processor.
  • the machine-readable medium can store such machine instructions non-transitorily, such as for example as would a non-transient solid-state memory or a magnetic hard drive or any equivalent storage medium.
  • the machine-readable medium can alternatively or additionally store such machine instructions in a transient manner, such as for example as would a processor cache or other random access memory associated with one or more physical processor cores.
  • one or more aspects or features of the subject matter described herein can be implemented on a computer having a display device, such as for example a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to a user and a keyboard and a pointing device, e.g., a mouse, a trackball, etc., by which a user may provide input to the computer.
  • a display device such as for example a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to a user and a keyboard and a pointing device, e.g., a mouse, a trackball, etc.
  • CTR cathode ray tube
  • LCD liquid crystal display
  • LED light emitting diode
  • Other kinds of devices can be used to provide for interaction with a user as well.
  • feedback provided to a user can be any form of sensory feedback, such as for example visual feedback, auditory feedback, or tactile feedback; and input from a user may be received in any form, including, but not limited to, acoustic, speech, or tactile input.
  • Other possible input devices include, but are not limited to, touch screens or other touch-sensitive devices such as single or multi-point resistive or capacitive trackpads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.
  • FIG. 7 illustrates an exemplary embodiment of a computer system 700.
  • the computer system 700 includes one or more processors 702 which can control the operation of the computer system 700.
  • processors are also referred to herein as “controllers.”
  • the processor(s) 702 can include any type of microprocessor or central processing unit (CPU), including programmable general-purpose or special-purpose microprocessors and/or any one of a variety of proprietary or commercially available single or multi-processor systems.
  • the computer system 700 can also include one or more memories 704, which can provide temporary storage for code to be executed by the processor(s) 702 or for data acquired from one or more users, storage devices, and/or databases.
  • the memory 704 can include read-only memory (ROM), flash memory, one or more varieties of random access memory (RAM) (e.g., static RAM (SRAM), dynamic RAM (DRAM), or synchronous DRAM (SDRAM)), and/or a combination of memory technologies.
  • ROM read-only memory
  • flash memory one or more varieties of random access memory (RAM) (e.g., static RAM (SRAM), dynamic RAM (DRAM), or synchronous DRAM (SDRAM)), and/or a combination of memory technologies.
  • RAM random access memory
  • SRAM static RAM
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • the various elements of the computer system 700 can be coupled to a bus system 712.
  • the illustrated bus system 712 is an abstraction that represents any one or more separate physical busses, communication lines/interfaces, and/or multi-drop or point-to-point connections, connected by appropriate bridges, adapters, and/or controllers.
  • the computer system 700 can also include one or more network interface(s) 706, one or more input/output (IO) interface(s) 708, and one or more storage device(s) 710.
  • IO input/output
  • the network interface(s) 706 can enable the computer system 700 to communicate with remote devices, e.g., other computer systems, over a network, and can be, for non-limiting example, remote desktop connection interfaces, Ethernet adapters, and/or other local area network (LAN) adapters.
  • the IO interface(s) 708 can include one or more interface components to connect the computer system 700 with other electronic equipment.
  • the 10 interface(s) 708 can include high speed data ports, such as universal serial bus (USB) ports, 1394 ports, Wi-Fi, Bluetooth, etc.
  • the computer system 700 can be accessible to a human user, and thus the IO interface(s) 708 can include displays, speakers, keyboards, pointing devices, and/or various other video, audio, or alphanumeric interfaces.
  • the storage device(s) 710 can include any conventional medium for storing data in a non-volatile and/or non-transient manner. The storage device(s) 710 can thus hold data and/or instructions in a persistent state, i.e., the value(s) are retained despite interruption of power to the computer system 700.
  • the storage device(s) 710 can include one or more hard disk drives, flash drives, USB drives, optical drives, various media cards, diskettes, compact discs, and/or any combination thereof and can be directly connected to the computer system 700 or remotely connected thereto, such as over a network.
  • the storage device(s) can include a tangible or non-transitory computer readable medium configured to store data, e.g., a hard disk drive, a flash drive, a USB drive, an optical drive, a media card, a diskette, a compact disc, etc.
  • FIG. 7 can be some or all of the elements of a single physical machine. In addition, not all of the illustrated elements need to be located on or in the same physical machine.
  • Exemplary computer systems include conventional desktop computers, workstations, minicomputers, laptop computers, tablet computers, personal digital assistants (PDAs), mobile phones, and the like.
  • PDAs personal digital assistants
  • the computer system 700 can include a web browser for retrieving web pages or other markup language streams, presenting those pages and/or streams (visually, aurally, or otherwise), executing scripts, controls and other code on those pages/streams, accepting user input with respect to those pages/streams (e.g., for purposes of completing input fields), issuing HyperText Transfer Protocol (HTTP) requests with respect to those pages/streams or otherwise (e.g., for submitting to a server information from the completed input fields), and so forth.
  • the web pages or other markup language can be in HyperText Markup Language (HTML) or other conventional forms, including embedded Extensible Markup Language (XML), scripts, controls, and so forth.
  • the computer system 700 can also include a web server for generating and/or delivering the web pages to client computer systems.
  • the computer system 700 can be provided as a single unit, e.g., as a single server, as a single tower, contained within a single housing, etc.
  • the single unit can be modular such that various aspects thereof can be swapped in and out as needed for, e.g., upgrade, replacement, maintenance, etc., without interrupting functionality of any other aspects of the system.
  • the single unit can thus also be scalable with the ability to be added to as additional modules and/or additional functionality of existing modules are desired and/or improved upon.
  • a computer system can also include any of a variety of other software and/or hardware components, including by way of non-limiting example, operating systems and database management systems. Although an exemplary computer system is depicted and described herein, it will be appreciated that this is for sake of generality and convenience. In other embodiments, the computer system may differ in architecture and operation from that shown and described here.
  • components of the invention described herein will be processed before use.
  • a new or used instrument is obtained and if necessary cleaned.
  • the instrument can then be sterilized.
  • the instrument is placed in a closed and sealed container, such as a plastic or TYVEK bag.
  • the container and instrument are then placed in a field of radiation that can penetrate the container, such as gamma radiation, x-rays, or high energy electrons.
  • the radiation kills bacteria on the instrument and in the container.
  • the sterilized instrument can then be stored in the sterile container.
  • the sealed container keeps the instrument sterile until it is opened in the medical facility.
  • the device is sterilized. This can be done by any number of ways known to those skilled in the art including beta or gamma radiation, ethylene oxide, steam, and a liquid bath (e.g., cold soak).
  • a liquid bath e.g., cold soak.
  • An exemplary embodiment of sterilizing a device including internal circuitry is described in more detail in U.S. Pat. No. 8,114,345 filed February 8, 2008 and entitled "System And Method Of Sterilizing An Implantable Medical Device.” It is preferred that device, if implanted, is hermetically sealed. This can be done by any number of ways known to those skilled in the art.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Veterinary Medicine (AREA)
  • Animal Behavior & Ethology (AREA)
  • Molecular Biology (AREA)
  • Medical Informatics (AREA)
  • Biomedical Technology (AREA)
  • Physics & Mathematics (AREA)
  • Pathology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Biophysics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Robotics (AREA)
  • Human Computer Interaction (AREA)
  • Optics & Photonics (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Vascular Medicine (AREA)
  • Theoretical Computer Science (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Surgical Instruments (AREA)
EP17832119.6A 2016-12-19 2017-12-05 Robotic surgical system with virtual control panel for tool actuation Active EP3554411B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/383,026 US10568701B2 (en) 2016-12-19 2016-12-19 Robotic surgical system with virtual control panel for tool actuation
PCT/US2017/064586 WO2018118410A1 (en) 2016-12-19 2017-12-05 Robotic surgical system with virtual control panel for tool actuation

Publications (3)

Publication Number Publication Date
EP3554411A1 EP3554411A1 (en) 2019-10-23
EP3554411C0 EP3554411C0 (en) 2024-03-20
EP3554411B1 true EP3554411B1 (en) 2024-03-20

Family

ID=61003364

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17832119.6A Active EP3554411B1 (en) 2016-12-19 2017-12-05 Robotic surgical system with virtual control panel for tool actuation

Country Status (4)

Country Link
US (3) US10568701B2 (zh)
EP (1) EP3554411B1 (zh)
CN (1) CN110099649B (zh)
WO (1) WO2018118410A1 (zh)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10918445B2 (en) * 2016-12-19 2021-02-16 Ethicon Llc Surgical system with augmented reality display
US10568701B2 (en) 2016-12-19 2020-02-25 Ethicon Llc Robotic surgical system with virtual control panel for tool actuation
US10357325B2 (en) * 2017-01-16 2019-07-23 The Aga Khan University Detection of surgical instruments on surgical tray
JP2021191320A (ja) * 2018-09-11 2021-12-16 ソニーグループ株式会社 手術支援システム、表示制御装置、および表示制御方法
CN109645935A (zh) * 2018-12-14 2019-04-19 张建国 一种内镜辅助用机械手
EP3934559A4 (en) * 2019-03-08 2022-12-07 Covidien LP ROBOTIC SURGICAL PROCESSES
CN110464468B (zh) * 2019-09-10 2020-08-11 深圳市精锋医疗科技有限公司 手术机器人及其末端器械的控制方法、控制装置
CN111513854A (zh) * 2020-05-11 2020-08-11 绍兴梅奥心磁医疗科技有限公司 导管的操控装置及***
JP2022088805A (ja) 2020-12-03 2022-06-15 株式会社メディカロイド ロボット手術システムおよび表示方法
CN113100938B (zh) * 2021-03-26 2022-08-16 复旦大学附属中山医院 外科能量器械及外科手术设备
US20230165658A1 (en) * 2021-11-30 2023-06-01 Stryker Corporation Systems and methods for connecting a medical imaging device to a medical imaging controller

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016530950A (ja) * 2013-08-23 2016-10-06 エシコン・エンド−サージェリィ・エルエルシーＥｔｈｉｃｏｎ Ｅｎｄｏ−Ｓｕｒｇｅｒｙ， ＬＬＣ 外科用器具アセンブリのためのエラー検出装置

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2000065461A1 (en) * 1999-04-22 2000-11-02 Fraunhofer Center For Research In Computer Graphics, Inc. Tools for interacting with virtual environments
US20040243147A1 (en) * 2001-07-03 2004-12-02 Lipow Kenneth I. Surgical robot and robotic controller
US8010180B2 (en) * 2002-03-06 2011-08-30 Mako Surgical Corp. Haptic guidance system and method
US8398541B2 (en) * 2006-06-06 2013-03-19 Intuitive Surgical Operations, Inc. Interactive user interfaces for robotic minimally invasive surgical systems
US10008017B2 (en) * 2006-06-29 2018-06-26 Intuitive Surgical Operations, Inc. Rendering tool information as graphic overlays on displayed images of tools
US8114345B2 (en) 2008-02-08 2012-02-14 Ethicon Endo-Surgery, Inc. System and method of sterilizing an implantable medical device
US8155479B2 (en) 2008-03-28 2012-04-10 Intuitive Surgical Operations Inc. Automated panning and digital zooming for robotic surgical systems
US8418073B2 (en) * 2009-03-09 2013-04-09 Intuitive Surgical Operations, Inc. User interfaces for electrosurgical tools in robotic surgical systems
CN102341046B (zh) * 2009-03-24 2015-12-16 伊顿株式会社 利用增强现实技术的手术机器人***及其控制方法
KR101991034B1 (ko) * 2011-05-31 2019-06-19 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 로봇 수술 기구 엔드 이펙터의 능동 제어
US9498231B2 (en) * 2011-06-27 2016-11-22 Board Of Regents Of The University Of Nebraska On-board tool tracking system and methods of computer assisted surgery
CN103764061B (zh) * 2011-06-27 2017-03-08 内布拉斯加大学评议会 工具承载的追踪***和计算机辅助外科方法
US9123155B2 (en) 2011-08-09 2015-09-01 Covidien Lp Apparatus and method for using augmented reality vision system in surgical procedures
KR101894951B1 (ko) * 2011-09-20 2018-10-15 삼성전자주식회사 촉각 전달 장치 및 촉각 전달 장치의 동작 방법
US10383699B2 (en) 2013-03-15 2019-08-20 Sri International Hyperdexterous surgical system
EP2988696A4 (en) * 2013-04-25 2016-11-30 Intuitive Surgical Operations CONTROL ENTRY VISUALIZATION FIELD FOR SURGICAL EQUIPMENT
US9283048B2 (en) * 2013-10-04 2016-03-15 KB Medical SA Apparatus and systems for precise guidance of surgical tools
EP3258876B1 (en) 2015-02-20 2023-10-18 Covidien LP Operating room and surgical site awareness
KR20230003408A (ko) 2015-03-17 2023-01-05 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 원격조작 의료 시스템에서 기구의 스크린상 식별을 위한 시스템 및 방법
US10398517B2 (en) * 2016-08-16 2019-09-03 Ethicon Llc Surgical tool positioning based on sensed parameters
EP3518730B1 (en) * 2016-10-03 2024-04-17 Verb Surgical Inc. Immersive three-dimensional display for robotic surgery
US10568701B2 (en) 2016-12-19 2020-02-25 Ethicon Llc Robotic surgical system with virtual control panel for tool actuation
EP3360502A3 (en) * 2017-01-18 2018-10-31 KB Medical SA Robotic navigation of robotic surgical systems
CA3075692A1 (en) * 2017-09-14 2019-03-21 Vicarious Surgical Inc. Virtual reality surgical camera system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016530950A (ja) * 2013-08-23 2016-10-06 エシコン・エンド−サージェリィ・エルエルシーＥｔｈｉｃｏｎ Ｅｎｄｏ−Ｓｕｒｇｅｒｙ， ＬＬＣ 外科用器具アセンブリのためのエラー検出装置
US20170196560A1 (en) * 2013-08-23 2017-07-13 Ethicon Llc End effector detection and firing rate modulation systems for surgical instruments

Also Published As

Publication number Publication date
EP3554411C0 (en) 2024-03-20
US10568701B2 (en) 2020-02-25
CN110099649B (zh) 2022-07-29
EP3554411A1 (en) 2019-10-23
US20200170719A1 (en) 2020-06-04
US20230225607A1 (en) 2023-07-20
US11547494B2 (en) 2023-01-10
CN110099649A (zh) 2019-08-06
US20180168733A1 (en) 2018-06-21
WO2018118410A1 (en) 2018-06-28

Similar Documents

Publication Publication Date Title
US11446098B2 (en) Surgical system with augmented reality display
US11547494B2 (en) Robotic surgical system with virtual control panel for tool actuation
US11213355B2 (en) Hot device indication of video display
US11197731B2 (en) Auxiliary image display and manipulation on a computer display in a medical robotic system
US11484220B2 (en) Surgical equipment control input visualization field
US20240216090A1 (en) Structural adjustment systems and methods for a teleoperational medical system
US10251716B2 (en) Robotic surgical system with selective motion control decoupling
CN114401691A (zh) 用于外科机器人的手持式用户界面设备

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190718

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: A61B 5/00 20060101ALI20230502BHEP

Ipc: A61B 34/00 20160101AFI20230502BHEP

INTG Intention to grant announced

Effective date: 20230519

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTC Intention to grant announced (deleted)
INTG Intention to grant announced

Effective date: 20231004

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017080251

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

U01 Request for unitary effect filed

Effective date: 20240412

U07 Unitary effect registered

Designated state(s): AT BE BG DE DK EE FI FR IT LT LU LV MT NL PT SE SI

Effective date: 20240422