US20150005622A1 - Methods of locating and tracking robotic instruments in robotic surgical systems - Google Patents
Methods of locating and tracking robotic instruments in robotic surgical systems Download PDFInfo
- Publication number
- US20150005622A1 US20150005622A1 US14/314,161 US201414314161A US2015005622A1 US 20150005622 A1 US20150005622 A1 US 20150005622A1 US 201414314161 A US201414314161 A US 201414314161A US 2015005622 A1 US2015005622 A1 US 2015005622A1
- Authority
- US
- United States
- Prior art keywords
- robotic
- tool
- information
- image
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 0 *CC1=CCC=C1 Chemical compound *CC1=CCC=C1 0.000 description 2
Images
Classifications
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/40—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/06—Devices, other than using radiation, for detecting or locating foreign bodies ; determining position of probes within or on the body of the patient
- A61B5/061—Determining position of a probe within the body employing means separate from the probe, e.g. sensing internal probe position employing impedance electrodes on the surface of the body
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00163—Optical arrangements
- A61B1/00193—Optical arrangements adapted for stereoscopic vision
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00163—Optical arrangements
- A61B1/00194—Optical arrangements adapted for three-dimensional imaging
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/313—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for introducing through surgical openings, e.g. laparoscopes
- A61B1/3132—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor for introducing through surgical openings, e.g. laparoscopes for laparoscopy
-
- A61B19/2203—
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B34/37—Master-slave robots
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/107—Measuring physical dimensions, e.g. size of the entire body or parts thereof
- A61B5/1076—Measuring physical dimensions, e.g. size of the entire body or parts thereof for measuring dimensions inside body cavities, e.g. using catheters
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
- B25J13/088—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices with position, velocity or acceleration sensors
- B25J13/089—Determining the position of the robot with reference to its environment
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1679—Programme controls characterised by the tasks executed
- B25J9/1689—Teleoperation
-
- G06K9/00624—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/255—Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B23/00—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes
- G09B23/28—Models for scientific, medical, or mathematical purposes, e.g. full-sized devices for demonstration purposes for medicine
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H30/00—ICT specially adapted for the handling or processing of medical images
- G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/012—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor characterised by internal passages or accessories therefor
- A61B1/018—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor characterised by internal passages or accessories therefor for receiving instruments
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2065—Tracking using image or pattern recognition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/06—Measuring instruments not otherwise provided for
- A61B2090/061—Measuring instruments not otherwise provided for for measuring dimensions, e.g. length
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B2090/364—Correlation of different images or relation of image positions in respect to the body
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/39—Markers, e.g. radio-opaque or breast lesions markers
- A61B2090/3983—Reference marker arrangements for use with image guided surgery
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/361—Image-producing devices, e.g. surgical cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J15/00—Gripping heads and other end effectors
- B25J15/0019—End effectors other than grippers
-
- G06K2209/057—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
- G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10068—Endoscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30241—Trajectory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
- G06V2201/034—Recognition of patterns in medical or anatomical images of medical instruments
Definitions
- the embodiments of the invention relate generally to robots and robotic tools or instruments. More particularly, the embodiments of the invention relate to the acquisition and tracking of the position and orientation of robotic tools or instruments.
- MIS Minimally invasive surgical
- robotic e.g., telerobotic
- An endoscopic camera is typically used to provide images to a surgeon of the surgical cavity so that the surgeon can manipulate robotic surgical tools therein.
- the robotic surgical tool is not in the field of view of the camera or it is otherwise hidden by tissue or other surgical tools, a surgeon may be left guessing how to move the robotic surgical tool when it is obscured from his view.
- tissue or organs of interest in a surgical cavity are often obscured from view.
- a surgeon may have to initially guess the location of an organ of interest within a surgical cavity and search around therein to place the organ and the robotic surgical tools within a field view of the endoscopic camera.
- optical devices such as light emitting diodes
- optical devices can interfere with endoscopic surgical procedures and may not provide sufficiently accurate position and orientation information for a minimally invasive surgical system.
- a magnetic device may be applied to a robotic surgical tool in an attempt to magnetically sense its location.
- robotic surgical tools are often formed of metal and a magnetic device may not work well due to the interference generated by the movement of metal-tools and electrical motors in a minimally invasive surgical system. Moreover, these may provide only a single clue of the position of a robotic surgical tool.
- FIG. 1A is a block diagram of a robotic medical system including a stereo viewer and an image guided surgery (IGS) system with a tool tracking sub-system.
- IGS image guided surgery
- FIG. 1B is a block diagram of a patient side cart including robotic surgical arms to support and move robotic instruments.
- FIG. 2 is a functional block diagram of the video portion of the IGS system to provide a stereo image in both left and right video channels to provide three-dimensional images in a stereo viewer.
- FIG. 3 is a perspective view of a robotic surgical master control console including a stereo viewer and an IGS system with tool tracking sub-system.
- FIG. 4 is a perspective view of the stereo viewer of the robotic surgical master control console.
- FIG. 5A is a perspective view of a sequence of video frames including video images of a robotic medical tool that may be used to perform tool tracking.
- FIG. 5B illustrates different tool positions of a pair of tools in the field of view of a camera based on kenematic information and video image information.
- FIG. 6A is a functional block diagram of a tool tracking architecture and methodology for a robotic system including one or more robotic instruments.
- FIG. 6B is a flow chart of a tool tracking library and its application.
- FIG. 7 is a block diagram illustrating various techniques that may be combined together to meet the challenges in tool tracking.
- FIG. 8 is a functional flow-chart of a tool tracking system.
- FIG. 9A is a figure to illustrate the process of pure image segmentation to localize a tool within an image.
- FIG. 9B is a figure to illustrate the process of sequence matching and/or model-based synthesis to localize a tool within an image.
- FIG. 9C is a more detailed figure to illustrate the process of model-based synthesis to localize a tool within an image.
- FIGS. 10A-10B illustrates elements of a state-space model to adaptively fuse robot kinematics information and vision-based information together.
- FIG. 11A-C illustrate various image matching techniques that may be used separately or collectively to determine pose information of a tool.
- FIG. 12A is a diagram illustrating adaptive fusion under different viewing conditions.
- FIG. 12B is a diagram illustrating a set up for parallel stereo.
- FIGS. 12C-12E are charts illustrating various view geometry statistics that may be used to enhance the performance of the state space model for adaptively fusing information sources together.
- FIGS. 13A-13B are diagrams illustrating sequence matching of a feature in a sequence of one or more images from different sources.
- FIG. 14 is a diagram illustrating appearance learning of objects within an image.
- FIG. 15 is a flow chart of the application of tool tracking to image guided surgery.
- FIG. 16 is a perspective view of overlaying a pre-scanned image of tissue onto a depth map of a surgical site to provide image-guided surgery.
- FIG. 17 is a perspective view of a surgical site with an ultrasound tool capturing ultrasound images for overlay onto a display.
- aspects of the invention include methods, apparatus and integrated systems for tool acquisition (locating) and tool tracking (kinematics-tracking (pose predicting) and full-tracking) of robotic medical tools.
- the method/system for tool tracking systematically and efficiently integrates robot kinematics and visual information to obtain pose (position/orientation) information, which can be used to obtain a more accurate pose of a robotic surgical tool than robot kinematics or visual information alone, in either a camera coordinate system or a base coordinate system.
- pose position/orientation
- Known kinematics transformation can be applied to the pose correction to achieve improved pose in any related coordinate system.
- a camera coordinate system is a coordinate system based on a chosen camera (for example, (X r , Y r , Z r ) in FIG.
- tool tracking explores prior available information, such as the CAD models of tools, and dynamically learns the image appearances of the robotic instruments.
- tool tracking may be markerless so as not to interfere with normal robotic surgical procedures.
- tool tracking may provide continuous pose information of the robotic instruments including their relationships (e.g. tool A is on top of tool B and hence partially occluding tool B) with other tools so that image-based segmentation of the tools may be avoided.
- a block diagram of a robotic surgery system 100 is illustrated to perform minimally invasive robotic surgical procedures using one or more robotic arms 158 .
- Aspects of system 100 include telerobotic and autonomously operating features. These robotic arms often support a robotic instrument.
- a robotic surgical arm e.g., the center robotic surgical arm 158 C
- a stereo or three-dimensional surgical image capture device 101 C such as a stereo endoscope (which may be any of a variety of structures such as a stereo laparoscope, arthroscope, hysteroscope, or the like), or, optionally, some other imaging modality (such as ultrasound, fluoroscopy, magnetic resonance imaging, or the like).
- Robotic surgery may be used to perform a wide variety of surgical procedures, including but not limited to open surgery, neurosurgical procedures (e.g., stereotaxy), endoscopic procedures (e.g., laparoscopy, arthroscopy, thoracoscopy), and the like.
- neurosurgical procedures e.g., stereotaxy
- endoscopic procedures e.g., laparoscopy, arthroscopy, thoracoscopy
- a user or operator O (generally a surgeon) performs a minimally invasive surgical procedure on patient P by manipulating control input devices 160 at a master control console 150 .
- a computer 151 of the console 150 directs movement of robotically controlled endoscopic surgical instruments 101 A- 101 C via control lines 159 , effecting movement of the instruments using a robotic patient-side system 152 (also referred to as a patient-side cart).
- the robotic patient-side system 152 includes one or more robotic arms 158 .
- the robotic patient-side system 152 includes at least three robotic surgical arms 158 A- 158 C (generally referred to as robotic surgical arms 158 ) supported by corresponding positioning set-up arms 156 .
- the central robotic surgical arm 158 C may support an endoscopic camera 101 C.
- the robotic surgical arms 158 A and 158 B to the left and right of center may support robotic instruments 101 A and 101 B, respectively, that may manipulate tissue.
- Robotic instruments are generally referred to herein by the reference number 101 .
- Robotic instruments 101 may be any instrument or tool that couples to a robotic arm that can be manipulated thereby and can report back kinematics information to the robotic system.
- Robotic instruments include, but are not limited to, surgical tools, medical tools, bio-medical tools, and diagnostic instruments (ultrasound, computer tomography (CT) scanner, magnetic resonance imager (MRI)).
- CT computer tomography
- MRI magnetic resonance imager
- the robotic patient-side system 152 includes a positioning portion and a driven portion.
- the positioning portion of the robotic patient-side system 152 remains in a fixed configuration during surgery while manipulating tissue.
- the driven portion of the robotic patient-side system 152 is actively articulated under the direction of the operator O generating control signals at the surgeon's console 150 during surgery.
- the driven portion of the robotic patient-side system 152 may include, but is not limited or restricted to robotic surgical arms 158 A- 158 C.
- the instruments 101 , the robotic surgical arms 158 A- 158 C, and the set up joints 156 , 157 may include one or more displacement transducers, positional sensors, and/or orientational sensors 185 , 186 to assist in acquisition and tracking of robotic instruments. From instrument tip to ground (or world coordinate) of the robotic system, the kinematics information generated by the transducers and the sensors in the robotic patient-side system 152 is reported back to the robotic system and a tool tracking and image guided surgery (IGS) system 351 .
- IGS tool tracking and image guided surgery
- the positioning portion of the robotic patient-side system 152 that is in a fixed configuration during surgery may include, but is not limited or restricted to set-up arms 156 .
- Each set-up arm 156 may include a plurality of links and a plurality of joints.
- Each set-up arm may mount via a first set-up-joint 157 to the patient side system 152 .
- An assistant A may assist in pre-positioning of the robotic patient-side system 152 relative to patient P as well as swapping tools or instruments 101 for alternative tool structures, and the like, while viewing the internal surgical site via an external display 154 .
- the external display 154 or another external display 154 may be positioned or located elsewhere so that images of the surgical site may be displayed to students or other interested persons during a surgery. Images with additional information may be overlaid onto the images of the surgical site by the robotic surgical system for display on the external display 154 .
- the robotic patient-side system 152 comprises a cart column 170 supported by a base 172 .
- One or more robotic surgical arms 158 are respectively attached to one or more set-up arms 156 that are a part of the positioning portion of robotic patient-side system 152 .
- the cart column 170 includes a protective cover 180 that protects components of a counterbalance subsystem and a braking subsystem (described below) from contaminants.
- each robotic surgical arm 158 is used to control robotic instruments 101 A- 101 C. Moreover, each robotic surgical arm 158 is coupled to a set-up arm 156 that is in turn coupled to a carriage housing 190 in one embodiment of the invention, as described below with reference to FIG. 3 . The one or more robotic surgical arms 158 are each supported by their respective set-up arm 156 , as is illustrated in FIG. 1B .
- the robotic surgical arms 158 A- 158 D may each include one or more displacement transducers, orientational sensors, and/or positional sensors 185 to generate raw uncorrected kinematics data, kinematics datum, and/or kinematics information to assist in acquisition and tracking of robotic instruments.
- the robotic instruments may also include a displacement transducer, a positional sensor, and/or orientation sensor 186 in some embodiments of the invention.
- one or more robotic instruments may include a marker 189 to assist in acquisition and tracking of robotic instruments.
- the stereo endoscopic camera 101 C includes an endoscope 202 for insertion into a patient, a camera head 204 , a left image forming device (e.g., a charge coupled device (CCD)) 206 L, aright image forming device 206 R, a left camera control unit (CCU) 208 L, and a right camera control unit (CCU) 208 R coupled together as shown.
- the stereo endoscopic camera 101 C generates a left video channel 220 L and a right video channel 220 R of frames of images of the surgical site coupled to a stereo display device 164 through a video board 218 .
- a lock reference signal is coupled between the left and right camera control units 208 L, 208 R.
- the right camera control unit generates the lock signal that is coupled to the left camera control unit to synchronize the left view channel to the right video channel.
- the left camera control unit 208 L may also generates the lock reference signal so that the right video channel synchronizes to the left video channel.
- the stereo display 164 includes a left monitor 230 L and a right monitor 230 R.
- the viewfinders or monitors 230 L, 230 R may be provided by a left display device 402 L and a right display device 402 R, respectively.
- the stereo images may be provided in color by a pair of color display devices 402 L, 402 R.
- Stereo images of a surgical site may be captured by other types of endoscopic devices and cameras with different structures. For example, a single optical channel may be used with a pair of spatially offset sensors to capture stereo images of the surgical site.
- the master control console 150 of the robotic surgical system 100 may include the computer 151 , a binocular or stereo viewer 312 , an arm support 314 , a pair of control input wrists and control input arms in a workspace 316 , foot pedals 318 (including foot pedals 318 A- 318 B), and a viewing sensor 320 .
- the master control console 150 may further include a tool tracking and image guided surgery system 351 coupled to the computer 151 for providing the tool images and tissue images overlaid on the visible surgical site images.
- the tool tracking and image guided surgery system 351 may be located elsewhere in the robotic surgical system 100 , such as the patient side cart 152 or a separate computer system.
- the stereo viewer 312 has two displays where stereo three-dimensional images of the surgical site may be viewed to perform minimally invasive surgery.
- the operator O typically sits in a chair, moves his or her head into alignment with the stereo viewer 312 to view the three-dimensional annotated images of the surgical site.
- the master control console 150 may include the viewing sensor 320 disposed adjacent the binocular display 312 .
- the system operator aligns his or her eyes with the binocular eye pieces of the display 312 to view a stereoscopic image of the surgical worksite, the operator's head sets off the viewing sensor 320 to enable the control of the robotic instruments 101 .
- the viewing sensor 320 can disable or stop generating new control signals in response to movements of the touch sensitive handles in order to hold the state of the robotic instruments.
- the processing required for tool tracking and image guided surgery may be entirely performed using computer 151 given a sufficiently capable computing platform.
- the arm support 314 can be used to rest the elbows or forearms of the operator O (typically a surgeon) while gripping touch sensitive handles of the control input wrists, one in each hand, in the workspace 316 to generate control signals.
- the touch sensitive handles are positioned in the workspace 316 disposed beyond the arm support 314 and below the viewer 312 . This allows the touch sensitive handles to be moved easily in the control space 316 in both position and orientation to generate control signals.
- the operator O can use his feet to control the foot-pedals 318 to change the configuration of the surgical system and generate additional control signals to control the robotic instruments 101 as well as the endoscopic camera.
- the computer 151 may include one or more microprocessors 302 to execute instructions and a storage device 304 to store software with executable instructions that may be used to generate control signals to control the robotic surgical system 100 .
- the computer 151 with its microprocessors 302 interprets movements and actuation of the touch sensitive handles (and other inputs from the operator O or other personnel) to generate control signals to control the robotic surgical instruments 101 in the surgical worksite.
- the computer 151 and the stereo viewer 312 map the surgical worksite into the controller workspace 316 so it feels and appears to the operator that the touch sensitive handles are working over the surgical worksite.
- the computer 151 may couple to the tool tracking and image guided surgery system 351 to execute software and perform computations for the elements of the image guided surgery unit.
- the tool tracking system described herein may be considered as operating in an open-loop fashion if the surgeon operating the master console is not considered part of the system. If the robotic instrument is to be automatically controlled with the tool tracking system, such as in visual serving systems used to control the pose of a robot's end-effector using visual information extracted from images, the tool tracking system may be considered to be operating in a closed visual-feedback loop.
- the viewer 312 includes stereo images for each eye including a left image 400 L and a right image 400 R of the surgical site including any robotic instruments 400 respectively in a left viewfinder 401 L and a right viewfinder 401 R.
- the images 400 L and 400 R in the viewfinders may be provided by a left display device 402 L and a right display device 402 R, respectively.
- the display devices 402 L, 402 R may optionally be pairs of cathode ray tube (CRT) monitors, liquid crystal displays (LCDs), or other type of image display devices (e.g., plasma, digital light projection, etc.).
- the images are provided in color by a pair of color display devices 402 L, 402 R; such as color CRTs or color LCDs.
- three dimensional maps (a depth map with respect to a camera coordinate system or equivalently a surface map of an object with respect to its local coordinate system is a plurality of three-dimensional points to illustrate a surface in three dimensions) of the anatomy, derived from alternative imaging modalities (e.g. CT scan, XRAY, or MRI), may also be provided to a surgeon by overlaying them onto the video images of the surgical site.
- alternative imaging modalities e.g. CT scan, XRAY, or MRI
- a right image 410 R rendered from a three dimensional map such as from a CT scan may be merged onto or overlaid on the right image 400 R being displayed by the display device 402 R.
- a rendered left image 410 L is merged into or overlaid on the left image 400 L of the surgical site provided by the display device 402 L.
- a stereo image may be displayed to map out organ location or tissue location information in the surgical site to the operator O in the control of the robotic instruments in the surgical site, augmenting the operator's view of the surgical site with information that may not be directly available or visible by an endoscopic camera in the surgical site.
- a stereo video endoscopic camera 101 C has been shown and described, a mono video endoscopic camera generating a single video channel of frames of images of the surgical site may also be used in a number of embodiments of the invention. Rendered images can also be overlaid onto the frames of images of the single video channel.
- Tool tracking has a number of applications in robotic surgical systems.
- One illustrative application of tool tracking is to automatically control the motion of the endoscopic camera so that a surgeon automatically views regions of interest in a surgical site, thus freeing the surgeon from the camera control task.
- robotic instruments are tracked so that the endoscopic camera is centered in the field of view of the surgical site.
- Another illustrative application for accurate tool tracking may be used to move a robotic instrument to reach a surgical target (e.g., a tumor) either automatically or by a surgeon.
- a surgical target e.g., a tumor
- other real-time imaging modalities such as ultra-sound or pre-scanned images, may be used with real-time tool tracking to move a robotic instrument to the tumor and remove it.
- GUI graphic user interface
- a tool tracking system and architecture that fully integrates kinematics information and visual information for robust and accurate tool tracking performance. Results of tool localization and tracking are made accurate and reliable by adaptively combining together robust kinematics information and accurate geometric information derived from video.
- the tool tracking system performs locating (determining absolute locations/poses with stereo video), tracking (integrating visual and kinematics) and predicting (kinematics while the tool or a portion thereof is not visible) functions.
- An analysis-by-synthesis capability makes it possible to explore the prior information that is of concern, such as information about the tools and not the tissue and surrounding environment.
- the basic procedure in analysis-by-synthesis is to synthesize an image based on the model (geometry and texture) and the current pose (position/location and orientation) of a tool and then compare it against real images.
- the error between the real and synthesized images is the driving force for better estimation of tool pose.
- appearance-based learning may be applied to update the model for a specific tool.
- matching may be performed using features, such as edges and/or corners, that are more robust to lighting variation.
- stereo imaging may be used along with the analysis-by-synthesis (ABS) techniques.
- ABS analysis-by-synthesis
- a stereo approach may be provided based on the tool (or some of its parts).
- stereo and ABS techniques may be applied to a sequence of images (e.g., the same location and different orientations or different locations and different orientation). Sequence-based matching makes the procedure less vulnerable to local minimum in the process of optimization/estimation.
- a sequential Bayesian approach may be applied to fuse visual information and robot kinematics to efficiently track tools.
- the states provide zero-order kinematics of the tools (e.g. current position and orientation, or pose), and the first-order kinematics of the tools (e.g. translational and angular velocity).
- higher-order or lower-order state space model may be adopted.
- a linear Kalman filtering may be applied when the noise can be approximated as Gaussian.
- an extended Kalman filtering may be used to filter out noise from observations and state dynamics.
- sequential Monte Carlo method e.g., particle filtering
- the point distribution function may be represented by a set of representative samples.
- the sources should be correctly characterized, for example, in terms of observation noise characteristics.
- observation noises e.g., pixel location of a feature point
- observation e.g., pixel location of a feature point
- a robotic instrument that is under a slanted viewing angle or is occluded by other tools has a large variance.
- tool tracking involves determining a pose of a robotic instrument 101 including its position or location (Xt,Yt,Zt) and its orientation in the camera coordinate system as it moves in, around, and out of the surgical site.
- a full pose description may not only include the location and orientation of a robotic instrument in a three dimensional space but may further include the pose of an end-effector, if any.
- Positional information or pose as used herein may be used to refer to one or both the location and orientation of a robotic instrument.
- a sequence of left video frames 500 L within a camera coordinate system and a sequence of right video frames 500 R within another camera coordinate system may be used for tool tracking in one embodiment of the invention.
- a single view with a single sequence of video frames may be used for tool tracking in another embodiment of the invention.
- a single video frame may be used for tool tracking in yet another embodiment of the invention providing partially corrected pose estimates.
- a marker 502 on the robotic instrument 101 may be used to assist in the tool tracking if visible or otherwise sensible.
- the marker 502 is a painted marker minimally altering the robotic instruments.
- markerless tool tracking is provided with no modification of the robotic instruments.
- natural image features of a robotic tool may be detected as natural markers and/or image appearance of the tools and the CAD model of tools may be used to provide tool tracking.
- video information from an endoscopic camera and kinematics of the robotic arm and robotic instrument are used as cues to determine the pose of the robotic instrument in the surgical site. If the robotic instrument is not visible to the camera, the video information alone is insufficient to determine position but the kinematics adds the missing information to determine robotic instrument pose. Moreover, even if the video information is available, the addition of the kinematics information makes the computation of pose more robust.
- a tool tracking system is provided based on robot kinematics and vision that requires little to no modifications to the robotic instruments 101 and the pre-existing surgical system 100 .
- Kinematics information provided by the surgical system 100 may include kinematic position k t P , kinematic orientation k t ⁇ , kinematic linear velocity ⁇ dot over (k) ⁇ t P , and kinematic angular velocity ⁇ dot over (k) ⁇ t ⁇ of one or more robotic instruments 101 .
- the kinematics information may be the result of movement of the robotic surgical arm 158 , the robotic instrument 101 , or both the robotic surgical arm 158 and robotic instrument 101 at a given time.
- the kinematics information provided by the surgical system 100 may also include the kinematic position k t P , kinematic orientation k t ⁇ , kinematic linear velocity ⁇ dot over (k) ⁇ t P , and kinematic angular velocity ⁇ dot over (k) ⁇ t ⁇ of the endoscopic camera to provide a frame of reference.
- FIG. 6A a functional block diagram of a tool tracking architecture and methodology for a surgical system is illustrated in accordance with embodiments of the invention.
- the main operational stages of tool tracking are illustrated in the middle column of FIG. 6A .
- Key technical capabilities associated with tool tracking are illustrated in the left column of FIG. 6A but for operational constraints 601 .
- the end results of the tool tracking methodology are illustrated in the right column of FIG. 6A .
- the key technical components of the methodology and architecture may be further categorized into basic building blocks and supporting blocks.
- the basic building blocks including image matching 609 and a state-space model 613 that are used to provide efficient tool tracking, each of which are responsive to visual information.
- the supporting blocks include model-based synthesis 611 , adaptive fusion 615 , and sequence matching 607 to support the implementation of robust and accurate tool tracking.
- Adaptive fusion 615 fully explores prior information that may be available, including prior kinematics information and prior visual information.
- vision information and kinematics information are typically available with known characteristics.
- Robot kinematics information is usually stable and often accurate but may drift during long periods of time.
- Vision-based information is very accurate when it can be reliably estimated. Otherwise vision-based information may be very inaccurate.
- adaptive fusion is used to obtain accurate information fusion from different sources of information as well as similar sources of information.
- the vision-based information is known to be accurate then the information fusion is heavily biased towards the vision-based information.
- robot kinematics information is used over the vision-based information to generate a more robust fusion of information. While the quality of robot kinematics is typically uniform, the quality of vision information in terms of image matching and 3D post estimation varies a lot. View geometry statistics may be used to determine the reliability and accuracy of video-based information.
- Adaptive fusion may also be used to obtain accurate information fusion from similar sources of information.
- Model-based synthesis is used herein to generally refer to generation or rendering of a template image for use in subsequent matching operations, and includes full synthesis, geometry only synthesis, and implicit synthesis.
- Full synthesis is a complete synthesis of an image of the robotic instrument.
- robotic instrument images are generated from a computer aided design (CAD) model based on its geometry and texture.
- CAD computer aided design
- Other prior information (the location/orientation of the model), not necessarily accurate, is presented along with the synthesized robotic instrument images for image matching 609 .
- Geometry-only synthesis is the case where the geometry of the robotic instrument is used to synthesize geometry-only images (e.g., edge images). Texture of the model is not used in geometry-only synthesis.
- Implicit synthesis is the case where images are not actually synthesized. Instead the model (either geometry or texture or both) is implicitly used to perform image matching.
- the geometric properties e.g., width, length, shape
- the geometric relationship among them e.g., markers forming a line
- sequence matching is where objects or features in a sequence of images captured from one camera view are matched against objects or features in a sequence of images captured from a different camera view. In another embodiment of the invention, sequence matching is where objects or features in a sequence of images from a camera view are matched against objects or features in a sequence of synthesized images.
- Tool acquisition may also be referred to as localization herein.
- the goal of the tool acquisition stage 604 is to obtain the absolute pose information (location and orientation) of the one or more robotic instruments within the field of view of one or more cameras, such as the stereo endoscopic camera 101 C.
- the tool acquisition stage 604 performs a locating function 614 resulting in the location and orientation of the tool.
- the goal of the tool tracking stage 606 is to dynamically update the absolute pose (location and orientation) of a moving robotic instrument.
- the tool tracking stage 606 may perform a full-tracking function 616 or a kinematics-tracking (pose prediction) function 618 respectively resulting in either a full-tracking state when both visual information and robot kinematics information are available or a kinematics-tracking state when visual information is not utilized (e.g., tool outside the field of view or occluded).
- the mode/stage of the tool tracking system changes from tool acquisition to tool tracking after the tool is initially located within the field of view.
- the tool tracking system may remain in the tool tracking mode/stage.
- the tool tracking system may change from a tool tracking mode/stage into a tool acquisition mode/stage if the tool is removed from the field of view and then returns into the field of view.
- the tool tracking system may optionally begin operation with an initialization procedure if there is only a single tool in the field of view. If additional tools are to be tracked, the optional initialization procedure may be skipped as other tools have been located and tracked. If the tools have no markers, the optional initialization procedure may involve tools moving around in order to obtain a robust localization via sequence matching.
- FIG. 6A starts at block 600 and goes to block 602 .
- an optional initialization of the tool tracking system may occur.
- Mono or stereo video 603 may be used in the tool tracking system and is initialized to begin generation of digital video frames of image data of the surgical site.
- Kinematics information 605 may also be used in the tool tracking system during initialization to form an initial pose of the robotic instrument.
- the kinematics information 605 may include positional information, including angular or linear information for example, from sensors located at various places along a robotic arm and the robotic instrument.
- the kinematics information 605 may be for both the endoscopic camera and robotic instruments such that the relationship between positional information for the robotic instruments and the camera may be determined.
- Initialization begins with a single tool in the field of view without any occlusions.
- the system may be initialized for additional robotic instruments in the field of view. If tools have already been located and tracked and a new tool is being added, the new tool can be initialized by placing it into the field of view with the previously located and tracked tools. If no tool has been located and tracked, each tool may be initialized by placing it within the field of view with all other tools outside the field of view.
- the robotic instrument being initialized may be placed in the center of the surgical site for optimal estimation across the whole space or as close to stereo endoscopic camera 101 C as possible that will allow for accurate stereo computation. With a markerless system, the robotic instrument may be moved and rotated for reliable sequence matching.
- FIG. 9B graphically illustrates the tool acquisition stage in a surgical site.
- Stereo video images 500 L, 500 R of the surgical site are captured by the endoscopic camera 101 C, including one or more robotic instruments 101 in the surgical site.
- Stereo video may be used to obtain an absolute initial pose of the one or more robotic instruments 101 in one embodiment of the invention.
- mono video may be used with kinematics information to estimate absolute initial pose (position and orientation) of the one or more robotic instruments 101 .
- the one or more robotic instruments 101 may include painted markers 502 to assist in tool acquisition and tool tracking in the surgical site.
- the tool acquisition stage performs a locating function 614 resulting in the initial pose of the one or more robotic instruments 101 in the surgical site.
- the methodology goes to block 606 .
- the tool tracking system enters a tool tracking mode or stage in the surgical site.
- the goal of tool tracking is to update the absolute pose information (location and orientation) based on incremental and/or partial information (visual and robot kinematics).
- the tool tracking system is at a full-tracking state 616 when visual and kinematics information is available. If a robotic instrument is not visible (e.g., tools inside an organ or occluded by other tools) in the surgical site, the tool tracking system is at a kinematics-tracking state 618 for estimating tool pose.
- the tool tracking system may transition from tool tracking 606 and return to tool acquisition 604 if a tracked tool gets out of field of view and then comes back into the field of view of the camera.
- a tool tracking application 650 is executed by a system 351 of the robotic surgical system 100 .
- the video board 218 illustrated in FIG. 2 may be a part of the IGS system 351 in order to receive the video images from the endoscopic camera over the surgical site.
- a kinematics application programming interface (API) 660 provides a software interface to receive the raw kinematics data from the surgical system 100 .
- the kinematics API 660 couples the kinematics information to the tool tracking application 650 and a tool tracking library 652 .
- the raw kinematics data 680 is received by an API streamer thread 658 which provides the physical interface to a communication channel (for example, fiber optic cable or Ethernet, and may buffer the raw kinematics data by storing it into a memory, a hard disk, or other data storage device).
- the tool tracking library 652 may issue data requests to the API streamer thread 658 .
- a video capture thread 656 is coupled to the endoscopic camera to receive the raw endoscopic video feed 670 .
- the raw video 670 may be mono video of a single channel or stereo video of left and right channels.
- the video capture thread 656 may buffer the raw video data by storing it into a frame buffer memory, a hard disk, or other data storage device.
- a video application programming interface (API) 659 provides the software interface to receive the raw video data from the surgical system into the tool tracking system.
- the tool tracking library 652 may issue data requests to the video capture thread 656 .
- the tool tracking library 652 contains the core functionality of combining kinematics (through kinematics API 660 ) and video (through video API 659 ) for accurate tracking of tools.
- the library also provides application program interface so it can be invoked in a certain way by a customer-designed tool tracking application 650
- the tool tracking library 652 In response to the video data and the kinematics data, the tool tracking library 652 generates corrected kinematics data for the pose of a robotic instrument.
- the raw kinematics data is corrected for orientation and position of the tools.
- the corrected kinematics data may be used in a number of applications, such as image guided surgery.
- the speed of raw kinematics 680 may be 100 to 200 Hertz (Hz) and the speed of raw video 670 may be 30 Hz to 60 hz and the speed of tool tracking maybe even slower.
- the speed of the corrected kinematics 690 should be substantially similar to the speed of the raw kinematics 680 for medical applications.
- the raw kinematics may be passed through.
- a correction matrix (rotation and translation) may then be used to correct the raw kinematics information from the tool tracking library.
- the corrected kinematics 690 may be directly output from the tool tracking library 652 where a correction matrix is applied to the raw kinematics. Either way is feasible because the correction matrix corrects the bias in the raw kinematics and the bias changes slowly, for example, slower than 1 Hz.
- FIG. 6A illustrated previously, illustrates a functional block diagram including operational stages of a tool tracking system.
- FIG. 7 is a block diagram illustrating the challenges of performing tool tracking.
- FIG. 8 graphically illustrates a functional block diagram of a tool tracking system 800 .
- the tool tracking system 800 adaptively fuses visual information and robot kinematics in order to achieve robust, accurate and efficient tool tracking.
- the unknown full pose of a robotic instrument, at a time instant t, is represented as a state s t 805 B in a Bayesian state-space model 802 .
- the state-space model 802 may use a plurality of posed states 805 A- 805 C to perform tool tracking in the surgical site.
- the state-space model 802 may generate the corrected kinematics information 690 of the robotic instrument.
- a CAD tool model 804 (geometry only or both geometry and texture) is used for synthesizing (explicitly or implicitly) an image under a given pose (i.e. state).
- the relative robot kinematics ⁇ dot over (k) ⁇ t 605 (where the dot above the k being used to represent that the relative or first-derivative measurements of the kinematics information) between time instances t to t+1 can be coupled into the state-space model 802 .
- Visual information 603 from captured images may be amplified and analyzed by an amplifier/filter 808 to control the influence of visual feedback 809 on the fusion of visual information and kinematics information.
- the amplifier/filter 808 generally implements how view geometry statistics are applied for adaptive fusion. If stereo images 803 are available, the spatial constraints 807 between left and right image pairs may also be explicitly or implicitly explored to assist in tool tracking.
- the natural challenges are those imposed by realistic operational scenarios.
- the technical challenges are those caused by proposed tool tracking algorithms when facing natural challenges.
- the natural challenges for example include cluttering and occlusion 702 , illumination variation and image appearance change 703 , and viewing singularity 704 .
- the technical challenges include image segmentation 710 and matching ambiguity 712 , for example.
- the natural challenge of illumination and image appearance 703 is where the same scene changes dramatically along with the motion of directional light sources.
- the image intensity of the same object can be different, depending on the distance of the object from the lighting source and the angle between the lighting source and local surface normal. This makes image-based processing less reliable.
- specularities from organs, blood that under directional endo-illumination make image processing more challenging.
- the natural challenge of viewing singularity 704 may occur when three dimensional geometry information is derived from two dimensional images.
- Three dimensional geometry information derived from two dimensional images is not reliable when the two dimensional projection of a three dimensional object is degenerated. For example, a three dimensional cylindrical tool is projected onto an image plane as a circle.
- the natural challenge of scene cluttering and occlusion 702 is the case where there could be more than one robotic instrument in the field of view. Additionally, the robotic instruments may be partially or fully submerged with complex and dynamic background of organ tissues, blood and smoke caused by electro-dissection.
- the technical challenges include image segmentation 710 and matching ambiguity 712 .
- a big technical challenge for tool tracking may be reliability and accuracy under realistic situations.
- model based synthesis techniques 722 may be used.
- model based synthesis 722 a CAD model of a robotic instrument may be used to render a clean tool image as a pattern to match against or search within a limited region constrained by the pose information of tool. As a result, pure image segmentation from the real images is avoided. Because the states of all robotic instruments are tracked, mutual occlusions of all these robotic instruments can be calculated thereby making image matching more reliable.
- many areas in an image look alike and non-corresponding areas of two images may appear to be more alike than two corresponding areas (for example, due to illumination variations), making region-based matching ambiguous.
- sequence matching 728 may be applied where a sequence of images will be matched against another sequence of images. Such a method is useful when we use robust and accurate relative kinematics information ⁇ dot over (k) ⁇ t .
- sequence matching 728 for the same three-state sequence, the situation changes significantly.
- the perfect or error-less relative kinematics information is a two-sequence [ ⁇ dot over (k) ⁇ t ⁇ 1 , ⁇ dot over (k) ⁇ t ] of kinematics information.
- the respective sequence of three images 811 A- 811 C [I t ⁇ 1 , I t , I t+1 ] and the respective sequence of three synthesized images [I S t ⁇ 1 , I S y , I S t+1 ] may be used to determine the unknown states. That is, if we know any one of the three states in the three-state sequence [s t ⁇ 1 , s t , s t+1 ], we can obtain other missing states through perfect or error-less relative kinematics.
- sequence matching 728 can provide a more robust and more reliable matching as the number of unknowns is reduced and the same number of observations (real images) are kept.
- appearance learning techniques 724 may be used to handle image or appearance changes 703 such as from illumination variations and natural wear of a tool, for example.
- appearance learning techniques handle appearance changes by training the tool tracking system on image samples of the same tool under different viewing conditions.
- Appearance learning techniques have been used extensively in object tracking to handle appearance change due to illumination variations. For example, parametric models have been built to handle illumination variations. Appearance learning techniques are further illustrated herein with reference to FIG. 14 with the use of face images instead of tool images.
- adaptive fusion techniques 726 may be used to handle the challenges of singularity of viewing geometry or viewing singularity 704 .
- the technique of adaptive fusion is used to explore the available pose information, i.e., predicted state (before correction) when feeding geometric information derived from video into the Bayesian state-space model 802 . More specifically, video-derived information has much less weight when fused with robot kinematics information under such conditions. In a Bayesian state-space model 802 , this manifests itself as large noise variance in the observation equation.
- Adaptive fusion may be used to handle the challenges of singularity of viewing geometry in order to provide robust and accurate kinematics information of the tools in a surgical, medical, or other type of robotic system.
- Pure image segmentation may be used by a tool tracking algorithm to localize tools. Pure image segmentation of tools from a two dimensional image is straightforward if the tools have distinctive features, such as color marks that may be used to identify a tool. However, operational conditions may make pure image segmentation techniques difficult if not impossible to perform.
- an image has a tool 901 hidden by an occlusion 902 .
- the occlusion 902 is so severe that it breaks key steps (e.g., color- and/or shape-based analysis) of pure image segmentation of the image such that the tool 901 cannot be found therein.
- the tool shape 901 S is substantially covered over by the occlusion shape 902 S in the image illustrated in FIG. 9A .
- an occlusion can only make it more difficult for pure image segmentation techniques to localize a tool.
- the image of the tool 901 is again hidden by an occlusion 902 .
- Techniques of sequence matching and/or model-based-synthesis matching may be used to localize the robotic instruments instead of pure image segmentation. Sequence matching was briefly discussed previously. Model based synthesis uses a priori knowledge regarding kinematics and appearance that may be available for the tools that are being tracked.
- a CAD model 904 A of the tool 901 is used to synthesize an image of the tool given the known or hypothesized pose information for the tool.
- the pose information for the tool may be determined from kinematics information or otherwise and a posed synthesized image 904 B may then be generated.
- the posed synthesized image 904 B of the tool 901 may then be used to perform image matching or an image search within the overall image of a surgical site to find the location of the tool 901 therein even though it may be partially occluded.
- This technique of tool localization may generally be referred to herein as an analysis-by-synthesis approach.
- image segmentation may be guided by exploring the available prior kinematics and image information. That is, image segmentation may be constrained to be performed within a limited region of the image of the surgical site based on rough pose information of the tool in response to the prior robot kinematics and the CAD model 904 A of the tool 901 .
- This technique of tool localization may generally be referred to herein as aided image segmentation in contrast to pure image segmentation that has no constraints.
- image synthesis also referred to herein as model-based synthesis
- image analysis/search are key steps in using analysis-by-synthesis methods for tool localization and tool tracking.
- the image synthesis 911 and image analysis/search 915 processes may be repeatedly performed in an iterative optimization approach to find the best tool pose parameter in response to a given cost function CF 913 .
- an initial pose hypothesis may be formulated to generate the initial synthesized model tool for computation of an initial cost function.
- the cost function CF 913 is a function of what corresponding features 912 , 914 are used for matching and how an image is synthesized during image synthesis 911 .
- a synthesize edge image 904 S of the tool may be synthesized during image synthesis 911 in response to the CAD geometry of the CAD model 904 A of the tool.
- a synthesized regular image 904 B of the tool may be synthesized during image synthesis 911 in response to the CAD geometry and CAD texture of the CAD model 904 A of the tool.
- the synthesize edge image 904 S of the tool may be used to perform image matching with edges in a video image 910 A of the tool.
- the synthesized regular image 904 B of the tool may be used to perform image matching with a regular video image 910 B of the tool.
- appearance learning may be used to augment the analysis-by-synthesis process for tool localization and tool tracking.
- an edge image such as illustrate in video image 910 A, is typically robust against lighting variations.
- the model will be decomposed into triangles, the 3D vertex coordinates of which will be described in a coordinate system attached to the model.
- the model coordinates will first be transformed to a world coordinate system, before being projected to a 2D display coordinate system by applying the camera model. Once in the 2D display coordinate system, each triangle will be rasterized.
- the synthesis of the final per-pixel color values may be computed via interpolation of color specified on a per-vertex basis, texture mapping and filtering, and the application of lighting models. (Reference: Computer Graphics: Principals and Practice, by James D. Foley, Andries van Dam , et. al, Addison-Wesley Professional; 2 edition, Aug. 4, 1995, ISBN: 978-0201848403).
- the function L is a mapping function that maps the model texture M t into the real image/appearance I s [x] because the real image varies with lighting conditions and other factors, such as occlusions.
- Camera coordinates of a 3D point 931 P on the tool that maps to x may be represented by [X,Y,Z] T .
- a local coordinate of the 3D point 931 P on the tool that is internal to the tool model may be represented as [X M ,Y M ,Z M ] T .
- a transformation T ⁇ P, ⁇ of the local coordinate of the 3D point 931 P on the tool to the camera coordinate of the tool as a function of the tool pose may be written as
- T ⁇ P, ⁇ is a four by four 3D-to-3D rigid transformation matrix that can be further decomposed into translational and rotational parts.
- T ⁇ P , ⁇ ⁇ * arg ⁇ ⁇ min T ⁇ P , ⁇ ⁇ ⁇ C ⁇ ( I s , I ) , ( Eq . ⁇ 3 )
- the synthesized tool image I s may be repeatedly generated in an iterative manner using updated pose information so that the cost function C of comparing the synthesized tool image I s with the video images of the surgical site are minimized and the tool pose is optimized.
- SSD sum of squared differences
- an SSD is a simple cost function, it is nonlinear (e.g., higher than quadratic) in terms of the pose parameters due to the camera geometric projection ⁇ that is nonlinear and the mapping function L to map model texture to real image/appearance that varies with lighting conditions that may be non-linear.
- Minimizing a nonlinear cost function C is a complex optimization problem.
- Different strategies can be applied to solve a problem of minimizing a nonlinear cost function C.
- random optimization methods may be used to solve and minimize a nonlinear cost function C problem in order to avoid an exhaustive parameter search.
- a quadratic approximation of the cost function may be use to iteratively solve the nonlinear problem.
- the complex optimization problem may be broken up into two different steps to more efficiently minimize the cost function C.
- the first step entails performing an image matching where the raw image pixels or extracted image features of I are used for matching against those of the respective synthesized tool image I s .
- the second step involves performing a geometry-only optimization in response to the result of the image matching between the raw image pixels or extracted image features and corresponding ones from the respective synthesized tool image I s .
- these two steps to solve the optimization problem of Eq. 3 may be formulated into the following two equations:
- T ⁇ P , ⁇ ⁇ * arg ⁇ ⁇ min T ⁇ P , ⁇ ⁇ ⁇ ⁇ ⁇ ( x m - f ⁇ ( X m ) ) 2 . ( Eq . ⁇ 5 )
- Eq. 4 represent the step of finding the corresponding 2D feature points x m from 1 and 3D points X m on the tool via image matching of I s and I.
- Eq. 5 represents the geometry-only optimization where optimal 3D-2D mapping T ⁇ P, ⁇ can be found given the matched 2D-3D pairs.
- the function f( ) is a nonlinear function in the following form ⁇ ( ⁇ T ⁇ P, ⁇ ).
- the image matching and analysis-by-synthesis processes may be incorporated into a sequential framework for fusion of vision and kinematics information to obtain more accurate positional information of a tool than would otherwise be available from each alone.
- appearance learning techniques may be used to handle image and appearance changes, such as from illumination variations and natural wear of a tool, for example.
- the appearance variations due to changes in illumination may exhibit illumination subspace/cone phenomena or spherical harmonics for example.
- Appearance learning techniques generally train the tool tracking system on image samples of the same tool under different viewing conditions. Pose specific learning techniques may be used as well as clustering or manifold learning techniques may be used to train the tool tracking system over a large number of samples.
- basis images for illumination variations 1401 A, 1401 B, 1401 C may be used to train the tool tracking system to generate one or more synthesized images 1402 A- 1402 B which are more closely matched to the respective real images 1045 A- 1405 B that may be captured under different lighting conditions.
- Appearance learning techniques have been used extensively in object tracking to handle appearance change due to illumination variations (reference: G. Hager and P. Belhumeur, “Efficient Region Tracking with Parametric Models of Geometry and Illumination,” IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 20, pp. 1025-1039, 1998).
- parametric models have been built to handle illumination variations (reference: H. Murase, S. Nayar, “Learning and Recognition of 3-D Objects from Brightness Images,” Proc. AAAI Fall Symposium, Machine Learning in Computer Vision, pp. 25-29, 1993.
- the next step to obtain more accurate positional information is the fusion of image positional information and kinematics positional information of the robotic instruments.
- the purpose of information fusion is to provide more robust and/or more accurate positional information for the robotic instruments in the surgical site such that tool tracking information may be applied in various ways to obtain accurate results, e.g., measurements of certain physical entities within a surgical site.
- Key to successfully fusing information together from similar sources or from different sources is determining how to adjust the contributions of each to the fusion. The contribution of sources to the information fusion may be adjusted in different ways, such as by a winner-take-all or a weighted averaging method, for example.
- Both raw robotic kinematics information 1010 of the robotic instrument and vision-based information 1011 can be used to generate the state variables 1000 A- 1000 D.
- a tool model 804 may be used to synthesize 611 the synthesized images 810 in response to the state variables 1000 A- 1000 D.
- An image analysis 806 , 609 is performed comparing the synthesized images 810 of the robotic instrument with the observed images 1011 .
- Some real-world data analysis tasks involve estimating unknown quantities from given observations. Moreover, a priori knowledge about phenomenon of a number of applications may be available to allow us to formulate Bayesian models involving probability and statistics.
- the unknown quantities of information fusion at corresponding time instances can be defined as state variables 1000 A- 1000 D (of the system) and a Markov chain model can be assumed among states at different time instances, then a state-space model may be formed.
- the state-space model may include 1) a dynamic/state model that relates state variables 1000 A- 1000 D at different time instances (t ⁇ 1, t, t+1, t+2) and 2) an observation model that relates state variables S 1000 A- 1000 D to observations O 1002 A 1002 D.
- the state-space model (a discrete version is shown—a continuous version is similar and involves temporal integration) may be described by the following set of mathematical equations,
- D and H are the dynamic matrix and observation matrix respectively.
- v t and w t are respectively the dynamic noise and the observation noise that have Gaussian distributions N( ⁇ s ,C d ) and N( ⁇ o ,C o ) respectively.
- C d and C o are covariance matrices for dynamic model and observation model respectively.
- State-space models haves been used in many disciplines of science and engineering, and may be referred to by different names such as Bayesian filtering, optimal filtering, stochastic filtering, on-line inference, for example.
- the state-space model is linear and the modeling noises of the system are Gaussian, then a substantially exact analytic expression can be derived to solve the on-line estimation problem for the state variables of information fusion.
- the analytic expression that is well-known and widely used is a Kalman filter (see R. E. Kalman, “A New Approach to Linear Filtering and Prediction Problems,” Trans of the ASM—Journal of Basic engineering, Vol. 82, pp. 35-45, 1960.).
- EKF Extended Kalman Filtering
- SMC Sequential Monte Carlo
- s represents a state vector of the states 1000 A- 1000 D.
- s may be a vector version of the matrix T ⁇ P, ⁇ (Eq. 1), or a collection of point positions [X,Y,Z] T , etc., that are equivalent in theory but may be chosen based on a particular application.
- Respective velocity information may be readily added to the state space model by taking the first derivative of the positional information.
- Position and velocity may be represented mathematically as
- the orientation and angular velocity of the robotic instrument may be represented mathematically using unit quaternion for its minimal-style representation and operation efficiency as
- the state vector may be represented mathematically in a covariance matrix as follows:
- the filter state-space model may be implemented with extended Kalman filtering (simple to implement but maybe not sufficient), unscented Kalman filtering (see S. J. Julier and J. K. Uhlmann. A New Extension of the Kalman Filter to Nonlinear Systems. In Proc. of AeroSense: The 11 th Int. Symp. on Aerospace/Defense Sensing, Simulation and Controls., 1997 (easy to implement and maybe sufficient)), or computationally expensive particle filtering.
- Extended Kalman filtering or particle filtering may be used for tool tracking because 1) the dynamic state space model is non-linear due to the quaternion representation, and 2) the observation model is non-linear in the case of using 2D images as observations and linear in the case of using stereo-derived 3D points as observations.
- Sets of observations are available to the state space model including robot kinematics (k t P and k t ⁇ ) 1010 and vision 1011 .
- image measurements, the 2D locations of featured points x i are directly used to construct the observation vector O.
- Such a choice make the observation equation for the vision part nonlinear due to the perspective projection of a 3D tool onto one 2D image in the case of monocular view, or left and right images in the case of stereo view.
- observation matrix C o consists of two parts: one for kinematics (the leading diagonal sub-matrix of Eq. 12) and one sub-matrix, a covariance matrix C o,v , for vision as follows:
- observation equation is linear for the vision part and we can construct the observation covariance matrix C u,v .
- observation covariance matrix C u,v we have the following covariance matrix for the case of parallel camera setup ( FIG. 12B ):
- view-geometry variance matrices C o,v,i for each 3D point X s,i ) are related to 1) the uncertainty (standard deviation) of matching stereo images, 2) the inverse of image resolution (for example, high-definition camera offers better accuracy than standard-definition camera), and 3) the square of the true values of X s,i .
- first-order kinematic information (velocity information) ( ⁇ dot over (k) ⁇ t P and ⁇ dot over (k) ⁇ t ⁇ ) is provided that may provide an extra constraint with respect to the evolution (e.g., changes from state to state) of the dynamic system.
- first-order vision observations may be derived by tracking image points (i.e., temporal image matching) to provide an extra constraint.
- estimated states e.g., tool poses
- observation vectors may be formed for the vision-based information and the observation equation is non-linear due to the perspective projection of 3D points to 2D points.
- Kalman filtering provides a perfect sequential solution to a batch least square optimization problem. Because of the Markovian assumption that given the present state future states are conditionally independent of the past states.
- tool tracking in reality is a nonlinear problem and it is often not easy to achieve an accurate solution with a sequential approach such as extended Kalman filtering.
- an iterative bundle adjustment process that combines all observations across multiple states may be used starting from initial results provided by the extended Kalman filtering 802 .
- bundle adjustment has wide applications.
- bundle adjustment can be used for sequential matching to be discussed later.
- bundle adjustment can be used for sequential matching to be discussed later.
- Bundle adjustment an optimization technique in photogrammetry, refers to the “bundles” of light rays leaving each 3D feature and converging on each camera center which are “adjusted” optimally with respect to both feature and camera positions (reference: C. Salma, C. Theurer, and S. Henrikson, Manual of Photogrammetry, American Society of Photogrammetry, Falls Church, 1980).
- a bundle adjustment problem is essentially just a large sparse geometric parameter problem that is nonlinear in nature.
- iterative procedures may be used, such as the well known Levenberg-Marqudart mathematical algorithm (see “Numerical Recipes in C: The Art of Scientific Computing”, William H. Press, Saul A. Teukolsky, William T. Vetterling, Brian P. Flannery, Second edition, Cambridge University Press, 1992).
- the following describes one example of applying bundle adjustment to improve the pose estimate for the problem of tool tracking.
- Bundle adjustment is used to optimize the estimated poses of the robotic instrument in a chosen coordinate system, e.g., the camera-centered coordinate system. That is, it is desirous to obtain the relative orientation of the tool with respect to the camera.
- bundle adjustment is applied whenever there is a new observation, e.g., a new pair of stereo images (see P. McLauchlan, “The Variable State Dimension Filter applied to Surface-Based Structure from Motion,” CVSSP Tech Report TR-4/99, University of Survey, 1999). Typically, some approximation is made to make the computations efficient.
- bundle adjustment is applied to a short sequence of states and measurements.
- bundle adjustment is applied to all available states and measurements.
- a batch process approach may be best, followed by a window-based approach, and finally a recursive/sequential approach in implementing bundle adjustment with the state-space model.
- a batch bundle adjustment may be applied at each time instance or at selected time instances based on all the measurements and state estimates that are available from extended Kalman filtering. Applying a batch bundle adjustment in the beginning of time where the state-space model is applied may be preferred because 1) quick convergence to the correct solution to a non-linear optimization problem is desirous from the beginning, and 2) the computation is efficient because there are only a small number of states and observations available.
- Image matching is used for incorporating vision information into the tool tracking system and its state space model.
- Image matching in the image analysis steps (Eqs. 4 and 5) finds the corresponding 2D feature image points and 3D points on the tool.
- Image matching also finds the corresponding image features between the left and right images from a stereo camera.
- Image matching may be facilitated by sequence matching of a temporal sequence of frames of video images.
- image matching may be facilitated by using a 2D or 3D model of a tool and a video image.
- Implementation of the two dimensional image matching may alternately be performed by simple intensity-based SSD (sum of squared difference), feature-based matching (for example, a point, a scale-invariant-feature-transform feature: D. Lowe, “Object recognition from local scale-invariant features,” Proc. Int. Conf. Computer Vision, 1999.), or probabilistic matching.
- simple intensity-based SSD sum of squared difference
- feature-based matching for example, a point, a scale-invariant-feature-transform feature: D. Lowe, “Object recognition from local scale-invariant features,” Proc. Int. Conf. Computer Vision, 1999.
- probabilistic matching When matching a 2D image with a 3D model synthesized image, more robust features such as edges can be used.
- the cost function would be the sum of distance measures from the image edge points to the closest synthesized curves/lines (reference: D. Lowe, “Fitting parameterized three-dimensional models to images,” IEEE Trans. on Pattern Analysis and Machine Intelligence , Vol. 13, pp. 441-450.).
- Image matching can be applied in different scenarios, such as temporal matching of images within sequences of images ( FIG. 11A ) or spatial matching of images across two stereo views ( FIG. 11C ).
- real images are used to match against synthesized images in the analysis-by-synthesis approach ( FIG. 11B ).
- two or more of these image matching techniques may be used together to perform image matching.
- image matching could be applied to corresponding artificial markers attached to instruments, natural image features or image appearances (e.g., instrument tips).
- the artificial markers are passive visual markers.
- FIG. 11A temporal image matching of a pair of video images 1101 V- 1101 V′ is illustrated.
- the video image 1101 V of the actual tool 101 (e.g., see FIG. 1A ) is taken at time t resulting in a tool image 101 V at time t.
- the video image 1101 V′ of the same actual tool 101 is taken at a different time, time t+1, by the same camera resulting in a tool image 101 V′ at time t+1.
- the camera may be fixed with respect to the robotic surgical system while the tool 101 may move relative to the robotic surgical system.
- markers 502 V on the tool image 101 V in the first video image 1101 V of the tool 101 may be compared with the markers 502 V′ on the tool image 101 V′ in the second video image 1101 V′ to help determine new pose information of the actual tool 101 .
- marker image matching other information may be used determine pose information of the actual tool 101 .
- synthesis image matching of a video image 1101 V and a synthesized image 1101 S is illustrated.
- the video image 1101 V of the actual tool 101 (e.g., see FIG. 1A ) is taken by a camera resulting in a tool image 101 V.
- the synthesized image 1101 S of the tool 101 is generated by a computer having prior knowledge of the actual tool 101 resulting in a synthesized tool image 101 S.
- the pose of synthesized tool image 101 S in the synthesized image 1101 S of the tool 101 attempts to match the pose of the tool represented by the tool image 101 V in the video image 1101 V at a moment in time.
- Various aspects of the video image 1101 V and the synthesized image 1101 S of the tool may be used to perform image matching. That is, one or more of a matching of markers 1110 , a matching of natural features 1111 , and/or an appearance matching 1112 may be performed. For example, markers 502 V in the first video image 1101 V of the tool image 101 V may be compared with the synthesized markers 502 S on the synthesized tool image 101 S in the synthesized image 1101 S of the same tool 101 to help determine new pose information of the actual tool 101 .
- FIG. 11C special image matching of a left video image 1101 VL and a right video image 1101 VR is illustrated.
- the left video image 1101 VL of the actual tool 101 e.g., see FIG. 1A
- the right video image 1101 VR of the same actual tool 101 is taken at a different position with respect to the tool, such as the right side, resulting in a right tool image 101 VR.
- the left video image 1101 VL and the right video image 1101 VR are captured at substantially the same time by their respective cameras.
- markers 502 VL on the left tool image 101 VL in the left video image 1101 VL of the tool 101 may be compared with the markers 502 VR on the right tool image 101 VR in the right video image 1101 VR of the same tool to determine new pose information of the actual tool 101 .
- these image matching techniques may be combined to generate better pose information for the tool. For example, it is natural to combine temporal image matching ( FIG. 11A ) with spatial image matching ( FIG. 11C ) or combine temporal image matching ( FIG. 11A ) with synthesis image matching ( FIG. 11D ). However, all three techniques may be used all together or flexibly in various combinations to try and obtain the best pose information for the tool.
- stereo images are used to construct 3D feature points, and these 3D points can be matched (for example, through the popular iterative closest point algorithm, reference: P. Besel and N, McKay, “A method for registration of 3-D shape,” IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 14, pp. 239-256, 1992) against corresponding 3D points, for example, markers, on the robotic instrument.
- matched for example, through the popular iterative closest point algorithm, reference: P. Besel and N, McKay, “A method for registration of 3-D shape,” IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 14, pp. 239-256, 1992
- corresponding 3D points for example, markers
- image matching has been described with respect to explicit image synthesis, such as a full image synthesis or a geometry-only image synthesis.
- image matching may be made using implicit image synthesis. That is, images are not actually synthesized. Rather, a computer aided design (CAD) model of the tools and prior pose information of the tools are implicitly used to facilitate image matching.
- CAD computer aided design
- artificial markers may be applied to the tools to assist in image matching as is described below.
- natural markers or features of a tool may be used to assist in image matching.
- the following descriptions apply equally well to the case when no artificial markers are present and features of the tool, i.e., natural markers, are detected directly from the instrument images.
- FIG. 5B illustrates video images 101 AV and 101 BV for a respective pair of tools 101 A and 101 B in the field of view 510
- FIG. 5B further illustrates pose information 101 AK and 101 BK based on kinematics for the respective tools 101 A and 101 B in and around the field of view 510
- the video images 101 AV and 101 BV and the pose information 101 AK and 101 BK for the respective tools 101 A and 101 B may be adaptively fused together to improve the overall pose information for each.
- a plurality of marker dots 502 A and 502 B or other types of markers may be affixed to the respective tools 101 A and 101 B.
- Video information of the marker dots 502 A′ and 502 B′ may be ascertained from the video images 101 AV and 101 BV of the respective tools 101 A and 101 B.
- image feature points maker dots 502 A′- 502 B′
- the image feature points form a pattern.
- the image matching problem is simplified from many dot-to-dot matchings to a single pattern matching.
- a pattern matching or association is straightforward if one tool is in the field of view. If more that one robotic instrument tool is in the field of view, there are two approaches available to solving the image matching problem that can be used alone or combined together.
- the robotic kinematics information will not change the spatial arrangements (positional and/or orientational) across instruments, especially in the beginning of surgical operations. For example, if two instruments are arranged to the left and to the right in the camera coordinate system, then the robotic kinematics should represent that arrangement. However, this is not a requirement on the absolute values of the robotic kinematics information.
- the robotic kinematics information may indicate that the one or both of the robotic surgical tools are outside the field of view. Resolving pattern association ambiguity in matching a first pattern in an image to the tool arranged to the left and a second pattern in the image to the tool arranged to the right can be carried out in either 2D image or 3D space.
- tool motion is used to resolve the pattern association ambiguity.
- pattern images are tracked individually through temporal image matching.
- the motion trajectories of the pattern images are then compared against the kinematic information of each of the tools. From the comparison, the correct association of pattern images to tools can then be made.
- a pattern association issue for a single tool in the field of view There may be a pattern association issue for a single tool in the field of view.
- a pattern may be directionally ambiguous, such as a line pattern with identical markers that has a directional ambiguity of a 180-degree flip.
- Pattern association for a single tool is not an issue if the pattern consisting of artificial markers or natural markers is unique. For example, if the pattern of markers has directional information embedded such that the markers at each of ends of the pattern are distinctive there is no issue.
- the design of the artificial markers on the tool provides directional information of the marker pattern.
- the first approach to solving the image matching problem is to use very rough robotic kinematics information to resolve any directional ambiguity.
- the robotic kinematics should not flip although it may be far away, such as outside the field of view after projection onto 2D image.
- the second approach to solving the image matching problem is to use motion trajectories of the pattern to remove the directional ambiguity.
- Image information may have quality issues (matching reliability and accuracy) regarding image-derived 3D information.
- a viewing-singularity happens when a cylindrical instrument with markers on the shaft is projected to an image of small circle (see 1222 S in FIG. 12A ). That is, all the markers become invisible, hence there is no vision-based observation.
- Another extreme case for perfect viewing is when the instrument shaft lies right in the field of view so that all the markers are fully visible. That is, a circular marker is projected onto image as a circle for example. In practice, we often have scenarios between these extreme cases.
- Viewing geometry is more than just pure geometry between a camera and a 3D object. Other information related to viewing geometry can impact the quality of image-derived information for fusion.
- the geometry of an illumination source 1210 A- 1210 B with respect to the camera 1211 A- 1211 B and the 3D object may be of interest to achieve accurate and reliable information.
- the position of light source 1210 A with respect to the camera 1211 A and the 3D object 1200 generating view 1 may differ from the position of light source 1210 B with respect to the camera 1211 B and the 3D object 1200 generating view 2.
- the different poses of the 3D object may change how a light source strikes its features and provide different views.
- the tip 1200 T and the shaft 1200 S of a 3D tool 1200 may generate different image-derived 3D information with different reliabilities.
- image 1221 A the shaft image 1221 S may be good to use while in image 1221 B the shaft image 1222 S may be poor to use in forming image-derived 3D information.
- image-derived 3D information also depends upon the particular image features used. For example, edge-based features are less sensitive to illumination change than intensity-based features.
- View-geometry statistics are used to represent how good the image-derived information for fusion is when compared to the ground-truth. That is, view-geometry statistics may be used to estimate uncertainty.
- the view-geometry statistics are used in the Bayesian state-space model for adaptively fusing image information and kinematics information. To be specific, the view geometry statistics may be represented in the covariance matrix of the observation equation (Eq. 13).
- view-geometry statistics Digitization error/image resolution; feature/algorithm related Image matching error; Distance from object to camera; Angle between object surface normal and line of sight; Illumination and specularity.
- noise assumptions e.g., independent Gaussian noise
- view-geometry statistics for these phenomenon may be computed.
- the plot in FIG. 12C is based on simulation of image digitization error [ ⁇ 0.5, 0.5] pixel and correct image matching to confirm the conclusion that the estimate uncertainty of a 3D point is proportional to the inverse of image resolution.
- the plot in FIG. 12D is based on simulation of varying depth to confirm the conclusion that the estimate uncertainty of a 3D point is proportional to the square of the true value of X/Y/Z:
- View-geometry statistical analysis can also be applied to the most general case of camera set up, for example, a non-parallel set up. It can also be applied to the case of using 2D images rather than 3D stereo-reconstructed points as observations.
- 2D images rather than 3D stereo-reconstructed points as observations.
- a technique is chosen for matching a pair of image features 1301 and 1302 such as from left and right images respectively in a stereo setting or a real image and a synthesized image.
- the pair of images features may be matched over one or more sequences of images 1301 A- 1301 E and 1302 A- 1302 E, for example.
- a rigid sequence matching (where the relative kinematics within each sequence are perfectly known or identical across sequences in the ideal case) may be employed where just one common motion parameter is used to estimate for all pairs between two sequences such as illustrated in FIG. 13A .
- the sequences 1301 and 1302 have identical motion relationship (relative kinematics) among the images 1301 A to 1301 E, and 1302 A to 1302 E.
- a flexible sequence matching (where the relative kinematics within each sequence are known with errors) may be employed as illustrated in FIG. 13B .
- the sequences 1303 and 1302 have a identical motion relationship (relative kinematics) among the images 1303 A to 1303 E, and 1302 A to 1302 E.
- the probability of matching image features can be improved over that of a single pair of images. Assuming statistical independence and that the chance or probability of matching a single pair of image features correctly is 60% the chance of having correct matching improves to 78% with a sequence of 3 images, 92% with a sequence of 5 images, and 99% with a sequence of just 10 images, for example. However, if the relative kinematics for each pair of images in the sequence is not accurate, the chance of having correct matches is less and a bundle adjustment procedure should be taken for improved matching. Within the bundle adjustment procedure, the relative kinematics and its uncertainty are taken into consideration.
- T s , i * arg ⁇ ⁇ min T s , i ⁇ ⁇ C ⁇ ⁇ F 1 , i ⁇ ( T s , i ⁇ ( X ) ) , F 2 , i ⁇ ( X ) ⁇ ( Eq . ⁇ 19 )
- C is the cost function of matching features F 1,i and F 2,i and X represents the location of the features in either two or three dimensions.
- the geometric transformation Y s,i may be in 3D and the features may be in 2D after known camera projection. If relative kinematics ⁇ dot over (k) ⁇ 1/2,i within each sequence are perfectly known, then the geometric transformation equation for sequence matching may become
- T s * arg ⁇ ⁇ min T S ⁇ ⁇ C ⁇ ⁇ F 1 , 1 ( T s k . 1 , 1 ⁇ ( X ) ) , F 2 , 1 ⁇ ( k . 2 , 1 ⁇ ( X ) ) ; ... ⁇ ; F 1 , n ( T s k . 1 , n ⁇ ( X ) ) , F 2 , n ⁇ ( k . 2 , n ⁇ ( X ) ) ⁇ ( Eq . ⁇ 20 )
- T s k 1,1 represents transformation from T s , typically a chosen T s,i , through relative known kinematics.
- Sequence matching methods may be applied in many different scenarios, e.g., a single view case (a sequence of real images against a sequence of synthesized images) or a stereo case (matching among two sequences of real images and two sequence of synthesized images).
- a single view case a sequence of real images against a sequence of synthesized images
- a stereo case matching among two sequences of real images and two sequence of synthesized images.
- relative kinematics may not be perfect and may change over time.
- bundle adjustment based flexible sequence matching may be applied to estimate the optimal parameters to more accurately locate and track tools.
- a tool tracking system for a robotic instrument has a number of applications.
- One application for a tool tracking system is image-guided surgery (IGS), or more specifically image-guided endoscopic surgery (IGES).
- IGS image-guided surgery
- IGES image-guided endoscopic surgery
- the basic goal of image guided surgery is to enhance a surgeon's experience by providing real time information derived from single or multiple imaging modalities (e.g., visual, x-ray, computerized topography (CT), magnetic resonance imaging (MRI), ultrasound) during surgery or training/simulation.
- imaging modalities e.g., visual, x-ray, computerized topography (CT), magnetic resonance imaging (MRI), ultrasound
- CT computerized topography
- MRI magnetic resonance imaging
- ultrasound e.g., magnetic resonance imaging
- Two particular benefits of IGS/IGES are 1) improved visualization for easier on-line diagnostics and 2) improved localization for reliable and precise surgery.
- Tool tracking is one technology used for IGS/IGES since instruments are used by surgeons to navigate, sense and operate (e.g., diagnostic, cut, suture, ablation etc.) in the areas of interest. That is, the tool tracking system described herein can enable image-guided surgery without significant added operational inconvenience and/or added equipment.
- the tool tracking system described herein may also be used in other applications, such as dynamically reconstructing the geometry of organs, surgical simulation, and training.
- Tool tracking may be used to provide automated camera control and guidance to maintain a robotic instrument in the field of view. Tool tracking can also be used to assist the surgeon to move the robotic instrument to reach a tumor either automatically or with a surgeon's assistance. Ultra-sound or pre-scanned images can also be used along with real-time tool tracking. Other applications of tool tracking include graphic user interface that facilities the entrance and re-entrance of the robotic instrument during surgery.
- Tool tracking can be used to take a number of measurements during surgery as well.
- tool tracking may be used to measure organ sizes by touching the robotic tool tip at different points of an organ.
- a pair of robotic tools being tracked can concurrent touch points of the organ and a distance along a line between their tips can be accurately measured with the assistance of tool tracking.
- tool tracking may be used to construct a 3D model of an organ.
- the tip of a single robotic tool may be used to touch points across the organ's surface to construct a 3D model of the organ.
- a perspective view of a surgical site 1700 includes a robotic ultrasound tool 1710 .
- the robotic ultrasound (US) tool 1710 has an attached or integrated ultrasound transducer 1710 A.
- the robotic ultrasound (US) tool 1710 may be used to guide and navigate other instruments to perform various medical or surgical procedures.
- two dimensional ultrasound images 1711 A may be captured in a two dimensional coordinate system 1703 .
- the two dimensional ultrasound images 1711 A may be translated from the two dimensional coordinate system 1703 into a camera coordinate system 1701 .
- the translated ultrasound images may then be overlaid onto video images of the surgical site 1700 displayed by the stereo viewer 312 , such as illustrated by the translated ultrasound images 1711 B- 1711 D in the surgical site 1700 illustrated in FIG. 17 .
- Tool tracking may be used to flagpole ultrasound by (1) determining the transformation of the ultrasound images 1711 A from the two dimensional coordinate system 1703 to the local ultrasound coordinate system 1702 in response to ultrasound calibration; (2) at the transducer 1710 A, determining the transformation from the ultrasound transducer coordinate system 1702 to the camera coordinate system 1701 by using tool tracking; and then; (3) cascading the transformations together to overlay the ultrasound image in the camera coordinate system 1701 onto the surgical site as illustrated by image 1711 B.
- the quality of image overlaying depends on tracking accuracy of the robotic ultrasound tool within a camera coordinate system ( FIG. 5A ) and the ultrasound calibration.
- Ultrasound calibration is described in the reference “A novel closed form solution for ultrasound calibration,” Boctor, E.; Viswanathan, A.; Choti, M.; Taylor, R. H.; Fichtinger, G.; Hager, G., Biomedical Imaging: Nano to Macro, 2004. IEEE International Symposium on Volume, Issue, 15-18 Apr. 2004 Page(s): 527-530 Vol. 1.
- Tool tracking may also be used to generate 3D/volumetric ultrasound images by stacking overlaid 2D images 1711 B-D from the 2D ultrasound transducer 1710 A generated by rotating the ultrasound tool 1701 as illustrated by the arrow 1720 .
- tool tracking may be used overlay one or more drop virtual point/marks 1650 A- 1650 B on images of the tissue surface 1600 in the surgical site by using one or multiple tools 1610 L, 1610 R (for example, a surgical tool or an ultrasound tool) to touch point of interest.
- tools 1610 L, 1610 R for example, a surgical tool or an ultrasound tool
- teaching surgeons can use tools to draw virtual marks to illustrate areas of interest to remote student surgeons on an external display.
- surgeon can use one type of tracked tool (e.g., an ultrasound tool) to draw marks to indicate regions of interest and then use a different type of tracked tool (e.g., a cautery tool) to operate or perform a surgical or other medical procedure in selected regions of interest.
- a different type of tracked tool e.g., a cautery tool
- the tool tracking system described herein may also be used for image-guided interventional radiology (IGIR) along with other sensors.
- active sensors/cameras e.g. electro-magnetic sensor, or active near Infra-Red illumination plus stereo camera
- IGIR image-guided interventional radiology
- active sensors/cameras e.g. electro-magnetic sensor, or active near Infra-Red illumination plus stereo camera
- FIG. 15 a flow chart 1500 of an IGS application with tool tracking is illustrated.
- the flow chart 1500 illustrates a robotic surgery for a tumor in or around a liver.
- IGS with tool tracking may be used for other medical and surgical procedures for different organs and tissue that are robotically controlled or robotically assisted.
- a liver or other organ/tissue prior to surgery may be scanned with a computer tomography scanner 1503 to obtain a number of images of the liver such that they may be used to reconstruct a complete 3D volume of a scanned object at block 1504 by a computer as desired.
- a computer tomography scanner 1503 may be used to obtain a number of images of the liver such that they may be used to reconstruct a complete 3D volume of a scanned object at block 1504 by a computer as desired.
- E CT computer tomography error
- a computer tomography volume that includes liver and surrounding area is selected to generate a computer tomographic (CT) segment that contains liver only.
- CT computer tomographic
- the CT volume segment is taken to register against a surface/depth map from other imaging modalities, e.g., stereo cameras.
- the real liver during surgery 1501 B may have some biological variation forming an error (E BIO ) from the prior scan taken by the CT scanner.
- One or more robotic instrument tips 1510 are inserted into the surgical site.
- One or more endoscopic cameras 1512 take sequences of images 1513 - 1514 of the surgical site including the liver 1501 B and the robotic instrument tips 1501 . These sequences of images 1513 - 1514 are typically stereo image pairs but could be a sequence of single images from a mono-view. The sequence of images 1513 may be used to determine the depth of surface features to make a surface/depth map 1515 . In one embodiment of the invention, a sequence of surface maps 1515 including a robotic instrument may be analyzed similar to a sequence of images 1513 as described herein to determine a location of the robotic instrument.
- the surface map 1515 may be overlaid with a model of an organ after surface registration.
- the surface map 1515 may be further annotated with a map, outline, or other indication of the estimated tumor location 1517 .
- an internal liver tumor that is not visually visible maybe easily visible in a CT scan of the liver.
- An error between the estimated tumor location and the actual tumor location during surgery may be a sum of errors of the scanning (Ect), surface registration (Ereg), biological changes (Ebio), and formation of the depth/surface map (Est). This error can be reduced with further information from tool tracking and touching a tool tip to the tissue.
- the sequences of images 1514 may be stereo image pairs or a sequence of single images from a mono-view.
- the sequence of images 1514 along with kinematics information may be adaptively fused together to translate a model of the tool tip (M ET ) into the endoscopic camera frame of reference (ECM).
- the model of the tool tip formed from the adaptive fusion based tool tracking may be used to estimate the tool tip location 1518 .
- An estimation error (E ET ) between the actual tip location 1520 and the estimated tip location 1518 may be made small by the tool tracking methods described herein.
- One or more of the robotic surgical tools may include a needle at its tip as the end effector.
- the operational error E OP between the actual location of the tumor and the actual location of tool tips is a sum of errors.
- a pre-scanned image 1602 A (for example, CT images) may be aligned to a camera coordinate system 1605 and then overlaid as an overlaid image 1602 B onto a surface map or depth map 1600 of the tissue surface.
- Visual information and kinematics information of the first robotic instrument 1610 L and the second robotic instrument 1610 R may be adaptively fused together to more accurately determine the position of the tools within the surgical site. With both position of the overlaid image 1602 B and the tools 1610 L, 1610 R aligned to the camera coordinate system 1605 , the tool may be automatically or manually guided to perform a procedure on the tissue.
- the overlaid image 1602 B may represent a tumor on the tissue surface or buried below the tissue surface that would otherwise be hidden from view.
- an image 1610 B of the portion of the tool below the surface may be synthesized in response to the tool tracking information and the adaptive fusion of kinematics with a priori video information of the tool.
- the one or more robotic surgical tools 16101 , and 1610 R may be used to take measurements or determine a surface profile. Tissue in a surgical site may be touched with the tool tip of the robotic tool 1610 L at a first point and tissue in the surgical site may be touched with the tool tip of the robotic tool 1610 R at a second point. A sequence of images of the surgical site including the robotic surgical tools 1610 L and 1610 R may be captured. Kinematics information and image information of the robotic tools may be adaptively fused together to accurately determine the tool tip locations at the first and second points. The pose of the first tool tip at the first point and the pose of the second tool tip at the second point may be compared to determine a distance between them. The tool tips may be touching external surfaces of a tumor to determine a diameter of the tumor. Alternatively, the tool tips may be touching external surfaces of an organ to measure the diameter of the organ.
- a sequence of depth maps including a robotic instrument may be analyzed to determine a location of robotic instruments.
- Robotic instruments can be located in the camera coordinate frame using any depth map in which the tool can be identified.
- Kinematics datum provides an approximate location for a robotic instrument that is to be tracked. This is a priori knowledge for the next iteration of the tracking problem.
- the depth map is analyzed in the environs of the approximate location to locate the robotic instrument. Other information may be employed to improve the a priori knowledge, such as a dynamic model of the robotic instrument, or knowledge of the type of procedure being performed under the camera capturing the images.
- the locating of the robotic instrument may be performed over any number of depth maps.
- the depth maps may be sequential views, arranged in time order, in which case there is a correlation between the successive views.
- Kinematics datum provides an approximate location for a robotic instrument that is to be tracked. This is a priori knowledge for the next iteration of the tracking problem.
- the depth map is analyzed in the environs of the approximate location to locate the robotic instrument.
- Other information may be employed to improve the a priori knowledge, such as a dynamic model of the robotic instrument, or knowledge of the type of procedure being performed under the camera capturing the images. If the robotic tool is obscured, a current optimal estimate of the location of the surgical instrument may be made (an a posteriori estimate) using the a priori knowledge and the depth map.
- Instantaneous (re-)correction of the kinematics datum may be computed by adaptively fusing together the available kinematics data, visual information, and/or a priori information.
- the correction to the current state is used to update the ongoing correction of future kinematics data.
- the correction is simply made to future data without regard for past corrections.
- the sequence of corrections is analyzed and an optimal correction based on all available past corrections is computed and used to correct the kinematics data. Analysis by synthesis and appearance learning techniques may be used to improve the correction to the current state.
- Algorithms that locate the surgical instrument in the depth map and provide the optimal kinematic correction can be further optimized by understanding of the relative variances in corrected kinematic residual error vs. the variances in computed robotic instrument location from the depth map.
- Kinematics is suspected of initially having a large DC bias, but relatively small variance.
- a well-design image processing subsystem may have substantially zero DC bias but a relatively large variance.
- An example of an optimal correction that accommodates these two differing noise processes is one generated by a Kalman filter.
- an image 1610 B of the portion of the tool below the surface may be synthesized.
- the synthesized image portion 1610 B may be included in the tool images 400 L, 400 R as synthesized image portion 400 B displayed on the display devices 402 L, 402 R of the stereo viewer 312 illustrated in FIG. 4 .
- the robotic surgical tool may exit the field of view 510 of a camera entirely such, as illustrated by the tool 101 F.
- the robotic surgical tool may no longer be in the field of view as a result of camera movement over the surgical site, robotic surgical tool movement, or a combination of both.
- the camera may move away from the position of the robotic surgical tool in a surgical site such that the robotic surgical tool is outside the field of view of the camera.
- the robotic surgical tool may move away from a position of the camera in a surgical site such that the robotic surgical tool is outside the field of view of the camera. In either case, a surgeon may be left guessing where the robotic surgical tool is outside the field of view unless some indication is provided to him in the stereo viewer 312 .
- compass icons or a compass rose may be displayed in the display devices of the stereo viewer 312 to provide an indication where the robotic surgical tool is located outside the field of view and a direction of tool reentrance into the field of view.
- compass icons for the directions North 420 N, South 420 S, East 420 E, and West 420 W as well as directions in between such as North-East 420 NE, North-West 420 NW, South-East 420 SE, and South-West 420 SW may be indicated in the stereo viewer to indicate tool reentrance into the field of view of the camera over a surgical site.
- a robotic surgical tool is tracked in and out of the field of view of the camera.
- a determination is made by comparing positions of the camera and a tool whether or not it is outside the field of view of the camera by using the available tool tracking information of the robotic surgical tool. It the robotic surgical tool is out of the field of view, then one of a plurality of compass icons 420 may be displayed in the field of view to show a direction of tool reentrance.
- Additional information may be conveyed to a surgeon looking in the stereo viewer 312 by somewhat altering the one compass icon 420 in the display that indicates the direction of the tool. For example, by flashing the icon fast or slow may indicate the robotic surgical tool is getting closer or farther away respectively from the field of view. As another example, arrowheads may be added to ends of a bar icon to indicate that the robotic surgical tool is move towards or away from the field of view. In an alternate implementation, the icon indicating direction or reentrance may be colored to indicate movement bringing the tool closer to the field of view (e.g., red for getting warmer) or movement taking the tool further away from the field of view (e.g., green for getting colder).
- the embodiments of the tool tracking system described herein provide an automatic integrated system that is accurate and reliable by adaptively fusing kinematics and visual information, synthesizing images based on a model and prior poses, and employing sequence matching.
- a number of elements of the tool tracking system are implemented in software and executed by a computer and its processor, such as computer 151 and its processor 302 .
- the elements of the embodiments of the invention are essentially the code segments to perform the necessary tasks.
- the program or code segments can be stored in a processor readable medium or transmitted by a computer data signal embodied in a carrier wave over a transmission medium or communication link.
- the processor readable medium may include any medium that can store or transfer information.
- Examples of the processor readable medium include an electronic circuit, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM), a floppy diskette, a CD-ROM, an optical disk, a hard disk, a fiber optic medium, a radio frequency (RF) link, etc.
- the computer data signal may include any signal that can propagate over a transmission medium such as electronic network channels, optical fibers, air, electromagnetic, RF links, etc.
- the code segments may be downloaded via computer networks such as the Internet, Intranet, etc.
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Medical Informatics (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Biomedical Technology (AREA)
- Physics & Mathematics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Animal Behavior & Ethology (AREA)
- Veterinary Medicine (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Robotics (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Radiology & Medical Imaging (AREA)
- General Physics & Mathematics (AREA)
- Optics & Photonics (AREA)
- Theoretical Computer Science (AREA)
- Primary Health Care (AREA)
- Epidemiology (AREA)
- Business, Economics & Management (AREA)
- Multimedia (AREA)
- Mechanical Engineering (AREA)
- Human Computer Interaction (AREA)
- Medicinal Chemistry (AREA)
- Chemical & Material Sciences (AREA)
- Mathematical Physics (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Analysis (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- Computational Mathematics (AREA)
- Algebra (AREA)
- Mathematical Optimization (AREA)
- General Business, Economics & Management (AREA)
- Urology & Nephrology (AREA)
- Computer Vision & Pattern Recognition (AREA)
Abstract
In one embodiment of the invention, a method is disclosed to locate a robotic instrument in the field of view of a camera. The method includes capturing sequential images in a field of view of a camera. The sequential images are correlated between successive views. The method further includes receiving a kinematic datum to provide an approximate location of the robotic instrument and then analyzing the sequential images in response to the approximate location of the robotic instrument. An additional method for robotic systems is disclosed. Further disclosed is a method for indicating tool entrance into the field of view of a camera.
Description
- The embodiments of the invention relate generally to robots and robotic tools or instruments. More particularly, the embodiments of the invention relate to the acquisition and tracking of the position and orientation of robotic tools or instruments.
- Minimally invasive surgical (MIS) procedures have become more common using robotic (e.g., telerobotic) surgical systems. An endoscopic camera is typically used to provide images to a surgeon of the surgical cavity so that the surgeon can manipulate robotic surgical tools therein. However if the robotic surgical tool is not in the field of view of the camera or it is otherwise hidden by tissue or other surgical tools, a surgeon may be left guessing how to move the robotic surgical tool when it is obscured from his view.
- Moreover, tissue or organs of interest in a surgical cavity are often obscured from view. A surgeon may have to initially guess the location of an organ of interest within a surgical cavity and search around therein to place the organ and the robotic surgical tools within a field view of the endoscopic camera.
- To better localize a surgical tool in the field of view, optical devices, such as light emitting diodes, have been attached to robotic surgical tools. However, optical devices can interfere with endoscopic surgical procedures and may not provide sufficiently accurate position and orientation information for a minimally invasive surgical system. A magnetic device may be applied to a robotic surgical tool in an attempt to magnetically sense its location. However, robotic surgical tools are often formed of metal and a magnetic device may not work well due to the interference generated by the movement of metal-tools and electrical motors in a minimally invasive surgical system. Moreover, these may provide only a single clue of the position of a robotic surgical tool.
- The embodiments of the invention are summarized by the claims that follow below.
-
FIG. 1A is a block diagram of a robotic medical system including a stereo viewer and an image guided surgery (IGS) system with a tool tracking sub-system. -
FIG. 1B is a block diagram of a patient side cart including robotic surgical arms to support and move robotic instruments. -
FIG. 2 is a functional block diagram of the video portion of the IGS system to provide a stereo image in both left and right video channels to provide three-dimensional images in a stereo viewer. -
FIG. 3 is a perspective view of a robotic surgical master control console including a stereo viewer and an IGS system with tool tracking sub-system. -
FIG. 4 is a perspective view of the stereo viewer of the robotic surgical master control console. -
FIG. 5A is a perspective view of a sequence of video frames including video images of a robotic medical tool that may be used to perform tool tracking. -
FIG. 5B illustrates different tool positions of a pair of tools in the field of view of a camera based on kenematic information and video image information. -
FIG. 6A is a functional block diagram of a tool tracking architecture and methodology for a robotic system including one or more robotic instruments. -
FIG. 6B is a flow chart of a tool tracking library and its application. -
FIG. 7 is a block diagram illustrating various techniques that may be combined together to meet the challenges in tool tracking. -
FIG. 8 is a functional flow-chart of a tool tracking system. -
FIG. 9A is a figure to illustrate the process of pure image segmentation to localize a tool within an image. -
FIG. 9B is a figure to illustrate the process of sequence matching and/or model-based synthesis to localize a tool within an image. -
FIG. 9C is a more detailed figure to illustrate the process of model-based synthesis to localize a tool within an image. -
FIGS. 10A-10B illustrates elements of a state-space model to adaptively fuse robot kinematics information and vision-based information together. -
FIG. 11A-C illustrate various image matching techniques that may be used separately or collectively to determine pose information of a tool. -
FIG. 12A is a diagram illustrating adaptive fusion under different viewing conditions. -
FIG. 12B is a diagram illustrating a set up for parallel stereo. -
FIGS. 12C-12E are charts illustrating various view geometry statistics that may be used to enhance the performance of the state space model for adaptively fusing information sources together. -
FIGS. 13A-13B are diagrams illustrating sequence matching of a feature in a sequence of one or more images from different sources. -
FIG. 14 is a diagram illustrating appearance learning of objects within an image. -
FIG. 15 is a flow chart of the application of tool tracking to image guided surgery. -
FIG. 16 is a perspective view of overlaying a pre-scanned image of tissue onto a depth map of a surgical site to provide image-guided surgery. -
FIG. 17 is a perspective view of a surgical site with an ultrasound tool capturing ultrasound images for overlay onto a display. - In the following detailed description of the embodiments of the invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be obvious to one skilled in the art that the embodiments of the invention may be practiced without these specific details. In other instances well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention.
- Aspects of the invention include methods, apparatus and integrated systems for tool acquisition (locating) and tool tracking (kinematics-tracking (pose predicting) and full-tracking) of robotic medical tools. The method/system for tool tracking systematically and efficiently integrates robot kinematics and visual information to obtain pose (position/orientation) information, which can be used to obtain a more accurate pose of a robotic surgical tool than robot kinematics or visual information alone, in either a camera coordinate system or a base coordinate system. Known kinematics transformation can be applied to the pose correction to achieve improved pose in any related coordinate system. A camera coordinate system is a coordinate system based on a chosen camera (for example, (Xr, Yr, Zr) in
FIG. 12B ), or a common reference coordinate system for multiple cameras (for example, (XS, YS, ZS) inFIG. 12B ). In some aspects, tool tracking explores prior available information, such as the CAD models of tools, and dynamically learns the image appearances of the robotic instruments. In some aspects, tool tracking may be markerless so as not to interfere with normal robotic surgical procedures. Furthermore, tool tracking may provide continuous pose information of the robotic instruments including their relationships (e.g. tool A is on top of tool B and hence partially occluding tool B) with other tools so that image-based segmentation of the tools may be avoided. - Referring now to
FIG. 1A , a block diagram of arobotic surgery system 100 is illustrated to perform minimally invasive robotic surgical procedures using one or more robotic arms 158. Aspects ofsystem 100 include telerobotic and autonomously operating features. These robotic arms often support a robotic instrument. For instance, a robotic surgical arm (e.g., the center roboticsurgical arm 158C) may be used to support a stereo or three-dimensional surgicalimage capture device 101C such as a stereo endoscope (which may be any of a variety of structures such as a stereo laparoscope, arthroscope, hysteroscope, or the like), or, optionally, some other imaging modality (such as ultrasound, fluoroscopy, magnetic resonance imaging, or the like). Robotic surgery may be used to perform a wide variety of surgical procedures, including but not limited to open surgery, neurosurgical procedures (e.g., stereotaxy), endoscopic procedures (e.g., laparoscopy, arthroscopy, thoracoscopy), and the like. - A user or operator O (generally a surgeon) performs a minimally invasive surgical procedure on patient P by manipulating
control input devices 160 at amaster control console 150. Acomputer 151 of theconsole 150 directs movement of robotically controlled endoscopicsurgical instruments 101A-101C viacontrol lines 159, effecting movement of the instruments using a robotic patient-side system 152 (also referred to as a patient-side cart). - The robotic patient-
side system 152 includes one or more robotic arms 158. Typically, the robotic patient-side system 152 includes at least three roboticsurgical arms 158A-158C (generally referred to as robotic surgical arms 158) supported by corresponding positioning set-uparms 156. The central roboticsurgical arm 158C may support anendoscopic camera 101C. The roboticsurgical arms robotic instruments - Robotic instruments are generally referred to herein by the
reference number 101.Robotic instruments 101 may be any instrument or tool that couples to a robotic arm that can be manipulated thereby and can report back kinematics information to the robotic system. Robotic instruments include, but are not limited to, surgical tools, medical tools, bio-medical tools, and diagnostic instruments (ultrasound, computer tomography (CT) scanner, magnetic resonance imager (MRI)). - Generally, the robotic patient-
side system 152 includes a positioning portion and a driven portion. The positioning portion of the robotic patient-side system 152 remains in a fixed configuration during surgery while manipulating tissue. The driven portion of the robotic patient-side system 152 is actively articulated under the direction of the operator O generating control signals at the surgeon'sconsole 150 during surgery. The driven portion of the robotic patient-side system 152 may include, but is not limited or restricted to roboticsurgical arms 158A-158C. - The
instruments 101, the roboticsurgical arms 158A-158C, and the set upjoints orientational sensors side system 152 is reported back to the robotic system and a tool tracking and image guided surgery (IGS)system 351. - As an exemplary embodiment, the positioning portion of the robotic patient-
side system 152 that is in a fixed configuration during surgery may include, but is not limited or restricted to set-uparms 156. Each set-uparm 156 may include a plurality of links and a plurality of joints. Each set-up arm may mount via a first set-up-joint 157 to thepatient side system 152. - An assistant A may assist in pre-positioning of the robotic patient-
side system 152 relative to patient P as well as swapping tools orinstruments 101 for alternative tool structures, and the like, while viewing the internal surgical site via anexternal display 154. Theexternal display 154 or anotherexternal display 154 may be positioned or located elsewhere so that images of the surgical site may be displayed to students or other interested persons during a surgery. Images with additional information may be overlaid onto the images of the surgical site by the robotic surgical system for display on theexternal display 154. - Referring now to
FIG. 1B , a perspective view of the robotic patient-side system 152 is illustrated. The robotic patient-side system 152 comprises acart column 170 supported by abase 172. One or more robotic surgical arms 158 are respectively attached to one or more set-uparms 156 that are a part of the positioning portion of robotic patient-side system 152. Situated approximately at a central location onbase 172, thecart column 170 includes aprotective cover 180 that protects components of a counterbalance subsystem and a braking subsystem (described below) from contaminants. - Excluding a
monitor arm 154, each robotic surgical arm 158 is used to controlrobotic instruments 101A-101C. Moreover, each robotic surgical arm 158 is coupled to a set-uparm 156 that is in turn coupled to acarriage housing 190 in one embodiment of the invention, as described below with reference toFIG. 3 . The one or more robotic surgical arms 158 are each supported by their respective set-uparm 156, as is illustrated inFIG. 1B . - The robotic
surgical arms 158A-158D may each include one or more displacement transducers, orientational sensors, and/orpositional sensors 185 to generate raw uncorrected kinematics data, kinematics datum, and/or kinematics information to assist in acquisition and tracking of robotic instruments. The robotic instruments may also include a displacement transducer, a positional sensor, and/ororientation sensor 186 in some embodiments of the invention. Moreover, one or more robotic instruments may include amarker 189 to assist in acquisition and tracking of robotic instruments. - Referring now to
FIG. 2 , the stereoendoscopic camera 101C includes anendoscope 202 for insertion into a patient, acamera head 204, a left image forming device (e.g., a charge coupled device (CCD)) 206L, arightimage forming device 206R, a left camera control unit (CCU) 208L, and a right camera control unit (CCU) 208R coupled together as shown. The stereoendoscopic camera 101C generates aleft video channel 220L and aright video channel 220R of frames of images of the surgical site coupled to astereo display device 164 through avideo board 218. To initially synchronize left and right frames of data, a lock reference signal is coupled between the left and rightcamera control units camera control unit 208L may also generates the lock reference signal so that the right video channel synchronizes to the left video channel. - The
stereo display 164 includes aleft monitor 230L and aright monitor 230R. As discussed further herein with reference toFIG. 4 , the viewfinders or monitors 230L,230R may be provided by aleft display device 402L and aright display device 402R, respectively. The stereo images may be provided in color by a pair ofcolor display devices - Additional details of a stereo endoscopic camera and a stereo display may be found in U.S. Pat. No. 5,577,991 entitled “Three Dimensional Vision Endoscope with Position Adjustment Means for Imaging Device and Visual Field Mask” filed on Jul. 7, 1995 by Akui et al; U.S. Pat. No. 6,139,490 entitled “Stereoscopic Endoscope with Virtual Reality Viewing” filed on Nov. 10, 1997 by Breidenthal et al; and U.S. Pat. No. 6,720,988 entitled “Stereo Imaging System and Method for use in Telerobotic Systems” filed on Aug. 20, 1999 by Gere et al.; all of which are incorporated herein by reference. Stereo images of a surgical site may be captured by other types of endoscopic devices and cameras with different structures. For example, a single optical channel may be used with a pair of spatially offset sensors to capture stereo images of the surgical site.
- Referring now to
FIG. 3 , a perspective view of the robotic surgicalmaster control console 150 is illustrated. Themaster control console 150 of the roboticsurgical system 100 may include thecomputer 151, a binocular orstereo viewer 312, anarm support 314, a pair of control input wrists and control input arms in aworkspace 316, foot pedals 318 (includingfoot pedals 318A-318B), and aviewing sensor 320. Themaster control console 150 may further include a tool tracking and image guidedsurgery system 351 coupled to thecomputer 151 for providing the tool images and tissue images overlaid on the visible surgical site images. Alternatively, the tool tracking and image guidedsurgery system 351 may be located elsewhere in the roboticsurgical system 100, such as thepatient side cart 152 or a separate computer system. - The
stereo viewer 312 has two displays where stereo three-dimensional images of the surgical site may be viewed to perform minimally invasive surgery. When using the master control console, the operator O typically sits in a chair, moves his or her head into alignment with thestereo viewer 312 to view the three-dimensional annotated images of the surgical site. To ensure that the operator is viewing the surgical site when controlling therobotic instruments 101, themaster control console 150 may include theviewing sensor 320 disposed adjacent thebinocular display 312. When the system operator aligns his or her eyes with the binocular eye pieces of thedisplay 312 to view a stereoscopic image of the surgical worksite, the operator's head sets off theviewing sensor 320 to enable the control of therobotic instruments 101. When the operator's head is removed the area of thedisplay 312, theviewing sensor 320 can disable or stop generating new control signals in response to movements of the touch sensitive handles in order to hold the state of the robotic instruments. Alternatively, the processing required for tool tracking and image guided surgery may be entirely performed usingcomputer 151 given a sufficiently capable computing platform. - The
arm support 314 can be used to rest the elbows or forearms of the operator O (typically a surgeon) while gripping touch sensitive handles of the control input wrists, one in each hand, in theworkspace 316 to generate control signals. The touch sensitive handles are positioned in theworkspace 316 disposed beyond thearm support 314 and below theviewer 312. This allows the touch sensitive handles to be moved easily in thecontrol space 316 in both position and orientation to generate control signals. Additionally, the operator O can use his feet to control the foot-pedals 318 to change the configuration of the surgical system and generate additional control signals to control therobotic instruments 101 as well as the endoscopic camera. - The
computer 151 may include one ormore microprocessors 302 to execute instructions and a storage device 304 to store software with executable instructions that may be used to generate control signals to control the roboticsurgical system 100. Thecomputer 151 with itsmicroprocessors 302 interprets movements and actuation of the touch sensitive handles (and other inputs from the operator O or other personnel) to generate control signals to control the roboticsurgical instruments 101 in the surgical worksite. In one embodiment of the invention, thecomputer 151 and thestereo viewer 312 map the surgical worksite into thecontroller workspace 316 so it feels and appears to the operator that the touch sensitive handles are working over the surgical worksite. Thecomputer 151 may couple to the tool tracking and image guidedsurgery system 351 to execute software and perform computations for the elements of the image guided surgery unit. - The tool tracking system described herein may be considered as operating in an open-loop fashion if the surgeon operating the master console is not considered part of the system. If the robotic instrument is to be automatically controlled with the tool tracking system, such as in visual serving systems used to control the pose of a robot's end-effector using visual information extracted from images, the tool tracking system may be considered to be operating in a closed visual-feedback loop.
- Referring now to
FIG. 4 , a perspective view of thestereo viewer 312 of themaster control console 150 is illustrated. To provide a three-dimensional perspective, theviewer 312 includes stereo images for each eye including aleft image 400L and a right image 400R of the surgical site including anyrobotic instruments 400 respectively in aleft viewfinder 401L and aright viewfinder 401R. Theimages 400L and 400R in the viewfinders may be provided by aleft display device 402L and aright display device 402R, respectively. Thedisplay devices color display devices - In the stereo viewer, three dimensional maps (a depth map with respect to a camera coordinate system or equivalently a surface map of an object with respect to its local coordinate system is a plurality of three-dimensional points to illustrate a surface in three dimensions) of the anatomy, derived from alternative imaging modalities (e.g. CT scan, XRAY, or MRI), may also be provided to a surgeon by overlaying them onto the video images of the surgical site. In the
right viewfinder 401R, aright image 410R rendered from a three dimensional map such as from a CT scan, may be merged onto or overlaid on the right image 400R being displayed by thedisplay device 402R. In theleft viewfinder 401L, a renderedleft image 410L is merged into or overlaid on theleft image 400L of the surgical site provided by thedisplay device 402L. In this manner, a stereo image may be displayed to map out organ location or tissue location information in the surgical site to the operator O in the control of the robotic instruments in the surgical site, augmenting the operator's view of the surgical site with information that may not be directly available or visible by an endoscopic camera in the surgical site. - While a stereo
video endoscopic camera 101C has been shown and described, a mono video endoscopic camera generating a single video channel of frames of images of the surgical site may also be used in a number of embodiments of the invention. Rendered images can also be overlaid onto the frames of images of the single video channel. - Tool tracking has a number of applications in robotic surgical systems. One illustrative application of tool tracking is to automatically control the motion of the endoscopic camera so that a surgeon automatically views regions of interest in a surgical site, thus freeing the surgeon from the camera control task. For example, robotic instruments are tracked so that the endoscopic camera is centered in the field of view of the surgical site. Another illustrative application for accurate tool tracking may be used to move a robotic instrument to reach a surgical target (e.g., a tumor) either automatically or by a surgeon. For a target such as a tumor that may be occluded, other real-time imaging modalities, such as ultra-sound or pre-scanned images, may be used with real-time tool tracking to move a robotic instrument to the tumor and remove it. Other illustrative applications of tool tracking include a graphic user interface (GUI) that facilities the entrance and re-entrance of robotic instrument during surgery. Tool tracking is very useful when robotic instruments are not in the field of view of the endoscopic camera or are otherwise obscured in the field of view of the camera. In such scenarios, robotic kinematics provides information through the proposed state-space model.
- In one embodiment of the invention, a tool tracking system and architecture is provided that fully integrates kinematics information and visual information for robust and accurate tool tracking performance. Results of tool localization and tracking are made accurate and reliable by adaptively combining together robust kinematics information and accurate geometric information derived from video. The tool tracking system performs locating (determining absolute locations/poses with stereo video), tracking (integrating visual and kinematics) and predicting (kinematics while the tool or a portion thereof is not visible) functions.
- Technical capabilities in the tool tracking system include an analysis-by-synthesis for image matching and a sequential Bayesian approach which fuses together visual and kinematic information.
- An analysis-by-synthesis capability makes it possible to explore the prior information that is of concern, such as information about the tools and not the tissue and surrounding environment. The basic procedure in analysis-by-synthesis is to synthesize an image based on the model (geometry and texture) and the current pose (position/location and orientation) of a tool and then compare it against real images. The error between the real and synthesized images is the driving force for better estimation of tool pose. To make this approach more robust, e.g., handling varying illumination and the natural wear of a tool; appearance-based learning may be applied to update the model for a specific tool. Alternatively, matching may be performed using features, such as edges and/or corners, that are more robust to lighting variation.
- To obtain the absolute pose (location and orientation), stereo imaging may be used along with the analysis-by-synthesis (ABS) techniques. Instead of just a feature point based stereo that may require markers, a stereo approach may be provided based on the tool (or some of its parts). To further improve the robustness, stereo and ABS techniques may be applied to a sequence of images (e.g., the same location and different orientations or different locations and different orientation). Sequence-based matching makes the procedure less vulnerable to local minimum in the process of optimization/estimation.
- A sequential Bayesian approach may be applied to fuse visual information and robot kinematics to efficiently track tools. In this approach, the states provide zero-order kinematics of the tools (e.g. current position and orientation, or pose), and the first-order kinematics of the tools (e.g. translational and angular velocity). Depending upon the complexity of the underlying physical system, higher-order or lower-order state space model may be adopted. For a linear space model, a linear Kalman filtering may be applied when the noise can be approximated as Gaussian. For a nonlinear state space model, an extended Kalman filtering may be used to filter out noise from observations and state dynamics. For a complex state space model that requires a point distribution function (pdf), sequential Monte Carlo method (e.g., particle filtering) can be applied where the point distribution function may be represented by a set of representative samples.
- One challenge in fusing information from the various sources is that the sources should be correctly characterized, for example, in terms of observation noise characteristics. Taking the simplest case of Gaussian noises that can be characterized by means and co-variance, we can estimate the co-variance matrices quite robustly since we know the geometry of the robotic instruments (location and orientation) and their relative positions. For example, observation (e.g., pixel location of a feature point) from a robotic instrument that is under a slanted viewing angle or is occluded by other tools has a large variance.
- Referring now to
FIG. 5A , tool tracking involves determining a pose of arobotic instrument 101 including its position or location (Xt,Yt,Zt) and its orientation in the camera coordinate system as it moves in, around, and out of the surgical site. A full pose description may not only include the location and orientation of a robotic instrument in a three dimensional space but may further include the pose of an end-effector, if any. Positional information or pose as used herein may be used to refer to one or both the location and orientation of a robotic instrument. - A sequence of left video frames 500L within a camera coordinate system and a sequence of right video frames 500R within another camera coordinate system (or one pair of video frames/images from left and rights views) may be used for tool tracking in one embodiment of the invention. Alternatively, a single view with a single sequence of video frames may be used for tool tracking in another embodiment of the invention. Alternatively, a single video frame may be used for tool tracking in yet another embodiment of the invention providing partially corrected pose estimates.
- A
marker 502 on therobotic instrument 101 may be used to assist in the tool tracking if visible or otherwise sensible. In one embodiment of the invention, themarker 502 is a painted marker minimally altering the robotic instruments. In other embodiments of the invention, markerless tool tracking is provided with no modification of the robotic instruments. For example, natural image features of a robotic tool may be detected as natural markers and/or image appearance of the tools and the CAD model of tools may be used to provide tool tracking. - In one embodiment of the invention, video information from an endoscopic camera and kinematics of the robotic arm and robotic instrument are used as cues to determine the pose of the robotic instrument in the surgical site. If the robotic instrument is not visible to the camera, the video information alone is insufficient to determine position but the kinematics adds the missing information to determine robotic instrument pose. Moreover, even if the video information is available, the addition of the kinematics information makes the computation of pose more robust. A tool tracking system is provided based on robot kinematics and vision that requires little to no modifications to the
robotic instruments 101 and the pre-existingsurgical system 100. - Kinematics information provided by the
surgical system 100 may include kinematic position kt P, kinematic orientation kt Ω, kinematic linear velocity {dot over (k)}t P, and kinematic angular velocity {dot over (k)}t Ω of one or morerobotic instruments 101. The kinematics information may be the result of movement of the robotic surgical arm 158, therobotic instrument 101, or both the robotic surgical arm 158 androbotic instrument 101 at a given time. The kinematics information provided by thesurgical system 100 may also include the kinematic position kt P, kinematic orientation kt Ω, kinematic linear velocity {dot over (k)}t P, and kinematic angular velocity {dot over (k)}t Ω of the endoscopic camera to provide a frame of reference. - Referring now to
FIG. 6A , a functional block diagram of a tool tracking architecture and methodology for a surgical system is illustrated in accordance with embodiments of the invention. The main operational stages of tool tracking are illustrated in the middle column ofFIG. 6A . Key technical capabilities associated with tool tracking are illustrated in the left column ofFIG. 6A but foroperational constraints 601. The end results of the tool tracking methodology are illustrated in the right column ofFIG. 6A . - The key technical components of the methodology and architecture may be further categorized into basic building blocks and supporting blocks. The basic building blocks including image matching 609 and a state-
space model 613 that are used to provide efficient tool tracking, each of which are responsive to visual information. The supporting blocks include model-basedsynthesis 611,adaptive fusion 615, and sequence matching 607 to support the implementation of robust and accurate tool tracking. -
Adaptive fusion 615 fully explores prior information that may be available, including prior kinematics information and prior visual information. - In a robotic surgery system, vision information and kinematics information are typically available with known characteristics. Robot kinematics information is usually stable and often accurate but may drift during long periods of time. Vision-based information is very accurate when it can be reliably estimated. Otherwise vision-based information may be very inaccurate.
- In embodiments of the invention, adaptive fusion is used to obtain accurate information fusion from different sources of information as well as similar sources of information. With adaptive fusion, if the vision-based information is known to be accurate then the information fusion is heavily biased towards the vision-based information. If the vision-based information is known to be unreliable or inaccurate, robot kinematics information is used over the vision-based information to generate a more robust fusion of information. While the quality of robot kinematics is typically uniform, the quality of vision information in terms of image matching and 3D post estimation varies a lot. View geometry statistics may be used to determine the reliability and accuracy of video-based information. Adaptive fusion may also be used to obtain accurate information fusion from similar sources of information.
- Model-based synthesis is used herein to generally refer to generation or rendering of a template image for use in subsequent matching operations, and includes full synthesis, geometry only synthesis, and implicit synthesis. Full synthesis, as the name implies, is a complete synthesis of an image of the robotic instrument. For example, robotic instrument images are generated from a computer aided design (CAD) model based on its geometry and texture. Other prior information (the location/orientation of the model), not necessarily accurate, is presented along with the synthesized robotic instrument images for image matching 609. Geometry-only synthesis is the case where the geometry of the robotic instrument is used to synthesize geometry-only images (e.g., edge images). Texture of the model is not used in geometry-only synthesis. Implicit synthesis is the case where images are not actually synthesized. Instead the model (either geometry or texture or both) is implicitly used to perform image matching. For example, the geometric properties (e.g., width, length, shape) of a marker and the geometric relationship among them (e.g., markers forming a line) when available in a marker-based tracking system may be used to improve image matching.
- In one embodiment of the invention, sequence matching is where objects or features in a sequence of images captured from one camera view are matched against objects or features in a sequence of images captured from a different camera view. In another embodiment of the invention, sequence matching is where objects or features in a sequence of images from a camera view are matched against objects or features in a sequence of synthesized images.
- There are two main operational stages in a tool tracking system. The two main operational stages are
tool acquisition 604 and tool tracking 606. Tool acquisition may also be referred to as localization herein. - The goal of the
tool acquisition stage 604 is to obtain the absolute pose information (location and orientation) of the one or more robotic instruments within the field of view of one or more cameras, such as the stereoendoscopic camera 101C. Thetool acquisition stage 604 performs a locatingfunction 614 resulting in the location and orientation of the tool. - The goal of the
tool tracking stage 606 is to dynamically update the absolute pose (location and orientation) of a moving robotic instrument. Thetool tracking stage 606 may perform a full-tracking function 616 or a kinematics-tracking (pose prediction)function 618 respectively resulting in either a full-tracking state when both visual information and robot kinematics information are available or a kinematics-tracking state when visual information is not utilized (e.g., tool outside the field of view or occluded). - The mode/stage of the tool tracking system changes from tool acquisition to tool tracking after the tool is initially located within the field of view. Provided that the tool remains in the field of view, the tool tracking system may remain in the tool tracking mode/stage. However, the tool tracking system may change from a tool tracking mode/stage into a tool acquisition mode/stage if the tool is removed from the field of view and then returns into the field of view. The tool tracking system may optionally begin operation with an initialization procedure if there is only a single tool in the field of view. If additional tools are to be tracked, the optional initialization procedure may be skipped as other tools have been located and tracked. If the tools have no markers, the optional initialization procedure may involve tools moving around in order to obtain a robust localization via sequence matching.
- The methodology of the tool tracking is now further described with reference to
FIG. 6A which starts atblock 600 and goes to block 602. - At
block 602, an optional initialization of the tool tracking system may occur. Mono orstereo video 603 may be used in the tool tracking system and is initialized to begin generation of digital video frames of image data of the surgical site.Kinematics information 605 may also be used in the tool tracking system during initialization to form an initial pose of the robotic instrument. Thekinematics information 605 may include positional information, including angular or linear information for example, from sensors located at various places along a robotic arm and the robotic instrument. Thekinematics information 605 may be for both the endoscopic camera and robotic instruments such that the relationship between positional information for the robotic instruments and the camera may be determined. - Initialization begins with a single tool in the field of view without any occlusions. The system may be initialized for additional robotic instruments in the field of view. If tools have already been located and tracked and a new tool is being added, the new tool can be initialized by placing it into the field of view with the previously located and tracked tools. If no tool has been located and tracked, each tool may be initialized by placing it within the field of view with all other tools outside the field of view. The robotic instrument being initialized may be placed in the center of the surgical site for optimal estimation across the whole space or as close to stereo
endoscopic camera 101C as possible that will allow for accurate stereo computation. With a markerless system, the robotic instrument may be moved and rotated for reliable sequence matching. - At
block 604, the tool tracking system enters a tool acquisition stage/mode in the surgical site.FIG. 9B graphically illustrates the tool acquisition stage in a surgical site.Stereo video images FIG. 5A , are captured by theendoscopic camera 101C, including one or morerobotic instruments 101 in the surgical site. Stereo video may be used to obtain an absolute initial pose of the one or morerobotic instruments 101 in one embodiment of the invention. In another embodiment of the invention, mono video may be used with kinematics information to estimate absolute initial pose (position and orientation) of the one or morerobotic instruments 101. The one or morerobotic instruments 101 may include paintedmarkers 502 to assist in tool acquisition and tool tracking in the surgical site. The tool acquisition stage performs a locatingfunction 614 resulting in the initial pose of the one or morerobotic instruments 101 in the surgical site. - After the robotic instruments have been acquired, the methodology goes to block 606.
- At
block 606, the tool tracking system enters a tool tracking mode or stage in the surgical site. The goal of tool tracking is to update the absolute pose information (location and orientation) based on incremental and/or partial information (visual and robot kinematics). In thetool tracking stage 606, the tool tracking system is at a full-trackingstate 616 when visual and kinematics information is available. If a robotic instrument is not visible (e.g., tools inside an organ or occluded by other tools) in the surgical site, the tool tracking system is at a kinematics-trackingstate 618 for estimating tool pose. - The tool tracking system may transition from tool tracking 606 and return to
tool acquisition 604 if a tracked tool gets out of field of view and then comes back into the field of view of the camera. - Referring now to
FIG. 6B , a flow chart of a tool tracking library and its application is illustrated. Atool tracking application 650 is executed by asystem 351 of the roboticsurgical system 100. Thevideo board 218 illustrated inFIG. 2 may be a part of theIGS system 351 in order to receive the video images from the endoscopic camera over the surgical site. A kinematics application programming interface (API) 660 provides a software interface to receive the raw kinematics data from thesurgical system 100. Thekinematics API 660 couples the kinematics information to thetool tracking application 650 and atool tracking library 652. Theraw kinematics data 680 is received by anAPI streamer thread 658 which provides the physical interface to a communication channel (for example, fiber optic cable or Ethernet, and may buffer the raw kinematics data by storing it into a memory, a hard disk, or other data storage device). Thetool tracking library 652 may issue data requests to theAPI streamer thread 658. - A
video capture thread 656 is coupled to the endoscopic camera to receive the rawendoscopic video feed 670. Theraw video 670 may be mono video of a single channel or stereo video of left and right channels. Thevideo capture thread 656 may buffer the raw video data by storing it into a frame buffer memory, a hard disk, or other data storage device. A video application programming interface (API) 659 provides the software interface to receive the raw video data from the surgical system into the tool tracking system. Thetool tracking library 652 may issue data requests to thevideo capture thread 656. - The
tool tracking library 652 contains the core functionality of combining kinematics (through kinematics API 660) and video (through video API 659) for accurate tracking of tools. The library also provides application program interface so it can be invoked in a certain way by a customer-designedtool tracking application 650 - In response to the video data and the kinematics data, the
tool tracking library 652 generates corrected kinematics data for the pose of a robotic instrument. The raw kinematics data is corrected for orientation and position of the tools. The corrected kinematics data may be used in a number of applications, such as image guided surgery. - As shown in
FIG. 6B , the speed ofraw kinematics 680 may be 100 to 200 Hertz (Hz) and the speed ofraw video 670 may be 30 Hz to 60 hz and the speed of tool tracking maybe even slower. However, the speed of the correctedkinematics 690 should be substantially similar to the speed of theraw kinematics 680 for medical applications. To maintain the speed in the kinematics information, the raw kinematics may be passed through. A correction matrix (rotation and translation) may then be used to correct the raw kinematics information from the tool tracking library. Alternatively the correctedkinematics 690 may be directly output from thetool tracking library 652 where a correction matrix is applied to the raw kinematics. Either way is feasible because the correction matrix corrects the bias in the raw kinematics and the bias changes slowly, for example, slower than 1 Hz. - Reference is now made to
FIGS. 6A , 7, and 8.FIG. 6A , described previously, illustrates a functional block diagram including operational stages of a tool tracking system.FIG. 7 is a block diagram illustrating the challenges of performing tool tracking.FIG. 8 graphically illustrates a functional block diagram of atool tracking system 800. - Referring now to
FIG. 8 , thetool tracking system 800 adaptively fuses visual information and robot kinematics in order to achieve robust, accurate and efficient tool tracking. The unknown full pose of a robotic instrument, at a time instant t, is represented as a state st 805B in a Bayesian state-space model 802. The state-space model 802 may use a plurality of posedstates 805A-805C to perform tool tracking in the surgical site. The state-space model 802 may generate the correctedkinematics information 690 of the robotic instrument. A CAD tool model 804 (geometry only or both geometry and texture) is used for synthesizing (explicitly or implicitly) an image under a given pose (i.e. state). - For updating the state information of a robotic instrument, the relative robot kinematics {dot over (k)}t 605 (where the dot above the k being used to represent that the relative or first-derivative measurements of the kinematics information) between time instances t to t+1 can be coupled into the state-
space model 802.Visual information 603 from captured images may be amplified and analyzed by an amplifier/filter 808 to control the influence ofvisual feedback 809 on the fusion of visual information and kinematics information. The amplifier/filter 808 generally implements how view geometry statistics are applied for adaptive fusion. Ifstereo images 803 are available, thespatial constraints 807 between left and right image pairs may also be explicitly or implicitly explored to assist in tool tracking. - As illustrated in
FIG. 7 , there are natural challenges and technical challenges to provide tool tracking. The natural challenges are those imposed by realistic operational scenarios. The technical challenges are those caused by proposed tool tracking algorithms when facing natural challenges. The natural challenges for example include cluttering andocclusion 702, illumination variation andimage appearance change 703, andviewing singularity 704. The technical challenges includeimage segmentation 710 and matchingambiguity 712, for example. - The natural challenge of illumination and
image appearance 703 is where the same scene changes dramatically along with the motion of directional light sources. For endoscopic operations, the image intensity of the same object can be different, depending on the distance of the object from the lighting source and the angle between the lighting source and local surface normal. This makes image-based processing less reliable. In addition, specularities from organs, blood that under directional endo-illumination make image processing more challenging. - The natural challenge of viewing
singularity 704 may occur when three dimensional geometry information is derived from two dimensional images. Three dimensional geometry information derived from two dimensional images is not reliable when the two dimensional projection of a three dimensional object is degenerated. For example, a three dimensional cylindrical tool is projected onto an image plane as a circle. - The natural challenge of scene cluttering and
occlusion 702 is the case where there could be more than one robotic instrument in the field of view. Additionally, the robotic instruments may be partially or fully submerged with complex and dynamic background of organ tissues, blood and smoke caused by electro-dissection. - As previously mentioned, the technical challenges include
image segmentation 710 and matchingambiguity 712. Moreover while efficiency is of concern, a big technical challenge for tool tracking may be reliability and accuracy under realistic situations. - Consider now for example,
pure image segmentation 710, i.e., segmentation of tools from a 2D image only, is a challenging task when the background is cluttered and/or objects are occluded. To handle this particular technical challenge, prior information is explored as a known robotic instrument is being tracked. More specifically, model basedsynthesis techniques 722 may be used. With model basedsynthesis 722, a CAD model of a robotic instrument may be used to render a clean tool image as a pattern to match against or search within a limited region constrained by the pose information of tool. As a result, pure image segmentation from the real images is avoided. Because the states of all robotic instruments are tracked, mutual occlusions of all these robotic instruments can be calculated thereby making image matching more reliable. - Another technical challenge in tool tracking, especially markerless tool tracking, is the matching ambiguity of a pair of
images 712, either between left and right images or between real and synthesized images. Fundamentally, many areas in an image look alike and non-corresponding areas of two images may appear to be more alike than two corresponding areas (for example, due to illumination variations), making region-based matching ambiguous. To reduce such ambiguity, sequence matching 728 may be applied where a sequence of images will be matched against another sequence of images. Such a method is useful when we use robust and accurate relative kinematics information {dot over (k)}t. - For example, consider a sequence of three
real images 811A-811C [It−1, It, It+1] and threecorresponding states 805A-805C [st−1, st, st+1]. For each state, one image can be rendered such that a sequence of three synthesized images [IS t−1, IS t, IS t+1] may be formed. Under a regular analysis-by-synthesis scheme, the real images It and the synthesized images IS t are compared. The difference determined by the comparison is used to update the corresponding state st. For a three-state sequence, three independent computations are used to update three states. Now if we use sequence matching 728 for the same three-state sequence, the situation changes significantly. For ease of explanation, suppose that the perfect or error-less relative kinematics information is a two-sequence [{dot over (k)}t−1,{dot over (k)}t] of kinematics information. This suggests that there is only one unknown (any one of the three states) for the three-state sequence [st−1, st, st+1] because st=st=st−1+{dot over (k)}t−1. With one known state of kinematic information, the respective sequence of threeimages 811A-811C [It−1, It, It+1] and the respective sequence of three synthesized images [IS t−1, IS y, IS t+1] may be used to determine the unknown states. That is, if we know any one of the three states in the three-state sequence [st−1, st, st+1], we can obtain other missing states through perfect or error-less relative kinematics. - The sequence of five
real images 811A-811C [It−1, It, It+1] and the sequence of three synthesized images [IS t−1, IS t, IS t+1] are then compared to determine a difference to update the current state st so that its underlying kinematics information is more accurate and reliable for use in tool acquisition and tool tracking. Thus, sequence matching 728 can provide a more robust and more reliable matching as the number of unknowns is reduced and the same number of observations (real images) are kept. - Additionally,
appearance learning techniques 724 may be used to handle image or appearance changes 703 such as from illumination variations and natural wear of a tool, for example. Generally, appearance learning techniques handle appearance changes by training the tool tracking system on image samples of the same tool under different viewing conditions. Appearance learning techniques have been used extensively in object tracking to handle appearance change due to illumination variations. For example, parametric models have been built to handle illumination variations. Appearance learning techniques are further illustrated herein with reference toFIG. 14 with the use of face images instead of tool images. - Moreover,
adaptive fusion techniques 726 may be used to handle the challenges of singularity of viewing geometry orviewing singularity 704. The technique of adaptive fusion is used to explore the available pose information, i.e., predicted state (before correction) when feeding geometric information derived from video into the Bayesian state-space model 802. More specifically, video-derived information has much less weight when fused with robot kinematics information under such conditions. In a Bayesian state-space model 802, this manifests itself as large noise variance in the observation equation. - Adaptive fusion may be used to handle the challenges of singularity of viewing geometry in order to provide robust and accurate kinematics information of the tools in a surgical, medical, or other type of robotic system.
- Pure image segmentation may be used by a tool tracking algorithm to localize tools. Pure image segmentation of tools from a two dimensional image is straightforward if the tools have distinctive features, such as color marks that may be used to identify a tool. However, operational conditions may make pure image segmentation techniques difficult if not impossible to perform.
- Referring now to
FIG. 9A , an image has atool 901 hidden by anocclusion 902. Theocclusion 902 is so severe that it breaks key steps (e.g., color- and/or shape-based analysis) of pure image segmentation of the image such that thetool 901 cannot be found therein. The tool shape 901S is substantially covered over by the occlusion shape 902S in the image illustrated inFIG. 9A . In the case of markerless tool tracking where the tools have no markings, an occlusion can only make it more difficult for pure image segmentation techniques to localize a tool. - Referring now to
FIG. 9B , the image of thetool 901 is again hidden by anocclusion 902. Techniques of sequence matching and/or model-based-synthesis matching may be used to localize the robotic instruments instead of pure image segmentation. Sequence matching was briefly discussed previously. Model based synthesis uses a priori knowledge regarding kinematics and appearance that may be available for the tools that are being tracked. - With model based synthesis, a
CAD model 904A of thetool 901 is used to synthesize an image of the tool given the known or hypothesized pose information for the tool. The pose information for the tool may be determined from kinematics information or otherwise and a posedsynthesized image 904B may then be generated. The posedsynthesized image 904B of thetool 901 may then be used to perform image matching or an image search within the overall image of a surgical site to find the location of thetool 901 therein even though it may be partially occluded. This technique of tool localization may generally be referred to herein as an analysis-by-synthesis approach. Using the synthesized 904B image as a pattern to search for thetool 901 within an image of the surgical site helps overcome the difficulty of anocclusion 902 that may cover thetool 901. Tool image fragments 901′ left over after theocclusion 902 is subtracted from the tool image is sufficient to use to determine tool localization. However if theocclusion 902 completely covers over thetool 901, image analysis alone cannot localize tools. - Alternatively in another embodiment of the invention, image segmentation may be guided by exploring the available prior kinematics and image information. That is, image segmentation may be constrained to be performed within a limited region of the image of the surgical site based on rough pose information of the tool in response to the prior robot kinematics and the
CAD model 904A of thetool 901. This technique of tool localization may generally be referred to herein as aided image segmentation in contrast to pure image segmentation that has no constraints. - Referring now to
FIG. 9C , image synthesis (also referred to herein as model-based synthesis) and image analysis/search are key steps in using analysis-by-synthesis methods for tool localization and tool tracking. Theimage synthesis 911 and image analysis/search 915 processes may be repeatedly performed in an iterative optimization approach to find the best tool pose parameter in response to a givencost function CF 913. With an iterative optimization approach, an initial pose hypothesis may be formulated to generate the initial synthesized model tool for computation of an initial cost function. Thecost function CF 913 is a function of what corresponding features 912,914 are used for matching and how an image is synthesized duringimage synthesis 911. For example, a synthesize edge image 904S of the tool may be synthesized duringimage synthesis 911 in response to the CAD geometry of theCAD model 904A of the tool. Alternatively, a synthesizedregular image 904B of the tool may be synthesized duringimage synthesis 911 in response to the CAD geometry and CAD texture of theCAD model 904A of the tool. The synthesize edge image 904S of the tool may be used to perform image matching with edges in avideo image 910A of the tool. The synthesizedregular image 904B of the tool may be used to perform image matching with aregular video image 910B of the tool. If the appearance of the tool has changed in the images, e.g., 910B, appearance learning may be used to augment the analysis-by-synthesis process for tool localization and tool tracking. Note that an edge image, such as illustrate invideo image 910A, is typically robust against lighting variations. - With a synthesized tool image Is being synthesized in response to a given tool pose in the camera coordinate system and the CAD model M (with geometry Mg and texture Mt); the synthesis process may be expressed by an equation as
-
I s [x]=L(M t[Φ;(P,Ω,M g)]) (Eq. 1) - where x=[x, y] is the image coordinate and Φ is the homogeneous camera geometric projection from three dimensions (3D) into two dimensions (2D). Thus, the model texture Mt can be mapped to the coordinate of image Is[x] as a function of the homogeneous camera geometric projection Φ and a combination of tool pose (position P and orientation Ω of the tool), and the geometry Mg of the tool model. For presentation clarity, we omit the nonlinear mapping step from the 2D homogeneous coordinates [xw, yw, w] after Φ projection to 2D in-homogeneous image coordinates
-
ψ:[x,y]=[x w /w,y w /w]. (Eq. 1A) - In an example image synthesis pipeline, the model will be decomposed into triangles, the 3D vertex coordinates of which will be described in a coordinate system attached to the model. The model coordinates will first be transformed to a world coordinate system, before being projected to a 2D display coordinate system by applying the camera model. Once in the 2D display coordinate system, each triangle will be rasterized. The synthesis of the final per-pixel color values may be computed via interpolation of color specified on a per-vertex basis, texture mapping and filtering, and the application of lighting models. (Reference: Computer Graphics: Principals and Practice, by James D. Foley, Andries van Dam, et. al, Addison-Wesley Professional; 2 edition, Aug. 4, 1995, ISBN: 978-0201848403).
- The function L is a mapping function that maps the model texture Mt into the real image/appearance Is[x] because the real image varies with lighting conditions and other factors, such as occlusions.
- The tool pose may be represented by the position P=[PX,PY,PZ]T of a chosen
reference point 931R (e.g., the control point before the tool wrist) and the orientation Ω of its local coordinatesystem 920 originated in thereference point 931R with respect to the camera coordinatesystem 921. Camera coordinates of a3D point 931P on the tool that maps to x may be represented by [X,Y,Z]T. A local coordinate of the3D point 931P on the tool that is internal to the tool model may be represented as [XM,YM,ZM]T. A transformation T{P,Ω} of the local coordinate of the3D point 931P on the tool to the camera coordinate of the tool as a function of the tool pose may be written as -
[X,Y,Z,1]T =T {P,Ω} [X M ,Y M ,Z M,1]T (Eq. 2) - where T{P,Ω} is a four by four 3D-to-3D rigid transformation matrix that can be further decomposed into translational and rotational parts.
- After image synthesis of the synthesized tool image Is, an image analysis or image search is performed to find the best estimate of the tool pose. Mathematically this is an optimization problem that may be written in equation form as
-
- where C is a cost function of comparing images. The synthesized tool image Is may be repeatedly generated in an iterative manner using updated pose information so that the cost function C of comparing the synthesized tool image Is with the video images of the surgical site are minimized and the tool pose is optimized.
- One of the simplest cost functions is a sum of squared differences (SSD) that may be used to compare the synthesized tool image Is with the video images of the surgical site. However even though an SSD is a simple cost function, it is nonlinear (e.g., higher than quadratic) in terms of the pose parameters due to the camera geometric projection Φ that is nonlinear and the mapping function L to map model texture to real image/appearance that varies with lighting conditions that may be non-linear. Minimizing a nonlinear cost function C is a complex optimization problem.
- Different strategies can be applied to solve a problem of minimizing a nonlinear cost function C. For example, random optimization methods may be used to solve and minimize a nonlinear cost function C problem in order to avoid an exhaustive parameter search. On the other hand, a quadratic approximation of the cost function may be use to iteratively solve the nonlinear problem.
- In one embodiment of the invention, the complex optimization problem may be broken up into two different steps to more efficiently minimize the cost function C. The first step entails performing an image matching where the raw image pixels or extracted image features of I are used for matching against those of the respective synthesized tool image Is. The second step involves performing a geometry-only optimization in response to the result of the image matching between the raw image pixels or extracted image features and corresponding ones from the respective synthesized tool image Is. Mathematically, these two steps to solve the optimization problem of Eq. 3 may be formulated into the following two equations:
-
- and the best T{P,Ω} is determined as
-
- Eq. 4 represent the step of finding the corresponding 2D feature points xm from 1 and 3D points Xm on the tool via image matching of Is and I. Eq. 5 represents the geometry-only optimization where optimal 3D-2D mapping T{P,Ω} can be found given the matched 2D-3D pairs. The function f( ) is a nonlinear function in the following form ψ(ΦT{P,Ω}).
- In cases where the initial pose hypothesis is not close to the true pose or in case where it is desirable to obtain a very accurate pose estimate, the foregoing steps to solve the optimization problem (Eqs. 4 and 5) combined with the synthesis step (Eq. 1) can be repeated in an iterative procedure.
- The image matching and analysis-by-synthesis processes may be incorporated into a sequential framework for fusion of vision and kinematics information to obtain more accurate positional information of a tool than would otherwise be available from each alone.
- Referring now to
FIG. 14 , a diagram illustrating appearance learning of objects within an image is illustrated. As discussed previously, appearance learning techniques may be used to handle image and appearance changes, such as from illumination variations and natural wear of a tool, for example. The appearance variations due to changes in illumination may exhibit illumination subspace/cone phenomena or spherical harmonics for example. Appearance learning techniques generally train the tool tracking system on image samples of the same tool under different viewing conditions. Pose specific learning techniques may be used as well as clustering or manifold learning techniques may be used to train the tool tracking system over a large number of samples. - In
FIG. 14 , basis images forillumination variations synthesized images 1402A-1402B which are more closely matched to the respective real images 1045A-1405B that may be captured under different lighting conditions. - Appearance learning techniques have been used extensively in object tracking to handle appearance change due to illumination variations (reference: G. Hager and P. Belhumeur, “Efficient Region Tracking with Parametric Models of Geometry and Illumination,” IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 20, pp. 1025-1039, 1998). For example, parametric models have been built to handle illumination variations (reference: H. Murase, S. Nayar, “Learning and Recognition of 3-D Objects from Brightness Images,” Proc. AAAI Fall Symposium, Machine Learning in Computer Vision, pp. 25-29, 1993.
- After image matching, such as through analysis-by-synthesis for example, the next step to obtain more accurate positional information is the fusion of image positional information and kinematics positional information of the robotic instruments. In general, the purpose of information fusion is to provide more robust and/or more accurate positional information for the robotic instruments in the surgical site such that tool tracking information may be applied in various ways to obtain accurate results, e.g., measurements of certain physical entities within a surgical site. Key to successfully fusing information together from similar sources or from different sources is determining how to adjust the contributions of each to the fusion. The contribution of sources to the information fusion may be adjusted in different ways, such as by a winner-take-all or a weighted averaging method, for example.
- Ideally, all sources of information should be fused together so that the information fusion constantly provides the best accuracy and the most robust tool tracking. However due to the dynamic nature of systems, there typically is a trade-off between accuracy and robustness in information fusion. As a result of the tradeoffs, the typical practical approaches to information fusion tend to have compromised results.
- Referring now to
FIGS. 10A-10B , a state-space model is now described to adaptively fuse together robot kinematics information and vision-based information. Both rawrobotic kinematics information 1010 of the robotic instrument and vision-basedinformation 1011 can be used to generate thestate variables 1000A-1000D. Atool model 804 may be used to synthesize 611 the synthesizedimages 810 in response to thestate variables 1000A-1000D. Animage analysis 806, 609 is performed comparing the synthesizedimages 810 of the robotic instrument with the observedimages 1011. - Some real-world data analysis tasks involve estimating unknown quantities from given observations. Moreover, a priori knowledge about phenomenon of a number of applications may be available to allow us to formulate Bayesian models involving probability and statistics.
- The unknown quantities of information fusion at corresponding time instances can be defined as
state variables 1000A-1000D (of the system) and a Markov chain model can be assumed among states at different time instances, then a state-space model may be formed. The state-space model may include 1) a dynamic/state model that relatesstate variables 1000A-1000D at different time instances (t−1, t, t+1, t+2) and 2) an observation model that relatesstate variables S 1000A-1000D toobservations O 1002A -
- where D and H are the dynamic matrix and observation matrix respectively. vt and wt are respectively the dynamic noise and the observation noise that have Gaussian distributions N(μs,Cd) and N(μo,Co) respectively. Cd and Co are covariance matrices for dynamic model and observation model respectively.
- State-space models haves been used in many disciplines of science and engineering, and may be referred to by different names such as Bayesian filtering, optimal filtering, stochastic filtering, on-line inference, for example.
- If the state-space model is linear and the modeling noises of the system are Gaussian, then a substantially exact analytic expression can be derived to solve the on-line estimation problem for the state variables of information fusion. In this case, the analytic expression that is well-known and widely used is a Kalman filter (see R. E. Kalman, “A New Approach to Linear Filtering and Prediction Problems,” Trans of the ASM—Journal of Basic engineering, Vol. 82, pp. 35-45, 1960.). In the case of a non-linear system and Gaussian noise, Extended Kalman Filtering (EKF) (reference: Greg Welch and Gary Bishop, An Introduction to Kalman Filter, Dept. Computer Science Tech Report 95-041, University of North Carolina, updated 2006.) can be applied where the non-linear system is approximated by linearizing the models (either non-linear dynamic model or non-linear observation model or non-linear both) based on previous estimate and applying Kalman filtering.
- However, for more complex problems that do not obey linear model and Gaussian noise assumption, a set of more general methods are required. The general method for estimating the state variables is often referred to as Sequential Monte Carlo (SMC) methods (reference: J. Liu and J. R. Chen, “Sequential Monte Carlo Methods for Dynamic Systems,” Journal of American Statistical Association, Vol. 93, pp. 1032-1044, 1998, In SMC methods (also referred to as particle filter), states are represented by posterior probability density function (pdf) and sampling techniques are used to generate the posterior probability density function (pdf).
- In a number of embodiments of the invention, a state-space model based approach is used to adaptively fuse robot kinematics and vision information. In the following discussion, s represents a state vector of the
states 1000A-1000D. In practice, s may be a vector version of the matrix T{P,Ω} (Eq. 1), or a collection of point positions [X,Y,Z]T, etc., that are equivalent in theory but may be chosen based on a particular application. - Respective velocity information may be readily added to the state space model by taking the first derivative of the positional information.
- Without any loss of generality, consider for example position P and orientation Ω and first derivatives {dot over (P)} and {dot over (Ω)} of a robotic instrument. Position and velocity may be represented mathematically as
-
- The orientation and angular velocity of the robotic instrument may be represented mathematically using unit quaternion for its minimal-style representation and operation efficiency as
-
- Combining the position and orientation vectors together, the state vector may be represented mathematically in a covariance matrix as follows:
-
- The filter state-space model may be implemented with extended Kalman filtering (simple to implement but maybe not sufficient), unscented Kalman filtering (see S. J. Julier and J. K. Uhlmann. A New Extension of the Kalman Filter to Nonlinear Systems. In Proc. of AeroSense: The 11th Int. Symp. on Aerospace/Defense Sensing, Simulation and Controls., 1997 (easy to implement and maybe sufficient)), or computationally expensive particle filtering.
- Extended Kalman filtering or particle filtering may be used for tool tracking because 1) the dynamic state space model is non-linear due to the quaternion representation, and 2) the observation model is non-linear in the case of using 2D images as observations and linear in the case of using stereo-derived 3D points as observations. In one embodiment of the invention, we adopt the following nonlinear equation to model the transfer of the system from state
-
- where the non-linear part comes from quaternion operation Ωt=q(Ωt−1,{dot over (Ω)}t−1). The Jacobian matrix for Eq. 9 is as follows:
-
- where Q is a skew-symmetric matrix given by
-
- Sets of observations are available to the state space model including robot kinematics (kt P and kt Ω) 1010 and
vision 1011. In one embodiments of the invention, image measurements, the 2D locations of featured points xi, are directly used to construct the observation vector O. Such a choice make the observation equation for the vision part nonlinear due to the perspective projection of a 3D tool onto one 2D image in the case of monocular view, or left and right images in the case of stereo view. - Alternatively, vision-derived (through matching stereo image, for example) 3D points Xs,i, observations due to vision, may be used to construct the observation vector O. In such case, the observation matrix Co consists of two parts: one for kinematics (the leading diagonal sub-matrix of Eq. 12) and one sub-matrix, a covariance matrix Co,v, for vision as follows:
-
- In such a case, the observation equation is linear for the vision part and we can construct the observation covariance matrix Cu,v. For example, we have the following covariance matrix for the case of parallel camera setup (
FIG. 12B ): -
- where the view-geometry variance matrices Co,v,i (for each 3D point Xs,i) are related to 1) the uncertainty (standard deviation) of matching stereo images, 2) the inverse of image resolution (for example, high-definition camera offers better accuracy than standard-definition camera), and 3) the square of the true values of Xs,i.
- In a number of embodiments of the invention, first-order kinematic information (velocity information) ({dot over (k)}t P and {dot over (k)}t Ω) is provided that may provide an extra constraint with respect to the evolution (e.g., changes from state to state) of the dynamic system. In other embodiments of the invention, first-order vision observations may be derived by tracking image points (i.e., temporal image matching) to provide an extra constraint.
- In embodiments of the invention with analysis-by-synthesis methodology, estimated states (e.g., tool poses) may be used to synthesize images for image matching against real images to establish correspondence between 3D points on the tool model and 2D points on the tool image. With visual correspondence, observation vectors may be formed for the vision-based information and the observation equation is non-linear due to the perspective projection of 3D points to 2D points.
- In the case of Gaussian noises and linear systems, Kalman filtering provides a perfect sequential solution to a batch least square optimization problem. because of the Markovian assumption that given the present state future states are conditionally independent of the past states. However, tool tracking in reality is a nonlinear problem and it is often not easy to achieve an accurate solution with a sequential approach such as extended Kalman filtering. To achieve accurate results with extended Kalman filtering given a non-linear system, an iterative bundle adjustment process that combines all observations across multiple states may be used starting from initial results provided by the extended Kalman filtering 802. As a general optimization method, bundle adjustment has wide applications.
- For example, bundle adjustment can be used for sequential matching to be discussed later. To given a specific example in the case of stereo image matching, we have spatial image matching across two views and temporal image matching within each image sequence. By combining all these redundant information together through bundle adjustment, we can achieve improved results compared single stereo pair based results.
- Bundle adjustment, an optimization technique in photogrammetry, refers to the “bundles” of light rays leaving each 3D feature and converging on each camera center which are “adjusted” optimally with respect to both feature and camera positions (reference: C. Salma, C. Theurer, and S. Henrikson, Manual of Photogrammetry, American Society of Photogrammetry, Falls Church, 1980). A bundle adjustment problem is essentially just a large sparse geometric parameter problem that is nonlinear in nature. To implement bundle adjustment iterative procedures may be used, such as the well known Levenberg-Marqudart mathematical algorithm (see “Numerical Recipes in C: The Art of Scientific Computing”, William H. Press, Saul A. Teukolsky, William T. Vetterling, Brian P. Flannery, Second edition, Cambridge University Press, 1992).
- The following describes one example of applying bundle adjustment to improve the pose estimate for the problem of tool tracking.
- In tool tracking, estimates of the poses of a robotic instrument are needed for the state space model. Bundle adjustment is used to optimize the estimated poses of the robotic instrument in a chosen coordinate system, e.g., the camera-centered coordinate system. That is, it is desirous to obtain the relative orientation of the tool with respect to the camera.
- There are a number of ways to use bundle adjustment with the state-space model. There is a batch approach, a window-based approach, and a recursive/sequential approach to apply bundle adjustment techniques to the state space model. With a recursive/sequential approach, bundle adjustment is applied whenever there is a new observation, e.g., a new pair of stereo images (see P. McLauchlan, “The Variable State Dimension Filter applied to Surface-Based Structure from Motion,” CVSSP Tech Report TR-4/99, University of Survey, 1999). Typically, some approximation is made to make the computations efficient. With a window-based approach, bundle adjustment is applied to a short sequence of states and measurements. With a batch approach, bundle adjustment is applied to all available states and measurements. Generally, a batch process approach may be best, followed by a window-based approach, and finally a recursive/sequential approach in implementing bundle adjustment with the state-space model.
- For example, a batch bundle adjustment may be applied at each time instance or at selected time instances based on all the measurements and state estimates that are available from extended Kalman filtering. Applying a batch bundle adjustment in the beginning of time where the state-space model is applied may be preferred because 1) quick convergence to the correct solution to a non-linear optimization problem is desirous from the beginning, and 2) the computation is efficient because there are only a small number of states and observations available.
- Image matching is used for incorporating vision information into the tool tracking system and its state space model. Image matching (Eq. 4) in the image analysis steps (Eqs. 4 and 5) finds the corresponding 2D feature image points and 3D points on the tool. Image matching also finds the corresponding image features between the left and right images from a stereo camera.
- Image matching may be facilitated by sequence matching of a temporal sequence of frames of video images. Alternatively, image matching may be facilitated by using a 2D or 3D model of a tool and a video image. Implementation of the two dimensional image matching may alternately be performed by simple intensity-based SSD (sum of squared difference), feature-based matching (for example, a point, a scale-invariant-feature-transform feature: D. Lowe, “Object recognition from local scale-invariant features,” Proc. Int. Conf. Computer Vision, 1999.), or probabilistic matching. When matching a 2D image with a 3D model synthesized image, more robust features such as edges can be used. In this case, the cost function would be the sum of distance measures from the image edge points to the closest synthesized curves/lines (reference: D. Lowe, “Fitting parameterized three-dimensional models to images,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 13, pp. 441-450.).
- Image matching can be applied in different scenarios, such as temporal matching of images within sequences of images (
FIG. 11A ) or spatial matching of images across two stereo views (FIG. 11C ). In another embodiment of the invention, real images are used to match against synthesized images in the analysis-by-synthesis approach (FIG. 11B ). In another embodiment of the invention, two or more of these image matching techniques (FIGS. 11A-11C ) may be used together to perform image matching. In each of these embodiments of the invention, image matching could be applied to corresponding artificial markers attached to instruments, natural image features or image appearances (e.g., instrument tips). The artificial markers are passive visual markers. - Referring now to
FIG. 11A , temporal image matching of a pair ofvideo images 1101V-1101V′ is illustrated. Thevideo image 1101V of the actual tool 101 (e.g., seeFIG. 1A ) is taken at time t resulting in atool image 101V at time t. Thevideo image 1101V′ of the sameactual tool 101 is taken at a different time,time t+ 1, by the same camera resulting in atool image 101V′ attime t+ 1. During the time the images are captured, the camera may be fixed with respect to the robotic surgical system while thetool 101 may move relative to the robotic surgical system. - Various aspects of the video images of the tool taken at different times may be used to perform image matching. That is one or more of a matching of
markers 1110, a matching ofnatural features 1111, and/or an appearance matching 1112 may be performed. For example,markers 502V on thetool image 101V in thefirst video image 1101V of thetool 101 may be compared with themarkers 502V′ on thetool image 101V′ in thesecond video image 1101V′ to help determine new pose information of theactual tool 101. Besides marker image matching, other information may be used determine pose information of theactual tool 101. - Referring now to
FIG. 11B , synthesis image matching of avideo image 1101V and a synthesized image 1101S is illustrated. Thevideo image 1101V of the actual tool 101 (e.g., seeFIG. 1A ) is taken by a camera resulting in atool image 101V. The synthesized image 1101S of thetool 101 is generated by a computer having prior knowledge of theactual tool 101 resulting in a synthesized tool image 101S. The pose of synthesized tool image 101S in the synthesized image 1101S of thetool 101 attempts to match the pose of the tool represented by thetool image 101V in thevideo image 1101V at a moment in time. - Various aspects of the
video image 1101V and the synthesized image 1101S of the tool may be used to perform image matching. That is, one or more of a matching ofmarkers 1110, a matching ofnatural features 1111, and/or an appearance matching 1112 may be performed. For example,markers 502V in thefirst video image 1101V of thetool image 101V may be compared with the synthesized markers 502S on the synthesized tool image 101S in the synthesized image 1101S of thesame tool 101 to help determine new pose information of theactual tool 101. - Referring now to
FIG. 11C , special image matching of a left video image 1101VL and a right video image 1101VR is illustrated. The left video image 1101VL of the actual tool 101 (e.g., seeFIG. 1A ) is taken with a camera in a first position with respect to the tool, such as a left side, resulting in a left tool image 101VL. The right video image 1101VR of the sameactual tool 101 is taken at a different position with respect to the tool, such as the right side, resulting in a right tool image 101VR. The left video image 1101VL and the right video image 1101VR are captured at substantially the same time by their respective cameras. - Various aspects of the left and right video images of the tool may be used to perform image matching. That is, one or more of a matching of
markers 1110, a matching ofnatural features 1111, and/or an appearance matching 1112 may be performed. For example, markers 502VL on the left tool image 101VL in the left video image 1101VL of thetool 101 may be compared with the markers 502VR on the right tool image 101VR in the right video image 1101VR of the same tool to determine new pose information of theactual tool 101. - As mentioned herein, these image matching techniques may be combined to generate better pose information for the tool. For example, it is natural to combine temporal image matching (
FIG. 11A ) with spatial image matching (FIG. 11C ) or combine temporal image matching (FIG. 11A ) with synthesis image matching (FIG. 11D ). However, all three techniques may be used all together or flexibly in various combinations to try and obtain the best pose information for the tool. - In another embodiment of the invention, stereo images are used to construct 3D feature points, and these 3D points can be matched (for example, through the popular iterative closest point algorithm, reference: P. Besel and N, McKay, “A method for registration of 3-D shape,” IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 14, pp. 239-256, 1992) against corresponding 3D points, for example, markers, on the robotic instrument. Before we have stereo-derived 3D points, we need to apply a two-step approach: 1) matching of image correspondences, and 2) 3D reconstruction based on the geometry of stereo cameras (
FIG. 12B ). - Previously, image matching has been described with respect to explicit image synthesis, such as a full image synthesis or a geometry-only image synthesis. However, image matching may be made using implicit image synthesis. That is, images are not actually synthesized. Rather, a computer aided design (CAD) model of the tools and prior pose information of the tools are implicitly used to facilitate image matching.
- Moreover, artificial markers may be applied to the tools to assist in image matching as is described below. However, natural markers or features of a tool may be used to assist in image matching. Thus, the following descriptions apply equally well to the case when no artificial markers are present and features of the tool, i.e., natural markers, are detected directly from the instrument images.
- Referring now to
FIG. 5B , image matching can be significantly simplified yet made robust by exploring prior knowledge and available robotic kinematics even in the presence of more than one robotic instrument in the field of view.FIG. 5B illustrates video images 101AV and 101BV for a respective pair oftools view 510,FIG. 5B further illustrates pose information 101AK and 101BK based on kinematics for therespective tools view 510. The video images 101AV and 101BV and the pose information 101AK and 101BK for therespective tools marker dots respective tools marker dots 502A′ and 502B′ may be ascertained from the video images 101AV and 101BV of therespective tools - For simplicity in explanation, it is assumed that image feature points (
maker dots 502A′-502B′) are reliably localized, and without loss of generality, it is further assumed that the image feature points form a pattern. Using the concept of a pattern, the image matching problem is simplified from many dot-to-dot matchings to a single pattern matching. A pattern matching or association is straightforward if one tool is in the field of view. If more that one robotic instrument tool is in the field of view, there are two approaches available to solving the image matching problem that can be used alone or combined together. - In one embodiment of the invention, it is assumed that the robotic kinematics information will not change the spatial arrangements (positional and/or orientational) across instruments, especially in the beginning of surgical operations. For example, if two instruments are arranged to the left and to the right in the camera coordinate system, then the robotic kinematics should represent that arrangement. However, this is not a requirement on the absolute values of the robotic kinematics information. For example, the robotic kinematics information may indicate that the one or both of the robotic surgical tools are outside the field of view. Resolving pattern association ambiguity in matching a first pattern in an image to the tool arranged to the left and a second pattern in the image to the tool arranged to the right can be carried out in either 2D image or 3D space.
- In another embodiment of the invention, tool motion is used to resolve the pattern association ambiguity. As the plurality of tools move differently (e.g., different directions and/or speed), pattern images are tracked individually through temporal image matching. The motion trajectories of the pattern images are then compared against the kinematic information of each of the tools. From the comparison, the correct association of pattern images to tools can then be made.
- There may be a pattern association issue for a single tool in the field of view. A pattern may be directionally ambiguous, such as a line pattern with identical markers that has a directional ambiguity of a 180-degree flip. Pattern association for a single tool is not an issue if the pattern consisting of artificial markers or natural markers is unique. For example, if the pattern of markers has directional information embedded such that the markers at each of ends of the pattern are distinctive there is no issue. In one embodiment of the invention, the design of the artificial markers on the tool provides directional information of the marker pattern.
- If there is a directional ambiguity for a given tool, there are two approaches to solving the image matching problem, similar to how the pattern associations for multiple tools is handled. The first approach to solving the image matching problem is to use very rough robotic kinematics information to resolve any directional ambiguity. The robotic kinematics should not flip although it may be far away, such as outside the field of view after projection onto 2D image. The second approach to solving the image matching problem is to use motion trajectories of the pattern to remove the directional ambiguity.
- Image information may have quality issues (matching reliability and accuracy) regarding image-derived 3D information. For example, a viewing-singularity happens when a cylindrical instrument with markers on the shaft is projected to an image of small circle (see 1222S in
FIG. 12A ). That is, all the markers become invisible, hence there is no vision-based observation. Another extreme case for perfect viewing is when the instrument shaft lies right in the field of view so that all the markers are fully visible. That is, a circular marker is projected onto image as a circle for example. In practice, we often have scenarios between these extreme cases. - Viewing geometry is more than just pure geometry between a camera and a 3D object. Other information related to viewing geometry can impact the quality of image-derived information for fusion.
- Referring now to
FIG. 12A , the geometry of anillumination source 1210A-1210B with respect to thecamera 1211A-1211B and the 3D object may be of interest to achieve accurate and reliable information. The position oflight source 1210A with respect to thecamera 1211A and the3D object 1200generating view 1 may differ from the position oflight source 1210B with respect to thecamera 1211B and the3D object 1200generating view 2. Moreover, the different poses of the 3D object may change how a light source strikes its features and provide different views. - Additionally, different parts of a 3D object may behave differently under the same or different viewing conditions. For example, the
tip 1200T and theshaft 1200S of a3D tool 1200 may generate different image-derived 3D information with different reliabilities. Inimage 1221A the shaft image 1221S may be good to use while inimage 1221B the shaft image 1222S may be poor to use in forming image-derived 3D information. - The quality of image-derived 3D information also depends upon the particular image features used. For example, edge-based features are less sensitive to illumination change than intensity-based features.
- These issues with regard to viewing geometry contribute toward how vision information should be generated and combined with robot kinematics for fused results that are reliable and accurate. The quality of viewing geometry may be described statistically. View-geometry statistics are used to represent how good the image-derived information for fusion is when compared to the ground-truth. That is, view-geometry statistics may be used to estimate uncertainty. The view-geometry statistics are used in the Bayesian state-space model for adaptively fusing image information and kinematics information. To be specific, the view geometry statistics may be represented in the covariance matrix of the observation equation (Eq. 13).
- The following may be considered for view-geometry statistics: Digitization error/image resolution; feature/algorithm related Image matching error; Distance from object to camera; Angle between object surface normal and line of sight; Illumination and specularity. Based on certain noise assumptions (e.g., independent Gaussian noise), view-geometry statistics for these phenomenon may be computed. To make a specific example, we assume the case of parallel stereo images (
FIG. 12B ) were 3D points are first reconstructed and then fed into the state space. Under this ideal assumption, we have yr=yl. And the x-directional image projection are -
- where Bs is the baseline distance between the two optical centers and fx is the common focal length in x-direction. Finally, the 3D reconstruction problem becomes a simple one with a parallel setup as follows:
-
- For presentation simplicity, we use dx=xl−xr to represent the image disparity for the matched left and right image point.
- In the following, we list equations and plots based on the assumption of independent Gaussian noise and the parallel stereo setup (Eq. 14), Assuming that we have image matching uncertainty σx, we can derive the following equation
-
- where symbols with a bar (e.g.,
Z s,d x) represent the true values. - From this and the plots based on simulation of exact camera model, we can conclude that the uncertainty (standard deviations) of the X/Y/Z estimate of a 3D point is proportional to the uncertainty of image matching, the inverse of image resolution, and the square of the true value of X/Y/Z. The plot in
FIG. 12E is based on simulation of exact model and Gaussian random matching error to confirm the conclusion that the estimate uncertainty of a 3D point is proportional to the uncertainty of image matching. - The plot in
FIG. 12C is based on simulation of image digitization error [−0.5, 0.5] pixel and correct image matching to confirm the conclusion that the estimate uncertainty of a 3D point is proportional to the inverse of image resolution. The plot inFIG. 12D is based on simulation of varying depth to confirm the conclusion that the estimate uncertainty of a 3D point is proportional to the square of the true value of X/Y/Z: - View-geometry statistical analysis can also be applied to the most general case of camera set up, for example, a non-parallel set up. It can also be applied to the case of using 2D images rather than 3D stereo-reconstructed points as observations. In the following, we give a simple example. More specifically, let's apply the following projective equations
-
x i=ψ(K[R C |−C]X i), (Eq. 16) - for both left and right cameras. ψ is the perspective projection (Eq. 1A) and K is the 3×3 camera intrinsic parameter matrix, while RC (3×3) and C (3×1) represent the camera orientation and position, the extrinsic parameters. Overall, we can use a 3×4 matrix A to represent K[RC|−C]. For left camera and right camera, we use Al and Ar respectively. Hence the stereo problem becomes solving the following array of equations:
-
x i l=ψ(A l X i), -
x i r=ψ(A r X i) (Eq. 17) - We can linearize this array of equations to be the following with B being the 2×3 Jacobian matrix and {tilde over (x)} being the mean-subtracted variable as follows;
-
{tilde over (x)} i l ≈B l X i, -
{tilde over (x)} i r ≈B r X i (Eq. 18)) - From Eq. 18, we can compute the Jacobian matrix required for computing the observation covariance matrix for EKF in the case of using 2D images as observations as follows:
-
- where [ ]+ represents a pseudo inverse.
- Referring now to
FIGS. 13A-13B , the concept of sequence matching is now described. A technique is chosen for matching a pair of image features 1301 and 1302 such as from left and right images respectively in a stereo setting or a real image and a synthesized image. The pair of images features may be matched over one or more sequences ofimages 1301A-1301E and 1302A-1302E, for example. - A rigid sequence matching (where the relative kinematics within each sequence are perfectly known or identical across sequences in the ideal case) may be employed where just one common motion parameter is used to estimate for all pairs between two sequences such as illustrated in
FIG. 13A . Thesequences images 1301A to 1301E, and 1302A to 1302E. Alternatively, a flexible sequence matching (where the relative kinematics within each sequence are known with errors) may be employed as illustrated inFIG. 13B . Thesequences - To match a single pair of
features - If two or more two
temporal sequences 1301A-1301E and 1302A-1302E of images are available for matching and the kinematics among images within each sequence is known or identical across sequences, the probability of matching image features can be improved over that of a single pair of images. Assuming statistical independence and that the chance or probability of matching a single pair of image features correctly is 60% the chance of having correct matching improves to 78% with a sequence of 3 images, 92% with a sequence of 5 images, and 99% with a sequence of just 10 images, for example. However, if the relative kinematics for each pair of images in the sequence is not accurate, the chance of having correct matches is less and a bundle adjustment procedure should be taken for improved matching. Within the bundle adjustment procedure, the relative kinematics and its uncertainty are taken into consideration. - In one example of applying sequence matching where there are two sequences of observations (e.g., image feature points) O1={F1,1 . . . , F1,n} and O2={F2,1, . . . , F2,n}, the problem is to find out the geometric transform Ts between the two sequences through sequence matching. In matching individual pairs F1,i and F2,i, a geometric transform Ts,i:F1,i→F2,i may be computed. In other words, we have the following equation for individual matching
-
- where C is the cost function of matching features F1,i and F2,i and X represents the location of the features in either two or three dimensions. For example, the geometric transformation Ys,i may be in 3D and the features may be in 2D after known camera projection. If relative kinematics {dot over (k)}1/2,i within each sequence are perfectly known, then the geometric transformation equation for sequence matching may become
-
- where Ts k
1,1 represents transformation from Ts, typically a chosen Ts,i, through relative known kinematics. - In the geometric transformation (Eq. 20), the same amount of available information is used to estimate just one parameter Ts as it is to estimate a series of parameters Ts,j (Eq. 19). Consequently, the use of sequence of images and relative kinematics provides a result that is much more accurate and robust. Sequence matching is more accurate and robust with as many diverse images in the sequence as possible. If the sequence of images are all the same, then the assumption of statistical independence per matching is false. As a result, the matching performance with a sequence of images will not be improved. For example, in the tool acquisition
operational stage 604 illustrated inFIG. 6A , diversified images sequences may be generated by translating and/or rotating the tools within the field of view of the camera. - Sequence matching methods may be applied in many different scenarios, e.g., a single view case (a sequence of real images against a sequence of synthesized images) or a stereo case (matching among two sequences of real images and two sequence of synthesized images). However, relative kinematics may not be perfect and may change over time. Hence, bundle adjustment based flexible sequence matching may be applied to estimate the optimal parameters to more accurately locate and track tools.
- A tool tracking system for a robotic instrument has a number of applications. One application for a tool tracking system is image-guided surgery (IGS), or more specifically image-guided endoscopic surgery (IGES). The basic goal of image guided surgery is to enhance a surgeon's experience by providing real time information derived from single or multiple imaging modalities (e.g., visual, x-ray, computerized topography (CT), magnetic resonance imaging (MRI), ultrasound) during surgery or training/simulation. Two particular benefits of IGS/IGES are 1) improved visualization for easier on-line diagnostics and 2) improved localization for reliable and precise surgery. Tool tracking is one technology used for IGS/IGES since instruments are used by surgeons to navigate, sense and operate (e.g., diagnostic, cut, suture, ablation etc.) in the areas of interest. That is, the tool tracking system described herein can enable image-guided surgery without significant added operational inconvenience and/or added equipment. The tool tracking system described herein may also be used in other applications, such as dynamically reconstructing the geometry of organs, surgical simulation, and training.
- Tool tracking may be used to provide automated camera control and guidance to maintain a robotic instrument in the field of view. Tool tracking can also be used to assist the surgeon to move the robotic instrument to reach a tumor either automatically or with a surgeon's assistance. Ultra-sound or pre-scanned images can also be used along with real-time tool tracking. Other applications of tool tracking include graphic user interface that facilities the entrance and re-entrance of the robotic instrument during surgery.
- Tool tracking can be used to take a number of measurements during surgery as well. For example, tool tracking may be used to measure organ sizes by touching the robotic tool tip at different points of an organ. A pair of robotic tools being tracked can concurrent touch points of the organ and a distance along a line between their tips can be accurately measured with the assistance of tool tracking. Additionally, tool tracking may be used to construct a 3D model of an organ. The tip of a single robotic tool may be used to touch points across the organ's surface to construct a 3D model of the organ.
- Tool tracking can be used to align different image modalities together. Referring now to
FIG. 17 , a perspective view of a surgical site 1700 includes arobotic ultrasound tool 1710. The robotic ultrasound (US)tool 1710 has an attached orintegrated ultrasound transducer 1710A. The robotic ultrasound (US)tool 1710 may be used to guide and navigate other instruments to perform various medical or surgical procedures. By touchingtissue 1705 in the surgical site with therobotic ultrasound tool 1710, two dimensional ultrasound images 1711A may be captured in a two dimensional coordinatesystem 1703. The two dimensional ultrasound images 1711A may be translated from the two dimensional coordinatesystem 1703 into a camera coordinatesystem 1701. The translated ultrasound images may then be overlaid onto video images of the surgical site 1700 displayed by thestereo viewer 312, such as illustrated by the translatedultrasound images 1711B-1711D in the surgical site 1700 illustrated inFIG. 17 . Tool tracking may be used to flagpole ultrasound by (1) determining the transformation of the ultrasound images 1711A from the two dimensional coordinatesystem 1703 to the local ultrasound coordinatesystem 1702 in response to ultrasound calibration; (2) at thetransducer 1710A, determining the transformation from the ultrasound transducer coordinatesystem 1702 to the camera coordinatesystem 1701 by using tool tracking; and then; (3) cascading the transformations together to overlay the ultrasound image in the camera coordinatesystem 1701 onto the surgical site as illustrated byimage 1711B. The quality of image overlaying depends on tracking accuracy of the robotic ultrasound tool within a camera coordinate system (FIG. 5A ) and the ultrasound calibration. Ultrasound calibration is described in the reference “A novel closed form solution for ultrasound calibration,” Boctor, E.; Viswanathan, A.; Choti, M.; Taylor, R. H.; Fichtinger, G.; Hager, G., Biomedical Imaging: Nano to Macro, 2004. IEEE International Symposium on Volume, Issue, 15-18 Apr. 2004 Page(s): 527-530 Vol. 1. Tool tracking may also be used to generate 3D/volumetric ultrasound images by stacking overlaid2D images 1711B-D from the2D ultrasound transducer 1710A generated by rotating theultrasound tool 1701 as illustrated by thearrow 1720. - As shown in
FIG. 16 , tool tracking may be used overlay one or more drop virtual point/marks 1650A-1650B on images of thetissue surface 1600 in the surgical site by using one ormultiple tools - The tool tracking system described herein may also be used for image-guided interventional radiology (IGIR) along with other sensors. For example, active sensors/cameras (e.g. electro-magnetic sensor, or active near Infra-Red illumination plus stereo camera) may be used to scan patient bodies for navigation or 3D reconstruction with tool tracking during surgery.
- Referring now to
FIG. 15 , aflow chart 1500 of an IGS application with tool tracking is illustrated. Theflow chart 1500 illustrates a robotic surgery for a tumor in or around a liver. However, IGS with tool tracking may be used for other medical and surgical procedures for different organs and tissue that are robotically controlled or robotically assisted. - At
block 1501A, a liver or other organ/tissue prior to surgery may be scanned with acomputer tomography scanner 1503 to obtain a number of images of the liver such that they may be used to reconstruct a complete 3D volume of a scanned object atblock 1504 by a computer as desired. There may be a slight computer tomography error ECT for 3D reconstruction that may be minimized. - At
block 1505, a computer tomography volume that includes liver and surrounding area is selected to generate a computer tomographic (CT) segment that contains liver only. At this point, the CT volume segment is taken to register against a surface/depth map from other imaging modalities, e.g., stereo cameras. - The real liver during
surgery 1501B may have some biological variation forming an error (EBIO) from the prior scan taken by the CT scanner. - One or more
robotic instrument tips 1510 are inserted into the surgical site. - One or more
endoscopic cameras 1512 take sequences of images 1513-1514 of the surgical site including theliver 1501B and the robotic instrument tips 1501. These sequences of images 1513-1514 are typically stereo image pairs but could be a sequence of single images from a mono-view. The sequence ofimages 1513 may be used to determine the depth of surface features to make a surface/depth map 1515. In one embodiment of the invention, a sequence ofsurface maps 1515 including a robotic instrument may be analyzed similar to a sequence ofimages 1513 as described herein to determine a location of the robotic instrument. - The
surface map 1515 may be overlaid with a model of an organ after surface registration. Thesurface map 1515 may be further annotated with a map, outline, or other indication of the estimatedtumor location 1517. For example an internal liver tumor that is not visually visible maybe easily visible in a CT scan of the liver. An error between the estimated tumor location and the actual tumor location during surgery may be a sum of errors of the scanning (Ect), surface registration (Ereg), biological changes (Ebio), and formation of the depth/surface map (Est). This error can be reduced with further information from tool tracking and touching a tool tip to the tissue. - The sequences of
images 1514 may be stereo image pairs or a sequence of single images from a mono-view. The sequence ofimages 1514 along with kinematics information may be adaptively fused together to translate a model of the tool tip (MET) into the endoscopic camera frame of reference (ECM). The model of the tool tip formed from the adaptive fusion based tool tracking may be used to estimate thetool tip location 1518. An estimation error (EET) between theactual tip location 1520 and the estimatedtip location 1518 may be made small by the tool tracking methods described herein. By using thetumor location 1550 andtool tip location 1520 during surgery, we can drive the tools to touch their tips to reach the tumor, one example of image-guided surgery. One or more of the robotic surgical tools may include a needle at its tip as the end effector. The operational error EOP between the actual location of the tumor and the actual location of tool tips is a sum of errors. - Referring now to
FIG. 16 , a perspective view of a surgical site to provide image guided surgery is illustrated. Apre-scanned image 1602A (for example, CT images) may be aligned to a camera coordinatesystem 1605 and then overlaid as an overlaidimage 1602B onto a surface map ordepth map 1600 of the tissue surface. - Visual information and kinematics information of the first
robotic instrument 1610L and the secondrobotic instrument 1610R may be adaptively fused together to more accurately determine the position of the tools within the surgical site. With both position of the overlaidimage 1602B and thetools system 1605, the tool may be automatically or manually guided to perform a procedure on the tissue. The overlaidimage 1602B may represent a tumor on the tissue surface or buried below the tissue surface that would otherwise be hidden from view. As portions of thetool image 1610B of the portion of the tool below the surface may be synthesized in response to the tool tracking information and the adaptive fusion of kinematics with a priori video information of the tool. - The one or more robotic
surgical tools 16101, and 1610R may be used to take measurements or determine a surface profile. Tissue in a surgical site may be touched with the tool tip of therobotic tool 1610L at a first point and tissue in the surgical site may be touched with the tool tip of therobotic tool 1610R at a second point. A sequence of images of the surgical site including the roboticsurgical tools - Within the presented framework for adaptive fusion of vision and kinematics for tool tracking, we can also incorporate the depth maps.
- In one embodiment of the invention, we use depth maps for localizing tools. A sequence of depth maps including a robotic instrument may be analyzed to determine a location of robotic instruments. Robotic instruments can be located in the camera coordinate frame using any depth map in which the tool can be identified. Kinematics datum provides an approximate location for a robotic instrument that is to be tracked. This is a priori knowledge for the next iteration of the tracking problem. The depth map is analyzed in the environs of the approximate location to locate the robotic instrument. Other information may be employed to improve the a priori knowledge, such as a dynamic model of the robotic instrument, or knowledge of the type of procedure being performed under the camera capturing the images. The locating of the robotic instrument may be performed over any number of depth maps.
- If a correlation exists between sequential views, the problem of sequential location of a target robotic surgical tool is a tool tracking problem. The depth maps may be sequential views, arranged in time order, in which case there is a correlation between the successive views.
- Kinematics datum provides an approximate location for a robotic instrument that is to be tracked. This is a priori knowledge for the next iteration of the tracking problem. The depth map is analyzed in the environs of the approximate location to locate the robotic instrument. Other information may be employed to improve the a priori knowledge, such as a dynamic model of the robotic instrument, or knowledge of the type of procedure being performed under the camera capturing the images. If the robotic tool is obscured, a current optimal estimate of the location of the surgical instrument may be made (an a posteriori estimate) using the a priori knowledge and the depth map. Instantaneous (re-)correction of the kinematics datum may be computed by adaptively fusing together the available kinematics data, visual information, and/or a priori information. The correction to the current state is used to update the ongoing correction of future kinematics data. In one embodiment of the invention, the correction is simply made to future data without regard for past corrections. In another embodiment of the invention, the sequence of corrections is analyzed and an optimal correction based on all available past corrections is computed and used to correct the kinematics data. Analysis by synthesis and appearance learning techniques may be used to improve the correction to the current state.
- Algorithms that locate the surgical instrument in the depth map and provide the optimal kinematic correction, can be further optimized by understanding of the relative variances in corrected kinematic residual error vs. the variances in computed robotic instrument location from the depth map. Kinematics is suspected of initially having a large DC bias, but relatively small variance. A well-design image processing subsystem may have substantially zero DC bias but a relatively large variance. An example of an optimal correction that accommodates these two differing noise processes is one generated by a Kalman filter.
- Previously described herein with reference to
FIG. 16 , if portions of thetool image 1610B of the portion of the tool below the surface may be synthesized. Thesynthesized image portion 1610B may be included in thetool images 400L,400R as synthesizedimage portion 400B displayed on thedisplay devices stereo viewer 312 illustrated inFIG. 4 . - Referring now to
FIG. 5B , while tracking a robotic surgical tool, it may exit the field ofview 510 of a camera entirely such, as illustrated by thetool 101F. The robotic surgical tool may no longer be in the field of view as a result of camera movement over the surgical site, robotic surgical tool movement, or a combination of both. For example, the camera may move away from the position of the robotic surgical tool in a surgical site such that the robotic surgical tool is outside the field of view of the camera. As another example, the robotic surgical tool may move away from a position of the camera in a surgical site such that the robotic surgical tool is outside the field of view of the camera. In either case, a surgeon may be left guessing where the robotic surgical tool is outside the field of view unless some indication is provided to him in thestereo viewer 312. - In
FIG. 4 , compass icons or a compass rose (generally referred to with reference number 420) may be displayed in the display devices of thestereo viewer 312 to provide an indication where the robotic surgical tool is located outside the field of view and a direction of tool reentrance into the field of view. For example one of a plurality of compass icons for thedirections North 420N, South 420S,East 420E, andWest 420W as well as directions in between such as North-East 420NE, North-West 420NW, South-East 420SE, and South-West 420SW may be indicated in the stereo viewer to indicate tool reentrance into the field of view of the camera over a surgical site. - To indicate tool reentrance into the field of view a robotic surgical tool is tracked in and out of the field of view of the camera. A determination is made by comparing positions of the camera and a tool whether or not it is outside the field of view of the camera by using the available tool tracking information of the robotic surgical tool. It the robotic surgical tool is out of the field of view, then one of a plurality of
compass icons 420 may be displayed in the field of view to show a direction of tool reentrance. - Additional information, such as getting closer or farther in distance away from the field of view, may be conveyed to a surgeon looking in the
stereo viewer 312 by somewhat altering the onecompass icon 420 in the display that indicates the direction of the tool. For example, by flashing the icon fast or slow may indicate the robotic surgical tool is getting closer or farther away respectively from the field of view. As another example, arrowheads may be added to ends of a bar icon to indicate that the robotic surgical tool is move towards or away from the field of view. In an alternate implementation, the icon indicating direction or reentrance may be colored to indicate movement bringing the tool closer to the field of view (e.g., red for getting warmer) or movement taking the tool further away from the field of view (e.g., green for getting colder). - The embodiments of the tool tracking system described herein provide an automatic integrated system that is accurate and reliable by adaptively fusing kinematics and visual information, synthesizing images based on a model and prior poses, and employing sequence matching.
- A number of elements of the tool tracking system are implemented in software and executed by a computer and its processor, such as
computer 151 and itsprocessor 302. When implemented in software, the elements of the embodiments of the invention are essentially the code segments to perform the necessary tasks. The program or code segments can be stored in a processor readable medium or transmitted by a computer data signal embodied in a carrier wave over a transmission medium or communication link. The processor readable medium may include any medium that can store or transfer information. Examples of the processor readable medium include an electronic circuit, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM), a floppy diskette, a CD-ROM, an optical disk, a hard disk, a fiber optic medium, a radio frequency (RF) link, etc. The computer data signal may include any signal that can propagate over a transmission medium such as electronic network channels, optical fibers, air, electromagnetic, RF links, etc. The code segments may be downloaded via computer networks such as the Internet, Intranet, etc. - While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the embodiments of the invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art. For example, some embodiments of the invention have been described with reference to a robotic surgical system. However, these embodiments are equally applicable to other robotic systems. Thus, the embodiments of the invention should be construed according to the claims that follow below.
Claims (25)
1-25. (canceled)
26. A robotic system comprising:
a first robotic tool having a first tool tip;
a second robotic tool having a second tool tip; and
a processor programmed to: fuse kinematics derived position information and image derived position information of the first robotic tool in a state-space model by using a tool tracking system to estimate a first tool tip location while the first tool tip is touching tissue at a first point; fuse kinematics derived position information and image derived position information of the second robotic tool in a state-space model by using the tool tracking system to estimate a second tool tip location while the second tool tip is touching the tissue at a second point; and compare the first tool tip location and the second tool tip location by using the tool tracking system to determine a distance between the first and second points on the tissue.
27. The robotic system of claim 26 , wherein
the tissue comprises a tumor;
the first point is on a first external surface of the tumor;
the second point is on a second external surface of the tumor; and
the comparing determines a diameter of the tumor.
28. The robotic system of claim 26 , wherein
the tissue comprises an organ;
the first point is on a first external surface of the organ;
the second point is on a second external surface of the organ; and
the comparing determines a diameter of the organ.
29. The robotic system of claim 26 , wherein the processor is programmed to:
process sensor data of the first robotic tool by using the tool tracking system to generate the kinematics derived position information of the first robotic tool; and
process sensor data of the second robotic tool by using the tool tracking system to generate the kinematics derived position information of the second robotic tool.
30. The robotic system of claim 26 , further comprising:
an image capture device for capturing a sequence of images of the first robotic tool and the second robotic tool at a surgical site;
wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using the tool tracking system to generate the image derived position information of the first and second robotic tools.
31. The robotic system of claim 30 , wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using an image-by-synthesis approach wherein synthesized images of computer models of the first and second robotic tools are compared against images of the first and second robotic tools in the sequence of images.
32. The robotic system of claim 30 , wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using a feature matching approach wherein features of the first and second robotic tools are identified in the sequence of images.
33. The robotic system of claim 30 , wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using a sequence matching approach wherein objects or features in a sequence of images captured from a view of the image capturing device are matched against objects or features in a sequence of images captured from a different view of the image capturing device.
34. The robotic system of claim 30 , wherein the processor is programmed to process the sequence of images of the first robotic tool and the second robotic tool at the surgical site by using a sequence matching approach wherein objects or features in a sequence of images captured from a view of the image capturing device are matched against objects or features in a sequence of synthesized images.
35. The robotic system of claim 26 , wherein the processor is programmed to fuse the kinematics derived position information and the image derived position information of the first robotic tool in the state-space model by using a sequential Bayesian approach; and fuse the kinematics derived position information and the image derived position information of the second robotic tool in the state-space model by using the sequential Bayesian approach.
36. A method for tracking movement of a robotic instrument, the method comprising:
receiving images of video frames from at least one camera;
receiving kinematics information related to robotic movement of the robotic instrument;
determining mechanical pose information from the kinematics information;
synthesizing model pose information of a computer aided design model of the robotic instrument using the mechanical pose information;
determining video pose information of the robotic instrument by using the synthesized model pose information as a pattern for pattern searching within the images; and
providing a state-space model of a sequence of states of corrected kinematics information for accurate pose information of the robotic instrument, the state-space model to receive raw kinematics information of mechanical pose information and to adaptively fuse the mechanical pose information and the video pose information together to generate the sequence of states of the corrected kinematics information for the robotic instrument.
37. The method of claim 36 , wherein the synthesized model pose information of the model of the robotic instrument includes one or more markers of the robotic instrument forming a pattern.
38. The method of claim 37 , wherein the one or more markers include artificial markers consisting of a pattern of dots.
39. The method of claim 37 , wherein the one or more markers include natural markers represented by consisting of geometry information of the computer aided design model.
40. A method for tracking movement of a robotic instrument, the method comprising:
receiving images of video frames from at least one camera;
determining video pose information of the robotic instrument within the images;
estimating uncertainty in the determined video pose information in light of video information in response to view geometry statistics;
receiving kinematics information related to robotic movement of the robotic instrument;
determining mechanical pose information from the kinematics information; and
fusing the mechanical pose information and the video pose information in a state-space model using a covariance matrix configured to compensate for the estimated uncertainty in the video pose information to generate estimated pose information for the robotic instrument.
41. The method of claim 40 , wherein the state-space model includes a dynamic model and an observation model respectively including dynamic noise and observation noise that respectively have dynamic and observation Gaussian distributions respectively characterized by dynamic and observation covariance matrices, wherein the observation covariance matrix includes a sub-matrix for vision; and wherein the method further comprises:
configuring the sub-matrix for vision to compensate for the estimated uncertainty of video information by modifying elements of the sub-matrix so as to adjust a standard deviation for the estimated pose information for the robotic instrument accordingly.
42. The method of claim 40 , wherein assuming an independent Gaussian noise model, the view geometry statistics are computed for one or more of digitization error/image resolution, feature/algorithm related image matching error, distance from object to camera, angle between object surface normal and line of sight, illumination, and specularity.
43. A robotic system comprising:
a camera;
a robotic instrument; and
one or more processors programmed so as to cooperatively perform the following tasks: receive images of video frames from the camera; receive kinematics information related to robotic movement of the robotic instrument; determine mechanical pose information from the kinematics information; synthesize model pose information of a computer aided design model of the robotic instrument using the mechanical pose information; determine video pose information of the robotic instrument by using the synthesized model pose information as a pattern for pattern searching within the images; and provide a state-space model of a sequence of states of corrected kinematics information for accurate pose information of the robotic instrument, the state-space model to receive raw kinematics information of mechanical pose information and to adaptively fuse the mechanical pose information and the video pose information together to generate the sequence of states of the corrected kinematics information for the robotic instrument.
44. The robotic system of claim 43 , wherein the synthesized model pose information of the model of the robotic instrument includes one or more markers of the robotic instrument forming a pattern.
45. The robotic system of claim 44 , wherein the one or more markers include artificial markers consisting of a pattern of dots.
46. The robotic system of claim 44 , wherein the one or more markers include natural markers represented by consisting of geometry information of the computer aided design model.
47. A robotic system comprising:
a camera;
a robotic instrument; and
one or more processors programmed so as to cooperatively perform the following tasks: receive images of video frames from the camera; determine video pose information of the robotic instrument within the images; estimate uncertainty in the determined video pose information in light of video information in response to view geometry statistics; receive kinematics information related to robotic movement of the robotic instrument; determine mechanical pose information from the kinematics information; and fuse the mechanical pose information and the video pose information in a state-space model using a covariance matrix configured to compensate for the estimated uncertainty in the video pose information to generate estimated pose information for the robotic instrument.
48. The robotic system of claim 47 , wherein the state-space model includes a dynamic model and an observation model respectively including dynamic noise and observation noise that respectively have dynamic and observation Gaussian distributions respectively characterized by dynamic and observation covariance matrices, wherein the observation covariance matrix includes a sub-matrix for vision; and wherein one of the one or more processors is programmed to: configure the sub-matrix for vision to compensate for the estimated uncertainty of video information by modifying elements of the sub-matrix so as to adjust a standard deviation for the estimated pose information for the robotic instrument accordingly.
49. The robotic system of claim 47 , wherein assuming an independent Gaussian noise model, and the view geometry statistics are computed for one or more of digitization error/image resolution, feature/algorithm related image matching error, distance from object to camera, angle between object surface normal and line of sight, illumination, and specularity.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/314,161 US20150005622A1 (en) | 2007-09-30 | 2014-06-25 | Methods of locating and tracking robotic instruments in robotic surgical systems |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/865,016 US8073528B2 (en) | 2007-09-30 | 2007-09-30 | Tool tracking systems, methods and computer products for image guided surgery |
US13/249,821 US8792963B2 (en) | 2007-09-30 | 2011-09-30 | Methods of determining tissue distances using both kinematic robotic tool position information and image-derived position information |
US14/314,161 US20150005622A1 (en) | 2007-09-30 | 2014-06-25 | Methods of locating and tracking robotic instruments in robotic surgical systems |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/249,821 Division US8792963B2 (en) | 2007-09-30 | 2011-09-30 | Methods of determining tissue distances using both kinematic robotic tool position information and image-derived position information |
Publications (1)
Publication Number | Publication Date |
---|---|
US20150005622A1 true US20150005622A1 (en) | 2015-01-01 |
Family
ID=40509160
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/865,016 Active 2030-08-29 US8073528B2 (en) | 2005-05-16 | 2007-09-30 | Tool tracking systems, methods and computer products for image guided surgery |
US13/249,821 Active 2028-01-29 US8792963B2 (en) | 2007-09-30 | 2011-09-30 | Methods of determining tissue distances using both kinematic robotic tool position information and image-derived position information |
US14/314,161 Abandoned US20150005622A1 (en) | 2007-09-30 | 2014-06-25 | Methods of locating and tracking robotic instruments in robotic surgical systems |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/865,016 Active 2030-08-29 US8073528B2 (en) | 2005-05-16 | 2007-09-30 | Tool tracking systems, methods and computer products for image guided surgery |
US13/249,821 Active 2028-01-29 US8792963B2 (en) | 2007-09-30 | 2011-09-30 | Methods of determining tissue distances using both kinematic robotic tool position information and image-derived position information |
Country Status (1)
Country | Link |
---|---|
US (3) | US8073528B2 (en) |
Cited By (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9682476B1 (en) | 2015-05-28 | 2017-06-20 | X Development Llc | Selecting robot poses to account for cost |
US9717607B1 (en) | 2016-10-28 | 2017-08-01 | International Business Machines Corporation | Augmented control of robotic prosthesis by a cognitive system |
US9724826B1 (en) | 2015-05-28 | 2017-08-08 | X Development Llc | Selecting physical arrangements for objects to be acted upon by a robot |
US20180078316A1 (en) * | 2016-09-22 | 2018-03-22 | Medtronic Navigation, Inc. | System for Guided Procedures |
WO2018217433A1 (en) * | 2017-05-25 | 2018-11-29 | Covidien Lp | Systems and methods for detection of objects within a field of view of an image capture device |
US10555775B2 (en) | 2005-05-16 | 2020-02-11 | Intuitive Surgical Operations, Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
WO2020146501A1 (en) * | 2019-01-10 | 2020-07-16 | Verily Life Sciences Llc | Surgical workflow and activity detection based on surgical videos |
WO2020176161A1 (en) * | 2019-02-25 | 2020-09-03 | Acessa Health Inc. | Automated ablation control systems |
CN112621789A (en) * | 2020-12-08 | 2021-04-09 | 广东联航智能科技有限公司 | Control system of robot for double-arm man-machine cooperative operation |
WO2021120554A1 (en) * | 2019-12-18 | 2021-06-24 | 苏州微创畅行机器人有限公司 | Registration apparatus, registration method, registration system, navigation operation system, electronic device, and storage medium |
WO2021146339A1 (en) * | 2020-01-14 | 2021-07-22 | Activ Surgical, Inc. | Systems and methods for autonomous suturing |
WO2021202697A1 (en) * | 2020-03-31 | 2021-10-07 | Georgia Tech Research Corporation | Method and system for mri-guided visual servoing |
WO2021199305A1 (en) * | 2020-03-31 | 2021-10-07 | 日本電気株式会社 | Control device, control system, control method, and recording medium with control program recorded thereon |
WO2021211516A1 (en) * | 2020-04-13 | 2021-10-21 | Kaliber Labs Inc. | Systems and methods for computer-assisted shape measurements in video |
US20220005226A1 (en) * | 2020-07-05 | 2022-01-06 | Asensus Surgical Us, Inc. | Camera calibration using measured motion |
US11553984B2 (en) | 2016-06-03 | 2023-01-17 | Covidien Lp | Robotic surgical system with an embedded imager |
US11553974B2 (en) | 2017-05-25 | 2023-01-17 | Covidien Lp | Systems and methods for detection of objects within a field of view of an image capture device |
US11594325B2 (en) * | 2018-09-12 | 2023-02-28 | Verb Surgical Inc. | Method and system for automatically tracking and managing inventory of surgical tools in operating rooms |
US11602402B2 (en) | 2018-12-04 | 2023-03-14 | Globus Medical, Inc. | Drill guide fixtures, cranial insertion fixtures, and related methods and robotic systems |
US11759271B2 (en) | 2017-04-28 | 2023-09-19 | Stryker Corporation | System and method for indicating mapping of console-based surgical systems |
US11901065B2 (en) | 2018-05-23 | 2024-02-13 | Verb Surgical Inc. | Surgery evaluation using machine-learning-based surgical video analysis |
US11944272B2 (en) | 2017-12-07 | 2024-04-02 | Medtronic Xomed, Inc. | System and method for assisting visualization during a procedure |
US11963676B2 (en) | 2018-10-16 | 2024-04-23 | Activ Surgical, Inc. | Autonomous methods and systems for tying surgical knots |
Families Citing this family (283)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8944070B2 (en) * | 1999-04-07 | 2015-02-03 | Intuitive Surgical Operations, Inc. | Non-force reflecting method for providing tool force information to a user of a telesurgical system |
US9155544B2 (en) * | 2002-03-20 | 2015-10-13 | P Tech, Llc | Robotic systems and methods |
US8827899B2 (en) * | 2004-09-24 | 2014-09-09 | Vivid Medical, Inc. | Disposable endoscopic access device and portable display |
US9943372B2 (en) * | 2005-04-18 | 2018-04-17 | M.S.T. Medical Surgery Technologies Ltd. | Device having a wearable interface for improving laparoscopic surgery and methods for use thereof |
US9492240B2 (en) | 2009-06-16 | 2016-11-15 | Intuitive Surgical Operations, Inc. | Virtual measurement tool for minimally invasive surgery |
US8971597B2 (en) * | 2005-05-16 | 2015-03-03 | Intuitive Surgical Operations, Inc. | Efficient vision and kinematic data fusion for robotic surgical instruments and other applications |
US9789608B2 (en) | 2006-06-29 | 2017-10-17 | Intuitive Surgical Operations, Inc. | Synthetic representation of a surgical robot |
US8073528B2 (en) | 2007-09-30 | 2011-12-06 | Intuitive Surgical Operations, Inc. | Tool tracking systems, methods and computer products for image guided surgery |
US8147503B2 (en) * | 2007-09-30 | 2012-04-03 | Intuitive Surgical Operations Inc. | Methods of locating and tracking robotic instruments in robotic surgical systems |
US20070265495A1 (en) * | 2005-12-15 | 2007-11-15 | Medivision, Inc. | Method and apparatus for field of view tracking |
US9266239B2 (en) * | 2005-12-27 | 2016-02-23 | Intuitive Surgical Operations, Inc. | Constraint based control in a minimally invasive surgical apparatus |
US7907166B2 (en) * | 2005-12-30 | 2011-03-15 | Intuitive Surgical Operations, Inc. | Stereo telestration for robotic surgery |
US9782229B2 (en) * | 2007-02-16 | 2017-10-10 | Globus Medical, Inc. | Surgical robot platform |
US20080065104A1 (en) * | 2006-06-13 | 2008-03-13 | Intuitive Surgical, Inc. | Minimally invasive surgical instrument advancement |
US20090192523A1 (en) | 2006-06-29 | 2009-07-30 | Intuitive Surgical, Inc. | Synthetic representation of a surgical instrument |
US10258425B2 (en) * | 2008-06-27 | 2019-04-16 | Intuitive Surgical Operations, Inc. | Medical robotic system providing an auxiliary view of articulatable instruments extending out of a distal end of an entry guide |
US9718190B2 (en) | 2006-06-29 | 2017-08-01 | Intuitive Surgical Operations, Inc. | Tool position and identification indicator displayed in a boundary area of a computer display screen |
US10008017B2 (en) | 2006-06-29 | 2018-06-26 | Intuitive Surgical Operations, Inc. | Rendering tool information as graphic overlays on displayed images of tools |
US7728868B2 (en) | 2006-08-02 | 2010-06-01 | Inneroptic Technology, Inc. | System and method of providing real-time dynamic imagery of a medical procedure site using multiple modalities |
CA2684475C (en) * | 2007-04-16 | 2016-01-12 | Neuroarm Surgical Ltd. | Frame mapping and force feedback methods, devices and systems |
US8903546B2 (en) | 2009-08-15 | 2014-12-02 | Intuitive Surgical Operations, Inc. | Smooth control of an articulated instrument across areas with different work space conditions |
US9084623B2 (en) | 2009-08-15 | 2015-07-21 | Intuitive Surgical Operations, Inc. | Controller assisted reconfiguration of an articulated instrument during movement into and out of an entry guide |
US8620473B2 (en) | 2007-06-13 | 2013-12-31 | Intuitive Surgical Operations, Inc. | Medical robotic system with coupled control modes |
US9089256B2 (en) | 2008-06-27 | 2015-07-28 | Intuitive Surgical Operations, Inc. | Medical robotic system providing an auxiliary view including range of motion limitations for articulatable instruments extending out of a distal end of an entry guide |
US9469034B2 (en) | 2007-06-13 | 2016-10-18 | Intuitive Surgical Operations, Inc. | Method and system for switching modes of a robotic system |
US9138129B2 (en) | 2007-06-13 | 2015-09-22 | Intuitive Surgical Operations, Inc. | Method and system for moving a plurality of articulated instruments in tandem back towards an entry guide |
US20090069804A1 (en) * | 2007-09-12 | 2009-03-12 | Jensen Jeffrey L | Apparatus for efficient power delivery |
JPWO2009084345A1 (en) * | 2007-12-28 | 2011-05-19 | オリンパスメディカルシステムズ株式会社 | Medical equipment system |
WO2009094646A2 (en) | 2008-01-24 | 2009-07-30 | The University Of North Carolina At Chapel Hill | Methods, systems, and computer readable media for image guided ablation |
US8340379B2 (en) | 2008-03-07 | 2012-12-25 | Inneroptic Technology, Inc. | Systems and methods for displaying guidance data based on updated deformable imaging data |
US8290240B2 (en) * | 2008-06-11 | 2012-10-16 | Sirona Dental Systems Gmbh | System, apparatus, method, and computer program product for determining spatial characteristics of an object using a camera and a search pattern |
US8121389B2 (en) * | 2008-06-11 | 2012-02-21 | Sirona Dental Systems Gmbh | System, apparatus, method and computer program product for optical position recognition |
US20090312629A1 (en) * | 2008-06-13 | 2009-12-17 | Inneroptic Technology Inc. | Correction of relative tracking errors based on a fiducial |
US8864652B2 (en) | 2008-06-27 | 2014-10-21 | Intuitive Surgical Operations, Inc. | Medical robotic system providing computer generated auxiliary views of a camera instrument for controlling the positioning and orienting of its tip |
US9023027B2 (en) | 2008-09-30 | 2015-05-05 | Biosense Webster (Israel), Ltd. | Current localization tracker |
US8456182B2 (en) * | 2008-09-30 | 2013-06-04 | Biosense Webster, Inc. | Current localization tracker |
US20100125284A1 (en) * | 2008-11-20 | 2010-05-20 | Hansen Medical, Inc. | Registered instrument movement integration |
US8830224B2 (en) * | 2008-12-31 | 2014-09-09 | Intuitive Surgical Operations, Inc. | Efficient 3-D telestration for local robotic proctoring |
US8184880B2 (en) * | 2008-12-31 | 2012-05-22 | Intuitive Surgical Operations, Inc. | Robust sparse image matching for robotic surgery |
US11464578B2 (en) | 2009-02-17 | 2022-10-11 | Inneroptic Technology, Inc. | Systems, methods, apparatuses, and computer-readable media for image management in image-guided medical procedures |
US8641621B2 (en) | 2009-02-17 | 2014-02-04 | Inneroptic Technology, Inc. | Systems, methods, apparatuses, and computer-readable media for image management in image-guided medical procedures |
US8690776B2 (en) | 2009-02-17 | 2014-04-08 | Inneroptic Technology, Inc. | Systems, methods, apparatuses, and computer-readable media for image guided surgery |
US8554307B2 (en) | 2010-04-12 | 2013-10-08 | Inneroptic Technology, Inc. | Image annotation in image-guided medical procedures |
US8423182B2 (en) | 2009-03-09 | 2013-04-16 | Intuitive Surgical Operations, Inc. | Adaptable integrated energy control system for electrosurgical tools in robotic surgical systems |
WO2010125481A1 (en) * | 2009-04-29 | 2010-11-04 | Koninklijke Philips Electronics, N.V. | Real-time depth estimation from monocular endoscope images |
US9155592B2 (en) * | 2009-06-16 | 2015-10-13 | Intuitive Surgical Operations, Inc. | Virtual measurement tool for minimally invasive surgery |
US8918212B2 (en) | 2009-06-24 | 2014-12-23 | Intuitive Surgical Operations, Inc. | Arm with a combined shape and force sensor |
US8423186B2 (en) * | 2009-06-30 | 2013-04-16 | Intuitive Surgical Operations, Inc. | Ratcheting for master alignment of a teleoperated minimally-invasive surgical instrument |
GB2472066A (en) | 2009-07-23 | 2011-01-26 | Medi Maton Ltd | Device for manipulating and tracking a guide tube with radiopaque markers |
US9492927B2 (en) | 2009-08-15 | 2016-11-15 | Intuitive Surgical Operations, Inc. | Application of force feedback on an input device to urge its operator to command an articulated instrument to a preferred pose |
US8918211B2 (en) | 2010-02-12 | 2014-12-23 | Intuitive Surgical Operations, Inc. | Medical robotic system providing sensory feedback indicating a difference between a commanded state and a preferred pose of an articulated instrument |
KR100956762B1 (en) * | 2009-08-28 | 2010-05-12 | 주식회사 래보 | Surgical robot system using history information and control method thereof |
CN102781336B (en) * | 2009-10-30 | 2016-01-20 | 约翰霍普金斯大学 | For visual tracking and the annotation of the anatomic landmark important clinically of surgical intervention |
CN102711650B (en) * | 2010-01-13 | 2015-04-01 | 皇家飞利浦电子股份有限公司 | Image integration based registration and navigation for endoscopic surgery |
CN102144927B (en) * | 2010-02-10 | 2012-12-12 | 清华大学 | Motion-compensation-based computed tomography (CT) equipment and method |
IT1398933B1 (en) * | 2010-03-04 | 2013-03-28 | Clementoni S P A | MULTIMEDIA SYSTEM FOR LEARNING AND PUZZLE GAMES. |
CN102802498B (en) * | 2010-03-24 | 2015-08-19 | 奥林巴斯株式会社 | Endoscope apparatus |
US9089292B2 (en) * | 2010-03-26 | 2015-07-28 | Medtronic Minimed, Inc. | Calibration of glucose monitoring sensor and/or insulin delivery system |
EP2449954B1 (en) * | 2010-05-31 | 2014-06-04 | Olympus Medical Systems Corp. | Endoscopic form detection device and form detecting method of insertion section of endoscope |
US9516207B2 (en) * | 2010-06-24 | 2016-12-06 | Marc S. Lemchen | Exam-cam robotic systems and methods |
EP2590551B1 (en) * | 2010-07-09 | 2019-11-06 | Edda Technology, Inc. | Methods and systems for real-time surgical procedure assistance using an electronic organ map |
JP5735227B2 (en) * | 2010-07-16 | 2015-06-17 | ルネサスエレクトロニクス株式会社 | Image conversion apparatus and image conversion system |
FR2963693B1 (en) | 2010-08-04 | 2013-05-03 | Medtech | PROCESS FOR AUTOMATED ACQUISITION AND ASSISTED ANATOMICAL SURFACES |
JP5065538B2 (en) | 2010-09-14 | 2012-11-07 | オリンパスメディカルシステムズ株式会社 | ENDOSCOPE SYSTEM AND METHOD OF OPERATING VISUAL DEFECT JUDGING SYSTEM |
US9805625B2 (en) | 2010-10-29 | 2017-10-31 | KindHeart, Inc. | Surgical simulation assembly |
US9342997B2 (en) | 2010-10-29 | 2016-05-17 | The University Of North Carolina At Chapel Hill | Modular staged reality simulator |
EP2635190B1 (en) * | 2010-11-05 | 2017-01-11 | Koninklijke Philips N.V. | Imaging apparatus for imaging an object |
US9486189B2 (en) | 2010-12-02 | 2016-11-08 | Hitachi Aloka Medical, Ltd. | Assembly for use with surgery system |
WO2012078989A1 (en) * | 2010-12-10 | 2012-06-14 | Wayne State University | Intelligent autonomous camera control for robotics with medical, military, and space applications |
US8712177B2 (en) * | 2011-01-25 | 2014-04-29 | Siemens Aktiengesellschaft | Motion compensated overlay |
US11412998B2 (en) | 2011-02-10 | 2022-08-16 | Karl Storz Imaging, Inc. | Multi-source medical display |
US10674968B2 (en) * | 2011-02-10 | 2020-06-09 | Karl Storz Imaging, Inc. | Adjustable overlay patterns for medical display |
US10631712B2 (en) * | 2011-02-10 | 2020-04-28 | Karl Storz Imaging, Inc. | Surgeon's aid for medical display |
US9449241B2 (en) | 2011-02-23 | 2016-09-20 | The Johns Hopkins University | System and method for detecting and tracking a curvilinear object in a three-dimensional space |
US9026247B2 (en) | 2011-03-30 | 2015-05-05 | University of Washington through its Center for Communication | Motion and video capture for tracking and evaluating robotic surgery and associated systems and methods |
WO2012178202A1 (en) * | 2011-06-23 | 2012-12-27 | Oblong Industries, Inc. | Adaptive tracking system for spatial input devices |
US9519341B2 (en) | 2011-08-04 | 2016-12-13 | Olympus Corporation | Medical manipulator and surgical support apparatus |
JP6005950B2 (en) | 2011-08-04 | 2016-10-12 | オリンパス株式会社 | Surgery support apparatus and control method thereof |
JP5936914B2 (en) | 2011-08-04 | 2016-06-22 | オリンパス株式会社 | Operation input device and manipulator system including the same |
JP6021353B2 (en) * | 2011-08-04 | 2016-11-09 | オリンパス株式会社 | Surgery support device |
EP2740434A4 (en) | 2011-08-04 | 2015-03-18 | Olympus Corp | Medical manipulator and method for controlling same |
JP6009840B2 (en) | 2011-08-04 | 2016-10-19 | オリンパス株式会社 | Medical equipment |
JP5841451B2 (en) | 2011-08-04 | 2016-01-13 | オリンパス株式会社 | Surgical instrument and control method thereof |
JP6000641B2 (en) | 2011-08-04 | 2016-10-05 | オリンパス株式会社 | Manipulator system |
JP5931497B2 (en) | 2011-08-04 | 2016-06-08 | オリンパス株式会社 | Surgery support apparatus and assembly method thereof |
JP6081061B2 (en) | 2011-08-04 | 2017-02-15 | オリンパス株式会社 | Surgery support device |
JP5953058B2 (en) | 2011-08-04 | 2016-07-13 | オリンパス株式会社 | Surgery support device and method for attaching and detaching the same |
US9161772B2 (en) | 2011-08-04 | 2015-10-20 | Olympus Corporation | Surgical instrument and medical manipulator |
JP6021484B2 (en) | 2011-08-04 | 2016-11-09 | オリンパス株式会社 | Medical manipulator |
US9757206B2 (en) | 2011-08-21 | 2017-09-12 | M.S.T. Medical Surgery Technologies Ltd | Device and method for assisting laparoscopic surgery—rule based approach |
US11561762B2 (en) * | 2011-08-21 | 2023-01-24 | Asensus Surgical Europe S.A.R.L. | Vocally actuated surgical control system |
US9204939B2 (en) | 2011-08-21 | 2015-12-08 | M.S.T. Medical Surgery Technologies Ltd. | Device and method for assisting laparoscopic surgery—rule based approach |
US10866783B2 (en) | 2011-08-21 | 2020-12-15 | Transenterix Europe S.A.R.L. | Vocally activated surgical control system |
US9795282B2 (en) | 2011-09-20 | 2017-10-24 | M.S.T. Medical Surgery Technologies Ltd | Device and method for maneuvering endoscope |
US8693730B2 (en) * | 2011-11-15 | 2014-04-08 | Macdonald Dettwiler & Associates Inc. | Method of real-time tracking of moving/flexible surfaces |
FR2983059B1 (en) * | 2011-11-30 | 2014-11-28 | Medtech | ROBOTIC-ASSISTED METHOD OF POSITIONING A SURGICAL INSTRUMENT IN RELATION TO THE BODY OF A PATIENT AND DEVICE FOR CARRYING OUT SAID METHOD |
KR101876386B1 (en) * | 2011-12-29 | 2018-07-11 | 삼성전자주식회사 | Medical robotic system and control method for thereof |
WO2013116240A1 (en) | 2012-01-30 | 2013-08-08 | Inneroptic Technology, Inc. | Multiple medical device guidance |
EP3760389A1 (en) * | 2012-02-15 | 2021-01-06 | Intuitive Surgical Operations, Inc. | Switching control of an instrument to an input device upon the instrument entering a display area viewable by an operator of the input device |
US9782147B2 (en) * | 2012-03-06 | 2017-10-10 | Analogic Corporation | Apparatus and methods for localization and relative positioning of a surgical instrument |
WO2013158636A1 (en) * | 2012-04-16 | 2013-10-24 | Azizian Mahdi | Dual-mode stereo imaging system for tracking and control in surgical and interventional procedures |
US10383765B2 (en) * | 2012-04-24 | 2019-08-20 | Auris Health, Inc. | Apparatus and method for a global coordinate system for use in robotic surgery |
US9105207B2 (en) | 2012-05-02 | 2015-08-11 | Empire Technology Development Llc | Four dimensional image registration using dynamical model for augmented reality in medical applications |
WO2013182224A1 (en) * | 2012-06-05 | 2013-12-12 | Brainlab Ag | Improving the accuracy of navigating a medical device |
US9220570B2 (en) | 2012-06-29 | 2015-12-29 | Children's National Medical Center | Automated surgical and interventional procedures |
JP5949242B2 (en) * | 2012-07-11 | 2016-07-06 | セイコーエプソン株式会社 | Robot system, robot, robot control apparatus, robot control method, and robot control program |
US9375196B2 (en) * | 2012-07-12 | 2016-06-28 | Covidien Lp | System and method for detecting critical structures using ultrasound |
US20140051049A1 (en) | 2012-08-17 | 2014-02-20 | Intuitive Surgical Operations, Inc. | Anatomical model and method for surgical training |
WO2014041491A1 (en) * | 2012-09-12 | 2014-03-20 | Virtamed Ag | A mixed reality simulation method and system |
EP2895098B1 (en) | 2012-09-17 | 2022-08-10 | Intuitive Surgical Operations, Inc. | Methods and systems for assigning input devices to teleoperated surgical instrument functions |
WO2014047348A1 (en) | 2012-09-19 | 2014-03-27 | Jeff Thramann | Generation of electrical energy in a ski or snowboard |
US10631939B2 (en) | 2012-11-02 | 2020-04-28 | Intuitive Surgical Operations, Inc. | Systems and methods for mapping flux supply paths |
WO2014093824A1 (en) * | 2012-12-14 | 2014-06-19 | The Trustees Of Columbia University In The City Of New York | Markerless tracking of robotic surgical tools |
US20140176661A1 (en) | 2012-12-21 | 2014-06-26 | G. Anthony Reina | System and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional [4-d] virtual mesh and multi-sensory annotation in standard file formats used for digital imaging and communications in medicine (dicom) |
US9091628B2 (en) | 2012-12-21 | 2015-07-28 | L-3 Communications Security And Detection Systems, Inc. | 3D mapping with two orthogonal imaging views |
WO2014106262A1 (en) * | 2012-12-31 | 2014-07-03 | Mako Surgical Corp. | System for image-based robotic surgery |
US10292887B2 (en) * | 2012-12-31 | 2019-05-21 | Mako Surgical Corp. | Motorized joint positioner |
US10507066B2 (en) | 2013-02-15 | 2019-12-17 | Intuitive Surgical Operations, Inc. | Providing information of tools by filtering image areas adjacent to or on displayed images of the tools |
KR20140108047A (en) | 2013-02-28 | 2014-09-05 | 삼성전자주식회사 | Method for tracking a moving object and a controlling apparatus capable of tracking a moving object |
US10314559B2 (en) | 2013-03-14 | 2019-06-11 | Inneroptic Technology, Inc. | Medical device guidance |
MY170323A (en) | 2013-03-15 | 2019-07-17 | Synaptive Medical Inc | Intelligent positioning system and methods therefore |
AU2014231341B2 (en) | 2013-03-15 | 2019-06-06 | Synaptive Medical Inc. | System and method for dynamic validation, correction of registration for surgical navigation |
US9629595B2 (en) | 2013-03-15 | 2017-04-25 | Hansen Medical, Inc. | Systems and methods for localizing, tracking and/or controlling medical instruments |
KR102117273B1 (en) * | 2013-03-21 | 2020-06-01 | 삼성전자주식회사 | Surgical robot system and method for controlling the same |
WO2014165740A1 (en) * | 2013-04-04 | 2014-10-09 | The Board Of Trustees Of The University Of Illinois | Systems and methods for identifying instruments |
KR20140121581A (en) * | 2013-04-08 | 2014-10-16 | 삼성전자주식회사 | Surgical robot system |
US9861295B2 (en) * | 2013-05-21 | 2018-01-09 | Autonomic Technologies, Inc. | System and method for surgical planning and navigation to facilitate placement of a medical device within a target region of a patient |
CN109199388B (en) * | 2013-07-29 | 2022-03-18 | 直观外科手术操作公司 | Shape sensor system with redundant sensing |
CN105188509B (en) * | 2013-08-07 | 2017-08-25 | 奥林巴斯株式会社 | The method of work of endoscopic system and endoscopic system |
WO2015022684A1 (en) * | 2013-08-10 | 2015-02-19 | Needleways Ltd. | Medical needle path display |
US10744646B2 (en) | 2013-08-29 | 2020-08-18 | Wayne State University | Camera control system and method |
JP6335460B2 (en) * | 2013-09-26 | 2018-05-30 | キヤノン株式会社 | Robot system control apparatus, command value generation method, and robot system control method |
JPWO2015056471A1 (en) * | 2013-10-17 | 2017-03-09 | オリンパス株式会社 | Endoscope apparatus and inspection method using endoscope |
JP6000928B2 (en) * | 2013-10-24 | 2016-10-05 | オリンパス株式会社 | Medical manipulator and initialization method for medical manipulator |
EP3063752A4 (en) * | 2013-10-31 | 2017-06-14 | Health Research, Inc. | System and method for a situation and awareness-based intelligent surgical system |
US8880151B1 (en) | 2013-11-27 | 2014-11-04 | Clear Guide Medical, Llc | Surgical needle for a surgical system with optical recognition |
US9622720B2 (en) | 2013-11-27 | 2017-04-18 | Clear Guide Medical, Inc. | Ultrasound system with stereo image guidance or tracking |
CN106030683B (en) * | 2013-12-20 | 2020-10-30 | 直观外科手术操作公司 | Simulator system for medical procedure training |
CN104827480A (en) * | 2014-02-11 | 2015-08-12 | 泰科电子(上海)有限公司 | Automatic calibration method of robot system |
CN106470634B (en) | 2014-03-17 | 2019-06-14 | 直观外科手术操作公司 | System and method for being indicated outside the screen of the instrument in remote control operation medical system |
WO2015148529A1 (en) * | 2014-03-24 | 2015-10-01 | University Of Houston | Interactive systems and methods for real-time laparoscopic navigation |
US10350009B2 (en) | 2014-03-28 | 2019-07-16 | Intuitive Surgical Operations, Inc. | Quantitative three-dimensional imaging and printing of surgical implants |
KR102397254B1 (en) | 2014-03-28 | 2022-05-12 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | Quantitative three-dimensional imaging of surgical scenes |
US10555788B2 (en) | 2014-03-28 | 2020-02-11 | Intuitive Surgical Operations, Inc. | Surgical system with haptic feedback based upon quantitative three-dimensional imaging |
CN111184577A (en) * | 2014-03-28 | 2020-05-22 | 直观外科手术操作公司 | Quantitative three-dimensional visualization of an instrument in a field of view |
KR20160138502A (en) * | 2014-03-28 | 2016-12-05 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | Alignment of q3d models with 3d images |
EP3125807B1 (en) | 2014-03-28 | 2022-05-04 | Intuitive Surgical Operations, Inc. | Quantitative three-dimensional imaging of surgical scenes from multiport perspectives |
ES2617307T3 (en) * | 2014-04-14 | 2017-06-16 | Softbank Robotics Europe | A procedure for locating a robot in a location plane |
KR20150128049A (en) * | 2014-05-08 | 2015-11-18 | 삼성전자주식회사 | Surgical robot and control method thereof |
US20170202624A1 (en) * | 2014-06-08 | 2017-07-20 | M.S.T. Medical Surgery Technologies Ltd | Device and method for assisting laparoscopic surgery utilizing a touch screen |
EP3169252A1 (en) * | 2014-07-14 | 2017-05-24 | KB Medical SA | Anti-skid surgical instrument for use in preparing holes in bone tissue |
USD779678S1 (en) | 2014-07-24 | 2017-02-21 | KindHeart, Inc. | Surgical tray |
WO2016014385A2 (en) * | 2014-07-25 | 2016-01-28 | Covidien Lp | An augmented surgical reality environment for a robotic surgical system |
GB201413991D0 (en) | 2014-08-07 | 2014-09-24 | Ubisense Ltd | Tool tracking |
US9740193B2 (en) | 2014-08-08 | 2017-08-22 | Roboticvisiontech, Inc. | Sensor-based safety features for robotic equipment |
US10070940B2 (en) * | 2014-09-15 | 2018-09-11 | Synaptive Medical (Barbados) Inc. | End effector for a positioning device |
US10828104B2 (en) * | 2014-09-15 | 2020-11-10 | Synaptive Medical (Barbados) Inc. | Surgical navigation system using image segmentation |
US9901406B2 (en) | 2014-10-02 | 2018-02-27 | Inneroptic Technology, Inc. | Affected region display associated with a medical device |
US9833254B1 (en) | 2014-10-03 | 2017-12-05 | Verily Life Sciences Llc | Controlled dissection of biological tissue |
US10499999B2 (en) | 2014-10-09 | 2019-12-10 | Auris Health, Inc. | Systems and methods for aligning an elongate member with an access site |
US10188467B2 (en) | 2014-12-12 | 2019-01-29 | Inneroptic Technology, Inc. | Surgical guidance intersection display |
US9905000B2 (en) * | 2015-02-19 | 2018-02-27 | Sony Corporation | Method and system for surgical tool localization during anatomical surgery |
CA2882968C (en) * | 2015-02-23 | 2023-04-25 | Sulfur Heron Cognitive Systems Inc. | Facilitating generation of autonomous control information |
KR20170127561A (en) | 2015-03-17 | 2017-11-21 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | System and method for on-screen identification of instruments in a remotely operated medical system |
EP3273894B1 (en) * | 2015-03-26 | 2023-03-15 | Biomet Manufacturing, LLC | Anatomic registration probes, systems and methods |
US10939962B1 (en) * | 2015-04-02 | 2021-03-09 | Mazor Robotics Ltd. | Cranial insertion placement verification |
US20160314717A1 (en) * | 2015-04-27 | 2016-10-27 | KindHeart, Inc. | Telerobotic surgery system for remote surgeon training using robotic surgery station coupled to remote surgeon trainee and instructor stations and associated methods |
US20160314712A1 (en) * | 2015-04-27 | 2016-10-27 | KindHeart, Inc. | Telerobotic surgery system for remote surgeon training using robotic surgery station and remote surgeon station and associated methods |
TWI569764B (en) * | 2015-05-20 | 2017-02-11 | 國立交通大學 | Method and system for recognizing multiple instruments during minimally invasive surgery |
US20230148987A1 (en) * | 2015-05-29 | 2023-05-18 | Koninklijke Philips N.V. | Interventional system |
KR20230054760A (en) | 2015-06-09 | 2023-04-25 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | Configuring surgical system with surgical procedures atlas |
US9949700B2 (en) | 2015-07-22 | 2018-04-24 | Inneroptic Technology, Inc. | Medical device approaches |
US20170042626A1 (en) * | 2015-07-31 | 2017-02-16 | Advanced Tactile Imaging, Inc. | Method and probe for providing tactile feedback in laparoscopic surgery |
US20170035287A1 (en) * | 2015-08-04 | 2017-02-09 | Novartis Ag | Dynamic surgical data overlay |
US10136949B2 (en) | 2015-08-17 | 2018-11-27 | Ethicon Llc | Gathering and analyzing data for robotic surgical systems |
US10347155B2 (en) * | 2015-08-20 | 2019-07-09 | Uti Limited Partnership | Suturing training device and method |
WO2017048922A1 (en) | 2015-09-16 | 2017-03-23 | KindHeart, Inc. | Surgical simulation system and associated methods |
US10058393B2 (en) * | 2015-10-21 | 2018-08-28 | P Tech, Llc | Systems and methods for navigation and visualization |
EP3373834A4 (en) | 2015-11-12 | 2019-07-31 | Intuitive Surgical Operations Inc. | Surgical system with training or assist functions |
GB201521922D0 (en) * | 2015-12-10 | 2016-01-27 | Cambridge Medical Robotics Ltd | User interface for a robot |
WO2017098259A1 (en) | 2015-12-10 | 2017-06-15 | Cambridge Medical Robotics Limited | Robotic system |
WO2017117369A1 (en) | 2015-12-31 | 2017-07-06 | Stryker Corporation | System and methods for performing surgery on a patient at a target site defined by a virtual object |
US10154886B2 (en) * | 2016-01-06 | 2018-12-18 | Ethicon Llc | Methods, systems, and devices for controlling movement of a robotic surgical system |
US9675319B1 (en) | 2016-02-17 | 2017-06-13 | Inneroptic Technology, Inc. | Loupe display |
US11064904B2 (en) | 2016-02-29 | 2021-07-20 | Extremity Development Company, Llc | Smart drill, jig, and method of orthopedic surgery |
US10413272B2 (en) | 2016-03-08 | 2019-09-17 | Covidien Lp | Surgical tool with flex circuit ultrasound sensor |
US10631838B2 (en) | 2016-05-03 | 2020-04-28 | Covidien Lp | Devices, systems, and methods for locating pressure sensitive critical structures |
WO2017205351A1 (en) * | 2016-05-23 | 2017-11-30 | Mako Surgical Corp. | Systems and methods for identifying and tracking physical objects during a robotic surgical procedure |
GB2551769B (en) * | 2016-06-30 | 2020-02-19 | Rolls Royce Plc | Methods, apparatus, computer programs and non-transitory computer readable storage mediums for controlling a robot within a volume |
KR102604919B1 (en) | 2016-07-14 | 2023-11-23 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | Secondary instrument control in a computer-assisted teleoperated system |
EP3278759B1 (en) * | 2016-08-02 | 2021-04-07 | P Tech, LLC | Systems for surgical navigation and visualization |
US10055667B2 (en) | 2016-08-03 | 2018-08-21 | X Development Llc | Generating a model for an object encountered by a robot |
US10695134B2 (en) | 2016-08-25 | 2020-06-30 | Verily Life Sciences Llc | Motion execution of a robotic system |
KR102265060B1 (en) * | 2016-10-03 | 2021-06-16 | 버브 서지컬 인크. | Immersive 3D display for robotic surgery |
US10278778B2 (en) | 2016-10-27 | 2019-05-07 | Inneroptic Technology, Inc. | Medical device navigation using a virtual 3D space |
WO2018098196A1 (en) * | 2016-11-23 | 2018-05-31 | Clear Guide Medical, Inc. | System and methods for navigating interventional instrumentation |
AU2017378250B2 (en) | 2016-12-16 | 2023-09-28 | Mako Surgical Corp. | Techniques for detecting errors or loss of accuracy in a surgical robotic system |
EP3574504A1 (en) * | 2017-01-24 | 2019-12-04 | Tietronix Software, Inc. | System and method for three-dimensional augmented reality guidance for use of medical equipment |
DE102017103198A1 (en) * | 2017-02-16 | 2018-08-16 | avateramedical GmBH | Device for determining and retrieving a reference point during a surgical procedure |
US10251709B2 (en) * | 2017-03-05 | 2019-04-09 | Samuel Cho | Architecture, system, and method for developing and robotically performing a medical procedure activity |
CN117885097A (en) * | 2017-03-07 | 2024-04-16 | 直观外科手术操作公司 | Systems and methods for controlling tools having articulatable distal portions |
WO2018208823A1 (en) * | 2017-05-09 | 2018-11-15 | Boston Scientific Scimed, Inc. | Operating room devices, methods, and systems |
JP7317723B2 (en) | 2017-06-28 | 2023-07-31 | オーリス ヘルス インコーポレイテッド | Electromagnetic field distortion detection |
JP7330902B2 (en) | 2017-06-28 | 2023-08-22 | オーリス ヘルス インコーポレイテッド | Electromagnetic distortion detection |
US11284955B2 (en) | 2017-06-29 | 2022-03-29 | Verb Surgical Inc. | Emulation of robotic arms and control thereof in a virtual reality environment |
US11011077B2 (en) | 2017-06-29 | 2021-05-18 | Verb Surgical Inc. | Virtual reality training, simulation, and collaboration in a robotic surgical system |
US10610303B2 (en) | 2017-06-29 | 2020-04-07 | Verb Surgical Inc. | Virtual reality laparoscopic tools |
US11270601B2 (en) * | 2017-06-29 | 2022-03-08 | Verb Surgical Inc. | Virtual reality system for simulating a robotic surgical environment |
US11259879B2 (en) | 2017-08-01 | 2022-03-01 | Inneroptic Technology, Inc. | Selective transparency to assist medical device navigation |
CA3065440C (en) | 2017-08-11 | 2023-10-03 | Brainlab Ag | Video based microscope adjustment |
WO2019040381A1 (en) * | 2017-08-21 | 2019-02-28 | RELIGN Corporation | Arthroscopic devices and methods |
EP3672496A4 (en) * | 2017-08-23 | 2021-04-28 | The General Hospital Corporation | Surgical decision support using a decision theoretic model |
US10772703B2 (en) | 2017-08-25 | 2020-09-15 | Titan Medical Inc. | Methods and apparatuses for positioning a camera of a surgical robotic system to capture images inside a body cavity of a patient during a medical procedure |
US10339670B2 (en) * | 2017-08-29 | 2019-07-02 | Trimble Inc. | 3D tool tracking and positioning using cameras |
JP2020532383A (en) | 2017-09-05 | 2020-11-12 | コヴィディエン リミテッド パートナーシップ | Camera control for surgical robot systems |
CN109589179B (en) * | 2017-10-02 | 2023-01-17 | 吕孝宇 | Mixed reality system and method for determining spatial coordinates of a dental instrument |
US10657726B1 (en) * | 2017-10-02 | 2020-05-19 | International Osseointegration Ai Research And Training Center | Mixed reality system and method for determining spatial coordinates of dental instruments |
US10464209B2 (en) | 2017-10-05 | 2019-11-05 | Auris Health, Inc. | Robotic system with indication of boundary for robotic arm |
US10016900B1 (en) | 2017-10-10 | 2018-07-10 | Auris Health, Inc. | Surgical robotic arm admittance control |
JP6880324B2 (en) | 2017-11-10 | 2021-06-02 | インテュイティブ サージカル オペレーションズ, インコーポレイテッド | Systems and methods for controlling robot manipulators or related tools |
US11534252B2 (en) * | 2017-11-16 | 2022-12-27 | Intuitive Surgical Operations, Inc. | Master/slave registration and control for teleoperation |
WO2019111512A1 (en) * | 2017-12-05 | 2019-06-13 | ソニー・オリンパスメディカルソリューションズ株式会社 | Medical information processing device and information processing method |
GB201800534D0 (en) | 2018-01-12 | 2018-02-28 | Ubisense Ltd | Tool tracking |
US11484365B2 (en) | 2018-01-23 | 2022-11-01 | Inneroptic Technology, Inc. | Medical image guidance |
US10881472B2 (en) * | 2018-02-20 | 2021-01-05 | Verb Surgical Inc. | Correcting a robotic surgery user interface device tracking input |
EP3557191A1 (en) * | 2018-04-20 | 2019-10-23 | CSEM Centre Suisse d'Electronique et de Microtechnique SA - Recherche et Développement | System and method for improving the accuracy of a position of an object |
US10963698B2 (en) | 2018-06-14 | 2021-03-30 | Sony Corporation | Tool handedness determination for surgical videos |
US11534246B2 (en) * | 2018-06-15 | 2022-12-27 | Verb Surgical Inc. | User input device for use in robotic surgery |
DE102019004233B4 (en) | 2018-06-15 | 2022-09-22 | Mako Surgical Corp. | SYSTEMS AND METHODS FOR TRACKING OBJECTS |
US20200015904A1 (en) | 2018-07-16 | 2020-01-16 | Ethicon Llc | Surgical visualization controls |
CN109226967B (en) * | 2018-07-25 | 2021-03-09 | 同高先进制造科技(太仓)有限公司 | Active laser vision steady weld joint tracking system for laser-arc hybrid welding |
US10299864B1 (en) | 2018-08-07 | 2019-05-28 | Sony Corporation | Co-localization of multiple internal organs based on images obtained during surgery |
US10413364B1 (en) | 2018-08-08 | 2019-09-17 | Sony Corporation | Internal organ localization of a subject for providing assistance during surgery |
US11344374B2 (en) * | 2018-08-13 | 2022-05-31 | Verily Life Sciences Llc | Detection of unintentional movement of a user interface device |
US10751883B2 (en) | 2018-08-16 | 2020-08-25 | Mitutoyo Corporation | Robot system with supplementary metrology position coordinates determination system |
US10871366B2 (en) | 2018-08-16 | 2020-12-22 | Mitutoyo Corporation | Supplementary metrology position coordinates determination system for use with a robot |
US11002529B2 (en) | 2018-08-16 | 2021-05-11 | Mitutoyo Corporation | Robot system with supplementary metrology position determination system |
US11745354B2 (en) | 2018-08-16 | 2023-09-05 | Mitutoyo Corporation | Supplementary metrology position coordinates determination system including an alignment sensor for use with a robot |
US11612438B2 (en) * | 2018-09-05 | 2023-03-28 | Point Robotics Medtech Inc. | Navigation system and method for medical operation by a robotic system using a tool |
US10913156B2 (en) | 2018-09-24 | 2021-02-09 | Mitutoyo Corporation | Robot system with end tool metrology position coordinates determination system |
CN109223183A (en) * | 2018-09-30 | 2019-01-18 | 深圳市精锋医疗科技有限公司 | Starting method, readable access to memory and the operating robot of operating robot |
EP3866720A4 (en) * | 2018-10-15 | 2022-07-06 | Mazor Robotics Ltd. | Versatile multi-arm robotic surgical system |
WO2020086345A1 (en) * | 2018-10-22 | 2020-04-30 | Intuitive Surgical Operations, Inc. | Systems and methods for master/tool registration and control for intuitive motion |
WO2020123671A1 (en) * | 2018-12-11 | 2020-06-18 | Project Moray, Inc. | Hybrid-dimensional, augmented reality, and/or registration of user interface and simulation systems for robotic catheters and other uses |
EP3930615A1 (en) * | 2019-02-28 | 2022-01-05 | Koninklijke Philips N.V. | Feedforward continuous positioning control of end-effectors |
US20220125500A1 (en) * | 2019-03-08 | 2022-04-28 | Covidien Lp | Robotic surgery methods |
JP7128135B2 (en) | 2019-03-08 | 2022-08-30 | 富士フイルム株式会社 | Endoscope image learning device, method and program, Endoscope image recognition device |
KR20220004950A (en) | 2019-05-01 | 2022-01-12 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | Systems and Methods for Imaging Devices and Integrated Motion |
DE102019114817B4 (en) * | 2019-06-03 | 2021-12-02 | Karl Storz Se & Co. Kg | Imaging system and method of observation |
US10963733B2 (en) * | 2019-06-17 | 2021-03-30 | Tianzhi Yang | Associating spatial point sets with candidate correspondences |
US11276148B2 (en) | 2019-06-20 | 2022-03-15 | Cilag Gmbh International | Super resolution and color motion artifact correction in a pulsed fluorescence imaging system |
US11360028B2 (en) | 2019-06-20 | 2022-06-14 | Cilag Gmbh International | Super resolution and color motion artifact correction in a pulsed hyperspectral, fluorescence, and laser mapping imaging system |
US11288772B2 (en) | 2019-06-20 | 2022-03-29 | Cilag Gmbh International | Super resolution and color motion artifact correction in a pulsed fluorescence imaging system |
US11793399B2 (en) | 2019-06-20 | 2023-10-24 | Cilag Gmbh International | Super resolution and color motion artifact correction in a pulsed hyperspectral imaging system |
US11398011B2 (en) | 2019-06-20 | 2022-07-26 | Cilag Gmbh International | Super resolution and color motion artifact correction in a pulsed laser mapping imaging system |
US10758309B1 (en) | 2019-07-15 | 2020-09-01 | Digital Surgery Limited | Methods and systems for using computer-vision to enhance surgical tool control during surgeries |
CN112336295A (en) * | 2019-08-08 | 2021-02-09 | 上海安翰医疗技术有限公司 | Method and device for controlling magnetic capsule endoscope, storage medium, and electronic device |
EP4025921A4 (en) | 2019-09-03 | 2023-09-06 | Auris Health, Inc. | Electromagnetic distortion detection and compensation |
US11832996B2 (en) | 2019-12-30 | 2023-12-05 | Cilag Gmbh International | Analyzing surgical trends by a surgical system |
US11648060B2 (en) | 2019-12-30 | 2023-05-16 | Cilag Gmbh International | Surgical system for overlaying surgical instrument data onto a virtual three dimensional construct of an organ |
US11219501B2 (en) | 2019-12-30 | 2022-01-11 | Cilag Gmbh International | Visualization systems using structured light |
US11744667B2 (en) | 2019-12-30 | 2023-09-05 | Cilag Gmbh International | Adaptive visualization by a surgical system |
US11896442B2 (en) | 2019-12-30 | 2024-02-13 | Cilag Gmbh International | Surgical systems for proposing and corroborating organ portion removals |
US11776144B2 (en) | 2019-12-30 | 2023-10-03 | Cilag Gmbh International | System and method for determining, adjusting, and managing resection margin about a subject tissue |
US11284963B2 (en) | 2019-12-30 | 2022-03-29 | Cilag Gmbh International | Method of using imaging devices in surgery |
US11759283B2 (en) | 2019-12-30 | 2023-09-19 | Cilag Gmbh International | Surgical systems for generating three dimensional constructs of anatomical organs and coupling identified anatomical structures thereto |
US11711596B2 (en) | 2020-01-23 | 2023-07-25 | Covidien Lp | System and methods for determining proximity relative to an anatomical structure |
CN111311557A (en) * | 2020-01-23 | 2020-06-19 | 腾讯科技(深圳)有限公司 | Endoscope image processing method, endoscope image processing device, electronic apparatus, and storage medium |
US11623086B2 (en) | 2020-01-24 | 2023-04-11 | Medtronic Xomed, Inc. | System and method for therapy |
US11666755B2 (en) | 2020-01-24 | 2023-06-06 | Medtronic Xomed, Inc. | System and method for therapy |
US11167127B2 (en) | 2020-01-24 | 2021-11-09 | Medtronic Xomed, Inc. | System and method for therapy |
US11167140B2 (en) | 2020-01-24 | 2021-11-09 | Medtronic Xomed, Inc. | System and method for therapy |
KR102426925B1 (en) * | 2020-06-23 | 2022-07-29 | (주)휴톰 | Method and program for acquiring motion information of a surgical robot using 3d simulation |
US20220015841A1 (en) * | 2020-07-15 | 2022-01-20 | Orthosoft Ulc | Robotic device and sterilization unit for surgical instrument |
US20230248456A1 (en) * | 2020-08-19 | 2023-08-10 | Covidien Lp | System and method for depth estimation in surgical robotic system |
CN112107363B (en) * | 2020-08-31 | 2022-08-02 | 上海交通大学 | Ultrasonic fat dissolving robot system based on depth camera and auxiliary operation method |
CN114831738A (en) * | 2020-10-08 | 2022-08-02 | 深圳市精锋医疗科技股份有限公司 | Surgical robot, graphical control device thereof and graphical display method |
US20220133228A1 (en) * | 2020-11-03 | 2022-05-05 | Biosense Webster (Israel) Ltd. | Identification and visualization of non-navigated objects in medical images |
US11992941B2 (en) * | 2020-12-11 | 2024-05-28 | Depuy Ireland Unlimited Company | High accuracy molded navigation arrays, systems, and methods |
CN113967926A (en) * | 2021-01-29 | 2022-01-25 | 航天智造(上海)科技有限责任公司 | Visual sensor equipment for bolt fastening and guiding of industrial robot |
CN113658178B (en) * | 2021-10-14 | 2022-01-25 | 北京字节跳动网络技术有限公司 | Tissue image identification method and device, readable medium and electronic equipment |
CN114098993B (en) * | 2021-11-24 | 2023-05-23 | 重庆金山医疗机器人有限公司 | Main hand pitching information acquisition method |
WO2023180963A1 (en) * | 2022-03-23 | 2023-09-28 | Verb Surgical Inc. | Video-based analysis of stapling events during a surgical procedure using machine learning |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070138992A1 (en) * | 2005-12-20 | 2007-06-21 | Intuitive Surgical Inc. | Medical robotic system with sliding mode control |
Family Cites Families (144)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4209852A (en) | 1974-11-11 | 1980-06-24 | Hyatt Gilbert P | Signal processing and memory arrangement |
US3740779A (en) * | 1971-11-26 | 1973-06-26 | Leveen H | Surgical device |
US4312363A (en) | 1980-02-26 | 1982-01-26 | Senco Products, Inc. | Surgical tissue thickness measuring instrument |
US4753569A (en) | 1982-12-28 | 1988-06-28 | Diffracto, Ltd. | Robot calibration |
US4614366A (en) | 1983-11-18 | 1986-09-30 | Exactident, Inc. | Nail identification wafer |
US4754415A (en) | 1984-10-12 | 1988-06-28 | Diffracto Ltd. | Robotic alignment and part simulation |
US5078140A (en) | 1986-05-08 | 1992-01-07 | Kwoh Yik S | Imaging device - aided robotic stereotaxis system |
US4725965A (en) | 1986-07-23 | 1988-02-16 | American Telephone And Telegraph Company | Method for calibrating a SCARA robot |
US4922909A (en) | 1987-07-17 | 1990-05-08 | Little James H | Video monitoring and reapposition monitoring apparatus and methods |
US4831549A (en) | 1987-07-28 | 1989-05-16 | Brigham Young University | Device and method for correction of robot inaccuracy |
US5579444A (en) | 1987-08-28 | 1996-11-26 | Axiom Bildverarbeitungssysteme Gmbh | Adaptive vision-based controller |
US4826391A (en) | 1988-02-23 | 1989-05-02 | The University Of British Columbia | Manipulator arm position sensing |
JP2698660B2 (en) | 1989-06-12 | 1998-01-19 | 株式会社日立製作所 | Manipulator control method and control device, and manipulator device |
FR2660185B1 (en) | 1990-03-30 | 1998-04-17 | Medirand Inc | SYSTEM AND METHOD FOR INDICATING A POSITION TO BE OPERATED IN THE BODY OF A PATIENT. |
US5086401A (en) | 1990-05-11 | 1992-02-04 | International Business Machines Corporation | Image-directed robotic system for precise robotic surgery including redundant consistency checking |
US5417210A (en) | 1992-05-27 | 1995-05-23 | International Business Machines Corporation | System and method for augmentation of endoscopic surgery |
US5279309A (en) | 1991-06-13 | 1994-01-18 | International Business Machines Corporation | Signaling device and method for monitoring positions in a surgical operation |
US5603318A (en) | 1992-04-21 | 1997-02-18 | University Of Utah Research Foundation | Apparatus and method for photogrammetric surgical localization |
US5657429A (en) | 1992-08-10 | 1997-08-12 | Computer Motion, Inc. | Automated endoscope system optimal positioning |
US7074179B2 (en) | 1992-08-10 | 2006-07-11 | Intuitive Surgical Inc | Method and apparatus for performing minimally invasive cardiac procedures |
US5388059A (en) | 1992-12-30 | 1995-02-07 | University Of Maryland | Computer vision system for accurate monitoring of object pose |
WO1994026167A1 (en) | 1993-05-14 | 1994-11-24 | Sri International | Remote center positioner |
JPH08164148A (en) | 1994-12-13 | 1996-06-25 | Olympus Optical Co Ltd | Surgical operation device under endoscope |
US5836869A (en) | 1994-12-13 | 1998-11-17 | Olympus Optical Co., Ltd. | Image tracking endoscope system |
US6690963B2 (en) | 1995-01-24 | 2004-02-10 | Biosense, Inc. | System for determining the location and orientation of an invasive medical instrument |
JP3539645B2 (en) | 1995-02-16 | 2004-07-07 | 株式会社日立製作所 | Remote surgery support device |
US5797849A (en) | 1995-03-28 | 1998-08-25 | Sonometrics Corporation | Method for carrying out a medical procedure using a three-dimensional tracking and imaging system |
US6122541A (en) | 1995-05-04 | 2000-09-19 | Radionics, Inc. | Head band for frameless stereotactic registration |
US5649021A (en) | 1995-06-07 | 1997-07-15 | David Sarnoff Research Center, Inc. | Method and system for object detection for instrument control |
DE19529950C1 (en) | 1995-08-14 | 1996-11-14 | Deutsche Forsch Luft Raumfahrt | Guiding method for stereo laparoscope in minimal invasive surgery |
EP0910300B1 (en) * | 1996-02-15 | 2003-12-03 | Biosense, Inc. | Site marking probe |
US5797900A (en) | 1996-05-20 | 1998-08-25 | Intuitive Surgical, Inc. | Wrist mechanism for surgical instrument for performing minimally invasive surgery with enhanced dexterity and sensitivity |
US6167296A (en) | 1996-06-28 | 2000-12-26 | The Board Of Trustees Of The Leland Stanford Junior University | Method for volumetric image navigation |
US5950201A (en) | 1996-12-06 | 1999-09-07 | International Business Machines Corporation | Computerized design automation method using a single logical PFVL paradigm |
US6132368A (en) | 1996-12-12 | 2000-10-17 | Intuitive Surgical, Inc. | Multi-component telepresence system and method |
US6331181B1 (en) | 1998-12-08 | 2001-12-18 | Intuitive Surgical, Inc. | Surgical robotic tools, data architecture, and use |
US6714839B2 (en) | 1998-12-08 | 2004-03-30 | Intuitive Surgical, Inc. | Master having redundant degrees of freedom |
US20030163142A1 (en) | 1997-11-27 | 2003-08-28 | Yoav Paltieli | System and method for guiding the movements of a device to a target particularly for medical applications |
CA2284533C (en) | 1998-01-22 | 2008-03-18 | Biosense, Inc. | Intrabody measurement |
US6529765B1 (en) | 1998-04-21 | 2003-03-04 | Neutar L.L.C. | Instrumented and actuated guidance fixture for sterotactic surgery |
EP1079756B1 (en) | 1998-05-28 | 2004-08-04 | Orthosoft, Inc. | Interactive computer-assisted surgical system |
US6425865B1 (en) | 1998-06-12 | 2002-07-30 | The University Of British Columbia | Robotically assisted medical ultrasound |
US5959425A (en) | 1998-10-15 | 1999-09-28 | Fanuc Robotics North America, Inc. | Vision guided automatic robotic path teaching method |
WO2000025185A1 (en) | 1998-10-27 | 2000-05-04 | Irobotics, Inc. | Robotic process planning using templates |
JP4101951B2 (en) | 1998-11-10 | 2008-06-18 | オリンパス株式会社 | Surgical microscope |
WO2000028882A2 (en) | 1998-11-18 | 2000-05-25 | Microdexterity Systems, Inc. | Medical manipulator for use with an imaging device |
US6659939B2 (en) | 1998-11-20 | 2003-12-09 | Intuitive Surgical, Inc. | Cooperative minimally invasive telesurgical system |
US6468265B1 (en) | 1998-11-20 | 2002-10-22 | Intuitive Surgical, Inc. | Performing cardiac surgery without cardioplegia |
US6951535B2 (en) | 2002-01-16 | 2005-10-04 | Intuitive Surgical, Inc. | Tele-medicine system that transmits an entire state of a subsystem |
US6459926B1 (en) | 1998-11-20 | 2002-10-01 | Intuitive Surgical, Inc. | Repositioning and reorientation of master/slave relationship in minimally invasive telesurgery |
US6493608B1 (en) | 1999-04-07 | 2002-12-10 | Intuitive Surgical, Inc. | Aspects of a control system of a minimally invasive surgical apparatus |
US6522906B1 (en) | 1998-12-08 | 2003-02-18 | Intuitive Surgical, Inc. | Devices and methods for presenting and regulating auxiliary information on an image display of a telesurgical system to assist an operator in performing a surgical procedure |
US6720988B1 (en) | 1998-12-08 | 2004-04-13 | Intuitive Surgical, Inc. | Stereo imaging system and method for use in telerobotic systems |
US6770081B1 (en) | 2000-01-07 | 2004-08-03 | Intuitive Surgical, Inc. | In vivo accessories for minimally invasive robotic surgery and methods |
US6799065B1 (en) | 1998-12-08 | 2004-09-28 | Intuitive Surgical, Inc. | Image shifting apparatus and method for a telerobotic system |
WO2000045229A1 (en) | 1999-01-29 | 2000-08-03 | Georgia Tech Research Corporation | Uncalibrated dynamic mechanical system controller |
US6470207B1 (en) | 1999-03-23 | 2002-10-22 | Surgical Navigation Technologies, Inc. | Navigational guidance via computer-assisted fluoroscopic imaging |
WO2000057767A2 (en) | 1999-03-31 | 2000-10-05 | Ultraguide Ltd. | Apparatus and methods for medical diagnostic and for medical guided interventions and therapy |
US6594552B1 (en) | 1999-04-07 | 2003-07-15 | Intuitive Surgical, Inc. | Grip strength with tactile feedback for robotic surgery |
US6424885B1 (en) | 1999-04-07 | 2002-07-23 | Intuitive Surgical, Inc. | Camera referenced control in a minimally invasive surgical apparatus |
US6817972B2 (en) | 1999-10-01 | 2004-11-16 | Computer Motion, Inc. | Heart stabilizer |
US6235038B1 (en) | 1999-10-28 | 2001-05-22 | Medtronic Surgical Navigation Technologies | System for translation of electromagnetic and optical localization systems |
US7689014B2 (en) | 2000-01-18 | 2010-03-30 | Z-Kat Inc | Apparatus and method for measuring anatomical objects using coordinated fluoroscopy |
US6725080B2 (en) | 2000-03-01 | 2004-04-20 | Surgical Navigation Technologies, Inc. | Multiple cannula image guided tool for image guided procedures |
US7819799B2 (en) | 2000-03-16 | 2010-10-26 | Immersion Medical, Inc. | System and method for controlling force applied to and manipulation of medical instruments |
US6856827B2 (en) | 2000-04-28 | 2005-02-15 | Ge Medical Systems Global Technology Company, Llc | Fluoroscopic tracking and visualization system |
US6856826B2 (en) | 2000-04-28 | 2005-02-15 | Ge Medical Systems Global Technology Company, Llc | Fluoroscopic tracking and visualization system |
US6484049B1 (en) | 2000-04-28 | 2002-11-19 | Ge Medical Systems Global Technology Company, Llc | Fluoroscopic tracking and visualization system |
US6782287B2 (en) | 2000-06-27 | 2004-08-24 | The Board Of Trustees Of The Leland Stanford Junior University | Method and apparatus for tracking a medical instrument based on image registration |
US6741911B2 (en) | 2000-09-20 | 2004-05-25 | John Castle Simmons | Natural robot control |
US7194118B1 (en) | 2000-11-10 | 2007-03-20 | Lucid, Inc. | System for optically sectioning and mapping surgically excised tissue |
DE10058123A1 (en) | 2000-11-22 | 2002-05-23 | Wella Ag | Roof support for positioning objects within a space or room using a swing arm, especially for medical use, is designed so that the driving electric motor is mounted high up with the support minimizing the requirement for cleaning |
US6690960B2 (en) | 2000-12-21 | 2004-02-10 | David T. Chen | Video-based surgical targeting system |
US7607440B2 (en) | 2001-06-07 | 2009-10-27 | Intuitive Surgical, Inc. | Methods and apparatus for surgical planning |
US20040243147A1 (en) | 2001-07-03 | 2004-12-02 | Lipow Kenneth I. | Surgical robot and robotic controller |
US7457698B2 (en) | 2001-08-31 | 2008-11-25 | The Board Of Regents Of The University And Community College System On Behalf Of The University Of Nevada, Reno | Coordinated joint motion control system |
US6728599B2 (en) | 2001-09-07 | 2004-04-27 | Computer Motion, Inc. | Modularity system for computer assisted surgery |
US6587750B2 (en) | 2001-09-25 | 2003-07-01 | Intuitive Surgical, Inc. | Removable infinite roll master grip handle and touch sensor for robotic surgery |
US7182770B2 (en) | 2001-10-10 | 2007-02-27 | Medical Designs, Llc | Needle positioning forceps |
AU2002343552B2 (en) | 2001-11-08 | 2006-04-27 | The Johns Hopkins University | System and method for robot targeting under flouroscopy based on image servoing |
CA2369845A1 (en) | 2002-01-31 | 2003-07-31 | Braintech, Inc. | Method and apparatus for single camera 3d vision guided robotics |
US20030210812A1 (en) * | 2002-02-26 | 2003-11-13 | Ali Khamene | Apparatus and method for surgical navigation |
US8010180B2 (en) | 2002-03-06 | 2011-08-30 | Mako Surgical Corp. | Haptic guidance system and method |
US7166112B2 (en) | 2002-03-25 | 2007-01-23 | Depuy Spine, Inc. | Device for determining distance between two points in a surgical site |
US6978167B2 (en) | 2002-07-01 | 2005-12-20 | Claron Technology Inc. | Video pose tracking system and method |
EP1531749A2 (en) | 2002-08-13 | 2005-05-25 | Microbotics Corporation | Microsurgical robot system |
AU2003263003A1 (en) | 2002-08-29 | 2004-03-19 | Computerized Medical Systems, Inc. | Methods and systems for localizing of a medical imaging probe and of a biopsy needle |
AU2003268554A1 (en) | 2002-09-09 | 2004-03-29 | Z-Kat, Inc. | Image guided interventional method and apparatus |
US6826257B2 (en) | 2002-09-13 | 2004-11-30 | Beekley Corporation | Device and method for margin marking of radiography specimens |
US7135992B2 (en) | 2002-12-17 | 2006-11-14 | Evolution Robotics, Inc. | Systems and methods for using multiple hypotheses in a visual simultaneous localization and mapping system |
US7458936B2 (en) | 2003-03-12 | 2008-12-02 | Siemens Medical Solutions Usa, Inc. | System and method for performing probabilistic classification and decision support using multidimensional medical image databases |
AU2004203173A1 (en) | 2003-07-14 | 2005-02-03 | Sunnybrook And Women's College And Health Sciences Centre | Optical image-based position tracking for magnetic resonance imaging |
US7545965B2 (en) | 2003-11-10 | 2009-06-09 | The University Of Chicago | Image modification and detection using massive training artificial neural networks (MTANN) |
US20050102063A1 (en) | 2003-11-12 | 2005-05-12 | Pierre Bierre | 3D point locator system |
US20050193451A1 (en) | 2003-12-30 | 2005-09-01 | Liposonix, Inc. | Articulating arm for medical procedures |
US7689321B2 (en) | 2004-02-13 | 2010-03-30 | Evolution Robotics, Inc. | Robust sensor fusion for mapping and localization in a simultaneous localization and mapping (SLAM) system |
US20050234679A1 (en) | 2004-02-13 | 2005-10-20 | Evolution Robotics, Inc. | Sequential selective integration of sensor data |
WO2005102202A1 (en) | 2004-04-26 | 2005-11-03 | Orthosoft Inc. | Method for permanent calibration based on actual measurement |
DE102004026813A1 (en) | 2004-06-02 | 2005-12-29 | Kuka Roboter Gmbh | Method and device for controlling handling devices |
EP1769390B1 (en) | 2004-06-04 | 2014-12-03 | Stereotaxis, Inc. | User interface for remote control of medical devices |
US7283654B2 (en) * | 2004-08-26 | 2007-10-16 | Lumeniq, Inc. | Dynamic contrast visualization (DCV) |
DE102004042489B4 (en) * | 2004-08-31 | 2012-03-29 | Siemens Ag | Medical examination or treatment facility with associated method |
US7464596B2 (en) | 2004-09-24 | 2008-12-16 | The Boeing Company | Integrated ultrasonic inspection probes, systems, and methods for inspection of composite assemblies |
US7756608B2 (en) | 2005-02-28 | 2010-07-13 | Abb Ab | System for calibration of an industrial robot and a method thereof |
US7664571B2 (en) | 2005-04-18 | 2010-02-16 | Honda Motor Co., Ltd. | Controlling a robot using pose |
US9526587B2 (en) | 2008-12-31 | 2016-12-27 | Intuitive Surgical Operations, Inc. | Fiducial marker design and detection for locating surgical instrument in images |
US8073528B2 (en) | 2007-09-30 | 2011-12-06 | Intuitive Surgical Operations, Inc. | Tool tracking systems, methods and computer products for image guided surgery |
US9867669B2 (en) | 2008-12-31 | 2018-01-16 | Intuitive Surgical Operations, Inc. | Configuration marker design and detection for instrument tracking |
US8971597B2 (en) | 2005-05-16 | 2015-03-03 | Intuitive Surgical Operations, Inc. | Efficient vision and kinematic data fusion for robotic surgical instruments and other applications |
US9789608B2 (en) | 2006-06-29 | 2017-10-17 | Intuitive Surgical Operations, Inc. | Synthetic representation of a surgical robot |
US9492240B2 (en) | 2009-06-16 | 2016-11-15 | Intuitive Surgical Operations, Inc. | Virtual measurement tool for minimally invasive surgery |
US10555775B2 (en) | 2005-05-16 | 2020-02-11 | Intuitive Surgical Operations, Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
US8147503B2 (en) | 2007-09-30 | 2012-04-03 | Intuitive Surgical Operations Inc. | Methods of locating and tracking robotic instruments in robotic surgical systems |
US8108072B2 (en) | 2007-09-30 | 2012-01-31 | Intuitive Surgical Operations, Inc. | Methods and systems for robotic instrument tool tracking with adaptive fusion of kinematics information and image information |
US8398541B2 (en) | 2006-06-06 | 2013-03-19 | Intuitive Surgical Operations, Inc. | Interactive user interfaces for robotic minimally invasive surgical systems |
EP2289453B1 (en) | 2005-06-06 | 2015-08-05 | Intuitive Surgical Operations, Inc. | Laparoscopic ultrasound robotic surgical system |
KR101270912B1 (en) | 2005-06-09 | 2013-06-03 | 이에프에 인더스트릴레 포르슝 운트 엔트빅룽 게엠베하 | Device and method for the contactless determination and measurement of a spatial position and/or a spatial orientation of bodies, method for calibrating and checking, in particular, medical tools, and patterns or structures on, in particular, medical tools |
US20090037278A1 (en) | 2005-07-01 | 2009-02-05 | Searete Llc, A Limited Liability Corporation Of The State Of Delaware | Implementing visual substitution options in media works |
US7835785B2 (en) | 2005-10-04 | 2010-11-16 | Ascension Technology Corporation | DC magnetic-based position and orientation monitoring system for tracking medical instruments |
EP3162318B1 (en) | 2005-10-20 | 2019-10-16 | Intuitive Surgical Operations, Inc. | Auxiliary image display and manipulation on a computer display in a medical robotic system |
US20070161854A1 (en) | 2005-10-26 | 2007-07-12 | Moshe Alamaro | System and method for endoscopic measurement and mapping of internal organs, tumors and other objects |
US20070167702A1 (en) | 2005-12-30 | 2007-07-19 | Intuitive Surgical Inc. | Medical robotic system providing three-dimensional telestration |
US7907166B2 (en) | 2005-12-30 | 2011-03-15 | Intuitive Surgical Operations, Inc. | Stereo telestration for robotic surgery |
WO2007090288A1 (en) | 2006-02-09 | 2007-08-16 | Northern Digital Inc. | Retroreflective marker-tracking systems |
US8526688B2 (en) | 2006-03-09 | 2013-09-03 | General Electric Company | Methods and systems for registration of surgical navigation data and image data |
EP1854425A1 (en) | 2006-05-11 | 2007-11-14 | BrainLAB AG | Position determination for medical devices with redundant position measurement and weighting to prioritise measurements |
US20080065104A1 (en) | 2006-06-13 | 2008-03-13 | Intuitive Surgical, Inc. | Minimally invasive surgical instrument advancement |
US10258425B2 (en) | 2008-06-27 | 2019-04-16 | Intuitive Surgical Operations, Inc. | Medical robotic system providing an auxiliary view of articulatable instruments extending out of a distal end of an entry guide |
US9718190B2 (en) | 2006-06-29 | 2017-08-01 | Intuitive Surgical Operations, Inc. | Tool position and identification indicator displayed in a boundary area of a computer display screen |
US7612805B2 (en) | 2006-07-11 | 2009-11-03 | Neal Solomon | Digital imaging system and methods for selective image filtration |
US20080013809A1 (en) | 2006-07-14 | 2008-01-17 | Bracco Imaging, Spa | Methods and apparatuses for registration in image guided surgery |
EP1886640B1 (en) | 2006-08-08 | 2009-11-18 | BrainLAB AG | Planning method and system for adjusting a free-shaped bone implant |
DE102007021185B4 (en) | 2007-05-05 | 2012-09-20 | Ziehm Imaging Gmbh | X-ray diagnostic device with a plurality of coded marks and a method for determining the position of device parts of the X-ray diagnostic device |
US7962313B2 (en) * | 2007-12-14 | 2011-06-14 | Palo Alto Research Center Incorporated | Method and apparatus for using mobile code for distributed data fusion in networked sensing systems |
US8473031B2 (en) | 2007-12-26 | 2013-06-25 | Intuitive Surgical Operations, Inc. | Medical robotic system with functionality to determine and display a distance indicated by movement of a tool robotically manipulated by an operator |
US20090171332A1 (en) | 2007-12-27 | 2009-07-02 | Intuitive Surgical, Inc. | Medical device with orientable tip for robotically directed laser cutting and biomaterial application |
US8808164B2 (en) | 2008-03-28 | 2014-08-19 | Intuitive Surgical Operations, Inc. | Controlling a robotic surgical tool with a display monitor |
US8169468B2 (en) | 2008-04-26 | 2012-05-01 | Intuitive Surgical Operations, Inc. | Augmented stereoscopic visualization for a surgical robot |
US8086026B2 (en) * | 2008-06-27 | 2011-12-27 | Waldean Schulz | Method and system for the determination of object positions in a volume |
US8830224B2 (en) | 2008-12-31 | 2014-09-09 | Intuitive Surgical Operations, Inc. | Efficient 3-D telestration for local robotic proctoring |
US8184880B2 (en) | 2008-12-31 | 2012-05-22 | Intuitive Surgical Operations, Inc. | Robust sparse image matching for robotic surgery |
US8223193B2 (en) | 2009-03-31 | 2012-07-17 | Intuitive Surgical Operations, Inc. | Targets, fixtures, and workflows for calibrating an endoscopic camera |
US9155592B2 (en) | 2009-06-16 | 2015-10-13 | Intuitive Surgical Operations, Inc. | Virtual measurement tool for minimally invasive surgery |
US8218727B2 (en) | 2009-09-04 | 2012-07-10 | Siemens Medical Solutions Usa, Inc. | System for medical image processing, manipulation and display |
-
2007
- 2007-09-30 US US11/865,016 patent/US8073528B2/en active Active
-
2011
- 2011-09-30 US US13/249,821 patent/US8792963B2/en active Active
-
2014
- 2014-06-25 US US14/314,161 patent/US20150005622A1/en not_active Abandoned
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070138992A1 (en) * | 2005-12-20 | 2007-06-21 | Intuitive Surgical Inc. | Medical robotic system with sliding mode control |
Cited By (34)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11672606B2 (en) | 2005-05-16 | 2023-06-13 | Intuitive Surgical Operations, Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
US11116578B2 (en) | 2005-05-16 | 2021-09-14 | Intuitive Surgical Operations, Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
US11478308B2 (en) | 2005-05-16 | 2022-10-25 | Intuitive Surgical Operations, Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
US10555775B2 (en) | 2005-05-16 | 2020-02-11 | Intuitive Surgical Operations, Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
US10792107B2 (en) | 2005-05-16 | 2020-10-06 | Intuitive Surgical Operations, Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
US10842571B2 (en) | 2005-05-16 | 2020-11-24 | Intuitive Surgical Operations, Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
US9682476B1 (en) | 2015-05-28 | 2017-06-20 | X Development Llc | Selecting robot poses to account for cost |
US9724826B1 (en) | 2015-05-28 | 2017-08-08 | X Development Llc | Selecting physical arrangements for objects to be acted upon by a robot |
US11553984B2 (en) | 2016-06-03 | 2023-01-17 | Covidien Lp | Robotic surgical system with an embedded imager |
US11839433B2 (en) * | 2016-09-22 | 2023-12-12 | Medtronic Navigation, Inc. | System for guided procedures |
US20180078316A1 (en) * | 2016-09-22 | 2018-03-22 | Medtronic Navigation, Inc. | System for Guided Procedures |
US9717607B1 (en) | 2016-10-28 | 2017-08-01 | International Business Machines Corporation | Augmented control of robotic prosthesis by a cognitive system |
US11759271B2 (en) | 2017-04-28 | 2023-09-19 | Stryker Corporation | System and method for indicating mapping of console-based surgical systems |
WO2018217433A1 (en) * | 2017-05-25 | 2018-11-29 | Covidien Lp | Systems and methods for detection of objects within a field of view of an image capture device |
US11553974B2 (en) | 2017-05-25 | 2023-01-17 | Covidien Lp | Systems and methods for detection of objects within a field of view of an image capture device |
US11918299B2 (en) | 2017-05-25 | 2024-03-05 | Covidien Lp | Systems and methods for detection of objects within a field of view of an image capture device |
US11944272B2 (en) | 2017-12-07 | 2024-04-02 | Medtronic Xomed, Inc. | System and method for assisting visualization during a procedure |
US11901065B2 (en) | 2018-05-23 | 2024-02-13 | Verb Surgical Inc. | Surgery evaluation using machine-learning-based surgical video analysis |
US11594325B2 (en) * | 2018-09-12 | 2023-02-28 | Verb Surgical Inc. | Method and system for automatically tracking and managing inventory of surgical tools in operating rooms |
US11963676B2 (en) | 2018-10-16 | 2024-04-23 | Activ Surgical, Inc. | Autonomous methods and systems for tying surgical knots |
US11602402B2 (en) | 2018-12-04 | 2023-03-14 | Globus Medical, Inc. | Drill guide fixtures, cranial insertion fixtures, and related methods and robotic systems |
US11605161B2 (en) | 2019-01-10 | 2023-03-14 | Verily Life Sciences Llc | Surgical workflow and activity detection based on surgical videos |
WO2020146501A1 (en) * | 2019-01-10 | 2020-07-16 | Verily Life Sciences Llc | Surgical workflow and activity detection based on surgical videos |
WO2020176161A1 (en) * | 2019-02-25 | 2020-09-03 | Acessa Health Inc. | Automated ablation control systems |
US11678942B2 (en) | 2019-12-18 | 2023-06-20 | Suzhou MicroPort Orthobot Co., Ltd. | Device, method and system for registration |
WO2021120554A1 (en) * | 2019-12-18 | 2021-06-24 | 苏州微创畅行机器人有限公司 | Registration apparatus, registration method, registration system, navigation operation system, electronic device, and storage medium |
WO2021146339A1 (en) * | 2020-01-14 | 2021-07-22 | Activ Surgical, Inc. | Systems and methods for autonomous suturing |
JPWO2021199305A1 (en) * | 2020-03-31 | 2021-10-07 | ||
WO2021199305A1 (en) * | 2020-03-31 | 2021-10-07 | 日本電気株式会社 | Control device, control system, control method, and recording medium with control program recorded thereon |
JP7323057B2 (en) | 2020-03-31 | 2023-08-08 | 日本電気株式会社 | Control device, control method, and control program |
WO2021202697A1 (en) * | 2020-03-31 | 2021-10-07 | Georgia Tech Research Corporation | Method and system for mri-guided visual servoing |
WO2021211516A1 (en) * | 2020-04-13 | 2021-10-21 | Kaliber Labs Inc. | Systems and methods for computer-assisted shape measurements in video |
US20220005226A1 (en) * | 2020-07-05 | 2022-01-06 | Asensus Surgical Us, Inc. | Camera calibration using measured motion |
CN112621789A (en) * | 2020-12-08 | 2021-04-09 | 广东联航智能科技有限公司 | Control system of robot for double-arm man-machine cooperative operation |
Also Published As
Publication number | Publication date |
---|---|
US20090088634A1 (en) | 2009-04-02 |
US8792963B2 (en) | 2014-07-29 |
US20120020547A1 (en) | 2012-01-26 |
US8073528B2 (en) | 2011-12-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8792963B2 (en) | Methods of determining tissue distances using both kinematic robotic tool position information and image-derived position information | |
US8108072B2 (en) | Methods and systems for robotic instrument tool tracking with adaptive fusion of kinematics information and image information | |
US8147503B2 (en) | Methods of locating and tracking robotic instruments in robotic surgical systems | |
WO2009045827A2 (en) | Methods and systems for tool locating and tool tracking robotic instruments in robotic surgical systems | |
US20230107693A1 (en) | Systems and methods for localizing, tracking and/or controlling medical instruments | |
US11504095B2 (en) | Three-dimensional imaging and modeling of ultrasound image data | |
EP3081184B1 (en) | System and method for fused image based navigation with late marker placement | |
US7889905B2 (en) | Fast 3D-2D image registration method with application to continuously guided endoscopy | |
US20180220883A1 (en) | Image-based global registration system and method applicable to bronchoscopy guidance | |
Mahmoud et al. | On-patient see-through augmented reality based on visual SLAM | |
US20110282151A1 (en) | Image-based localization method and system | |
US20120063644A1 (en) | Distance-based position tracking method and system | |
US20130123801A1 (en) | Method of real-time tracking of moving/flexible surfaces | |
EP2223150A1 (en) | System and method for automatic calibration of tracked ultrasound | |
US10980509B2 (en) | Deformable registration of preoperative volumes and intraoperative ultrasound images from a tracked transducer | |
Doignon et al. | Pose estimation and feature tracking for robot assisted surgery with medical imaging | |
US20230123621A1 (en) | Registering Intra-Operative Images Transformed from Pre-Operative Images of Different Imaging-Modality for Computer Assisted Navigation During Surgery | |
US10832422B2 (en) | Alignment system for liver surgery | |
EP3931799B1 (en) | Interventional device tracking | |
Niemann et al. | Image-Based Modeling and its Application in Image Processing | |
Deguchi et al. | Camera motion tracking of real bronchoscope using epipolar geometry analysis and CT-derived bronchoscopic images | |
Cheng | Stereoscopic Motion Tracking in Biomedical Visualisation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |