EP3566672A1 - Methods for offscreen indication of instruments in a teleoperational medical system - Google Patents
Methods for offscreen indication of instruments in a teleoperational medical system Download PDFInfo
- Publication number
- EP3566672A1 EP3566672A1 EP19181852.5A EP19181852A EP3566672A1 EP 3566672 A1 EP3566672 A1 EP 3566672A1 EP 19181852 A EP19181852 A EP 19181852A EP 3566672 A1 EP3566672 A1 EP 3566672A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- instrument
- view
- tip
- field
- imaging
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 128
- 238000003384 imaging method Methods 0.000 claims abstract description 101
- 230000000007 visual effect Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 abstract description 16
- 230000033001 locomotion Effects 0.000 abstract description 13
- 238000002059 diagnostic imaging Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 46
- 230000003287 optical effect Effects 0.000 description 9
- 238000003780 insertion Methods 0.000 description 5
- 230000037431 insertion Effects 0.000 description 5
- 230000008447 perception Effects 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 239000004065 semiconductor Substances 0.000 description 4
- 210000003484 anatomy Anatomy 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 238000001356 surgical procedure Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000000712 assembly Effects 0.000 description 2
- 238000000429 assembly Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000002405 diagnostic procedure Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 238000005096 rolling process Methods 0.000 description 2
- 230000001225 therapeutic effect Effects 0.000 description 2
- 208000002847 Surgical Wound Diseases 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000001574 biopsy Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000012636 effector Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000002262 irrigation Effects 0.000 description 1
- 238000003973 irrigation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000002560 therapeutic procedure Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- 210000000707 wrist Anatomy 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B34/35—Surgical robots for telesurgery
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
- A61B34/37—Master-slave robots
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
- A61B2017/00017—Electrical control of surgical instruments
- A61B2017/00115—Electrical control of surgical instruments with audible or visual output
- A61B2017/00119—Electrical control of surgical instruments with audible or visual output alarm; indicating an abnormal situation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/08—Accessories or related features not otherwise provided for
- A61B2090/0807—Indication means
- A61B2090/0811—Indication means for the position of a particular part of an instrument with respect to the rest of the instrument, e.g. position of the anvil of a stapling instrument
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/371—Surgical systems with images on a monitor during operation with simultaneous use of two cameras
Definitions
- the present disclosure is directed to systems and methods for performing a teleoperational medical procedure and more particularly to systems and methods for providing an indication of the location of a teleoperational instruments located outside of an endoscope's field of view.
- Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during invasive medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians may insert medical tools to reach a target tissue location.
- Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments.
- Minimally invasive medical tools may also include imaging instruments such as endoscopic instruments. Imaging instruments provide a user with a field of view within the patient anatomy. Some minimally invasive medical tools and imaging instruments may be teleoperated or otherwise computer-assisted.
- instruments may be controlled without being visible to a user in the field of view provided by the imaging instrument. Inadvertent movement of the instrument outside of the field of view, creates a safety risk. Additionally, clinicians may lose track of instruments that are located outside of the field of view. Systems and methods are needed to provide a clinician with an indication of the location of instruments outside of the field of view, while minimizing the occurrence of false indications.
- a medical imaging system comprises a teleoperational assembly configured to control the movement of a medical instrument including an instrument tip and a processing unit including one or more processors.
- the processing unit is configured to determine an instrument tip position and determine a position error associated with the instrument tip position.
- the processing unit is also configured to determine at least one instrument tip bounding volume based upon the position error and determine if the instrument tip is within a field of view of an imaging instrument.
- a method of imaging comprises determining an instrument tip position for a medical instrument controlled by a teleoperational assembly and determining a position error associated with the instrument tip position. The method further comprises determining at least one instrument tip bounding volume based upon the position error and determining if the instrument tip is within a field of view of an imaging instrument.
- position refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian X, Y, Z coordinates).
- orientation refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom - e.g., roll, pitch, and yaw).
- the term “pose” refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (up to six total degrees of freedom).
- the term “shape” refers to a set of poses, positions, or orientations measured along an object.
- a teleoperational medical system for use in, for example, medical procedures including diagnostic, therapeutic, or surgical procedures, is generally indicated by the reference numeral 10.
- the teleoperational medical systems of this disclosure are under the teleoperational control of a surgeon.
- a teleoperational medical system may be under the partial control of a computer programmed to perform the procedure or sub-procedure.
- a fully automated medical system under the full control of a computer programmed to perform the procedure or sub-procedure, may be used to perform procedures or sub-procedures. As shown in FIG.
- the teleoperational medical system 10 generally includes a teleoperational assembly 12 mounted to or near an operating table O on which a patient P is positioned.
- the teleoperational assembly 12 may be referred to as a patient side cart.
- a medical instrument system 14 and an endoscopic imaging system 15 are operably coupled to the teleoperational assembly 12.
- An operator input system 16 allows a surgeon or other type of clinician S to view images of or representing the surgical site and to control the operation of the medical instrument system 14 and/or the endoscopic imaging system 15.
- the operator input system 16 may be located at a surgeon's console, which is usually located in the same room as operating table O. It should be understood, however, that the surgeon S can be located in a different room or a completely different building from the patient P.
- Operator input system 16 generally includes one or more control device(s) for controlling the medical instrument system 14.
- the control device(s) may include one or more of any number of a variety of input devices, such as hand grips, joysticks, trackballs, data gloves, trigger-guns, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and the like.
- control device(s) will be provided with the same degrees of freedom as the medical instruments of the teleoperational assembly to provide the surgeon with telepresence, the perception that the control device(s) are integral with the instruments so that the surgeon has a strong sense of directly controlling instruments as if present at the surgical site.
- the control device(s) may have more or fewer degrees of freedom than the associated medical instruments and still provide the surgeon with telepresence.
- the control device(s) are manual input devices which move with six degrees of freedom, and which may also include an actuatable handle for actuating instruments (for example, for closing grasping jaws, applying an electrical potential to an electrode, delivering a medicinal treatment, and the like).
- the teleoperational assembly 12 supports and manipulates the medical instrument system 14 while the surgeon S views the surgical site through the console 16.
- An image of the surgical site can be obtained by the endoscopic imaging system 15, such as a stereoscopic endoscope, which can be manipulated by the teleoperational assembly 12 to orient the endoscope 15.
- An electronics cart 18 can be used to process the images of the surgical site for subsequent display to the surgeon S through the surgeon's console 16.
- the number of medical instrument systems 14 used at one time will generally depend on the diagnostic or surgical procedure and the space constraints within the operating room among other factors.
- the teleoperational assembly 12 may include a kinematic structure of one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place, generally referred to as a set-up structure) and a teleoperational manipulator.
- the teleoperational assembly 12 includes a plurality of motors that drive inputs on the medical instrument system 14. These motors move in response to commands from the control system (e.g., control system 20).
- the motors include drive systems which when coupled to the medical instrument system 14 may advance the medical instrument into a naturally or surgically created anatomical orifice.
- motorized drive systems may move the distal end of the medical instrument in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motors can be used to actuate an articulable end effector of the instrument for grasping tissue in the jaws of a biopsy device or the like.
- degrees of linear motion e.g., linear motion along the X, Y, Z Cartesian axes
- rotational motion e.g., rotation about the X, Y, Z Cartesian axes
- the teleoperational medical system 10 also includes a control system 20.
- the control system 20 includes at least one memory and at least one processor (not shown), and typically a plurality of processors, for effecting control between the medical instrument system 14, the operator input system 16, and an electronics system 18.
- the control system 20 also includes programmed instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described in accordance with aspects disclosed herein. While control system 20 is shown as a single block in the simplified schematic of FIG. 1A , the system may include two or more data processing circuits with one portion of the processing optionally being performed on or adjacent the teleoperational assembly 12, another portion of the processing being performed at the operator input system 16, and the like.
- control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.
- control system 20 may include one or more servo controllers that receive force and/or torque feedback from the medical instrument system 14. Responsive to the feedback, the servo controllers transmit signals to the operator input system 16. The servo controller(s) may also transmit signals instructing teleoperational assembly 12 to move the medical instrument system(s) 14 and/ or endoscopic imaging system 15 which extend into an internal surgical site within the patient body via openings in the body. Any suitable conventional or specialized servo controller may be used. A servo controller may be separate from, or integrated with, teleoperational assembly 12. In some embodiments, the servo controller and teleoperational assembly are provided as part of a teleoperational arm cart positioned adjacent to the patient's body.
- the teleoperational medical system 10 may further include optional operation and support systems (not shown) such as illumination systems, steering control systems, irrigation systems, and/or suction systems.
- the teleoperational system may include more than one teleoperational assembly and/or more than one operator input system.
- the exact number of manipulator assemblies will depend on the surgical procedure and the space constraints within the operating room, among other factors.
- the operator input systems may be collocated, or they may be positioned in separate locations. Multiple operator input systems allow more than one operator to control one or more manipulator assemblies in various combinations.
- FIG. 1B is a perspective view of the surgeon's console 16.
- the surgeon's console 16 includes a left eye display 32 and a right eye display 34 for presenting the surgeon S with a coordinated stereo view of the surgical site that enables depth perception.
- the console 16 further includes one or more input control devices 36, which in turn cause the teleoperational assembly 12 to manipulate one or more instruments or the endoscopic imaging system.
- the input control devices 36 can provide the same degrees of freedom as their associated instruments 14 to provide the surgeon S with telepresence, or the perception that the input control devices 36 are integral with the instruments 14 so that the surgeon has a strong sense of directly controlling the instruments 14.
- position, force, and tactile feedback sensors may be employed to transmit position, force, and tactile sensations from the instruments 14 back to the surgeon's hands through the input control devices 36.
- FIG. 1C is a perspective view of the electronics cart 18.
- the electronics cart 18 can be coupled with the endoscope 15 and can include a processor to process captured images for subsequent display, such as to a surgeon on the surgeon's console, or on another suitable display located locally and/or remotely.
- the electronics cart 18 can process the captured images to present the surgeon with coordinated stereo images of the surgical site.
- Such coordination can include alignment between the opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope.
- image processing can include the use of previously determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations.
- the electronics cart 18 may also include a display monitor and components of the control system 20.
- FIG. 1D is a perspective view of one embodiment of a teleoperational assembly 12 which may be referred to as a patient side cart.
- the patient side cart 12 shown provides for the manipulation of three surgical tools 26 (e.g., instrument systems 14) and an imaging device 28 (e.g., endoscopic imaging system 15), such as a stereoscopic endoscope used for the capture of images of the site of the procedure.
- the imaging device may transmit signals over a cable 56 to the electronics cart 18.
- Manipulation is provided by teleoperative mechanisms having a number of joints.
- the imaging device 28 and the surgical tools 26 can be positioned and manipulated through incisions in the patient so that a kinematic remote center is maintained at the incision to minimize the size of the incision.
- Images of the surgical site can include images of the distal ends of the surgical tools 26 when they are positioned within the field-of-view of the imaging device 28.
- the patient side cart 22 includes a drivable base 58.
- the drivable base 58 is connected to a telescoping column 57, which allows for adjustment of the height of the arms 54.
- the arms 54 may include a rotating joint 55 that both rotates and moves up and down.
- Each of the arms 54 may be connected to an orienting platform 53.
- the orienting platform 53 may be capable of 360 degrees of rotation.
- the patient side cart 22 may also include a telescoping horizontal cantilever 52 for moving the orienting platform 53 in a horizontal direction.
- each of the arms 54 connects to a manipulator arm 51.
- the manipulator arms 51 may connect directly to a medical instrument 26 via a manipulator spar 59.
- the manipulator arms 51 may be teleoperatable.
- the arms 54 connecting to the orienting platform are not teleoperatable. Rather, such arms 54 are positioned as desired before the surgeon 18 begins operation with the teleoperative components.
- Endoscopic imaging systems may be provided in a variety of configurations including rigid or flexible endoscopes.
- Rigid endoscopes include a rigid tube housing a relay lens system for transmitting an image from a distal end to a proximal end of the endoscope.
- Flexible endoscopes transmit images using one or more flexible optical fibers.
- Endoscopes may be provided with different viewing angles including a 0° viewing angle for forward axial viewing or viewing angles between 0°-90° for forward oblique viewing.
- Digital image based endoscopes have a "chip on the tip" design in which a distal digital sensor such as a one or more charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device store image data.
- CCD charge-coupled device
- CMOS complementary metal oxide semiconductor
- FIG. 2A illustrates a rigid off-axis stereo endoscopic imaging instrument 100 including a handle 102 and a shaft 104 rigidly coupled to the handle.
- a roll adaptor 106 is coupled rotatably coupled to the shaft 104 and/or the handle 102.
- the shaft includes a distal end 108 and a proximal end 110 and houses a distal image storage device, lens systems, optical fibers, or other stereo image capture and transmission components (not shown).
- the shaft 104 extends along an optical axis OA.
- the instrument 100 has an angle of view 112 and a conical optical field of view 114.
- the angle of view is approximately 30° but may be any angle suitable for oblique angle viewing with respect to the optical axis OA.
- the teleoperational assembly e.g., assembly 12
- the imaging instrument 100 may be operated to rotate the imaging instrument 100, including the instrument body 102 and the shaft 104, about the optical axis OA.
- FIG. 2A illustrates the imaging instrument 100 with a -30° or downward angle with respect to the optical axis OA.
- FIG. 2B illustrates the imaging instrument 100 rotated 180° with a +30° or upward angle with respect to the optical axis OA.
- the terms “up”, “down”, “upward, and “downward” are used for illustrative purposes only to label generally opposite directions and are not intended to be limiting.
- FIG. 3 illustrates a surgical workspace 200 surrounding an imaging instrument field of view 202 for an imaging instrument 204 (e.g., imaging instruments 15, 100).
- the imaging instrument 204 has a distal end 205.
- Two surgical instruments 206, 208 e.g., instruments 14
- field of view 202 has a three-dimensional pyramidal frustum shape. If the imaging instrument 204 is a stereoscopic imaging instrument with two imaging devices, the field of view of the imaging instrument is the combined volume of the three-dimensional pyramidal frustums for each imaging device of the imaging instrument.
- the three-dimensional field of view for the imaging instrument or imaging devices may be a conical frustum shape.
- the medical instrument 206 includes a distal instrument tip 210 which in this embodiment is a two-finger actuatable tip including a proximal tip portion 212 a first distal tip portion 214 and a second distal tip portion 216.
- the medical instrument 208 includes a distal instrument tip 218 which in this embodiment is a two-finger actuatable tip including a proximal tip portion 220 a first distal tip portion 222 and a second distal tip portion 224.
- the medical instruments 206, 208 are outside of the field of view of the imaging instrument 204.
- the determination of whether the instruments 206, 208 are inside or outside of the field of view of the imaging instrument 204 may be based upon the calculated location of the instrument distal tips 210, 218 as is described in further detail at FIG. 7 .
- the surgical workspace of FIG. 3 has a world coordinate system (X W , Y W , Z W ) for world space and the distal end of the imaging instrument has an imaging or endoscope coordinate system (X I , Y I , Z I ) for endoscope space.
- FIG. 4A illustrates a displayed image 230 of a field of view 231 of the imaging instrument 204 in the surgical workspace 200.
- the instruments 206, 208 are visible in the displayed image 230.
- the instruments 206, 208 are considered be "on-screen," that is, on a display screen displaying the image 230.
- the field of view 231 has a boundary 232.
- the displayed image has a display or screen coordinate system (X S , Y S , Z S ) for display or screen space. In a stereoscopic imaging system, there may be separate screen coordinate systems for each of the left and right screens.
- FIG. 4B illustrates a displayed image 240 of a field of view 241 of the imaging instrument 204 in the surgical workspace 200.
- the field of view 241 has a boundary 242.
- the field of view of the instrument 204 may be changed, for example, by zooming-in, zooming-out, moving in a pitch motion, moving in a yaw motion, or rolling between an upward viewing angle and a downward viewing angle.
- the instrument 206 remains within the field of view 241 and thus is visible in the displayed image 240.
- the instrument 206 is thus considered to be on-screen.
- the instrument 208 is not in the field of view 241 and thus is not visible in the displayed image 240.
- the instrument 208 is thus considered to be offscreen.
- out-of-view instrument indicators may be visually or audibly presented to increase the clinician's awareness of the location of instruments not visible within the field of view. For example, as shown in FIG. 4B , an out-of-view instrument indicator 244 is provided along the boundary 242 of the field of view 241 to indicate that instrument 208 is located out of the field of view, in the general direction of the indicator.
- the indicator 244 is a graphical bar, but in other embodiments may be a series of dots, or an icon, an alpha-numeric indicator.
- an audible out-of-view indicator such as a beeping sound or a language-based instruction may alert the clinician that the instrument 208 is out of the field of view.
- the audible cue may pan between left and right speakers of the surgeon's console to reinforce the instrument position relative to the view.
- the audible cue may emit from the left or right speaker in correspondence with the left or right hand control associated with the instrument.
- textual information 246 related to the out-of view instrument may be provided to alert the clinician and/or to provide identifying information about the instrument or an instruction to visualize the instrument.
- the use of an out-of-view indicator may be limited to avoid becoming a distraction to the clinician.
- the use of the out-of-view indicator may be context-sensitive such that the out-of-view indicator may only be displayed during certain modes of operation of the teleoperational system.
- the out-of-view indicator may be displayed during a mode of the system in which the operator controls movement of the imaging system, a mode which may be known as a camera control mode.
- the out-of-view indicator may be displayed while the system awaits input from the operator to take control of an associated instrument.
- the out-of-view indicator may be displayed for a few seconds after initiating a mode of the system in which the operator controls movement of the instruments, a mode which may be known as a following mode.
- the out-of-view indicator may be disabled or selectively enabled when the clinician wants to learn about the location of out-of-view instruments.
- the clinician must provide an acknowledgement that the instrument tip is outside of the field of view before operation of the out of view instrument is enabled. Additional warnings or acknowledgements may be used for energy emitting devices, sharp devices, or devices that provide an increased patient risk if used without visualization.
- an out-of-view indicator may be provided for instruments that are within the field of view but not visible due to occluding tissue or other structures.
- FIG. 4C illustrates a displayed image 250 of a field of view 251 of the imaging instrument 204 in the surgical workspace 200 after moving the imaging instrument 204 from the position or orientation it was in for FIG. 4B .
- the field of view 251 has a boundary 252.
- the instrument 206 remains within the field of view 251 and thus is visible in the displayed image 250.
- the instrument 208 is out of the field of view 251 and thus is not visible in the displayed image 250.
- the out-of-view indicator 244 has relocated to a different location along the boundary 252 to reflect the location of the out of view instrument after the imaging instrument has moved.
- the total length of the indicator 244 may remain the same as the imaging instrument moves.
- the indicator may scale to indicate a distance of the out of view instrument from the field of view.
- FIG. 4D illustrates a displayed image 260 of a field of view 261 of the imaging instrument 204 in the surgical workspace 200 after moving the imaging instrument 204 from the position or orientation it was in for FIG. 4C .
- the field of view 261 has a boundary 262.
- the instrument 206 remains within the field of view 261 and thus is visible in the displayed image 260.
- the instrument 208 is out of the field of view 261 and thus is not visible in the displayed image 260.
- the out-of-view indicator 244 has further relocated along the boundary 262 to reflect the location of the out of view instrument after the imaging instrument has moved.
- FIG. 4E illustrates a displayed image 270 of a field of view 271 of the imaging instrument 204 in the surgical workspace 200 after rolling the imaging instrument 204 from the viewing angle up configuration it was in for FIG. 4A to a viewing angle down configuration.
- the field of view 271 has a boundary 272.
- the instruments 206 and 208 are both out of the field of view 271 and thus are not visible in the displayed image 270.
- the out-of-view indicator 244 indicates the general direction of the out of view instrument 208 and an out-of-view indicator 274 indicates the general direction of the out of view instrument 210.
- FIG. 4F illustrates a displayed image 280 of a field of view 281 of the imaging instrument 204 in the surgical workspace 200.
- the field of view 271 has a boundary 282.
- the instruments 206 and 208 are both within the field of view 281 and thus are visible in the displayed image 280.
- a warning indicator 284 may be provided to indicate that the instrument tip 218 is nearing the boundary 282.
- the warning indicator 284 is a ring, but in alternative embodiments, may be another type of graphical symbol or an auditory indicator.
- the indicator may change (e.g., flash) as the tip 218 moves nearer to the boundary 282.
- the warning indicator 284 may appear in the displayed image 280 when any of the instrument tips move within a predefined distance (e.g., 1 cm.) from the boundary 282.
- the determination of whether the instruments 206, 208 are inside or outside of the field of view of the imaging instrument 204 may be based upon the calculated location of the instrument distal tips 210, 218. Because of small error factors associated with the teleoperational system, the instrument, and/or the imaging system, the determination of the location of the instrument distal tips 210, 218 with respect to the imaging instrument 204 has an associated cumulative error factor. To avoid providing false-positive out-of-view indicators to a clinician, the determination of whether an instrument tip is out of the imaging system field of view may be biased by estimating the range of possible locations for the distal tip and suppressing an out-of-view indicator if any or a designated percentage of the estimated possible locations for the distal tip are within the field of view. The sensitivity of the bias may be adjusted based upon the clinician's tolerance for false-positive out-of-view indicators.
- FIG. 5 illustrates the surgical workspace 200 surrounding the imaging instrument field of view 202 for the imaging instrument 204.
- the position errors associated with the instrument tips 210, 218 of medical instruments 206, 208 are shown.
- a set of error bounding volumes is associated with each instrument tip 210, 218.
- the error bounding volumes may represent the predicted locations of the distal and proximal ends of the tip portions to within a high degree of certainty such as 90-99%.
- An error bounding volume 290 represents the predicted locations of the proximal tip portion 212 of the instrument tip 210.
- An error bounding volume 292 represents the predicted locations of the distal tip portions 214, 216 of the instrument tip 210.
- the distal tip portions 214, 216 are arranged in a grip-closed configuration (e.g., an angle between the tip portions is approximately zero) in which the distal tip portions are close together.
- a single bounding volume 292 may be used to approximate the distal tip portions 214, 216.
- a separate bounding volume may be used to approximate the location of each distal tip portion.
- An error bounding volume 294 represents the predicted locations of the proximal tip portion 220 of the instrument tip 218.
- An error bounding volume 296 represents the predicted locations of the distal tip portions 222, 224 of the instrument tip 218.
- the bounding volumes may be used as the basis for determining whether to present an out-of-view indicator to a clinician. For example, both boundary volumes 290, 292 are outside of the field of view 202, and thus an out-of-view indicator would be generated for instrument 206. At least one of the bounding volumes 294, 296 is within the field of view 202 and thus an out-of view indicator would not be generated for instrument 208.
- a bounding volume may also be provided for the field of view in recognition of error associated with the boundaries of the field of view. Similarly, if the bounding volumes of the field of view intersect the bounding volumes of an instrument, an out-of-view indicator will not be presented to a clinician.
- FIG. 6 illustrates a method 300 for determining whether an instrument is outside of a field of view.
- the method 300 includes evaluating instrument tip positions in the endoscope tip coordinate space as described in further detail in FIG. 7 .
- the method 300 includes estimating instrument tip position error with respect to the distal tip of the endoscope as described in further detail in FIG. 8 .
- the method 300 includes determining the instrument tip bounding volumes in screen coordinate space as described in further detail in FIG. 9 .
- the method 300 includes determining if the instrument tip is outside of the endoscope field of view as described in further detail in FIG. 10 .
- an imaging instrument is referred to as an endoscope in FIGS. 6-10 , other imaging instruments may be used.
- FIG. 7 illustrates a method 310 for evaluating instrument tip positions in the endoscope tip coordinate space.
- the forward kinematics of the teleoperational system and medical instruments is evaluated to determine the position of landmarks on the instrument tips, such as the proximal and distal tip portions, within the endoscope coordinate system.
- the evaluation uses forward kinematic and rigid body transformations.
- a process 312 includes evaluating the manipulator (e.g. manipulator 51) and instrument (e.g., instrument 206, 208) forward kinematic position data in the world coordinate space.
- a process 314 includes evaluating the manipulator and endoscope (e.g., imaging instrument 204) forward kinematic position data in the world coordinate space.
- a process 316 includes transforming the instrument tip position data into endoscope tip coordinate space.
- a process 320 includes determining a gripping angle (e.g., the angle between the distal tip portions 214, 216) for the distal tip portions.
- a process 322 includes determining a length of the instrument tip (e.g., a length between the proximal tip portion 212 and the distal tip portions 214, 216).
- the gripping angle and instrument tip length are used to evaluate the position of the proximal and distal instrument tip points.
- the proximal tip portion position is determined in endoscope tip coordinate space.
- a first distal tip portion (e.g., portion 214) position is determined in the endoscope tip coordinate space.
- a second distal tip portion (e.g., portion 216) position is determined in the endoscope tip coordinate space.
- FIG. 8 illustrates a method 330 for estimating instrument tip position error with respect to the endoscope.
- the instrument tip position error can be highly dependent upon configuration with variations ranging, for example, from 0.5 cm to 2.0 cm. An overall upper bound on error may be assumed for all configurations, but this overall approach would likely overestimate error as compared to an estimation for a particular configuration.
- the tip position error is modeled as a summation of error contributed by the major constituent parts of the kinematic chain. For example, when the instruments are working close to the remote center (i.e., a small insertion depth) the majority of the tip position error may be contributed by the set-up joints while very little of the error is contributed by the manipulator.
- the method 330 includes a process 332 for determining the instrument manipulator arm remote center position error and a process 334 for determining the endoscope manipulator arm remote center position error.
- the remote center position of a manipulator arm is determined by its setup joint sensors and fixed mechanical offsets between the links of the kinematic chain of the teleoperational assembly. An end-to-end kinematic calibration procedure may be performed to account for variability in manufacturing and assembly. The resultant position error may be due to non-linearity in the setup joint sensors, resolution of the sensors, and deflection of the assembly during calibration.
- the remote center position of a manipulator arm is independent of whether the arm has a medical instrument or imaging instrument installed.
- the manipulator arm remote center position error is relative between two arms, one of which has an instrument and the other of which has an imaging instrument.
- the method 330 further includes a process 336 for determining an instrument manipulator arm orientation error and a process 342 for determining an instrument manipulator arm orientation error.
- the arm orientation error relates to the error in in the pointing direction of the manipulator spar which is affected by the accuracy of the outer pitch and yaw joints of the manipulator and any misalignment in mounting of the manipulator to the setup structure of the teleoperational assembly.
- the instrument arm orientation error is combined with the insertion depth.
- the endoscope arm orientation error is combined with the endoscope insertion depth.
- an instrument arm unobservable error is determined, and at process 346 an endoscope arm unobservable error is determined.
- the unobservable errors account for errors that cannot be observed from the joint position sensors.
- the primary error is deflection of the setup joints due to loads incurred at the patient body wall. This directly affects the position of the remote center.
- Another error factor that may account for deflection of the instrument shaft is compliance, which may be a function of insertion depth.
- all of the error factors determined in processes 332-346 are combined to determine the proximal portion tip error with respect to the endoscope.
- the method further includes a process 350 for determining instrument wrist error and a process 352 for determining instrument tip length.
- the error factors from processes 350 and 352 are combined with the proximal portion tip error from process 348 to determine distal instrument tip error with respect to the endoscope.
- FIG. 9 illustrates a method 360 for determining the instrument tip bounding volumes in the screen coordinate space.
- the method 360 combines the instrument tip positions from method 310 with the estimated position errors from method 330.
- the position errors are mapped to the screen and the amount of the error to use as a bias for field of view testing is determined.
- the position error is the basis for determining bias for in/out of field of view testing.
- the position error can thus artificially grow or shrink the bounding volumes of the instrument tip portions to affect the balance of false positive and false negative errors when detecting whether an instrument tip is inside or outside of the imaging instruments field of view.
- the method 360 includes a process 362 of retrieving the instrument tip portion positions in endoscope tip space, a process 364 of retrieving a calibrated stereo camera model, and a process 366 of retrieving estimated instrument tip position errors in endoscope tip coordinate space.
- the tip position errors are projected to screen-space (e.g., from the stereo camera model) coordinate system position errors.
- the projected position errors are mapped a bias. When the bias is zero, the system will report a mix of false positive (i.e., incorrectly reporting out-of-view indicator) and false negative (i.e., out-of-view indicator not presented when it should have been presented) errors.
- the system When the bias is large (e.g., 100% bias), the system will only detect an instrument is out of the field of view when it is substantially outside of the field of view.
- the process of mapping the estimated position error to a bias allows the trade-off between no-bias and full-bias to be tuned. Minimize overt false positive errors is important because out-of-view indicators can be distracting and confusing if presented when the instrument tip is clearly visible in the field of view. Too many false positive errors can desensitize the clinician to the alerts such that the clinician comes to ignore subsequent true positive detections.
- FIGS. 11-15 illustrate in greater detail the use of bias.
- the insertion depth of the instrument tip is determined.
- bounding volumes e.g., 290, 292, 294, 296, 522, 524. are created around instrument tip points.
- the instrument tip bounding volumes are determined in the screen space coordinate system. Ordinarily the bounding volumes are not displayed to a user, but optionally they may be.
- FIG. 10 is a method for determining if an instrument tip is outside of an imaging instrument field of view.
- the instrument tip bounding volumes are retrieved for the left eye screen coordinate space, and at process 384, the instrument tip bounding volumes are retrieved for the right eye screen coordinate space.
- a test is conducted to determine whether the bounding volumes are inside or outside of the left field of view.
- a test is conducted to determine whether the bounding volumes are inside or outside of the right eye field of view.
- an out-of-view indicator is presented to the clinician if all of the bounding volumes are outside of the fields of view.
- FIG. 11 illustrates a bias profile 400 for mapping the projected position error to a bias factor to determine how much of the position error to use for biasing the tool out-of-view indicators.
- the bias profile reduces the bias (e.g., bias factor ⁇ 1) when the apparent size of the error is small (e.g., when the position error size is less than 20% of the screen size). This generally occurs when the instrument tip is far from the endoscope.
- This mapping method provides a direct way to tune out-of-view detection process based upon perceived error and may be independent of assumptions about instrument depth.
- FIG. 12-15 illustrates the use of the bias profile 400.
- FIG. 12 illustrates position error 500 for the proximal tip portion 212 of the instrument tip 210.
- a position error 502 is shown for the distal tip portion 214 and a position error 504 is shown for the distal tip portion 216.
- the position error circles 500, 502, 504 represent the estimate of the upper bound of the instrument position error for the tip portion.
- An error radius 506 is also identified.
- a dimension 508 is 10% of the screen size, and a dimension 510 is 20% of the screen size.
- Points 512, 514, 516 correspond to points 212, 214, and 216, respectively and represent an overlay of the system's absolute sense of the instrument tip position relative to the endoscope tip.
- the error radius 506 is smaller than 10% of the screen size. Based upon the bias profile 400, no bias is applied to the tip positions 512-516 and instead the positions 512-516 are used.
- the connecting segments themselves are used as the bounding volumes to determine whether the instrument tip is inside or outside of the field of view.
- FIG. 13 illustrates the instrument tip 210 at a medium distance from the endoscope tip.
- the error radius 520 of the position error is between the 10% dimension 508 and the 20% dimension 510.
- a bounding volume 522 is determined as a percentage (between 0% and 100%) of the full position errors. This bounding volume 522 is used to determine whether the instrument tip is inside or outside of the field of view.
- FIG. 14 illustrates the instrument tip 210 at a near distance from the endoscope tip.
- the error radius is greater than the 20% dimension 510.
- a bounding volume 524 is determined to be the full position error. This bounding volume 524 is used to determine whether the instrument tip is inside or outside of the field of view. As shown in FIG. 15 , if the points 512, 514, 516 were used to determine whether the instrument tip was inside or outside of the field of view, the system would falsely report an out-of-view indicator because points 512, 514, 516 are outside of the field of view.
- the bounding volume 524 is used to test inside/outside field of view, the system determines that the bounding volume 524 is inside the field of view and thus does not report an out-of-view indicator.
- One or more elements in embodiments of the invention may be implemented in software to execute on a processor of a computer system such as control processing system.
- the elements of the embodiments of the invention are essentially the code segments to perform the necessary tasks.
- the program or code segments can be stored in a processor readable storage medium or device that may have been downloaded by way of a computer data signal embodied in a carrier wave over a transmission medium or a communication link.
- the processor readable storage device may include any medium that can store information including an optical medium, semiconductor medium, and magnetic medium.
- Processor readable storage device examples include an electronic circuit; a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device,
- the code segments may be downloaded via computer networks such as the Internet, Intranet, etc.
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Surgery (AREA)
- Life Sciences & Earth Sciences (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Robotics (AREA)
- Medical Informatics (AREA)
- Heart & Thoracic Surgery (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Gynecology & Obstetrics (AREA)
- Radiology & Medical Imaging (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Pathology (AREA)
- Endoscopes (AREA)
Abstract
Description
- This patent application claims priority to and the benefit of the filing date of
U.S. Provisional Patent Application 61/954,442 - The present disclosure is directed to systems and methods for performing a teleoperational medical procedure and more particularly to systems and methods for providing an indication of the location of a teleoperational instruments located outside of an endoscope's field of view.
- Minimally invasive medical techniques are intended to reduce the amount of tissue that is damaged during invasive medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques may be performed through natural orifices in a patient anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians may insert medical tools to reach a target tissue location. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools may also include imaging instruments such as endoscopic instruments. Imaging instruments provide a user with a field of view within the patient anatomy. Some minimally invasive medical tools and imaging instruments may be teleoperated or otherwise computer-assisted. In a teleoperational medical system, instruments may be controlled without being visible to a user in the field of view provided by the imaging instrument. Inadvertent movement of the instrument outside of the field of view, creates a safety risk. Additionally, clinicians may lose track of instruments that are located outside of the field of view. Systems and methods are needed to provide a clinician with an indication of the location of instruments outside of the field of view, while minimizing the occurrence of false indications.
- The embodiments of the invention are summarized by the claims that follow below.
- In one embodiment, a medical imaging system comprises a teleoperational assembly configured to control the movement of a medical instrument including an instrument tip and a processing unit including one or more processors. The processing unit is configured to determine an instrument tip position and determine a position error associated with the instrument tip position. The processing unit is also configured to determine at least one instrument tip bounding volume based upon the position error and determine if the instrument tip is within a field of view of an imaging instrument.
- In another embodiment, a method of imaging comprises determining an instrument tip position for a medical instrument controlled by a teleoperational assembly and determining a position error associated with the instrument tip position. The method further comprises determining at least one instrument tip bounding volume based upon the position error and determining if the instrument tip is within a field of view of an imaging instrument.
- Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures. It is emphasized that, in accordance with the standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion. In addition, the present disclosure may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed.
-
FIG. 1A is a schematic view of a teleoperational medical system, in accordance with an embodiment of the present disclosure. -
FIG. 1B is a perspective view of a surgeon's control console for a teleoperational medical system, in accordance with many embodiments. -
FIG. 1C is a perspective view of a teleoperational medical system electronics cart, in accordance with many embodiments. -
FIG. 1D is a perspective view of a patient side cart, according to one example of principles described herein. -
FIG. 2A illustrates an off-axis endoscopic imaging instrument in first orientation. -
FIG. 2B illustrates the off-axis endoscopic imaging instrument ofFIG. 2A in a second orientation. -
FIG. 2C illustrates the distal end of the off-axis endoscopic imaging instrument ofFIG. 2A . -
FIG. 3 illustrates a surgical workspace surrounding an imaging instrument field of view. -
FIG. 4A illustrates a field of view of a surgical workspace with two instruments visible in the field of view. -
FIG. 4B-4D illustrate other fields of view of the surgical workspace ofFIG. 4A , with one of the instruments outside of the field of view. -
FIG. 4E illustrates another field of view of the surgical workspace ofFIG. 4A , with both of the instruments outside of the field of view. -
FIG. 4F illustrates another field of view of the surgical workspace ofFIG. 4A with one of the instruments located near the boundary of the field of view. -
FIG. 5 illustrates a surgical workspace surrounding an imaging instrument field of view and including instruments with error bounding volumes. -
FIG. 6 is a method for determining whether an instrument is outside of a field of view. -
FIG. 7 is a method for evaluating instrument tip positions. -
FIG. 8 is a method for estimating instrument tip position error in an imaging instrument coordinate space. -
FIG. 9 is a method for determining an instrument tip bounding volume in a screen coordinate space. -
FIG. 10 is a method for determining if an instrument tip is outside of an imaging instrument field of view. -
FIG. 11 is a bias profile used in the method ofFIG. 6 . -
FIGS. 12-15 illustrate the influence of instrument tip distance from the imaging instrument in determining whether the instrument tip is outside of the imaging instrument field of view. - For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosure is intended. In the following detailed description of the aspects of the invention, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be obvious to one skilled in the art that the embodiments of this disclosure may be practiced without these specific details. In other instances well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention.
- Any alterations and further modifications to the described devices, instruments, methods, and any further application of the principles of the present disclosure are fully contemplated as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and/or steps described with respect to one embodiment may be combined with the features, components, and/or steps described with respect to other embodiments of the present disclosure. In addition, dimensions provided herein are for specific examples and it is contemplated that different sizes, dimensions, and/or ratios may be utilized to implement the concepts of the present disclosure. To avoid needless descriptive repetition, one or more components or actions described in accordance with one illustrative embodiment can be used or omitted as applicable from other illustrative embodiments. For the sake of brevity, the numerous iterations of these combinations will not be described separately. For simplicity, in some instances the same reference numbers are used throughout the drawings to refer to the same or like parts.
- The embodiments below will describe various instruments and portions of instruments in terms of their state in three-dimensional space. As used herein, the term "position" refers to the location of an object or a portion of an object in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three degrees of rotational freedom - e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or a portion of an object in at least one degree of translational freedom and to the orientation of that object or portion of the object in at least one degree of rotational freedom (up to six total degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an object.
- Referring to
FIG.1A of the drawings, a teleoperational medical system for use in, for example, medical procedures including diagnostic, therapeutic, or surgical procedures, is generally indicated by thereference numeral 10. As will be described, the teleoperational medical systems of this disclosure are under the teleoperational control of a surgeon. In alternative embodiments, a teleoperational medical system may be under the partial control of a computer programmed to perform the procedure or sub-procedure. In still other alternative embodiments, a fully automated medical system, under the full control of a computer programmed to perform the procedure or sub-procedure, may be used to perform procedures or sub-procedures. As shown inFIG. 1A , the teleoperationalmedical system 10 generally includes ateleoperational assembly 12 mounted to or near an operating table O on which a patient P is positioned. Theteleoperational assembly 12 may be referred to as a patient side cart. Amedical instrument system 14 and anendoscopic imaging system 15 are operably coupled to theteleoperational assembly 12. Anoperator input system 16 allows a surgeon or other type of clinician S to view images of or representing the surgical site and to control the operation of themedical instrument system 14 and/or theendoscopic imaging system 15. - The
operator input system 16 may be located at a surgeon's console, which is usually located in the same room as operating table O. It should be understood, however, that the surgeon S can be located in a different room or a completely different building from the patient P.Operator input system 16 generally includes one or more control device(s) for controlling themedical instrument system 14. The control device(s) may include one or more of any number of a variety of input devices, such as hand grips, joysticks, trackballs, data gloves, trigger-guns, hand-operated controllers, voice recognition devices, touch screens, body motion or presence sensors, and the like. In some embodiments, the control device(s) will be provided with the same degrees of freedom as the medical instruments of the teleoperational assembly to provide the surgeon with telepresence, the perception that the control device(s) are integral with the instruments so that the surgeon has a strong sense of directly controlling instruments as if present at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instruments and still provide the surgeon with telepresence. In some embodiments, the control device(s) are manual input devices which move with six degrees of freedom, and which may also include an actuatable handle for actuating instruments (for example, for closing grasping jaws, applying an electrical potential to an electrode, delivering a medicinal treatment, and the like). - The
teleoperational assembly 12 supports and manipulates themedical instrument system 14 while the surgeon S views the surgical site through theconsole 16. An image of the surgical site can be obtained by theendoscopic imaging system 15, such as a stereoscopic endoscope, which can be manipulated by theteleoperational assembly 12 to orient theendoscope 15. An electronics cart 18 can be used to process the images of the surgical site for subsequent display to the surgeon S through the surgeon'sconsole 16. The number ofmedical instrument systems 14 used at one time will generally depend on the diagnostic or surgical procedure and the space constraints within the operating room among other factors. Theteleoperational assembly 12 may include a kinematic structure of one or more non-servo controlled links (e.g., one or more links that may be manually positioned and locked in place, generally referred to as a set-up structure) and a teleoperational manipulator. Theteleoperational assembly 12 includes a plurality of motors that drive inputs on themedical instrument system 14. These motors move in response to commands from the control system (e.g., control system 20). The motors include drive systems which when coupled to themedical instrument system 14 may advance the medical instrument into a naturally or surgically created anatomical orifice. Other motorized drive systems may move the distal end of the medical instrument in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and in three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motors can be used to actuate an articulable end effector of the instrument for grasping tissue in the jaws of a biopsy device or the like. - The teleoperational
medical system 10 also includes acontrol system 20. Thecontrol system 20 includes at least one memory and at least one processor (not shown), and typically a plurality of processors, for effecting control between themedical instrument system 14, theoperator input system 16, and anelectronics system 18. Thecontrol system 20 also includes programmed instructions (e.g., a computer-readable medium storing the instructions) to implement some or all of the methods described in accordance with aspects disclosed herein. Whilecontrol system 20 is shown as a single block in the simplified schematic ofFIG. 1A , the system may include two or more data processing circuits with one portion of the processing optionally being performed on or adjacent theteleoperational assembly 12, another portion of the processing being performed at theoperator input system 16, and the like. Any of a wide variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a number of separate programs or subroutines, or they may be integrated into a number of other aspects of the teleoperational systems described herein. In one embodiment,control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry. - In some embodiments,
control system 20 may include one or more servo controllers that receive force and/or torque feedback from themedical instrument system 14. Responsive to the feedback, the servo controllers transmit signals to theoperator input system 16. The servo controller(s) may also transmit signals instructingteleoperational assembly 12 to move the medical instrument system(s) 14 and/ orendoscopic imaging system 15 which extend into an internal surgical site within the patient body via openings in the body. Any suitable conventional or specialized servo controller may be used. A servo controller may be separate from, or integrated with,teleoperational assembly 12. In some embodiments, the servo controller and teleoperational assembly are provided as part of a teleoperational arm cart positioned adjacent to the patient's body. - The teleoperational
medical system 10 may further include optional operation and support systems (not shown) such as illumination systems, steering control systems, irrigation systems, and/or suction systems. In alternative embodiments, the teleoperational system may include more than one teleoperational assembly and/or more than one operator input system. The exact number of manipulator assemblies will depend on the surgical procedure and the space constraints within the operating room, among other factors. The operator input systems may be collocated, or they may be positioned in separate locations. Multiple operator input systems allow more than one operator to control one or more manipulator assemblies in various combinations. -
FIG. 1B is a perspective view of the surgeon'sconsole 16. The surgeon'sconsole 16 includes aleft eye display 32 and aright eye display 34 for presenting the surgeon S with a coordinated stereo view of the surgical site that enables depth perception. Theconsole 16 further includes one or moreinput control devices 36, which in turn cause theteleoperational assembly 12 to manipulate one or more instruments or the endoscopic imaging system. Theinput control devices 36 can provide the same degrees of freedom as their associatedinstruments 14 to provide the surgeon S with telepresence, or the perception that theinput control devices 36 are integral with theinstruments 14 so that the surgeon has a strong sense of directly controlling theinstruments 14. To this end, position, force, and tactile feedback sensors (not shown) may be employed to transmit position, force, and tactile sensations from theinstruments 14 back to the surgeon's hands through theinput control devices 36. -
FIG. 1C is a perspective view of theelectronics cart 18. The electronics cart 18 can be coupled with theendoscope 15 and can include a processor to process captured images for subsequent display, such as to a surgeon on the surgeon's console, or on another suitable display located locally and/or remotely. For example, where a stereoscopic endoscope is used, the electronics cart 18 can process the captured images to present the surgeon with coordinated stereo images of the surgical site. Such coordination can include alignment between the opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope. As another example, image processing can include the use of previously determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations. The electronics cart 18 may also include a display monitor and components of thecontrol system 20. -
FIG. 1D is a perspective view of one embodiment of ateleoperational assembly 12 which may be referred to as a patient side cart. Thepatient side cart 12 shown provides for the manipulation of three surgical tools 26 (e.g., instrument systems 14) and an imaging device 28 (e.g., endoscopic imaging system 15), such as a stereoscopic endoscope used for the capture of images of the site of the procedure. The imaging device may transmit signals over acable 56 to theelectronics cart 18. Manipulation is provided by teleoperative mechanisms having a number of joints. Theimaging device 28 and thesurgical tools 26 can be positioned and manipulated through incisions in the patient so that a kinematic remote center is maintained at the incision to minimize the size of the incision. Images of the surgical site can include images of the distal ends of thesurgical tools 26 when they are positioned within the field-of-view of theimaging device 28. - The patient side cart 22 includes a
drivable base 58. Thedrivable base 58 is connected to atelescoping column 57, which allows for adjustment of the height of thearms 54. Thearms 54 may include a rotating joint 55 that both rotates and moves up and down. Each of thearms 54 may be connected to anorienting platform 53. The orientingplatform 53 may be capable of 360 degrees of rotation. The patient side cart 22 may also include a telescopinghorizontal cantilever 52 for moving the orientingplatform 53 in a horizontal direction. - In the present example, each of the
arms 54 connects to amanipulator arm 51. Themanipulator arms 51 may connect directly to amedical instrument 26 via amanipulator spar 59. Themanipulator arms 51 may be teleoperatable. In some examples, thearms 54 connecting to the orienting platform are not teleoperatable. Rather,such arms 54 are positioned as desired before thesurgeon 18 begins operation with the teleoperative components. - Endoscopic imaging systems (e.g.,
systems 15, 28) may be provided in a variety of configurations including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube housing a relay lens system for transmitting an image from a distal end to a proximal end of the endoscope. Flexible endoscopes transmit images using one or more flexible optical fibers. Endoscopes may be provided with different viewing angles including a 0° viewing angle for forward axial viewing or viewing angles between 0°-90° for forward oblique viewing. Digital image based endoscopes have a "chip on the tip" design in which a distal digital sensor such as a one or more charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device store image data. Endoscopic imaging systems may provide two- or three- dimensional images to the viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereo endoscopic images may provide the viewer with more accurate depth perception. Stereo endoscopic instruments employ stereo cameras to capture stereo images of the patient anatomy.FIG. 2A illustrates a rigid off-axis stereoendoscopic imaging instrument 100 including ahandle 102 and ashaft 104 rigidly coupled to the handle. Aroll adaptor 106 is coupled rotatably coupled to theshaft 104 and/or thehandle 102. The shaft includes adistal end 108 and aproximal end 110 and houses a distal image storage device, lens systems, optical fibers, or other stereo image capture and transmission components (not shown). Theshaft 104 extends along an optical axis OA. As shown inFIG. 2C , theinstrument 100 has an angle ofview 112 and a conical optical field ofview 114. In this embodiment, the angle of view is approximately 30° but may be any angle suitable for oblique angle viewing with respect to the optical axis OA. Responsive to manual or teleoperational control, the teleoperational assembly (e.g., assembly 12) may be operated to rotate theimaging instrument 100, including theinstrument body 102 and theshaft 104, about the optical axis OA.FIG. 2A illustrates theimaging instrument 100 with a -30° or downward angle with respect to the optical axis OA.FIG. 2B illustrates theimaging instrument 100 rotated 180° with a +30° or upward angle with respect to the optical axis OA. The terms "up", "down", "upward, and "downward" are used for illustrative purposes only to label generally opposite directions and are not intended to be limiting. -
FIG. 3 illustrates asurgical workspace 200 surrounding an imaging instrument field ofview 202 for an imaging instrument 204 (e.g.,imaging instruments 15, 100). Theimaging instrument 204 has adistal end 205. Twosurgical instruments 206, 208 (e.g., instruments 14) are operable within theworkspace 200. In this embodiment, field ofview 202 has a three-dimensional pyramidal frustum shape. If theimaging instrument 204 is a stereoscopic imaging instrument with two imaging devices, the field of view of the imaging instrument is the combined volume of the three-dimensional pyramidal frustums for each imaging device of the imaging instrument. In alternative embodiments, the three-dimensional field of view for the imaging instrument or imaging devices may be a conical frustum shape. Themedical instrument 206 includes adistal instrument tip 210 which in this embodiment is a two-finger actuatable tip including a proximal tip portion 212 a firstdistal tip portion 214 and a seconddistal tip portion 216. Similarly, themedical instrument 208 includes adistal instrument tip 218 which in this embodiment is a two-finger actuatable tip including a proximal tip portion 220 a firstdistal tip portion 222 and a seconddistal tip portion 224. - In the configuration of
FIG. 3 , themedical instruments imaging instrument 204. The determination of whether theinstruments imaging instrument 204 may be based upon the calculated location of the instrumentdistal tips FIG. 7 . The surgical workspace ofFIG. 3 has a world coordinate system (XW, YW, ZW) for world space and the distal end of the imaging instrument has an imaging or endoscope coordinate system (XI, YI, ZI) for endoscope space. -
FIG. 4A illustrates a displayedimage 230 of a field ofview 231 of theimaging instrument 204 in thesurgical workspace 200. Theinstruments image 230. In this configuration, theinstruments image 230. The field ofview 231 has aboundary 232. The displayed image has a display or screen coordinate system (XS, YS, ZS) for display or screen space. In a stereoscopic imaging system, there may be separate screen coordinate systems for each of the left and right screens. -
FIG. 4B illustrates a displayedimage 240 of a field ofview 241 of theimaging instrument 204 in thesurgical workspace 200. The field ofview 241 has aboundary 242. The field of view of theinstrument 204 may be changed, for example, by zooming-in, zooming-out, moving in a pitch motion, moving in a yaw motion, or rolling between an upward viewing angle and a downward viewing angle. Theinstrument 206 remains within the field ofview 241 and thus is visible in the displayedimage 240. Theinstrument 206 is thus considered to be on-screen. Theinstrument 208 is not in the field ofview 241 and thus is not visible in the displayedimage 240. Theinstrument 208 is thus considered to be offscreen. - Because the
instruments FIG. 4B , an out-of-view instrument indicator 244 is provided along theboundary 242 of the field ofview 241 to indicate thatinstrument 208 is located out of the field of view, in the general direction of the indicator. In this embodiment, theindicator 244 is a graphical bar, but in other embodiments may be a series of dots, or an icon, an alpha-numeric indicator. In addition to or alternative to thevisual indicator 244, an audible out-of-view indicator such as a beeping sound or a language-based instruction may alert the clinician that theinstrument 208 is out of the field of view. The audible cue may pan between left and right speakers of the surgeon's console to reinforce the instrument position relative to the view. Alternatively, the audible cue may emit from the left or right speaker in correspondence with the left or right hand control associated with the instrument. In addition to or alternative to thevisual indicator 244,textual information 246 related to the out-of view instrument may be provided to alert the clinician and/or to provide identifying information about the instrument or an instruction to visualize the instrument. - In various embodiments, the use of an out-of-view indicator may be limited to avoid becoming a distraction to the clinician. The use of the out-of-view indicator may be context-sensitive such that the out-of-view indicator may only be displayed during certain modes of operation of the teleoperational system. For example, the out-of-view indicator may be displayed during a mode of the system in which the operator controls movement of the imaging system, a mode which may be known as a camera control mode. As another example, the out-of-view indicator may be displayed while the system awaits input from the operator to take control of an associated instrument. As another example, the out-of-view indicator may be displayed for a few seconds after initiating a mode of the system in which the operator controls movement of the instruments, a mode which may be known as a following mode. In still other alternative embodiments, the out-of-view indicator may be disabled or selectively enabled when the clinician wants to learn about the location of out-of-view instruments. In some embodiments, the clinician must provide an acknowledgement that the instrument tip is outside of the field of view before operation of the out of view instrument is enabled. Additional warnings or acknowledgements may be used for energy emitting devices, sharp devices, or devices that provide an increased patient risk if used without visualization. In some embodiments an out-of-view indicator may be provided for instruments that are within the field of view but not visible due to occluding tissue or other structures.
-
FIG. 4C illustrates a displayedimage 250 of a field ofview 251 of theimaging instrument 204 in thesurgical workspace 200 after moving theimaging instrument 204 from the position or orientation it was in forFIG. 4B . The field ofview 251 has aboundary 252. Theinstrument 206 remains within the field ofview 251 and thus is visible in the displayedimage 250. Theinstrument 208 is out of the field ofview 251 and thus is not visible in the displayedimage 250. The out-of-view indicator 244 has relocated to a different location along theboundary 252 to reflect the location of the out of view instrument after the imaging instrument has moved. In some embodiments, the total length of theindicator 244 may remain the same as the imaging instrument moves. In alternative embodiments, the indicator may scale to indicate a distance of the out of view instrument from the field of view. -
FIG. 4D illustrates a displayedimage 260 of a field ofview 261 of theimaging instrument 204 in thesurgical workspace 200 after moving theimaging instrument 204 from the position or orientation it was in forFIG. 4C . The field ofview 261 has aboundary 262. Theinstrument 206 remains within the field ofview 261 and thus is visible in the displayedimage 260. Theinstrument 208 is out of the field ofview 261 and thus is not visible in the displayedimage 260. The out-of-view indicator 244 has further relocated along theboundary 262 to reflect the location of the out of view instrument after the imaging instrument has moved. -
FIG. 4E illustrates a displayedimage 270 of a field ofview 271 of theimaging instrument 204 in thesurgical workspace 200 after rolling theimaging instrument 204 from the viewing angle up configuration it was in forFIG. 4A to a viewing angle down configuration. The field ofview 271 has aboundary 272. Theinstruments view 271 and thus are not visible in the displayedimage 270. The out-of-view indicator 244 indicates the general direction of the out ofview instrument 208 and an out-of-view indicator 274 indicates the general direction of the out ofview instrument 210. -
FIG. 4F illustrates a displayedimage 280 of a field ofview 281 of theimaging instrument 204 in thesurgical workspace 200. The field ofview 271 has aboundary 282. Theinstruments view 281 and thus are visible in the displayedimage 280. Awarning indicator 284 may be provided to indicate that theinstrument tip 218 is nearing theboundary 282. In this embodiment, thewarning indicator 284 is a ring, but in alternative embodiments, may be another type of graphical symbol or an auditory indicator. The indicator may change (e.g., flash) as thetip 218 moves nearer to theboundary 282. Thewarning indicator 284 may appear in the displayedimage 280 when any of the instrument tips move within a predefined distance (e.g., 1 cm.) from theboundary 282. - As explained above, the determination of whether the
instruments imaging instrument 204 may be based upon the calculated location of the instrumentdistal tips distal tips imaging instrument 204 has an associated cumulative error factor. To avoid providing false-positive out-of-view indicators to a clinician, the determination of whether an instrument tip is out of the imaging system field of view may be biased by estimating the range of possible locations for the distal tip and suppressing an out-of-view indicator if any or a designated percentage of the estimated possible locations for the distal tip are within the field of view. The sensitivity of the bias may be adjusted based upon the clinician's tolerance for false-positive out-of-view indicators. -
FIG. 5 illustrates thesurgical workspace 200 surrounding the imaging instrument field ofview 202 for theimaging instrument 204. In this illustration, the position errors associated with theinstrument tips medical instruments instrument tip error bounding volume 290 represents the predicted locations of theproximal tip portion 212 of theinstrument tip 210. Anerror bounding volume 292 represents the predicted locations of thedistal tip portions instrument tip 210. In this configuration, thedistal tip portions single bounding volume 292 may be used to approximate thedistal tip portions distal tip portions error bounding volume 294 represents the predicted locations of theproximal tip portion 220 of theinstrument tip 218. Anerror bounding volume 296 represents the predicted locations of thedistal tip portions instrument tip 218. The bounding volumes may be used as the basis for determining whether to present an out-of-view indicator to a clinician. For example, bothboundary volumes view 202, and thus an out-of-view indicator would be generated forinstrument 206. At least one of the boundingvolumes view 202 and thus an out-of view indicator would not be generated forinstrument 208. In alternative embodiments, a bounding volume may also be provided for the field of view in recognition of error associated with the boundaries of the field of view. Similarly, if the bounding volumes of the field of view intersect the bounding volumes of an instrument, an out-of-view indicator will not be presented to a clinician. -
FIG. 6 illustrates amethod 300 for determining whether an instrument is outside of a field of view. Atprocess 302, themethod 300 includes evaluating instrument tip positions in the endoscope tip coordinate space as described in further detail inFIG. 7 . Atprocess 304, themethod 300 includes estimating instrument tip position error with respect to the distal tip of the endoscope as described in further detail inFIG. 8 . Atprocess 306, themethod 300 includes determining the instrument tip bounding volumes in screen coordinate space as described in further detail inFIG. 9 . Atprocess 308, themethod 300 includes determining if the instrument tip is outside of the endoscope field of view as described in further detail inFIG. 10 . Although an imaging instrument is referred to as an endoscope inFIGS. 6-10 , other imaging instruments may be used. -
FIG. 7 illustrates amethod 310 for evaluating instrument tip positions in the endoscope tip coordinate space. In themethod 310, the forward kinematics of the teleoperational system and medical instruments is evaluated to determine the position of landmarks on the instrument tips, such as the proximal and distal tip portions, within the endoscope coordinate system. The evaluation uses forward kinematic and rigid body transformations. Aprocess 312 includes evaluating the manipulator (e.g. manipulator 51) and instrument (e.g.,instrument 206, 208) forward kinematic position data in the world coordinate space. Aprocess 314 includes evaluating the manipulator and endoscope (e.g., imaging instrument 204) forward kinematic position data in the world coordinate space. Aprocess 316 includes transforming the instrument tip position data into endoscope tip coordinate space. Aprocess 320 includes determining a gripping angle (e.g., the angle between thedistal tip portions 214, 216) for the distal tip portions. Aprocess 322 includes determining a length of the instrument tip (e.g., a length between theproximal tip portion 212 and thedistal tip portions 214, 216). At aprocess 318, the gripping angle and instrument tip length are used to evaluate the position of the proximal and distal instrument tip points. At aprocess 324, the proximal tip portion position is determined in endoscope tip coordinate space. At aprocess 326, a first distal tip portion (e.g., portion 214) position is determined in the endoscope tip coordinate space. At aprocess 328, a second distal tip portion (e.g., portion 216) position is determined in the endoscope tip coordinate space. -
FIG. 8 illustrates amethod 330 for estimating instrument tip position error with respect to the endoscope. Because of the large number of links in the kinematic chain of the teleoperational system and the variability of the manipulator poses, the instrument tip position error can be highly dependent upon configuration with variations ranging, for example, from 0.5 cm to 2.0 cm. An overall upper bound on error may be assumed for all configurations, but this overall approach would likely overestimate error as compared to an estimation for a particular configuration. Inmethod 330 the tip position error is modeled as a summation of error contributed by the major constituent parts of the kinematic chain. For example, when the instruments are working close to the remote center (i.e., a small insertion depth) the majority of the tip position error may be contributed by the set-up joints while very little of the error is contributed by the manipulator. - The
method 330 includes aprocess 332 for determining the instrument manipulator arm remote center position error and aprocess 334 for determining the endoscope manipulator arm remote center position error. The remote center position of a manipulator arm is determined by its setup joint sensors and fixed mechanical offsets between the links of the kinematic chain of the teleoperational assembly. An end-to-end kinematic calibration procedure may be performed to account for variability in manufacturing and assembly. The resultant position error may be due to non-linearity in the setup joint sensors, resolution of the sensors, and deflection of the assembly during calibration. The remote center position of a manipulator arm is independent of whether the arm has a medical instrument or imaging instrument installed. The manipulator arm remote center position error is relative between two arms, one of which has an instrument and the other of which has an imaging instrument. - The
method 330 further includes aprocess 336 for determining an instrument manipulator arm orientation error and aprocess 342 for determining an instrument manipulator arm orientation error. The arm orientation error relates to the error in in the pointing direction of the manipulator spar which is affected by the accuracy of the outer pitch and yaw joints of the manipulator and any misalignment in mounting of the manipulator to the setup structure of the teleoperational assembly. Atprocess 338 the instrument arm orientation error is combined with the insertion depth. At 340, the endoscope arm orientation error is combined with the endoscope insertion depth. Atprocess 344 an instrument arm unobservable error is determined, and atprocess 346 an endoscope arm unobservable error is determined. The unobservable errors account for errors that cannot be observed from the joint position sensors. The primary error is deflection of the setup joints due to loads incurred at the patient body wall. This directly affects the position of the remote center. Another error factor that may account for deflection of the instrument shaft is compliance, which may be a function of insertion depth. Atprocess 348, all of the error factors determined in processes 332-346 are combined to determine the proximal portion tip error with respect to the endoscope. The method further includes aprocess 350 for determining instrument wrist error and aprocess 352 for determining instrument tip length. At process 354, the error factors fromprocesses process 348 to determine distal instrument tip error with respect to the endoscope. -
FIG. 9 illustrates amethod 360 for determining the instrument tip bounding volumes in the screen coordinate space. Themethod 360 combines the instrument tip positions frommethod 310 with the estimated position errors frommethod 330. Inmethod 360, the position errors are mapped to the screen and the amount of the error to use as a bias for field of view testing is determined. The position error is the basis for determining bias for in/out of field of view testing. The position error can thus artificially grow or shrink the bounding volumes of the instrument tip portions to affect the balance of false positive and false negative errors when detecting whether an instrument tip is inside or outside of the imaging instruments field of view. - The
method 360 includes aprocess 362 of retrieving the instrument tip portion positions in endoscope tip space, aprocess 364 of retrieving a calibrated stereo camera model, and aprocess 366 of retrieving estimated instrument tip position errors in endoscope tip coordinate space. At process 370, the tip position errors are projected to screen-space (e.g., from the stereo camera model) coordinate system position errors. At aprocess 372, the projected position errors are mapped a bias. When the bias is zero, the system will report a mix of false positive (i.e., incorrectly reporting out-of-view indicator) and false negative (i.e., out-of-view indicator not presented when it should have been presented) errors. When the bias is large (e.g., 100% bias), the system will only detect an instrument is out of the field of view when it is substantially outside of the field of view. The process of mapping the estimated position error to a bias allows the trade-off between no-bias and full-bias to be tuned. Minimize overt false positive errors is important because out-of-view indicators can be distracting and confusing if presented when the instrument tip is clearly visible in the field of view. Too many false positive errors can desensitize the clinician to the alerts such that the clinician comes to ignore subsequent true positive detections. When the instrument tip get close to the endoscope tip (i.e., the instrument tip takes up a large portion of the field of view), the size of the tip position error approaches the cross-sectional dimensions of the field of view volume. In this configuration, without the use of a bias, there would be a high rate of false positive detections.FIGS. 11-15 illustrate in greater detail the use of bias. - At
process 368, the insertion depth of the instrument tip is determined. Atprocess 374, bounding volumes (e.g., 290, 292, 294, 296, 522, 524) are created around instrument tip points. Atprocess 376, the instrument tip bounding volumes are determined in the screen space coordinate system. Ordinarily the bounding volumes are not displayed to a user, but optionally they may be. -
FIG. 10 is a method for determining if an instrument tip is outside of an imaging instrument field of view. At a process 382, the instrument tip bounding volumes are retrieved for the left eye screen coordinate space, and atprocess 384, the instrument tip bounding volumes are retrieved for the right eye screen coordinate space. At aprocess 386, a test is conducted to determine whether the bounding volumes are inside or outside of the left field of view. At aprocess 388, a test is conducted to determine whether the bounding volumes are inside or outside of the right eye field of view. At aprocess 390 an out-of-view indicator is presented to the clinician if all of the bounding volumes are outside of the fields of view. -
FIG. 11 illustrates abias profile 400 for mapping the projected position error to a bias factor to determine how much of the position error to use for biasing the tool out-of-view indicators. Thebias profile 400 applies full bias (bias factor =1) when the apparent size of the position error is large (e.g., when the position error size is 20% or greater of the screen size). This generally occurs when the instrument tip is very close to the endoscope tip. The bias profile reduces the bias (e.g., bias factor <1) when the apparent size of the error is small (e.g., when the position error size is less than 20% of the screen size). This generally occurs when the instrument tip is far from the endoscope. This mapping method provides a direct way to tune out-of-view detection process based upon perceived error and may be independent of assumptions about instrument depth. -
FIG. 12-15 illustrates the use of thebias profile 400.FIG. 12 illustratesposition error 500 for theproximal tip portion 212 of theinstrument tip 210. Aposition error 502 is shown for thedistal tip portion 214 and aposition error 504 is shown for thedistal tip portion 216. The position error circles 500, 502, 504 represent the estimate of the upper bound of the instrument position error for the tip portion. Anerror radius 506 is also identified. Adimension 508 is 10% of the screen size, and adimension 510 is 20% of the screen size.Points points error radius 506 is smaller than 10% of the screen size. Based upon thebias profile 400, no bias is applied to the tip positions 512-516 and instead the positions 512-516 are used. The connecting segments themselves are used as the bounding volumes to determine whether the instrument tip is inside or outside of the field of view. -
FIG. 13 illustrates theinstrument tip 210 at a medium distance from the endoscope tip. In this configuration theerror radius 520 of the position error is between the 10% dimension 508 and the 20% dimension 510. Based upon thebias profile 400, a boundingvolume 522 is determined as a percentage (between 0% and 100%) of the full position errors. This boundingvolume 522 is used to determine whether the instrument tip is inside or outside of the field of view. -
FIG. 14 illustrates theinstrument tip 210 at a near distance from the endoscope tip. In this configuration the error radius is greater than the 20% dimension 510. Based upon thebias profile 400, a boundingvolume 524 is determined to be the full position error. This boundingvolume 524 is used to determine whether the instrument tip is inside or outside of the field of view. As shown inFIG. 15 , if thepoints points volume 524 is used to test inside/outside field of view, the system determines that the boundingvolume 524 is inside the field of view and thus does not report an out-of-view indicator. - One or more elements in embodiments of the invention may be implemented in software to execute on a processor of a computer system such as control processing system. When implemented in software, the elements of the embodiments of the invention are essentially the code segments to perform the necessary tasks. The program or code segments can be stored in a processor readable storage medium or device that may have been downloaded by way of a computer data signal embodied in a carrier wave over a transmission medium or a communication link. The processor readable storage device may include any medium that can store information including an optical medium, semiconductor medium, and magnetic medium. Processor readable storage device examples include an electronic circuit; a semiconductor device, a semiconductor memory device, a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device, The code segments may be downloaded via computer networks such as the Internet, Intranet, etc.
- Note that the processes and displays presented may not inherently be related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the operations described. The required structure for a variety of these systems will appear as elements in the claims. In addition, the embodiments of the invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the invention as described herein.
- While certain exemplary embodiments of the invention have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the embodiments of the invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.
- The present application also includes the following numbered clauses:
- 1. A medical imaging system comprising:
- a teleoperational assembly configured to control the movement of a medical instrument including an instrument tip; and
- a processing unit including one or more processors, wherein the processing unit is configured to
determine an instrument tip position;
determine a position error associated with the instrument tip position;
determine at least one instrument tip bounding volume based upon the position error; and
indicate when the instrument tip is outside a field of view of an imaging instrument based on the at least one instrument tip bounding volume.
- 2. The system of
clause 1 wherein indicating when the instrument tip is outside a field of view includes generating an out-of-view instrument indicator for display with an image of the field of view. - 3. The system of
clause 2 wherein the out-of-view instrument indicator includes a graphical bar extending along at least one edge of the image of the field of view. - 4. The system of
clause 2 wherein the out-of-view instrument indicator includes a textual display related to the medical instrument. - 5. The system of
clause 1 wherein indicating when the instrument tip is outside a field of view includes generating an audible out-of-view instrument indication. - 6. The system of
clause 1 wherein the field of view of the imaging instrument is a stereo field of view of a stereo imaging instrument. - 7. The system of
clause 1 wherein the processing unit is further configured to
recognize a context for persistent or temporary display of the out-of-view indicator if the instrument tip is determined to not be within the field of view of the imaging instrument. - 8. The system of
clause 1 wherein the position error associated with the instrument tip is determined from kinematic variability in the teleoperational assembly. - 9. The system of
clause 1 wherein the position error associated with the instrument tip is determined from kinematic variability in the medical instrument. - 10. The system of
clause 1 wherein the at least one instrument tip bounding volume is further based upon a distance of the instrument tip from a distal end of the imaging instrument. - 11. The system of
clause 1 wherein the at least one instrument tip bounding volume includes a first volume associated with a distal end of the instrument tip and a second volume associated with a proximal end of the instrument tip. - 12. The system of
clause 1 wherein indicating when the instrument tip is outside a field of view of an imaging instrument includes determining that the entire instrument tip bounding volume is outside the field of view of the imaging instrument. - 13. The system of
clause 1 wherein the processing unit is further configured to
generate a warning indicator if the instrument tip is within the field of view of the imaging instrument and within a predefined proximity to an edge of the field of view. - 14. The system of clause 13 wherein the warning indicator includes a visual indicator for display with an image of the field of view.
- 15. The system of
clause 1 further comprising the instrument. - 16. A method of imaging, the method comprising:
- determining an instrument tip position for a medical instrument controlled by a teleoperational assembly;
- determining a position error associated with the instrument tip position;
- determining at least one instrument tip bounding volume based upon the position error; and
- indicating when the instrument tip is outside a field of view of an imaging instrument based on the at least one instrument tip bounding volume.
- 17. The method of
clause 16 indicating when the instrument tip is outside a field of view includes generating an out-of-view instrument indicator for display with an image of the field of view. - 18. The method of clause 17 wherein the out-of-view instrument indicator includes a graphical bar extending along at least one edge of the image of the field of view.
- 19. The method of clause 17 wherein the out-of-view instrument indicator includes textual information about the medical instrument.
- 20. The method of
clause 16 wherein indicating when the instrument tip is outside a field of view includes generating an audible out-of-view instrument indication - 21. The method of
clause 16 wherein the field of view of the imaging instrument is a stereo field of view of a stereo imaging instrument. - 22. The method of
clause 16 wherein determining a position error associated with the instrument tip position includes determining the position error from kinematic variability in the teleoperational assembly. - 23. The method of
clause 16 wherein determining a position error associated with the instrument tip position includes determining the position error from kinematic variability in the medical instrument. - 24. The method of
clause 16 wherein determining the at least one instrument tip bounding volume includes determining the bounding volume based upon a distance of the instrument tip from a distal end of the imaging instrument. - 25. The method of
clause 16 wherein determining the at least one instrument tip bounding volume includes determining the bounding volume based upon a projected size of the position error on a display. - 26. The method of
clause 16 wherein the at least one instrument tip bounding volume includes a first volume associated with a distal end of the instrument tip and a second volume associated with a proximal end of the instrument tip. - 27. The method of
clause 16 wherein indicating when the instrument tip is outside a field of view of an imaging instrument includes determining that the entire instrument tip bounding volume is outside the field of view of the imaging instrument. - 28. The method of
clause 16 further comprising generating a warning indicator if the instrument tip is within the field of view of the imaging instrument and within a predefined proximity to an edge of the field of view. - 29. The method of
clause 28 wherein the warning indicator includes a visual indicator for display with an image of the field of view. - 30. The method of
clause 16 further comprising
receiving an acknowledgement of an out-of-view instrument status for the medical instrument and
responsive to the receipt of the out-of-view instrument status, enabling functionality of the medical instrument.
Claims (15)
- A method of imaging, the method comprising:determining an instrument tip position for a medical instrument controlled by a teleoperational assembly;determining a position error associated with the instrument tip position;determining at least one instrument tip bounding volume based upon the position error; andindicating when the instrument tip is outside a field of view of an imaging instrument based on the at least one instrument tip bounding volume by generating an out-of-view instrument indicator.
- The method of claim 1 wherein generating an out-of-view instrument indicator includes generating the out-of-view instrument indicator for display with an image of the field of view.
- The method of claim 2 wherein the out-of-view instrument indicator includes a graphical bar extending along at least one edge of the image of the field of view.
- The method of claim 2 wherein the out-of-view instrument indicator includes textual information about the medical instrument.
- The method of claim 1 wherein generating an out-of-view instrument indicator includes generating an audible out-of-view instrument indication.
- The method of claim 1 wherein the field of view of the imaging instrument is a stereo field of view of a stereo imaging instrument.
- The method of claim 1 wherein determining a position error associated with the instrument tip position includes determining the position error from kinematic variability in the teleoperational assembly.
- The method of claim 1 wherein determining a position error associated with the instrument tip position includes determining the position error from kinematic variability in the medical instrument.
- The method of claim 1 wherein determining the at least one instrument tip bounding volume includes determining a bounding volume based upon a distance of the instrument tip from a distal end of the imaging instrument.
- The method of claim 1 wherein determining the at least one instrument tip bounding volume includes determining a bounding volume based upon a projected size of the position error on a display.
- The method of claim 1 wherein the at least one instrument tip bounding volume includes a first volume associated with a distal end of the instrument tip and a second volume associated with a proximal end of the instrument tip.
- The method of claim 1 wherein indicating when the instrument tip is outside a field of view of an imaging instrument includes determining that at least one instrument tip bounding volume is entirely outside the field of view of the imaging instrument.
- The method of claim 1 further comprising generating a warning indicator if the instrument tip is within the field of view of the imaging instrument and within a predefined proximity to an edge of the field of view.
- The method of claim 13 wherein the warning indicator includes a visual indicator for display with an image of the field of view.
- The method of claim 1 further comprising:receiving an acknowledgement of an out-of-view instrument status for the medical instrument; andresponsive to the received out-of-view instrument status, enabling functionality of the medical instrument.
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201461954442P | 2014-03-17 | 2014-03-17 | |
PCT/US2015/021109 WO2015142956A1 (en) | 2014-03-17 | 2015-03-17 | Systems and methods for offscreen indication of instruments in a teleoperational medical system |
EP15765724.8A EP3119339B1 (en) | 2014-03-17 | 2015-03-17 | Systems and methods for offscreen indication of instruments in a teleoperational medical system |
Related Parent Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15765724.8A Division EP3119339B1 (en) | 2014-03-17 | 2015-03-17 | Systems and methods for offscreen indication of instruments in a teleoperational medical system |
EP15765724.8A Division-Into EP3119339B1 (en) | 2014-03-17 | 2015-03-17 | Systems and methods for offscreen indication of instruments in a teleoperational medical system |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3566672A1 true EP3566672A1 (en) | 2019-11-13 |
EP3566672B1 EP3566672B1 (en) | 2023-08-23 |
Family
ID=54145247
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15765724.8A Active EP3119339B1 (en) | 2014-03-17 | 2015-03-17 | Systems and methods for offscreen indication of instruments in a teleoperational medical system |
EP19181852.5A Active EP3566672B1 (en) | 2014-03-17 | 2015-03-17 | Methods for offscreen indication of instruments in a teleoperational medical system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15765724.8A Active EP3119339B1 (en) | 2014-03-17 | 2015-03-17 | Systems and methods for offscreen indication of instruments in a teleoperational medical system |
Country Status (6)
Country | Link |
---|---|
US (2) | US11317979B2 (en) |
EP (2) | EP3119339B1 (en) |
JP (3) | JP6542252B2 (en) |
KR (1) | KR102363661B1 (en) |
CN (2) | CN106470634B (en) |
WO (1) | WO2015142956A1 (en) |
Families Citing this family (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106470634B (en) | 2014-03-17 | 2019-06-14 | 直观外科手术操作公司 | System and method for being indicated outside the screen of the instrument in remote control operation medical system |
CN106455920B (en) * | 2014-09-05 | 2018-08-07 | 奥林巴斯株式会社 | Endoscopic system |
JP6766061B2 (en) | 2015-03-17 | 2020-10-07 | インテュイティブ サージカル オペレーションズ, インコーポレイテッド | Systems and methods for rendering on-screen identification of instruments in remote-controlled medical systems |
US10610315B2 (en) | 2015-03-17 | 2020-04-07 | Intuitive Surgical Operations, Inc. | Systems and methods for onscreen identification of instruments in a teleoperational medical system |
CN109496135B (en) * | 2016-06-01 | 2021-10-26 | 恩达马斯特有限公司 | Endoscopy system component |
EP3496646A4 (en) * | 2016-08-12 | 2020-04-01 | Intuitive Surgical Operations Inc. | Systems and methods for onscreen menus in a teleoperational medical system |
US11370113B2 (en) * | 2016-09-06 | 2022-06-28 | Verily Life Sciences Llc | Systems and methods for prevention of surgical mistakes |
JP2020510474A (en) * | 2017-03-07 | 2020-04-09 | インテュイティブ サージカル オペレーションズ, インコーポレイテッド | System and method for controlling a tool having an articulatable distal portion |
JP2018202032A (en) * | 2017-06-08 | 2018-12-27 | 株式会社メディカロイド | Remote control apparatus for medical equipment |
CN117598791A (en) * | 2017-07-13 | 2024-02-27 | 直观外科手术操作公司 | System and method for switching control between multiple instrument arms |
WO2019032450A1 (en) * | 2017-08-08 | 2019-02-14 | Intuitive Surgical Operations, Inc. | Systems and methods for rendering alerts in a display of a teleoperational system |
KR102222651B1 (en) | 2017-11-10 | 2021-03-05 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | Systems and methods for controlling robot manipulators or associated tools |
WO2019148286A1 (en) * | 2018-02-02 | 2019-08-08 | Intellijoint Surgical Inc. | Operating room remote monitoring |
US20210030497A1 (en) * | 2019-07-31 | 2021-02-04 | Auris Health, Inc. | Apparatus, systems, and methods to facilitate instrument visualization |
EP3804630A1 (en) * | 2019-10-10 | 2021-04-14 | Koninklijke Philips N.V. | Ultrasound object zoom tracking |
US11844497B2 (en) * | 2020-02-28 | 2023-12-19 | Covidien Lp | Systems and methods for object measurement in minimally invasive robotic surgery |
CN114099005B (en) * | 2021-11-24 | 2023-09-15 | 重庆金山医疗机器人有限公司 | Method for judging whether instrument is in visual field or is shielded or not and energy display method |
JPWO2023100234A1 (en) * | 2021-11-30 | 2023-06-08 | ||
US20230210579A1 (en) * | 2021-12-30 | 2023-07-06 | Verb Surgical Inc. | Real-time surgical tool presence/absence detection in surgical videos |
JP2023180371A (en) * | 2022-06-09 | 2023-12-21 | 株式会社メディカロイド | Surgery system |
WO2024033828A1 (en) * | 2022-08-10 | 2024-02-15 | Medical Microinstruments Inc. | Method for controlling a slave device, controlled by a master device in a robotic system for medical or surgical teleoperation, taking into account limits of a field of view, and related robotic system |
WO2024141948A1 (en) * | 2022-12-29 | 2024-07-04 | Medical Microinstruments Inc. | Control method for the movement of a robotic surgical instrument exiting or entering the field of view of a viewing system, and related robotic system for surgery |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090088773A1 (en) * | 2007-09-30 | 2009-04-02 | Intuitive Surgical, Inc. | Methods of locating and tracking robotic instruments in robotic surgical systems |
US20090192523A1 (en) * | 2006-06-29 | 2009-07-30 | Intuitive Surgical, Inc. | Synthetic representation of a surgical instrument |
Family Cites Families (37)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4150326A (en) * | 1977-09-19 | 1979-04-17 | Unimation, Inc. | Trajectory correlation and error detection method and apparatus |
US6659939B2 (en) * | 1998-11-20 | 2003-12-09 | Intuitive Surgical, Inc. | Cooperative minimally invasive telesurgical system |
AU2001292836A1 (en) | 2000-09-23 | 2002-04-02 | The Board Of Trustees Of The Leland Stanford Junior University | Endoscopic targeting method and system |
US7607440B2 (en) | 2001-06-07 | 2009-10-27 | Intuitive Surgical, Inc. | Methods and apparatus for surgical planning |
JP4047567B2 (en) * | 2001-10-05 | 2008-02-13 | オリンパス株式会社 | Surgical system |
US7170677B1 (en) * | 2002-01-25 | 2007-01-30 | Everest Vit | Stereo-measurement borescope with 3-D viewing |
US7145478B2 (en) * | 2002-12-17 | 2006-12-05 | Evolution Robotics, Inc. | Systems and methods for controlling a density of visual landmarks in a visual simultaneous localization and mapping system |
JP2005040205A (en) * | 2003-07-23 | 2005-02-17 | Olympus Corp | Three-dimensional endoscope apparatus |
JP2005143918A (en) | 2003-11-17 | 2005-06-09 | Olympus Corp | Remote operation support system |
US20050137751A1 (en) | 2003-12-05 | 2005-06-23 | Cox Damon K. | Auto-diagnostic method and apparatus |
US7972298B2 (en) * | 2004-03-05 | 2011-07-05 | Hansen Medical, Inc. | Robotic catheter system |
US7626569B2 (en) * | 2004-10-25 | 2009-12-01 | Graphics Properties Holdings, Inc. | Movable audio/video communication interface system |
US9789608B2 (en) * | 2006-06-29 | 2017-10-17 | Intuitive Surgical Operations, Inc. | Synthetic representation of a surgical robot |
US8073528B2 (en) * | 2007-09-30 | 2011-12-06 | Intuitive Surgical Operations, Inc. | Tool tracking systems, methods and computer products for image guided surgery |
US8398541B2 (en) * | 2006-06-06 | 2013-03-19 | Intuitive Surgical Operations, Inc. | Interactive user interfaces for robotic minimally invasive surgical systems |
US20070005002A1 (en) * | 2005-06-30 | 2007-01-04 | Intuitive Surgical Inc. | Robotic surgical instruments for irrigation, aspiration, and blowing |
CN103142309B (en) * | 2005-10-20 | 2015-06-17 | 直观外科手术操作公司 | Auxiliary image display and manipulation on computer display in medical robotic system |
US8190238B2 (en) * | 2005-12-09 | 2012-05-29 | Hansen Medical, Inc. | Robotic catheter system and methods |
US8469945B2 (en) | 2006-01-25 | 2013-06-25 | Intuitive Surgical Operations, Inc. | Center robotic arm with five-bar spherical linkage for endoscopic camera |
US7841980B2 (en) * | 2006-05-11 | 2010-11-30 | Olympus Medical Systems Corp. | Treatment system, trocar, treatment method and calibration method |
US10008017B2 (en) * | 2006-06-29 | 2018-06-26 | Intuitive Surgical Operations, Inc. | Rendering tool information as graphic overlays on displayed images of tools |
US9718190B2 (en) | 2006-06-29 | 2017-08-01 | Intuitive Surgical Operations, Inc. | Tool position and identification indicator displayed in a boundary area of a computer display screen |
US9089256B2 (en) * | 2008-06-27 | 2015-07-28 | Intuitive Surgical Operations, Inc. | Medical robotic system providing an auxiliary view including range of motion limitations for articulatable instruments extending out of a distal end of an entry guide |
DE102007030907A1 (en) | 2007-07-03 | 2009-01-08 | Manroland Ag | Method for driving a paddle wheel and control device and drive device therefor |
US8808164B2 (en) * | 2008-03-28 | 2014-08-19 | Intuitive Surgical Operations, Inc. | Controlling a robotic surgical tool with a display monitor |
US8086026B2 (en) * | 2008-06-27 | 2011-12-27 | Waldean Schulz | Method and system for the determination of object positions in a volume |
US8418073B2 (en) * | 2009-03-09 | 2013-04-09 | Intuitive Surgical Operations, Inc. | User interfaces for electrosurgical tools in robotic surgical systems |
US8579012B2 (en) | 2009-03-27 | 2013-11-12 | Novelis Inc. | Continuous casting apparatus for casting strip of variable width |
US8423186B2 (en) | 2009-06-30 | 2013-04-16 | Intuitive Surgical Operations, Inc. | Ratcheting for master alignment of a teleoperated minimally-invasive surgical instrument |
US8918211B2 (en) * | 2010-02-12 | 2014-12-23 | Intuitive Surgical Operations, Inc. | Medical robotic system providing sensory feedback indicating a difference between a commanded state and a preferred pose of an articulated instrument |
US9125669B2 (en) * | 2011-02-14 | 2015-09-08 | Mako Surgical Corporation | Haptic volumes for reaming during arthroplasty |
CN102909728B (en) * | 2011-08-05 | 2015-11-25 | 鸿富锦精密工业(深圳)有限公司 | The vision correction methods of robot tooling center points |
WO2013033566A1 (en) * | 2011-09-02 | 2013-03-07 | Stryker Corporation | Surgical instrument including a cutting accessory extending from a housing and actuators that establish the position of the cutting accessory relative to the housing |
WO2013192598A1 (en) * | 2012-06-21 | 2013-12-27 | Excelsius Surgical, L.L.C. | Surgical robot platform |
CN106470634B (en) | 2014-03-17 | 2019-06-14 | 直观外科手术操作公司 | System and method for being indicated outside the screen of the instrument in remote control operation medical system |
JP6854237B2 (en) * | 2014-03-28 | 2021-04-07 | インテュイティブ サージカル オペレーションズ, インコーポレイテッド | Quantitative 3D visualization of instruments in the field of view |
US11737663B2 (en) * | 2020-03-30 | 2023-08-29 | Auris Health, Inc. | Target anatomical feature localization |
-
2015
- 2015-03-17 CN CN201580025502.2A patent/CN106470634B/en active Active
- 2015-03-17 CN CN201910406219.4A patent/CN110236693B/en active Active
- 2015-03-17 KR KR1020167028341A patent/KR102363661B1/en active IP Right Grant
- 2015-03-17 EP EP15765724.8A patent/EP3119339B1/en active Active
- 2015-03-17 WO PCT/US2015/021109 patent/WO2015142956A1/en active Application Filing
- 2015-03-17 JP JP2016558114A patent/JP6542252B2/en active Active
- 2015-03-17 US US15/262,867 patent/US11317979B2/en active Active
- 2015-03-17 EP EP19181852.5A patent/EP3566672B1/en active Active
-
2019
- 2019-06-12 JP JP2019109394A patent/JP6851427B2/en active Active
-
2021
- 2021-01-14 JP JP2021004074A patent/JP7295153B2/en active Active
-
2022
- 2022-03-31 US US17/710,435 patent/US11903665B2/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090192523A1 (en) * | 2006-06-29 | 2009-07-30 | Intuitive Surgical, Inc. | Synthetic representation of a surgical instrument |
US20090088773A1 (en) * | 2007-09-30 | 2009-04-02 | Intuitive Surgical, Inc. | Methods of locating and tracking robotic instruments in robotic surgical systems |
Also Published As
Publication number | Publication date |
---|---|
US20220218426A1 (en) | 2022-07-14 |
US11903665B2 (en) | 2024-02-20 |
US11317979B2 (en) | 2022-05-03 |
CN106470634A (en) | 2017-03-01 |
US20170165013A1 (en) | 2017-06-15 |
WO2015142956A1 (en) | 2015-09-24 |
JP2017515523A (en) | 2017-06-15 |
CN110236693A (en) | 2019-09-17 |
JP6542252B2 (en) | 2019-07-10 |
JP2021072900A (en) | 2021-05-13 |
EP3119339B1 (en) | 2019-08-28 |
JP7295153B2 (en) | 2023-06-20 |
EP3119339A4 (en) | 2017-11-29 |
JP6851427B2 (en) | 2021-03-31 |
CN106470634B (en) | 2019-06-14 |
CN110236693B (en) | 2022-11-04 |
EP3119339A1 (en) | 2017-01-25 |
JP2019162511A (en) | 2019-09-26 |
KR20160133515A (en) | 2016-11-22 |
EP3566672B1 (en) | 2023-08-23 |
KR102363661B1 (en) | 2022-02-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11903665B2 (en) | Systems and methods for offscreen indication of instruments in a teleoperational medical system | |
US10905506B2 (en) | Systems and methods for rendering onscreen identification of instruments in a teleoperational medical system | |
US11872006B2 (en) | Systems and methods for onscreen identification of instruments in a teleoperational medical system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED |
|
AC | Divisional application: reference to earlier application |
Ref document number: 3119339 Country of ref document: EP Kind code of ref document: P |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20200513 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: A61B 34/37 20160101ALI20230221BHEP Ipc: A61B 90/00 20160101AFI20230221BHEP |
|
INTG | Intention to grant announced |
Effective date: 20230313 |
|
RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: MOHR, PAUL, W. Inventor name: HOFFMAN, BRIAN, D. Inventor name: ITKOWITZ, BRANDON, D. |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230510 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AC | Divisional application: reference to earlier application |
Ref document number: 3119339 Country of ref document: EP Kind code of ref document: P |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015085346 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20230823 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1601674 Country of ref document: AT Kind code of ref document: T Effective date: 20230823 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231124 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231223 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231226 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231123 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231223 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20231124 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20240328 Year of fee payment: 10 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602015085346 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20230823 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |