US20180338804A1 - System and method of registration between devices with movable arms - Google Patents
System and method of registration between devices with movable arms Download PDFInfo
- Publication number
- US20180338804A1 US20180338804A1 US16/053,049 US201816053049A US2018338804A1 US 20180338804 A1 US20180338804 A1 US 20180338804A1 US 201816053049 A US201816053049 A US 201816053049A US 2018338804 A1 US2018338804 A1 US 2018338804A1
- Authority
- US
- United States
- Prior art keywords
- pose
- movable arm
- image capturing
- images
- capturing device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 161
- 230000033001 locomotion Effects 0.000 claims abstract description 158
- 230000006870 function Effects 0.000 claims description 28
- 238000003384 imaging method Methods 0.000 abstract description 6
- 210000001503 joint Anatomy 0.000 description 33
- 230000009471 action Effects 0.000 description 15
- 230000008569 process Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 13
- 238000012545 processing Methods 0.000 description 13
- 238000002604 ultrasonography Methods 0.000 description 13
- 238000010276 construction Methods 0.000 description 10
- 239000000523 sample Substances 0.000 description 10
- 230000003287 optical effect Effects 0.000 description 9
- 230000004044 response Effects 0.000 description 9
- 238000005094 computer simulation Methods 0.000 description 8
- 210000000707 wrist Anatomy 0.000 description 8
- 210000003414 extremity Anatomy 0.000 description 7
- 210000003484 anatomy Anatomy 0.000 description 6
- 238000002591 computed tomography Methods 0.000 description 5
- 239000003550 marker Substances 0.000 description 4
- 238000000926 separation method Methods 0.000 description 4
- 210000002310 elbow joint Anatomy 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 210000000323 shoulder joint Anatomy 0.000 description 3
- 210000003857 wrist joint Anatomy 0.000 description 3
- 239000004606 Fillers/Extenders Substances 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 238000002594 fluoroscopy Methods 0.000 description 2
- 230000009474 immediate action Effects 0.000 description 2
- 230000009191 jumping Effects 0.000 description 2
- 238000002595 magnetic resonance imaging Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 238000001356 surgical procedure Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001427 coherent effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000007408 cone-beam computed tomography Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000006880 cross-coupling reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000012636 effector Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 210000003709 heart valve Anatomy 0.000 description 1
- 230000003116 impacting effect Effects 0.000 description 1
- 238000013152 interventional procedure Methods 0.000 description 1
- 230000001846 repelling effect Effects 0.000 description 1
- 210000004872 soft tissue Anatomy 0.000 description 1
- 238000003325 tomography Methods 0.000 description 1
- 210000000515 tooth Anatomy 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/30—Surgical robots
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/10—Safety means specially adapted therefor
- A61B6/102—Protection against mechanical damage, e.g. anti-collision devices
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/44—Constructional features of apparatus for radiation diagnosis
- A61B6/4429—Constructional features of apparatus for radiation diagnosis related to the mounting of source units and detector units
- A61B6/4435—Constructional features of apparatus for radiation diagnosis related to the mounting of source units and detector units the source unit and the detector unit being coupled by a rigid structure
- A61B6/4441—Constructional features of apparatus for radiation diagnosis related to the mounting of source units and detector units the source unit and the detector unit being coupled by a rigid structure the rigid structure being a C-arm or U-arm
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
- A61B6/44—Constructional features of apparatus for radiation diagnosis
- A61B6/4429—Constructional features of apparatus for radiation diagnosis related to the mounting of source units and detector units
- A61B6/4458—Constructional features of apparatus for radiation diagnosis related to the mounting of source units and detector units the source unit or the detector unit being attached to robotic arms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/361—Image-producing devices, e.g. surgical cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1674—Programme controls characterised by safety, monitoring, diagnostic
- B25J9/1676—Avoiding collision or forbidden zones
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00064—Constructional details of the endoscope body
- A61B1/00071—Insertion part of the endoscope body
- A61B1/0008—Insertion part of the endoscope body characterised by distal tip features
- A61B1/00087—Tools
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00147—Holding or positioning arrangements
- A61B1/00149—Holding or positioning arrangements using articulated arms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00163—Optical arrangements
- A61B1/00174—Optical arrangements characterised by the viewing angles
- A61B1/00183—Optical arrangements characterised by the viewing angles for variable viewing angles
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
- A61B1/05—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances characterised by the image sensor, e.g. camera, being in the distal end portion
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B17/00—Surgical instruments, devices or methods, e.g. tourniquets
- A61B2017/00017—Electrical control of surgical instruments
- A61B2017/00216—Electrical control of surgical instruments with eye tracking or head position tracking control
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2055—Optical tracking systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2059—Mechanical position encoders
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B2034/2046—Tracking techniques
- A61B2034/2065—Tracking using image or pattern recognition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/30—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
- A61B2090/306—Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure using optical fibres
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/371—Surgical systems with images on a monitor during operation with simultaneous use of two cameras
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/376—Surgical systems with images on a monitor during operation using X-rays, e.g. fluoroscopy
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
- A61B2090/378—Surgical systems with images on a monitor during operation using ultrasound
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B90/00—Instruments, implements or accessories specially adapted for surgery or diagnosis and not covered by any of the groups A61B1/00 - A61B50/00, e.g. for luxation treatment or for protecting wound edges
- A61B90/36—Image-producing devices or illumination devices not otherwise provided for
- A61B90/37—Surgical systems with images on a monitor during operation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S901/00—Robots
- Y10S901/02—Arm motion controller
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S901/00—Robots
- Y10S901/46—Sensing device
- Y10S901/47—Optical
Definitions
- the present invention generally relates to robotic systems and in particular, to collision avoidance during controlled movement of image capturing device and manipulatable device robot arms.
- Robotic systems and computer-assisted devices often include robot or movable arms to manipulate instruments for performing a task at a work site and at least one robot or movable arm for supporting an image capturing device which captures images of the work site.
- robot arms When the robot arms are operating within close proximity to each other, there is a possibility that the arms may collide with one another and by so doing, cause damage to the arms.
- the robot arms When the robot arms are commonly operated, preventing such collisions may be straightforward.
- collision avoidance may be much more challenging.
- a robot arm comprises a plurality of links coupled together by one or more actively controlled joints.
- a plurality of actively controlled joints may be provided.
- the robot arm may also include one or more passive joints, which are not actively controlled, but comply with movement of an actively controlled joint.
- Such active and passive joints may be revolute or prismatic joints.
- the configuration of the robot arm may then be determined by the positions of the joints and knowledge of the structure and coupling of the links.
- One action that the controller may take to avoid collisions is to warn an operator who is commanding a robot arm that a collision with another robot arm is imminent.
- U.S. 2009/0192524 A1 entitled “Synthetic Representation of a Surgical Robot,” which is incorporated herein by reference describes a patient side cart upon which a plurality of robot arms is mounted.
- U.S. 2009/0326553 entitled “Medical Robotic System Providing an Auxiliary View of Articulatable Instruments Extending out of a Distal End of an Entry Guide,” which is incorporated herein by reference describes a medical robotic system having a plurality of articulated instruments extending out of an entry guide.
- Collision avoidance may also be performed automatically by a controller which is controlling movement of one or more robot arms.
- a controller which is controlling movement of one or more robot arms.
- the robot arms have redundancy so that multiple configurations are possible for each arm to achieve a desired position and orientation of its held instrument.
- Each controller commands movement of its associated robot arm subject to a secondary constraint that results in eliminating possible arm configurations that would result in a collision with another robot arm.
- one object of one or more aspects of the present invention is a robotic system and method implemented therein that automatically performs collision avoidance of robot arms.
- Another object of one or more aspects of the present invention is a robotic system and method implemented therein that automatically performs collision avoidance of independently operated robot arms.
- Another object of one or more aspects of the present invention is a robotic system and method implemented therein that automatically performs collision avoidance of robot arms without preventing any of the robot arms from performing its intended task.
- one aspect is a robotic system including: a first robot arm holding a manipulatable device having a working end; and a controller configured to control movement of at least one of the first robot arm and a second robot arm while avoiding a collision between the first and second robot arms, wherein the second robot arm holds an image capturing device for capturing a plurality of images of the working end of the manipulatable device from which a three-dimensional computer model of the working end of the manipulatable device is generatable; wherein the controller is configured to avoid the collision between the first and second robot arms: by determining a position and an orientation of the working end of the manipulatable device relative to a reference frame of the image capturing device by using at least one image of the plurality of images of the working end of the manipulatable device, wherein the reference frame of the image capturing device corresponds to a perspective of the captured images; by determining a configuration and position of one of the first and second robot arms relative to a common reference
- Another aspect is a method implemented in a robotic system for avoiding collisions between first and second robot arms, wherein the first robot arm holds a manipulatable device having a working end, wherein the second robot arm holds an image capturing device for capturing a plurality of images of the working end of the manipulatable device from which a three-dimensional computer model of the working end of the manipulatable device is generatable, and wherein the method includes: determining a position and an orientation of the working end of the manipulatable device relative to a reference frame of the image capturing device by using at least one image of the plurality of images of the working end of the manipulatable device, wherein the reference frame of the image capturing device corresponds to a perspective of the captured images; determining a configuration and position of one of the first and second robot arms relative to a common reference frame by using the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device; determining a configuration and position of the other of the first and second robot arms relative to the common reference frame
- a robotic system including: a first robot arm that is holding a manipulatable device having a working end; and a processor programmed to register the first robot arm to a second robot arm that is holding an image capturing device, wherein the processor performs the registration by: performing a low accuracy registration of the first and second robot arms at initial positions of the first and second robot arms using external tracking data and kinematics data of at least one of the first and second robot arms; performing a mid accuracy registration of the first and second robot arms relative to a first setup position using at least one image captured by the image capturing device of at least a part of the first robot arm and kinematics data of at least one of the first and second robot arms, wherein the first setup position includes a first safety margin indicative of the low accuracy registration; and performing a high accuracy registration of the first and second robot arms relative to a second setup position using at least one image captured by the image capturing device of the working end of the manipulatable device and kinematics data of at least one of the first and second robot arms,
- the first robot arm holds a manipulatable device having a working end.
- the second robot arm holds an image capturing device.
- the method includes performing a low accuracy registration of the first and second robot arms at initial positions of the first and second robot arms using external tracking data and kinematics data of at least one of the first and second robot arms; performing a mid accuracy registration of the first and second robot arms relative to a first setup position using at least one image captured by the image capturing device of at least a part of the first robot arm and kinematics data of at least one of the first and second robot arms, wherein the first setup position includes a first safety margin indicative of the low accuracy registration; and performing a high accuracy registration of the first and second robot arms relative to a second setup position using at least one image captured by the image capturing device of the working end of the manipulatable device and kinematics data of at least one of the first and second robot arms, wherein the second setup position includes a second safety margin indicative of the
- a movement control system includes a controller.
- the controller includes one or more processors and memory coupled to the one or more processors.
- the controller is coupled to a computer-assisted surgical device having a first movable arm coupled to a manipulatable device having a working end.
- the controller is further coupled to a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device.
- the controller is configured to receive one or more first configurations for the first movable arm; receive one or more second configurations for the second movable arm; receive a first plurality of images of the working end from the image capturing device; determine, based on at least one of the first plurality of images, a position and an orientation of the working end in a common reference frame; determine, based on the first configurations, a first movable arm position and a first movable arm trajectory for the first movable arm in the common reference frame; determine, based on the second configurations, a second movable arm position and a second movable arm trajectory for the second movable arm in the common reference frame; based on the first movable arm position, the first movable arm trajectory, the second movable arm position, and the second movable arm trajectory, determine whether motion of the first movable arm, motion of the second movable arm, or motions of the first and second movable arms together will result in an undesirable relationship between the first and second movable arms; and send
- the common reference frame is a reference frame of the image capturing device and the controller is further configured to determine the first movable arm position in the common reference frame further based on the position and the orientation of the working end in the common reference frame.
- the common reference frame is a reference frame of the computer-assisted surgical device and the controller is further configured to determine the second movable arm position in a reference frame of the image capturing device, determine the position and the orientation of the working end in the reference frame of the image capturing device, and transform the second movable arm position and the position and orientation of the working end from the reference frame of the image capturing device to the common reference frame.
- the first plurality of images are a plurality of two-dimensional images from a image capturing device and the controller is further configured to determine the position and the orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device.
- the system further includes a viewer adapted to display a second plurality images of a work space of the working end of the manipulatable device and an input unit configured to receive information of a user specified region of interest within the images being displayed on the viewer.
- the controller is further configured to send second movement commands to the second movable arm so that the image capturing device captures images of the user specified region of interest.
- the input unit includes a gaze tracking unit configured to track a gaze of a user on a display screen of the viewer and the gaze tracking unit includes an indicator operable by the user to indicate the region of interest by using a current gaze point of the user on the display screen.
- the input unit includes a telestrator unit configured to receive one of the second plurality of images and display the received one of the second plurality of images on a display screen of the telestrator and the telestrator unit includes a marker unit operable by a user to indicate the region of interest on the displayed one of the second plurality of images.
- the system further includes a camera, and the second plurality of images are captured by the camera.
- system further includes an ultrasound probe, and the second plurality of images are captured by the ultrasound probe.
- the first movable arm has redundant degrees of freedom so that for each controllable position and orientation of the working end of the manipulatable device there are a first plurality of possible positions and orientations for the first movable arm
- the second movable arm has redundant degrees of freedom so that for each controllable position and orientation of the image capturing device there are a second plurality of possible positions and orientations for the second movable arm
- the first movement command sent to the first movable arm or the second movable arm directs the first movable arm to move to one of the first plurality of possible positions and orientations or directs the second movable arm to move to one of the second plurality of possible positions and orientations that avoid the undesirable relationship.
- the controller is further configured to determine which one of the first movable arm and the second movable arms is to be sent the first movement command based on differences between the first movable arm position and the first plurality of possible positions and orientations and differences between the second movable arm position and the second plurality of possible positions and orientations and the determination is made so as to minimize a cost function.
- the first movement command directs the first movable arm to move from the first movable arm position to a first selected one of the first plurality of possible positions and orientations or directs the second movable arm to move from the second movable arm position to a second selected one of the second plurality of possible positions and orientations based on which of the first selected one of the first plurality of possible positions and orientations and the second selected one of the second plurality of possible positions and orientations minimizes the cost function.
- the undesirable relationship is selected from a group consisting of a collision between the first movable arm and the second movable arm, too close a proximity between the first movable arm and the second movable arm, and obstruction of a region of interest of the image capturing device by the first movable arm.
- a method of controlling movement in a medical system includes receiving one or more first configurations for a first movable arm of a computer-assisted surgical device, the first movable arm being coupled to a manipulatable device having a working end; receiving one or more second configurations for a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device; receiving a first plurality of images of the working end from the image capturing device; determining, based on at least one of the first plurality of images, a position and an orientation of the working end in a common reference frame; determining, based on the first configurations, a first movable arm position and a first movable arm trajectory for the first movable arm in the common reference frame; determining, based on the second configurations, a second movable arm position and a second movable arm trajectory for the second movable arm in the common reference frame; based on the first movable arm position, the first movable arm trajectory, the
- the common reference frame is a reference frame of the image capturing device, and the method further includes determining the first movable arm position in the common reference frame further based on the position and the orientation of the working end in the common reference frame.
- the common reference frame is a reference frame of the computer-assisted surgical device and the method further includes determining the second movable arm position in a reference frame of the image capturing device, determining the position and the orientation of the working end in the reference frame of the image capturing device, and transforming the second movable arm position and the position and orientation of the working end from the reference frame of the image capturing device to the common reference frame.
- the first plurality of images are a plurality of two-dimensional images from a image capturing device and the method further includes determining the position and the orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device. In some examples, the method further includes determining the first movement command so as to minimize a cost function.
- a movement control system includes a tracking system and a controller coupled to the tracking system.
- the controller includes one or more processors and memory coupled to the one or more processors.
- the controller is coupled to a computer-assisted surgical device having a first movable arm coupled to a manipulatable device having a working end.
- the controller is further coupled to a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device.
- the controller is configured to receive first kinematic data for the first movable arm; receive second kinematic data for the second movable arm; receive first tracking data for the first movable arm from the tracking system; receive second tracking data for the second movable arm from the tracking system; determine, based on the first kinematic data and the first tracking data, a first location of the first movable arm; determine, based on the second kinematic data and the second tracking data, a second location of the second movable arm; send a first movement command to the second movable arm that directs the second movable arm to move into a first set-up position to capture images of at least a portion of the manipulatable device while maintaining a first safety margin between the first movable arm and the second movable arm, the first movement command being based on the first location and the second location; receive third kinematic data for the first movable arm; receive fourth kinematic data for the second movable arm; receive one or more first images from the image capturing device
- the controller is further configured to receive fifth kinematic data for the first movable arm; receive sixth kinematic data for the second movable arm; receive one or more second images from the image capturing device, the one or more second images capturing at least a portion of the working end; determine, based on the fifth kinematic data and the one or more second images, a fifth location of the first movable arm; determine, based on the sixth kinematic data and the one or more second images, a sixth location of the second movable arm; and send a third movement command to the second movable arm that directs the second movable arm to move into a third set-up position, different from the first set-up position and the second set-up position, to capture images of the working end while maintaining a third safety margin between the first movable arm and the second movable arm, the third movement command being based on the fifth location and the sixth location.
- the second safety margin is an order of magnitude smaller than the first safety margin. In some examples, maintaining the first safety margin keeps a separation between the first movable arm and the second movable arm of at least ten centimeters and maintaining the second safety margin keeps a separation between the first movable arm and the second movable arm of at least one centimeter. In some examples, the third safety margin is an order of magnitude smaller than the second safety margin. In some examples, maintaining the second safety margin keeps a separation between the first movable arm and the second movable arm of at least one centimeter and maintaining the third safety margin keeps a separation between the first movable arm and the second movable arm of at least one millimeter.
- the first images are two-dimensional images from a image capturing device and the controller is further configured to determine a position and an orientation of the manipulatable device from the two-dimensional images even when the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device.
- the second images are two-dimensional images from a image capturing device and the controller is further configured to determine a position and an orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device.
- the system further includes a viewer adapted to display one more second images received from the image capturing device and an input unit configured to receive information of the region of interest within the second images.
- the input unit includes a gaze tracking unit configured to track a gaze of a user on a display screen of the viewer and the gaze tracking unit includes an indicator operable by the user to indicate the region of interest by using a current gaze point of the user on the display screen.
- the input unit includes a telestrator unit configured to receive one of the second of images and display the received one of the second images on a display screen of the telestrator and the telestrator unit includes a marker unit operable by a user to indicate the region of interest on the displayed one of the second images.
- the system further includes a camera and the second images are captured by the camera.
- the system further includes an ultrasound probe and the second images are captured by the ultrasound probe.
- the first movement command minimizes a cost function.
- the second movable arm has redundant degrees of freedom so that for each controllable position and orientation of the image capturing device there are a plurality of possible positions and orientations for the second movable arm, the first movement command sent to the second movable arm directs the second movable arm to move to one of the plurality of possible positions and orientations that maintain the first safety margin, the second movement command sent to the second movable arm directs the second movable arm to move to one of the plurality of possible positions and orientations that maintain the second safety margin, and the third movement command sent to the second movable arm directs the second movable arm to move to one of the plurality of possible positions and orientations that maintain the third safety margin.
- a method of controlling movement in a medical system includes receiving first kinematic data for a first movable arm of a computer-assisted surgical device, the first movable arm being coupled to a manipulatable device having a working end; receiving second kinematic data for a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device; receiving first tracking data for the first movable arm from a tracking system; receiving second tracking data for the second movable arm from the tracking system; determining, based on the first kinematic data and the first tracking data, a first location of the first movable arm; determining, based on the second kinematic data and the second tracking data, a second location of the second movable arm; sending a first movement command to the second movable arm that directs the second movable arm to move into a first set-up position to capture images of at least a portion of the manipulatable device while maintaining a first safety margin between the first movable
- the method further includes receiving fifth kinematic data for the first movable arm; receiving sixth kinematic data for the second movable arm; receiving one or more second images from the image capturing device, the one or more second images capturing at least a portion of the working end; determining, based on the fifth kinematic data and the one or more second images, a fifth location of the first movable arm; determining, based on the sixth kinematic data and the one or more second images, a sixth location of the second movable arm; and sending a third movement command to the second movable arm that directs the second movable arm to move into a third set-up position, different from the first set-up position and the second set-up position, to capture images of the working end while maintaining a third safety margin between the first movable arm and the second movable arm, the third movement command being based on the fifth location and the sixth location.
- the method further includes determining the third movement command so as to minimize a cost function.
- the first images are two-dimensional images from a image capturing device and the method further includes determining a position and an orientation of the manipulatable device from the two-dimensional images even when the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device.
- the second images are two-dimensional images from a image capturing device and the method further includes determining a position and an orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device.
- a robotic system includes a first robot arm holding a manipulatable device having a working end and a controller configured to control movement of at least one of the first robot arm and a second robot arm while avoiding a collision between the first and second robot arms.
- the second robot arm holds an image capturing device for capturing a plurality of images of the working end of the manipulatable device from which a three-dimensional computer model of the working end of the manipulatable device is generatable.
- the controller is configured to avoid the collision between the first and second robot arms by determining a position and an orientation of the working end of the manipulatable device relative to a reference frame of the image capturing device by using at least one image of the plurality of images of the working end of the manipulatable device, wherein the reference frame of the image capturing device corresponds to a perspective of the captured images; by determining a configuration and position of one of the first and second robot arms relative to a common reference frame by using the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device; and by determining a configuration and position of the other of the first and second robot arms relative to the common reference frame by using joint position information received for the other of the first and second robot arms; by determining an imminent collision between the first and second robot arms using the determined configurations and positions of the first and second robot arms relative to the common reference frame; and by commanding an action to be taken by one of the first and second robot arms to avoid the imminent collision.
- the image capturing device includes a tomographic image capturing device for capturing a plurality of two-dimensional images from which the working end of the manipulatable device is discernible even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device.
- the controller is configured to determine the configuration and position of the first robot arm relative to the reference frame of the image capturing device by using information of the construction and geometry of the first robot arm and the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device.
- the controller is configured to determine the configuration and position of the second robot arm relative to a reference frame of the manipulatable device by using a transform to translate points in the reference frame of the image capturing device to the reference frame of the manipulatable device.
- the transform has been determined by using the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device.
- the reference frame of the manipulatable device is the common reference frame.
- the system further includes a viewer adapted to display images derived from captured images of a work space of the working end of the manipulatable device and an input unit configured to receive information of a user specified region of interest within the images being displayed on the viewer.
- the controller is configured to command and control movement of the second robot arm so that the image capturing device captures images of the user specified region of interest in the plurality of images of the working end of the manipulatable device.
- the input unit includes a gaze tracking unit configured to track a gaze of a user on a display screen of the viewer.
- the gaze tracking unit includes an indicator operable by the user to indicate the region of interest by using a current gaze point of the user on the display screen.
- the input unit includes a telestrator unit configured to receive an image being displayed on the viewer and display the received image on a display screen of the telestrator.
- the telestrator unit includes a marker unit operable by a user to indicate the region of interest on the image being displayed on the display screen of the telestrator unit.
- the system further includes a camera and the captured images of the work space of the working end of the manipulatable device are captured by the camera.
- the system further includes an ultrasound probe and the captured images of the work space of the working end of the manipulatable device are captured by the ultrasound probe.
- the controller is associated with the first robot arm to control movement of the first robot arm while avoiding a collision with the second robot arm and a reference frame of the manipulatable device is the common reference frame.
- the controller is associated with the second robot arm to control movement of the second robot arm while avoiding a collision with the first robot arm, and the reference frame of the image capturing device is the common reference frame.
- the first robot arm includes a first plurality of joints and a first plurality of links that are coupled together to provide redundant degrees of freedom movement of the first robot arm so that for each controllable position and orientation of the working end of the manipulatable device there are a plurality of possible configurations for the first robot arm.
- the controller is configured to control movement of the first robot arm by commanding the first robot arm to be configured in one of the plurality of possible configurations for the first robot arm according to a desired position and orientation of the working end of the manipulatable device and according to a secondary constraint for avoiding a collision between the first and second robot arms.
- the second robot arm includes a second plurality of joints and a second plurality of links that are coupled together to provide redundant degrees of freedom movement of the second robot arm so that for each controllable position and orientation of the image capturing device there are a plurality of possible configurations for the second robot arm.
- the controller is configured to control movement of the second robot arm by commanding the second robot arm to be configured in one of the plurality of possible configurations for the second robot arm according to a desired position and orientation of the image capturing device and according to a secondary constraint for avoiding a collision between the first and second robot arms.
- the first robot arm includes a first plurality of joints and a first plurality of links that are coupled together to provide redundant degrees of freedom movement of the first robot arm so that for each controllable position and orientation of the working end of the manipulatable device there are a plurality of possible configurations for the first robot arm.
- the second robot arm includes a second plurality of joints and a second plurality of links that are coupled together to provide redundant degrees of freedom movement of the second robot arm so that for each controllable position and orientation of the image capturing device there are a plurality of possible configurations for the second robot arm.
- the controller is configured to control movement of one of the first and second robot arms to be configured in one of its plurality of possible configurations to avoid a collision with the other one of the first and second robot arms.
- the controller is configured to determine which one of the first and second robot arms is to be configured in one of its plurality of possible configurations to avoid a collision with the other one of the first and second robot arms by processing differences between current configurations of the first and second robot arms and their respective pluralities of possible configurations to minimize a cost function. In some examples, the controller is configured to determine which one of the first and second robot arms is to be configured in one of its plurality of possible configurations to avoid a collision with the other one of the first and second robot arms by processing required joint movements of the first and second robotic systems to move from their current configurations to others of their respective pluralities of possible configurations to minimize a cost function.
- Consistent with some embodiments is a method for avoiding collisions between first and second robot arms.
- the first robot arm holds a manipulatable device having a working end.
- the second robot arm holds an image capturing device for capturing a plurality of images of the working end of the manipulatable device from which a three-dimensional computer model of the working end of the manipulatable device is generatable.
- the method includes determining a position and an orientation of the working end of the manipulatable device relative to a reference frame of the image capturing device by using at least one image of the plurality of images of the working end of the manipulatable device, wherein the reference frame of the image capturing device corresponds to a perspective of the captured images; determining a configuration and position of one of the first and second robot arms relative to a common reference frame by using the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device; determining a configuration and position of the other of the first and second robot arms relative to the common reference frame by using joint position information received for the other of the first and second robot arms; determining an imminent collision between the first and second robot arms using the determined configurations and positions of the first and second robot arms relative to the common reference frame; and commanding an action to be taken by one of the first and second robot arms to avoid the imminent collision.
- a robotic system includes a first robot arm that is holding a manipulatable device having a working end and a processor programmed to register the first robot arm to a second robot arm that is holding an image capturing device.
- the processor performs the registration by performing a low accuracy registration of the first and second robot arms at initial positions of the first and second robot arms using external tracking data and kinematics data of at least one of the first and second robot arms; performing a mid accuracy registration of the first and second robot arms relative to a first setup position using at least one image captured by the image capturing device of at least a part of the first robot arm and kinematics data of at least one of the first and second robot arms, wherein the first setup position includes a first safety margin indicative of the low accuracy registration; and performing a high accuracy registration of the first and second robot arms relative to a second setup position using at least one image captured by the image capturing device of the working end of the manipulatable device and kinematics data of at least one of the first and second robot arms,
- Consistent with some embodiments is a method implemented by a processor for registering first and second robot arms.
- the first robot arm holds a manipulatable device having a working end.
- the second robot arm holds an image capturing device.
- the method includes performing a low accuracy registration of the first and second robot arms at initial positions of the first and second robot arms using external tracking data and kinematics data of at least one of the first and second robot arms, performing a mid accuracy registration of the first and second robot arms relative to a first setup position using at least one image captured by the image capturing device of at least a part of the first robot arm and kinematics data of at least one of the first and second robot arms, wherein the first setup position includes a first safety margin indicative of the low accuracy registration, and performing a high accuracy registration of the first and second robot arms relative to a second setup position using at least one image captured by the image capturing device of the working end of the manipulatable device and kinematics data of at least one of the first and second robot arms, wherein the
- FIG. 1 illustrates a block diagram of a robotic system utilizing aspects of the present invention.
- FIG. 2 illustrates a flow diagram of a method, implemented in a robotic system utilizing aspects of the present invention, for performing multi-accuracy registration of an image capturing device robot arm and a manipulatable device robot arm as the image capturing device robot arm moves relative to the manipulatable device robot arm.
- FIG. 3 illustrates a flow diagram of a method, implemented in a robotic system utilizing aspects of the present invention, for automatically avoiding a collision between independently operated robot arms from the perspective of an image capturing device reference frame.
- FIG. 4 illustrates a flow diagram of the method, implemented in a robotic system utilizing aspects of the present invention, for automatically avoiding a collision between independently operated robot arms from the perspective of a manipulatable device reference frame.
- FIG. 5 illustrates a flow diagram of a method, implemented in a robotic system utilizing aspects of the present invention, for controlling an image capturing device robot arm so that its image capturing device captures images of a user selected region of interest while avoiding a collision with another robot arm.
- FIG. 6 illustrates a flow diagram of a method, implemented in a robotic system utilizing aspects of the present invention, for controlling movement of an image capturing device robot arm according to a plan of motion while avoiding a collision with another robot arm.
- FIG. 7 illustrates a block diagram of a portion of a robotic system utilizing aspects of the present invention, which is used for collision avoidance of two robot arms operating in a coupled-control mode.
- FIG. 8 illustrates a perspective view of an operating room employing a multiple aperture medical robotic system in which aspects of the present invention are usable.
- FIG. 9 illustrates a front view of a Patient-Side Cart for a multiple aperture medical robotic system in which aspects of the present invention are usable.
- FIG. 10 illustrates a perspective view of an instrument usable in a multiple aperture medical robotic system in which aspects of the present invention are usable.
- FIG. 11 illustrates a front view of a console of a robotic system utilizing aspects of the present invention.
- FIG. 12 illustrates a perspective view of an image capturing system usable in a robotic system utilizing aspects of the present invention.
- FIG. 13 illustrates a perspective view of a Patient-Side Cart for a single aperture medical robotic system in which aspects of the present invention are usable.
- FIG. 14 illustrates a perspective view of a distal end of an entry guide with articulated instruments extending out of it as used in a single aperture medical robotic system in which aspects of the present invention are usable.
- FIG. 15 illustrates a cross-sectional view of an entry guide as used in a single aperture medical robotic system in which aspects of the present invention are usable.
- FIG. 1 illustrates, as an example, a block diagram of various components of a robotic or computer-assisted system 1000 in which methods 2000 , 2500 , and 4000 are implemented for automatically avoiding a collision between two or more robot or movable arms.
- FIG. 2 describes a multi-step approach to registration of robot or movable arms.
- FIGS. 3, 4 are provided to describe the methods 2000 , 2500 which are generally applicable to independently operated robot or movable arms.
- FIGS. 5-7 are provided to describe the method 4000 which is generally applicable to controlling movement of an image capturing device robot or movable arm though a controller which is controlling a manipulatable device robot or movable arm.
- FIGS. 8-15 Examples of the robotic system 1000 in which the methods 2000 , 2500 , and 4000 may be used are described in reference to FIGS. 8-15 .
- FIGS. 8-12 are provided to describe a robotic or computer-assisted system 3000 using a Patient-Side Cart 3010 with multiple robot or movable arms which is suitable for performing procedures with multiple apertures for entry to a work site.
- FIGS. 13-15 are provided to describe an alternative Patient-Side Cart 4010 for the robotic system 3000 , in which the Patient-Side Cart 4010 has a single robot or movable arm which is suitable for performing procedures with a single aperture for entry to a work site.
- the robotic or computer-assisted system 3000 will first be described to provide context and additional details on implementations of the robotic or computer-assisted system 1000 .
- a medical robotic or computer-assisted surgical system is described herein as an example of the robotic or computer-assisted system 1000 , it is to be appreciated that the various aspects of the invention as claimed herein are not to be limited to such types of robotic or computer-assisted systems.
- a perspective view of an operating room is illustrated in which a medical robotic system 3000 is provided for a Surgeon to perform a medical procedure on a Patient.
- the medical robotic system in this case is a Minimally Invasive Robotic Surgical (MIRS) system including a Patient-Side Cart 3010 , an Image Capturing System 3020 , and a Console 3030 .
- MIRS Minimally Invasive Robotic Surgical
- the Patient-Side Cart 3010 has a plurality of robot arms for holding and manipulating a plurality of devices such as instruments and at least one endoscope.
- a plurality of robot arms for holding and manipulating a plurality of devices such as instruments and at least one endoscope.
- each of the devices being held by the plurality of robot or movable arms is introduced through its own entry aperture into a Patient.
- the Console 3030 includes input devices for commanding movement of associated ones of the plurality of robot arms of the Patient-Side Cart 3010 and the operation of their respectively held devices.
- the Console 3030 and the Patient-Side Cart 3010 communicate through a cable 3031 .
- the Image Capturing System 3020 captures a plurality of images, such as a series of two-dimensional image projections, of one or more objects within a specified region of interest at a work site in the Patient.
- the plurality of images may then be used to generate a three-dimensional computer model of the one or more objects without requiring prior knowledge of the three-dimensional shapes of the one or more objects, in a conventional manner such as used in computed tomography.
- a control unit 3021 may be provided beneath or to the side of an articulated operating table 3040 , so that an Assistant may manually control movement of an image capturing device of the Image Capturing System 3020 using a joy stick or other control input provided on the control unit 3021 .
- the control unit 3021 and the Image Capturing System 3020 communicate through cable 3022 or using wireless technology.
- the control unit 3021 may be provided at another location and communicate with the Image Capturing System 3020 as a wired or wireless mobile entity.
- the control unit 3021 may include at least one processing unit and memory. In some examples, the processing unit may control operation and/or execution of hardware and/or software in control unit 3021 .
- the processing unit may include one or more central processing units (CPUs), multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), custom processors/application specific integrated circuits (ASICs), and/or the like.
- the memory may be used to store one or more software and/or firmware applications as well as various data structures to be used by control unit 3021 .
- the memory may also include one or more types of machine readable media.
- machine readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.
- movement of the image capturing device of the Image Capturing System 3020 may be controlled by the Surgeon operating an associated one of the input devices of the Console 3030 according to aspects of the present invention.
- movement of the image capturing device of the Image Capturing System 3020 may be controlled automatically by a processor of the Console 3030 so that the image capturing device automatically captures a plurality of images of a user specified region of interest in the work site while avoiding a collision with the robot arms of the Patient-Side Cart 3010 according to aspects of the present invention.
- the Console 3030 and the Image Capturing System 3020 communicate through cable 3050 .
- FIG. 9 illustrates, as an example, a front view of the Patient-Side Cart 3010 .
- robot arms 34 , 36 hold instruments 33 , 35 and robot arm 38 holds stereo endoscope 37 .
- the fourth robot arm 32 is available so that another instrument 31 may be introduced at the work site along with the instruments 33 , 35 and endoscope 37 .
- the fourth robot arm 32 may be used for introducing a second endoscope or another image capturing device, such as an ultrasound transducer, to the work site.
- Each of the robot or movable arms is conventionally formed of links which are coupled together and manipulated through actuatable joints.
- Each of the robot arms includes a setup arm and a device manipulator.
- the setup arm positions its held device so that a pivot point occurs at its entry aperture into the Patient.
- the device manipulator may then manipulate its held device so that it may be pivoted about the pivot point, inserted into and retracted out of the entry aperture, and rotated about its shaft axis.
- the robot arms are mounted on a movable base 3015 of the Patient-Side Cart 3010 .
- the robot arms may be attached to sliders on a side of the operating table, to sliders on a wall or to sliders on the ceiling of the operating room.
- FIG. 10 illustrates, as an example, an instrument 100 that may be used for either instrument 33 , 35 or 31 .
- the instrument 100 comprises an interface housing 108 , a shaft 104 , a working end 102 , and a wrist mechanism 106 which includes one or more wrist joints.
- the interface housing 108 is removably attached to a robot arm so as to be mechanically coupled to actuators (such as motors) in the instrument manipulator of the attached robot arm. Cables or rods, that are coupled to the actuators of the instrument manipulator and extend through the shaft 104 from the interface housing 108 to the one or more wrist joints of the wrist mechanism 106 and to the jaws of the instrument's end effector 102 , actuate the wrist joints and jaws in a conventional manner.
- actuators such as motors
- the instrument manipulator may also manipulate the instrument in pitch and yaw angular rotations about its pivot point at the entry aperture, manipulate the instrument in a roll angular rotation about the instrument's shaft axis, and insert and retract the instrument along a rail on the robot arm as commanded by the processor 43 .
- FIG. 11 illustrates, as an example, a front view of the Console 3030 .
- the Console 3030 has left and right input devices 41 , 42 which the user may grasp respectively with his/her left and right hands to manipulate associated devices being held by the plurality of robot or movable arms of the Patient-Side Cart 3010 in preferably six degrees-of-freedom (“DOF”).
- Foot pedals 44 with toe and heel controls are provided on the Console 3030 so the user may control movement and/or actuation of devices associated with the foot pedals.
- a processor 43 is provided in the Console for control and other purposes.
- a stereo vision display 45 is also provided in the Console so that the user may view the work site in stereo vision from images captured by the stereoscopic camera of the endoscope 37 .
- Left and right eyepieces, 46 and 47 are provided in the stereo vision display 45 so that the user may view left and right two-dimensional (“2D”) display screens inside the display 45 respectively with the user's left and right eyes.
- 2D two-dimensional
- the processor 43 performs various functions in the medical robotic system.
- One important function that it performs is to translate and transfer the mechanical motion of input devices 41 , 42 to command actuators in their associated device manipulators to actuate their respective joints so that the Surgeon can effectively manipulate devices, such as the tool instruments 31 , 33 , 35 and endoscope 37 , which are associated with the input devices 41 , 42 at the time.
- Another function of the processor 43 is to implement the methods, cross-coupling control logic, and controllers described herein.
- processor 43 may be implemented by any combination of hardware, software and firmware. Also, its functions as described herein may be performed by one unit or divided up among a number of subunits, each of which may be implemented in turn by any combination of hardware, software and firmware. Further, although being shown as part of or being physically adjacent to the Console, the processor 43 may also be distributed as subunits throughout the system 3000 .
- FIG. 12 illustrates, as an example, the Image Capturing System 3020 which has a robot or movable arm 1101 which is mounted to a base 1106 .
- the robot arm 1101 comprises a carousel 1107 , first link 1102 , second link 1103 , wrist 1104 , and C-arm 1105 .
- the carousel 1107 is rotatable (as indicted by arrow “a”) relative to the base 1106 using a carousel joint 1108 .
- the first link 1102 is rotatable (as indicated by arrow “b”) relative to the carousel 1107 using a shoulder joint 1109 .
- the second link 1103 is rotatable (as indicated by arrow “c”) relative to the first link 1102 using an elbow joint 1110 .
- the wrist 1104 is rotatable (as indicated by arrow “d”) relative to the second link 1103 using a roll joint 1111 .
- the C-arm 1105 is rotatable (as indicated by arrow “e”) relative to the wrist 1104 using a pitch joint 1112 and rotatable (as indicated by arrow “j”) relative to the wrist 1104 using a yaw joint 1121 .
- the C-arm 1105 comprises a first limb 1113 , second limb 1114 , and a central portion 1115 .
- the first and second limbs 1113 , 1114 are extendable away from and towards (as indicated by arrow “f”) the central element 1115 using an extender joint 1120 .
- the first limb 1113 has an X-ray detector 1116 disposed at its distal end and the second limb 1114 has an X-ray source 1117 disposed at its distal end.
- the X-ray detector 1116 is rotatable (as indicated by arrow “g”) relative to the distal end of the first limb 1113 .
- the X-ray source 1117 is rotatable (as indicated by arrow “h”) relative to the distal end of the second limb 1114 .
- the X-ray source and X-ray detector are disposed on opposing ends of the C-arm 1105 so as to form an image capturing device.
- the C-arm 1105 may be positioned relative to the Patient so that the C-arm 1105 may be moved so that the image capturing device (comprising X-ray source 1117 and X-ray detector 1116 ) may capture a series of two-dimensional projections of one or more objects within a specified region of interest at a work site in the Patient.
- the series of two-dimensional projections may then be used to generate a three-dimensional computer model of the one or more objects in a conventional manner for cone beam computed tomography.
- FIG. 13 illustrates, as an example, an alternative Patient-Side Cart 4010 which is usable in the robotic system 3000 to introduce a plurality of articulated instruments to a work site through a single entry aperture in the Patient by an entry guide 200 .
- the aperture may be a minimally invasive incision or a natural body orifice.
- the entry guide 200 is a cylindrical structure which is held and manipulated by a robot arm 4011 , which is mounted on base 4015 and includes a setup arm 4012 and an entry guide manipulator 4013 .
- the setup arm 4012 comprises a plurality of links and joints which are used to position the entry guide 200 at the aperture.
- the setup arm 4012 includes a prismatic joint for adjusting the height of the setup arm 4012 (as indicated by arrow “A”) and a plurality of rotary joints for adjusting the horizontal position of the setup arm 4012 (as indicated by arrows “B” and “C”).
- the entry guide manipulator 4013 is used to robotically pivot the entry guide 200 (and the articulated instruments disposed within it at the time) in yaw, pitch and roll angular rotations about the pivot point as indicated by arrows D, E and F, respectively.
- Articulated instrument manipulators (not shown) reside in housing 4014 .
- the entry guide 200 has articulated instruments such as articulated surgical instruments 231 , 241 and an articulated stereo camera instrument 211 (or other image capturing device instrument) extending out of its distal end.
- the camera instrument 211 has a pair of stereo image capturing elements 311 , 312 and a fiber optic cable 313 (coupled at its proximal end to a light source) housed in its tip.
- the surgical instruments 231 , 241 have working ends 331 , 341 .
- the entry guide 200 may guide additional instruments as required for performing a medical procedure at a work site in the Patient. For example, as shown in a cross-sectional view of the entry guide 200 in FIG.
- a passage 351 is available for extending another articulated surgical tool through the entry guide 200 and out through its distal end.
- Passages 431 , 441 are respectively used by the articulated surgical tool instruments 231 , 241 , and passage 321 is used for the articulated camera instrument 211 .
- the stereo vision display 45 of the Console 3030 displays stereo images derived from the stereo images captured by the articulated camera instrument 211 .
- the processor 43 of the Console 3030 translates and transfers the mechanical motion of the input devices 41 , 42 to actuate joints of devices, such as the entry guide 200 , articulated surgical instruments 231 , 241 , and articulated camera instrument 211 , which are associated at the time with the input devices 41 , 42 .
- Each of the articulated instruments comprises a plurality of actuatable joints and a plurality of links coupled to the joints.
- the second articulated instrument 241 comprises first, second, and third links 322 , 324 , 326 , first and second joints 323 , 325 , and a wrist assembly 327 .
- the first joint 323 couples the first and second links 322 , 324 and the second joint 325 couples the second and third links 324 , 326 so that the second link 324 may pivot about the first joint 323 in pitch and yaw while the first and third links 322 , 326 remain parallel to each other.
- the first articulated instrument 231 and the camera articulated instrument 211 may be similarly constructed and operated.
- the robotic system 1000 has a first manipulatable device 1001 , a first image capturing device 1002 , and a camera 1003 .
- the first manipulatable device 1001 may be an instrument such as the one of the instruments 33 , 35 held and manipulated by robot arms of the Patient-Side Cart 3010 .
- the first manipulatable device 1001 may be an entry guide through which articulated instruments extend such as the entry guide 200 held and manipulated by the robot arm of the Patient-Side Cart 4010 .
- a second manipulatable device 1004 is also shown which may be another device such as the first manipulatable device 1001 .
- a second image capturing device 1005 is also shown which may provide a different imaging modality than that of the first image capturing device 1002 .
- two manipulatable devices and two image capturing devices are shown for illustrative purposes, it is to be appreciated that in practice, more or less of each of such devices may be included in the robotic system 1000 to perform a task or procedure on an object at a work site. Additional cameras may also be included.
- a device controller 1021 controls movement of the robot arm 1011 to position and orient a working end of the first manipulatable device 1001 in response to commands from an input unit 1031 .
- the input unit 1031 may be a user operated input device, such as one of the input devices 41 , 42 or the foot pedal 44 of the console 3030 .
- the input unit 1031 may be a processor, such as the processor 43 of the console 3030 , executing stored program instructions.
- the input unit 1031 may be coupled control logic which communicates through bus 1040 with one or more of the controllers 1022 , 1023 , 1024 , 1025 and/or input units 1032 , 1033 , 1034 , 1035 associated with the first image capturing device 1002 , the camera 1003 , the second manipulatable device 1004 , and the second image capturing device 1005 .
- a device controller 1024 controls movement of the robot arm 1014 to position and orient a working end of the second manipulatable device 1004 in response to commands from an input unit 1034 .
- the input unit 1034 may be a user operated input device, such as one of the input devices 41 , 42 or the foot pedal 44 of the console 3030 .
- the input unit 1034 may be a processor, such as the processor 43 of the console 3030 , executing stored program instructions.
- the input unit 1034 may be coupled control logic which communicates through bus 1040 with one or more of the controllers 1021 , 1022 , 1023 , 1025 and/or input units 1031 , 1032 , 1033 , 1035 associated with the first manipulatable device 1001 , the first image capturing device 1002 , the camera 1003 , and the second image capturing device 1005 .
- the first image capturing device 1002 is manipulatable by its robot arm 1012 (or other motorized mechanism) to capture a plurality of two-dimensional image slices or projections of an object at the work site from which a three-dimensional model of the object may be computer generated without prior knowledge of the shape of the object using an imaging modality such as ultrasound, X-ray fluoroscopy, Computed Tomography (CT), and Magnetic Resonance Imaging (MRI).
- a controller 1022 controls movement of the robot arm 1012 to position and orient the first image capturing device 1002 in response to commands from an input unit 1032 .
- the input unit 1032 may be a user operated input device, such as one of the input devices 41 , 42 of the console 3030 .
- the input unit 1032 may be a processor, such as the processor 43 of the console 3030 , executing stored program instructions.
- the input unit 1032 may be coupled control logic which communicates through bus 1040 with one or more of the controllers 1021 , 1023 , 1024 , 1025 and/or input units 1031 , 1033 , 1034 , 1035 associated with the first manipulatable device 1001 , the camera 1003 , the second manipulatable device 1004 , and the second image capturing device 1005 .
- the first image capturing device 1002 and its robot arm 1012 may be combined to form an image capturing system such as the Image Capturing System 3020 .
- the second image capturing device 1005 may be similarly constructed and operated as the first image capturing device 1002 to capture a plurality of two-dimensional image projections of an object at the work site from which a three-dimensional model of the object may be computer generated without prior knowledge of the shape of the object using an imaging modality such as ultrasound, X-ray fluoroscopy, Computed Tomography (CT), and Magnetic Resonance Imaging (MRI).
- an imaging modality such as ultrasound, X-ray fluoroscopy, Computed Tomography (CT), and Magnetic Resonance Imaging (MRI).
- CT Computed Tomography
- MRI Magnetic Resonance Imaging
- different imaging modalities are provided by the first and second image capturing devices 1002 , 1005 .
- a controller 1025 controls movement of the robot arm 1015 to position and orient the second image capturing device 1005 in response to commands from an input unit 1035 .
- the input unit 1035 may be a user operated input device, such as one of the input devices 41 , 42 of the console 3030 .
- the input unit 1035 may be a processor, such as the processor 43 of the console 3030 , executing stored program instructions.
- the input unit 1035 may be coupled control logic which communicates through bus 1040 with one or more of the controllers 1021 , 1022 , 1023 , 1024 , and/or input units 1031 , 1032 , 1033 , 1034 , associated with the first manipulatable device 1001 , the first image capturing device 1002 , the camera 1003 , and the second manipulatable device 1004 .
- the second image capturing device 1005 and its robot arm 1015 may be combined to form an image capturing system such as the Image Capturing System 3020 .
- the second image capturing device 1005 may be constructed differently than that of the first image capturing device 1002 .
- the second image capturing device 1005 may be held, positioned, and oriented by one of the first and second manipulatable devices 1001 , 1004 rather than having its own robot arm, controller, and input unit.
- Examples of such a second image capturing device include a drop-in ultrasound probe or an optical coherent tomography probe.
- an ultrasound probe see, e.g., U.S. 2007/0021738 A1 entitled “Laparoscopic Ultrasound Robotic Surgical System,” which is incorporated herein by reference.
- the camera 1003 may be held and manipulated by a robot arm 1013 to capture stereoscopic images of work site, such as the endoscope 37 of FIG. 9 .
- the camera 1003 may be the articulated camera instrument 211 of FIG. 14 .
- a controller 1023 controls movement of the robot arm 1013 (or joints of the articulated camera instrument 211 ) to position and orient an image capturing element of the camera 1003 in response to commands from an input unit 1033 .
- the input unit 1033 may be a user operated input device, such as one of the input devices 41 , 42 of the console 3030 .
- the input unit 1033 may be a processor, such as the processor 43 of the console 3030 , executing stored program instructions.
- the input unit 1033 may be coupled control logic which communicates through bus 1040 with one or more of the controllers 1021 , 1022 , 1024 , 1025 and/or input units 1031 , 1032 , 1034 , 1035 associated with the first manipulatable device 1001 , the image capturing device 1002 , the second manipulatable device 1004 , and the second image capturing device 1005 .
- the camera 1003 may be held, positioned, and oriented by one of the first and second manipulatable devices 1001 , 1004 rather than having its own robot arm, controller, and input unit.
- the camera may be a tethered camera which is orientable by a manipulatable device pulling on the tether(s). In this case, the camera would not have its own robot arm, controller, and input unit.
- An example of such a tethered camera is described in U.S. Patent Application Publication No. 2012/02900134, entitled “Estimation of a Position and Orientation of a Frame Used in Controlling Movement of a Tool,” which is incorporated herein by reference.
- Each of the robot arms 1011 , 1012 , 1013 , 1014 , 1015 includes a plurality of links and a plurality of actuatable joints whose positions and/or velocities are sensed by a plurality of joint sensors.
- Information from the plurality of joint sensors of each robot arm 1011 , 1012 , 1013 , 1014 , 1015 is provided to its respective controller 1021 , 1022 , 1023 , 1024 , 1025 for control purposes and may be provided to one or more of the other controllers over bus 1040 for collision avoidance purposes, since this joint information indicates configuration information for the robot arms 1011 , 1012 , 1013 , 1014 , 1015 .
- Each of the control units 1021 - 1025 may include at least one processing unit and memory.
- the processing unit may control operation and/or execution of hardware and/or software in the respective control unit 1021 - 1025 .
- Each of the processing units may include one or more central processing units (CPUs), multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), custom processors/application specific integrated circuits (ASICs), and/or the like.
- the memory may be used to store one or more software and/or firmware applications as well as various data structures to be used by the respective control unit 1021 - 1025 .
- the memory may also include one or more types of machine readable media.
- machine readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.
- the robotic system 1000 also includes a stereo viewer or display 1051 for displaying stereo images which have been generated by an image processor 1050 from images captured by the stereo camera 1003 and/or three-dimensional computer models of one or more objects which have been generated by the image processor 1050 from the plurality of two-dimensional image slices of the one or more objects captured by the first image capturing device 1002 and/or second image capturing device 1005 .
- a stereo viewer or display 1051 for displaying stereo images which have been generated by an image processor 1050 from images captured by the stereo camera 1003 and/or three-dimensional computer models of one or more objects which have been generated by the image processor 1050 from the plurality of two-dimensional image slices of the one or more objects captured by the first image capturing device 1002 and/or second image capturing device 1005 .
- the image processor 1050 Before displaying images derived from both the camera 1003 and one or both of the first and second image capturing devices 1002 , 1005 concurrently on the display 1051 , the image processor 1050 registers the images so that the three-dimensional computer models of the one or more objects are properly superimposed on and positioned relative to images of the one or more objects in the stereo images derived from images captured by the stereo camera 1003 .
- FIGS. 3, 4 respectively illustrate, as examples, flow diagrams of methods 2000 , 2500 which may be implemented in the robotic system 1000 for avoiding a collision between independently operated image capturing device and manipulatable device robot arms.
- the methods 2000 , 2500 are essentially the same method taken from different perspectives.
- the method 2000 is from the perspective of an image capturing device reference frame and may be performed by an image capturing device controller and the method 2500 is from the perspective of a manipulatable device reference frame and may be performed by a manipulatable device controller.
- manipulatable device when the working end of more than one device is viewable within the field of view of the image capturing device, each of those devices are to be processed as the described manipulatable device according to the methods so that collisions may be avoided between their robot arms and the robot arm of the image capturing device.
- the robotic system 1000 is the medical robotic system 3000 of FIG. 8 and it includes the Patient-Side Cart 3010 of FIG. 9 , the relevant blocks of both methods 2000 , 2500 are performed for each of the manipulatable devices 31 , 33 , 35 , 37 that is viewable in images captured by the image capturing device 3020 .
- the robotic system 1000 is the medical robotic system 3000 of FIG.
- only one of the methods 2000 , 2500 is performed to avoid collisions between robot arms.
- only one of the device controllers preferably performs the collision avoidance method. The controller taking such action may be pre-established, user selected, or selected by certain criteria.
- the Surgeon may interact with a menu being displayed on the stereo viewer 45 to select either the control unit 3021 to perform the method 2000 or the processor 43 to perform the method 2500 .
- the control unit 3021 controls movement of the robot arm 1101 which holds and manipulates the image capturing device 1116 , 1117 of the Image Capturing System 3020 .
- the processor 43 implements a controller which controls movement of the robot arm 34 that holds and manipulates the instrument 33 , which may be the manipulatable device for the purpose of this example.
- the manipulatable device referred to in methods 2000 , 2500 is the entry guide 200 rather than one of the articulated instruments 211 , 231 , 241 .
- it is the entry guide manipulator 4013 that is at risk of colliding with or being struck by the robot arm 1101 of the Image Capturing System 3020 .
- the articulated instruments 211 , 231 , 241 do not have significant portions of any robot arms extending outside the Patient's body. They merely have manipulators whose movements are generally confined to be within a housing area 4014 of the entry guide manipulator 4013 as shown in FIG. 13 .
- collision avoidance between the robot arm 1101 of the Image Capturing System 3020 and the entry guide manipulator 4013 may be simpler than collision avoidance between the robot arm 1101 of the Image Capturing System 3020 and the robot arms 32 , 34 , 36 , 38 of the Patient-Side Cart 3010 .
- FIG. 2 illustrates, as an example, a flow diagram of a method 9000 , which is implemented in the robotic system 1000 , for registering an image capturing device robot arm and a manipulatable device robot arm.
- a method 9000 which is implemented in the robotic system 1000 , for registering an image capturing device robot arm and a manipulatable device robot arm.
- higher accuracy registrations are performed as the image capturing device robot arm moves closer to the manipulatable device robot arm.
- This multi-step approach provides high accuracy registration in a safe manner.
- the method performs a low accuracy registration (e.g., within an accuracy range of tens of centimeters) of the image capturing device robot arm and the manipulatable device robot arm relative to a common reference frame at their initial positions using an external tracker system and kinematic data for the robot arms.
- the external tracker system may be of the transmitter/receiver type which conventionally employs transmitters, which are strategically disposed on known locations of the robot arms of the image capturing device and the manipulatable device, and one or multiple receivers, which are disposed within transmission distance of the transmitters.
- the external tracker system may be of the optical type which conventionally employs optically discernible targets, which are strategically disposed on known locations of the robot arms of the image capturing device and the manipulatable device, and one or multiple optical detectors, which are disposed so as to have an unobstructed view of the targets.
- the kinematic data may be provided, for example, by encoders disposed to sense joint positions of the image capturing device robot arm and the manipulatable device robot arm. The joint positions may then be combined in a conventional manner using knowledge of the construction, shapes, and sizes of the robot arms to estimate the configurations of the robot arms.
- the image capturing device is moved towards the manipulatable device robot arm to assume a low-risk setup position that is close enough to allow the image capturing device being held by the image capturing device robot arm to capture images of at least a portion of the manipulatable device robot arm while being sufficiently far enough away from the manipulatable device robot arm to ensure that the image capturing device robot arm does not collide with the manipulatable device robot arm.
- the method also takes into account the current low accuracy level of the registration through a first safety margin that ensures no collision will occur between the image capturing device robot arm and the manipulatable device robot arm.
- the first safety margin may maintain a distance of at least ten centimeters between the image capturing device robot arm and the manipulatable device robot arm to account for the current low accuracy in determining the relative positions of those two robot arms.
- the movement of the image capturing device robot arm may be performed, for example, by an operator commanding such movement through a robot arm controller with assistance from the robot arm controller to inhibit unsafe movement.
- the robot arm controller may provide force feedback to the operator through a haptic device, such as a joystick, to provide resistance against the unsafe movement. Alternatively, it may be performed automatically by the robot arm controller in either a direct mode or cross-coupled mode.
- the method performs a mid accuracy registration (e.g., within an accuracy range of centimeters) of the image capturing device robot arm and the manipulatable device robot arm relative to the common reference frame at their current positions using one or more captured images of at least a part of the manipulatable device robot arm and kinematics data for the robot arms.
- a mid accuracy registration e.g., within an accuracy range of centimeters
- the method then waits for an operator's command to initiate movement of the image capturing device robot arm to capture images of a user specified region of interest.
- the image capturing device is moved towards the manipulatable device robot arm to assume an image capturing setup position that is close enough to allow the image capturing device being held by the image capturing device robot arm to capture images of a user specified region of interest while being sufficiently far enough away from the manipulatable device robot arm to ensure that the image capturing device robot arm does not collide with the manipulatable device robot arm.
- the method takes into account the current mid accuracy level of the registration through a second safety margin that ensures no collision will occur between the image capturing device robot arm and the manipulatable device robot arm.
- the second safety margin may maintain a distance of at least one centimeter between the image capturing device robot arm and the manipulatable device robot arm to account for the current mid accuracy in determining the relative positions of those two robot arms.
- the movement of the image capturing device robot arm may be performed, for example, by an operator commanding such movement through a robot arm controller with assistance from the robot arm controller to inhibit unsafe movement.
- the robot arm controller may provide force feedback to the operator through a haptic device, such as a joystick, to provide resistance against the unsafe movement.
- the movement may be performed automatically by the image capturing device controller directly or through a coupled control mode with the manipulatable device controller.
- the method commands the image capturing device robot arm to move relative to the region of interest and commands the image capturing device being held by the image capturing device robot arm to capture images of the region of interest during such movement while avoiding a collision of the image capturing device robot arm and the manipulatable device robot arm.
- the image capturing device captures images of at least a working end of a manipulatable device being held by the manipulatable device robot arm. In this case, the working end of the manipulatable device is proximate to the region of interest so as to be within the field of view of the image capturing device.
- the method performs at least one high accuracy registration (e.g., within an accuracy range of millimeters) of the image capturing device robot arm and the manipulatable device robot arm relative to the common reference frame at their current positions using one or more of the captured images of the working end of the manipulatable device and kinematics data for the robot arms.
- at least one high accuracy registration e.g., within an accuracy range of millimeters
- the method which is preferably performed by the image capturing device controller 1022 , receives information of the configuration of the robot arm that holds and manipulates the image capturing device.
- the robot arm comprises a plurality of links coupled together by a plurality of joints
- the robot arm configuration is determinable from sensed positions of the joints and known geometries of the links and other structure making up the robot arm.
- such robot arm configuration information may be provided from a plurality of sensors in the image capturing device robot arm, such as encoders which are coupled to actuators of the joints. Alternatively, it may be joint positions being commanded by the image capturing device controller 1022 in response to commands from the input device 1032 .
- the method determines whether or not it is time to perform a registration of the image capturing device to the manipulatable device. Such registration may be performed once at the start of the method or it may be performed periodically to correct any registration errors that may accrue over time. If the determination in block 2002 is YES, then the method performs the registration by performing blocks 2003 , 2004 . On the other hand, if the determination in block 2002 is NO, then registration is skipped by the method jumping to block 2005 .
- the method receives an image, which has been captured by the image capturing device, of the working end of the manipulatable device.
- the received image will be a two-dimensional slice or projection of the working end of the manipulatable device and any other objects within an image capturing cone of the image capturing device.
- the method determines the position and orientation (i.e., pose) of the working end of the manipulatable device relative to a reference frame of the image capturing device, which corresponds to a position and orientation of the image capturing cone or field of view of the image capturing device.
- the method conventionally uses artificial and/or natural features of the working end that are discernible in the received image.
- artificial features include such things as markings or structure specifically placed on the working end to aid in determining their pose (i.e., position and orientation).
- Natural features include such things as the shape and known geometries of structure of the working end of the manipulatable device. The pose of the working end may be determinable even though the working end may be partially occluded.
- the working end may be occluded by portions of a patient's anatomy, other portions of the manipulatable device, portions of the image capturing device, other medical instruments, and/or the like.
- the patient's anatomy may include soft tissue, bone, teeth, and/or the like.
- the sequence of images captured by the image capturing device and received as the method loops through blocks 2001 - 2009 may be used to refine the determination of the pose.
- conventional tool tracking techniques may be used that aid in and serve to further refine the determination of the pose of the working end of the manipulatable device. For additional details on such pose determining techniques and artifacts, see U.S. Pat. No.
- the method receives information of the configuration of the robot arm that holds and manipulates the manipulatable device.
- the robot arm configuration is determinable from sensed positions of the joints and known geometries of the links and other structure making up the robot arm.
- such robot arm configuration information may be provided from a plurality of sensors of the manipulatable device robot arm, such as encoders which are coupled to actuators of the joints.
- it may be joint positions being commanded by the manipulatable device controller 1021 in response to commands from the input device 1031 .
- it may be some joint positions being provided by tracking the working end of instruments in endoscopic images which may provide better accuracy compared to encoder measurements.
- the information of the configuration of the robot arm preferably includes either the sensed joint positions or commanded joint positions of the manipulatable device robot arm. If, on the other hand, there is only one robot arm configuration that may correspond to the pose of the working end of its held manipulatable device, then the robot arm configuration may theoretically be determined from the determined pose of the working end of the manipulatable device, the known construction and geometries (e.g., size and shape) of the manipulatable device and its robot arm, and the position of the base of the manipulatable device robot arm.
- DOF degree-of-freedom
- the base position of the manipulatable device robot arm is necessary.
- the base position is fixed and only needs to be determined once and stored in a memory.
- the base position may be determined, for example, by external sensors such as pressure sensors strategically disposed on the floor.
- the base position may be determined by a transmitter/receiver system in which one or more transmitters are disposed on the base with one or more receivers disposed at fixed locations.
- the base position (as well as the robot arm configuration) may be determined by an optical tracking system or by any other well known position sensing means.
- the method determines the configuration and position of the manipulatable device robot arm in the image capturing device frame of reference. Since the pose of the working end of the manipulatable device relative to the image capturing device reference frame has already been determined in block 2004 , the determination of the configuration and position of its robot arm in the image capturing device frame of reference is determined in this case by using the known construction and geometries of the manipulatable device and its robot arm along with the information of the manipulatable device robot arm configuration received in block 2005 .
- the method determines the configuration and position of the image capturing device robot arm in the image capturing device reference frame. Since the image capturing device reference frame is defined by the pose of the distal end of the image capturing device robot arm, it is a simple matter to determine the configuration and position of the image capturing device robot arm in the image capturing device reference frame using the known construction and geometries of the image capturing device and its robot arm along with the information of the image capturing device robot arm configuration received in block 2001 .
- the method determines whether an imminent collision is threatened between the image capturing device robot arm and the manipulatable device robot arm.
- the determination is made by using the information received or determined in blocks 2001 - 2007 for the current process cycle and previous process cycles.
- time series information not only can the trajectories of the two arms be predicted, but the rates at which they are moving may be estimated. With this information, a collision prediction may be made. When the predicted collision is within a specified time period, then the collision is considered to be imminent, requiring immediate action.
- the collision prediction may use a minimum distance between the image capturing device robot arm and the manipulatable device robot arm.
- the method preferably approximates links of the robot arms with geometric shapes that are dimensionally slightly larger than the actual links for safety purposes. Since the geometric shapes of the image capturing device robot arm and the geometric shapes of the manipulating device robot arm occupy known positions and orientations relative to each other at this point, it is a straightforward calculation to determine a minimum distance between the geometric shapes representing the image capturing device robot arm and the geometric shapes representing the manipulating device robot arm.
- block 2008 may be used to detect additional undesirable relationships between the manipulating device robot arm and the image capturing device robot arm other than an imminent collision.
- block 2008 may be used to detect when the manipulating device robot arm and the image capturing device robot arm are in too close a proximity to each other, even though a collision is not imminent.
- block 2008 may be used to detect that the manipulating device robot arm is obstructing a region of interest for which the image capturing device is to capture images thereof.
- the method jumps back to block 2001 to perform blocks 2001 - 2008 for a next process cycle.
- the method commands the image capturing device robot arm to take an action to avoid a collision with the robot arm of the manipulatable device.
- the commanded action may be to temporarily halt movement of the image capturing device robot arm until the manipulatable device robot arm has moved to a collision safe position.
- the speed of its movement may be adjusted instead to avoid collision with the manipulatable device robot arm.
- an alternative arm configuration may be commanded to avoid collision with the manipulatable device robot arm.
- the movement of the image capturing device robot arm may be halted and the collision avoidance task may be passed over to the manipulatable device controller to perform the method 2500 of FIG. 4 .
- the method After performing block 2009 , the method then loops back to block 2001 to process information for a next process cycle.
- the method which is preferably performed by the manipulatable device controller 1021 , receives information of the configuration of the robot arm that holds and manipulates the manipulatable device.
- the robot arm comprises a plurality of links coupled together by a plurality of joints
- the robot arm configuration is determinable from sensed positions of the joints and known geometries of the links and other structure making up the robot arm.
- such robot arm configuration information may be provided from a plurality of sensors in the image capturing device robot arm, such as encoders which are coupled to actuators of the joints.
- it may be joint positions being commanded by the manipulatable device controller 1021 in response to commands from the input device 1031 .
- it may be some joint positions being provided by tracking the working end of instruments in endoscopic images which may provide better accuracy compared to encoder measurements.
- the method determines whether or not it is time to perform a registration of the image capturing device to the manipulatable device. Such registration may be performed once at the start of the method or it may be performed periodically to correct any registration errors that may accrue over time. If the determination in block 2502 is YES, then the method performs the registration by performing blocks 2503 , 2504 . On the other hand, if the determination in block 2502 is NO, then registration is skipped by the method jumping to block 2505 .
- the method receives an image, which has been captured by the image capturing device, of the working end of the manipulatable device.
- the received image will be a two-dimensional slice or projection of the working end of the manipulatable device and any other objects within an image capturing cone or field of view of the image capturing device.
- the image may be received by the manipulatable device controller 1021 over a bus 1040 (or cable 3050 of FIG. 8 ) from the first image capturing device 1002 or a processor in an image capturing system that includes the first image capturing device 1002 .
- the method determines the position and orientation (i.e., pose) of the working end of the manipulatable device relative to a reference frame of the image capturing device in the same manner as described in reference to block 2004 of FIG. 3 .
- the method receives information of the configuration of the robot arm that holds and manipulates the image capturing device.
- the robot arm comprises a plurality of links coupled together by a plurality of joints
- the robot arm configuration is determinable from sensed positions of the joints and known geometries of the links and other structure making up the robot arm.
- such robot arm configuration information may be provided from a plurality of sensors of the manipulatable device robot arm, such as encoders which are coupled to actuators of the joints. Alternatively, it may be joint positions being commanded by the image capturing device controller 1022 in response to commands from the input device 1032 .
- the method determines the configuration and position of the manipulatable device robot arm in the manipulatable device reference frame. Since the manipulatable device reference frame is defined by the pose of the distal end of the manipulatable device robot arm, it is a simple matter to determine the configuration and position of the manipulatable device robot arm in the manipulatable device reference frame using the information received in block 2501 and the known construction and geometries of the manipulatable device and its robot arm.
- the method determines the configuration and position of the image capturing device robot arm in the manipulatable device frame of reference.
- One way to do this is to first determine the configuration and position of the image capturing device robot arm in the image capturing device reference frame, such as determined in block 2007 of FIG. 3 , then use the following transformation equation to translate points of the image capturing device robot arm from the image capturing device reference frame to manipulatable device reference frame:
- M P is a point in the manipulatable device reference frame “M”
- M I T is the image capturing device reference frame “I” to manipulatable device reference frame “M” transform
- I P is a point in the image capturing device reference frame “I”.
- the method may determine the transform M I T by comparing points of the working end of the manipulatable device in the image reference frame, using information of its pose determined in block 2504 , with corresponding points of the working end of the manipulatable device in the manipulatable device reference frame, using information of its pose determined from the information of the manipulatable device robot arm configuration which was received in block 2501 and prior known information of the size, shape, and construction of the manipulatable device.
- block 2508 the method determines whether an imminent collision is threatened between the image capturing device robot arm and the manipulatable device robot arm in a similar manner as described in reference to block 2008 of FIG. 3 .
- block 2508 may also be used to detect additional undesirable relationships between the manipulating device robot arm and the image capturing device robot arm other than an imminent collision.
- the method may command either the manipulatable device robot arm or the image capturing device robot arm, through the image capturing device controller 1022 , to take a collision avoidance action.
- the entity performing the collision avoidance action and the particular collision avoidance action taken may take different forms depending upon system factors. One factor is the nature of the tasks being performed at the time by the image capturing device and the manipulatable device. Another factor is the type or structure of the robot arms or their corresponding velocity and acceleration at the moment when an imminent collision is detected.
- the robotic system 1000 is the medical robotic system 3000 and the manipulatable device is one of the instruments 33 , 35 which is being used at the time to perform a delicate surgery on a Patient, then it is desirable not to disturb the robot arm holding the instrument. Therefore, in this case it may be preferable to take action by modifying the trajectory of the image capturing device robot arm 1101 to avoid a collision. To avoid such collision, movement of the image capturing device robot arm 1101 may be temporarily halted until the manipulatable device robot arm has moved to a collision safe position. Alternatively, rather than halting all movement of the image capturing device robot arm 1101 , the speed and/or direction of its movement may be adjusted instead to avoid collision with the manipulatable device robot arm.
- the image capturing device and manipulatable device robot arms may have redundant degrees-of-freedom (DOF)
- DOF degrees-of-freedom
- one of the robot arms with such redundant DOF may be configured in an alternative configuration without affecting the pose of its held device.
- the manipulatable device robot arm has redundant DOF, then it would not be necessary to modify the trajectory of the image capturing device since the manipulatable device robot arm may be configured instead to an alternative configuration without significantly impacting the manipulation of its held manipulatable device.
- one of the first and second robot arms is selected for using an alternative configuration to avoid the imminent collision by processing differences between current configurations of the first and second robot arms and their respective pluralities of possible configurations to minimize a cost function.
- the selection may be made by processing required joint movements of the first and second robotic systems to move from their current configurations to others of their respective pluralities of possible configurations to minimize a cost function.
- One possible cost function may be based on an inverse square of the minimum distance between the first and second robot arms.
- a second possible cost function may be based on a weighted average of an inverse square of distances between links of the first and second robot arm, with each link being associated with a passive or active degree of freedom.
- a third possible cost function may be based on the second cost function, but may account for only distances which are below a threshold, such as a threshold within an order of magnitude of a desired safety margin.
- a fourth possible cost function may be based on a modified version of the second cost function where the distances are measured between virtual objects or buffer regions surrounding the links of the first and second robot arms.
- a fifth possible cost function may include the buffer regions of the fourth cost function and the distance threshold of the third cost function.
- a sixth possible cost function may be based on a constrained manipulability index for the first and second robot arms with the manipulability index estimating an ability of the first or second robot arm to move in arbitrary directions around the alternative configuration.
- a seventh possible cost function may be based on virtual potential fields applied to the links and/or joints of the first and second robot arms. As an example, each link and/or joint of the first and second robot arms may have a virtual electric charge assigned to it and the induced virtual repelling force may be integrated between the first and second robot arms to determine the seventh cost function.
- the method After performing block 2509 , the method then loops back to block 2501 to process information for a next process cycle.
- an update or recalibration of initial registration transforms for work site objects may be performed.
- the robotic system 1000 is the medical robotic system 3000
- the work site objects include the instruments 31 , 33 , 35 , the endoscope 37 , the image capturing device 1116 , 1117 of the Image Capturing System 3020 , and the Patient's anatomy.
- registration errors may accumulate or otherwise occur in some manner.
- the image capturing device and the manipulatable device are independently operated typically by different people.
- an Assistant standing next to the Operating Table 3040 may operate the control unit 3021 to control movement of the Image Capturing System 3020 and the Surgeon may operate the input devices 41 , 42 of the Console 3030 to control movement of the manipulatable devices 33 , 35 .
- the Surgeon may want to supplement those images with three-dimensional X-ray images of an interior view of the anatomical structure.
- a method for automatically controlling the Image Capturing System 3020 to provide the Surgeon's desired view of a region of interest is useful.
- FIG. 5 illustrates, as an example, a method 4000 implemented in the robotic system 1000 for automatically controlling movement of the image capturing device so as to capture images of a user specified region of interest while avoiding a collision between its robot arm and another robot arm.
- the method receives information of a user specified region of interest at the work site.
- the region of interest is specified relative to an image of the work site being displayed at the time on the display 1051 .
- the images being displayed on the display 1051 may have been derived from images captured by a camera 1003 such as a stereoscopic endoscope. Alternatively, they may have been derived from images captured by a second image capturing device 1005 such as an ultrasound probe. Alternatively, the images being displayed on the display 1051 may include both images derived from those captured by a camera 1003 and images derived from those captured by a second image capturing device 1005 by registering and superimposing the images.
- the user may specify the region of interest by simply using an input device such as a mouse to control movement of a cursor on the display 1051 so that the path of the cursor defines the region of interest.
- the user may use a gaze tracker and a user controlled switch to indicate when the user's current gaze point on the display 1051 is to specify a center of a region of interest.
- the shape of the region of interest may be predefined or it may be determined by the shape of an object upon which the user is currently gazing.
- the user may telerobotically control movement of a working end of the manipulatable device so that it touches an object at the work site to indicate that the region of interest should be the object that has been touched.
- the method receives information of the initial arm configurations and base positions of the image capturing device robot arm and a manipulatable device robot arm.
- Information of the initial arm configurations may be provided by information of their construction and information of their joint positions which is received from a plurality of sensors of the robot arms.
- Information of the base positions in the world reference frame may be provided by external sensors which provide such information.
- One example of such an external sensor is an optical sensor that is used in an optical tracking system.
- a pressure transducer that is one of an array of pressure transducers strategically placed at known positions on the floor of the work site.
- the method determines a set-up position for the image capturing device so that it is properly positioned to capture the requested images of the user specified region of interest.
- the method first translates the region of interest from the display reference frame to an image capturing device reference frame using a previously determined transform and information received in block 4002 . It then translates the region of interest from the image capturing device reference frame to the world reference frame using a previously determined transform and information received in block 4002 . With the region of interest known in the world reference frame, the method then determines a set-up position from which the image capturing device may capture the requested images of the user specified region of interest without colliding with the initial position in the world reference frame of the manipulatable device robot arm.
- the method determines a plan of motion for the image capturing device to capture the requested images of the user specified region of interest.
- the plan of motion may comprise planned movement of the links 1102 , 1103 , the wrist 1104 and the C-arm 1105 .
- the method commands movement of the image capturing device to its set-up position after receiving a command to do so from the user.
- the commanded movement typically entails rotating the carousel joint 1108 so that the robot arm 1101 is facing the region of interest and rotating the shoulder joint 1109 , elbow joint 1110 , roll joint 1111 , pitch joint 1112 , and yaw joint 1121 so that the X-ray source 1117 and X-ray detector 1116 (i.e., the image capturing device) are properly positioned to start capturing the requested images of the user specified region of interest.
- the method commands incremental movement of the image capturing device in accordance with the plan of motion while avoiding a collision between the image capturing device robot arm and the manipulatable device robot arm. Additional details of tasks performed by the method in block 4006 are described in reference to FIG. 6 .
- the method determines whether the plan of motion is completed. If the determination in block 4007 is NO, then the method jumps back to block 4006 to command another incremental movement of the image capturing device according to the plan of motion while avoiding a collision between the image capturing device robot arm and the manipulatable device robot arm.
- the method commands the image capturing device to move back to its initial position or a predefined parking position upon receiving a command to do so from the user, so that it is out of the way and presents no further risk of collision with the manipulatable device robot arm.
- FIG. 6 illustrates, as an example, a flow diagram of a method for performing block 4006 of the method 4000 .
- FIG. 7 illustrates a block diagram of a portion of a robotic system which is used for collision avoidance between two independently operated robot arms.
- a manipulatable device robot arm simulator 6003 an imminent collision detector 6021 , an image capturing device robot arm simulator 6013 , and an incremental motion generator 6004 are preferably implemented in a conventional manner by a processor executing program code.
- the method receives a movement command for the manipulatable device robot arm.
- the output of the manipulatable device controller 6011 provides joint commands for the manipulatable device robot arm actuators 6012 .
- the method determines the position and configuration for the manipulatable device robot arm. For example, referring to FIG. 7 , the manipulatable device robot arm simulator unit 6003 generates information of the position and configuration for the manipulatable device robot arm using the joint commands from the manipulatable device controller 6011 and prior knowledge of the construction of the manipulatable device robot arm.
- the method determines a desired incremental movement of the image capturing device.
- the incremental motion generator 6004 generates a desired incremental movement of the image capturing device using information of the plan of motion determined in block 4004 of FIG. 5 and stored in memory 6005 .
- the desired incremental movement in this case depends upon a desired velocity for the movement of the image capturing device and a process cycle period for looping through blocks 4006 - 4007 of FIG. 5 .
- the method determines the position and configuration for the image capturing device robot arm. For example, referring to FIG. 7 , the image capturing device robot arm simulator unit 6013 generates information of the position and configuration for the image capturing device robot arm using the desired incremental motion commands provided by the incremental motion generator 6004 , knowledge of the current position and configuration of the image capturing device robot arm, and prior knowledge of the construction of the image capturing device robot arm.
- the method determines whether an imminent collision is threatened between the image capturing device robot arm and the manipulatable device robot arm in a similar manner as performed in block 2008 of FIG. 3 .
- the imminent collision detector 6021 determines whether a collision between the two arms is imminent by using the information provided by the robot arm simulators 6003 , 6013 for the current process cycle and previous process cycles. By using time series information, not only can the trajectories of the two robot arms be predicted, but the rates at which they are moving may be estimated. With this information, a collision prediction may be made. When the predicted collision is within a specified time period, then the collision is considered to be imminent, requiring immediate action.
- the method commands the image capturing device robot arm to move according to the desired incremental movement.
- the method modifies the incremental movement command to avoid the imminent collision, using as examples, one of the actions described in reference to block 2009 of FIG. 3 .
- the imminent collision detector 6021 After the imminent collision detector 6021 has determined that a collision between the image capturing device robot arm and the manipulatable device robot arm is imminent, it sends an indication of such to the manipulatable device controller 6011 , which in turn, relays the indication to the image capturing device controller 6001 so that it can take corrective action to avoid the collision.
- control units 1021 - 1025 and/or 3021 may include non-transient, tangible, machine readable media that include executable code that when run by one or more processors may cause the one or more processors (e.g., the processing units of control units 1021 - 1025 and/or 3021 ) to perform the processes of methods 2000 , 2500 , 4000 , and/or 9000 as described above.
- processors e.g., the processing units of control units 1021 - 1025 and/or 3021
- Some common forms of machine readable media that may include the processes of methods 2000 , 2500 , 4000 , and/or 9000 are, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Medical Informatics (AREA)
- Surgery (AREA)
- Public Health (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Veterinary Medicine (AREA)
- Robotics (AREA)
- Pathology (AREA)
- Biophysics (AREA)
- High Energy & Nuclear Physics (AREA)
- Optics & Photonics (AREA)
- Physics & Mathematics (AREA)
- Radiology & Medical Imaging (AREA)
- Mechanical Engineering (AREA)
- Automation & Control Theory (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Manipulator (AREA)
Abstract
Description
- This application is a continuation of U.S. patent application Ser. No. 14/862,692 filed Sep. 23, 2015, which is a continuation of U.S. Pat. No. 9,259,282 filed Dec. 10, 2013, and claims priority to U.S. Provisional Application No. 61/735,170 filed Dec. 10, 2012 and entitled “Collision Avoidance During Controlled Movement of Image Capturing Device and Manipulatable Robot Arms”, each of which is incorporated herein in their entirety by reference.
- The present invention generally relates to robotic systems and in particular, to collision avoidance during controlled movement of image capturing device and manipulatable device robot arms.
- Robotic systems and computer-assisted devices often include robot or movable arms to manipulate instruments for performing a task at a work site and at least one robot or movable arm for supporting an image capturing device which captures images of the work site. When the robot arms are operating within close proximity to each other, there is a possibility that the arms may collide with one another and by so doing, cause damage to the arms. When the robot arms are commonly operated, preventing such collisions may be straightforward. However, when the robot arms are independently operated, collision avoidance may be much more challenging.
- A robot arm comprises a plurality of links coupled together by one or more actively controlled joints. In many embodiments, a plurality of actively controlled joints may be provided. The robot arm may also include one or more passive joints, which are not actively controlled, but comply with movement of an actively controlled joint. Such active and passive joints may be revolute or prismatic joints. The configuration of the robot arm may then be determined by the positions of the joints and knowledge of the structure and coupling of the links.
- To avoid collisions between robot arms it is useful to have information of the configurations of the arms and information of their respective positions in a common reference frame. With this information, a controller that is controlling the movement of one of the robot arms can take action to avoid a collision with another robot arm.
- One action that the controller may take to avoid collisions is to warn an operator who is commanding a robot arm that a collision with another robot arm is imminent. As one example of such a warning system, U.S. 2009/0192524 A1 entitled “Synthetic Representation of a Surgical Robot,” which is incorporated herein by reference, describes a patient side cart upon which a plurality of robot arms is mounted. As another example of such a warning system, U.S. 2009/0326553 entitled “Medical Robotic System Providing an Auxiliary View of Articulatable Instruments Extending out of a Distal End of an Entry Guide,” which is incorporated herein by reference, describes a medical robotic system having a plurality of articulated instruments extending out of an entry guide. As yet another example of such a warning system, U.S. 2009/0234444 A1 entitled “Method and Apparatus for Conducting an Interventional Procedure Involving Heart Valves using a Robot-based X-ray Device,” which is incorporated herein by reference, describes a medical robotic system in which an X-ray source and detector are mounted on opposing ends of a C-arm so that X-ray images of a patient's anatomy can be captured during the performance of a medical procedure using a catheter control robot.
- Collision avoidance may also be performed automatically by a controller which is controlling movement of one or more robot arms. As an example of such an automatic collision avoidance system, U.S. Pat. No. 8,004,229 entitled “Software center and highly configurable robotic systems for surgery and other uses,” which is incorporated herein by reference, describes a medical robotic system that is configured to avoid collisions between its robot arms. The robot arms have redundancy so that multiple configurations are possible for each arm to achieve a desired position and orientation of its held instrument. Each controller commands movement of its associated robot arm subject to a secondary constraint that results in eliminating possible arm configurations that would result in a collision with another robot arm.
- When a plurality of robot arms is controlled by a plurality of controllers, however, confusion and unintended consequences may result if more than one of the plurality of controllers is attempting to avoid collisions. This problem is exacerbated when there is none or only limited communication of information between the controllers, such as may be the case when the robotic system employs independently operated robot arms. To avoid robot arm collision problems, the independently operated robot arms may be used sequentially, but not concurrently at the work site. However, it may be advantageous to use the robot arms concurrently during the performance of a procedure or task at a work site.
- Accordingly, one object of one or more aspects of the present invention is a robotic system and method implemented therein that automatically performs collision avoidance of robot arms.
- Another object of one or more aspects of the present invention is a robotic system and method implemented therein that automatically performs collision avoidance of independently operated robot arms.
- Another object of one or more aspects of the present invention is a robotic system and method implemented therein that automatically performs collision avoidance of robot arms without preventing any of the robot arms from performing its intended task.
- These and additional objects are accomplished by the various aspects of the present invention, wherein briefly stated, one aspect is a robotic system including: a first robot arm holding a manipulatable device having a working end; and a controller configured to control movement of at least one of the first robot arm and a second robot arm while avoiding a collision between the first and second robot arms, wherein the second robot arm holds an image capturing device for capturing a plurality of images of the working end of the manipulatable device from which a three-dimensional computer model of the working end of the manipulatable device is generatable; wherein the controller is configured to avoid the collision between the first and second robot arms: by determining a position and an orientation of the working end of the manipulatable device relative to a reference frame of the image capturing device by using at least one image of the plurality of images of the working end of the manipulatable device, wherein the reference frame of the image capturing device corresponds to a perspective of the captured images; by determining a configuration and position of one of the first and second robot arms relative to a common reference frame by using the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device; and by determining a configuration and position of the other of the first and second robot arms relative to the common reference frame by using joint position information received for the other of the first and second robot arms; by determining an imminent collision between the first and second robot arms using the determined configurations and positions of the first and second robot arms relative to the common reference frame; and by commanding an action to be taken by one of the first and second robot arms to avoid the imminent collision.
- Another aspect is a method implemented in a robotic system for avoiding collisions between first and second robot arms, wherein the first robot arm holds a manipulatable device having a working end, wherein the second robot arm holds an image capturing device for capturing a plurality of images of the working end of the manipulatable device from which a three-dimensional computer model of the working end of the manipulatable device is generatable, and wherein the method includes: determining a position and an orientation of the working end of the manipulatable device relative to a reference frame of the image capturing device by using at least one image of the plurality of images of the working end of the manipulatable device, wherein the reference frame of the image capturing device corresponds to a perspective of the captured images; determining a configuration and position of one of the first and second robot arms relative to a common reference frame by using the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device; determining a configuration and position of the other of the first and second robot arms relative to the common reference frame by using joint position information received for the other of the first and second robot arms; determining an imminent collision between the first and second robot arms using the determined configurations and positions of the first and second robot arms relative to the common reference frame; and commanding an action to be taken by one of the first and second robot arms to avoid the imminent collision.
- Another aspect is a robotic system including: a first robot arm that is holding a manipulatable device having a working end; and a processor programmed to register the first robot arm to a second robot arm that is holding an image capturing device, wherein the processor performs the registration by: performing a low accuracy registration of the first and second robot arms at initial positions of the first and second robot arms using external tracking data and kinematics data of at least one of the first and second robot arms; performing a mid accuracy registration of the first and second robot arms relative to a first setup position using at least one image captured by the image capturing device of at least a part of the first robot arm and kinematics data of at least one of the first and second robot arms, wherein the first setup position includes a first safety margin indicative of the low accuracy registration; and performing a high accuracy registration of the first and second robot arms relative to a second setup position using at least one image captured by the image capturing device of the working end of the manipulatable device and kinematics data of at least one of the first and second robot arms, wherein the second setup position includes a second safety margin indicative of the mid accuracy registration.
- Another aspect is a method implemented by a processor for registering first and second robot arms. The first robot arm holds a manipulatable device having a working end. The second robot arm holds an image capturing device. The method includes performing a low accuracy registration of the first and second robot arms at initial positions of the first and second robot arms using external tracking data and kinematics data of at least one of the first and second robot arms; performing a mid accuracy registration of the first and second robot arms relative to a first setup position using at least one image captured by the image capturing device of at least a part of the first robot arm and kinematics data of at least one of the first and second robot arms, wherein the first setup position includes a first safety margin indicative of the low accuracy registration; and performing a high accuracy registration of the first and second robot arms relative to a second setup position using at least one image captured by the image capturing device of the working end of the manipulatable device and kinematics data of at least one of the first and second robot arms, wherein the second setup position includes a second safety margin indicative of the mid accuracy registration.
- Consistent with some embodiments, a movement control system includes a controller. The controller includes one or more processors and memory coupled to the one or more processors. The controller is coupled to a computer-assisted surgical device having a first movable arm coupled to a manipulatable device having a working end. The controller is further coupled to a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device. The controller is configured to receive one or more first configurations for the first movable arm; receive one or more second configurations for the second movable arm; receive a first plurality of images of the working end from the image capturing device; determine, based on at least one of the first plurality of images, a position and an orientation of the working end in a common reference frame; determine, based on the first configurations, a first movable arm position and a first movable arm trajectory for the first movable arm in the common reference frame; determine, based on the second configurations, a second movable arm position and a second movable arm trajectory for the second movable arm in the common reference frame; based on the first movable arm position, the first movable arm trajectory, the second movable arm position, and the second movable arm trajectory, determine whether motion of the first movable arm, motion of the second movable arm, or motions of the first and second movable arms together will result in an undesirable relationship between the first and second movable arms; and send a first movement command to the first movable arm or the second movable arm to avoid the undesirable relationship.
- In some examples, the common reference frame is a reference frame of the image capturing device and the controller is further configured to determine the first movable arm position in the common reference frame further based on the position and the orientation of the working end in the common reference frame. In some examples, the common reference frame is a reference frame of the computer-assisted surgical device and the controller is further configured to determine the second movable arm position in a reference frame of the image capturing device, determine the position and the orientation of the working end in the reference frame of the image capturing device, and transform the second movable arm position and the position and orientation of the working end from the reference frame of the image capturing device to the common reference frame.
- In some examples, the first plurality of images are a plurality of two-dimensional images from a image capturing device and the controller is further configured to determine the position and the orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device. In some examples, the system further includes a viewer adapted to display a second plurality images of a work space of the working end of the manipulatable device and an input unit configured to receive information of a user specified region of interest within the images being displayed on the viewer. The controller is further configured to send second movement commands to the second movable arm so that the image capturing device captures images of the user specified region of interest.
- In some examples, the input unit includes a gaze tracking unit configured to track a gaze of a user on a display screen of the viewer and the gaze tracking unit includes an indicator operable by the user to indicate the region of interest by using a current gaze point of the user on the display screen. In some examples, the input unit includes a telestrator unit configured to receive one of the second plurality of images and display the received one of the second plurality of images on a display screen of the telestrator and the telestrator unit includes a marker unit operable by a user to indicate the region of interest on the displayed one of the second plurality of images.
- In some examples, the system further includes a camera, and the second plurality of images are captured by the camera. In some examples, system further includes an ultrasound probe, and the second plurality of images are captured by the ultrasound probe. In some examples, the first movable arm has redundant degrees of freedom so that for each controllable position and orientation of the working end of the manipulatable device there are a first plurality of possible positions and orientations for the first movable arm, the second movable arm has redundant degrees of freedom so that for each controllable position and orientation of the image capturing device there are a second plurality of possible positions and orientations for the second movable arm, and the first movement command sent to the first movable arm or the second movable arm directs the first movable arm to move to one of the first plurality of possible positions and orientations or directs the second movable arm to move to one of the second plurality of possible positions and orientations that avoid the undesirable relationship.
- In some examples, the controller is further configured to determine which one of the first movable arm and the second movable arms is to be sent the first movement command based on differences between the first movable arm position and the first plurality of possible positions and orientations and differences between the second movable arm position and the second plurality of possible positions and orientations and the determination is made so as to minimize a cost function.
- In some examples, the first movement command directs the first movable arm to move from the first movable arm position to a first selected one of the first plurality of possible positions and orientations or directs the second movable arm to move from the second movable arm position to a second selected one of the second plurality of possible positions and orientations based on which of the first selected one of the first plurality of possible positions and orientations and the second selected one of the second plurality of possible positions and orientations minimizes the cost function. In some examples, the undesirable relationship is selected from a group consisting of a collision between the first movable arm and the second movable arm, too close a proximity between the first movable arm and the second movable arm, and obstruction of a region of interest of the image capturing device by the first movable arm.
- Consistent with some embodiments, a method of controlling movement in a medical system includes receiving one or more first configurations for a first movable arm of a computer-assisted surgical device, the first movable arm being coupled to a manipulatable device having a working end; receiving one or more second configurations for a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device; receiving a first plurality of images of the working end from the image capturing device; determining, based on at least one of the first plurality of images, a position and an orientation of the working end in a common reference frame; determining, based on the first configurations, a first movable arm position and a first movable arm trajectory for the first movable arm in the common reference frame; determining, based on the second configurations, a second movable arm position and a second movable arm trajectory for the second movable arm in the common reference frame; based on the first movable arm position, the first movable arm trajectory, the second movable arm position, and the second movable arm trajectory, determining whether motion of the first movable arm, motion of the second movable arm, or motions of the first and second movable arms together will result in an undesirable relationship between the first and second movable arms; and sending a first movement command to the first movable arm or the second movable arm to avoid the undesirable relationship.
- In some examples, the common reference frame is a reference frame of the image capturing device, and the method further includes determining the first movable arm position in the common reference frame further based on the position and the orientation of the working end in the common reference frame. In some examples, the common reference frame is a reference frame of the computer-assisted surgical device and the method further includes determining the second movable arm position in a reference frame of the image capturing device, determining the position and the orientation of the working end in the reference frame of the image capturing device, and transforming the second movable arm position and the position and orientation of the working end from the reference frame of the image capturing device to the common reference frame.
- In some examples, the first plurality of images are a plurality of two-dimensional images from a image capturing device and the method further includes determining the position and the orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device. In some examples, the method further includes determining the first movement command so as to minimize a cost function.
- Consistent with some embodiments, a movement control system includes a tracking system and a controller coupled to the tracking system. The controller includes one or more processors and memory coupled to the one or more processors. The controller is coupled to a computer-assisted surgical device having a first movable arm coupled to a manipulatable device having a working end. The controller is further coupled to a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device. The controller is configured to receive first kinematic data for the first movable arm; receive second kinematic data for the second movable arm; receive first tracking data for the first movable arm from the tracking system; receive second tracking data for the second movable arm from the tracking system; determine, based on the first kinematic data and the first tracking data, a first location of the first movable arm; determine, based on the second kinematic data and the second tracking data, a second location of the second movable arm; send a first movement command to the second movable arm that directs the second movable arm to move into a first set-up position to capture images of at least a portion of the manipulatable device while maintaining a first safety margin between the first movable arm and the second movable arm, the first movement command being based on the first location and the second location; receive third kinematic data for the first movable arm; receive fourth kinematic data for the second movable arm; receive one or more first images from the image capturing device, the one or more first images capturing at least a portion of the manipulatable device; determine, based on the third kinematic data and the one or more first images, a third location of the first movable arm; determine, based on the fourth kinematic data and the one or more first images, a fourth location of the second movable arm; and send a second movement command to the second movable arm that directs the second movable arm to move into a second set-up position, different from the first set-up position, to capture images of a region of interest while maintaining a second safety margin between the first movable arm and the second movable arm, the second movement command being based on the third location and the fourth location.
- In some examples, the controller is further configured to receive fifth kinematic data for the first movable arm; receive sixth kinematic data for the second movable arm; receive one or more second images from the image capturing device, the one or more second images capturing at least a portion of the working end; determine, based on the fifth kinematic data and the one or more second images, a fifth location of the first movable arm; determine, based on the sixth kinematic data and the one or more second images, a sixth location of the second movable arm; and send a third movement command to the second movable arm that directs the second movable arm to move into a third set-up position, different from the first set-up position and the second set-up position, to capture images of the working end while maintaining a third safety margin between the first movable arm and the second movable arm, the third movement command being based on the fifth location and the sixth location.
- In some examples, the second safety margin is an order of magnitude smaller than the first safety margin. In some examples, maintaining the first safety margin keeps a separation between the first movable arm and the second movable arm of at least ten centimeters and maintaining the second safety margin keeps a separation between the first movable arm and the second movable arm of at least one centimeter. In some examples, the third safety margin is an order of magnitude smaller than the second safety margin. In some examples, maintaining the second safety margin keeps a separation between the first movable arm and the second movable arm of at least one centimeter and maintaining the third safety margin keeps a separation between the first movable arm and the second movable arm of at least one millimeter.
- In some examples, the first images are two-dimensional images from a image capturing device and the controller is further configured to determine a position and an orientation of the manipulatable device from the two-dimensional images even when the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device. In some examples, the second images are two-dimensional images from a image capturing device and the controller is further configured to determine a position and an orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device. In some examples, the system further includes a viewer adapted to display one more second images received from the image capturing device and an input unit configured to receive information of the region of interest within the second images.
- In some examples, the input unit includes a gaze tracking unit configured to track a gaze of a user on a display screen of the viewer and the gaze tracking unit includes an indicator operable by the user to indicate the region of interest by using a current gaze point of the user on the display screen. In some examples, the input unit includes a telestrator unit configured to receive one of the second of images and display the received one of the second images on a display screen of the telestrator and the telestrator unit includes a marker unit operable by a user to indicate the region of interest on the displayed one of the second images. In some examples, the system further includes a camera and the second images are captured by the camera. In some examples, the system further includes an ultrasound probe and the second images are captured by the ultrasound probe. In some examples, the first movement command minimizes a cost function.
- In some examples, the second movable arm has redundant degrees of freedom so that for each controllable position and orientation of the image capturing device there are a plurality of possible positions and orientations for the second movable arm, the first movement command sent to the second movable arm directs the second movable arm to move to one of the plurality of possible positions and orientations that maintain the first safety margin, the second movement command sent to the second movable arm directs the second movable arm to move to one of the plurality of possible positions and orientations that maintain the second safety margin, and the third movement command sent to the second movable arm directs the second movable arm to move to one of the plurality of possible positions and orientations that maintain the third safety margin.
- Consistent with some embodiments, a method of controlling movement in a medical system includes receiving first kinematic data for a first movable arm of a computer-assisted surgical device, the first movable arm being coupled to a manipulatable device having a working end; receiving second kinematic data for a second movable arm coupled to an image capturing device separate from the computer-assisted surgical device; receiving first tracking data for the first movable arm from a tracking system; receiving second tracking data for the second movable arm from the tracking system; determining, based on the first kinematic data and the first tracking data, a first location of the first movable arm; determining, based on the second kinematic data and the second tracking data, a second location of the second movable arm; sending a first movement command to the second movable arm that directs the second movable arm to move into a first set-up position to capture images of at least a portion of the manipulatable device while maintaining a first safety margin between the first movable arm and the second movable arm, the first movement command being based on the first location and the second location; receiving third kinematic data for the first movable arm; receiving fourth kinematic data for the second movable arm; receiving one or more first images from the image capturing device, the one or more first images capturing at least a portion of the manipulatable device; determining, based on the third kinematic data and the one or more first images, a third location of the first movable arm; determining, based on the fourth kinematic data and the one or more first images, a fourth location of the second movable arm; and sending a second movement command to the second movable arm that directs the second movable arm to move into a second set-up position, different from the first set-up position, to capture images of a region of interest while maintaining a second safety margin between the first movable arm and the second movable arm, the second movement command being based on the third location and the fourth location.
- In some examples, the method further includes receiving fifth kinematic data for the first movable arm; receiving sixth kinematic data for the second movable arm; receiving one or more second images from the image capturing device, the one or more second images capturing at least a portion of the working end; determining, based on the fifth kinematic data and the one or more second images, a fifth location of the first movable arm; determining, based on the sixth kinematic data and the one or more second images, a sixth location of the second movable arm; and sending a third movement command to the second movable arm that directs the second movable arm to move into a third set-up position, different from the first set-up position and the second set-up position, to capture images of the working end while maintaining a third safety margin between the first movable arm and the second movable arm, the third movement command being based on the fifth location and the sixth location.
- In some examples, the method further includes determining the third movement command so as to minimize a cost function. In some examples, the first images are two-dimensional images from a image capturing device and the method further includes determining a position and an orientation of the manipulatable device from the two-dimensional images even when the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device. In some examples, the second images are two-dimensional images from a image capturing device and the method further includes determining a position and an orientation of the working end from the two-dimensional images even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device.
- Consistent with some embodiments, a robotic system includes a first robot arm holding a manipulatable device having a working end and a controller configured to control movement of at least one of the first robot arm and a second robot arm while avoiding a collision between the first and second robot arms. The second robot arm holds an image capturing device for capturing a plurality of images of the working end of the manipulatable device from which a three-dimensional computer model of the working end of the manipulatable device is generatable. The controller is configured to avoid the collision between the first and second robot arms by determining a position and an orientation of the working end of the manipulatable device relative to a reference frame of the image capturing device by using at least one image of the plurality of images of the working end of the manipulatable device, wherein the reference frame of the image capturing device corresponds to a perspective of the captured images; by determining a configuration and position of one of the first and second robot arms relative to a common reference frame by using the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device; and by determining a configuration and position of the other of the first and second robot arms relative to the common reference frame by using joint position information received for the other of the first and second robot arms; by determining an imminent collision between the first and second robot arms using the determined configurations and positions of the first and second robot arms relative to the common reference frame; and by commanding an action to be taken by one of the first and second robot arms to avoid the imminent collision.
- In some examples, the image capturing device includes a tomographic image capturing device for capturing a plurality of two-dimensional images from which the working end of the manipulatable device is discernible even when the working end of the manipulatable device is occluded by one or more objects disposed between the image capturing device and the working end of the manipulatable device. In some examples, the controller is configured to determine the configuration and position of the first robot arm relative to the reference frame of the image capturing device by using information of the construction and geometry of the first robot arm and the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device.
- In some examples, the controller is configured to determine the configuration and position of the second robot arm relative to a reference frame of the manipulatable device by using a transform to translate points in the reference frame of the image capturing device to the reference frame of the manipulatable device. The transform has been determined by using the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device. The reference frame of the manipulatable device is the common reference frame. In some examples, the system further includes a viewer adapted to display images derived from captured images of a work space of the working end of the manipulatable device and an input unit configured to receive information of a user specified region of interest within the images being displayed on the viewer. The controller is configured to command and control movement of the second robot arm so that the image capturing device captures images of the user specified region of interest in the plurality of images of the working end of the manipulatable device.
- In some examples, the input unit includes a gaze tracking unit configured to track a gaze of a user on a display screen of the viewer. The gaze tracking unit includes an indicator operable by the user to indicate the region of interest by using a current gaze point of the user on the display screen. In some examples, the input unit includes a telestrator unit configured to receive an image being displayed on the viewer and display the received image on a display screen of the telestrator. The telestrator unit includes a marker unit operable by a user to indicate the region of interest on the image being displayed on the display screen of the telestrator unit.
- In some examples, the system further includes a camera and the captured images of the work space of the working end of the manipulatable device are captured by the camera. In some examples, the system further includes an ultrasound probe and the captured images of the work space of the working end of the manipulatable device are captured by the ultrasound probe. In some examples, the controller is associated with the first robot arm to control movement of the first robot arm while avoiding a collision with the second robot arm and a reference frame of the manipulatable device is the common reference frame. In some examples, the controller is associated with the second robot arm to control movement of the second robot arm while avoiding a collision with the first robot arm, and the reference frame of the image capturing device is the common reference frame.
- In some examples, the first robot arm includes a first plurality of joints and a first plurality of links that are coupled together to provide redundant degrees of freedom movement of the first robot arm so that for each controllable position and orientation of the working end of the manipulatable device there are a plurality of possible configurations for the first robot arm. The controller is configured to control movement of the first robot arm by commanding the first robot arm to be configured in one of the plurality of possible configurations for the first robot arm according to a desired position and orientation of the working end of the manipulatable device and according to a secondary constraint for avoiding a collision between the first and second robot arms.
- In some examples, the second robot arm includes a second plurality of joints and a second plurality of links that are coupled together to provide redundant degrees of freedom movement of the second robot arm so that for each controllable position and orientation of the image capturing device there are a plurality of possible configurations for the second robot arm. The controller is configured to control movement of the second robot arm by commanding the second robot arm to be configured in one of the plurality of possible configurations for the second robot arm according to a desired position and orientation of the image capturing device and according to a secondary constraint for avoiding a collision between the first and second robot arms.
- In some examples, the first robot arm includes a first plurality of joints and a first plurality of links that are coupled together to provide redundant degrees of freedom movement of the first robot arm so that for each controllable position and orientation of the working end of the manipulatable device there are a plurality of possible configurations for the first robot arm. The second robot arm includes a second plurality of joints and a second plurality of links that are coupled together to provide redundant degrees of freedom movement of the second robot arm so that for each controllable position and orientation of the image capturing device there are a plurality of possible configurations for the second robot arm. The controller is configured to control movement of one of the first and second robot arms to be configured in one of its plurality of possible configurations to avoid a collision with the other one of the first and second robot arms.
- In some examples, the controller is configured to determine which one of the first and second robot arms is to be configured in one of its plurality of possible configurations to avoid a collision with the other one of the first and second robot arms by processing differences between current configurations of the first and second robot arms and their respective pluralities of possible configurations to minimize a cost function. In some examples, the controller is configured to determine which one of the first and second robot arms is to be configured in one of its plurality of possible configurations to avoid a collision with the other one of the first and second robot arms by processing required joint movements of the first and second robotic systems to move from their current configurations to others of their respective pluralities of possible configurations to minimize a cost function.
- Consistent with some embodiments is a method for avoiding collisions between first and second robot arms. The first robot arm holds a manipulatable device having a working end. The second robot arm holds an image capturing device for capturing a plurality of images of the working end of the manipulatable device from which a three-dimensional computer model of the working end of the manipulatable device is generatable. The method includes determining a position and an orientation of the working end of the manipulatable device relative to a reference frame of the image capturing device by using at least one image of the plurality of images of the working end of the manipulatable device, wherein the reference frame of the image capturing device corresponds to a perspective of the captured images; determining a configuration and position of one of the first and second robot arms relative to a common reference frame by using the determined position and orientation of the working end of the manipulatable device relative to the reference frame of the image capturing device; determining a configuration and position of the other of the first and second robot arms relative to the common reference frame by using joint position information received for the other of the first and second robot arms; determining an imminent collision between the first and second robot arms using the determined configurations and positions of the first and second robot arms relative to the common reference frame; and commanding an action to be taken by one of the first and second robot arms to avoid the imminent collision.
- Consistent with some embodiments, a robotic system includes a first robot arm that is holding a manipulatable device having a working end and a processor programmed to register the first robot arm to a second robot arm that is holding an image capturing device. The processor performs the registration by performing a low accuracy registration of the first and second robot arms at initial positions of the first and second robot arms using external tracking data and kinematics data of at least one of the first and second robot arms; performing a mid accuracy registration of the first and second robot arms relative to a first setup position using at least one image captured by the image capturing device of at least a part of the first robot arm and kinematics data of at least one of the first and second robot arms, wherein the first setup position includes a first safety margin indicative of the low accuracy registration; and performing a high accuracy registration of the first and second robot arms relative to a second setup position using at least one image captured by the image capturing device of the working end of the manipulatable device and kinematics data of at least one of the first and second robot arms, wherein the second setup position includes a second safety margin indicative of the mid accuracy registration.
- Consistent with some embodiments is a method implemented by a processor for registering first and second robot arms. The first robot arm holds a manipulatable device having a working end. The second robot arm holds an image capturing device. The method includes performing a low accuracy registration of the first and second robot arms at initial positions of the first and second robot arms using external tracking data and kinematics data of at least one of the first and second robot arms, performing a mid accuracy registration of the first and second robot arms relative to a first setup position using at least one image captured by the image capturing device of at least a part of the first robot arm and kinematics data of at least one of the first and second robot arms, wherein the first setup position includes a first safety margin indicative of the low accuracy registration, and performing a high accuracy registration of the first and second robot arms relative to a second setup position using at least one image captured by the image capturing device of the working end of the manipulatable device and kinematics data of at least one of the first and second robot arms, wherein the second setup position includes a second safety margin indicative of the mid accuracy registration.
- Additional objects, features and advantages of the various aspects of the present invention will become apparent from the following description of its preferred embodiment, whose description should be taken in conjunction with the accompanying drawings.
-
FIG. 1 illustrates a block diagram of a robotic system utilizing aspects of the present invention. -
FIG. 2 illustrates a flow diagram of a method, implemented in a robotic system utilizing aspects of the present invention, for performing multi-accuracy registration of an image capturing device robot arm and a manipulatable device robot arm as the image capturing device robot arm moves relative to the manipulatable device robot arm. -
FIG. 3 illustrates a flow diagram of a method, implemented in a robotic system utilizing aspects of the present invention, for automatically avoiding a collision between independently operated robot arms from the perspective of an image capturing device reference frame. -
FIG. 4 illustrates a flow diagram of the method, implemented in a robotic system utilizing aspects of the present invention, for automatically avoiding a collision between independently operated robot arms from the perspective of a manipulatable device reference frame. -
FIG. 5 illustrates a flow diagram of a method, implemented in a robotic system utilizing aspects of the present invention, for controlling an image capturing device robot arm so that its image capturing device captures images of a user selected region of interest while avoiding a collision with another robot arm. -
FIG. 6 illustrates a flow diagram of a method, implemented in a robotic system utilizing aspects of the present invention, for controlling movement of an image capturing device robot arm according to a plan of motion while avoiding a collision with another robot arm. -
FIG. 7 illustrates a block diagram of a portion of a robotic system utilizing aspects of the present invention, which is used for collision avoidance of two robot arms operating in a coupled-control mode. -
FIG. 8 illustrates a perspective view of an operating room employing a multiple aperture medical robotic system in which aspects of the present invention are usable. -
FIG. 9 illustrates a front view of a Patient-Side Cart for a multiple aperture medical robotic system in which aspects of the present invention are usable. -
FIG. 10 illustrates a perspective view of an instrument usable in a multiple aperture medical robotic system in which aspects of the present invention are usable. -
FIG. 11 illustrates a front view of a console of a robotic system utilizing aspects of the present invention. -
FIG. 12 illustrates a perspective view of an image capturing system usable in a robotic system utilizing aspects of the present invention. -
FIG. 13 illustrates a perspective view of a Patient-Side Cart for a single aperture medical robotic system in which aspects of the present invention are usable. -
FIG. 14 illustrates a perspective view of a distal end of an entry guide with articulated instruments extending out of it as used in a single aperture medical robotic system in which aspects of the present invention are usable. -
FIG. 15 illustrates a cross-sectional view of an entry guide as used in a single aperture medical robotic system in which aspects of the present invention are usable. -
FIG. 1 illustrates, as an example, a block diagram of various components of a robotic or computer-assistedsystem 1000 in whichmethods FIG. 2 describes a multi-step approach to registration of robot or movable arms.FIGS. 3, 4 are provided to describe themethods FIGS. 5-7 are provided to describe themethod 4000 which is generally applicable to controlling movement of an image capturing device robot or movable arm though a controller which is controlling a manipulatable device robot or movable arm. Examples of therobotic system 1000 in which themethods FIGS. 8-15 .FIGS. 8-12 are provided to describe a robotic or computer-assistedsystem 3000 using a Patient-Side Cart 3010 with multiple robot or movable arms which is suitable for performing procedures with multiple apertures for entry to a work site.FIGS. 13-15 are provided to describe an alternative Patient-Side Cart 4010 for therobotic system 3000, in which the Patient-Side Cart 4010 has a single robot or movable arm which is suitable for performing procedures with a single aperture for entry to a work site. - Before describing details of the
methods FIGS. 1-7 , the robotic or computer-assistedsystem 3000 will first be described to provide context and additional details on implementations of the robotic or computer-assistedsystem 1000. Although a medical robotic or computer-assisted surgical system is described herein as an example of the robotic or computer-assistedsystem 1000, it is to be appreciated that the various aspects of the invention as claimed herein are not to be limited to such types of robotic or computer-assisted systems. - Referring to
FIG. 8 , a perspective view of an operating room is illustrated in which a medicalrobotic system 3000 is provided for a Surgeon to perform a medical procedure on a Patient. The medical robotic system in this case is a Minimally Invasive Robotic Surgical (MIRS) system including a Patient-Side Cart 3010, anImage Capturing System 3020, and aConsole 3030. - The Patient-
Side Cart 3010, as described in detail in reference toFIG. 9 , has a plurality of robot arms for holding and manipulating a plurality of devices such as instruments and at least one endoscope. When using the Patient-Side Cart 3010, each of the devices being held by the plurality of robot or movable arms is introduced through its own entry aperture into a Patient. - The
Console 3030, as described in detail in reference toFIG. 11 , includes input devices for commanding movement of associated ones of the plurality of robot arms of the Patient-Side Cart 3010 and the operation of their respectively held devices. TheConsole 3030 and the Patient-Side Cart 3010 communicate through acable 3031. - The
Image Capturing System 3020, as described in detail in reference toFIG. 12 , captures a plurality of images, such as a series of two-dimensional image projections, of one or more objects within a specified region of interest at a work site in the Patient. The plurality of images may then be used to generate a three-dimensional computer model of the one or more objects without requiring prior knowledge of the three-dimensional shapes of the one or more objects, in a conventional manner such as used in computed tomography. - A
control unit 3021 may be provided beneath or to the side of an articulated operating table 3040, so that an Assistant may manually control movement of an image capturing device of theImage Capturing System 3020 using a joy stick or other control input provided on thecontrol unit 3021. Thecontrol unit 3021 and theImage Capturing System 3020 communicate throughcable 3022 or using wireless technology. Alternatively, thecontrol unit 3021 may be provided at another location and communicate with theImage Capturing System 3020 as a wired or wireless mobile entity. Thecontrol unit 3021 may include at least one processing unit and memory. In some examples, the processing unit may control operation and/or execution of hardware and/or software incontrol unit 3021. The processing unit may include one or more central processing units (CPUs), multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), custom processors/application specific integrated circuits (ASICs), and/or the like. The memory may be used to store one or more software and/or firmware applications as well as various data structures to be used bycontrol unit 3021. The memory may also include one or more types of machine readable media. Some common forms of machine readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read. - Alternatively, movement of the image capturing device of the
Image Capturing System 3020 may be controlled by the Surgeon operating an associated one of the input devices of theConsole 3030 according to aspects of the present invention. Alternatively, movement of the image capturing device of theImage Capturing System 3020 may be controlled automatically by a processor of theConsole 3030 so that the image capturing device automatically captures a plurality of images of a user specified region of interest in the work site while avoiding a collision with the robot arms of the Patient-Side Cart 3010 according to aspects of the present invention. TheConsole 3030 and theImage Capturing System 3020 communicate throughcable 3050. -
FIG. 9 illustrates, as an example, a front view of the Patient-Side Cart 3010. In a typical application,robot arms hold instruments robot arm 38 holdsstereo endoscope 37. Thefourth robot arm 32 is available so that anotherinstrument 31 may be introduced at the work site along with theinstruments endoscope 37. Alternatively, thefourth robot arm 32 may be used for introducing a second endoscope or another image capturing device, such as an ultrasound transducer, to the work site. - Each of the robot or movable arms is conventionally formed of links which are coupled together and manipulated through actuatable joints. Each of the robot arms includes a setup arm and a device manipulator. The setup arm positions its held device so that a pivot point occurs at its entry aperture into the Patient. The device manipulator may then manipulate its held device so that it may be pivoted about the pivot point, inserted into and retracted out of the entry aperture, and rotated about its shaft axis. In this example, the robot arms are mounted on a
movable base 3015 of the Patient-Side Cart 3010. Alternatively, the robot arms may be attached to sliders on a side of the operating table, to sliders on a wall or to sliders on the ceiling of the operating room. -
FIG. 10 illustrates, as an example, aninstrument 100 that may be used for eitherinstrument instrument 100 comprises aninterface housing 108, ashaft 104, a workingend 102, and awrist mechanism 106 which includes one or more wrist joints. Theinterface housing 108 is removably attached to a robot arm so as to be mechanically coupled to actuators (such as motors) in the instrument manipulator of the attached robot arm. Cables or rods, that are coupled to the actuators of the instrument manipulator and extend through theshaft 104 from theinterface housing 108 to the one or more wrist joints of thewrist mechanism 106 and to the jaws of the instrument'send effector 102, actuate the wrist joints and jaws in a conventional manner. The instrument manipulator may also manipulate the instrument in pitch and yaw angular rotations about its pivot point at the entry aperture, manipulate the instrument in a roll angular rotation about the instrument's shaft axis, and insert and retract the instrument along a rail on the robot arm as commanded by theprocessor 43. -
FIG. 11 illustrates, as an example, a front view of theConsole 3030. TheConsole 3030 has left andright input devices Side Cart 3010 in preferably six degrees-of-freedom (“DOF”).Foot pedals 44 with toe and heel controls are provided on theConsole 3030 so the user may control movement and/or actuation of devices associated with the foot pedals. Aprocessor 43 is provided in the Console for control and other purposes. Astereo vision display 45 is also provided in the Console so that the user may view the work site in stereo vision from images captured by the stereoscopic camera of theendoscope 37. Left and right eyepieces, 46 and 47, are provided in thestereo vision display 45 so that the user may view left and right two-dimensional (“2D”) display screens inside thedisplay 45 respectively with the user's left and right eyes. - The
processor 43 performs various functions in the medical robotic system. One important function that it performs is to translate and transfer the mechanical motion ofinput devices tool instruments endoscope 37, which are associated with theinput devices processor 43 is to implement the methods, cross-coupling control logic, and controllers described herein. - Although described as a processor, it is to be appreciated that the
processor 43 may be implemented by any combination of hardware, software and firmware. Also, its functions as described herein may be performed by one unit or divided up among a number of subunits, each of which may be implemented in turn by any combination of hardware, software and firmware. Further, although being shown as part of or being physically adjacent to the Console, theprocessor 43 may also be distributed as subunits throughout thesystem 3000. - U.S. Pat. No. 6,659,939 B2 entitled “Cooperative Minimally Invasive Telesurgical System,” which is incorporated herein by reference, provides additional details on a multiple aperture medical robotic system such as described herein.
-
FIG. 12 illustrates, as an example, theImage Capturing System 3020 which has a robot ormovable arm 1101 which is mounted to abase 1106. Therobot arm 1101 comprises acarousel 1107,first link 1102,second link 1103,wrist 1104, and C-arm 1105. Thecarousel 1107 is rotatable (as indicted by arrow “a”) relative to thebase 1106 using a carousel joint 1108. Thefirst link 1102 is rotatable (as indicated by arrow “b”) relative to thecarousel 1107 using ashoulder joint 1109. Thesecond link 1103 is rotatable (as indicated by arrow “c”) relative to thefirst link 1102 using an elbow joint 1110. Thewrist 1104 is rotatable (as indicated by arrow “d”) relative to thesecond link 1103 using a roll joint 1111. The C-arm 1105 is rotatable (as indicated by arrow “e”) relative to thewrist 1104 using a pitch joint 1112 and rotatable (as indicated by arrow “j”) relative to thewrist 1104 using ayaw joint 1121. - The C-
arm 1105 comprises afirst limb 1113,second limb 1114, and acentral portion 1115. The first andsecond limbs central element 1115 using anextender joint 1120. Thefirst limb 1113 has anX-ray detector 1116 disposed at its distal end and thesecond limb 1114 has anX-ray source 1117 disposed at its distal end. TheX-ray detector 1116 is rotatable (as indicated by arrow “g”) relative to the distal end of thefirst limb 1113. TheX-ray source 1117 is rotatable (as indicated by arrow “h”) relative to the distal end of thesecond limb 1114. In this arrangement, the X-ray source and X-ray detector are disposed on opposing ends of the C-arm 1105 so as to form an image capturing device. By actuating the carousel joint 1108,shoulder joint 1109, elbow joint 1110, roll joint 1111, pitch joint 1112, yaw joint 1121, and extender joint 1120, the C-arm 1105 may be positioned relative to the Patient so that the C-arm 1105 may be moved so that the image capturing device (comprisingX-ray source 1117 and X-ray detector 1116) may capture a series of two-dimensional projections of one or more objects within a specified region of interest at a work site in the Patient. The series of two-dimensional projections may then be used to generate a three-dimensional computer model of the one or more objects in a conventional manner for cone beam computed tomography. - Previously incorporated by reference
U.S. Published Application 2009/0234444 A1 provides additional details on such an Image Capturing System and its use during the performance of a medical procedure on a patient. -
FIG. 13 illustrates, as an example, an alternative Patient-Side Cart 4010 which is usable in therobotic system 3000 to introduce a plurality of articulated instruments to a work site through a single entry aperture in the Patient by anentry guide 200. The aperture may be a minimally invasive incision or a natural body orifice. Theentry guide 200 is a cylindrical structure which is held and manipulated by arobot arm 4011, which is mounted onbase 4015 and includes asetup arm 4012 and anentry guide manipulator 4013. Thesetup arm 4012 comprises a plurality of links and joints which are used to position theentry guide 200 at the aperture. As indicated in the figure, thesetup arm 4012 includes a prismatic joint for adjusting the height of the setup arm 4012 (as indicated by arrow “A”) and a plurality of rotary joints for adjusting the horizontal position of the setup arm 4012 (as indicated by arrows “B” and “C”). Theentry guide manipulator 4013 is used to robotically pivot the entry guide 200 (and the articulated instruments disposed within it at the time) in yaw, pitch and roll angular rotations about the pivot point as indicated by arrows D, E and F, respectively. Articulated instrument manipulators (not shown) reside inhousing 4014. - As shown in
FIG. 14 , theentry guide 200 has articulated instruments such as articulatedsurgical instruments 231, 241 and an articulated stereo camera instrument 211 (or other image capturing device instrument) extending out of its distal end. Thecamera instrument 211 has a pair of stereoimage capturing elements surgical instruments 231, 241 have workingends instruments 231, 241 are shown, theentry guide 200 may guide additional instruments as required for performing a medical procedure at a work site in the Patient. For example, as shown in a cross-sectional view of theentry guide 200 inFIG. 15 , apassage 351 is available for extending another articulated surgical tool through theentry guide 200 and out through its distal end.Passages surgical tool instruments 231, 241, andpassage 321 is used for the articulatedcamera instrument 211. - When the Patient-
Side Cart 4010 is used in therobotic system 3000, thestereo vision display 45 of theConsole 3030 displays stereo images derived from the stereo images captured by the articulatedcamera instrument 211. Also, theprocessor 43 of theConsole 3030 translates and transfers the mechanical motion of theinput devices entry guide 200, articulatedsurgical instruments 231, 241, and articulatedcamera instrument 211, which are associated at the time with theinput devices - Each of the articulated instruments comprises a plurality of actuatable joints and a plurality of links coupled to the joints. As an example, as shown in
FIG. 14 , the second articulated instrument 241 comprises first, second, andthird links second joints wrist assembly 327. The first joint 323 couples the first andsecond links third links second link 324 may pivot about the first joint 323 in pitch and yaw while the first andthird links instrument 231 and the camera articulatedinstrument 211, may be similarly constructed and operated. - U.S. Pat. No. 7,725,214 entitled “Minimally Invasive Surgical System,” which is incorporated herein by reference, provides additional details on a single aperture medical robotic system such as described herein.
- Now referring back to
FIG. 1 , a block diagram of components of therobotic system 1000 is illustrated to describe various aspects of the present invention. In this example, therobotic system 1000 has a firstmanipulatable device 1001, a firstimage capturing device 1002, and acamera 1003. The firstmanipulatable device 1001 may be an instrument such as the one of theinstruments Side Cart 3010. Alternatively, the firstmanipulatable device 1001 may be an entry guide through which articulated instruments extend such as theentry guide 200 held and manipulated by the robot arm of the Patient-Side Cart 4010. A secondmanipulatable device 1004 is also shown which may be another device such as the firstmanipulatable device 1001. A secondimage capturing device 1005 is also shown which may provide a different imaging modality than that of the firstimage capturing device 1002. Although two manipulatable devices and two image capturing devices are shown for illustrative purposes, it is to be appreciated that in practice, more or less of each of such devices may be included in therobotic system 1000 to perform a task or procedure on an object at a work site. Additional cameras may also be included. - A
device controller 1021 controls movement of therobot arm 1011 to position and orient a working end of the firstmanipulatable device 1001 in response to commands from aninput unit 1031. Theinput unit 1031 may be a user operated input device, such as one of theinput devices foot pedal 44 of theconsole 3030. Alternatively, theinput unit 1031 may be a processor, such as theprocessor 43 of theconsole 3030, executing stored program instructions. Alternatively, theinput unit 1031 may be coupled control logic which communicates throughbus 1040 with one or more of thecontrollers input units image capturing device 1002, thecamera 1003, the secondmanipulatable device 1004, and the secondimage capturing device 1005. - Likewise, a
device controller 1024 controls movement of therobot arm 1014 to position and orient a working end of the secondmanipulatable device 1004 in response to commands from aninput unit 1034. Theinput unit 1034 may be a user operated input device, such as one of theinput devices foot pedal 44 of theconsole 3030. Alternatively, theinput unit 1034 may be a processor, such as theprocessor 43 of theconsole 3030, executing stored program instructions. Alternatively, theinput unit 1034 may be coupled control logic which communicates throughbus 1040 with one or more of thecontrollers input units manipulatable device 1001, the firstimage capturing device 1002, thecamera 1003, and the secondimage capturing device 1005. - The first
image capturing device 1002 is manipulatable by its robot arm 1012 (or other motorized mechanism) to capture a plurality of two-dimensional image slices or projections of an object at the work site from which a three-dimensional model of the object may be computer generated without prior knowledge of the shape of the object using an imaging modality such as ultrasound, X-ray fluoroscopy, Computed Tomography (CT), and Magnetic Resonance Imaging (MRI). Acontroller 1022 controls movement of therobot arm 1012 to position and orient the firstimage capturing device 1002 in response to commands from aninput unit 1032. Theinput unit 1032 may be a user operated input device, such as one of theinput devices console 3030. Alternatively, theinput unit 1032 may be a processor, such as theprocessor 43 of theconsole 3030, executing stored program instructions. Alternatively, theinput unit 1032 may be coupled control logic which communicates throughbus 1040 with one or more of thecontrollers input units manipulatable device 1001, thecamera 1003, the secondmanipulatable device 1004, and the secondimage capturing device 1005. The firstimage capturing device 1002 and itsrobot arm 1012 may be combined to form an image capturing system such as theImage Capturing System 3020. - The second
image capturing device 1005 may be similarly constructed and operated as the firstimage capturing device 1002 to capture a plurality of two-dimensional image projections of an object at the work site from which a three-dimensional model of the object may be computer generated without prior knowledge of the shape of the object using an imaging modality such as ultrasound, X-ray fluoroscopy, Computed Tomography (CT), and Magnetic Resonance Imaging (MRI). Typically, different imaging modalities are provided by the first and secondimage capturing devices image capturing device 1002, acontroller 1025 controls movement of therobot arm 1015 to position and orient the secondimage capturing device 1005 in response to commands from aninput unit 1035. Theinput unit 1035 may be a user operated input device, such as one of theinput devices console 3030. Alternatively, theinput unit 1035 may be a processor, such as theprocessor 43 of theconsole 3030, executing stored program instructions. Alternatively, theinput unit 1035 may be coupled control logic which communicates throughbus 1040 with one or more of thecontrollers input units manipulatable device 1001, the firstimage capturing device 1002, thecamera 1003, and the secondmanipulatable device 1004. The secondimage capturing device 1005 and itsrobot arm 1015 may be combined to form an image capturing system such as theImage Capturing System 3020. - Alternatively, the second
image capturing device 1005 may be constructed differently than that of the firstimage capturing device 1002. For example, the secondimage capturing device 1005 may be held, positioned, and oriented by one of the first and secondmanipulatable devices - The
camera 1003 may be held and manipulated by arobot arm 1013 to capture stereoscopic images of work site, such as theendoscope 37 ofFIG. 9 . As an alternative example, thecamera 1003 may be the articulatedcamera instrument 211 ofFIG. 14 . Acontroller 1023 controls movement of the robot arm 1013 (or joints of the articulated camera instrument 211) to position and orient an image capturing element of thecamera 1003 in response to commands from aninput unit 1033. Theinput unit 1033 may be a user operated input device, such as one of theinput devices console 3030. Alternatively, theinput unit 1033 may be a processor, such as theprocessor 43 of theconsole 3030, executing stored program instructions. Alternatively, theinput unit 1033 may be coupled control logic which communicates throughbus 1040 with one or more of thecontrollers input units manipulatable device 1001, theimage capturing device 1002, the secondmanipulatable device 1004, and the secondimage capturing device 1005. - Alternatively, the
camera 1003 may be held, positioned, and oriented by one of the first and secondmanipulatable devices - Each of the
robot arms robot arm respective controller bus 1040 for collision avoidance purposes, since this joint information indicates configuration information for therobot arms - Each of the control units 1021-1025 may include at least one processing unit and memory. In some examples, the processing unit may control operation and/or execution of hardware and/or software in the respective control unit 1021-1025. Each of the processing units may include one or more central processing units (CPUs), multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), custom processors/application specific integrated circuits (ASICs), and/or the like. The memory may be used to store one or more software and/or firmware applications as well as various data structures to be used by the respective control unit 1021-1025. The memory may also include one or more types of machine readable media. Some common forms of machine readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any other medium from which a processor or computer is adapted to read.
- The
robotic system 1000 also includes a stereo viewer ordisplay 1051 for displaying stereo images which have been generated by animage processor 1050 from images captured by thestereo camera 1003 and/or three-dimensional computer models of one or more objects which have been generated by theimage processor 1050 from the plurality of two-dimensional image slices of the one or more objects captured by the firstimage capturing device 1002 and/or secondimage capturing device 1005. Before displaying images derived from both thecamera 1003 and one or both of the first and secondimage capturing devices display 1051, theimage processor 1050 registers the images so that the three-dimensional computer models of the one or more objects are properly superimposed on and positioned relative to images of the one or more objects in the stereo images derived from images captured by thestereo camera 1003. -
FIGS. 3, 4 respectively illustrate, as examples, flow diagrams ofmethods robotic system 1000 for avoiding a collision between independently operated image capturing device and manipulatable device robot arms. Although shown for explanatory purposes as two different methods, it is to be appreciated that themethods method 2000 is from the perspective of an image capturing device reference frame and may be performed by an image capturing device controller and themethod 2500 is from the perspective of a manipulatable device reference frame and may be performed by a manipulatable device controller. - Although only one manipulatable device is mentioned in describing the methods herein, it is to be appreciated that when the working end of more than one device is viewable within the field of view of the image capturing device, each of those devices are to be processed as the described manipulatable device according to the methods so that collisions may be avoided between their robot arms and the robot arm of the image capturing device. For example, when the
robotic system 1000 is the medicalrobotic system 3000 ofFIG. 8 and it includes the Patient-Side Cart 3010 ofFIG. 9 , the relevant blocks of bothmethods manipulatable devices image capturing device 3020. On the other hand, when therobotic system 1000 is the medicalrobotic system 3000 ofFIG. 8 and it includes the Patient-Side Cart 4010 ofFIG. 13 , the relevant blocks of bothmethods entry guide manipulator 4013, but the working ends of articulated instruments extending out of theentry guide 200 are treated as the working end of theentry guide 200. - To avoid confusion and unintended consequences, only one of the
methods - As an example of user specification, when the
robotic system 1000 is the medicalrobotic system 3000 ofFIG. 8 and it includes the Patient-Side Cart 3010 ofFIG. 9 , the Surgeon may interact with a menu being displayed on thestereo viewer 45 to select either thecontrol unit 3021 to perform themethod 2000 or theprocessor 43 to perform themethod 2500. As previously explained, thecontrol unit 3021 controls movement of therobot arm 1101 which holds and manipulates theimage capturing device Image Capturing System 3020. Theprocessor 43, on the other hand, implements a controller which controls movement of therobot arm 34 that holds and manipulates theinstrument 33, which may be the manipulatable device for the purpose of this example. - When the
robotic system 1000 is the medicalrobotic system 3000 ofFIG. 8 and it includes the Patient-Side Cart 4010 ofFIG. 13 , then the manipulatable device referred to inmethods entry guide 200 rather than one of the articulatedinstruments entry guide manipulator 4013 that is at risk of colliding with or being struck by therobot arm 1101 of theImage Capturing System 3020. The articulatedinstruments housing area 4014 of theentry guide manipulator 4013 as shown inFIG. 13 . It is notable that collision avoidance between therobot arm 1101 of theImage Capturing System 3020 and theentry guide manipulator 4013 may be simpler than collision avoidance between therobot arm 1101 of theImage Capturing System 3020 and therobot arms Side Cart 3010. This is because not only are theremore robot arms Side Cart 3010 for therobot arm 1101 of theImage Capturing System 3020 to collide with, but also because therobot arms Side Cart 3010 may move more often than the entryguide robot manipulator 4013 of the Patient-Side Cart 4010. -
FIG. 2 illustrates, as an example, a flow diagram of amethod 9000, which is implemented in therobotic system 1000, for registering an image capturing device robot arm and a manipulatable device robot arm. Using the method, higher accuracy registrations are performed as the image capturing device robot arm moves closer to the manipulatable device robot arm. This multi-step approach provides high accuracy registration in a safe manner. - In
block 9001, the method performs a low accuracy registration (e.g., within an accuracy range of tens of centimeters) of the image capturing device robot arm and the manipulatable device robot arm relative to a common reference frame at their initial positions using an external tracker system and kinematic data for the robot arms. As an example, the external tracker system may be of the transmitter/receiver type which conventionally employs transmitters, which are strategically disposed on known locations of the robot arms of the image capturing device and the manipulatable device, and one or multiple receivers, which are disposed within transmission distance of the transmitters. As another example, the external tracker system may be of the optical type which conventionally employs optically discernible targets, which are strategically disposed on known locations of the robot arms of the image capturing device and the manipulatable device, and one or multiple optical detectors, which are disposed so as to have an unobstructed view of the targets. The kinematic data may be provided, for example, by encoders disposed to sense joint positions of the image capturing device robot arm and the manipulatable device robot arm. The joint positions may then be combined in a conventional manner using knowledge of the construction, shapes, and sizes of the robot arms to estimate the configurations of the robot arms. - In
block 9002, the image capturing device is moved towards the manipulatable device robot arm to assume a low-risk setup position that is close enough to allow the image capturing device being held by the image capturing device robot arm to capture images of at least a portion of the manipulatable device robot arm while being sufficiently far enough away from the manipulatable device robot arm to ensure that the image capturing device robot arm does not collide with the manipulatable device robot arm. In determining the low-risk setup position, the method also takes into account the current low accuracy level of the registration through a first safety margin that ensures no collision will occur between the image capturing device robot arm and the manipulatable device robot arm. For example, the first safety margin may maintain a distance of at least ten centimeters between the image capturing device robot arm and the manipulatable device robot arm to account for the current low accuracy in determining the relative positions of those two robot arms. The movement of the image capturing device robot arm may be performed, for example, by an operator commanding such movement through a robot arm controller with assistance from the robot arm controller to inhibit unsafe movement. For example, the robot arm controller may provide force feedback to the operator through a haptic device, such as a joystick, to provide resistance against the unsafe movement. Alternatively, it may be performed automatically by the robot arm controller in either a direct mode or cross-coupled mode. - In
block 9003, the method performs a mid accuracy registration (e.g., within an accuracy range of centimeters) of the image capturing device robot arm and the manipulatable device robot arm relative to the common reference frame at their current positions using one or more captured images of at least a part of the manipulatable device robot arm and kinematics data for the robot arms. - In block 9004, the method then waits for an operator's command to initiate movement of the image capturing device robot arm to capture images of a user specified region of interest.
- In
block 9005, the image capturing device is moved towards the manipulatable device robot arm to assume an image capturing setup position that is close enough to allow the image capturing device being held by the image capturing device robot arm to capture images of a user specified region of interest while being sufficiently far enough away from the manipulatable device robot arm to ensure that the image capturing device robot arm does not collide with the manipulatable device robot arm. In determining the image capturing setup position, the method takes into account the current mid accuracy level of the registration through a second safety margin that ensures no collision will occur between the image capturing device robot arm and the manipulatable device robot arm. For example, the second safety margin may maintain a distance of at least one centimeter between the image capturing device robot arm and the manipulatable device robot arm to account for the current mid accuracy in determining the relative positions of those two robot arms. The movement of the image capturing device robot arm may be performed, for example, by an operator commanding such movement through a robot arm controller with assistance from the robot arm controller to inhibit unsafe movement. For example, the robot arm controller may provide force feedback to the operator through a haptic device, such as a joystick, to provide resistance against the unsafe movement. Alternatively, the movement may be performed automatically by the image capturing device controller directly or through a coupled control mode with the manipulatable device controller. - In block 9006, the method commands the image capturing device robot arm to move relative to the region of interest and commands the image capturing device being held by the image capturing device robot arm to capture images of the region of interest during such movement while avoiding a collision of the image capturing device robot arm and the manipulatable device robot arm. In addition to capturing images of the region of interest, the image capturing device captures images of at least a working end of a manipulatable device being held by the manipulatable device robot arm. In this case, the working end of the manipulatable device is proximate to the region of interest so as to be within the field of view of the image capturing device. During the image capturing process, the method performs at least one high accuracy registration (e.g., within an accuracy range of millimeters) of the image capturing device robot arm and the manipulatable device robot arm relative to the common reference frame at their current positions using one or more of the captured images of the working end of the manipulatable device and kinematics data for the robot arms.
- Additional details describing aspects of block 9006 are described below in reference to
FIGS. 3, 4 and additional details describing aspects ofblocks 9005, 9006 are described below in reference toFIG. 5 . - Referring now to the
method 2000 ofFIG. 3 , inblock 2001, the method, which is preferably performed by the imagecapturing device controller 1022, receives information of the configuration of the robot arm that holds and manipulates the image capturing device. When the robot arm comprises a plurality of links coupled together by a plurality of joints, the robot arm configuration is determinable from sensed positions of the joints and known geometries of the links and other structure making up the robot arm. As previously described in reference toFIG. 1 , such robot arm configuration information may be provided from a plurality of sensors in the image capturing device robot arm, such as encoders which are coupled to actuators of the joints. Alternatively, it may be joint positions being commanded by the imagecapturing device controller 1022 in response to commands from theinput device 1032. - In
block 2002, the method determines whether or not it is time to perform a registration of the image capturing device to the manipulatable device. Such registration may be performed once at the start of the method or it may be performed periodically to correct any registration errors that may accrue over time. If the determination inblock 2002 is YES, then the method performs the registration by performingblocks block 2002 is NO, then registration is skipped by the method jumping to block 2005. - In
block 2003, the method receives an image, which has been captured by the image capturing device, of the working end of the manipulatable device. Generally, the received image will be a two-dimensional slice or projection of the working end of the manipulatable device and any other objects within an image capturing cone of the image capturing device. - In
block 2004, the method determines the position and orientation (i.e., pose) of the working end of the manipulatable device relative to a reference frame of the image capturing device, which corresponds to a position and orientation of the image capturing cone or field of view of the image capturing device. To do this, the method conventionally uses artificial and/or natural features of the working end that are discernible in the received image. In this case, artificial features include such things as markings or structure specifically placed on the working end to aid in determining their pose (i.e., position and orientation). Natural features include such things as the shape and known geometries of structure of the working end of the manipulatable device. The pose of the working end may be determinable even though the working end may be partially occluded. For example, the working end may be occluded by portions of a patient's anatomy, other portions of the manipulatable device, portions of the image capturing device, other medical instruments, and/or the like. The patient's anatomy may include soft tissue, bone, teeth, and/or the like. To aid in the determination, the sequence of images captured by the image capturing device and received as the method loops through blocks 2001-2009 may be used to refine the determination of the pose. Also, conventional tool tracking techniques may be used that aid in and serve to further refine the determination of the pose of the working end of the manipulatable device. For additional details on such pose determining techniques and artifacts, see U.S. Pat. No. 8,108,072 entitled “Methods and systems for robotic instrument tool tracking with adaptive fusion of kinematics information and image information,” which is incorporated herein by reference, and U.S. Publication No. 2010/0168763 A1 entitled “Configuration marker design and detection for instrument tracking,” which is incorporated herein by reference. - In
block 2005, the method receives information of the configuration of the robot arm that holds and manipulates the manipulatable device. Where the robot arm comprises a plurality of links coupled together by a plurality of joints, the robot arm configuration is determinable from sensed positions of the joints and known geometries of the links and other structure making up the robot arm. As previously described in reference toFIG. 1 , such robot arm configuration information may be provided from a plurality of sensors of the manipulatable device robot arm, such as encoders which are coupled to actuators of the joints. Alternatively, it may be joint positions being commanded by themanipulatable device controller 1021 in response to commands from theinput device 1031. Alternatively, it may be some joint positions being provided by tracking the working end of instruments in endoscopic images which may provide better accuracy compared to encoder measurements. - If the manipulatable device robot arm is a redundant degree-of-freedom (DOF) arm, then the information of the configuration of the robot arm preferably includes either the sensed joint positions or commanded joint positions of the manipulatable device robot arm. If, on the other hand, there is only one robot arm configuration that may correspond to the pose of the working end of its held manipulatable device, then the robot arm configuration may theoretically be determined from the determined pose of the working end of the manipulatable device, the known construction and geometries (e.g., size and shape) of the manipulatable device and its robot arm, and the position of the base of the manipulatable device robot arm. In this case, it may not be necessary to receive information of the sensed or commanded joint positions in
block 2005, only the base position of the manipulatable device robot arm is necessary. When the manipulatable device robot arm is mounted to a base that doesn't move, then the base position is fixed and only needs to be determined once and stored in a memory. When the manipulatable device robot arm is mounted to a movable base, then the base position may be determined, for example, by external sensors such as pressure sensors strategically disposed on the floor. As another example, the base position may be determined by a transmitter/receiver system in which one or more transmitters are disposed on the base with one or more receivers disposed at fixed locations. As another example, the base position (as well as the robot arm configuration) may be determined by an optical tracking system or by any other well known position sensing means. - In
block 2006, the method determines the configuration and position of the manipulatable device robot arm in the image capturing device frame of reference. Since the pose of the working end of the manipulatable device relative to the image capturing device reference frame has already been determined inblock 2004, the determination of the configuration and position of its robot arm in the image capturing device frame of reference is determined in this case by using the known construction and geometries of the manipulatable device and its robot arm along with the information of the manipulatable device robot arm configuration received inblock 2005. - In
block 2007, the method determines the configuration and position of the image capturing device robot arm in the image capturing device reference frame. Since the image capturing device reference frame is defined by the pose of the distal end of the image capturing device robot arm, it is a simple matter to determine the configuration and position of the image capturing device robot arm in the image capturing device reference frame using the known construction and geometries of the image capturing device and its robot arm along with the information of the image capturing device robot arm configuration received inblock 2001. - In
block 2008, the method determines whether an imminent collision is threatened between the image capturing device robot arm and the manipulatable device robot arm. Preferably, the determination is made by using the information received or determined in blocks 2001-2007 for the current process cycle and previous process cycles. By using time series information, not only can the trajectories of the two arms be predicted, but the rates at which they are moving may be estimated. With this information, a collision prediction may be made. When the predicted collision is within a specified time period, then the collision is considered to be imminent, requiring immediate action. The collision prediction may use a minimum distance between the image capturing device robot arm and the manipulatable device robot arm. In determining the minimum, the method preferably approximates links of the robot arms with geometric shapes that are dimensionally slightly larger than the actual links for safety purposes. Since the geometric shapes of the image capturing device robot arm and the geometric shapes of the manipulating device robot arm occupy known positions and orientations relative to each other at this point, it is a straightforward calculation to determine a minimum distance between the geometric shapes representing the image capturing device robot arm and the geometric shapes representing the manipulating device robot arm. - One or ordinary skill would also understand that the determination of
block 2008 may be used to detect additional undesirable relationships between the manipulating device robot arm and the image capturing device robot arm other than an imminent collision. In some embodiments,block 2008 may be used to detect when the manipulating device robot arm and the image capturing device robot arm are in too close a proximity to each other, even though a collision is not imminent. In some embodiments,block 2008 may be used to detect that the manipulating device robot arm is obstructing a region of interest for which the image capturing device is to capture images thereof. - If the
determination block 2008 is NO, then the method jumps back to block 2001 to perform blocks 2001-2008 for a next process cycle. On the other hand, if the determination is YES, then inblock 2009, the method commands the image capturing device robot arm to take an action to avoid a collision with the robot arm of the manipulatable device. The commanded action may be to temporarily halt movement of the image capturing device robot arm until the manipulatable device robot arm has moved to a collision safe position. Alternatively, rather than halting all movement of the image capturing device robot arm, the speed of its movement may be adjusted instead to avoid collision with the manipulatable device robot arm. Alternatively, if the image capturing device robot arm is a redundant DOF arm, then an alternative arm configuration may be commanded to avoid collision with the manipulatable device robot arm. Alternatively, the movement of the image capturing device robot arm may be halted and the collision avoidance task may be passed over to the manipulatable device controller to perform themethod 2500 ofFIG. 4 . - After performing
block 2009, the method then loops back to block 2001 to process information for a next process cycle. - Referring now to the
method 2500 ofFIG. 4 , inblock 2501, the method, which is preferably performed by themanipulatable device controller 1021, receives information of the configuration of the robot arm that holds and manipulates the manipulatable device. Where the robot arm comprises a plurality of links coupled together by a plurality of joints, the robot arm configuration is determinable from sensed positions of the joints and known geometries of the links and other structure making up the robot arm. As previously described in reference toFIG. 1 , such robot arm configuration information may be provided from a plurality of sensors in the image capturing device robot arm, such as encoders which are coupled to actuators of the joints. Alternatively, it may be joint positions being commanded by themanipulatable device controller 1021 in response to commands from theinput device 1031. Alternatively, it may be some joint positions being provided by tracking the working end of instruments in endoscopic images which may provide better accuracy compared to encoder measurements. - In
block 2502, the method determines whether or not it is time to perform a registration of the image capturing device to the manipulatable device. Such registration may be performed once at the start of the method or it may be performed periodically to correct any registration errors that may accrue over time. If the determination inblock 2502 is YES, then the method performs the registration by performingblocks block 2502 is NO, then registration is skipped by the method jumping to block 2505. - In
block 2503, the method receives an image, which has been captured by the image capturing device, of the working end of the manipulatable device. Generally, the received image will be a two-dimensional slice or projection of the working end of the manipulatable device and any other objects within an image capturing cone or field of view of the image capturing device. As an example, the image may be received by themanipulatable device controller 1021 over a bus 1040 (orcable 3050 ofFIG. 8 ) from the firstimage capturing device 1002 or a processor in an image capturing system that includes the firstimage capturing device 1002. - In
block 2504, the method determines the position and orientation (i.e., pose) of the working end of the manipulatable device relative to a reference frame of the image capturing device in the same manner as described in reference to block 2004 ofFIG. 3 . - In
block 2505, the method receives information of the configuration of the robot arm that holds and manipulates the image capturing device. Where the robot arm comprises a plurality of links coupled together by a plurality of joints, the robot arm configuration is determinable from sensed positions of the joints and known geometries of the links and other structure making up the robot arm. As previously described in reference toFIG. 1 , such robot arm configuration information may be provided from a plurality of sensors of the manipulatable device robot arm, such as encoders which are coupled to actuators of the joints. Alternatively, it may be joint positions being commanded by the imagecapturing device controller 1022 in response to commands from theinput device 1032. - In
block 2506, the method determines the configuration and position of the manipulatable device robot arm in the manipulatable device reference frame. Since the manipulatable device reference frame is defined by the pose of the distal end of the manipulatable device robot arm, it is a simple matter to determine the configuration and position of the manipulatable device robot arm in the manipulatable device reference frame using the information received inblock 2501 and the known construction and geometries of the manipulatable device and its robot arm. - In
block 2507, the method determines the configuration and position of the image capturing device robot arm in the manipulatable device frame of reference. One way to do this is to first determine the configuration and position of the image capturing device robot arm in the image capturing device reference frame, such as determined inblock 2007 ofFIG. 3 , then use the following transformation equation to translate points of the image capturing device robot arm from the image capturing device reference frame to manipulatable device reference frame: -
M P= M I T· I P (1) - where MP is a point in the manipulatable device reference frame “M”, M IT is the image capturing device reference frame “I” to manipulatable device reference frame “M” transform, and IP is a point in the image capturing device reference frame “I”.
- The method may determine the transform M IT by comparing points of the working end of the manipulatable device in the image reference frame, using information of its pose determined in
block 2504, with corresponding points of the working end of the manipulatable device in the manipulatable device reference frame, using information of its pose determined from the information of the manipulatable device robot arm configuration which was received inblock 2501 and prior known information of the size, shape, and construction of the manipulatable device. For additional details on such reference frame transformations, see previously incorporated by reference U.S. Patent Application Publication No. 2012/02900134, entitled “Estimation of a Position and Orientation of a Frame Used in Controlling Movement of a Tool.” - In
block 2508, the method determines whether an imminent collision is threatened between the image capturing device robot arm and the manipulatable device robot arm in a similar manner as described in reference to block 2008 ofFIG. 3 . One of ordinary skill would also understand that similar to the determination ofblock 2008, block 2508 may also be used to detect additional undesirable relationships between the manipulating device robot arm and the image capturing device robot arm other than an imminent collision. - If the
determination block 2508 is NO, then the method jumps back to block 2501 to perform blocks 2501-2509 for a next process cycle. On the other hand, if the determination is YES, then inblock 2509, the method may command either the manipulatable device robot arm or the image capturing device robot arm, through the imagecapturing device controller 1022, to take a collision avoidance action. The entity performing the collision avoidance action and the particular collision avoidance action taken may take different forms depending upon system factors. One factor is the nature of the tasks being performed at the time by the image capturing device and the manipulatable device. Another factor is the type or structure of the robot arms or their corresponding velocity and acceleration at the moment when an imminent collision is detected. - As an example of the nature of the tasks being performed at the time by the image capturing device and the manipulatable device, if the
robotic system 1000 is the medicalrobotic system 3000 and the manipulatable device is one of theinstruments device robot arm 1101 to avoid a collision. To avoid such collision, movement of the image capturingdevice robot arm 1101 may be temporarily halted until the manipulatable device robot arm has moved to a collision safe position. Alternatively, rather than halting all movement of the image capturingdevice robot arm 1101, the speed and/or direction of its movement may be adjusted instead to avoid collision with the manipulatable device robot arm. - As an example of the type or structure of the robot arms, if either or both of the image capturing device and manipulatable device robot arms has redundant degrees-of-freedom (DOF), then one of the robot arms with such redundant DOF may be configured in an alternative configuration without affecting the pose of its held device. In particular, if the manipulatable device robot arm has redundant DOF, then it would not be necessary to modify the trajectory of the image capturing device since the manipulatable device robot arm may be configured instead to an alternative configuration without significantly impacting the manipulation of its held manipulatable device. When both the image capturing device and manipulatable device robot arms have redundant DOF, then one of the first and second robot arms is selected for using an alternative configuration to avoid the imminent collision by processing differences between current configurations of the first and second robot arms and their respective pluralities of possible configurations to minimize a cost function. Alternatively, the selection may be made by processing required joint movements of the first and second robotic systems to move from their current configurations to others of their respective pluralities of possible configurations to minimize a cost function.
- Any of several possible cost functions may be minimized by the selection of an alternative configuration. One possible cost function may be based on an inverse square of the minimum distance between the first and second robot arms. A second possible cost function may be based on a weighted average of an inverse square of distances between links of the first and second robot arm, with each link being associated with a passive or active degree of freedom. A third possible cost function may be based on the second cost function, but may account for only distances which are below a threshold, such as a threshold within an order of magnitude of a desired safety margin. A fourth possible cost function may be based on a modified version of the second cost function where the distances are measured between virtual objects or buffer regions surrounding the links of the first and second robot arms. A fifth possible cost function may include the buffer regions of the fourth cost function and the distance threshold of the third cost function. A sixth possible cost function may be based on a constrained manipulability index for the first and second robot arms with the manipulability index estimating an ability of the first or second robot arm to move in arbitrary directions around the alternative configuration. A seventh possible cost function may be based on virtual potential fields applied to the links and/or joints of the first and second robot arms. As an example, each link and/or joint of the first and second robot arms may have a virtual electric charge assigned to it and the induced virtual repelling force may be integrated between the first and second robot arms to determine the seventh cost function.
- After performing
block 2509, the method then loops back to block 2501 to process information for a next process cycle. - During or after the performance of
methods robotic system 1000 is the medicalrobotic system 3000, then it is common practice to register each of the work site objects to a world reference frame. In this case, the work site objects include theinstruments endoscope 37, theimage capturing device Image Capturing System 3020, and the Patient's anatomy. During the performance of a procedure, however, registration errors may accumulate or otherwise occur in some manner. Thus, it may be beneficial to update their initial registration transforms with respect to the fixed reference frame using images of the working ends of theinstruments endoscope 37, which have been captured by theImage Capturing System 3020. - In the
methods FIGS. 2, 3 , the image capturing device and the manipulatable device are independently operated typically by different people. For example, when therobotic system 1000 is the medicalrobotic system 3000, an Assistant standing next to the Operating Table 3040 may operate thecontrol unit 3021 to control movement of theImage Capturing System 3020 and the Surgeon may operate theinput devices Console 3030 to control movement of themanipulatable devices Image Capturing System 3020 during the performance of a medical procedure. For example, when endoscopic images of an exterior view of an anatomical structure are being displayed on thestereo viewer 45 of theConsole 3030, the Surgeon may want to supplement those images with three-dimensional X-ray images of an interior view of the anatomical structure. However, since the Surgeon is occupied with the performance of the medical procedure, a method for automatically controlling theImage Capturing System 3020 to provide the Surgeon's desired view of a region of interest is useful. -
FIG. 5 illustrates, as an example, amethod 4000 implemented in therobotic system 1000 for automatically controlling movement of the image capturing device so as to capture images of a user specified region of interest while avoiding a collision between its robot arm and another robot arm. - In
block 4001, the method receives information of a user specified region of interest at the work site. When the user is viewing the work site on thedisplay 1051, the region of interest is specified relative to an image of the work site being displayed at the time on thedisplay 1051. The images being displayed on thedisplay 1051 may have been derived from images captured by acamera 1003 such as a stereoscopic endoscope. Alternatively, they may have been derived from images captured by a secondimage capturing device 1005 such as an ultrasound probe. Alternatively, the images being displayed on thedisplay 1051 may include both images derived from those captured by acamera 1003 and images derived from those captured by a secondimage capturing device 1005 by registering and superimposing the images. - As an example of one way for the user to specify the region of interest relative to the image being displayed at the time is to use a telestrator to draw a closed curve around the region of interest on a telestration screen which is displaying the same image being displayed at the time on the
display 1051. For details on such a telestrator see U.S. Published Application No. 2007/0167702 A1 entitled “Medical robotic system providing three-dimensional telestration,” which is incorporated herein by reference. As another example, the user may specify the region of interest by simply using an input device such as a mouse to control movement of a cursor on thedisplay 1051 so that the path of the cursor defines the region of interest. As another example of a way for the user to specify the region of interest, the user may use a gaze tracker and a user controlled switch to indicate when the user's current gaze point on thedisplay 1051 is to specify a center of a region of interest. In this case, the shape of the region of interest may be predefined or it may be determined by the shape of an object upon which the user is currently gazing. As yet another example of a way for the user to specify a region of interest, the user may telerobotically control movement of a working end of the manipulatable device so that it touches an object at the work site to indicate that the region of interest should be the object that has been touched. - In block 4002, the method receives information of the initial arm configurations and base positions of the image capturing device robot arm and a manipulatable device robot arm. Information of the initial arm configurations may be provided by information of their construction and information of their joint positions which is received from a plurality of sensors of the robot arms. Information of the base positions in the world reference frame may be provided by external sensors which provide such information. One example of such an external sensor is an optical sensor that is used in an optical tracking system. Another example is a pressure transducer that is one of an array of pressure transducers strategically placed at known positions on the floor of the work site.
- In block 4003, the method determines a set-up position for the image capturing device so that it is properly positioned to capture the requested images of the user specified region of interest. In order to do this, the method first translates the region of interest from the display reference frame to an image capturing device reference frame using a previously determined transform and information received in block 4002. It then translates the region of interest from the image capturing device reference frame to the world reference frame using a previously determined transform and information received in block 4002. With the region of interest known in the world reference frame, the method then determines a set-up position from which the image capturing device may capture the requested images of the user specified region of interest without colliding with the initial position in the world reference frame of the manipulatable device robot arm.
- In
block 4004, the method determines a plan of motion for the image capturing device to capture the requested images of the user specified region of interest. When the image capturing device is theimage capturing device 3020 ofFIG. 12 , the plan of motion may comprise planned movement of thelinks wrist 1104 and the C-arm 1105. - In
block 4005, the method commands movement of the image capturing device to its set-up position after receiving a command to do so from the user. When the image capturing device is theimage capturing device Image Capturing System 3020 ofFIG. 12 , the commanded movement typically entails rotating the carousel joint 1108 so that therobot arm 1101 is facing the region of interest and rotating theshoulder joint 1109, elbow joint 1110, roll joint 1111, pitch joint 1112, and yaw joint 1121 so that theX-ray source 1117 and X-ray detector 1116 (i.e., the image capturing device) are properly positioned to start capturing the requested images of the user specified region of interest. - In
block 4006, the method commands incremental movement of the image capturing device in accordance with the plan of motion while avoiding a collision between the image capturing device robot arm and the manipulatable device robot arm. Additional details of tasks performed by the method inblock 4006 are described in reference toFIG. 6 . - In
block 4007, the method determines whether the plan of motion is completed. If the determination inblock 4007 is NO, then the method jumps back to block 4006 to command another incremental movement of the image capturing device according to the plan of motion while avoiding a collision between the image capturing device robot arm and the manipulatable device robot arm. - On the other hand, if the determination in
block 4007 is YES, then inblock 4008, the method commands the image capturing device to move back to its initial position or a predefined parking position upon receiving a command to do so from the user, so that it is out of the way and presents no further risk of collision with the manipulatable device robot arm. -
FIG. 6 illustrates, as an example, a flow diagram of a method for performingblock 4006 of themethod 4000. To supplement the description,FIG. 7 illustrates a block diagram of a portion of a robotic system which is used for collision avoidance between two independently operated robot arms. In the example illustrated byFIG. 7 , a manipulatable devicerobot arm simulator 6003, animminent collision detector 6021, an image capturing device robot arm simulator 6013, and anincremental motion generator 6004 are preferably implemented in a conventional manner by a processor executing program code. - In
block 4201, the method receives a movement command for the manipulatable device robot arm. For example, referring toFIG. 7 , the output of themanipulatable device controller 6011 provides joint commands for the manipulatable devicerobot arm actuators 6012. - In
block 4202, the method determines the position and configuration for the manipulatable device robot arm. For example, referring toFIG. 7 , the manipulatable device robotarm simulator unit 6003 generates information of the position and configuration for the manipulatable device robot arm using the joint commands from themanipulatable device controller 6011 and prior knowledge of the construction of the manipulatable device robot arm. - In
block 4203, the method determines a desired incremental movement of the image capturing device. For example, referring toFIG. 7 , theincremental motion generator 6004 generates a desired incremental movement of the image capturing device using information of the plan of motion determined inblock 4004 ofFIG. 5 and stored inmemory 6005. The desired incremental movement in this case depends upon a desired velocity for the movement of the image capturing device and a process cycle period for looping through blocks 4006-4007 ofFIG. 5 . - In
block 4204, the method determines the position and configuration for the image capturing device robot arm. For example, referring toFIG. 7 , the image capturing device robot arm simulator unit 6013 generates information of the position and configuration for the image capturing device robot arm using the desired incremental motion commands provided by theincremental motion generator 6004, knowledge of the current position and configuration of the image capturing device robot arm, and prior knowledge of the construction of the image capturing device robot arm. - In
block 4205, the method determines whether an imminent collision is threatened between the image capturing device robot arm and the manipulatable device robot arm in a similar manner as performed inblock 2008 ofFIG. 3 . For example, referring toFIG. 7 , theimminent collision detector 6021 determines whether a collision between the two arms is imminent by using the information provided by therobot arm simulators 6003, 6013 for the current process cycle and previous process cycles. By using time series information, not only can the trajectories of the two robot arms be predicted, but the rates at which they are moving may be estimated. With this information, a collision prediction may be made. When the predicted collision is within a specified time period, then the collision is considered to be imminent, requiring immediate action. - If the determination in
block 4205 is NO, then inblock 4206, the method commands the image capturing device robot arm to move according to the desired incremental movement. On the other hand, if the determination inblock 4205 is YES, then inblock 4207, the method modifies the incremental movement command to avoid the imminent collision, using as examples, one of the actions described in reference to block 2009 ofFIG. 3 . For example, after theimminent collision detector 6021 has determined that a collision between the image capturing device robot arm and the manipulatable device robot arm is imminent, it sends an indication of such to themanipulatable device controller 6011, which in turn, relays the indication to the imagecapturing device controller 6001 so that it can take corrective action to avoid the collision. - Some embodiments of the control units 1021-1025 and/or 3021 may include non-transient, tangible, machine readable media that include executable code that when run by one or more processors may cause the one or more processors (e.g., the processing units of control units 1021-1025 and/or 3021) to perform the processes of
methods methods - Although the various aspects of the present invention have been described with respect to a preferred embodiment, it will be understood that the invention is entitled to full protection within the full scope of the appended claims.
Claims (21)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/053,049 US11007023B2 (en) | 2012-12-10 | 2018-08-02 | System and method of registration between devices with movable arms |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201261735170P | 2012-12-10 | 2012-12-10 | |
US14/101,769 US9259282B2 (en) | 2012-12-10 | 2013-12-10 | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms |
US14/862,692 US10064682B2 (en) | 2012-12-10 | 2015-09-23 | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms |
US16/053,049 US11007023B2 (en) | 2012-12-10 | 2018-08-02 | System and method of registration between devices with movable arms |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/862,692 Continuation US10064682B2 (en) | 2012-12-10 | 2015-09-23 | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms |
Publications (2)
Publication Number | Publication Date |
---|---|
US20180338804A1 true US20180338804A1 (en) | 2018-11-29 |
US11007023B2 US11007023B2 (en) | 2021-05-18 |
Family
ID=50881820
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/101,769 Active US9259282B2 (en) | 2012-12-10 | 2013-12-10 | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms |
US14/862,692 Active 2034-12-03 US10064682B2 (en) | 2012-12-10 | 2015-09-23 | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms |
US16/053,049 Active 2034-10-24 US11007023B2 (en) | 2012-12-10 | 2018-08-02 | System and method of registration between devices with movable arms |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/101,769 Active US9259282B2 (en) | 2012-12-10 | 2013-12-10 | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms |
US14/862,692 Active 2034-12-03 US10064682B2 (en) | 2012-12-10 | 2015-09-23 | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms |
Country Status (6)
Country | Link |
---|---|
US (3) | US9259282B2 (en) |
EP (2) | EP2928407B1 (en) |
JP (2) | JP6293777B2 (en) |
KR (1) | KR102218244B1 (en) |
CN (3) | CN110279427B (en) |
WO (1) | WO2014093367A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109620410A (en) * | 2018-12-04 | 2019-04-16 | 微创(上海)医疗机器人有限公司 | The method and system of mechanical arm anticollision, medical robot |
US20210187751A1 (en) * | 2018-09-12 | 2021-06-24 | Canon Kabushiki Kaisha | Robot system, control apparatus of robot system, control method of robot system, imaging apparatus, and storage medium |
US11980426B2 (en) | 2020-08-03 | 2024-05-14 | Warsaw Orthopedic, Inc. | System and method for preliminary registration |
Families Citing this family (221)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9948885B2 (en) * | 2003-12-12 | 2018-04-17 | Kurzweil Technologies, Inc. | Virtual encounters |
US20110160905A1 (en) * | 2008-09-03 | 2011-06-30 | Honda Motor Co., Ltd. | Workpiece mounting system, workpiece mounting method, sunroof unit holding device, and sunroof unit holding method |
US11871901B2 (en) | 2012-05-20 | 2024-01-16 | Cilag Gmbh International | Method for situational awareness for surgical network or surgical network connected device capable of adjusting function based on a sensed situation or usage |
KR102218244B1 (en) | 2012-12-10 | 2021-02-22 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms |
US9649765B2 (en) | 2013-03-11 | 2017-05-16 | Siemens Aktiengesellschaft | Reducing energy consumption of industrial robots by using new methods for motion path programming |
BR112015022187A2 (en) | 2013-03-14 | 2017-07-18 | Stanford Res Inst Int | minimally invasive surgical tools |
US10383699B2 (en) | 2013-03-15 | 2019-08-20 | Sri International | Hyperdexterous surgical system |
EP2969404B1 (en) | 2013-03-15 | 2021-08-11 | Intuitive Surgical Operations, Inc. | Systems for using the null space to emphasize anipulator joint motion anisotropically |
CN105050528B (en) * | 2013-03-15 | 2019-04-19 | 直观外科手术操作公司 | It is converted in the operation of tool in robotic surgical system |
JP5713047B2 (en) * | 2013-04-18 | 2015-05-07 | 株式会社安川電機 | Mobile robot, mobile robot positioning system, and mobile robot positioning method |
JP6005299B2 (en) * | 2013-11-28 | 2016-10-12 | 三菱電機株式会社 | Robot system and control method of robot system |
DK3076906T3 (en) * | 2013-12-05 | 2018-10-22 | Now Tech Zrt | Passenger vehicle and steering gear and steering method therefor |
EP3578119B1 (en) | 2013-12-11 | 2021-03-17 | Covidien LP | Wrist and jaw assemblies for robotic surgical systems |
JP5850958B2 (en) * | 2014-01-24 | 2016-02-03 | ファナック株式会社 | Robot programming device for creating a robot program for imaging a workpiece |
CN110236675B (en) | 2014-03-17 | 2023-05-02 | 直观外科手术操作公司 | Method and apparatus for stage pose tracking using fiducial markers |
CN111616666A (en) * | 2014-03-19 | 2020-09-04 | 直观外科手术操作公司 | Medical devices, systems, and methods using eye gaze tracking |
EP3119343A4 (en) | 2014-03-19 | 2017-12-20 | Intuitive Surgical Operations, Inc. | Medical devices, systems, and methods integrating eye gaze tracking for stereo viewer |
US9922144B2 (en) | 2014-03-26 | 2018-03-20 | Siemens Industry Software Ltd. | Energy and cycle time efficiency based method for robot positioning |
US20150277398A1 (en) * | 2014-03-26 | 2015-10-01 | Siemens Industry Software Ltd. | Object manipulation driven robot offline programming for multiple robot system |
EP2930462B1 (en) * | 2014-04-08 | 2017-09-13 | Hexagon Technology Center GmbH | Method for generating information about a sensor chain of a coordinate measuring machine (CMM) |
US9701011B2 (en) * | 2014-05-08 | 2017-07-11 | Siemens Industry Software Ltd. | Method for robotic energy saving tool search |
US9469029B2 (en) | 2014-07-31 | 2016-10-18 | Siemens Industry Software Ltd. | Method and apparatus for saving energy and reducing cycle time by optimal ordering of the industrial robotic path |
US9815201B2 (en) | 2014-07-31 | 2017-11-14 | Siemens Industry Software Limited | Method and apparatus for industrial robotic energy saving optimization using fly-by |
US10390853B2 (en) | 2014-08-13 | 2019-08-27 | Covidien Lp | Robotically controlling mechanical advantage gripping |
US9457469B2 (en) | 2014-08-14 | 2016-10-04 | Siemens Industry Software Ltd. | Method and apparatus for automatic and efficient location generation for cooperative motion |
CN106852114A (en) * | 2014-10-22 | 2017-06-13 | 卡尔斯特里姆保健公司 | Portable radiation imaging apparatus |
KR102480765B1 (en) | 2014-10-27 | 2022-12-23 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | Medical device with active brake release control |
CN110236853B (en) * | 2014-10-27 | 2021-06-04 | 直观外科手术操作公司 | System and method for registration to an operating table |
KR102574095B1 (en) | 2014-10-27 | 2023-09-06 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | System and method for instrument disturbance compensation |
CN107072729B (en) | 2014-10-27 | 2020-03-20 | 直观外科手术操作公司 | System and method for integrated surgical table motion |
KR20230096131A (en) | 2014-10-27 | 2023-06-29 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | System and method for integrated surgical table |
CN107072728B (en) | 2014-10-27 | 2020-07-17 | 直观外科手术操作公司 | System and method for monitoring control points during reactive motion |
WO2016126914A1 (en) * | 2015-02-05 | 2016-08-11 | Intuitive Surgical Operations, Inc. | System and method for anatomical markers |
EP3258874B1 (en) | 2015-02-19 | 2024-01-17 | Covidien LP | Input device for robotic surgical system |
CN113040921A (en) | 2015-03-10 | 2021-06-29 | 柯惠Lp公司 | Measuring health of connector components of a robotic surgical system |
DE102015104821A1 (en) * | 2015-03-27 | 2016-09-29 | Medineering Gmbh | Surgical instrument, system, arm and procedure |
US10959783B2 (en) * | 2015-04-15 | 2021-03-30 | Mobius Imaging, Llc | Integrated medical imaging and surgical robotic system |
WO2016196238A1 (en) | 2015-06-03 | 2016-12-08 | Covidien Lp | Offset instrument drive unit |
GB201509887D0 (en) * | 2015-06-08 | 2015-07-22 | Univ Strathclyde | Remote operations system |
DE102016104970A1 (en) | 2015-06-15 | 2016-12-15 | Knorr-Bremse Systeme für Nutzfahrzeuge GmbH | Disc brake for a commercial vehicle and brake pad set |
JP6761822B2 (en) | 2015-06-16 | 2020-09-30 | コヴィディエン リミテッド パートナーシップ | Robot Surgical System Torque Conversion Detection |
AU2016284040B2 (en) | 2015-06-23 | 2020-04-30 | Covidien Lp | Robotic surgical assemblies |
CN108025445A (en) | 2015-07-23 | 2018-05-11 | 斯里国际 | Robots arm and robotic surgical system |
US10034716B2 (en) | 2015-09-14 | 2018-07-31 | Globus Medical, Inc. | Surgical robotic systems and methods thereof |
US10806454B2 (en) | 2015-09-25 | 2020-10-20 | Covidien Lp | Robotic surgical assemblies and instrument drive connectors thereof |
US10299740B2 (en) * | 2015-09-29 | 2019-05-28 | General Electric Company | Methods and systems for cone-beam computed tomography |
CN112842228A (en) | 2015-10-23 | 2021-05-28 | 柯惠Lp公司 | Surgical system for detecting gradual changes in perfusion |
US10639108B2 (en) | 2015-10-30 | 2020-05-05 | Auris Health, Inc. | Process for percutaneous operations |
US9955986B2 (en) | 2015-10-30 | 2018-05-01 | Auris Surgical Robotics, Inc. | Basket apparatus |
US9949749B2 (en) | 2015-10-30 | 2018-04-24 | Auris Surgical Robotics, Inc. | Object capture with a basket |
CA3004167C (en) * | 2015-11-03 | 2019-02-05 | Synaptive Medical (Barbados) Inc. | Dual zoom and dual field-of-view microscope |
CN108348297B (en) | 2015-11-19 | 2021-08-31 | 柯惠Lp公司 | Optical force sensor for robotic surgical system |
US10130429B1 (en) | 2016-01-06 | 2018-11-20 | Ethicon Llc | Methods, systems, and devices for controlling movement of a robotic surgical system |
US10154886B2 (en) * | 2016-01-06 | 2018-12-18 | Ethicon Llc | Methods, systems, and devices for controlling movement of a robotic surgical system |
US10219868B2 (en) | 2016-01-06 | 2019-03-05 | Ethicon Llc | Methods, systems, and devices for controlling movement of a robotic surgical system |
US20190021699A1 (en) * | 2016-01-15 | 2019-01-24 | Koninklijke Philips N.V. | Automatic probe steering to clinical views using annotations in a fused image guidance system |
CN108289719B (en) * | 2016-01-20 | 2021-09-14 | 直观外科手术操作公司 | System and method for rapidly pausing and resuming motion deviation in a medical device repositionable arm |
CN108883296A (en) * | 2016-01-20 | 2018-11-23 | 富士设计科技有限公司 | Boron neutron capture therapy method system |
EP3406221A4 (en) * | 2016-01-22 | 2019-10-02 | Olympus Corporation | Medical manipulator system |
CN108882908B (en) * | 2016-02-01 | 2022-06-14 | 伊美姬艾利斯有限公司 | Radiation imaging apparatus |
EP3415071A4 (en) * | 2016-02-10 | 2019-11-20 | Olympus Corporation | Endoscope system |
WO2017146890A1 (en) | 2016-02-26 | 2017-08-31 | Intuitive Surgical Operations, Inc. | System and method for collision avoidance using virtual boundaries |
US10864050B2 (en) | 2016-02-26 | 2020-12-15 | Think Surgical, Inc. | Method and system for guiding user positioning of a robot |
US11576562B2 (en) | 2016-04-07 | 2023-02-14 | Titan Medical Inc. | Camera positioning method and apparatus for capturing images during a medical procedure |
WO2017179624A1 (en) * | 2016-04-15 | 2017-10-19 | 川崎重工業株式会社 | Control method of surgical operation system and surgical operation system |
US10531926B2 (en) * | 2016-05-23 | 2020-01-14 | Mako Surgical Corp. | Systems and methods for identifying and tracking physical objects during a robotic surgical procedure |
US10736219B2 (en) | 2016-05-26 | 2020-08-04 | Covidien Lp | Instrument drive units |
EP3463146A4 (en) | 2016-05-26 | 2020-03-11 | Covidien LP | Robotic surgical assemblies |
WO2017210500A1 (en) | 2016-06-03 | 2017-12-07 | Covidien Lp | Robotic surgical system with an embedded imager |
CN109195543A (en) | 2016-06-03 | 2019-01-11 | 柯惠Lp公司 | Passive axle system for robotic surgical system |
CN109275333B (en) | 2016-06-03 | 2022-05-17 | 柯惠Lp公司 | System, method and computer readable program product for controlling a robotic delivery manipulator |
US11446099B2 (en) | 2016-06-03 | 2022-09-20 | Covidien Lp | Control arm for robotic surgical systems |
US10427305B2 (en) * | 2016-07-21 | 2019-10-01 | Autodesk, Inc. | Robotic camera control via motion capture |
US10131053B1 (en) * | 2016-09-14 | 2018-11-20 | X Development Llc | Real time robot collision avoidance |
WO2018053282A1 (en) | 2016-09-16 | 2018-03-22 | GYS Tech, LLC d/b/a Cardan Robotics | System and method for mounting a robotic arm in a surgical robotic system |
CN106406312B (en) * | 2016-10-14 | 2017-12-26 | 平安科技(深圳)有限公司 | Guide to visitors robot and its moving area scaling method |
EP3528735A4 (en) | 2016-10-21 | 2020-04-29 | Mobius Imaging LLC | Methods and systems for setting trajectories and target locations for image guided surgery |
CN111417354B (en) | 2016-10-25 | 2023-12-12 | 莫比乌斯成像公司 | Method and system for robotic assisted surgery |
DE102016221222A1 (en) * | 2016-10-27 | 2018-05-03 | Siemens Healthcare Gmbh | A method of operating a collision protection system for a medical surgical device, medical surgical device, computer program and data carrier |
CN110114046B (en) | 2016-11-28 | 2021-07-13 | 威博外科公司 | Robotic surgical system with reduced undesirable vibrations |
AU2017378250B2 (en) | 2016-12-16 | 2023-09-28 | Mako Surgical Corp. | Techniques for detecting errors or loss of accuracy in a surgical robotic system |
WO2018112424A1 (en) * | 2016-12-16 | 2018-06-21 | Intuitive Surgical Operations, Inc. | Systems and methods for teleoperated control of an imaging instrument |
CA3048039A1 (en) | 2017-02-15 | 2018-08-23 | Covidien Lp | System and apparatus for crush prevention for medical robot applications |
JP6603890B2 (en) * | 2017-02-20 | 2019-11-13 | パナソニックIpマネジメント株式会社 | Working apparatus and electronic device assembling method |
JP6445604B2 (en) * | 2017-03-07 | 2018-12-26 | 上銀科技股▲フン▼有限公司 | Robot arm |
US10223821B2 (en) * | 2017-04-25 | 2019-03-05 | Beyond Imagination Inc. | Multi-user and multi-surrogate virtual encounters |
US10792119B2 (en) | 2017-05-22 | 2020-10-06 | Ethicon Llc | Robotic arm cart and uses therefor |
US11717361B2 (en) | 2017-05-24 | 2023-08-08 | Covidien Lp | Electrosurgical robotic system having tool presence detection |
JP7130003B2 (en) | 2017-05-25 | 2022-09-02 | コヴィディエン リミテッド パートナーシップ | Systems and methods for detection of objects within the field of view of an image capture device |
CN110662507A (en) | 2017-05-25 | 2020-01-07 | 柯惠Lp公司 | Robotic surgical system with automatic guidance |
JP2020520694A (en) | 2017-05-25 | 2020-07-16 | コヴィディエン リミテッド パートナーシップ | Drapes for covering robotic surgery systems and components of robotic surgery systems |
US10919152B1 (en) * | 2017-05-30 | 2021-02-16 | Nimble Robotics, Inc. | Teleoperating of robots with tasks by mapping to human operator pose |
US10856948B2 (en) | 2017-05-31 | 2020-12-08 | Verb Surgical Inc. | Cart for robotic arms and method and apparatus for registering cart to surgical table |
US10485623B2 (en) | 2017-06-01 | 2019-11-26 | Verb Surgical Inc. | Robotic arm cart with fine position adjustment features and uses therefor |
US10913145B2 (en) | 2017-06-20 | 2021-02-09 | Verb Surgical Inc. | Cart for robotic arms and method and apparatus for cartridge or magazine loading of arms |
WO2019006028A1 (en) | 2017-06-28 | 2019-01-03 | Intuitive Surgical Operations, Inc. | Systems and methods for projecting an endoscopic image to a three-dimensional volume |
CN107361849B (en) | 2017-07-31 | 2023-11-21 | 成都博恩思医学机器人有限公司 | Console for actuating an actuator |
WO2019032450A1 (en) | 2017-08-08 | 2019-02-14 | Intuitive Surgical Operations, Inc. | Systems and methods for rendering alerts in a display of a teleoperational system |
CN111132631A (en) * | 2017-08-10 | 2020-05-08 | 直观外科手术操作公司 | System and method for interactive point display in a teleoperational assembly |
EP3664741B1 (en) | 2017-08-11 | 2023-02-22 | Mobius Imaging LLC | Apparatus for attaching a reference marker to a patient |
EP3668439B1 (en) | 2017-08-16 | 2023-11-15 | Covidien LP | Synthesizing spatially-aware transitions between multiple camera viewpoints during minimally invasive surgery |
CN111050686B (en) * | 2017-09-05 | 2023-06-16 | 柯惠Lp公司 | Camera control for surgical robotic system |
JP7349992B2 (en) | 2017-09-05 | 2023-09-25 | コヴィディエン リミテッド パートナーシップ | Collision handling algorithms for robotic surgical systems |
CN111132629B (en) | 2017-09-06 | 2024-04-16 | 柯惠Lp公司 | Boundary scaling for surgical robots |
EP3461415A1 (en) * | 2017-09-27 | 2019-04-03 | Koninklijke Philips N.V. | System and method for positioning a mobile medical imaging system |
US10610307B2 (en) * | 2017-09-28 | 2020-04-07 | General Electric Company | Workflow assistant for image guided procedures |
EP3461410A1 (en) * | 2017-09-29 | 2019-04-03 | Koninklijke Philips N.V. | Kinematical joints for x-ray systems |
EP3691545A4 (en) | 2017-10-04 | 2022-02-16 | Mobius Imaging, LLC | Systems and methods for performing lateral-access spine surgery |
WO2019071189A2 (en) | 2017-10-05 | 2019-04-11 | GYS Tech, LLC d/b/a Cardan Robotics | Methods and systems for performing computer assisted surgery |
US11510741B2 (en) | 2017-10-30 | 2022-11-29 | Cilag Gmbh International | Method for producing a surgical instrument comprising a smart electrical system |
US11911045B2 (en) | 2017-10-30 | 2024-02-27 | Cllag GmbH International | Method for operating a powered articulating multi-clip applier |
US20190125320A1 (en) | 2017-10-30 | 2019-05-02 | Ethicon Llc | Control system arrangements for a modular surgical instrument |
US11801098B2 (en) | 2017-10-30 | 2023-10-31 | Cilag Gmbh International | Method of hub communication with surgical instrument systems |
CN111417356A (en) | 2017-12-01 | 2020-07-14 | 柯惠Lp公司 | Drape management assembly for robotic surgical system |
JP6654180B2 (en) * | 2017-12-14 | 2020-02-26 | 株式会社メディカロイド | Remote control device and remote operation system |
US11213359B2 (en) * | 2017-12-28 | 2022-01-04 | Cilag Gmbh International | Controllers for robot-assisted surgical platforms |
US11464559B2 (en) | 2017-12-28 | 2022-10-11 | Cilag Gmbh International | Estimating state of ultrasonic end effector and control system therefor |
US11969142B2 (en) | 2017-12-28 | 2024-04-30 | Cilag Gmbh International | Method of compressing tissue within a stapling device and simultaneously displaying the location of the tissue within the jaws |
US11857152B2 (en) | 2017-12-28 | 2024-01-02 | Cilag Gmbh International | Surgical hub spatial awareness to determine devices in operating theater |
US11389164B2 (en) | 2017-12-28 | 2022-07-19 | Cilag Gmbh International | Method of using reinforced flexible circuits with multiple sensors to optimize performance of radio frequency devices |
US11903601B2 (en) | 2017-12-28 | 2024-02-20 | Cilag Gmbh International | Surgical instrument comprising a plurality of drive systems |
US11744604B2 (en) | 2017-12-28 | 2023-09-05 | Cilag Gmbh International | Surgical instrument with a hardware-only control circuit |
US20190206569A1 (en) | 2017-12-28 | 2019-07-04 | Ethicon Llc | Method of cloud based data analytics for use with the hub |
US11202570B2 (en) | 2017-12-28 | 2021-12-21 | Cilag Gmbh International | Communication hub and storage device for storing parameters and status of a surgical device to be shared with cloud based analytics systems |
US20190201594A1 (en) | 2017-12-28 | 2019-07-04 | Ethicon Llc | Method of sensing particulate from smoke evacuated from a patient, adjusting the pump speed based on the sensed information, and communicating the functional parameters of the system to the hub |
US11678881B2 (en) | 2017-12-28 | 2023-06-20 | Cilag Gmbh International | Spatial awareness of surgical hubs in operating rooms |
US20190201039A1 (en) | 2017-12-28 | 2019-07-04 | Ethicon Llc | Situational awareness of electrosurgical systems |
US11666331B2 (en) | 2017-12-28 | 2023-06-06 | Cilag Gmbh International | Systems for detecting proximity of surgical end effector to cancerous tissue |
US11864728B2 (en) | 2017-12-28 | 2024-01-09 | Cilag Gmbh International | Characterization of tissue irregularities through the use of mono-chromatic light refractivity |
US11257589B2 (en) | 2017-12-28 | 2022-02-22 | Cilag Gmbh International | Real-time analysis of comprehensive cost of all instrumentation used in surgery utilizing data fluidity to track instruments through stocking and in-house processes |
US11786251B2 (en) | 2017-12-28 | 2023-10-17 | Cilag Gmbh International | Method for adaptive control schemes for surgical network control and interaction |
US11818052B2 (en) | 2017-12-28 | 2023-11-14 | Cilag Gmbh International | Surgical network determination of prioritization of communication, interaction, or processing based on system or device needs |
US11844579B2 (en) | 2017-12-28 | 2023-12-19 | Cilag Gmbh International | Adjustments based on airborne particle properties |
US11166772B2 (en) | 2017-12-28 | 2021-11-09 | Cilag Gmbh International | Surgical hub coordination of control and communication of operating room devices |
US11109866B2 (en) | 2017-12-28 | 2021-09-07 | Cilag Gmbh International | Method for circular stapler control algorithm adjustment based on situational awareness |
US11832899B2 (en) | 2017-12-28 | 2023-12-05 | Cilag Gmbh International | Surgical systems with autonomously adjustable control programs |
US11896443B2 (en) | 2017-12-28 | 2024-02-13 | Cilag Gmbh International | Control of a surgical system through a surgical barrier |
US11896322B2 (en) | 2017-12-28 | 2024-02-13 | Cilag Gmbh International | Sensing the patient position and contact utilizing the mono-polar return pad electrode to provide situational awareness to the hub |
US11672605B2 (en) | 2017-12-28 | 2023-06-13 | Cilag Gmbh International | Sterile field interactive control displays |
US11998193B2 (en) | 2017-12-28 | 2024-06-04 | Cilag Gmbh International | Method for usage of the shroud as an aspect of sensing or controlling a powered surgical device, and a control algorithm to adjust its default operation |
US11937769B2 (en) | 2017-12-28 | 2024-03-26 | Cilag Gmbh International | Method of hub communication, processing, storage and display |
US20190201139A1 (en) | 2017-12-28 | 2019-07-04 | Ethicon Llc | Communication arrangements for robot-assisted surgical platforms |
US11969216B2 (en) | 2017-12-28 | 2024-04-30 | Cilag Gmbh International | Surgical network recommendations from real time analysis of procedure variables against a baseline highlighting differences from the optimal solution |
US10758310B2 (en) | 2017-12-28 | 2020-09-01 | Ethicon Llc | Wireless pairing of a surgical device with another device within a sterile surgical field based on the usage and situational awareness of devices |
US11132462B2 (en) | 2017-12-28 | 2021-09-28 | Cilag Gmbh International | Data stripping method to interrogate patient records and create anonymized record |
US10595887B2 (en) | 2017-12-28 | 2020-03-24 | Ethicon Llc | Systems for adjusting end effector parameters based on perioperative information |
US11551810B2 (en) | 2018-01-03 | 2023-01-10 | Siemens Healthcare Gmbh | Method for acquiring and for altering a configuration of a number of objects in a procedure room and corresponding device |
JP2021510562A (en) | 2018-01-04 | 2021-04-30 | コヴィディエン リミテッド パートナーシップ | Systems and assemblies for mounting surgical accessories on a robotic surgery system and providing access through it |
US12029510B2 (en) | 2018-01-10 | 2024-07-09 | Covidien Lp | Determining positions and conditions of tools of a robotic surgical system utilizing computer vision |
WO2019139815A1 (en) | 2018-01-12 | 2019-07-18 | Duke University | Apparatus, method and article to facilitate motion planning of an autonomous vehicle in an environment having dynamic objects |
US11189379B2 (en) | 2018-03-06 | 2021-11-30 | Digital Surgery Limited | Methods and systems for using multiple data structures to process surgical data |
US11628024B2 (en) | 2018-03-08 | 2023-04-18 | Covidien Lp | Surgical robotic systems |
US11464532B2 (en) | 2018-03-08 | 2022-10-11 | Cilag Gmbh International | Methods for estimating and controlling state of ultrasonic end effector |
US11589915B2 (en) | 2018-03-08 | 2023-02-28 | Cilag Gmbh International | In-the-jaw classifier based on a model |
US11259830B2 (en) | 2018-03-08 | 2022-03-01 | Cilag Gmbh International | Methods for controlling temperature in ultrasonic device |
WO2019183141A1 (en) | 2018-03-21 | 2019-09-26 | Realtime Robotics, Inc. | Motion planning of a robot for various environments and tasks and improved operation of same |
US11090047B2 (en) | 2018-03-28 | 2021-08-17 | Cilag Gmbh International | Surgical instrument comprising an adaptive control system |
US11259806B2 (en) | 2018-03-28 | 2022-03-01 | Cilag Gmbh International | Surgical stapling devices with features for blocking advancement of a camming assembly of an incompatible cartridge installed therein |
EP3782573A4 (en) | 2018-04-17 | 2022-01-26 | Chengdu Borns Medical Robotics Inc. | Laparoscope-holding robot system for laparoscopic surgery |
EP3781073A4 (en) | 2018-04-20 | 2022-01-26 | Covidien LP | Compensation for observer movement in robotic surgical systems having stereoscopic displays |
CN111971150A (en) | 2018-04-20 | 2020-11-20 | 柯惠Lp公司 | System and method for surgical robot cart placement |
US20190336093A1 (en) * | 2018-05-02 | 2019-11-07 | Epica International, Inc. | System for performing robotic surgery |
CN108527374A (en) * | 2018-06-29 | 2018-09-14 | 德淮半导体有限公司 | Anti-collision system and method applied to mechanical arm |
CN112105312A (en) | 2018-07-03 | 2020-12-18 | 柯惠Lp公司 | Systems, methods, and computer-readable media for detecting image degradation during a surgical procedure |
US11597086B2 (en) | 2018-09-13 | 2023-03-07 | The Charles Stark Draper Laboratory, Inc. | Food-safe, washable interface for exchanging tools |
CN112739282A (en) | 2018-09-17 | 2021-04-30 | 柯惠Lp公司 | Surgical robot system |
US11109746B2 (en) | 2018-10-10 | 2021-09-07 | Titan Medical Inc. | Instrument insertion system, method, and apparatus for performing medical procedures |
US11117759B2 (en) * | 2018-11-29 | 2021-09-14 | Invia Robotics, Inc. | Depalletizing robots for an autonomous warehouse |
US11586106B2 (en) | 2018-12-28 | 2023-02-21 | Titan Medical Inc. | Imaging apparatus having configurable stereoscopic perspective |
DE102019200803A1 (en) * | 2019-01-23 | 2020-07-23 | Siemens Healthcare Gmbh | Medical robot, medical system, method for its operation, computer program and storage medium |
US11717355B2 (en) | 2019-01-29 | 2023-08-08 | Covidien Lp | Drive mechanisms for surgical instruments such as for use in robotic surgical systems |
US11576733B2 (en) | 2019-02-06 | 2023-02-14 | Covidien Lp | Robotic surgical assemblies including electrosurgical instruments having articulatable wrist assemblies |
US11484372B2 (en) | 2019-02-15 | 2022-11-01 | Covidien Lp | Articulation mechanisms for surgical instruments such as for use in robotic surgical systems |
US11291444B2 (en) | 2019-02-19 | 2022-04-05 | Cilag Gmbh International | Surgical stapling assembly with cartridge based retainer configured to unlock a closure lockout |
CN113613852B (en) * | 2019-03-20 | 2024-05-10 | 柯惠Lp公司 | Robot surgical collision detection system |
JP7511259B2 (en) | 2019-04-17 | 2024-07-05 | リアルタイム ロボティクス, インコーポレーテッド | MOTION PLANNING GRAPH GENERATION USER INTERFACE, SYSTEM, METHODS, AND ARTICLES - Patent application |
US11369386B2 (en) | 2019-06-27 | 2022-06-28 | Auris Health, Inc. | Systems and methods for a medical clip applier |
WO2020263949A1 (en) | 2019-06-28 | 2020-12-30 | Auris Health, Inc. | Medical instruments including wrists with hybrid redirect surfaces |
CN110448378B (en) * | 2019-08-13 | 2021-03-02 | 北京唯迈医疗设备有限公司 | Immersive intervention operation integrated console |
CN110464465B (en) * | 2019-08-14 | 2020-07-10 | 重庆文理学院 | Auxiliary robot for total knee joint replacement operation |
US11548140B2 (en) * | 2019-08-15 | 2023-01-10 | Covidien Lp | System and method for radio based location of modular arm carts in a surgical robotic system |
US11896330B2 (en) | 2019-08-15 | 2024-02-13 | Auris Health, Inc. | Robotic medical system having multiple medical instruments |
CN114502094A (en) * | 2019-09-26 | 2022-05-13 | 奥瑞斯健康公司 | System and method for collision detection and avoidance |
US11737845B2 (en) | 2019-09-30 | 2023-08-29 | Auris Inc. | Medical instrument with a capstan |
US11691292B2 (en) * | 2019-10-14 | 2023-07-04 | Boston Dynamics, Inc. | Robot choreographer |
CN110613511B (en) * | 2019-10-16 | 2021-03-16 | 武汉联影智融医疗科技有限公司 | Obstacle avoidance method for surgical robot |
JP7276071B2 (en) * | 2019-10-21 | 2023-05-18 | 株式会社島津製作所 | X-ray fluoroscope |
US11737835B2 (en) | 2019-10-29 | 2023-08-29 | Auris Health, Inc. | Braid-reinforced insulation sheath |
WO2021102587A1 (en) * | 2019-11-29 | 2021-06-03 | Innovere Medical Inc. | Systems and methods for passive collision control during medical imaging or therapeutic procedures |
CN113001536B (en) * | 2019-12-20 | 2022-08-23 | 中国科学院沈阳计算技术研究所有限公司 | Anti-collision detection method and device for multiple cooperative robots |
JP2023506355A (en) * | 2019-12-23 | 2023-02-16 | ソニーグループ株式会社 | Computer-assisted surgical system, surgical control device and surgical control method |
WO2021137104A1 (en) | 2019-12-31 | 2021-07-08 | Auris Health, Inc. | Dynamic pulley system |
CN114901200A (en) | 2019-12-31 | 2022-08-12 | 奥瑞斯健康公司 | Advanced basket drive mode |
US11919175B2 (en) | 2020-04-15 | 2024-03-05 | Mujin, Inc. | Robotic system with collision avoidance mechanism and method of operation thereof |
US20210322112A1 (en) * | 2020-04-21 | 2021-10-21 | Mazor Robotics Ltd. | System and method for aligning an imaging device |
EP3906882A1 (en) * | 2020-05-05 | 2021-11-10 | Ecential Robotics | Surgical robotic system |
US12030195B2 (en) | 2020-05-27 | 2024-07-09 | Covidien Lp | Tensioning mechanisms and methods for articulating surgical instruments such as for use in robotic surgical systems |
WO2022003485A1 (en) | 2020-06-29 | 2022-01-06 | Auris Health, Inc. | Systems and methods for detecting contact between a link and an external object |
US11357586B2 (en) | 2020-06-30 | 2022-06-14 | Auris Health, Inc. | Systems and methods for saturated robotic movement |
USD963851S1 (en) | 2020-07-10 | 2022-09-13 | Covidien Lp | Port apparatus |
WO2022014255A1 (en) * | 2020-07-14 | 2022-01-20 | Sony Group Corporation | Determination of surgical performance level |
CA3173684A1 (en) * | 2020-08-19 | 2022-02-24 | Kai Xu | Robot system and control method |
EP4201606A1 (en) * | 2020-08-19 | 2023-06-28 | Beijing Surgerii Robotics Company Limited | Robot system and control method therefor |
WO2022037356A1 (en) * | 2020-08-19 | 2022-02-24 | 北京术锐技术有限公司 | Robot system and control method |
CN112043397B (en) * | 2020-10-08 | 2021-09-24 | 深圳市精锋医疗科技有限公司 | Surgical robot and motion error detection method and detection device thereof |
CN112043396B (en) * | 2020-10-08 | 2022-03-04 | 深圳市精锋医疗科技股份有限公司 | Surgical robot, graphical control device thereof and graphical display method |
CN113967071B (en) | 2020-10-23 | 2023-09-29 | 成都博恩思医学机器人有限公司 | Control method and device for movement of mechanical arm of surgical robot along with operation bed |
CN112426225B (en) * | 2020-11-06 | 2021-09-03 | 哈尔滨工业大学 | Radiotherapy operation system based on motion capture |
JP2022101850A (en) * | 2020-12-25 | 2022-07-07 | 川崎重工業株式会社 | Surgery support robot, surgery support system, and surgery support robot control method |
CN112618024A (en) * | 2021-01-05 | 2021-04-09 | 上海交通大学医学院附属第九人民医院 | Multi-arm cooperative surgical robot |
CN112932671A (en) * | 2021-01-28 | 2021-06-11 | 哈尔滨思哲睿智能医疗设备有限公司 | Voice prompt control method and system for laparoscopic surgery robot |
US20220280238A1 (en) * | 2021-03-05 | 2022-09-08 | Verb Surgical Inc. | Robot-assisted setup for a surgical robotic system |
WO2022205207A1 (en) * | 2021-03-31 | 2022-10-06 | 深圳市大疆创新科技有限公司 | Gimbal, and configuration recognition method and apparatus therefor |
CN115338855A (en) * | 2021-05-14 | 2022-11-15 | 台达电子工业股份有限公司 | Double-arm robot assembling system |
US11948226B2 (en) | 2021-05-28 | 2024-04-02 | Covidien Lp | Systems and methods for clinical workspace simulation |
US20220395342A1 (en) * | 2021-06-10 | 2022-12-15 | Mazor Robotics Ltd. | Multi-arm robotic systems and methods for monitoring a target or performing a surgical procedure |
CN114504427B (en) * | 2021-10-03 | 2024-05-31 | 崔迪 | Ophthalmic surgery robot and ophthalmic surgery equipment |
TW202348377A (en) * | 2022-03-11 | 2023-12-16 | 美商即時機器人股份有限公司 | Motion planning and control for robots in shared workspace employing staging poses |
WO2023237922A1 (en) * | 2022-06-06 | 2023-12-14 | Lem Surgical Ag | Robotic active and dynamic collision avoidance system |
WO2024049444A1 (en) * | 2022-09-02 | 2024-03-07 | Intel Corporation | Post-attack real-time trajectory recovery of collaborative robotic arms |
CN117681214B (en) * | 2024-02-04 | 2024-04-12 | 泓浒(苏州)半导体科技有限公司 | Wafer transfer-based multi-mechanical arm collision early warning method and system |
Family Cites Families (58)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
SE456048B (en) | 1982-02-24 | 1988-08-29 | Philips Norden Ab | SET AND DEVICE FOR DETERMINING THE RISK OF COLLISION FOR TWO INBOARD'S LOVELY BODIES |
US4644237A (en) | 1985-10-17 | 1987-02-17 | International Business Machines Corp. | Collision avoidance system |
US5737500A (en) * | 1992-03-11 | 1998-04-07 | California Institute Of Technology | Mobile dexterous siren degree of freedom robot arm with real-time control system |
US6659939B2 (en) | 1998-11-20 | 2003-12-09 | Intuitive Surgical, Inc. | Cooperative minimally invasive telesurgical system |
US6788018B1 (en) * | 1999-08-03 | 2004-09-07 | Intuitive Surgical, Inc. | Ceiling and floor mounted surgical robot set-up arms |
US8004229B2 (en) * | 2005-05-19 | 2011-08-23 | Intuitive Surgical Operations, Inc. | Software center and highly configurable robotic systems for surgery and other uses |
US7607440B2 (en) * | 2001-06-07 | 2009-10-27 | Intuitive Surgical, Inc. | Methods and apparatus for surgical planning |
US8010180B2 (en) * | 2002-03-06 | 2011-08-30 | Mako Surgical Corp. | Haptic guidance system and method |
US6678582B2 (en) | 2002-05-30 | 2004-01-13 | Kuka Roboter Gmbh | Method and control device for avoiding collisions between cooperating robots |
US6927395B2 (en) * | 2002-06-14 | 2005-08-09 | Koninklijke Philips Electronics N.V. | Gamma camera collision avoidance |
US7046765B2 (en) | 2004-03-31 | 2006-05-16 | Accuray, Inc. | Radiosurgery x-ray system with collision avoidance subsystem |
US7979157B2 (en) * | 2004-07-23 | 2011-07-12 | Mcmaster University | Multi-purpose robotic operating system and method |
WO2006075331A2 (en) * | 2005-01-13 | 2006-07-20 | Mazor Surgical Technologies Ltd. | Image-guided robotic system for keyhole neurosurgery |
US8108072B2 (en) * | 2007-09-30 | 2012-01-31 | Intuitive Surgical Operations, Inc. | Methods and systems for robotic instrument tool tracking with adaptive fusion of kinematics information and image information |
US9789608B2 (en) | 2006-06-29 | 2017-10-17 | Intuitive Surgical Operations, Inc. | Synthetic representation of a surgical robot |
US8971597B2 (en) * | 2005-05-16 | 2015-03-03 | Intuitive Surgical Operations, Inc. | Efficient vision and kinematic data fusion for robotic surgical instruments and other applications |
US9867669B2 (en) | 2008-12-31 | 2018-01-16 | Intuitive Surgical Operations, Inc. | Configuration marker design and detection for instrument tracking |
DE102005023165A1 (en) | 2005-05-19 | 2006-11-23 | Siemens Ag | Medical imaging system with a patient traversable part and collision protection method |
EP2289452A3 (en) | 2005-06-06 | 2015-12-30 | Intuitive Surgical Operations, Inc. | Laparoscopic ultrasound robotic surgical system |
US20070167702A1 (en) | 2005-12-30 | 2007-07-19 | Intuitive Surgical Inc. | Medical robotic system providing three-dimensional telestration |
US7379533B2 (en) | 2006-03-10 | 2008-05-27 | Siemens Medical Solutions Usa, Inc. | Collision resolution in x-ray imaging systems |
CA2542932C (en) | 2006-03-28 | 2011-09-13 | Imris | Detection of collisions in medical procedures |
US9549663B2 (en) * | 2006-06-13 | 2017-01-24 | Intuitive Surgical Operations, Inc. | Teleoperated surgical retractor system |
US8849679B2 (en) * | 2006-06-15 | 2014-09-30 | Intouch Technologies, Inc. | Remote controlled robot system that provides medical images |
US10258425B2 (en) | 2008-06-27 | 2019-04-16 | Intuitive Surgical Operations, Inc. | Medical robotic system providing an auxiliary view of articulatable instruments extending out of a distal end of an entry guide |
DE102006037564C5 (en) | 2006-08-10 | 2010-09-23 | Siemens Ag | Method for X-ray image recording with a robot-guided C-arm system and recording device for X-ray image recording |
DE102007021769B4 (en) | 2007-05-09 | 2015-06-25 | Siemens Aktiengesellschaft | Angiography apparatus and associated recording method with a mechansimus for collision avoidance |
US9138129B2 (en) | 2007-06-13 | 2015-09-22 | Intuitive Surgical Operations, Inc. | Method and system for moving a plurality of articulated instruments in tandem back towards an entry guide |
US8655429B2 (en) * | 2007-06-29 | 2014-02-18 | Accuray Incorporated | Robotic arm for a radiation treatment system |
US8792964B2 (en) | 2008-03-12 | 2014-07-29 | Siemens Aktiengesellschaft | Method and apparatus for conducting an interventional procedure involving heart valves using a robot-based X-ray device |
US8167793B2 (en) | 2008-04-26 | 2012-05-01 | Intuitive Surgical Operations, Inc. | Augmented stereoscopic visualization for a surgical robot using time duplexing |
US8315738B2 (en) * | 2008-05-21 | 2012-11-20 | Fanuc Robotics America, Inc. | Multi-arm robot system interference check via three dimensional automatic zones |
US8126114B2 (en) * | 2008-09-12 | 2012-02-28 | Accuray Incorporated | Seven or more degrees of freedom robotic manipulator having at least one redundant joint |
US8781630B2 (en) | 2008-10-14 | 2014-07-15 | University Of Florida Research Foundation, Inc. | Imaging platform to provide integrated navigation capabilities for surgical guidance |
US8830224B2 (en) * | 2008-12-31 | 2014-09-09 | Intuitive Surgical Operations, Inc. | Efficient 3-D telestration for local robotic proctoring |
WO2010102384A1 (en) | 2009-03-10 | 2010-09-16 | Mcmaster University | Mobile robotic surgical system |
KR100974464B1 (en) * | 2009-04-21 | 2010-08-10 | 주식회사 래보 | Coupling structure of surgical instrument |
US8423186B2 (en) * | 2009-06-30 | 2013-04-16 | Intuitive Surgical Operations, Inc. | Ratcheting for master alignment of a teleoperated minimally-invasive surgical instrument |
US8386080B2 (en) * | 2009-09-15 | 2013-02-26 | Harris Corporation | Robotic apparatus implementing collision avoidance scheme and associated methods |
US8623028B2 (en) * | 2009-09-23 | 2014-01-07 | Intuitive Surgical Operations, Inc. | Surgical port feature |
WO2011040769A2 (en) * | 2009-10-01 | 2011-04-07 | 주식회사 이턴 | Surgical image processing device, image-processing method, laparoscopic manipulation method, surgical robot system and an operation-limiting method therefor |
JP5571432B2 (en) * | 2010-03-30 | 2014-08-13 | カール シュトルツ ゲゼルシャフト ミット ベシュレンクテル ハフツング ウント コンパニー コマンディートゲゼルシャフト | Medical robot system |
IT1401669B1 (en) * | 2010-04-07 | 2013-08-02 | Sofar Spa | ROBOTIC SURGERY SYSTEM WITH PERFECT CONTROL. |
US9400503B2 (en) * | 2010-05-20 | 2016-07-26 | Irobot Corporation | Mobile human interface robot |
US8460236B2 (en) * | 2010-06-24 | 2013-06-11 | Hansen Medical, Inc. | Fiber optic instrument sensing system |
US8855812B2 (en) | 2010-07-23 | 2014-10-07 | Chetan Kapoor | System and method for robot safety and collision avoidance |
JP2012051043A (en) * | 2010-08-31 | 2012-03-15 | Yaskawa Electric Corp | Robot system or robot control device |
US8848874B2 (en) * | 2010-09-20 | 2014-09-30 | Siemens Medical Solutions Usa, Inc. | System for recovering from collision of components of an X-ray imaging unit |
EP2468207A1 (en) * | 2010-12-21 | 2012-06-27 | Renishaw (Ireland) Limited | Method and apparatus for analysing images |
US9259289B2 (en) | 2011-05-13 | 2016-02-16 | Intuitive Surgical Operations, Inc. | Estimation of a position and orientation of a frame used in controlling movement of a tool |
GB201115586D0 (en) * | 2011-09-09 | 2011-10-26 | Univ Bristol | A system for anatomical reduction of bone fractures |
EP2744438B1 (en) * | 2011-09-13 | 2015-04-22 | Koninklijke Philips N.V. | Ablation planning with lesion coverage feedback |
WO2013038301A2 (en) * | 2011-09-13 | 2013-03-21 | Koninklijke Philips Electronics N.V. | Automatic online registration between a robot and images |
US10925564B2 (en) * | 2012-04-20 | 2021-02-23 | Siemens Medical Solutions Usa, Inc. | Medical imaging system with range imaging-based control |
CN102646148B (en) * | 2012-04-25 | 2014-09-24 | 浙江大学 | Motion trajectory planning method of mechanical arm of humanoid robot for preventing collision |
KR102218244B1 (en) | 2012-12-10 | 2021-02-22 | 인튜어티브 서지컬 오퍼레이션즈 인코포레이티드 | Collision avoidance during controlled movement of image capturing device and manipulatable device movable arms |
US11014240B2 (en) * | 2017-09-05 | 2021-05-25 | Abb Schweiz Ag | Robot having dynamic safety zones |
US10058396B1 (en) * | 2018-04-24 | 2018-08-28 | Titan Medical Inc. | System and apparatus for insertion of an instrument into a body cavity for performing a surgical procedure |
-
2013
- 2013-12-10 KR KR1020157018123A patent/KR102218244B1/en active IP Right Grant
- 2013-12-10 EP EP13862685.8A patent/EP2928407B1/en active Active
- 2013-12-10 EP EP21190869.4A patent/EP3932628A1/en active Pending
- 2013-12-10 WO PCT/US2013/074163 patent/WO2014093367A1/en active Application Filing
- 2013-12-10 JP JP2015545931A patent/JP6293777B2/en active Active
- 2013-12-10 CN CN201910317935.5A patent/CN110279427B/en active Active
- 2013-12-10 CN CN201380064534.4A patent/CN105188590B/en active Active
- 2013-12-10 CN CN202311823402.7A patent/CN117770979A/en active Pending
- 2013-12-10 US US14/101,769 patent/US9259282B2/en active Active
-
2015
- 2015-09-23 US US14/862,692 patent/US10064682B2/en active Active
-
2018
- 2018-02-14 JP JP2018024100A patent/JP6543742B2/en active Active
- 2018-08-02 US US16/053,049 patent/US11007023B2/en active Active
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20210187751A1 (en) * | 2018-09-12 | 2021-06-24 | Canon Kabushiki Kaisha | Robot system, control apparatus of robot system, control method of robot system, imaging apparatus, and storage medium |
US11992960B2 (en) * | 2018-09-12 | 2024-05-28 | Canon Kabushiki Kaisha | Robot system, control apparatus of robot system, control method of robot system, imaging apparatus, and storage medium |
CN109620410A (en) * | 2018-12-04 | 2019-04-16 | 微创(上海)医疗机器人有限公司 | The method and system of mechanical arm anticollision, medical robot |
US11980426B2 (en) | 2020-08-03 | 2024-05-14 | Warsaw Orthopedic, Inc. | System and method for preliminary registration |
Also Published As
Publication number | Publication date |
---|---|
CN117770979A (en) | 2024-03-29 |
US10064682B2 (en) | 2018-09-04 |
EP2928407A1 (en) | 2015-10-14 |
CN105188590A (en) | 2015-12-23 |
CN110279427B (en) | 2024-01-16 |
JP2016504077A (en) | 2016-02-12 |
JP2018110873A (en) | 2018-07-19 |
CN110279427A (en) | 2019-09-27 |
US11007023B2 (en) | 2021-05-18 |
US20160008078A1 (en) | 2016-01-14 |
KR102218244B1 (en) | 2021-02-22 |
JP6293777B2 (en) | 2018-03-14 |
US20140163736A1 (en) | 2014-06-12 |
WO2014093367A1 (en) | 2014-06-19 |
KR20150120944A (en) | 2015-10-28 |
JP6543742B2 (en) | 2019-07-10 |
EP2928407B1 (en) | 2021-09-29 |
EP3932628A1 (en) | 2022-01-05 |
CN105188590B (en) | 2019-04-26 |
US9259282B2 (en) | 2016-02-16 |
EP2928407A4 (en) | 2016-10-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11007023B2 (en) | System and method of registration between devices with movable arms | |
CN108472095B (en) | System, controller and method for robotic surgery using virtual reality devices | |
KR102105142B1 (en) | Switching control of an instrument to an input device upon the instrument entering a display area viewable by an operator of the input device | |
JP6662785B2 (en) | System and method for re-centering an imaging device and input controls | |
KR102482803B1 (en) | Secondary mechanism control in computer-assisted remote control system | |
KR20170035831A (en) | Intelligent positioning system and methods therefore | |
CN115605157A (en) | Passive and active arm control schemes with sensor integration to support remote operation and direct manual interaction | |
CN114502094A (en) | System and method for collision detection and avoidance | |
CN115734765A (en) | Robotic medical system with collision proximity indicator | |
CN115802970A (en) | System and method for saturating robot motion | |
US20230064265A1 (en) | Moveable display system | |
US20230126611A1 (en) | Information processing apparatus, information processing system, and information processing method | |
Bihlmaier et al. | Endoscope robots and automated camera guidance | |
US20230190394A1 (en) | Instrument insertion assistant | |
US20220296323A1 (en) | Moveable display unit on track | |
US20210030502A1 (en) | System and method for repositioning input control devices | |
US20240070875A1 (en) | Systems and methods for tracking objects crossing body wallfor operations associated with a computer-assisted system | |
CN116546931A (en) | Techniques for adjusting a field of view of an imaging device based on head movement of an operator | |
CN115397355A (en) | System and method for mounting robotic components on load sensors |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
AS | Assignment |
Owner name: INTUITIVE SURGICAL OPERATIONS, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:AZIZIAN, MAHDI;SORGER, JONATHAN M.;REEL/FRAME:055044/0104 Effective date: 20150831 Owner name: SIEMENS HEALTHCARE GMBH, GERMANY Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BLOHM, LUTZ;KUNZE, HOLGER;NIEBLER, CHRISTINE;SIGNING DATES FROM 20150907 TO 20150908;REEL/FRAME:055045/0428 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: SIEMENS HEALTHINEERS AG, GERMANY Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SIEMENS HEALTHCARE GMBH;REEL/FRAME:066267/0346 Effective date: 20231219 |