When operators telemanipulate robotic devices, they are hampered by perception barriers challenging their situational awareness. A surgeon telemanipulating a surgical robot is limited in their understanding of the surgical scene and of the robot's interaction with the anatomy. The robotics research community has dealt with these challenges by focusing on ways of providing force feedback to the surgeons and by providing assistive control laws (called virtual fixtures) that superimpose a safety barrier or help the surgeon follow a desired path while avoiding critical anatomy. These solutions are limited by the reliance on medical image registration (a process by which pre-operative anatomy images are related to the intraoperative scene). Moreover, these solutions relied predominantly on pre-operative geometric definitions of a surgical plan to construct the assistive virtual fixtures.

Figure 1 shows the approach followed in this collaborative research along with some scenarios where the surgical perception is lacking. According to our approach, the robot is used for manipulation augmentation and for perception augmentation by fusing  intraoperative sensory data and imaging (e.g. tissue stiffness, computer vision) with preoperative models and images of the anatomy.  Figures 1b and 1c show scenarios where such perception and situational awareness augmentation would be critical for safe operation. In both images, the robot can have multiple contacts outside the surgeon's visual field of view and there is a need for a robot that can discern such contacts and a high-level controller that uses this information to adapt the telemanipulation behaviors to enable safe operation.  

The concept of robot situational awareness was investigated as part of a paradigm in which intraoperative sensory information was used to inform the update of a surgical plan and its corresponding virtual fixtures. In addition to using geometry, the use of force-controlled palpation and exploration of the anatomy has been explored and demonstrated to allow adaptive surgical plans. Figure 2 shows a robot using force-controlled scan of a mock organ to account for organ deformation relative to a pre-operative model of the organ. Using such intraoperative information, advanced statistical methods were used to take advantage of intraoperative sensing and preoperative information to improve the computer?s model of the patient?s anatomy and the surgical plan. Figure 3 shows a result of force-controlled exploration where an organ stiffness map is generated and used with geometry to inform the process of updating the model of the anatomy. Figure 4 shows steps in an efficient method for real-time stiffness mapping during telemanipulation of the robot. The method allows an interactive rate annotation of the anatomy model with stiffness information which could be used and in figure 3 for identifying possible locations of tumors or for identifying a hidden artery as in figure 4.  

These tools enabled a rigorous exploration of new hybrid assistive telemanipulation frameworks that allow the robot high-level controller to specify behaviors where the robot controls motion or regulates force while allowing the user to telemanipulate the robot tip for remote palpation. Figure 5 shows a subset of these conditions. Automated and semi-automated telemanipulation with superimposed end-effector excitations have been developed to allow the high level controller to discern information tantamount to that obtained during palpation. Ways of relaying this information to users have also been explored through a user study with the aim of determining the potential benefits of these approaches. 

We also focused on sensing using continuum robots to assist with surgical perception. To achieve this, a new approach to model force and motion transmission losses. It was shown that the high level controller can use these modeling techniques to allow assistive behaviors of palpation to support force-controlled virtual fixture model update and to enable regulation of force while carrying out tasks of ablation and knot tying. Figure 6 shows one of our robotic platforms used to test our new sensing and control approaches. The figure shows a new approach for hybrid force/motion control using estimation of tip forces and two sample use scenarios (force regulated knot tying and ablation).

These contributions will facilitate future development of human-robot cooperative systems with applications for surgery, space robotics, search and rescue and potentially robot-worker collaboration in manufacturing.

Other broader impacts of this award supported the training of two post-docs, eight Ph.D. students, and 6 undergraduate students. Thirty-nine archival publications were presented in national and international conferences and journals. Four Ph.D. dissertations were published. In addition, 37 high school female students received STEM and robotics training in three winter classes with each class spanning three weeks. Seven Ph.D.s and two postdocs trained on this program have joined industry research groups in medical robotics and research in human-robot collaboration. One of the Ph.D. students trained on this award started a tenure-track faculty position in the U.S.. 

The project public page is http://nri-csa.vuse.vanderbilt.edu/joomla/ where there are also public data sets and computer code related to this project.

Last Modified: 12/22/2020
Modified by: Howard M Choset