Automatic recognition and classification of tasks in robotic surgery is an important stepping stone toward automated surgery and surgical training. Recently, technical breakthroughs in gathering data make data-driven model development possible. In this paper, we propose a framework for high-level robotic surgery task recognition using motion data. We present a novel classification technique that is used to classify three important surgical tasks through quantitative analyses of motion: knot tying, needle passing and suturing. The proposed technique integrates state-of-the-art data mining and time series analysis methods. The first step of this framework consists of developing a time series distance-based similarity measure using derivative dynamic time warping (DDTW). The distance-weighted k-nearest neighbor algorithm was then used to classify task instances. The framework was validated using an extensive dataset. Our results demonstrate the strength of the proposed framework in recognizing fundamental robotic surgery tasks.
@cite_4: Objective: Evaluation of the laparoscopic surgical skills of surgical residents is usually a subjective process carried out in the operating room by senior surgeons. The two hypotheses of the current study were: (1) haptic information and tool tissue interactions (types and transitions) performed in laparoscopic surgery are skill-dependent, and (2) statistical models (Hidden Markov Models—HMMs) incorporating these data are capable of objectively evaluating laparoscopic surgical skills.Materials and Methods: Eight subjects (six residents—two first-year (R1), two third-year (R3), and two fifth-year (R5)—and two expert laparoscopic surgeons) performed laparoscopic cholecystectomy on pigs using an instrumented grasper equipped with force torque (F T) sensors at the hand tool interface, and F T data was synchronized with video of the operative maneuvers. Fourteen types of tool tissue (T T) interactions, each associated with unique F T signatures, were defined from frame-by-frame video analysis. HMMs for each s...
@cite_33: COMPUTATIONAL MODELING APPROACHES FOR TASK ANALYSIS IN ROBOTIC-ASSISTED SURGERY by MAHTAB JAHANBANI FARD May 2016 Advisor: Dr. R. Darin Ellis Major: Industrial Engineering Degree: Doctor of Philosophy Surgery is continuously subject to technological innovations including the introduction of robotic surgical devices. The ultimate goal is to program the surgical robot to perform certain difficult or complex surgical tasks in an autonomous manner. The feasibility of current robotic surgery systems to record quantitative motion and video data motivates developing descriptive mathematical models to recognize, classify and analyze surgical tasks. Recent advances in machine learning research for uncovering concealed patterns in huge data sets, like kinematic and video data, offer a possibility to better understand surgical procedures from a system point of view. This dissertation focuses on bridging the gap between these two lines of the research by developing computational models for task analysis in robotic-assisted surgery. The key step for advance study in robotic-assisted surgery and autonomous skill assess- ment is to develop techniques that are capable of recognizing fundamental surgical tasks intelligently. Surgical tasks and at a more granular level, surgical gestures, need to be quan- tified to make them amenable for further study. To answer to this query, we introduce a new framework, namely DTW-kNN, to recognize and classify three important surgical tasks including suturing, needle passing and knot tying based on kinematic data captured using da Vinci robotic surgery system. Our proposed method needs minimum preprocessing that
@cite_8: Although researchers have developed robust approaches for estimating, location, and user identity, estimating user activities has proven much more challenging. Human activities are so complex and dynamic that it's often unclear what information is even relevant for modeling activities. Robust approaches to recognize user activities requires identifying the relevant information to be sensed and the appropriate sensing technologies. In our effort to develop an approach for automatically estimating hospital-staff activities, we trained a discrete hidden Markov model (HMM) to map contextual information to a user activity. We trained the model and evaluated it using data captured from almost 200 hours of detailed observation and documentation of hospital workers. In this article, we discuss our approach, the results, and how activity recognition could empower our vision of the hospital as a smart environment.
@cite_29: The need for a better integration of the new generation of computer-assisted-surgical systems has been recently emphasized. One necessity to achieve this objective is to retrieve data from the operating room (OR) with different sensors, then to derive models from these data. Recently, the use of videos from cameras in the OR has demonstrated its efficiency. In this paper, we propose a framework to assist in the development of systems for the automatic recognition of high-level surgical tasks using microscope videos analysis. We validated its use on cataract procedures. The idea is to combine state-of-the-art computer vision techniques with time series analysis. The first step of the framework consisted in the definition of several visual cues for extracting semantic information, therefore, characterizing each frame of the video. Five different pieces of image-based classifiers were, therefore, implemented. A step of pupil segmentation was also applied for dedicated visual cue detection. Time series classification algorithms were then applied to model time-varying data. Dynamic time warping and hidden Markov models were tested. This association combined the advantages of all methods for better understanding of the problem. The framework was finally validated through various studies. Six binary visual cues were chosen along with 12 phases to detect, obtaining accuracies of 94 .
@cite_21: Evaluating surgical skill is a time consuming, subjective, and difficult process. This paper compares two methods of identifying the skill level of a subject given motion data from a benchtop surgical task. In the first method, we build discrete Hidden Markov Models at the task level , and test against these models. In the second method, we build discrete Hidden Markov Models of surgical gestures, called surgemes , and evaluate skill at this level. We apply these techniques to 57 data sets collected from the da Vinci surgical system. Our current techniques have achieved accuracy levels of 100 using task level models and known gesture segmentation, 95 with task level models and unknown gesture segmentation, and 100 with the surgeme level models in correctly identifying the skill level. We observe that, although less accurate, the second method requires less prior label information. Also, the surgeme level classification provided more insights into what subjects did well, and what they did poorly.
@cite_24: 
@cite_0: 
@cite_27: 
@cite_15: 
@cite_34: This paper reports our progress in developing techniques for “parsing” raw motion data from a simple surgical task into a labeled sequence of surgical gestures. The ability to automatically detect and segment surgical motion can be useful in evaluating surgical skill, providing surgical training feedback, or documenting essential aspects of a procedure. If processed online, the information can be used to provide context-specific information or motion enhancements to the surgeon. However, in every case, the key step is to relate recorded motion data to a model of the procedure being performed.Robotic surgical systems such as the da Vinci system from Intuitive Surgical provide a rich source of motion and video data from surgical procedures. The application programming interface (API) of the da Vinci outputs 192 kinematics values at 10 Hz. Through a series of feature-processing steps, tailored to this task, the highly redundant features are projected to a compact and discriminative space. The resulting class...
@cite_25: 
@cite_20: The best method of training for laparoscopic surgical skills is controversial. Some advocate observation in the operating room, while others promote animal and simulated models or a combination of surgery-related tasks. A crucial process in surgical education is to evaluate the level of surgical skills. For laparoscopic surgery, skill evaluation is traditionally performed subjectively by experts grading a video of a procedure performed by a student. By its nature, this process uses fuzzy criteria. The objective of the current study was to develop and assess a skill scale using Markov models (MMs). Ten surgeons [five novice surgeons (NS); five expert surgeons (ES)] performed a cholecystectomy and Nissen fundoplication in a porcine model. An instrumented laparoscopic grasper equipped with a three-axis force torque (F T) sensor was used to measure the forces torques at the hand tool interface synchronized with a video of the tool operative maneuvers. A synthesis of frame-by-frame video analysis and a vector quantization algorithm, allowed to define F T signatures associated with 14 different types of tool tissue interactions. The magnitude of F T applied by NS and ES were significantly different (p<0.05) and varied based on the task being performed. High F T magnitudes were applied by NS compared with ES while performing tissue manipulation and vice versa in tasks involved tissue dissection. From each step of the surgical procedures, two MMs were developed representing the performance of three surgeons out of the five in the ES and NS groups. The data obtained by the remaining two surgeons in each group were used for evaluating the performance scale. The final result was a surgical performance index which represented a ratio of statistical similarity between the examined surgeon's MM and the MM of NS and ES. The difference between the performance index value, for a surgeon under study, and the NS ES boundary, indicated the level of expertise in the surgeon's own group. Preliminary data suggest that a performance index based on MM and F T signatures provides an objective means of distinguishing NS from ES. In addition, this methodology can be further applied to evaluate haptic virtual reality surgical simulators for improving realism in surgical education.
In recent years, recognizing and understanding surgical procedures at different levels of granularity has been a focus of research @cite_33 @cite_24 . Surgical procedures can be generally broken down to four main levels, from higher to lower: phases, steps, tasks and motions @cite_25 . At the higher level of surgical process modeling, statistical models have been proposed using recorded force and motion data @cite_20 @cite_0 , surgical tool usage @cite_4 and video data @cite_29 to classify surgery phases. Most existing work has addressed the recognition of activities using different techniques such as neural networks and Hidden Markov Models (HMM) @cite_8 @cite_15 . At the lower level, effort has been applied to detect surgical motion @cite_34 @cite_27 or model surgical gestures and classify them using different methods such as HMM and Linear Discriminant Analysis (LDA) @cite_21 . A common drawback in these methods is that they are time consuming and require significant human interaction and pre-processing.