title: Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision
authors: David Z. Li Masaru Ishii Russell H. Taylor Gregory D. Hager Ayushi Sinha
text: 
  
  
  Representations of Endoscopic Videos        
  
  
  
  
  
  Li et al. 
  
  
  Department of Computer Science, The Johns Hopkins University, USA
  
              dli44@alumni.jhu.edu
              Johns Hopkins Medical Institutions, USA
              Laboratory for Computational Sensing and Robotics, The Johns Hopkins University, USA
  
  
  
  Learning Representations of Endoscopic Videos to Detect Tool Presence Without Supervision
      David Z. Li 1Masaru Ishii 2 Russell H. Taylor 1 Gregory D. Hager 1Ayushi Sinha 3
      December 30, 2023
  =========================================================================================
  
  
  
  
  In this work, we explore whether it is possible to learn representations of endoscopic video frames to perform tasks such as identifying surgical tool presence without supervision. We use a maximum mean discrepancy (MMD) variational autoencoder (VAE) to learn low-dimensional latent representations of endoscopic videos and manipulate these representations to distinguish frames containing tools from those without tools. 
  We use three different methods to manipulate these latent representations in order to predict tool presence in each frame.
  Our fully unsupervised methods can identify whether endoscopic video frames contain tools with average precision of 71.56, 73.93, and 76.18, respectively, comparable to supervised methods.
  Our code is available at <https://github.com/zdavidli/tool-presence/>.
  
  
  
  
  
  § INTRODUCTION
  
  Despite the abundance of medical image data, progress in learning from such data has been impeded by the lack of labels and the difficulty in acquiring accurate labels. With increase in minimally invasive procedures <cit.>, an increasing number of endoscopic videos (Fig. <ref>) are available. This can open up the opportunity for video-based surgical education and skill assessment. Prior work <cit.> has shown that both experts and non-experts can produce valid objective skill assessment via pairwise comparisons of surgical videos. However, watching individual videos is time consuming and tedious. Therefore, much work is being done in automating skill assessment using supervised <cit.> and unsupervised <cit.> learning. These prior methods used kinematic data from tools to learn surgical motion. However, many endoscopic procedures do not capture kinematic data. Therefore, we want to explore whether we can work towards automated skill assessment directly from endoscopic video.
  
  Since videos contain more information than just kinematics, we want to first isolate tool motion from camera motion in endoscopic videos. If the two types of motion can be disentangled, then representations of video frames with and without tools should be distinct enough to allow separation between the two. Therefore, our aim in this work is to evaluate whether we can detect frames that contain tools. In order to do this, we use a variational autoencoder (VAE) <cit.> to learn latent representations of endoscopic video frames since VAEs have the ability to learn underlying low-dimensional latent representations from complex, high-dimensional input data.
  Specifically, we use maximum mean discrepancy (MMD) VAE <cit.>, which uses a modified objective function in order to avoid overfitting and promote learning a more informative latent representation.
  
  We then manipulate these learned representations or encodings using three different methods. First, we directly use the encodings produced by our MMD-VAE to evaluate whether encodings of frames with and without tools can be separated. Second, we model the tool presence as a binary latent factor and train a Bayesian mixture model to learn the clusters over our encodings and classify each frame as containing or not containing tools. Third, we use sequences of our encodings to perform future prediction and evaluate whether temporal context can better inform our prediction of tool presence. 
  Our evaluation methods identify frames containing tools with average precision of 71.56, 73.93, and 76.18, respectively, without any explicit labels.
  
  
  
  
  
  § PRIOR WORK
  
  
  Prior work has shown that surgical motion can be learned from robot kinematics. 
  Lea et al. <cit.> and DiPietro et al. <cit.> showed that supervised learning methods can accurately model and recognize surgical activities from robot kinematics. 
  DiPietro et al. <cit.> further showed that encodings learned from robot kinematics in an unsupervised manner also clustered according to high-level activities. 
  However, these methods rely on robot kinematics which provide information like gripper angle, velocity, etc. Endoscopic procedures that do not use robotic manipulation do not produce kinematics data, but do produce endoscopic videos.
  
  Much work has also been dedicated to extracting tools from video frames
  using supervised tool segmentation and tool tracking methods. 
  Many methods ignore the temporal aspect of videos and compute segmentation on a frame-by-frame basis. 
  Several methods use established network architectures, like U-Net, to compute segmentations <cit.>. 
  
  Some methods have tried to tie in the temporal aspect of videos by using recurrent neural networks (RNNs) to segment tools <cit.>, while others have combined simpler fully convolutional networks with optical flow to capture the temporal dimension <cit.>. 
  
  
  More recently, unsupervised methods for learning representations of videos have also been presented <cit.>. While Srivastava et al. <cit.> use RNNs to encode sequences of video frames, which can grow large quickly, we will explore whether unsupervised learning of video representations on a per-frame basis will give us sufficient information to discriminate between frames with and without tools. 
  
  
  
  § METHOD
  
  
  
  
   §.§ Dataset
  
  We use a publicly-available sinus endoscopy video consisting of five segments of continuous endoscope movement from the front of the nasal cavity to the back <cit.>. The video was initially collected at 1080p resolution and split into individual frames.
  Frames that depicted text or where the endoscope was outside the nose were discarded. A total of 1551 frames, downsampled from 1080p resolution to a height of 64 pixels and centrally cropped to 64× 64 pixels, were extracted. 
  This downsampling was necessary due to GPU limitations. 
  
  We held out 20% of the frames, sampled from throughout our video sequence, as the test set. 
  Each frame was manually labeled for tool presence.
  These annotations were used for evaluation only. 
  In the training set, 65.8% of frames were labeled as containing a tool,
  and in the test set 67.4% of frames were labeled as containing a tool.
  
  
  
   §.§ Variational autoencoder
  
  
  We use variational autoencoders (VAEs) <cit.> to learn low dimensional latent representations that can encode our endoscopic video data. VAEs and their extensions <cit.> are based on the idea that each data point, x∈𝐗, is generated from a d-dimensional latent random variable, z∈ℝ^d
  
  , with probability p_θ (x | z ), where z is sampled from the prior, p_θ(z) ∼𝒩(μ, σ^2), parameterized by θ <cit.>. However, since optimizing over the probability density function (PDF) P is intractable,
  the optimization is solved over a simpler PDF, Q, to find q_ϕ that best approximates p_θ <cit.>. To ensure that q best approximates p, θ and ϕ are jointly optimized by maximizing the evidence lower bound (ELBO) <cit.>:
  
      log p(x) ≥E_q_ϕ(z | x)[log p_θ(x | z)]-KL(q_ϕ(z | x)  p(z)).
  
  
  In order to encourage VAEs to learn more informative encodings without overfitting to the data, Zhao et al. <cit.> introduced the maximum mean discrepancy (MMD) VAE, which maximizes the mutual information between the data and the encodings. 
  
  MMD-VAE changes the objective function by replacing the KL-divergence term with MMD and introduces a regularization term, λ <cit.>:
  
      log p(x) ≥E_q_ϕ(z | x)[log p_θ(x | z)]-λMMD(q_ϕ(z | x)  p(z)).
  
  
  
  
   §.§ Training
  
  
  
  
  We used a convolutional neural network encoder and decoder each with two convolutional
  layers and three fully-connected layers (Fig. <ref>).
  We performed a hyperparameter sweep over latent dimension and regularization coefficient of our MMD-VAE implementation in PyTorch <cit.> and evaluated each model based on the criteria in Section <ref>.
  
  The best performing model from our sweep was trained for 80 epochs using stochastic gradient descent (SGD) on a single
  NVIDIA Quadro K620 GPU with 2GB memory with minibatch size of 32 and Adam optimizer <cit.> with default parameters except for learning rate which was set to 10^-3. This model had a latent dimension d=20 and regularization coefficient λ=5.
  
  
  
   §.§ Model Evaluation
  
  
  
  
  
    §.§.§ Direct Evaluation.
   First, we directly evaluate the encodings produced by our MMD-VAE implementation using a query-based evaluation. We compute the cosines between encodings of each test frame, i, containing a tool and all other test frames, j ≠ i, and threshold the products to separate high and low responses.
  Since the query (i.e., test frame, i) contains a tool, all other test frames containing tools should produce high response, while those without tools should produce low response. 
  To evaluate our results, we compute the average precision (AP) score <cit.> over responses from each test frame. AP summarizes the precision-recall curve by computing the weighted mean of precision values computed at each threshold, weighted by the increase in recall from the previous threshold. In simpler terms, AP computes the area under the precision-recall curve.
  
  
  
    §.§.§ Approximate Inference.
   Next, we evaluate our encodings using approximate inference. We estimate p (tool presence | latent encoding) by modeling the space of encodings as a finite mixture model with a categorical latent variable C which has K=2 Gaussian states (tool present and not present). We use Markov chain Monte Carlo (MCMC) sampling <cit.> to approximate the posterior distribution by constructing a Markov chain whose states are assignments of the model parameters and whose stationary distribution is p: 
  
      p(z_i | θ, μ_c_i, σ_c_i) = ∑_c_i=1^K θ_c_i^⊤𝒩(z_i | μ_c_i, σ_c_i).
  
  Here, z_i∈ℝ^d is the ith encoding generated from our MMD-VAE, d∈ℕ is the dimension of the encoding sample, and each of the K configurations are normally distributed according to parameters μ, σ∈ℝ^K and mixing probabilities θ∈ℝ^d× K. 
  We assume each encoding z_i is generated by θ and a latent state 1≤ c_i ≤ K, described by 𝒩(μ_c_i, σ_c_i).
  By running the Markov chain for B burn-in steps, we reach the stationary distribution, p. 
  
  We then sample the chain for N iterations which form samples from p <cit.>.
  The parameters of the mixture model, μ, σ,  and θ, are learned using the No-U-turn sampler <cit.> on Eq. <ref>.
  Finally, for a fixed z_i, we can estimate the probability that encoding z_i
  comes from latent cluster c_i to predict tool presence: 
  p(c_i | z_i) ∝ p(z_i|c_i)p(c_i) = p(z_i, c_i).
  
  For evaluation, we learned the parameters in PyStan <cit.>
  with four Markov chains with B=N=2500 and default hyperparameters. 
  The posterior probability of each sample belonging to each cluster was then computed and used to predict labels. 
  We evaluate the separation between the two latent states and compute the AP score for our label predictions.
  
  
  
    §.§.§ Future Prediction.
   Finally, we evaluate our encodings by training a future prediction model using a recurrent neural network (RNN) encoder-decoder <cit.> to observe a sequence of past video frame encodings and reconstruct a sequence of future frame encodings <cit.>. The intuition behind this approach is that models capable of future prediction must encode contextually relevant information <cit.>. Both the encoder and decoder have long short-term memory (LSTM) <cit.> architectures to avoid the vanishing gradient problem, and each frame of the future sequence is associated with its own mixture of multivariate Gaussians in order not to blur distinct futures together under a unimodal Gaussian <cit.>.
  
  Our PyTorch <cit.> implementation of the future prediction model was similar to that presented by DiPietro et al. <cit.>. We used 5 frame sequences of past and future encodings, and Adam <cit.> for optimization at a learning rate of 0.005 and other hyperparameters at their default values. The latent dimension was set to 64, the number of Gaussian mixture components to 16, and the model was trained for 1000 epochs with a batch size of 50.
  
  As in direct evaluation, we evaluate the encodings produced by future prediction by computing the cosines between encodings from each test sequence, s_i, containing a tool and all other test sequences, s_j ≠ s_i, taking the maximum per-frame, and thresholding, as before, to separate high and low responses. A per-frame maximum is computed here since each frame belongs to multiple adjacent sequences, s_j, producing multiple responses. Specifically, since we used 5 frame sequences, each frame produces 5 responses, of which we pick the maximum.
  Finally, we compute the AP over responses from each test sequence.
  
  
  
  § RESULTS
  
  
  The results from our three experiments are described in this section. Here, the direct evaluation will be referred to by MMD-VAE, approximate inference method by MMD-VAE + MCMC, and future prediction by MMD-VAE + FP. 
  
  We also compare our unsupervised methods to frame-level tool presence predictions from 4 supervised methods. 
  Twinanda et al. <cit.> use a supervised CNN based on AlexNet <cit.> to perform tool presence detection in a multi-task manner. 
  Sahu et al. <cit.> use a transfer learning approach to combine ImageNet <cit.> features with time-series analysis to detect tool presence.
  Raju et al.<cit.> combine features from GoogleNet <cit.> and VGGNet <cit.> for tool presence detection. 
  Jin et al.<cit.> use a supervised region-based CNN to spatially localize tools and use these detections to drive frame-level tool presence detection. Results are summarized in Table <ref>.
  
  
  
  
  
  
  
  
    §.§.§ Direct Evaluation.
   
  The encodings produced by our implementation of MMD–VAE show some amount of separation (Fig. <ref>a).
  Therefore, we expect our queries to produce high response when evaluated against frames with tools. 
  However, we also expect queries to produce higher response against nearby frames with tools compared to frames that are further away from the queries. This is because our per frame encodings may not be able to disentangle the presence and absence of tools over the varied anatomy present in our video sequences (Fig. <ref>). Direct evaluation achieves an AP of 71.56 in identifying frames with tools.
  
  
  
    §.§.§ Approximate Inference.
   
  Since the encodings from the MMD–VAE show some separation,
  we expect a trained mixture of two Gaussians to capture the separation in the encodings
  and generalize to labeling our held-out test set. We found that the trained mixture model learned two clusters with distinct means and no overlap (Fig. <ref>). 
  By treating each cluster as a binary tool indicator, we can predict labels for the encodings in test set. Compared to the direct evaluation method, we show improvement with an AP of 73.93.  We hypothesize that our two assumptions that (1) the data can be represented as a mixture of two Gaussians, and (2) a sample belonging to a hidden configuration directly indicates tool presence or absence may be too strong and, therefore, limiting the improvement in AP.
  Instead, the clusters likely capture a combination of tool presence and anatomy variance, and precision could be further improved by either relaxing the assumptions or increasing the model complexity.
  
  
  
  
  
    §.§.§ Future Prediction.
  We expect the addition of temporal information to allow for better disentanglement between tool motion from camera motion. We observe this improvement in the slightly greater separation in the encodings produced by MMD-VAE + FP than those produced without FP (Fig. <ref>b). This translates to further improvement in AP at 76.18. 
  We hypothesize that larger gains in AP were again limited due to the short 5 frame sequence of encodings used to train the future prediction network. Using longer sequences may allow detection of tools over larger variations in anatomy and, therefore, improve overall results.
  
  
  
  § CONCLUSION AND FUTURE WORK
  
  
  We showed through our evaluations that it is possible to learn representations of endoscopic videos that allow us to identify surgical tool presence without supervision. 
  We are able to detect frames containing tools directly from MMD–VAE encodings with an AP of 71.56. By performing approximate inference on these encodings, we are able to improve the AP of frame-level tool presence detection to 73.93. Finally, using the MMD–VAE encodings to perform future prediction allows us to further improve our AP to 76.18. 
  
  These results are comparable to those achieved by prior supervised methods evaluated on the M2cai16-tool dataset<cit.>. This dataset consists of 15 videos of cholecystectomy procedure, where each frame is labeled with the presence or absence of seven possible surgical tools in a multi-label fashion. As this work was evaluated on a a different dataset, our immediate next step is to re-evaluate our unsupervised method on the M2cai16-tool dataset. This comparison will not only allow us to better understand how our methods compare against supervised methods, but also allow us to evaluate whether our methods can learn generalized representations across various surgical tools.
  
  Going forward, we will explore whether variations in latent dimension and regularization for training our MMD-VAE can improve our ability to discriminate between frames with and without tools. 
  We will also explore whether classification can be improved by accommodating the variance in anatomy by relaxing the assumption that our encodings are a mixture of two Gaussian states. Another space to explore will be whether larger sequences of video frame encodings allow us to better separate tool motion from camera motion. Although our initial work is on a limited endoscopic video dataset, our results are promising and our method can be easily applied to larger datasets with wider range of tools and anatomy since we do not rely on labels for training. 
  
  The ability to reliably identify frames containing tools can help the annotation process and can also enable further research in many different areas. For instance, methods that rely on endoscopic video frames without tools <cit.> can easily discard frames that are labeled as containing tools. Further, by treating features like optical flow vectors from sequences of frames with and without tools differently, we can work on identifying pixels containing tools without supervision. Unsupervised segmentation of tools, in turn, can enable unsupervised tool tracking and can have great impact on research toward video-based surgical activity recognition and skill assessment.
  
  
  
  
    §.§.§ Acknowledgements
  
  This work was supported by the Johns Hopkins University Provost's Postdoctoral fellowship, NVIDIA GPU grant, and other Johns Hopkins University internal funds.
  We would also like to thank Daniel Malinsky and Robert DiPietro for their invaluable feedback. We would also like to acknowledge the JHU Department of Computer Science providing a research GPU cluster.
  
  splncs04
  
  