title: Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion
authors: Daochang Liu Yuhui Wei Tingting Jiang Yizhou Wang Rulin Miao Fei Shan Ziyu Li
text: 
  
  
  
  
  
  
  Unsupervised Surgical Instrument Segmentation
  
  
  
  
  
  
  
  
  
  
  
  
  
  D. Liu et al.
  
  
  
  
  NELVT, Department of Computer Science, Peking University, Beijing, China Peking University Cancer Hospital, Beijing, China Center on Frontiers of Computing Studies, Peking University, Beijing, China 
   Advanced Institute of Information Technology, Peking University, Hangzhou, China Deepwise AI Lab, Beijing, China
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Unsupervised Surgical Instrument Segmentation via Anchor Generation and Semantic Diffusion
      
  Daochang Liu1,5Equal contribution Yuhui Wei11 Tingting Jiang1 Yizhou Wang1,3,4 Rulin Miao2 Fei Shan2 Ziyu Li2
      December 30, 2023
  ==================================================================================================================
  
  
  
  
  
  Surgical instrument segmentation is a key component in developing context-aware operating rooms.  
  Existing works on this task heavily rely on the supervision of a large amount of labeled data, which involve laborious and expensive human efforts.
  In contrast, a more affordable unsupervised approach is developed in this paper. 
  To train our model, we first generate anchors as pseudo labels for instruments and background tissues respectively by fusing coarse handcrafted cues. 
  Then a semantic diffusion loss is proposed to resolve the ambiguity in the generated anchors via the feature correlation between adjacent video frames.
  In the experiments on the binary instrument segmentation task of the 2017 MICCAI EndoVis Robotic Instrument Segmentation Challenge dataset, the proposed method achieves 0.71 IoU and 0.81 Dice score without using a single manual annotation, which is promising to show the potential of unsupervised learning for surgical tool segmentation.
  
  
  
  
  
   
  
  
  
  
  
  
  § INTRODUCTION
  
  
  Instrument segmentation in minimally invasive surgery is fundamental for various advanced computer-aided intervention techniques such as automatic surgical skill assessment and intra-operative guidance systems <cit.>. 
  Given its importance, surgical instrument segmentation has witnessed remarkable progress from early traditional methods <cit.> to recent approaches using deep learning <cit.>. 
  However, such success is largely built upon supervised learning from a large amount of annotated data, which are very expensive and time-consuming to collect in the medical field, especially for the segmentation task on video data.
  Besides, the generalization ability of supervised methods is almost inevitably hindered by the domain gaps in real-world scenarios across different hospitals and procedure types. 
  
  
  
  
  In the literature, several attempts have been made to handle the lack of manual annotations <cit.>. 
  Image level annotations of tool presence were utilized in <cit.> to train neural networks in a weakly-supervised manner. 
  Jin  <cit.> propagated the ground truth across neighboring video frames using motion flow for semi-supervised learning, while Ross  <cit.> reduced the number of necessary labeled images by employing re-colorization as a pre-training task. Recently, a self-supervised approach was introduced to generate labels using the kinematic signal in the robot-assisted surgery <cit.>.
  Compared to prior works, this study steps further to essentially eliminate the demand for manual annotations or external signals by proposing an unsupervised method for the binary segmentation of surgical tools.
  Unsupervised learning has been successfully investigated in other surgical domains such as surgical workflow analysis <cit.> and surgical motion prediction <cit.>, implying its possibility in instrument segmentation.
  
  
  
  
  Our method, which includes anchor generation and semantic diffusion, learns from the general prior knowledge about surgical tools.
  As for the anchor generation, we present a new perspective to train neural networks in the absence of labeled data, , generating reliable pseudo labels from simple cues.
  A diverse collection of cues, including color, objectness, and location cues, are fused to be positive anchors and negative anchors, which correspond to pixels highly likely to be instruments and backgrounds respectively.
  Although individual cues are coarse and biased, our fusion process leverages the complementary information in these cues and thereby suppresses the noise.
  The segmentation model is then trained based on these anchors.
  However, since the anchors only cover a small portion of image pixels, a semantic diffusion loss is designed to propagate supervisory signals from anchor pixels to remaining pixels that are ambiguous to be instruments or not.
  The core idea of this loss is to exploit the temporal correlation in surgery videos.
  Specifically, adjacent video frames should share similar semantic representations in both instrument and background regions.
  
  
  In the experiments on the EndoVis 2017 dataset <cit.>, the proposed method achieves encouraging results (0.71 IoU and 0.81 Dice) on the binary segmentation task without using a single manual annotation, indicating its potential to reduce cost in clinical applications.
  Our method can also be easily extended to the semi-supervised setting and obtain performance comparable to the state-of-the-art. 
  In addition, experiments on the ISIC 2016 dataset <cit.> demonstrate that the proposed model is inherently flexible to be applied in other domains like skin lesion segmentation.
  In summary, our contributions are three-fold: 1) An unsupervised approach for binary segmentation of surgical tools 2) A training strategy by generating anchor supervision from coarse cues 3) A semantic diffusion loss to explore the inter-frame semantic similarity.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § METHOD
  
  
  As illustrated in Fig. <ref>, our unsupervised framework[By unsupervised, we mean no manual annotation of surgical instruments is used.] consists of two aspects, 1) generating anchors to provide initial training supervision 2) augmenting the supervision by a semantic diffusion loss. 
  Our framework is elaborated as follows.
  
  
  
   §.§ Anchor Generation
  
  
  In conventional supervised methods, human knowledge is passed to the segmentation model through annotating large-scale databases.
  To be free of annotation, we encode the knowledge about surgical instruments into hand-designed cues instead and generate pseudo labels for training.
  The selection of such cues should adhere to two principles, , simplicity and diversity. The simplicity of cues prevents virtually transferring intensive efforts from the data annotation to the design process of cues, while the diversity enriches the valuable information that we can take advantage of in different cues.
  Based on these principles, three cues are computed, including color, objectness and location cues.
  Given a video frame 𝐱_t∈ℝ^HW at time t with height H and width W, probability maps 𝐜_t^color∈[0,1]^HW, 𝐜_t^obj∈[0,1]^HW, 𝐜_t^loc∈[0,1]^HW are extracted according to the three cues respectively, which are fused as pseudo labels later.
  
  Color. 
  Color is an obvious visual characteristic to distinguish instruments from the surrounding backgrounds. 
  Surgical instruments are mostly of grayish and plain colors, while the background tissues tend to be reddish and highly-saturated. 
  Therefore, we multiply the inverted A channel in the LAB space, i.e., one minus the channel, and the inverted S channel in the HSV space to yield the probability map 𝐜_t^color.
  
  Objectness. 
  Another cue can be derived from to what extent the image region is object-like.
  Surgical instruments are often with well-defined boundaries, while the background elements scatter around and fail to form a concrete shape. 
  In detail, the objectness map 𝐜_t^obj is retrieved using a class-agnostic object detector <cit.>.
  Although this detector originally targets at daily scenes, we find it also give rich information in abdominal views.
  
  
  
  Location. 
  The third cue is based on the pixel location of instrument in the screen. 
  Instead of a fixed location prior, an adaptive and video-specific location probability map is obtained by averaging the color maps across the whole video: 𝐜_t^loc = 1/T∑_t𝐜_t^color, where T is the video length.
  The location map roughly highlights the image areas where instruments frequently appear in this video.
  
  Anchor Generation.
  As shown in Fig. <ref>, the resultant cue maps are very coarse and noisy. 
  Therefore, anchors are generated from these cues to suppress the noise.
  Concretely, the positive anchor 𝐚_t^pos∈[0,1]^HW[[0, 1] means values are between 0 and 1 both inclusively.] is defined as the element-wise product of all cues: 𝐚_t^pos = 𝐜_t^color𝐜_t^obj𝐜_t^loc, which captures the confident instrument regions that satisfy all the cues.
  Similarly, the negative anchor 𝐚_t^neg∈[0,1]^HW is defined as the element-wise product of all inverted cues: 𝐚_t^neg = (1 - 𝐜_t^color)(1 - 𝐜_t^obj)(1 - 𝐜_t^loc), which captures the confident background regions that satisfy none of the cues.
  As in Fig. <ref>, the false response is considerably minimized in the generated anchors.
  
  
  Anchor Loss.
  The anchors are then regarded as pseudo labels to train the segmentation network, a vanilla U-Net <cit.> in this paper.
  We propose an anchor loss to encourage network activation on the positive anchor and inhibit activation on the negative anchor:
  
      ℒ_anc(𝐱_t) = 1/HW∑_i - 𝐚^pos_t,i𝐩_t,i - 𝐚^neg_t,i(1 - 𝐩_t,i)
  
  where 𝐩_t∈[0,1]^HW denotes the prediction map from the network and i is the pixel index. 
  The loss is computed for each pixel and averaged over the whole image. 
  Compared to the standard binary cross-entropy, this anchor loss only imposes supervision on the pixels that are confident to be instruments or backgrounds, keeping the network away from being disrupted by the noisy cues.
  However, the anchors only amount to a minority of image pixels. 
  On the remaining ambiguous pixels outside the anchors, the network is not supervised and its behavior is undefined.
  Such a problem is tackled by the following semantic diffusion loss.
  
  
  
   §.§ Semantic Diffusion
  
  Apart from the cues mentioned above, temporal coherence is another natural source of knowledge for unsupervised learning in the sequential data.
  We argue that the instruments in adjacent video frames usually share similar semantics, termed as inter-frame instrument-instrument similarity.
  This temporal similarity is assumed to be stronger than the semantic similarity between the instrument and the background within a single frame, , the intra-frame instrument-background similarity.
  To this end, the semantic feature maps from a pre-trained convolutional neural network (CNN) are first aggregated within the instrument and background regions respectively using the prediction map:
  
      𝐟^fg_t = ∑_i𝐩_t,i𝐅_t,i        𝐟^bg_t = ∑_i (1 - 𝐩_t,i)𝐅_t,i
  
  where 𝐅_t∈ℝ^HW × D represents the CNN feature maps of frame 𝐱_t, and 𝐅_t,i∈ℝ^D denotes the features at pixel i, and D is the channel number, 𝐟^fg_t∈ℝ^D and 𝐟^bg_t∈ℝ^D are the aggregated features for the instrument and the background correspondingly.
  Then given two adjacent frames 𝐱_t and 𝐱_t+1, a semantic diffusion loss in a quadruplet form is proposed to constrain the inter-frame instrument-instrument similarity to be higher than the intra-frame instrument-background similarities by a margin:
  
      ℒ_dif^fg(𝐱_t, 𝐱_t+1) = max(ϕ(𝐟^fg_t, 𝐟^bg_t) + ϕ(𝐟^fg_t+1, 𝐟^bg_t+1) - 2 ϕ(𝐟^fg_t,𝐟^fg_t+1) + m^fg, 0)
  
  where ϕ(·,·) denotes the cosine similarity between two features and m^fg is a hyperparameter controlling the margin. 
  Likewise, another semantic diffusion loss can be formulated to enforce the inter-frame background-background similarity:
  
      ℒ_dif^bg(𝐱_t, 𝐱_t+1) = max(ϕ(𝐟^fg_t, 𝐟^bg_t) + ϕ(𝐟^fg_t+1, 𝐟^bg_t+1) - 2 ϕ(𝐟^bg_t,𝐟^bg_t+1) + m^bg, 0).
  
  Lastly, the anchor loss and the semantic diffusion loss are optimized collectively:
  
      ℒ_full(𝐱_t, 𝐱_t+1) = ℒ_anc(𝐱_t) + ℒ_anc(𝐱_t+1) + ℒ_dif^fg(𝐱_t, 𝐱_t+1) + ℒ_dif^bg(𝐱_t, 𝐱_t+1).
  
  Driven by the semantic diffusion loss, the initial signals on the confident anchor pixels are propagated to remaining ambiguous pixels.
  Our network benefits from such augmented supervision and outputs accurate and complete segmentation.
  Note that the semantic diffusion loss is generally not restricted to adjacent frames and can be also imposed on any image pair exhibiting inter-image similarity.
  
  
  
  § EXPERIMENT
  
  
  Dataset.
  Our method is evaluated on the dataset of the 2017 MICCAI EndoVis Robotic Instrument Segmentation Challenge <cit.> (EndoVis 2017), which consists of 10 abdominal porcine procedures videotaped by the da Vinci Xi systems.
  Our work focuses on the binary instrument segmentation task, where each frame is separated into instruments and backgrounds. 
  As our method is unsupervised, we do not use any annotations during the training process.
  Note that the ground truth of the test set is still held out by the challenge organizer. 
  
  
  Setup.
  Experiments are carried out in two different settings.
  1) Train Test (TT): This setting is common for supervised methods, where the learning and the inference are performed on two different sets of data. 
  In this setting, we follow the previous convention <cit.> and conduct 4-fold cross-validation on the released 8 training videos of EndoVis 2017, with the same splits as prior works.
  Our method can attain real-time online inference speed in this setting.
  2) Single Stage (SS): This is a specific setting for our unsupervised method.
  Since the learning involves no annotation, we can directly place the learning and the inference on the same set of data, , the released training set of EndoVis 2017.
  In application scenarios, the model needs to be re-trained when new unseen data comes, therefore this setting is more suitable for the offline batch analysis. 
  Following previous work <cit.>, we use intersection-over-union (IoU) and Dice coefficient to measure our performance.
  
  
  Implementation Details. 
  We extract the semantic feature maps from the relu5_3 layer of the VGG16 <cit.> pre-trained on ImageNet, which are interpolated to the same size as the prediction map. 
  The VGG16 extractor is frozen when training our U-Net.
  The margin factors m^fg and m^bg are set as 0.2 and 0.8.
  The prediction map is thresholded to be final segmentation mask using the Otsu algorithm <cit.>. Our implementation uses official pre-trained CNN models and parameters in PyTorch <cit.>. Codes will be released to offer all details.
  
  
  
   §.§ Results on EndoVis 2017
  
  
  Results on EndoVis 2017 are reported in Table <ref>. Firstly we assess the network performance only using the anchor loss ℒ_anc based on our cues, where we get the basic performance. 
  After we combine the semantic diffusion losses, especially the background semantic diffusion loss ℒ_dif^bg, the performance is strikingly improved. 
  This result proves our assumption that adjacent video frames are similar to each other in both foreground and background regions. 
  Since the background area is relatively more similar between the video frames, it is seen from Table <ref> that ℒ_dif^bg brings more improvement on the performance than ℒ_dif^fg.
  
  
  
  
  
  
   §.§ The Choice of Cues
  
  
  Different combinations of cues are examined to research their effects on the network performance. 
  Here we run the Otsu thresholding algorithm <cit.> not only on the network prediction map 𝐩 but also on the corresponding positive anchor 𝐚^pos and the inverted negative anchor 1-𝐚^neg to generate segmentation masks.
  The Otsu algorithm is adaptive to the disparate intensity level of the probabilistic maps.
  The resultant masks are then evaluated against the ground truth.
  As the results shown in Table <ref>, the best network performance comes from the combination of all three cues, because more kinds of cues can provide extra information from different aspects.
  Meanwhile, a single kind of cue may produce good results on the anchors, but may not be helpful to the final network prediction, because a single cue may contain lots of noise and it needs to be filtered out by the fusion with other useful cues.
  Also, different kinds of cues have varying effects on the network performance.
  For example, it is noticed that the c^color and c^obj cues are more important than the c^loc from the table.
  
  
  
  
  
   §.§ Compared to Supervised Methods
  
  
  At present, unsupervised instrument segmentation is still less explored, with few methods that can be directly compared with.
  Therefore, to provide an indirect reference, our method is adjusted to the semi-supervised and fully-supervised settings and compared with previous supervised methods in Table <ref>.
  When fully-supervised, we substitute the anchors by the ground truth on all the frames. 
  Since our contribution is not in the network architecture and we do not use special modules such as attention beyond the U-Net, our fully-supervised performance is close to some earlier works, which can be thought of as an upper bound of our unsupervised solution.
  When semi-supervised, the anchors are replaced with the ground truth on 50% frames in the same periodical way as in <cit.>.
  Our method has competitive performance with the state-of-the-art in the semi-supervised setting. 
  Lastly, in the last two rows without using any annotation, we achieve the preeminent performance.
  More data is exploited for learning in the setting SS than in the setting TT, which explains why the setting SS has better results. 
  
  
  
  
  
   §.§ Qualitative Result
  
  
  In this section, some visual results from our method are plotted. 
  Firstly, as seen in Fig. <ref>, the three cues are very coarse, , the background can still be found on the color cue maps.
  By the fusion of noisy cues, the anchors become purer, which are nonetheless very sparse. 
  Then via the semantic diffusion loss, which augments the signals on the anchors, the network can find the lost instrument region in those anchor pictures, as shown in the success cases in Fig. <ref>a and Fig. <ref>b.
  Although in some pictures there are difficulties such as complicated scenes and lighting inconstancy, we can also get good performance in these cases. 
  However, there are still some failure cases, such as the special probe (Fig. <ref>e) that is not thought of as the instrument in the ground truth. 
  Also, the dimmed light and the dark organ (Fig. <ref>d) can also have negative effects on the reliability of cues.
  A video demo is attached in the supplementary material.
  
  
  
  
  
   §.§ Extension to Other Domain
  
  
  An exploratory experiment is conducted on the skin lesion segmentation task of ISIC 2016 benchmark <cit.> to inspect whether our model can be migrated to other domains.
  We conform to the official train-test split.
  Due to the dramatic color variations of lesions, the color cue is excluded.
  The location cue is set as a fixed 2D Gaussian center prior since ISIC is not a video dataset.
  In view of the background similarity shared by most images, we sample random image pairs for semantic diffusion.
  Our flexibility is provisionally supported by the results in Table <ref>.
  Specific cues for skin lesions can be designed in future for better results.
  
  
  
  
  
  
  
  § CONCLUSION AND FUTURE WORK
  
  This work proposes an unsupervised surgical instrument segmentation method via anchor generation and semantic diffusion, whose efficacy and flexibility are validated by empirical results.
  The current framework is still limited to binary segmentation. 
  In future works, multiple class-specific anchors could be generated for multi-class segmentation, while additional grouping strategies could be incorporated as post-processing to support instance or part segmentation.
  
  
  
  
   §.§ Acknowledgments
   This work was partially supported by MOST-2018AAA0102004 and the Natural Science Foundation of China under contracts 61572042, 61527804, 61625201. We also acknowledge the Clinical Medicine Plus X-Young Scholars Project, and High-Performance Computing Platform of Peking University for providing computational resources.
  Thank Boshuo Wang for making the video demo.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  splncs04
  
  
  
  