author: Jinbin Huang, Aditi Mishra, Bum-Chul Kwon, Chris Bryan
bibliography: template.bib
title: ConceptExplainer: Interactive Explanation for Deep Neural Networks from a Concept Perspective

Deep learning has led to tremendous advances across a number of fields, including natural language processing, speech recognition, medical applications, computer vision, and autonomous vehicles . Unfortunately, the complexity and inherent opaqueness of neural networks imposes significant difficulty in understanding the behavior and inner workings of models . As deep learning is increasingly employed in today's society, it is important that people (including those who are non-experts in AI) are able to understand and interpret the predictions made by AI models  — the explainable AI (XAI) subfield supports the development of tools and techniques to support this process .

Generally, interpretability of trained neural networks is achieved either by (1) revealing the neural architecture, and how signals flow through the pipeline, or (2) using explainable substitutions to approximate the neural network's mental model on a task . While the first option is straightforward (i.e., making transparent the inner logic and algorithmic functions of the model) it requires users that are equipped with deep learning expertise, and is thus inappropriate as a paradigm for non-experts . In contrast, the latter option fits users with little deep learning expertise, but it must be carefully employed as it comes with inherent information loss and potentially false reflection . One interpretability approach that is suitable for non-experts is to demonstrate how input perturbation affects model outputs. In image classification, saliency and activation maps  measure the importance of the input features (i.e., image pixels) when classifying an image. The drawback is that these approaches only provide *local* or sample-level explanations for individual images. Cross-grained insights which can help users understand *class* or *global*-level model behaviors — e.g., measuring the influence that "ear" and "fur" have on predicting a classification of a "cat" — is not possible . Such understanding can be important, however, for non-expert users in the interpretability process, as it allows them to compare their own mental models of how classification should work to the neural network's mental model . To facilitate such explanations, concept-based methods have recently been introduced by the AI community  . For image classification, a concept refers to a group of pixels in a sample that represents an important part of the class object within the image . To be considered as a concept for a class, the concept needs to be easily understandable to human users, consistent within each concept but separable from other concepts, and necessary for the prediction of the class. Importantly, concepts can be used to generate explanations at the different levels of granularity. At a high level, we can compute the concepts that are influential to classes (e.g., "ear", "fur", "tail" are likely important to the "cat" class). Locally (for an instance in a class), we can measure how much each concept influences the correct or incorrect classification of the class. When concepts and classes are aggregated together, there is the potential to form a global perspective of the neural network's mental model, to understand the large-scale behavior based on what the network has learned. Although concepts are appropriate for use as an interpretability technique for non-experts, there has been little broad application of this technique. This is likely because of two shared complexities that make concept generation and presentation a difficult exercise: (1) The "normal" procedure to create concepts is by hand (i.e., manually). To automatically generate the massive concept space that covers an entire neural network is a non-trivial exercise. (2) Structuring, navigating, and probing such a concept space in a way that supports different (global and local) explanation granularities is likewise non-trivial. To address these two challenges, we propose ConceptExplainer. ConceptExplainer consists of two primary software components: (1) a backend pipeline interactively (and automatically) generates and structures a concept space for a neural network, and (2) a coordinated, visualization-driven frontend lets users interactively explore and probe the concept space. **ConceptExplainer represents the first interactive visualization system designed for non-expert users that supports concept-based explanations of deep learning models at different levels of granularity.** At the global level, ConceptExplainer can reveal which concepts are broadly influencing the neural network's decision making. At the class level, users can visualize which concepts are influential for that class, and can explore the overlaps of influential concepts which are shared between classes. At the instance level, individual images can be reviewed to see how present/non-present concepts affect the model's correct/incorrect predictions. In developing and evaluating ConceptExplainer, we make the following research contributions: **(1) Identifying challenges and goals.** Based on an analysis of challenges in XAI for non-experts, we identify salient **design challenges** and **design goals** which are important for concept-based explanation for non-expert users, particularly in an interactive visualization context. **(2) A novel visual analytics tool for concept-based explanation.** To support the ideated challenges and goals, we design and implement ConceptExplainer, an interactive system that automatically generates a concept space for an image classification neural network, and supports non-expert users to explore and probe concept-based explanations at multiple levels of granularity. ConceptExplainer is designed based on an iterative prototyping process, formatively validated through multiple evaluations (usage scenarios and a user study), and the codebase is open-sourced to promote adoption and replication. **(3) Empirical findings and generalizable takeaways.** Based on the process of designing, implementing, and validating ConceptExplainer, we discuss several lessons and takeaways on the topic of how visualization-driven concept-based explanation can support XAI tasks, such as identifying issues (or biases) in data samples, and how tools like ConceptExplainer can be extended in the future (such as supporting expert users).

# Related Work

## Deep Learning Interpretability using Concepts

There are two primary approaches for using concepts to improve deep learning interpretability: (1) training inherently interpretable models with concept-based constraints. (2) constructing post-hoc explanations based on concepts.

Regarding the first approach, Koh et al.  proposed Concept Bottleneck Model (CBM), which restricts neural networks to behave in an auto-encoder manner where they first map inputs to human-interpretable concepts and then predict based on those concepts. Chen et al.   proposed Concept Whitening (CW) layer as a substitution for the normalization layer (i.e., batch normalization) to achieve higher interpretablity. While these methods employ concepts to offer embedded interpretablity with no extra dependency, they do not provide ways to gain insights into a trained model. Our work differs from this line of research as we seek to interpret fixed models with no alteration.

For the second approach, TCAV  provides a means to computationally define a concept and calculate its class-specific influence on model predictions (called the TCAV score). Building upon TCAV, ACE  is a technique to automatically extract influential class-specific concepts from a dataset. For a formal description of ACE and TCAV, see Sect. , where we describe how both techniques are incorporated into ConceptExplainer's backend to automatically define and extract concepts.

Similarly, Ge at al.  proposed the Visual Reasoning Explanation (VRX) framework that answers interpretability questions such as *Why?* and *Why not?*   from a concept perspective by extracting and organizing class-specific concepts using trained structural concept graphs (SCGs) based on pairwise concept relationships. Though VRX provides insightful instance analysis, it requires training an SCG for each class. This makes VRX difficult to implement within interactive scenarios (i.e., as a part of a user interface) due to computational cost.

## Visual Analytics in Concept-based Interpretability

A large amount of neural network interpretability research in the visualization community has utilized (non-concept-based) conventional methods, including visualizing features, saliency and activation maps, and visualizing the model neurons or architectures (e.g., ); for more information, there are several recent surveys that discuss XAI from a visualization perspective . For research that employs visualization to support concept-based tasks, there are two recent systems that are highly relevant to ConceptExplainer:

ConceptExtract  utilizes visualization within a human-in-the-loop pipeline for concept extraction and fine-tuning. The system generates initial concept segments using ACE; the user then interactively refines the set of concept images. The user can also participate in refining neural network-based binary concept classifiers by interactively supplying labels. The intent of the system is to counter the potential drawbacks of automatic concept extraction, such as human incomprehensible patches, by incorporating human oversight and manual labeling, with the ultimate goal of enabling efficient handcrafting of high-quality concepts.

NeuroCartography  aims to interactively reveal neuron-concept relations by grouping a model's similarly-activated neurons in the same latent layer by the same set of data instances. The corresponding set of data instances is viewed as a set of concepts. Concepts discovered in this way are layer-dependent. They evolve as layers go deeper. The system is thus capable of letting users analyze concept evolution in the context of the neural architecture, which is potentially beneficial to expert users (e.g., model developers) in debugging.

While these two systems combine concept-based explanations and visual analytics, both differ from ConceptExplainer in two important ways: (1) They are designed for expert users; in contrast, we focus on AI non-experts. (2) They support different tasks (creating/refining concepts, and revealing concept/neuron relations); our system is designed to probe and explore concepts as a way to understand model behavior. To the best of our knowledge, ConceptExplainer is the first interactive visual analytics system designed for non-experts that employs concept as a means of explanation.

## Deep Learning Interpretability for Non-Expert Users

The popularity of deep learning in today's society has led to increased demands for XAI techniques that are friendly to non-expert users  . Desiderata for this type of explanation includes faithfulness, consistency with prior beliefs and generality . Liao et al.   proposed a set of design principles for end users, targeting explanation system in form of a question bank, where global explanations for a model's decision making and local explanations for a particular decision on an instance were ranked as top interests. Similarly, Hohman et al.  , in their categorization of end users (which include model developers/builders, model users, and non-experts) suggest that model users who apply deep learning models to their domain tasks mostly need tools that support exploration of model behaviors on local neighborhoods and at global scale. Concept-based explanations are a good fit for these needs, as they require minimal prior knowledge about neural networks (e.g., it is not necessary to know technical concepts like backpropagation) while remaining intuitive and accessible. ConceptExplainer supports probing and exploring of concepts at multiple levels to support interpretability tasks for non-experts.

# Design Challenges and Goals

Similar to prior visualization design studies in XAI  , we identify a salient set of design challenges (C1–C4) which are important for concept-based explanation for non-expert users, based on reviewing of state-of-the-art publications that discuss challenges in XAI, concept-based explanations, and AI explanations for non-experts (see Section ). We distill these into a set of four design goals (G1–G4), which we use to guide ConceptExplainer's development.

## Design Challenges

**(C1) Extracting human-understandable concepts for classes.** There is increasing evidence that neural networks predict based on a combination of the concepts present in instances.   It is helpful for the user to understand neural network behavior by demonstrating what concepts a neural network relies on for predictions and the extent of concept influence on predictions. For example, when the neural network classifies a *tiger*, is the "bushy background" concept more influential or the "tiger stripe"? By extracting influential concepts for each class, the user can intuitively understand how the network understands each class and verify if it works in a sensible way. However, discovering concepts and measuring concept influences can be difficult as noise is inevitable and there are different ways to measure influence . In our work, we adopt the ACE method for concept extraction and TCAV method for measuring concept influence, as they are the most widespread approaches for concept-based explanations. We also propose a concept clustering process after concept discovery to structure the concept space and facilitate concept navigation.

**(C2) Multi-scale concept visualization for large datasets.** System scalability has gained increasing attention in the visualization community. The issue becomes salient when visual interfaces for XAI are concerned because models to be explained are trained on large datasets. Concept explanations also face this issue because the number of concepts discovered grows linearly with the number of classes in the dataset. To demonstrate the concept space and enable users to gain insights from navigating the space, we need more than a naive navigation mechanism which can cause the user to be lost in the deluge of concept information without a clear navigational goal. Apart from that, the data structure of concepts – quantitative influences and images – requires specifically designed visualizations to present.

**(C3) Revealing conceptual overlap between classes.** The conceptual overlap (i.e., common influential features) between classes can reveal to the user why one class might be easily misclassified as another (e.g., Samoyeds classified as wolfs due to the influence of "snowy background" concept). Among a group of classes, understanding what concepts are shared (or not shared) can help discover features that cause confusion  and/or serve as the unique "signature" of a class in the neural network's mental model. For example, Fig. , shows a network has learned a "white fur" concept from both the *white wolf* and *Samoyed* classes, which provides an idea about the commonalities the network sees between these classes. In contrast, the lack of overlap between *fire engine* and *white wolf* also suggests that the network discriminates animals from vehicles.

**(C4) Balancing global explanations and local explanations.** Global explanations without much detail are generally easier to understand. However, since global explanations are a summary of local behavior solely revealing global explanations might lead to unfaithful interpretations. To ensure faithfulness of the explanations we need to show information at both global and local levels and integrate the analysis process at both levels in an organic workflow . The user benefits from having a holistic view of neural network behavior at various levels (see Fig.  ) in a consistent concept-based framework so they can (1) recognize contradictory explanations (2) verify their hypothesis at different levels.

## Design Goals

Based on the design challenges C1–C4, we identify four design goals for the ConceptExplainer system to support interactive concept-based explanations of neural networks' behavior for non-experts. Roughly, these goals can be ordered in terms of their granularity: at the global, class, and instance levels of analysis and explanation.

**(G1) Navigating the global concept space.** We aim to facilitate navigation in the concept space of a large dataset (in our case, we use ImageNet, which contains 1.2 M images for 1000 classes, see Sect. ) (C2). The methodology we use is transferable to other image datasets as long as a trained deep learning model is available. To discover the concept space while preventing noise (C1), we first use ACE algorithm to extract influential concepts from the dataset. To facilitate structured concept navigation we cluster the extracted concepts into multiple concept clusters. The clustering of concepts makes the navigation easier – the user can start their navigation by peeking at various concept clusters to pick up the one they want to explore and get into the cluster to see concepts inside. By gradually understanding the concept clusters the user can understand the concept space. The transition from concept cluster to individual concept reinforces an overview-first-detail-on-demand workflow. To help the user keep track of their investigation process in the concept space and understand the concept clusters by labeling them (taking notes of them) in a user-driven way, annotation functionality should also be available in the system.

**(G2) Supporting intra-class concept analysis** For a class, the discovered, influential concepts can be represented as a collection of image patches that have similar high dimensional activations. These influential concepts intuitively reveal how the model understands a class. The system needs to present these concepts such that the user can understand what they are and the extent of their influence (C2, C3). Users can even inspect the most influential concepts to see if the network has learned sensible relations (e.g., noisy concepts should be less influential than relevant concepts).

**(G3) Supporting inter-class concept analysis.** Understanding conceptual commonalities between classes can help users to understand how the network perceives similar/different classes and pinpoint concept-based root causes for misclassifications. To this end, we seek to demonstrate the links between similar concepts of different classes (C3). It is necessary that inter-class concept analysis go beyond "comparing two classes," as several classes may share commonalities of interest and the user may want to understand how the neural network considers them comprehensively.

**(G4) Supporting instance analysis.** Only explaining neural network behavior from a global or class level might lead the user to overlook important details, resulting in misinterpretations of how the network should work. To enable detailed inspection, while not overwhelming by revealing too many details, concept influences can be measured for individual instances. Such options keep the analytical framework unified: the user does not need other tools for instance explanations/analysis. In addition, instance-level analysis also helps users investigate edge cases ("*Why did the model misclassify this image?*") by listing influential concepts of the image.

# System Design

Based on the design goals G1–G4, we develop ConceptExplainer, an interactive system that provides non-expert users with the ability to probe and analyze concept-based explanations at both the global and local levels. As shown in Fig. , the system consists of an integrated backend and frontend. In the **backend**, we design a processing pipeline that leverages concept-extraction methods (specifically, TCAV and ACE) to automatically extract and organize concepts for a given image dataset. Extracted concepts are organized into meaningful clusters to facilitate interactive concept navigation and concept overlap inspection. The **frontend** interface (Fig. ) consists of four coordinated panels: (1) the header (M) contains controls widgets, including for manipulating the backend processing (setting parameters for the TCAV, ACE, and clustering methods), (2) the left panel (A – C) provides overview, navigation and analysis for classes, (3) the right panel (D – F) provides a clutter-free locality-retaining overview for the concept space and navigation mechanisms reinforcing the overview-first-detail-on-demand mantra. (4) the central panel (G – L) provides support for class/instance level analysis – an overview of class-specific concepts and inter-class concept links combined with a detailed concept inspection view account for both class and instance level model behaviors.

## Iterative Prototyping Process

To facilitate our design process, we employed an iterative prototyping methodology . Over an approximately six months period, we sketched and prototyped a number of user interface and interaction designs. Designs were holistically reviewed and discussed by the project team, with additional feedback solicited from XAI researchers who work on interpretability techniques for non-expert users. Designs deemed suboptimal were either discarded or refactored (Fig.  in the Appendix shows two examples of "old" interfaces), while well-received designs were iteratively refined to develop the current system version. As an example of this process' impact, we adopted a three-panel layout in the interface (as shown in Fig.  and described in Sect. ). The left **class navigation panel** supports class-based analysis and navigation; in parallel, the right **concept-navigation panel** supports concept-based analysis and navigation. Within each of these panels, users can transit fluently from global perspectives of the model (i.e., visualizing all classes/concepts together) down to analyzing individual instances. Classes, concepts, and instances are synthesized together in the center **class-concept panel**.

## Dataset and Model

To demonstrate a real-world application throughout the rest of this paper, we use the ImageNet dataset , which is a well-known image dataset consisting of 1.2M training instances. As a model, we employ GoogleNet , a convolutional neural network consisting of 22 layers. GoogleNet provides a pre-trained model for ImageNet, which classifies images into 1,000 object categories. Thus, our task is to use ConceptExplainer to explain GoogleNet's behavior in the context of instances and classes for the ImageNet dataset.

## Backend: Concept Extraction and Clustering

We use a combination of TCAV  and ACE  as the methodological backbone for concept-based explanation. Combining these methods together lets us automatically derive influential concepts for each class predicted by GoogleNet and the extent of the class' influences. By subsequently clustering the ensemble of derived concepts, we impose a human-understandable structure on the concepts that approximates the model's mental model in classification.

### TCAV

TCAV, or *testing with concept activation vectors*, is a concept-based explanation method that measures the importance of human understandable concepts to the neural network's inference — e.g., how much a "stripe" concept affects a classifier when predicting a zebra image. The TCAV approach views concepts as a set of images of similar traits. A set of images without these traits forms a set of counterexamples (i.e,. non-concept examples).

TCAV defines a concept activation vector, or CAV, as the normal to the hyperplane that linearly separates high-dimensional representations of concept examples from that of non-concept examples. To measure the influence of a concept to an instance's prediction, TCAV computes pixel gradients of the instance at a target layer and compares its direction with the CAV's (both are in the same latent space) by taking inner product of the two. If the gradient vector lines up with the CAV (indicated by a positive inner product) it means the concept is positively influencing the prediction ("stripe" makes an instance more likely to be a zebra). Conversely, if the two vectors part (indicated by negative inner product), the concept negatively influences the prediction.

The TCAV score, which measures conceptual influence, is defined as: $TCAV_{Q_{C,K,L}} = \frac{|\{x \in X_k : S_{C,k,l}(x) > 0\}|}{|X_k|}$ where the fraction of *k*-class inputs whose *l*-layer activation vector was positively influenced by concept *C*. $S_{C,k,l}(x)$, is the aforementioned innerproduct of pixel gradients and CAVs. To prevent meaningless concepts (concepts with influence $\sim$`<!-- -->`{=html}0.5), multiple CAVs can be trained using the same set of concept examples against various sets of counter examples. A meaningful concept should lead to TCAV scores that rejects the hypothesis of a 0.5 TCAV score with statistical significance (we use a threshold of $p>0.01$).

In our system, we train 20 CAVs for each concept on concept examples (this parameter can be edited via ConceptExplainer's frontend) and randomly formed counter examples after it is generated using ACE method. TCAV scores are computed for each CAV and averaged.

### ACE

TCAV provides a mechanism for measuring the influence of human-understandable concepts on the neural network for a specific class, but in the base TCAV approach, forming a set of examples for a concept is a manual process. To automate the process of concept generation and extraction, Ghorbani et al.  proposed ACE, or *automatic concept-based explanations*.

ACE extracts a set of concepts for each class the model predicts on, by segmenting the images of the class and grouping segments that form clusters in the high-dimensional space of specific latent layer; the TCAV procedure can be followed to filter out meaningless collections. This combined TCAV-plus-ACE process makes efficient and automated concept generation tractable. In our system, we default to sampling 50 images for each class, and segment them using SLIC  with three resolutions (15, 50, and 80 segments) per image. The segments are embedded in the network's "mixed4c" layer. $K$-means clustering is then performed to cluster their latent representations into 10 concepts (high-dimensional clusters), though each of these parameters is controllable from ConceptExplainer's frontend.

### Concept Clustering

Running ACE will extract a large ensemble of concepts. For example, we extracted 1,211 concepts from 143 randomly selected classes in ImageNet using the default parameters mentioned above. This concept space is unorganized (i.e., there's no ranking or structure imposed on them), which makes it difficult to review, explore, and compare them.

An additional consideration (or complexity) is that concepts discovered in different classes might be semantically similar (e.g., "snow background" in husky and "snowy background" in wolf); such similarity needs to be demonstrated because it is useful for understanding the model from a concept perspective.

To enable structured and guided navigation within this the concepts space (G1), we apply clustering to the extracted concepts. The intent is to group semantically similar concepts into the same cluster, which will let us leverage cluster identity as a means to demonstrate conceptual overlap across classes, and to form a meaningful navigation structure that the user can rely on to probe and explore the concept space.

The similarity between two concepts is quantified by their cluster identity (if two concepts come from the same cluster their similarity is 1, otherwise 0). In the frontend, users can choose between $k$-means and agglomerative clustering, and select associated cluster parameters. Based on feedback from our user study evaluation (see Sect. ), this provides good flexibility by allowing users to explore different clustering levels and granularities, though the system is extensible to other clustering methods (i.e., as future work) and heuristics. In particular, clustering itself can benefit from the use of XAI techniques, though exploring such activities is beyond the scope of the current paper.

## Frontend: ConceptExplainer Interface

Fig.  shows the ConceptExplainer interface, which consists of four primary panels and several coordinated views (A–N).

### Class Navigation Panel

The leftmost **class navigation panel** provides three visualizations (A–C) to facilitate navigation and selection of classes of interest for concept exploration. This panel primarily supports exploring the model from a class-based perspective, and allows users to analyze the concept space at a global level (G1), comparatively analyze classes (G2 and G3), and inspect specific instances within a class (G4).

\(A\) The **class performance view** summarizes model performance. Users can begin exploration by brushing histogram bins to filter out undesired classes (e.g., show only low performance classes).

\(B\) Such brushing updates the **class navigation view**. This view provides a global perspective of all classes using t-SNE , which maps a high dimensional representation of classes (computed as the average of latent vectors of all images of the same class) into a two dimensional embedding. To prevent overplotting, we aggregate classes into "clique" circles, sized by the number of total classes and colored based on average accuracy of classes in the clique. Hovering on a circle displays a tooltip (see (O)) showing individual classes belonging to the clique with with a representative image per class. Users can select a class of interest for subsequent analysis.

\(C\) Selected classes are displayed in the **class confusion matrix view**. The confusion matrix supports inter-class and intra-class analysis by summarizing the classification performance against ground truth labels across the selected classes. Clicking on a cell shows a list of images that belong to the corresponding category of the cell below the confusion matrix. For example, Fig. (C) shows the 11 instances of *tiger cat* which were misclassified as *tiger*.

Hovering over an instance shows the concepts that affect the instance's prediction. For example, in Fig. (J) (part of usage scenario \#3), a *police van* image was misclassified as *ambulance*. Hovering on the instance shows how influential the concepts of these two classes affect the prediction. In the usage scenario, the *ambulance* concepts have slightly higher, influence scores than *police van* concepts, which explains why the model misclassified the image as *ambulance*.

### Concept Navigation Panel

The **concept navigation panel** on the right side of the interface contains three views that facilitate navigation of the concept space in a structured way (G1) while providing context during class-concept and instance analysis (G2–G4).

\(D\) The entire concept space is displayed in a hexagon plot in the **concept navigation view**. The design of this view underwent multiple iterations during the iterative prototyping process. For example, Fig. (top) in the Appendix shows an early visualization design of the concept space, based on dimensionality reduction where each concept was plotted as a point. This design had several limitations, including overplotting and difficulty in distinguishing and selecting individual concept clusters.

The hexagon design avoids these issues. Instead of plotting a point cloud, we use the IsoMatch method  to plot organize concepts within a hexagonal grid layout like a heatmap. Each concept is visualized as a hex tile, assigned to a unique location such that (a) clutter and overlap is avoided, (b) cluster boundaries are easily demarcated using dark borders, (c) concepts are evenly distributed across the panel's available space, and (d) concept clusters are easily selectable.

\(E\) Clicking on a cluster loads it into the **concept cluster detail view**. This chart summarizes the concepts belonging to a concept cluster. Concepts are ordered along the x-axis by their influence scores, which are mapped to the y-axis. Circles are colored using a diverging blue-to-red color scale based on positive (above 0.5) or negative (less than 0.5) influence score; the influence score mid-point (0.5) is shown as a dotted line, and the average influence score over all concepts in the cluster is shown as a solid horizontal line.

\(F\) Hovering or clicking on a concept loads it into the **individual concept view**, which loads all image patches for that concept. By skimming through these, the user can quickly gain a semantic understanding of what that concept is. Skimming multiple concepts in this way can help show why they belong to the same concept cluster. As an additional feature for this view, we provide users with an annotation feature to input descriptive information about a concept cluster. This information is saved for future reference.

### Class-Concept Panel

The central **class-concept panel** synthesizes information from the left (class-focused) and right (concept-focused) panels, supporting interpretation to the underlying model's behavior on the given data at inter-class, intra-class and instance levels (G2–G4). It consists of three views, organized around a workflow going from inter-class comparison, to intra-class comparison, to instance analysis.

\(G\) The **concept card view** provides an overview of selected classes and concepts (e.g., selecting a set of classes from the class navigation view). Each class is represented by a card, with the class name at top. Below each class title, a histogram shows the distribution of influence scores for the concepts in that class. If the histogram skews to one side, it means the concepts are likewise skewed either positively or negatively influential. In contrast, a normal distribution indicates the majority of concepts in that class are closer to neutral (i.e., close to the 0.5 influence mid-point).

Below the histograms, a detailed dot plot shows individual concepts as circles. Circles are organized into rows corresponding to their concept clusters (labeled at left, e.g., the first row "CC9" stands for "concept cluster \#9"). Circles located on the same row belong to the same concept cluster (i.e., they would be in the same cluster in the concept navigation view in (D)), even when they are distributed among different classes. The left-to-right position of a circle (along with its corresponding red-to-blue color scale) within its class card is based on its influence score along the negative-to-neutral-to-positive scale.

Concepts in the same row are also linked via horizontal lines to emphasize their concept cluster grouping (supporting inter-class analysis, G3). This design is inspired by Lex et al.'s work . Clicking or hovering on these links highlights the corresponding concept cluster in the hexagon plot in (D), and loads the specific concept cluster in the detail view in (E).

To support additional concept analysis, we adopted the idea of periphery plots  to supply condensed insights to the left and right of the class cards. (I) The box plots to the left summarize the influence scores for the concepts for each row (i.e., for each concept cluster). (H) To the right, the bar chart shows the number of concepts belonging to that concept cluster, based on the current selection of classes. A higher frequency number indicates that this concept cluster is likely a common characteristic of these classes in the model's mental map.

\(J\) Clicking the title bar of a class card loads the **class-specific concept view**. This view facilitate intra-class concept inspection (G2). Each row shows one concept, sorted based on decreasing influence score (i.e., the top concept rows are more positively influential to the prediction of the class). Users can review and verify if the model is making inferences about a class based on a set of reasonable concepts. For example, if the user sees a concept relating to an image's background, as opposed to important semantic features (e.g., for a *zebra*, "grassland" as opposed to "stripes"), it may indicate that the neural network is assigning too much weight to the background of the image while ignoring foreground information that might reasonably be important (at least from a human's perspective).

In each row, we present five representative image patches of the corresponding concept so that the user can form a rough idea of what the concept is about. Additional samples can be loaded by clicking the "+" icons beside each label.

\(K\) To the right of the class-specific concept view, the **instance influence matrix** visualizes local explanations about the models behavior for individual instances. This heatmap plots each row as a concept (aligned with the concepts to the left in (J)), where each column indicates an image sample in the dataset. Thus, each cell indicates, for a particular instance, how influential that concept was in its prediction, colored using the same red-to-blue color scale from previous views. Above each column, a check mark or X indicates if the prediction was correct.

Instances are sorted based on their classification accuracy and confidence (i.e., the left-most column is the correctly classified instance with the highest confidence and the right-most column the misclassified instance with the highest confidence). In Fig. , the user has scrolled all the way to right, showing incorrect predictions with the highest confidence.

\(L\) Clicking on a column loads its instance into the **instance view**. This card provides details about the instance (G4), including its image with concept patches that can be toggled on as overlaid semi-transparent boundaries, its ground truth labeling, and the model's prediction and confidence. At the bottom of the card, a lollipop chart shows the concept influences specific to that particular instance.

### Header Bar Panel

\(M\) The **header bar** provides control widgets allowing users to update parameters and re-run the backend pipeline. The menu includes selecting a dataset and model, selecting the model layers where the concepts are extracted, setting TCAV and ACE parameters, specifying the concept clustering methods and number of clusters (i.e., the parameter $k$ in $k$-means). The number of clusters is set by default to the optimal value based on the silhouette score . Users can hover on other numbers of clusters to see their scores as shown in (M).

# Usage Scenarios

To help demonstrate how ConceptExplainer supports concept-based explanation to learn about the model behavior, we present three usage scenarios exploring GoogleNet on the ImageNet dataset. Each focuses on a different type of exploration and analysis: (#1) for individual classes and their instances, (#2) comparing the concepts between two classes, and (#3) understanding concept commonalities across many classes. We tell these stories from the perspective of Michael, a non-expert in AI.

## Usage Scenario \#1: Verifying GoogleNet's knowledge about individual classes and samples

As Fig. (A) shows, Michael begins by reviewing clusters in the concept navigation view, focusing on clusters with darker colors (i.e, containing more influential concepts). Loading Cluster \#13 in the concept cluster detail view, he realizes the two most-influential concepts (from the *zebra* and *lionfish* classes) look like stripes. He loads the *zebra* and *lionfish* classes into the central class-specific concepts panel for subsequent investigation.

In the class-specific concepts panel, Michael sees several cross-class links between the *lionfish* and *zebra* classes, which further confirms that conceptual overlap exists (Fig. (B)). Reviewing these, he sees that the shared concepts primarily relate to stripe patterns. As both zebras and lionfish have prominent stripe patterns, GoogleNet has learned (or formed a mental model) that stripe patterns are a way to recognize both zebras and lionfish.

Michael next inspects the individual concepts of the *zebra* and *lionfish* classes (Fig. (C)). For the *zebra* class, the three most positively influential concepts pertain to stripe patterns. As influence decreases, the concepts become more associated with noise and background features, such as representing desert or grassland. In this case, GoogleNet is aligning with Michael's mental model, as background information is largely irrelevant to classifying these images as zebras.

Reviewing individual samples using the influence instance matrix and the instance view, Michael notices a misclassified image of *zebra* predicted as the *llama* (Fig. (D)). The zebras in this image are very small, and "background concepts" (such as grasses and sky) are the main concepts present. In other words, the model predicted using less-influential concepts that were present, as stripe concepts were not found. This happens several times in instances predicted as *zebra* or *lionfish*, and the model regularly predicts wrongly in such cases.

## Usage Scenario \#2: Spotting unreasonable concepts and data quality issues

Using the class navigation view, Michael looks at other classes similar to *zebra*. Hovering on the circle containing the *zebra* class displays a tooltip, which indicates the *tiger* class (another animal with prominent stripes) is also contained in this clique (Fig (O)). An adjacent clique in this view is colored in red (indicating below-average classification accuracy); the tooltip shows that this clique contains a *tiger cat* and *tabby* class. Intrigued, he selects the *tiger*, *tiger cat*, and *tabby* classes.

The class confusion matrix (Fig. (C)) view shows many misclassifications between *tiger cat* and *tiger*; for example, there are 11 images of *tiger cat* (ground truth) that were predicted as *tiger* by GoogleNet. Selecting this cell loads these images; to Michael's eyes, they all indeed appear to be *tigers*, not *tiger cats*. Already, Michael can see that there is a data labeling issue in ImageNet: instances that should have been labeled as *tigers* were wrongly labeled as *tiger cats*.

Most *tiger cat* concepts appear to have negative influences in concept card view (Fig. (G)). Michael reviews these in the class-specific concept view (Fig. (J)).

He quickly scans through the concepts, and unsurprisingly, many of them seemed to be random or incomprehensible patches that do not form a collective theme. In other words, GoogleNet did not form a sensible mental model about *tiger cat*. Interestingly, the most negatively influential concept at the bottom row presents a "tiger stripe" pattern, which helps explain why there are so many *tiger cats* misclassified as *tigers*.

Because of this, Michael decides to review *tiger cat* instances that contain the "stripe" concept using the instance influence matrix and the instance view (Fig. (N)). Reviewing misclassified instances, he again can see that there are several images labeled as *tiger cat* but actually including *tigers* in them. Overall, Michael observes that mislabeled instances tend to include small objects of tigers. This explains why the tiger stripe was not positively influential – because it was not easily detected due to its size.

## Usage Scenario \#3: Understanding the neural network's knowledge on a group of classes

In Fig. (A), Michael investigates GoogleNet's global mental model by reviewing the class navigation view. He notices a big circle in the bottom right corner of the visualization, indicating a clique containing many classes. The tooltip shows that the classes (7 in total) are all vehicles. He selects all of them. Reviewing these in the class-specific concepts view, he sees (as expected) many cross-class links in the class-specific concepts panel which connect influential concepts for the various vehicles. This means the neural network has likely learned common features across different cars.

Michael scans through the horizontal links across classes one by one. Fig. (B) and (C) show that the highest-ranked concept cluster (labeled CC14, for concept cluster \#14) contains window-like patches in the *beach wagon* class, so an important "vehicle commonality" the network has likely learned is "car windows." This hypothesis is confirmed by reviewing across other concepts in this concept cluster (e.g., Fig.  (C) also contains window patches from the *police van* class). The next set of links in Fig.  (D) and (E) tend to show "car side" concepts, which are relevant features regardless of the vehicle type (in this case, between *jeep* and *cab*. Further investigation reveals both "urban background" in Fig.  (F) and (G), and "wheel" as influential concepts in Fig.  (H) and (I). Both concepts make sense at a high level ("*Cars have wheels and car pictures are usually taken in urban environments.*"), but interestingly, the *jeep* class does not consider the "urban background" concept as influential. Reviewing *jeep* instances reveals why: many *jeep* pictures are taken with the background in nature (desert or woodland settings); he cannot find *jeep* pictures taken in urban settings.

While many of the concepts make sense upon review, Michael is surprised to realize that GoogleNet considers "windows" and "car sides" as the *most salient* discriminators or influencers for predicting cars, while other features like "wheels" or "headlights" are not chosen.

# User Study

To evaluate ConceptExplainer, we designed a user study to answer two primary questions relating to design goals ("*How well does the system support the design goals (G1–G4)?*") and overall usability ("*What is the overall usability of the system?*"). Our user study consisted of two stages: i) a task stage where participants completed three representative analysis tasks in line with the four design goals listed in Sect. ; and ii) a freeform analysis stage where participants freely probed model behavior using the interface. We recruited ten participants who were non-experts in deep learning, had not heard of concept-based explanations, and had not used ImageNet and GoogleNet before. We collected both quantitative and qualitative data, which allowed us to robustly evaluate both the ability of ConceptExplainer to support the design goals G1–G4, and also to understand the system's overall usability.

## Study Design

Participants took the following procedure for the study:

**(1) Training.** Participants first completed a short demographics questionnaire. Next, they were given a brief (high-level) introduction on TCAV, ACE, and what concept influence means. The study administrator then walked the participant through available features and functionalities of ConceptExplainer; participants completed a simple training task to help familiarize themselves with the system. During the training, participants could ask questions at anytime, and were allowed to play around with the interface until they felt comfortable to proceed.

**(2) Task.** Participants completed the following three tasks (T1-T3):

**T1:** Inspect the influential concepts for a given class: (1) Identify the comprehensible concepts and state why they are comprehensible. (2) Identify the incomprehensible concepts and state why they are incomprehensible. (3) Review the concept influences; what are the items that make sense to you, and what are the items that do not make sense to you? (4) Take a look at the instances contained in the influence matrix, find interesting cases and describe why they are interesting. This task primarily focuses *instance-level* and *single class-level analysis*, and supports G1 and G2. Each participant completed T1 on two classes: one high-performance (accuracy $>0.9$) and one poor-performance (accuracy $\sim0.1-0.3$); class choices for each participant were randomly selected from a high-performance set (*zebra*, *tench*, *lionfish*) and a poor-performance set (*tiger cat*, *appenzeller*, *seashore*).

**T2:** Given two classes that are similar and have overlapping misclassifications (i.e., images which should belong to $C_2$ are classified as $C_3$, and vice versa): (1) Identify what are the commonalities between their influential concepts. (2) Identify distinguishable influential concepts unique to each class. This task represents *instance-level* and *multi-class analysis*, and supports G2 and G3. The class pairs for T2 were randomly selected from the following sets: {*tiger cat*, *tiger*}, {*eskimo dog*, *siberian husky*}, and {*police van*, *ambulance*}. Note that if *tiger cat* was chosen during T1, it was not an option in T2.

**T3:** Given a group of seven related classes representing various vehicles (*fire engine*, *police van*, *school bus*, *jeep*, *cab*, *ambulance*, *beach wagon*)): (1) What are the common influential concepts across this group? Reason about the vehicle knowledge that the neural network has learned and what vehicle knowledge it lacks. This task represents *multi-class* and *global-level analysis*, and supports all four design goals (G1–G4). All participants used the same set of seven classes for this task. (Two participants had previously seen the *police van* and *ambulance* classes in T2, however their results were in line with other participants, so we do not believe this caused any study confounds during this task.)

The task order was consistent. Each participant was given a verbal description of the task (which was also available on a sheet of paper) by the administrator. The think-aloud protocol was employed during this stage; the administrator listened to verbal utterances to help confirm that the participant was correctly performing the task and to check if the participant's interpretations of tasks and system features was correct. Upon completion of a task, the participant verbally summarized their answer(s) to the administrator.

**(3) Freeform analysis.** In this stage, participants conducted an undirected, freeform analysis of ImageNet to gain insights on model behavior using ConceptExplainer. Participants were told to use the tool until they were satisfied, but were required to spend at least 10 minutes. To prevent participants from becoming lost or frustrated, we prepared an initial (optional) motivation scenario which was only given if participants asked for guidance: "Check if the neural network is working sensibly on well performing classes." Only one participant asked for this, as we found that the others had ideated their own exploratory goals upon completing the task stage. Participants also verbally reported their thought processes during this stage.

**(4) Review.** Participants completed a short usability survey to rate various system components using 7-point Likert scales; they were also allowed to provide comments or critiques about the system.

## Participants and Apparatus

We recruited ten participants: nine graduate computer science students at Arizona State University and one analyst with two years experience in a data company (average age = 25.8, SD = 2.08; 8 males, 2 females). Although a couple of the participants had colloquial familiarity with AI/ML, none had expertise in deep learning development or analysis, and all were unfamiliar with concept-based explanation methods. Study duration averaged 86 minutes (SD = 15).

ConceptExplainer was shown using Google Chrome in full-screen mode on a 30" monitor at $3840\times2160$ resolution. Study sessions were conducted in a quiet, office environment with no distractions.

## Study Results

To analyze the study, we first report quantitative ratings from the review stages' usability survey. We next report on qualitative verbal comments and responses given by participants, which were collected via the think-aloud protocol and summary answers during their tasks.

To analyze these verbal comments, we used a grounded theory procedure  to qualitatively code comments based on assessing how ConceptExplainer promotes insights and supports the design goals G1–G4.

### System Usability Ratings

Fig.  summarizes system usability based on participants ratings from the survey completed during the review stage. Ratings are broken into two types: (Q1–Q3) the overall usability and effectiveness of the system, (Q4–Q13) the usability of individual features.

In general, feedback was positive. We highlight that the system was considered easy to use (Q1), learn (Q2), and understand (Q3), and that the individual interface panels, the class navigation panel (Q5–Q6), the class concept view (Q7–Q11), and the concept navigation panel (Q12–Q13) were all positively regarded.

### Feedback from Novice Users

The survey ratings show that ConceptExplainer achieves good usability. Here, by analyzing the verbal comments and responses of participants, we reflect on the types of insights users can generate with the system, specifically in the context of the design goals (G1–G4).

**(G1) Contextualizing concepts using the concept navigation view.** The concept navigation view was heavily used and considered positively as a way to explore new concepts. In particular, several participants inspected similar concepts using thi view as a way to verify their intuition or mental model about a concept of interest. "*I'm trying to see what are concepts similar to this one because I want to know if my thought is correct*" (p5). "*This map thing \[the concept navigation view\] is cool. I can compare this \[concept\] with similar ones*" (p8). "*The navigation view helps me understand a concept better*" (p9).

**(G2) The concept view is effective revealing class-specific concepts.** Several participants regarded the class-specific concept view within class-concept view as effective in revealing influential concepts. Specifically, the list of concepts ranked by concept influences provided them with intuitive insights for network behaviors. "*This is helpful because I can immediately see what the neural network learned about the class*" (p3). "*It doesn't make sense that top concepts are background patches...middle concepts should be moved up. It's clear in the view*" (p8).

Despite liking the class-specific concept view, participants sometimes encountered incomprehensible concepts during their analysis. While most concepts had "good quality," and participants could understand them at a glance, some other concepts did not present a uniform pattern, which sometimes led participants to get stuck. Trying to make sense of such concepts seemed infeasible. In part, this is a byproduct of the concept generation pipeline used in ConceptExplainer's backend, which automatically extracts and defines concepts (bottom up) as opposed to creating them by hand using some rules (top down). Unfortunately, this process is not error-prone as it can extract and define noisy, unhelpful concepts, too. When users encounter with such concepts, they can slow down their analysis process. We discuss this issue more in Sect. .

**(G3) Concept links are effective revealing inter-class concept overlap.** Participants reported it was easy to identify conceptual links between classes and understand them by using the concept view. "*I can see that concept cluster 5 are green areas in the two classes.*" (p2, analyzing the *tiger cat* and *tiger* classes), "*The first link is white-fur stuff.*" (p5, analyzing the *Eskimo dog* and *Siberian husky* classes), "*I can see silhouette of cars, tire and window \[being the common concepts\].*" (p7, analyzing the *police van* and *ambulance* classes). As an extension of the current system, two participants (p1, p9) requested the ability to compare two or multiple concepts in parallel.

**(G4) Instance analysis was found to be useful for identifying data quality issues.**

A couple of participants identified (unprompted) data quality issues solely using instance analysis view. "*Clearly, real *tigers* are mislabeled as *tiger cats* in the testing set*" (p3). "*Why is there a tiger stripe in *tiger*\_cat_concept_8? It doesn't make sense ... Oh I see why, because there are tiger pictures in ground truths*" (p8).

Although participants in general like the instance analysis views, four participants (p1, p5, p6, p8) mentioned it was hard for them to link concepts with their segments in the image instance view. Three participants (p1, p3, p7) mentioned that instances sometimes seemed counterintuitive. "*Some instances are highly (positively) influenced by the concepts but are still misclassified, why?*" (p7, while analyzing *tiger cat*). "*Why are 'police' letters negatively influential here?*" (p3, analyzing *police van*). "*Why are cab cases all negatively influenced but correctly classified?*" (p1, comparatively analyzing multiple car classes). The current interface does not support answering these sorts of "*why?*" questions; see Sect.  for discussion on this.

# Discussion and Conclusion

We have presented ConceptExplainer, an interactive visual analytics system that supports non-expert users to explore and probe concept-based explanations of deep learning models. Based on a set of usage scenarios and a robust user study, we demonstrate that ConceptExplainer addresses a number of design challenges (C1–C4) and goals (G1–G4) that are important to the problem of concept-based explanation for non-expert users. In particular, we show how non-experts can effectively reason about model behavior at different analytic scales (i.e., instance, class, and global levels). Below, we discuss takeaways and lessons learned from our experiences in designing and evaluating ConceptExplainer, which include identifying current system limitations and opportunities for future research directions.

## User Study Takeaways

In general, the user study demonstrated the overall performance of our system from a human-centric perspective, indicating that it provides good usability and successfully supports the design goals (G1–G4), while also illustrating nuances and complexities in concept-based explanations. For example, some concepts in the system were not comprehensible to participants, which could slow down (or derail) their analytic process. Unfortunately, there is no easy way around this issue; in fact, it is an acknowledged problem that automatic concept discovery invariably introduces some amount of noise . As a future workaround, ConceptExplainer employs a modularized design. As novel concept discovery methods are developed (which hopefully reduce noise), they can be integrated into our system.

Another interesting takeaway from our study is that participants tended to use the instance analysis view more than we expected; based on feedback, we see several potential avenues for future extensions to enrich functionality. For example, the current instance analysis view provides instance-level explanations in terms of individual concept influences. A logical next step could be illustrating how the interplay of different concepts influence an instance's prediction. For example, both "snowy background" and "white fur" are positively influential for the *white wolf* class. When these two concepts co-occur in an instance, do they increase the likelihood of a *white wolf* prediction, compared to if only one concept is present? To achieve this type of fine-grained analysis, we plan to adopt techniques such as VRX  in future versions of ConceptExplainer.

## Serendipitously Supporting an Unexpected Task

Our user study also revealed an explanatory task supported by ConceptExplainer that we did not intentionally support. During the freeform analysis stage, several participants analyzed if the model "understood" a class by quickly tabbing through instances in the class-specific concept view ((J) in Fig. ). This allowed them to comprehend a rough estimate of the types of instances making up the class, and to use this as a basis to understand the influential concepts that the model has learned for the class. This "fact-checking" action was unexpected to us, but demonstrates the flexibility of the system's visual analytics to support diverse actions to interpret deep learning behavior.

## ConceptExplainer's Suitability for Expert Users

To understand ConceptExplainer's suitability for *expert* users, we conducted a set of pair analytic sessions  and semi-structured interviews with three deep learning experts (Ph.D. students with $3+$ years experience in AI research). The intent was to formatively assess the system's potential applicability for expert users, and to investigate how the system could be extended to better support them.

Each opined that the system was easy to understand once the concept method was illustrated and a walk through of the interface was given.

In terms of altering the system to support expert usage scenarios, it was suggested that concept explanations could be juxtaposed with visualizing the inner architecture and logic of the neural network. Such "opening up the black box," when juxtaposed with concepts, could enable novel ways to investigate and understand peculiar or unexpected model behavior. For example, when training a model, tools like ConceptExplainer can be used to identify the conceptual root cause of misclassifications between two similar classes, or to identify issues in ground truth labeling. Along these lines, one expert also suggested we augment the interface to let users focus on revealing concept distinguishability, so it becomes more explicitly clear on what basis the neural network discriminates between classes. In the future, we plan to explore ways to integrate techniques that combine concepts with visualizing the model's inner architecture, though such designs are likely unsuitable for non-experts.

## Concept-based Explanations in Other Domains

Currently, concept-based explanations have primarily been applied in image classification scenarios. Likewise, our system is optimized for this type of deep learning task, as opposed to other domains (e.g., natural language processing). As a future step, we would like to explore the use of visual analytics as a modality for probing and exploring concept-based explanations applied to other (currently unexplored) domains, such as speech recognition or time series prediction. Such domains bring their own complexities; e.g., in image classification, a user can directly observe the pixels in a set of image patches representing a concept, but there are no direct analogs to this process in many other domains. Despite this, visualization may prove to be a key approach for this, due to the power of visualization in being able to graphically render abstract data in ways that reveal patterns and insights.


author: Donald Arseneau ([`email@example.com`](mailto:firstname.lastname@example.com))
date: 2012/05/31
title: The framed package[^1]

Create framed, shaded, or differently highlighted regions that can break across pages. The environments defined are

|              |                                                   |
|:-------------|:--------------------------------------------------|
| framed       | ordinary frame box (`\`) with edge at margin      |
| oframed      | framed with open top/bottom at page breaks        |
| shaded       | shaded background (                               |
| shaded\*     | shaded background (                               |
| snugshade    | shaded with tight fit around text (esp. in lists) |
| snugshade\*  | like snugshade with shading edge at margin        |
| leftbar      | thick vertical line in left margin                |
| titled-frame | frame with title-bar; template for others         |

to be used like

        \begin{framed}
          copious text
        \end{framed}

But the more general purpose of this package is to facilitate the definition of new environments that take multi-line material, wrap it with some non-breakable formatting (some kind of box or decoration) and allow page breaks in the material. Such environments are defined to declare (or use) `\` for applying the boxy decoration, and `\``{`*settings*`}` … `\` wrapped around the main text (environment body).

The "`framed`" environment uses `\`, by default, as its `\` with the additional settings `\fboxrule=\FrameRule` and `\fboxsep=\FrameSep`. You can change these lengths (using `\`) and you can change the definition of `\` to use much fancier boxes.

In fact, the "`shaded`" environment just redefines `\` to be `\colorbox{shadecolor}` (and you have to define the color "`shadecolor`": `\definecolor{shadecolor}...`).

Although the intention is for other packages to define the varieties of decoration, a command `\` is defined for frames with open tops or bottoms, and used for the "`oframed`" environment. This facility is based on a more complex and capable command `\` which can be used for a wider range of frame styles. One such style of a title-bar frame with continuation marks is provided as an example. It is used by the "`titled-frame`" environment. To make use of `titled-frame` in your document, or the `\` command in your own environment definitions, you must define the colors `TFFrameColor` (for the frame) and a contrasting `TFTitleColor` (for the title text).

A page break is allowed, and even encouraged, before the `framed` and other environments. If you want to attach some text (a box title) to the frame, then the text should be inserted by `\` so it cannot be separated from the body.

The contents of the framed regions are restricted: Floats, footnotes, marginpars and head-line entries will be lost. (Some of these may be handled in a later version.) This package will not work with the page breaking of multicol.sty, or other systems that perform column-balancing.

The `MakeFramed` environment does the work. Its "*settings*" argument should contain any adjustments to the text width (via a setting of `\`). Here, the parameter `\` gives the measured extra width added by the frame, so a common setting is "`\advance\hsize-\width`" which reduces the width of the text just enough that the outer edge of the frame aligns with the margins. The "*settings*" should also include a 'restore' command – `\``parboxrestore` or `\` or something similar; for instance, the snugshade environment uses settings to eliminate list indents and vertical space, but uses `\` in its `\` to reproduce the list margin ouside the shading.

There are actually four variants of `\` to allow different formatting for each part of an environment broken over pages. Unbroken text is adorned by `\`, whereas split text first uses `\`, possibly followed by `\`, and finishing with `\`. The default definitions for these three just invokes `\`, so that all portions are framed the same way. See the `oframe` environment for use of distinct First/Mid/Last frames.

# Expert commands:

-   `\`, `\`: the "`MakeFramed`" environment

-   `\`: command to draw the frame around its argument

-   `\`: the frame for the first part of a split environment

-   `\`: frame for the last portion

-   `\`: for any intermediate segments

-   `\`: restore some text settings, but fewer than `\``parbox``restore`

-   `\`: length register; `\` for default "`framed`".

-   `\`: length register; `\` for default "`framed`".

-   `\`: macro; height of frame above baseline at top of page

-   `\`: vertical space before and after the framed env.; defaults to `\`

This is still a 'pre-production' version because I can think of many features/improvements that should be made. Also, a detailed manual needs to be written. Nevertheless, starting with version 0.5 it should be bug-free.

# ToDo:

-   Test more varieties of list

-   Improve and correct documentation

-   Propagation of `\`

-   Handle footnotes (how??) floats (?) and marginpars.

-   Stretchability modification.

[^1]: This manual corresponds to framed v0.96, dated 2011/10/22.


abstract: Our aim is to investigate the tidal interaction in High-mass X-ray Binaries and Symbiotic stars in order to determine in which objects the rotation of the mass donors is synchronized or pseudosynchronized with the orbital motion of the compact companion. We find that the Be/X-ray binaries are not synchronized and the orbital periods of the systems are greater than the rotational periods of the mass donors. The giant and supergiant High-mass X-ray binaries and symbiotic stars are close to synchronization. We compare the rotation of mass donors in symbiotics with the projected rotational velocities of field giants and find that the M giants in S-type symbiotics rotate on average 1.5 times faster than the field M giants. We find that the projected rotational velocity of the red giant in symbiotic star MWC 560 is $v\: \sin i$$= 8.2 \pm 1.5$ km$\,$s$^{-\!1}$, and estimate its rotational period to be $P_{\rm rot}$ = 144 - 306 days. Using the theoretical predictions of tidal interaction and pseudosynchronization, we estimate the orbital eccentricity $e=0.68-0.82$.
author: Kiril Stoyanov & Radoslav Zamanov
title: Rotation of the Mass Donors in High-mass X-ray Binaries and Symbiotic Stars

```latex
\begin{multicols}{2}
\section{Introduction}
A High-mass X-ray binary system consists of a compact object (a neutron star or a black
hole) accreting material from an O or B companion star.
They are divided into Be/X-ray binaries (main-sequence star as a companion) and Giant/Supergiant X-ray binaries (giant or supergiant star as a companion). Accretion of matter is different for both types of X-ray binaries.                                                   In the Be/X-ray binaries, the compact object crosses the circumstellar disc and accretes matter from that disk.
In the giant/supergiant X-ray binaries, the mass donor ejects a slow and dense wind radially outflowing from the equator and the
compact object directly accretes the stellar wind through Bondi-Hoyle-Lyttleton accretion.

Symbiotic stars are interacting binaries, consisting of an evolved giant (either a
normal red giant in S-types symbiotics or a Mira-type variable in D-types symbiotics) transferring mass to a hot and luminous white dwarf or neutron star. The symbiotic stars are surrounded by a rich and luminous nebula resulting from the presence of both
an evolved giant with a heavy mass-loss and of a hot companion abundant in ionizing photons and often emanating its own wind.

\section{Synchronization and Pseudosynchronization}
In a binary with a circular orbit the rotational period of the primary, P$_{rot}$,
reaches an equilibrium value at the orbital period, $P_{orb}=P_{rot}$.
In other words the synchronous rotation (synchronization) means that
the rotational period is equal to the orbital period.
In a binary with an eccentric orbit,
the corresponding equilibrium is reached at a value of $P_{rot}$ which is
less than $P_{orb}$, the amount less being a function solely of the orbital
eccentricity $e$. In practice, in a binary with an eccentric orbit the
tidal force acts to synchronize the rotation of the mass donor with
the motion of the compact object at the periastron -
pseudosynchronous rotation.
To calculate the period of pseudosynchronization, P$_{ps}$, we use (Hut 1981):
\begin{equation}
P_{ps} = \frac{(1+3e^2+\frac{3}{8}e^4)(1-e^2)^\frac{3}{2}}{1+\frac{15}{2}e^2+
\frac{45}{8}e^4+\frac{5}{16}e^6} P_{orb}.
\label{Eq-ps}
\end{equation}

\subsection{Stars with  radiative envelopes}
Following Hurley, Tout \& Pols (2002) the circularization
timescale for stars with  radiative envelopes is:
\begin{equation}
\frac{1}{{\tau}_{circ}}=
\frac{21}{2} \left( \frac{G M_1}{R_1^3} \right)^\frac{1}{2}
q_2 \left(1 + q_2 \right)^\frac{11}{6} E_2 \left(\frac{R_1}{a} \right)^\frac{21}{2},
\label{circ_re}
\end{equation}
where M$_1$ and R$_1$ are the mass and the radius of the primary respectively,
q$_2$ is the mass ratio M$_2/$M$_1$, and $a$ is the semi-major axis.
The second-order tidal coefficient
E$_2$ = 1.592 $\times 10^{-9} M_1^{2.84}$.

The synchronization time scale is given as,
\begin{equation}
\tau_{sync} = K \tau_{circ},
\label{sync_re}
\end{equation}
where K  is:
\begin{equation}
K \approx \frac{0.015}{r_g} \, \frac{1 + q_2}{q_2} \left( \frac{R_1}{a} \right)^2.
\end{equation}
For the gyration radius of the primary $r_g$
we adopt $r_g\approx0.16$ for giants, and $r_g\approx0.25$ for main sequence stars
(Claret \& Gimenez, 1989).

\subsection{Stars with convective envelopes}
Following Hurley, Tout \& Pols (2002) the synchronization timescale for stars with convective
envelopes is:
\begin{equation}
 \tau_{\rm syn} \approx 800 \left( \frac{ M_{\rm 1}  R_{\rm 1}}{ L_{\rm 1}}\right)^{1/3}
 \frac{M_{\rm 1}^2 (\frac{M_{\rm 1}}{M_{\rm 2}} + 1)^2}{R_{\rm 1}^6} P_{\rm orb}^4\ \;   {\rm yr},
\label{sync}
\end{equation}
where $L_{\rm 1}$ is the luminosity of the giant.

The circularization time scale is:
\begin{equation}
 \frac{1}{{\tau}_{\rm circ}} = \frac{21}{2} \left(\frac{k}{T}\right) q{_2} (1+q{_2})
 \left(\frac{R_{\rm 1}}{a}\right)^8 .
\label{circ}
\end{equation}
In  Eq.~\ref{circ}, $(k/T)$ is:
\begin{equation}
\left(\frac{k}{T}\right) = \frac{2}{21} \frac{f_{\rm conv}}{{\tau}_{\rm conv}} \frac{M_{\rm env}}{M_{\rm 1}}
\;  {\rm yr}^{-1} ,
\label{KT}
\end{equation}
where $R_{\rm env}$ is the depth of the convective envelope, $M_{\rm env}$ is the envelope's mass, and
\begin{equation}
{\tau}_{\rm conv} = 0.4311 \left(\frac{M_{\rm env}R_{\rm env}(R_{\rm 1}-\frac{1}{2}R_{\rm env})}{3 L_{\rm 1}}\right) ^{\frac{1}{3}}  {\rm yr}
\label{tau}
\end{equation}
is the eddy turnover time scale (the time scale on which the largest convective cells turnover).
The numerical factor $f_{\rm conv}$ is
\begin{equation}
{f}_{\rm conv} = {\rm min} \left[1, \left( \frac{P_{\rm tid}}{2{\tau}_{\rm conv}} \right) ^2 \right],
\label{fconv}
\end{equation}
where $P_{\rm tid}$ is the tidal pumping time scale given by
\begin{equation}
\frac{1}{P_{\rm tid}} = \left|\frac{1}{P_{\rm orb}} - \frac{1}{P_{\rm rot}}\right|.
\label{ptid}
\end{equation}

The pseudosynchronization timescale is $\tau_{\rm ps}$ = (7/3($\alpha$ - 3)) $\tau_{\rm circ}$,
where $\alpha$ is a dimensionless quantity, representing the ratio of the orbital and rotational angular
momentum:

\begin{equation}
\alpha = \frac {q_2}{1+q_2} \frac {1}{r_{\rm g}^{2}} \left( \frac {a}{R_{\rm 1}} \right) ^2 .
\end{equation}
For a red giant we adopt $r_{\rm g} \approx 0.3$ (Claret, 2007).

In all equations, the masses, the radii and the lumunosities are in solar units.

\section{High-mass X-ray Binaries}
\vspace{-8mm}
The orbital and stellar parameters of 13 High-mass X-ray binaries are given in Table~1
and Table~2 in Stoyanov \& Zamanov (2009). We add 2 more objects - 4U~2206+54 and MWC~148. The
orbital and stellar parameters are taken from Rib{\'o} et. al. (2006) and Casares et al. (2012) respectively.

Using Eq.\ref{circ_re} and Eq.\ref{sync_re} we estimate the circularization and
synchronization timescales. The results are given in Table~\ref{t-times}.
The lifetime of a star on the main sequence can be estimated as
$\tau_{MS} = 10^{10} ({M_\odot}/{M})^{2.5} \; \; {\rm years}$ (Hansen \& Kawaler, 1994).
Comparing these  lifetimes with $\tau_{sync}$
from Table~\ref{t-times},
we see that among the Be/X-ray binaries only for LSI+61$^0$303 is
$\tau_{sync} \sim \tau_{MS}$.
This is the only Be/X-ray binary for which we can expect
considerable changes of the rotation  of the primary
during the lifetime of the Be star.

The lifetime of the giant is comparable or longer then
$\tau_{circ}$ \ and  \  $\tau_{sync}$  \  for the giant/supergiant systems
with short orbital periods. The exceptions are V725~Tau and BP~Cru, for
which $\tau_{sync}$ and $\tau_{circ}$ are longer than the lifetime of the giant/supergiant stage.

On Fig.1 in Stoyanov \& Zamanov (2009) is plotted P$_{rot}$ versus P$_{ps}$.
The giant/supergiant systems are located close to the line
P$_{ps} = P_{rot}$,
while those with mass donors from spectral class V are far away from the equilibrium.

In the Be/X-ray systems BQ~Cam, V635 Cas, V725 Tau and 4U~2206+54, the tidal force spinning down the donor star.
For the system LSI+61$^0$303, the rotation of the mass donor
is close to pseudosynchronization.
This is the only Be/X-ray binary in which  $\tau_{sync}$ is comparable with the
life-time of the binary. In the binaries X~Per and MWC~148, the neutron star is far away from the Be star
and the tidal force is weak.

Giant and supergiant systems are close to (pseudo)synchronization.
In these binaries
the rotation of the mass donors is influenced by the presence
of the compact object.
In LMC X-4 and Cen X-3, the mass donors
are synchronized and the orbits are circularized.
With respect to the rotation of the mass donor, V725~Tau
is similar to the Be/X-ray binaries.
Cyg X-1 is synchronized and almost circularized.
V830~Cen is pseudosynchronized but not circularized yet.
The systems LSI$+65^0010$ and Vela X-1 are close to
pseudosynchronization and the tidal force accelerates
the rotation of the mass donors.
In the case of SMC X-1, the tidal force acts as a decelerator
of the rotation of the mass donor.
In BP~Cru, a gas stream from the mass donor  exists,
probably resulting from the strong tidal force and spin-up of the mass donor (Leahy \& Kostka, 2008).

\begin{mytable}
\caption{Calculated time scales. Given here are
  the name of the object,
  synchronization time scale,
  circularization time scale, and the lifetime.}
\label{t-times}
{\begin{tabular}{lllc}
\hline
object   &    $\tau_{sync}$ [yr] & $\tau_{circ}$ [yr] & lifetime [yr]  \\
\hline
\multicolumn{2}{l}{{\bf Be/X-ray binaries}}  \\
LSI+61$^0$303*&   2.8\t10$^7$    & 2.4\t10$^8$    &5.6\t10$^7$ \\
X~Per         &   6.2\t10$^{17}$ & 1.8\t10$^{21}$ &1.1\t10$^7$ \\
BQ~Cam        &   3.5\t10$^{11}$ & 7.6\t10$^{13}$ &3.9\t10$^6$ \\
V635~Cas      &   1.4\t10$^{11}$ & 9.5\t10$^{12}$ &7.3\t10$^6$ \\
4U~2206+54    &   4.9\t10$^9$    & 3.7\t10$^{11}$ &7.3\t10$^7$ \\
MWC~148       &   1.2\t10$^{17}$ & 5.1\t10$^{20}$ &9.8\t10$^7$ \\
\multicolumn{2}{l}{{\bf Giant systems}}  \\
V725~Tau      &   2.8\t10$^{12}$ & 8.0\t10$^{14}$ &4\t$10^5$   \\
LMC X-4       &   4.5\t10$^2$    & 7.7\t10$^2$    &1\t$10^6$   \\
Cen~X-3       &   2.3\t10$^3$    & 4.2\t10$^3$    &5\t$10^5$   \\
\multicolumn{2}{l}{{\bf Supergiant systems}} \\
V830~Cen      &   7.5\t10$^3$ & 1.4\t10$^4$ & 1\t10$^6$   \\
LSI+65$^0$010 &   1.3\t10$^4$ & 3.9\t10$^4$ & 1\t10$^6$  \\
Vela~X-1      &   1.0\t10$^4$ & 2.8\t10$^4$ & 3.9\t10$^5$      \\
SMC X-1       &   3.3\t10$^4$ & 8.2\t10$^4$ & 8.8\t10$^5$    \\
BP Cru        &   1.8\t10$^6$ & 8.8\t10$^6$ & 8\t10$^4$     \\
Cyg X-1       &     $<1$      & $<1$       & 1\t10$^5$    \\
\hline
\end{tabular}}
*assuming neutron star as a secondary component.
\end{mytable}

\vspace{-10mm}
\section{S-type symbiotic stars}
43 symbiotic stars have been observed with FEROS spectrograph  at the 2.2m ESO
telescope of the La Silla Observatory (Zamanov et al. 2007).
The data for the rotation of 55 field red giants are taken from the
literature. M giants in S-type symbiotics rotate faster than
the field M giants.
Histograms of the available \vsi\  data  for the red giants
are plotted in Fig.2 in Zamanov \& Stoyanov (2012).
For the field  M0III-M6III giants we calculate a
mean \vsi$=$5.0~\kms,  median \vsi$=$4.3~\kms, and
standard deviation of the mean $\sigma=4.0$~\kms.
For the M0III-M6III giants in symbiotics,
we get a mean \vsi$=7.8$~\kms,   median \vsi$=$8.0~\kms,
and standard deviation of the mean $\sigma=$2.1~\kms.

There are 5 objects in our sample that deviate from the synchronization.
These objects are
RS~Oph, MWC~560, CH~Cyg, CD-43$^\circ$14304 and Z~And.
In three of them  collimated jets are detected:
Z~And   (Skopal et al. 2009);
CH~Cyg  (Crocker et al. 2002), MWC~560 (Tomov et al. 1990).
Additionally to the jets,  ejection of blobs are detected from  RS~Oph and CH~Cyg
(Iijima et al. 1994). This confirms  the  suggestions that in the jet-ejecting symbiotics
the mass donors  rotate faster than the orbital periods.
Probably there is a link between the jets
and the mass donor rotation.

On Fig.\ref{stoyanov-fig1} are plotted together the High-mass X-ray binaries and the S-type symbiotic stars.
It shows that none of the objects in our sample is above the line of synchronization.

\begin{myfigure}
\centerline{\resizebox{90mm}{!}{\includegraphics{stoyanov_CVs_2013_01_fig01.eps}}}
\caption{P$_{rot}$ versus P$_{orb}$ on a logarithmic scale.
      The blue circles indicates the High-mass X-ray Binaries - filled symbols for giant/supergiant systems and
       open symbols for Be/X-ray binaries.
      The red triangles indicates the S-type symbiotic stars. }
\label{stoyanov-fig1}
\end{myfigure}

\vspace{-10mm}
\section{Orbital eccentricity of MWC~560}
MWC~560 is a symbiotic star,
which consists of a red giant and a white dwarf
(Tomov et al. 1990).
The most spectacular features of this object are the collimated ejections
of matter with velocities of up to $\sim 6000$~\kms\ (Tomov et al. 1992)
and the resemblance of its emission line spectrum to that of
the low-redshift quasars (Zamanov \& Marziani, 2002).
The jet ejections are along the line of sight and the
system is seen almost pole-on ($i < 16^\circ$).
This makes it difficult to obtain the orbital eccentricity of the system in a conventional way.

For the system we adopt
$R_{\rm g} = 140 \pm 7~R_\odot$, $L_{\rm g} \sim 2400~L_\odot$,
 $M_{\rm g} = 1.7~M_\odot$,
$M_{\rm wd}=0.65~M_\odot$, $P_{\rm orb}=1931 \pm 162$~day (Gromadzki et al. 2007), $R_{\rm env}=0.9~R_{\rm g}$ and $M_{\rm env}= 1.0~M_\odot$ (Herwig 2005).
With the above values of the parameters assumed, we derive the semi-major axis of the orbit
to be $a \approx 860~R_\odot$.
Using these parameters, we calculate from Eq.~\ref{sync} and Eq.~\ref{circ} the synchronization and circularization time scales:
$\tau_{\rm sync} = 2.6 \times 10^4$~yr and
$\tau_{\rm circ} = 3.1 \times 10^6$~yr.
The typical lifetime of a symbiotic star is $\tau_{\rm ss} \sim 10^5$~yr (Yungelson et al., 1995).
From the rate of accretion on the white dwarf,
$\dot M_{\rm acc} \approx 5 \times 10^{-7}~M_\odot$ (Schmid et al. 2001), we can estimate, that it will take $10^6$~yr
to accrete $\sim 0.5~M_\odot$ from the envelope of the red giant companion.
Because the giant also losses mass via stellar wind, we find that the lifetime
of the symbiotic phase of MWC~560 should be
$\tau_{\rm ss} \leq 10^6$~yr.
For MWC~560 we have therefore the situation in which $\tau_{\rm ps} < \tau_{\rm syn} < \tau_{\rm ss} < \tau_{\rm circ}$.
This means that the symbiotic phase is long enough that the tidal forces can
(pseudo)synchronize the rotation of the red giant.
On the other hand, the value of $\tau_{\rm circ}$ demonstrates that
the symbiotic lifetime of MWC~560 is shorter than the circularization time,
and therefore the orbit can be eccentric.
This is in agreement with the observational evidences found by
Fekel et al. (2007) that the symbiotic stars with P$_{\rm orb} > 800$~days
tend to have eccentric orbits. The above implies that in MWC 560, the red giant is probably synchronized,
but the orbit is not circularized.
To determine the orbital eccentricity of  MWC~560, we need
to calculate $P_{\rm rot}$ for the mass donor.
We analyzed 21 high resolution spectra of MWC~560 and obtained value for \vsi $= 8.2 \pm 1.5$~\kms.
Using $R_{\rm g} = 140 \pm 7~R_\odot$ and  $\; i=12^\circ-16^\circ$,
we calculate $P_{\rm rot}=144 - 306$~days. This value
is less than the orbital period.
MWC~560 should
be close to synchronization or pseudosynchronization,
and $P_{\rm rot} =P_{\rm ps}$.
Using Eq.~\ref{Eq-ps} we can estimate the orbital eccentricity to be
$e=0.68-0.82$.

\vspace{-5mm}
\section{Conclusions}
\vspace{-4mm}
Using rotational velocity measurements and the theory of synchronization/pseudosynchronization we:

(1) find that the Be/X-ray binaries are far away from
(pseudo)synchronization.
The tidal force in the Be/X-ray binaries acts as a decelerator
of the rotation of the mass donors.  The only Be/X-ray binary
which is close to pseudosynchronization is the LSI$+61^0303$.
The objects containing mass donors of spectral class I
and III typically have $P_{rot} \sim P_{ps}$ and are
close to (pseudo)synchronization;

(2) demonstrate that the M giants in symbiotic stars rotate faster than the field giants.
Most symbiotics with orbital period less than 1000~d are synchronized;

(3) show that the High-mass X-ray binaries and the S-type symbiotic stars are either on the line of
synchronization or they are under the line. None of the objects in our sample is above the line of synchronization.

(4) calculate that the orbit of the symbiotic star MWC~560 should be highly eccentric, with $e\sim$ 0.7.

\thanks
We thank the anonymous referee for constructive comments. This work was supported by the OP ``HRD``, ESF and Bulgarian Ministry of Education, Youth and Science under the contract
BG051PO001-3.3.06-0047.

\vspace{-4mm}
\begin{thebibliography}{99}

\bibitem[1]{2012MNRAS.421.1103C} Casares, J., Rib{\'o}, M., Ribas, I., Paredes, J.~M., Vilardell, F., \& Negueruela, I.: 2012, MNRAS, 421, 1103
\bibitem[2]{2007A&A...467.1389C} Claret, A.: 2007, A\&A~467, 1389
\bibitem[3]{1989A&AS...81...37C} Claret, A., \& Gimenez, A.: 1989, A\&AS, 81, 37
\bibitem[4]{2002MNRAS.335.1100C} Crocker, M.~M., Davis, R.~J., Spencer, R.~E., et al.: 2002, MNRAS, 335, 1100
\bibitem[5]{2007AJ....133...17F} Fekel, F.~C., Hinkle, K.~H., Joyce, R.~R., Wood, P.~R., Lebzelter, T.: 2007, AJ~133, 17
\bibitem[6]{2007A&A...463..703G} Gromadzki, M., Miko{\l}ajewska, J., Whitelock, P.~A., Marang, F.: 2007, A\&A~463, 703
\bibitem[7]{1994sipp.book.....H} Hansen, C.~J., \& Kawaler, S.~D.: 1994, {\it Stellar Interiors.~ Physical Principles, Structure, and Evolution}, Springer-Verlag
\bibitem[8]{2005ARA&A..43..435H} Herwig, F.: 2005, ARA\&A~43, 435
\bibitem[9]{2002MNRAS.329..897H} Hurley, J.~R., Tout, C.~A., \& Pols, O.~R.: 2002, MNRAS, 329, 897
\bibitem[10]{LPH} Hut, P.: 1981, A\&A, 99, 126
\bibitem[11]{1994A&A...283..919I} Iijima, T., Strafella, F., Sabbadin, F., \& Bianchini, A.: 1994, A\&A, 283, 919
\bibitem[12]{2008MNRAS.384..747L} Leahy, D.~A., \& Kostka, M.: 2008, MNRAS, 384, 747
\bibitem[13]{2006A&A...449..687R} Rib{\'o}, M., Negueruela, I., Blay, P., Torrej{\'o}n, J.~M., \& Reig, P.: 2006, A\&A, 449, 687
\bibitem[14]{2001A&A...377..206S} Schmid, H.~M., Kaufer, A., Camenzind, M., Rivinius, T., Stahl, O., Szeifert, T., Tubbesing, S.,
Wolf, B.: 2001, A\&A~377, 206
\bibitem[15]{2009ApJ...690.1222S} Skopal, A., Pribulla, T., Budaj, J., et al.: 2009, ApJ, 690, 1222
\bibitem[16]{2009AN....330..727S} Stoyanov, K.~A., \& Zamanov, R.~K.: 2009, Astronomische Nachrichten, 330, 727
\bibitem[17]{1992MNRAS.258...23T} Tomov, T., Zamanov, R., Kolev, D., Georgiev, L., Antov, A., Mikolajewski, M.,
Esipov, V.: 1992, MNRAS~258, 23
\bibitem[18]{1990Natur.346..637T} Tomov, T., Kolev, D., Zamanov, R., Georgiev, L., Antov, A.: 1990, Nature~346, 637
\bibitem[19]{1995ApJ...447..656Y} Yungelson, L., Livio, M., Tutukov, A., \& Kenyon, S.~J.: 1995, ApJ~447, 656
\bibitem[20]{2007MNRAS.380.1053Z} Zamanov, R.~K., Bode, M.~F., Melo, C.~H.~F., et al.: 2007, MNRAS, 380, 1053
\bibitem[21]{2002ApJ...571L..77Z} Zamanov, R., Marziani, P.: 2002, ApJL~571, L77
\bibitem[22]{2012BlgAJ..18c..41Z} Zamanov, R.~K., \& Stoyanov, K.~A.: 2012, Bulgarian Astronomical Journal, 18, 41

\end{thebibliography}

\end{multicols}
```