\section*{Acknowledgements}
This work was partially funded by the UKRI Turing AI Fellowship EP/V024817/1.
\clearpage
\newpage
\bibliographystyle{splncs04}

\section{Discussion \& Future Work}\label{sec:discussion}
In this paper we presented an alternative approach for understanding and informing the construction of local surrogates. We have discussed and shown the interaction between coverage, fidelity and complexity. We have proposed a workflow for interpretable local surrogates which is interactive and where we consider a range of localities instead of a fixed, predetermined locality.

Future work will involve further exploration of the different problems and limitations posed in the paper, namely:
\begin{itemize}
    \item\texttt{\textbf{Coverage:}} We have introduced coverage as an important property of local surrogates. We model coverage as the radius of a hyper-sphere centered on the instance to be explained (from which we sample the neighbourhood). This is easy to understand and implement, but also very generic. However, one could argue whether this is the best way of defining coverage. Also, ideally we will want to design procedures for increasing/decreasing coverage that can adapt to the user's needs. For example, a straightforward extension to the hyper-sphere could be a hyper-ellipsoid where radii along different dimensions are derived from the user's needs, or preferences.
\item\texttt{\textbf{Sampling:}} Here we  used a simple sampling regime where we sampled the neighbourhood uniformly at random from the hyper-sphere. We also did not make use of weights during training. However, sampling can change to make training more efficient, allowing training the surrogate with fewer training samples. Also, classification tasks with high-dimensionality are particularly challenging as they would require a larger size for a neighbourhood, rather than a fixed size.

\item \texttt{\textbf{Uncertainty:}} Uncertainty is important in interpretability as it signals trust to the output of the model; be it a predictive model, or be it a local surrogate explainer. In this work we have presented the use of bootstrapping the surrogate by training on multiple (different) draws of the neighbourhood. This is of course computationally expensive. There might be better, computationally efficient methods for deriving uncertainty estimates for local surrogates. For Generalised Linear Models (GLM) there exist ways of deriving such uncertainty estimates without having to resort to multiple runs, but these would depend on assumptions to hold true.
\end{itemize}
\section{Experiments}\label{sec:experiments}

\begin{figure}[!t]
    \centering
    \includegraphics[width=\textwidth]{figures/experiments/example_rac_curve_lr.png}
    \caption{Surrogate models fitted on neighbourhoods generated from a range of radii. \textbf{top row:} decision boundary of a black-box (a neural network) that has been trained on the half-moons data. \textbf{bottom row:} decision boundaries of the surrogates (logistic regression). The small embedded bar-plot presents the coefficients of the two features. On the titles we note the \textit{true-positive rate} (TPR) and the \textit{true negative rate} (TNR) as measures of fidelity to the black-box. In all cases we plot the instance to be explained (green star) and the hyper-spheres (green circles) from which we sample the neighbourhood uniformly at random.}
    \label{fig:local_surrogates_lr}
\end{figure}
In this section we present experiments on toy datasets to illustrate the advantages of viewing locality as a continuum rather than keeping it fixed. In all that follows we model coverage by introducing a hyper-sphere, of radius $r$, centered on $\boldsymbol{x}$ -- the instance to be explained. The neighbourhood is then generated by sampling uniformly at random from within this hyper-sphere. Unless specified otherwise, the size of the neighbourhood is fixed at $2000$ points.

In Figure \ref{fig:local_surrogates_lr} we consider a range of radii and the surrogates obtained by training on the corresponding neighbourhoods. On the top row we see the decision boundary of a black-box (a neural network) that has been trained on the \texttt{half-moons} dataset. We also see the instance to be explained (green star) and the hyper-spheres (green circles) from which we sample the neighbourhoods. On the second row we have the decision boundaries of the surrogates (logistic regression) and the small embedded bar-plot presents the coefficients of the two features. On every sub-figure on the second row we also note the \textit{true-positive rate} (TPR) and the \textit{true negative rate} (TNR) as measures of fidelity to the black-box.

We see that the \texttt{radius} -- which controls the coverage -- has significant impact on both the accuracy metrics and the coefficients of the surrogate models learned. In the far-left pair of sub-figures -- which is the most global -- of the four, the coefficient corresponding to the first feature is positive, while then it turns negative. We also see an increase in the accuracy metrics when moving from the first case to the second, and when moving from the second to third. When comparing the last two we see that coefficients and accuracy have almost settled, and the choice would depend on the user.

In Figure \ref{fig:local_surrogates_different_coverage_lr} we consider a finer spectrum of radii. This gives the bigger picture of how models trained on different levels of coverage perform (with regards to accuracy) and what their learned parameters imply for the instance to be explained.

\begin{figure}
    \centering
    \includegraphics[width=\textwidth]{figures/experiments/two_moons_different_coverages_pareto.png}
    \caption{Surrogate models fitted on neighbourhoods generated from a range of radii. We show radius on the x-axis, and accuracy on the y-axis. The two embedded bar-plots correspond to the coefficients of the surrogate models (logistic regression) trained on neighbourhoods with radius as shown by the arrows.}
    \label{fig:local_surrogates_different_coverage_lr}
\end{figure}

In Figure \ref{fig:bootstrap_local_surrogates_lr} we introduce uncertainty in the explanations produced (for the same instance) to enhance the understanding of the behaviour of the models through the use of bootstrapping \cite{bootstrap}. We highlight four regions in the plots that were manually annotated accordingly. As we vary the coverage of (bootstrapped) local surrogates, we see that both the accuracy score and the coefficients change. We observe a general trend of decreasing accuracy (top sub-figure) as the radius increases, as well as a small increase in the standard deviation between the bootstrapped surrogate models. We also observe how the coefficients of the models vary with the radius, and how one of them transitions from negative to positive.
\begin{figure}[!t]
    \centering
    \includegraphics[width=\textwidth]{figures/experiments/bootstrap_local_surrogates.png}
    \caption{Bootstrapping the local surrogate. For every value of \texttt{radius} we train $500$ local surrogates on different draws of a neighbourhood. In this case we consider neighbourhoods of size $200$ to gain a better sense of variability (as opposed to neighbourhoods of size $2000$ as in other experiments). \textbf{top:} scores of bootstrapped surrogate models. \textbf{bottom:} coefficients of bootstrapped surrogate models. In both cases we plot the mean $\pm$1std. Vertical shaded regions are manually annotated to denote different behaviours of bootstrapped models and how they could be interpreted. }
    \label{fig:bootstrap_local_surrogates_lr}
\end{figure}

\begin{figure}[!ht]
    \centering
    \includegraphics[width=\textwidth]{figures/experiments/example_rac_curve_dtc.png}
    \caption{Surrogate models fitted on neighbourhoods generated from a range of radii. \textbf{top row:} decision boundary of a black-box (a neural network) that has been trained on the \texttt{circles data}. \textbf{second row:} decision boundaries of the surrogates (decision tree). \textbf{third \& fourth rows:} decision boundaries of the surrogates but this time with \texttt{max\_depth} fixed to $4$ and $1$ respectively. On the titles we note the \textit{true-positive rate} (TPR) and the \textit{true negative rate} (TNR) as measures of fidelity to the black-box. In all cases we plot the instance to be explained (green star) and the hyper-spheres (green circles) from which we sample the neighbourhood. The small embedded bar-plot presents the feature importance measures extracted from the surrogate}
    \label{fig:local_surrogate_dtc}
\end{figure}

In Figure \ref{fig:local_surrogate_dtc} we let our interpretable model class be decision trees. We now explore the setting where we constraint complexity of the surrogate model, and observe the model's accuracy. In the second row of the figure we see a succession of models trained on neighbourhoods with decreasing radius. With no complexity constraints the models achieve good fidelity scores and we also observe a decrease in complexity with a decrease in coverage. In the last two rows of the figure we impose complexity constraints on the training of the model: on the third row we restrict models to have at most $depth=4$ and on the last row we restrict models to have at most $depth=1$.

In Figure \ref{fig:bootstrap_local_surrogates_lr} we kept the complexity fixed (number of non-zero coefficients) and experimented with the interaction of coverage and accuracy. In problems on higher-dimensions we can also experiment with number of non-zero coefficients. A popular formulation for linear models is the Lasso, where the introduced $l_1$-norm penalty leads to sparser solutions. It is used in core machine learning and statistics, and in interpretable machine learning for this reason. In Figure \ref{fig:lasso_paths} we experiment with the \texttt{diabetes} dataset. Our model class is logistic regression, but now with $l_1$-norm penalty. A useful tool for analysis and visualisation for this model class is the so-called `regularisation path' of multiple solutions to the lasso formulation. A solution is obtained for successively relaxed penalty parameters - reducing the effect of the sparsity inducing $l_1$-norm penalty. Then for every solution you can observe how the coefficients vector varies. Examples of such paths are shown in the first row of Figure \ref{fig:lasso_paths}, where every colour corresponds to one coefficient and shows how it varies along the `path' of decreasing regularisation strength ($C$). In the same figure for a range of radii we plot the lasso path (top), accuracy scores of the models (middle), and the number of non-zero coefficients of the solutions (bottom). We observe a general trend of the accuracy scores path shifting upwards, as the radius decreases, and we also observe the beginning of the complexity paths (bottom) to shift downwards. This follows along the lines of previous observations where as we decrease the coverage, we can obtain a models with non-decreasing accuracy and non-increasing complexity.

\begin{figure}[!t]
    \centering
    \includegraphics[width=\textwidth]{figures/experiments/lasso_paths_idx_10.png}
    \caption{Lasso paths together with the corresponding accuracy and $l_0$-norm of solution. For this we use the \texttt{diabetes} dataset and transform it to a classification task by thresholding on the median value of the target variable. \textbf{top row:} regularisation paths for lasso logistic regression. Several models are trained on a range of decreasing regularisation parameters. On the left-end, regularisation ($C$) is strongest -- and all coefficients are zero -- while on the right-end regularisation is at its weakest. Each colour corresponds to the `path' of a different coefficient. \textbf{middle row:} accuracy scores for the models along the path. \textbf{bottom row:} number of non-zero coefficients for the models along the path. \textbf{columns:} each column corresponds to models trained on neighbourhoods generated from a different radius -- which is specified at the top of each column.}
    \label{fig:lasso_paths}
\end{figure}

\section{Introduction}

\textit{Local surrogate explainers} is a class of \textit{post-hoc} interpretability methods whose goal is to help explain a prediction for an instance, produced by \textit{any} machine learning model. These are usually considered \textit{black-boxes}, i.e. we have no access to the model's internals; we can only observe the output for a given input. For example, let $f: \mathbb{R}^d \to \{0, 1\}$ denote a binary classifier. Then the black-box allows us to feed an instance $\boldsymbol{x}\in\mathbb{R}^d$ through $f$, and then \textit{only} observe the output, $f(\boldsymbol{x})\in\{0, 1\}$.

The goal of local surrogates is to model the \textit{local} classification behaviour of the black-box with the instance of interest as the reference. In this context, `local' is a loose term that implies closeness with respect to a distance metric, where this distance can be geometric, e.g. Euclidean distance or weighted Manhattan distance; it can be based on semantics, e.g. all images of a certain class; or it can be any other measure that reflects closeness -- determined by the application or imposed by the user \cite{hepburn2021}.

As discussed in \cite{poyiadzi2021overlooked}, existing methods that fall in this category (such as LIME \cite{ribeiro2016should}, PALEX \cite{jia2019palex}, GSLS \cite{laugel2018b}, LEAP \cite{localembeddings}, LORE \cite{guidotti2018local}) follow a similar recipe for explaining an instance $\boldsymbol{x}$, but differ on their objectives and their understanding of locality. Practitioners and researchers alike, are now faced with a myriad of approaches with no simple way of deciding which one to use, and what the properties and assumptions of the different approaches are \cite{poyiadzi2021overlooked,kacper2019}. The common core algorithm followed by these techniques presented in \cite{poyiadzi2021overlooked} is shown in Algorithm \ref{algo:local_surrogate_recipy}. The major difference between all these approaches is how they sample the neighbourhood (line 1) and how they assign weights (line 3). These control the information to be modelled by the local surrogate.

In this work we take a step back and try to understand local surrogates from a different point of view. We start off by discussing global surrogates and the trade-off between \textit{accuracy} and \textit{complexity}. We then introduce the notion of locality -- and, local surrogates -- as a means of narrowing down the area of interest with the aim of making the conditions of the accuracy-complexity trade-off amenable and more favourable. We consider the whole spectrum ranging from global to local and refer to this idea as \textit{coverage}. Instead of assuming the existence of a `right level of locality', we propose and study a dynamic and interactive view of locality that could lead to more information extracted from local surrogate modelling. We argue that high coverage is a favourable property of surrogates as it implies a higher level of trust, and a better understanding of generalisation of the surrogate.

\renewcommand{\algorithmicrequire}{\textbf{Input:}}
\renewcommand{\algorithmicensure}{\textbf{Output:}}

\begin{algorithm}[!t]
  \caption{Fitting an interpretable local surrogate}\label{algo:local_surrogate_recipy}
  \begin{algorithmic}[1]
      \Require{black-box $f$, instance to be explained $\boldsymbol{x}$}
      \Ensure{Interpretable Local Surrogate}
      \State Sample, or generate, the neighbourhood $\boldsymbol{X_{N}}$ which is a means of encouraging locality to $\boldsymbol{x}$.
      \State Obtain labels for $\boldsymbol{X_{N}}$ from the black-box: $y_N = f(\boldsymbol{X_{N}})$.
      \State (Optional) Compute weights, $\boldsymbol{w_{N}}$, that measure closeness to $\boldsymbol{x}$, for all instances in $\boldsymbol{X_{N}}$.
      \State Fit a local surrogate model -from an interpretable model class- using $(\boldsymbol{X_{N}}, \boldsymbol{y_{N}}, \boldsymbol{w_{N}})$.
  \end{algorithmic}
\end{algorithm}
Our contributions are as follows:
\begin{itemize}
    \item We introduce an important property of local surrogates: coverage. Coverage controls the region of interest and determines how local or global the surrogate is.
    \item We discuss and empirically illustrate the interplay between coverage, fidelity and complexity.
    \item We propose a new interactive workflow for local surrogate modelling that depends on considering ranges of localities, rather a fixed, predetermined one.
\end{itemize}
\section{From Global to Local Surrogate Explainers}\label{sec:local_surrogates}

\subsection{Global Surrogates}

\begin{figure}[!t]
    \centering
    \includegraphics[width=\textwidth]{figures/global_surrogates_comparison_2.png}
    \caption{Toy example that illustrates the trade-off between complexity and accuracy, faced by global surrogates. Complexity, in this case, is measured by depth and number of leaves. \textbf{Accuracy is measured with respect to the black-box model, and not the original data,} $\boldsymbol{X}$. In this toy example we do this in an exhaustive manner by considering a meshgrid of points $\boldsymbol{X}_s$ spanning the range of the data, and then obtaining predictions from the black-box $f(\boldsymbol{X}_s)$. (This procedure is not recommended in general, and is only carried out for illustration purposes.)}
    \label{fig:global_surrogates_2}
\end{figure}

Surrogates refer to models fitted on top of a given initial model $f$, trained on $(\boldsymbol{X}, \boldsymbol{y})$. A surrogate $g_{s}$ would then be trained on $(\boldsymbol{X}_{s}, f(\boldsymbol{X}_{s}))$, where it could be that $\boldsymbol{X}_{s} = \boldsymbol{X}$, or not. In what follows we consider $f$ to be either a black-box or highly complex model that we wish to approximate with $g \in G$, where $G$ is a interpretable model class, such as linear models, decision trees or decision rules.

Our assumptions on the high complexity of $f$ combined with the complexity restriction on $g$ give rise to {two, potentially conflicting, objectives: \textbf{fidelity} and \textbf{complexity}.} We want to obtain a simple interpretable surrogate $g$ that matches as close as possible the behaviour of a potentially complex model $f$.

Figure \ref{fig:global_surrogates_2} illustrates this trade-off: complexity (as measured by depth and number of leaves of a decision tree) against accuracy\footnotemark. On the other hand, accuracy is computed with respect to the black-box model, and not to the original data ($\boldsymbol{X}$) only. In this toy example, we measure accuracy in an exhaustive manner by considering a meshgrid of points $\boldsymbol{X}_s$ spanning the range of the data, and then obtaining predictions from the black-box $f(\boldsymbol{X}_s)$.
\footnotetext{In this work we use fidelity and accuracy interchangeably to refer to metrics that measure how well the local surrogate models the behaviour of the black-box. The metrics we consider are: (1) the proportion of correct predictions, and (2) true positive rate and true negative rate.}

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.45\textwidth]{figures/focus_locally.png}
    \caption{A black-box's decision boundary to illustrate how focusing on a smaller region  makes modelling easier. Approximating the black-box model in its entirety would be challenging for a linear model, but focusing on just the right region makes it possible to achieve a high level of fidelity with the desired level of complexity.}
    \label{fig:focus_locally}
\end{figure}

{But, what if we do not care about being accurate everywhere? If we \textit{only} care about a specific instance, can we do better than a global surrogate?} The motivation is that if instead of trying to model the black-box's behaviour \textit{everywhere}, we only model it regionally, then the task would require a surrogate model with a lower complexity. This is exemplified in Figure \ref{fig:focus_locally}. Approximating the black-box model in its entirety would be challenging for a linear model, but focusing on just the right region makes it possible to achieve a high level of fidelity, with the desired level of complexity. This is what local surrogate explainers aim to do. A key problem is \textit{how} to define the relevant region, and then proceed to model it.

\subsection{Local Surrogate Explainers}

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.55\textwidth]{figures/pareto_fronts_1.png}
    \caption{Pareto frontiers for different types of surrogates. This illustrates how the Pareto frontier for a global surrogate (blue) can be made more favourable by only focusing locally (green and orange).}
    \label{fig:pareto_fronts}
\end{figure}

Local surrogate explainers, as compared to global surrogate explainers, (implicitly) tackle the accuracy-complexity trade-off by first relaxing the objective of accuracy: from requiring the approximation to be accurate everywhere, to requiring it to be accurate \textit{locally} only. By reducing the focus from global to local -- as shown, for example, in Figure \ref{fig:focus_locally} -- we can shift the Pareto frontier further out, making conditions more favourable, as shown in Figure \ref{fig:pareto_fronts}. This Figure illustrates the advantages of reducing the region of interest for the surrogate, and considering a local, rather than a global model. The blue line presents a Pareto frontier for a global surrogate, for example the one presented in Figure \ref{fig:global_surrogates_2}. Green and oranges lines show Pareto frontiers for two local surrogates, that highlight that when moving from a global model to a local one, better accuracy can be achieved under the same complexity, or, alternatively, a model with lower complexity can be obtained with a similar accuracy score. For example, consider the second sub-plot from Figure \ref{fig:global_surrogates_2} that shows the decision boundary of a decision tree with \texttt{depth = 5} and \texttt{number of leaves = 7} achieving an accuracy of $0.92$. Then, a decision tree fitted on the enclosed region in Figure \ref{fig:focus_locally} with \texttt{depth = 5} and \texttt{number of leaves = 7} could achieve an accuracy of $\geq 0.92$, or that for a desired accuracy of $0.92$, a decision tree with less complexity would be needed. This, of course, does not come for free - reducing the area of focus comes with its own implications. Consider shifting from the global case to a very small region, such that you can fit an accurate model that is also interpretable. How useful would it be? Probably not much. This illustrates the need for the third objective: \textbf{coverage}. The preceding argument does not imply that more coverage is necessarily better, although it might as well be. For the same accuracy and complexity, one should be prefer a model with a higher coverage as it would be more general.
By moving from global coverage to partial/local coverage, we improve the Pareto frontier on accuracy and complexity.

To summarise, we identify the following properties of interest for local surrogates that we will further analyse empirically.

    \begin{enumerate}
        \item \texttt{\textbf{Complexity (interpretability)}.} While this depends on the application and the user, a few generally accepted model classes are decision trees, linear models and rule-based models. These are not always human-interpretable. A deep decision tree, or a linear model with many non-zero entries, can be difficult to interpret and offer little to no information. Therefore, complexity constraints might need to be considered.
        \item \texttt{\textbf{Fidelity (accuracy)}.} The fitted surrogate model should have as high fidelity as possible with respect to the black-box model.
        \item \texttt{\textbf{Coverage (region of interest)}.} In general, high coverage is preferable to low coverage, as it would make the explanations easier to generalise and more trustworthy.
    \end{enumerate}
The final objectives for local surrogate explainers will depend on these three ingredients under different formulations. Example scenarios could be as follows:

\begin{enumerate}
    \item For a fixed coverage, obtain the most accurate linear model with at most $k$ non-zero coefficients.
    \item For an accuracy that is greater than $x\%$, understand the interaction of coverage and complexity.
    \item For a linear model with at most $k$ non-zero coefficients, understand the interaction of coverage and accuracy.
\end{enumerate}

In the next section we present experiments to illustrate the practical interest of considering a wide range of localities and also show the interplay of the three properties -- coverage, fidelity and complexity. Finally, we build upon these to show how fitting multiple surrogate models -- instead of just one -- can lead to more informed decisions and extract meaningful interpretations. This is related to the field of uncertainty estimation in interpretable machine learning. In \cite{slack2020much} the authors estimate Credible Intervals (CIs) that capture the uncertainty associated with each feature importance in local explanations. They then use the CIs to inform sampling; with regards to the size of the sample and the location of the sampling. In \cite{gosiewska2019ibreakdown} authors use bootstrapping to approximate the distribution of variable contributions in local explanations, with the goal of understanding model level uncertainty. In our case, we demonstrate how bootstrapping \cite{bootstrap} can be used to provide uncertainty estimates for the learned models to enhance understanding.


\section{Introduction}

Businesses are increasingly turning to machine learning systems to automate and enhance their operations and decision-making. By making use of complex modeling techniques, they are able to create models with high and sometimes super-human predictive performance. However, given their complexity, these models are often used as black-boxes for which it is unclear how predictions are made.

As a consequence, the business environment is and will increasingly be confronted with demands regarding explainability for various reasons. On the one hand, companies face internal challenges with the adoption of automated decision making systems due to limited explainability (trust in model, business insights, internal control)~\cite{arrieta2020, samek2019}. Moreover, the question of the impact of ML/AI models on business processes and the society is more salient than ever: external players will oblige companies to put explainability on the agenda as a priority.

Hence, data scientists are and will be increasingly confronted with explainability demands from stakeholders beyond the traditional focus on predictive performance. This need of explainability ignited a whole new research field, often referred to as explainable artificial intelligence (XAI) or interpretable machine learning (IML). Over the past years, a multitude of methods has been proposed to explain machine learning models and decisions thereof~\cite{adadi2018, arrieta2020, molnar2020, 2018guidotti}. These methods significantly differ from each other in terms of output (explanation) and the way these explanations are generated and can therefore be more or less suitable for a specific use case and/or stakeholder.
There is thus a risk to fail to achieve a \textit{transparent} use of interpretability, which would be in contradiction with the initial objective of enlightening and empowering model's stakeholders. This issue has been noticed in previous works~\cite{bhatt2020, langer2021}, but a concrete methodology is lacking in the existing body of literature. The goal of this paper is to make an argument for the need of prescriptive guidance to match stakeholder needs with an appropriate explanation method. Moreover, we provide an overview of our ongoing work to introduce a methodology which is a first step towards bridging this gap.

This paper is divided in two parts.
The first one is devoted to the state of the art on implementing explainability in practice, from a user perspective and from a machine learning perspective.
We highlight the absence of thorough and sufficient cross-referencing between these two fields of research and in particular a clear methodology for a machine learning practitioner.
The second section proposes a reflection and a sketch of such a methodology for a sound scientific implementation of explainability in practice.

\section{Implementing explainability: current state of the literature}
\label{sec:current_state}

The question of practically using machine learning explainability has been poorly covered in the existing literature.
One notable exception is Bhatt et al.~\cite{bhatt2020}, which study the use of explanation methods in practice and thus show that, currently, they are mainly used by machine learning engineers in an \textit{ad hoc} way as sanity checks for the models they build and deploy.
One reason identified by the authors is that organizations lack frameworks for making decisions regarding explainability, leaving these methods only understandable by people with a background in machine learning and obscure to others.
This lack of focus on the perspective of the stakeholders to whom explainability is provided has also been observed by Langer et al.~\cite{langer2021}, who argue that more extensive stakeholder involvement is needed for the development and selection of explanation methods.
They provide a conceptual model of the problem space in which they make a distinction between the explainability needs from stakeholders (so-called desiderata) and the explanation method that is used to address them. In their model, the success of an explanation method depends on the extent to which this method satisfies these needs and the careful consideration of these needs is therefore crucial.

However, as the rather user/business-oriented explainability needs and the rather machine learning-oriented properties of explanation methods are completely different in nature, the mapping between the two is not straightforward, leaving the question of the use of XAI in practice unanswered.

Below, we give an overview of relevant work regarding the implementation of interpretability, which can thus be decomposed into two objectives: the collection of explainability needs from stakeholders and the selection of an appropriate explanation method. We conclude the section with a discussion on the lack of methodological guidance provided by the literature.

\subsection{Understanding stakeholder needs}
Typical categorizations of stakeholders are based on their role in an organization~\cite{tomsett2018, preece2018, bhatt2020, langer2021}, their machine learning experience~\cite{yu2018} or a combination of the two~\cite{suresh2021}. Also for the categorization of the stakeholder needs regarding explainability, different propositions are made in the literature. Some authors mention possible high-level goals of explainability, such as model debugging, monitoring etc.~\cite{bhatt2020} or revealing (un)known (un)knowns~\cite{preece2018}. Langer et al.~\cite{langer2021} provide a list of more detailed needs such as privacy, fairness, legal compliance, etc. Suresh et al.~\cite{suresh2021} propose an hierarchical framework of so-called tasks (e.g., understand influence of features on output), objectives (e.g., model debugging and compliance) and goals (e.g., trust and understanding) that contribute to each other~\cite{suresh2021}.

In order to gather the needs from a stakeholder in a specific use case, different methods are introduced. Most of them take an approach stemming from information systems and software development research, where the collection of user needs is a well-known and well-studied problem.
Köhl et al.~\cite{kohl2019} provide an analytical framework to elicitate and specify explainability needs and consider them as non-functional requirements which should be satisficed instead of satisfied. These can be translated into Softgoal Interdependency Graphs to represent their relationships. Other scholars claim that data scientists might have difficulties to directly discuss XAI solutions with stakeholders and, therefore, propose to rather focus on what a stakeholder wants to achieve with explainability~\cite{wolf2019, cirqueira2020}. In essence, they try to get a good understanding of the stakeholder's background, capabilities and goals. In this way, they create textual and/or visual scenarios that describe where and when explainability is needed.
Another approach is taken by Liao et al.~\cite{liao2020}, who assume that an explanation can be seen as an answer to one or more questions of the stakeholder. They provide a set of possible questions for which the stakeholder might require answers in the form of an explanation. Eiband et al~\cite{eiband2018} propose a process consisting of different (iterative) phases which incorporates the position of different stakeholders and in which the first phases focus on what must be explained and the later phases look into how this can be explained.

\subsection{Specifying explanation method properties}
On the other side of assessing stakeholder needs, different scholars propose a set of general characteristics or requirements to describe an explanation method. These allow to create a generalized identity card for explanation methods which can serve as documentation and a means to compare them. Both Hall et al.~\cite{hall2019} and Sokol and Flach.~\cite{sokol2020} introduced a framework of explanation method characteristics that can be filled in for a specific explanation method.

The framework of Hall et al.~\cite{hall2019} consists of characteristics that are divided over the dimensions effectiveness, versatility, constraints, explanator types and categories, explanation properties and personal considerations.
Sokol and Flach~\cite{sokol2020} provide a framework with five dimensions of so-called requirements of an explanation method. The functional requirements consists of requirements that determine whether it is practically feasible to use an explanation method for a specific use case. The operational requirements relates to the interaction of users with the system and their expectations. The usability requirements consists of properties that are of importance for the receiver of explanations. Finally, the safety and validation requirements respectively focus on aspects as privacy and security, and the validation of the explanation method.

Without going into further detail about the individual explanation method properties in both frameworks, it can be argued that these dimensions not only cover generic properties of explanation methods but also (possible) stakeholder needs (e.g., constraints and personal considerations in Hall et al.~\cite{hall2019}, and operational and usability requirements in Sokol and Flach~\cite{sokol2020}). Compared to the stakeholder needs categorizations we mentioned above, the proposed templates do not give an exhaustive overview of all possible (future) needs, so certain of them might not be assessed and/or addressed. Moreover, describing how an explanation method addresses a specific need in a general way might limit practical usability of the framework. For instance, privacy is a broad concept that might imply different needs in different use cases. It can be about disclosure of the model, certain instances, certain features, etc. Similarly, the complexity of an explanation is a concept that can (simultaneously) cover different aspects. It can be impacted by the size of an explanation, its format, the type of features used in it, etc. Filling in this type of characteristics for a specific approach might lead to a description that is too general to confront with a specific use case. Another option is to fill in the template completely tailored to the use case at hand, which limits reusability and therefore to redo the exercise in each use case.

\subsection{There is a lack of methodological guidance}
Given the explainability needs of a stakeholder and the properties of an explanation method, a logical next step is to assess whether the properties are capable of satisfying the needs. However, this is a cumbersome task, since there is no clear mapping between the properties and the needs.
First, the explainability needs of a stakeholder and the explanation method properties are from a completely different nature. The former come from the domain of the (business) stakeholder and focus on information needs and domain constraints such as privacy and complexity. The latter rather relate to the XAI domain and focus more on the technical (algorithmic) details of an explanation method. As both sides use their respective jargon, a translation in between is considered necessary.
Second, there might be a many-to-many relationship between needs and properties. A single property of an explanation method might contribute to multiple stakeholder needs, while the fulfillment of a single stakeholder need can be impacted by multiple and seemingly unrelated explanation method properties.

To our knowledge, the XAI literature lacks prescriptive work on how to perform the mapping between explainability needs and explanation method properties. Approaches on gathering explainability needs typically ignore the properties of explanation methods or only discuss it to a limited extent (e.g., Liao et al.~\cite{liao2020} map their questions to a high-level taxonomy of explanation methods). More general frameworks to select explanation methods do mention the fact that a translation or mapping between needs and properties is needed, but provide no concrete guidance on how this can be done. For instance, Hall et al.~\cite{hall2019} only mention the mapping between explanation methods and stakeholder needs as a step in their methodology. Langer et al.~\cite{langer2021} provide some hypothetical scenarios to inspire future research on the mapping. It can be argued that creating a generalized identity card for an explanation method is a useful tool for data scientists to select an appropriate approach for a use case. However, we believe that this identity card should consist of characteristics that are closely related to the details of the explanation generation process, irrespective of potential needs that might be satisfied by them.

In this work, we make a first step towards bridging this gap by proposing a methodology to translate explainability needs from stakeholders to the lower-level properties of explanation methods. This methodology aims at  guiding data scientists in the process of providing explainability to business stakeholders.

\section{Proposed methodology}
The goal of this work is to provide a (first step towards a) methodology that can guide a data scientist in tackling an explainability use case. We revisit both the  explainability needs and explanation method properties, in order to get them at a level of detail that allows to match them. Based on existing work, we define a template for them that can be used to document both aspects. The current versions of the templates are included in the appendix of this paper. Moreover, we plan to provide guidance on how the templates can be completed and how they can be used to find an appropriate explanation method for an explainability use case. An overview is shown in Figure~\ref{fig:methodology}.

\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{img/methodology.png}
\caption{Illustration of the proposed methodology and its different components.}
\label{fig:methodology}
\end{figure}

\subsection{Explanation method properties}
Using Hall et al.~\cite{hall2019} and Sokol et al.~\cite{sokol2020} as a starting point, we consider it valuable to create an identity card (later referred to as "ID card") to describe an individual explanation method in detail. However, we make a clear distinction with (and therefore not include) the explainability needs that might be satisfied by a method, thus differentiating it from currently existing versions~\cite{hall2019,sokol2020}. The current version of our identity card, which is included in Appendix~\ref{app:properties} for the sake of clarity, consists of properties that are organized into the four dimensions discussed below.

\subsubsection{Compatibility properties}
This set of properties allows to determine whether it is practically feasible to apply an explanation method to a certain use case. It covers aspects such as the type of the task, the scope of an explanation, the model and the training data.

\subsubsection{Explanation properties}
This set of properties describes the explanation that is produced by an explanation method. This comprises the information that is provided, the language and the relationship with the model and the real world.

\subsubsection{Method usage}
This section describes how an explanation method can be used in practical terms. It discusses whether code is available and how it can be used.

\subsubsection{Process properties}
This set of properties describe the process an explanation method uses to produce explanations. It covers aspects that are not necessarily visible from an explanation itself, but that can play an important role in which final explanation is obtained.

\subsection{Explainability needs}
We also propose a card that gives a detailed overview of stakeholder needs. To construct this document, we started from the existing literature discussed in Section~\ref{sec:current_state}. Subsequently, we selected needs that were considered relevant and detailed enough to work with throughout a brainstorm between the authors of this paper. For these needs, we created a questionnaire that can be used to reveal the needs from a stakeholder. The resulting needs card and questionnaire were further refined by applying them to actual use cases and by integrating the feedback. The current version of the needs card, which is included in Appendix~\ref{app:needs}, is organized in the four dimensions discussed below. The corresponding questionnaire is outlined in Appendix~\ref{app:questions}.

\subsubsection{Use case context}
This dimension gives an overview of the use case for which explainability is required. It contains the business process in which the ML system fits and a detailed description of the ML system itself.

\subsubsection{Stakeholder}
In order to provide explainability to a stakeholder, it is necessary to have a good understanding of his role and his background, both regarding the business domain and machine learning in general.

\subsubsection{Stakeholder needs}
This dimension bundles the aspects that express what information a stakeholder wants to derive from the explanation. In traditional information systems terms, this dimension considers the functional requirements of the explanation.

\subsubsection{Stakeholder constraints}
This dimension bundles aspects that put restrictions on the explanation (process) that are relevant to the stakeholder. In traditional information systems term, these can be called the non-functional requirements of an explanation.

\subsection{Information collection}
The completion of the template for the explanation method properties is considered a task the research community that is in charge of providing explanation methods. Existing and future explanation methods should be subjected to a thorough investigation in order to assess their properties. To this end, experts can rely on existing documentation (research papers, reports, code base etc.) and experience by experimenting with the method. We acknowledge this is a cumbersome task, but we believe that this documentation, if properly composed, will form a valuable and comparable knowledge base that can be reused for future use cases.

To gather the information on stakeholder needs for a specific use case, the XAI expert should probe the relevant stakeholder. Since certain aspects might be more technical in nature, it is considered necessary to also involve the people that are responsible for the development and deployment of the model that is used in the use case (if they are not the stakeholder themselves). Our questionnaire can be used as a tool to guide this process, but also more approaches that are more tailored to the specific stakeholder might be explored (see the literature we discussed in Section~\ref{sec:current_state}).

\subsection{Matching stakeholder needs with explanation method properties}

After the collection and documentation of the stakeholder needs, the next step is to confront them with the knowledge base of detailed explanation method properties in order to select the (most) appropriate explanation method(s). As argued in Section~\ref{sec:current_state}, their is no clear relation between stakeholder needs and corresponding explanation method properties that can be generalized over use cases. However, the separation of the two sides makes their different nature explicit and aids to prevent the oversimplification of the problem. This should encourage the XAI expert to perform a thorough analysis and resulting mapping for each use case. While acknowledging that the current specifications of the stakeholder needs and explanation method properties are not final and should be further revised and refined, we believe they provide the XAI expert with useful tools in that process.

We advocate to document in detail the analysis and the resulting choice of explanation method(s). This encourages the XAI expert to show to what extent the stakeholder needs are addressed and to point at disadvantages or risks that are related to using an specific explanation method. This should result in a substantiated and nuanced advice for a specific use case.

\section{Ongoing and Future Research}

In this paper, we discussed our on-going work of designing a methodology to help XAI stakeholders choosing an explainability method. Besides identifying and discussing the issue and the current state of the art, our contributions include a prototype of ID cards and questionnaire to identify user needs (both provided in Appendix).

The following step in this research project is to apply this methodology in practice.
First, we plan on enriching the current knowledge base of ID cards with the most common explanation methods and their properties.
Alongside, we are currently conducting interviews with stakeholders working on specific real-life use cases requiring interpretability. These semi-structured interviews allow us to validate and refine the current stakeholder needs specification and the corresponding questionnaire, which is still an ongoing process. By considering use cases from diverse domains and involving different (types of) stakeholders (e.g. data scientists, non-expert customers, auditors...), we plan on covering as many different needs as possible. In particular, we also pay close attention to the extent to which stakeholders generally can or cannot answer questions related to a specific need.

The next step is to perform the analysis to match the stakeholder needs with an appropriate explanation method. We will do so for each use case collected in the interviews and present a (set of) explanation method(s) that can be considered a suitable option. Subsequently, the proposed solution(s) will be discussed with and demonstrated to the stakeholder as a validation step. This will allow us to assess the effectiveness and discover the limitations of our methodology.
By submitting this paper to the XKDD workshop, we hope to have interesting discussions with leading XAI researchers to challenge our in-the-making  methodology and thus help solve what we believe to be a fundamental issue of the field today.

\bibliographystyle{splncs04}

\section{Explanation method properties (ID cards)}
\label{app:properties}

\subsection{Compatibility properties}

\begin{itemize}
    \item Problem supervision level: unsupervised, supervised, reinforcement
    \item Problem type: classification, regression, clustering, etc.
    \item Explanation target: data, model, predictions
    \item Explanation scope: instance-level or global
    \item Applicable model class: model-agnostic or model-specific
    \item Ante-hoc (by design) of post-hoc (a posteriori)
    \item Compatible data types: tabular, image, text, etc.
    \item Compatible feature types: numerical, ordinal, categorical
    \item Training data-agnostic or training data access needed
\end{itemize}

\subsection{Explanation properties}
\begin{itemize}
    \item Explanation family: association between antecedent and consequent, contrasts and differences, causal mechanisms
    \item Explanatory medium: representation of explanation
    \item Explanation domain: original feature domain or transformed domain
\end{itemize}

\subsection{Method usage}
\begin{itemize}
    \item Code availability: source code
    \item Inputs: detailed description of arguments
    \item Outputs: detailed description of return values
\end{itemize}

\subsection{Process properties}
\begin{itemize}
    \item Computational complexity: big-O notation or based on empirical evaluation
    \item Fidelity: extent to which underlying model is closely mimicked
    \item Optimality: exact or heuristic explanation generation
    \item Sparsity: size of explanation
    \item Coverage: guarantee of explanation
    \item Plausibility: extent to which explanation is within data manifold
    \item Consistency: extent to which similar instances get similar explanations
    \item Stability: extent to which explanation generation is subject to randomness
    \item Interactiveness: extent to which end user can interact with explanation generation
\end{itemize}

\section{Explainability needs}
\label{app:needs}

\subsection{Use case context}
\begin{itemize}
    \item Business process and role of ML system
    \item Model inputs (data and features)
    \item Model output
    \item Model type
\end{itemize}

\subsection{Stakeholder}
\begin{itemize}
    \item Role in the organization and the business process
    \item Background regarding machine learning in general and regarding the application domain
\end{itemize}

\subsection{Stakeholder needs}

\begin{itemize}
    \item Information needs: information that must be provided by explanation to answer questions from stakeholder
    \item Language of the explanation
    \begin{itemize}
        \item Format type: textual/visual/…
        \item Feature specificity: fine-grained features vs. higher-level features
        \item All model features vs. subset of features
    \end{itemize}
    \item Truthfulness to model: extent to which an approximation of the model is allowed.
    \item Faithfulness to real world
    \begin{itemize}
        \item If actionability: feasibility of actions
        \item If case: realisticness of case (in data distribution)
    \end{itemize}

\end{itemize}

\subsection{Shared stakeholder constraints}
\begin{itemize}
    \item Timeliness: time frame in which explanation must be provided
    \item Explanation complexity: degree of difficulty that can be processed by stakeholder within given time frame
    \item Privacy / Intellectual property: disclosure of instances, features or model reasoning
    \item Robustness: extent to which explanations can differ between instances, models and over time
\end{itemize}

\clearpage
\section{Questionnaire to reveal needs}
\label{app:questions}

\subsection{Use case context}

\begin{itemize}
    \item What is the business process and where does the ML system fit?
    \item What are the model inputs?
    \item What are the model outputs?
    \item What type of model is used?
\end{itemize}

\subsection{Stakeholder}

\begin{itemize}
    \item Can you describe the stakeholder and his role in the business process?
    \item Background:
    \begin{itemize}
        \item Do the individual model features make sense to the stakeholder?
        \item What is the ML background of the stakeholder?
    \end{itemize}
\end{itemize}

\subsection{Stakeholder needs}

\begin{itemize}
    \item Information needs
    \begin{itemize}
        \item Does stakeholder want general insight in the system or insight for a specific case?
        \item What does stakeholder want to achieve/do with the explanation?
        \item If global:
        \begin{itemize}
            \item Does stakeholder want to know which features are most/least important?
            \item Does stakeholder want to have insight in the general logic of the system?
            \item Does stakeholder want to have (a-)typical instances?
        \end{itemize}
        \item If local:
        \begin{itemize}
            \item Does stakeholder want to know the logic of the model regarding this decision?
            \item Does stakeholder want to know which features contributed the most/least?
            \item Does stakeholder want information on how to change the decision (actionability)?
            \item Does stakeholder want to know how changes to the instance affect the decision?
            \item Does stakeholder want to see comparable or opposite cases?
        \end{itemize}
    \end{itemize}

    \item Language of the explanation
    \begin{itemize}
        \item Does stakeholder prefer a textual explanation, visual explanation, rule etc.?
        \item Does stakeholder want an explanation in terms of the individual features, or is an higher abstraction level considered sufficient or desirable?
        \item Is it allowed to only consider a subset of understandable features and ignore other features that might also have impacted decision-making?
    \end{itemize}

    \item Truthfulness to model
    \begin{itemize}
        \item Is using an inherently transparent model for this use case an option?
        \item Is it allowed to make an approximation, meaning that the explanation might not be completely truthful to the model?
    \end{itemize}

    \item Faithfulness to real world
    \begin{itemize}
        \item In case stakeholder wants actionability: Do the guidelines need to be actionable/realistic for the stakeholder?
        \item In case stakeholder wants case or counterfactual: Should instance be close to data distribution
    \end{itemize}
\end{itemize}

\subsection{Stakeholder constraints}
\begin{itemize}
    \item Timeliness
    \begin{itemize}
        \item Does stakeholder always need an explanation?
        \item If not, how quickly does stakeholder want an explanation after requesting one?
    \end{itemize}

    \item Explanation complexity
    \begin{itemize}
        \item How large can an explanation be?
        \item How much time does stakeholder have to process the explanation?
    \end{itemize}

    \item Privacy / IP
    \begin{itemize}
        \item Are there model features that cannot be shown?
        \item Can (a part of) the model logic be disclosed?
        \item Can other data instances be shown?
    \end{itemize}

    \item Robustness
    \begin{itemize}
        \item Should similar instances have similar explanations?
        \item Should explanation for certain instance or model be the same if model changes over time?
    \end{itemize}
\end{itemize}