---
abstract: 'Concept-based explainability aims to fill the model interpretability gap for non-technical humans-in-the-loop. Previous work has focused on providing concepts for specific models (e.g, neural networks) or data types (e.g., images), and by either trying to extract concepts from an already trained network or training self-explainable models through multi-task learning. In this work, we propose ConceptDistil, a method to bring concept explanations to any black-box classifier using knowledge distillation. Our method uses a surrogate neural network that approximates the predictions of a black-box classifier while simultaneously producing concept explanations. We validate our proposed concept-based knowledge distillation explainer in a real world use-case, showing that it achieves alignment with the black-box classifier while attaining high performance on the explainability task, providing high-level domain explanations. Concept-based explanations aims to fill the model interpretability gap for non-technical humans-in-the-loop. Previous work has focused on providing concepts for specific models (e.g, neural networks) or data types (e.g., images), and by either trying to extract concepts from an already trained network or training self-explainable models through multi-task learning. In this work, we propose ConceptDistil, a method to bring concept explanations to any black-box classifier using knowledge distillation. ConceptDistil is decomposed into two components: (1) a concept model that predicts which domain concepts are present in a given instance, and (2) a distillation model that tries to mimic the predictions of a black-box model using the concept model predictions. We validate ConceptDistil in a real world use-case, showing that it is able to optimize both tasks, bringing concept-explainability to any black-box model.'
author:
- |
    João Bento, Ricardo Moreira, Vladimir Balayan, Pedro Saleiro, Pedro Bizarro\
    Feedzai\
    `{michellevasquez@example.org`\
bibliography:
- 'pair2struct.bib'
title: 'ConceptDistil: Model-Agnostic Distillation of Concept Explanations'
---

Introduction {#sec:introduction}
============

![ConceptDistil explaining a black-box applied to the fraud detection domain.<span data-label="fig:teaser"></span>](images/FinalTeaser.png){width="95.00000%"}

no nosso dominio o recall é imperativo, e neste momento nao se dá 1% recalll de recall no nosso caso de finantial domains é lightgbm que ganha -meter valores

queremos concept explainability porque é um hot topic, entao criamos o conceptTab para poder trazer concepte xplainability to lgbm models, e conseuqnetemente all black boxes

ao contrario de imagens, em tabular nao é facil aplicar conceitos em tabular a melhor cenas é lgbm, e concep expl é boa forma entao queremos explicar o melhor modelo em tabular, lgbms/boosting,

queremos trazer concept explainability ao state of the art of tabular data

meter referencias ao longo da intro front page + metade da proxima

New Paper: ConceptDistil

Intro: - Popular methods focus on feature attribution -&gt; bad for the HITL -&gt; concept explainability is the way to go; ✔ - Previous work focused on global concept explainability, also not good for HITL who needs to make real-time decisions; ✔ - Previous methods focus on creating concept explainability within the classification model (self-explainable). Due to the known trade-offs of MTL (ref. Multi-Task Learning as Multi-Objective Optimization), this will generally result in poorer classification performance. ✔ - Another problem for concept explainability is the lack of labels (mainly in the tabular domain -&gt; hard to identify and costly to pay experts) ✔ - Moreover, for the tabular domain, it's not clear that Deep Learning is the SOTA in terms of performance (all the refs about the issue) - Our method removes the burden of the concept explainability from the classification model and instead builds a surrogate model which approximates the classifier while producing concept explanations. This has 2 advantages: the classification model keeps it's performance; concept explainability becomes model agnostic (crucial for the tabular domain). The problem of faithfulness might arise, but it's possible to assess that through the real-time measurement of fidelity. If fidelity is assured, the explanations are faithful. - Furthermore, we propose a method for obtaining concept labels through weak supervision - All things considered, we can say that we finally cracked all problems of concept explainability for critical decision making in the tabular domain; - We test our approach in the financial fraud application

Contributions: - Surrogate - Application in financial fraud detection

Abstract: More concrete about the contributions

Method:

Experimental Setup: - In the financial fraud use-case we verified that LigthGBM was the best in terms of performance, se, we'll distillate that

Concept-based explainability is a promising research direction in explainable AI that proposes the use of high-level domain explanations (*e.g.*, fraudulent patterns for fraud analysts, such as "Suspicious Email", "Suspicious Items"). These explanations encompass complex data patterns into recognizable domain concepts, allowing non-ML technical humans-in-the-loop to interact with ML models with ease.

Previous work [@Kim2018; @Ghorbani2019; @Koh2020conceptbottleneck; @li2018deep; @melis2018towards; @nanda2020unifying] focuses on the image domain, presenting itself as attractive for concept-explanations as concepts can be more clearly identified on images by domain experts (*e.g.* Doctors on medical imaging) or even non-experts (*e.g.* Paws on a bird). Due to this, datasets for concept-explainability in the image domain (*e.g.* CUB [@WelinderEtal2010]) usually have a high amount of concept labels.

Mainstream methods to interpret black-box models produce explanations based on feature-attribution coefficients [@Lundberg2017; @Ribeiro2016; @Sundararajan2017]. While useful for a technical persona (*e.g.*, a data scientist), these low-level explanations are difficult to grasp for most humans-in-the-loop (*e.g.*, medical doctors, fraud analysts). Recently, concept-based explainability has been a promising line of research in which explanations are based on the importance of high-level semantic concepts instead of model features.

Previous work on concept-based explainability, such as TCAV or ACE [@Kim2018; @Ghorbani2019], rely on *post-hoc* learning of concepts and subsequent assessment of their importance for the prediction of a given class. Although relevant for a global understanding of a model, this family of methods does not help humans-in-the-loop making individual decisions (*e.g.*, fraud analyst that discern an individual transaction). Another family of concept-based explainability methods aims to produce instance-level concept explanations by changing the model learning itself. These approaches use multi-task learning consisting of a main classification task and an explainability task [@balayan2020teaching; @Koh2020conceptbottleneck; @li2018deep; @melis2018towards; @nanda2020unifying]. Although promising, these approaches attain sub-par performance in the main classification task due to known trade-offs in multi-task learning [@sener2018multi].

The aforementioned approaches are model-specific as they they assume a structure based on neural networks (NN). However, for applications that work with tabular data, gradient boosting models are still quite popular  [@borisov2021deep; @shwartz2022tabular; @shavitt2018regularization; @gorishniy2021revisiting; @popov2019neural]. This urges the need for a concept explainability method for any type of model.

In this work we propose ConceptDistil, illustrated in Figure \[fig:teaser\], which introduces concept explainability as an independent component without changes to the classification model. It builds a distillation model that approximates the classifier while also producing concept explanations. The advantages of this approach are two fold: the classification model maintains its original performance, and concept explainability becomes model-agnostic, which is crucial in the context of tabular data. ConceptDistil is divided in two sequential components, a concept model which predicts which domain concepts are present on a given instance, and an attention-based distillation model that approximates the predictions of the black-box classifier. The distillation model explains the classifier as a weighted sum of the concept model's predictions. This offers two separate explanations types: (1) a data explanation that lists which concepts are present on a given instance and (2) a classifier explanation which gives the contributions of each concept to the classification score.

We validate our method in a real-world fraud detection tabular dataset. We show that ConceptDistil is able to approximate the behaviour of the explained model while still achieving high performance on the explainability task.

The main contributions of our work are:

-   We propose a method for concept explainability that does not compromise classification performances;

-   Our method is model-agnostic which makes it applicable to a wider range of domains;

-   Our method provides both data explanations and concept score contributions which correspond to model explanations.

Bullet points intro: Explain what is concept explainability: - bridges the gap between non-techinical HITL and ML models - encompass data relationships present in the data - domain knowledge

Concept explainability is popular in vision: - referencias para os papers de vision - Concepts are easy to identify + labels are easy to come by (non-experts can annotate)

concept explainability applied to tabular data is not as popular: - referencia para balayan et all - it presents hard challenges: - Concepts are hard to identify - labels are hard to come by

Another challenge to apply concept-explainability to tabular data is that that best architectures are boosting-based models (referencia para esse paper). This is a challenge as most literature focuses on applying concept explainability to DL models (referencias para os N papers)

If we want to explain the best models for tabular data, we need to explain both DL models and boosting-type models, as such, we propose ConceptTab - describe concept tab - describe our experiments - describe our findings

Concept-based knowledge distillation explainer {#sec:surrogate_exp}
==============================================

Concept-based explainability bridges the gap between non-ML technical humans-in-the-loop and ML-models, by providing high-level, human-grounded, domain explanations. These explanations are especially desirable in our domain, as fraud analysts who interact with the model in their decision making process might not be ML-knowledgeable to understand low-level explanations. Additionally, concept-based explanations can encompass complex interactions between several features, which can be difficult to understand in a tabular domain.

Regarding predictive tasks on tabular domains, gradient-boosting-based models, (e.g. LightGBM) are documented to out-perform deep learning based models. In our specific financial domain, we corroborate this finding with LightGBM models reaching 72.4% recall at 5% FPR while deep learning models achieved 68% recall at 5% FPR. When considering the financial domain, predictive performance is the single metric of interest. As a result, stakeholders are reluctant to trade performance for explainability, restraining our explainability approach to one that does not hinder model performance.

We propose a concept-based knowledge distillation explainer that is able to explain a black-box ML model by distilling its knowledge into concept-based explanations. This surrogate explainer is framed as an hierarchical multi-task network, where one task is responsible to predict domain concepts to serve as explanations (yellow Data Explainer and $\displaystyle \Ls_E$ in \[fig:concepttab\]) and as the inputs for the knowledge distillation task (blue Distillation Model and $\displaystyle \Ls_{KD}$ in \[fig:concepttab\]). This design forces the surrogate to approximate the complex black-box model behaviours using as input the domain concept predictions. One of the main requirements for this approach is a training set with concept annotations. Those could be obtained with manual annotation process, which is very ardours and sometimes unfeasible, or through a weak supervision technique, producing hard or soft (probabilistic) concept labels.

**Preliminaries.** We frame $D$ as a binary classification task, $\displaystyle \vy_D \in \sY_D = \{0, 1\}$ and $\mathcal{C}$ as a black-box binary classification model of the form $\displaystyle \mathcal{C}: \sX \rightarrow \sY_D$ where $\mathcal{C}(\vx) = \hat{\vy}_D$, where the random variable $\displaystyle (\vx, \vy_D)$ has an unknown joint distribution in $\sX \times \sY_D$ and $\hat{\vy}_D$ is the classifier estimate of $\vy_D$. We want to explain $\mathcal{C}$ by distilling its knowledge into an explainer $f$ that jointly predicts domain concepts to serve as data explanations as well as the behaviour (score) of the explained model.

We frame our explainer as a multi-task surrogate model with two individual tasks: a knowledge distillation task, $KD$, where $f$ learns to approximate the classification scores of the black-box $\mathcal{C}$ given by $\displaystyle \vy_{KD} \in \sY_{KD} = [0, 1]$, and a multi-label classification task, $E$, with targets $\displaystyle \vy_E \in \sY_E = \{0, 1\}^K$, where $f$ learns to predict $k$ concepts which serve as the data explanations. We define our explainer as $\displaystyle f: \sX \rightarrow \sY_{E} \times \sY_{KD}$, where $f(\vx) = (\hat{\vy}_E , \hat{\vy}_{KD})$ is the $(k+1)$-dimensional output vector of $k$ concept predictions plus the predicted knowledge distillation score. We seek to maximize the explainer's performance in both tasks during training. Let $\displaystyle \Ls_{KD}(\hat{\vy}_{KD}, \mathcal{C}(\vx))$ and $\displaystyle \Ls_E(\hat{\vy}_E, \vy_E)$ represent the losses incurred by the model at the knowledge distillation and explainability tasks, respectively. The surrogate explainer model $f$ minimizes the weighted combination of both losses, as defined by: $$\label{eq:final_loss}
\displaystyle \Ls(\hat{\vy}, \vy) = \lambda \, \Ls_{KD}(\hat{\vy}_{KD}, \mathcal{C}(\vx)) + (1 - \lambda) \, \Ls_E(\hat{\vy}_E, \vy_E)$$ Given the inherent fidelity-explainability trade-off, we may assume $\displaystyle \lambda \in [0, 1]$ as a hyperparameter to weight the relative importance of the knowledge distillation task with respect to the explainability task. A high $\lambda$ allows for a higher fidelity of $f$ to the actual predictions of $\mathcal{C}$ at the cost of lower performance on the explainability task. On the contrary, a low $\lambda$ allows for $f$ to have an higher explainability performance but at the risk of it being less coupled with the black-box model $\mathcal{C}$. We show the effect of this parameter on appendix \[appendix:hyperparams\_lambda\]. Depending on the nature of each task, different loss functions may be used. In this work, we opted for the Binary Cross-Entropy (BCE) loss as the $\displaystyle \Ls_{KD}$ and used the mean BCE loss over the $K$ concepts as the $\displaystyle\Ls_E$. More formally: $$\displaystyle\Ls_E(\hat{\vy}_E, \vy_{E}) = \frac{1}{K}\sideset{}{_{i=0}^K}\sum \text{CE}(\hat{\vy}_E^{(i)}, \vy_E^{(i)})$$

![Diagram of ConceptDistil architecture and its main components.<span data-label="fig:concepttab"></span>](images/JOELInputAttention.png){width="85.00000%"}

<span>.6</span> ![ConceptDistil explanation output example. <span data-label="fig:explanation_example"></span>](images/CDInput.png "fig:"){width="100.00000%"}

<span>.36</span> ![ConceptDistil explanation output example. <span data-label="fig:explanation_example"></span>](images/Explanation_Example.png "fig:"){width="1\linewidth"}

We propose the explainer architecture illustrated in Figure \[fig:concepttab\], composed by $C$ feedforward (FF) layers, parameterized by $\vtheta_{C}$, which are shared across all concepts, $M \times K$ layers, parameterized by $\{\vtheta_{M}^{(i)}\}_{i=1}^{K}$ which are concept-specific, and $A$ layers, parameterized by $\vtheta_{A}$, which are responsible for the attention-based distillation task.

The output of the explainability task is $\displaystyle \hat{\vy}_E$, which can be divided into $K$ components for each concept $i$, of the form: $$\displaystyle \hat{\vy}_E^{(i)} = h(h(\vx; \vtheta_{C}); \vtheta_{M}^{(i)})$$ Where $\displaystyle h(\vx; \vtheta)$ represents the neurons activation resulting from inputting $\displaystyle \vx$ into a network parameterized by $\displaystyle \vtheta$. The output of the knowledge distillation task is $\hat{\vy}_{KD} = \sum_{i=0}^{K} \hat{\vy}_E^{(i)} \times \alpha_i$, where $\alpha_i$ represents the attention coefficient for the $i$-th concept, calculated by a simple attention mechanism [@bahdanau2014neural]. This attention mechanism is parameterized by $\vtheta_{A}$ and receives as input the same input $\vx$ as the black-box model and the concept model. The attention coefficients are then defined as:

$$\begin{gathered}
    \alpha_i = softmax(e_i, e)\\
    e = h(\vx; \vtheta_A)\end{gathered}$$

**Model and Data Explainability.** The aforementioned architecture, illustrated on Figure \[fig:concepttab\], is therefore a composition of two models: (1) a concept model, consisting of $C$ common layers and $M \times K$ concept-specific layers, that predicts which concepts are present on a given input; and (2) a distillation model, composed of $A$ layers, which, for a given input, attributes attention-based importances to the predicted concepts in order to obtain the black-box assigned score. Considering the two models, our method produces two types of explanations which can be provided to users: (1) the concepts present on a given input, as predicted by the concept model; (2) how much each concept prediction is relevant to obtain the final black-box model output score.

**Task Contamination.** One problem that arise from jointly learning **ConceptDistil** is the problem of task contamination. This phenomena happens when, to improve the knowledge distillation task, the gradients force the already correctly learned concept model to degrade its explainability performance. To erase the possibility of task contamination we propose an alternative, named **ConceptDistil – No gradient**, where the propagation of gradients back from the attention mechanism into the concept model is blocked. This allows ConceptDistil to separate the knowledge distillation and the explainability tasks while training. We further propose another alternative to overcome task contamination, dubbed as **ConceptDistil – 2-staged**, where we train the two components of ConceptDistil in separate stages. In this training framework, the concept model is trained on the concept labels, and only after is the distillation model trained using as inputs the predictions of the already trained concept model.

Dizer que desta forma a attention n ta a poluir o data explainer.

Finalizar isto o que sao as explicaçoes - data explainer, explica os dados attention está a alinhar o data explainer com o modelo

Falar das varias explainability informations - que é o resultado do attention weight, com a prediction etc

Experimental setup {#sec:experimental_setup}
==================

Use-case: - We are working in financial fraud use-case where - In the financial fraud use-case we verified that LigthGBM was the best in terms of performance, se, we'll distillate that

Data: - Fraud Dataset ✔️ (how it's sampled and why) ✔ - Golden Dataset (why it's a small dataset) ✔ - Concept labels (how are being generated - weak supervision) ✔

- To evaluate the fraud performance, we train and evaluate on the full dataset. (mention this in the specific section of the results) ✔

Models and Hyperparameter optimization: ✔ - Architectures used - TPE search - Nr of trials - Ranges of Parameters varied

Baselines: ✔ - Only concepts - Surrogate model

Evaluation Metrics: ✔ - Fraud -&gt; Recall@5%FPR - Distillation -&gt; 1 - Mean Absolute Error (Abs. Fidelity) - Explainability -&gt; Mean ROC AUC across concepts

We validate our method in real world financial fraud detection application where the data is typically in the tabular form. In our use-case, predictive performance is often the single metric of interest. Therefore, it is crucial to deploy the model with maximum performance. Thus, we ran an experiment (details in Appendix \[appendix:tabular\_performance\]) to find models with higher performance for the predictive task. Therefore, we decided to evaluate our ConceptDistil approaches using the best LightGBM and FeedForward Neural Network (FFNN) models as the base classifiers. **Dataset.** For our experiments, we used a privately held online retailer fraud detection dataset, comprised of approximately 5.5M transactions where 2% represent fraudulent behaviour. The dataset is then divided into three sequential subsets: train, validation and test, composed of 4.8M, 200k and 1M instances respectively. Because our goal is model explainability, we created a sample of the transactions which would most likely need to be explained, containing the transactions of highest uncertainty for the historical fraud detection system. This sample resulted in 460k instances for training, 17k for validation and 107k for test. Our ConceptDistil models were trained and tested using these sampled datasets.

**Golden Concept Labels.** To evaluate our explainability task, we randomly sampled a small set from the sampled dataset, and presented those transactions to a group of human experts (fraud analysts with the knowledge of the most common fraud patterns latent on this online retailer dataset). We then asked the fraud analysts to annotate the existing concepts on each transaction. We present more details about this golden concept dataset in Appendix Section \[appendix:golden\_dataset\]

**Concept Weak Supervision.** To train our explainability models using the full and sampled datasets we employed a weak supervision methodology. This consisted of creating a set of *Concept Teachers* of the form $\displaystyle f: \sX \rightarrow \sY_{E}$ trained using a small golden training set with concept labels obtained with the method described above. The *Concept Teachers* were then used to infer the probabilistic labels $p(\vy_E^{(i)}|\vx)$ of the concept $i$ being present in transaction $\vx$. Our explainability models were trained using these probabilistic labels as targets. We further detail how we developed our *Concept Teachers* in the Appendix Section \[appendix:exp\_setup\].

**Hyperparameter Optimization.** We selected the best hyperparameters using a Tree-structured Parzen Estimator (TPE) [@bergstra2011algorithms] optimization of 50 trials for each variant. We further detail the hyperparameters in appendix \[appendix:hyperparams\].

**Evaluation Metrics.** To evaluate the knowledge distillation task we used a measure of fidelity derived from the Mean Absolute Error (MAE) given by: $$fidelity = 1 - MAE = 1 - \frac{\sum_{i=1}^{n} |\hat{y}_{KD}^{(i)}-y_{KD}^{(i)}|}{n},$$ where $n$ is the total number of instances in the evaluation set. Because our knowledge distillation task approximates a classification score between 0 and 1, we can also see that this measure is always within the same interval. To evaluate the explainability task, because we do not wish to focus on any specific range of False Positive Rate (FPR), we measure the area under the ROC curve (ROC AUC) of each individual concept. We then take the average of these measures across concepts to obtain a single evaluation measure for the explainability task. The explainability task evaluation is performed on the small golden set of human-annotated concept labels. To evaluate the performance of models on the classification task (fraud task) we consider the True Positive Rate (TPR) (or *Recall*), at 5% FPR. We select 5% FPR because it was the target metric for the The fraud performance evaluations were done using the full test dataset (not sampled) since we don't want to focus solely on the subset of transactions which require good explanations (FP, FN, uncertain) but instead on the whole set of transaction upon which the fraud model would take an approve/decline/review decision. We provide results for this task in Appendix Section \[appendix:tabular\_performance\].

Baselines
---------

Along with our ConceptDistil models, we created 2 baseline models which help us understand the performance bounds on the two individual tasks.

**Concept Explainabilty Baseline.** Our first baseline is a model with the architecture of the concept data explainer from ConceptDistil which, given an input $\vx$, predicts the probability $p(\vy_E|\vx)$. When evaluated on the golden test set, this model gives us an upper bound of the explainability performance. Additionally, the best model obtained for the concept model baseline is the one used for the 2-staged ConceptDistil.

**Knowledge Distillation baseline** The second baseline model is a standard surrogate which, given an input $\vx$, predicts the output of the black-box classifier $\vy_{KD}$. This standard surrogate has a multi-layer architecture where the number of layers were varied between 4 and 9 and the layer dimensions were varied between 2 and 4096. When evaluated on the sampled test set, this standard surrogate model gives us an upper bound for the knowledge distillation task performance. The hyperparameters of both baseline models were optimized using the same methodology and ranges described above.

Results
=======

Results of best performing LGBM model vs the best self-explainable in terms of fraud.

Table with results for the surrogate models along with the distillation (FF surrogate) and explainability (only concepts) baselines.

Examples of explanations obtained for the fraud use-case

  **Baselines**                                                **Fidelity (%)**   **Mean ROC AUC (%)**
  ----------------------------------------------------------- ------------------ ----------------------
  Standard MLP (only knowledge distillation task)                 **93.31**                -
  Concept model (only explainability task)                            -                **78.12**
  **Variants**                                                                   
  ConceptDistil (with leakage)                                      91.47                 73.9
  ConceptDistil (without leakage)                                   88.46                77.35
  Two-staged surrogate (pre-trained concept data explainer)         86.86                78.12

  : Performance comparison between different variations of proposed concept-based knowledge distillation explainers.

\[tab:results\_table\]

In total, we trained around 1600 models, and summarized the obtained results in the Table \[tab:results\_table\]. Along with our ConceptDistil models, we created 2 baseline models which help us understand the performance bounds on the two individual tasks. We detail an implementation for those baselines in appendix \[appendix:baselines\].

The results of Table \[tab:results\_table\] show a clear difference in distillation performance between the two black-box classifiers used. The fidelity performance achieved when distilling the FFNN model is higher for all surrogate architectures tested. We believe that the distillation of the FFNN is a simpler task when compared with the distillation of LightGBM due to the similarity of inductive biases between the FFNN classifier and the surrogate architectures. **Baselines comparison.** We can see in Table \[tab:results\_table\] that our best Distillation baseline achieved 93.31% fidelity when distilling the LightGBM classifier and 98.44% when distilling the FFNN classifier, which is superior to the best fidelity obtained by ConceptDistil explainers (91.43% for LightGBM and 97.10% for FFNN), showing approximately a 1.9 and 1.3 percentage points (pp) gap in fidelity for the LightGBM and FFNN distillation, respectively. The biggest gap in fidelity (approximately 6.1 pp) occurred for the ConceptDistil – *2-staged* when distilling the LightGBM classifier. In terms of the explainability baseline, the best model attains 78.12% of mean ROC AUC. Since the ConceptDistil – *2-staged* uses this explainability baseline as its own data explainer, its explainability performance is the same as the baseline. In contrast, for jointly learned ConceptDistil models, there is a drop of 1.6 pp and 1 pp in explainability performance when distilling LightGBM and FFNN, respectively. These results confirm the existence of a trade-off between knowledge distillation and explainability tasks since both single-task baselines outperform the proposed multi-task approaches.

  ------------------------------- ---------------------------- ----------------------------
                                   **Knowledge Distillation**   **Concept Explainability**
  **Models**                            **Fidelity (%)**           **Mean ROC AUC (%)**
  Standard Surrogate (baseline)            **93.31**                        -
  Concept model (baseline)                     -                        **78.12**
  ConceptDistil – 2-staged                   86.86                        78.12
  ConceptDistil – No gradient                88.46                        77.35
  ConceptDistil                              91.47                        73.90
  ------------------------------- ---------------------------- ----------------------------

  : Performance comparison between different concept-based knowledge distillation explainers.

\[tab:results\_table\]

  ------------------------------- -------------- ----------------- -------------- -----------------
                                   **Fidelity**      **Mean**       **Fidelity**      **Mean**
                                     **(%)**      **ROC AUC (%)**     **(%)**      **ROC AUC (%)**
  Distillation model (baseline)     **93.31**            -           **98.44**            -
  Concept model (baseline)              -            **78.12**           -            **78.12**
  ConceptDistil – 2-staged            88.07            78.12           94.73            78.12
  ConceptDistil – No gradient         87.20            76.28           95.26            77.58
  ConceptDistil                       91.43            76.43           97.10            77.16
  ------------------------------- -------------- ----------------- -------------- -----------------

  : Performance comparison between different concept-based knowledge distillation explainers.

\[tab:results\_table\]

  -- ---------------------- ----------- ------------- ----------- -------------
                             Fidelity       Mean       Fidelity       Mean
                                (%)      ROC AUC (%)      (%)      ROC AUC (%)
     Distillation Model      **93.31**        -        **98.44**        -
     Explainability Model        -        **78.12**        -        **78.12**
     Default                 **91.43**      76.43      **97.10**      77.16
     2-staged                  88.07      **78.12**      94.73      **78.12**
     No gradient               87.20        76.28        95.26        77.58
  -- ---------------------- ----------- ------------- ----------- -------------

  : Performance comparison between different concept-based knowledge distillation explainers.

\[tab:results\_table\]

**ConceptDistil – 2-staged vs ConceptDistil – No gradient.** When comparing the results of these variants, we see that, when distilling the LightGBM, the *No gradient* variant performs worse than the *2-staged* on both tasks. When distilling the FFNN, the *No gradient* achieves better distillation but worse explainability which corresponds to a softer trade-of between distillation and explainability than the *2-staged*. - Similar explainability performance (no grad slightly worse due to bigger search space) - On distillation performance no grad is 1.8 pp better which is unexpected since the *2-staged* trains with a static concept model which should allow for a more stable distillation learning. - We hypothesize that the distillation model of the jointly learned (no grad) enjoys more regularization from the Dropout and Batch Normalization of the concept model training

**ConceptDistil – 2-staged/No gradient vs ConceptDistil.** When comparing *2-staged* and *No gradient* variants with the vanilla ConceptDistil, we see that the *2-staged* achieves higher explainability performance while vanilla ConceptDistil attains better fidelity when distilling the LightGBM model. For the FFNN, vanilla ConceptDistil attains better fidelity, without much sacrifice in explainability performance (0.96 pp loss). **ConceptDistil – No gradient vs ConceptDistil.** When comparing the *No gradient* variant with the Standard ConceptDistil, we again observe the aforementioned trade-off between the knowledge distillation and explainability tasks. Table \[tab:results\_table\] shows better explainability performance for the *No gradient* variant (4.7 pp) when compared to the Standard ConceptDistil, but with lower in fidelity (3.0 pp). This interaction between the two metrics, is again due to the Standard ConceptDistil allowing the information flow from knowledge distillation task into the concept predictions. These concept predictions will in turn be tuned for a better knowledge distillation task at the cost of the concept explainability task.

Conclusion
==========

We present a method for obtaining concept explainability for any black-box classifier. In contrast with previous approaches to concept explainability, our method is a surrogate model which swaps the classification task a for knowledge distillation task, never compromising the classification performance. This is specially important in applications where classification performance not negotiable when introducing explainability

We present ConceptDistil, a concept-based knowledge distillation method that brings concept explainability to any black-box classifier. ConceptDistil is decomposed into two sub-models: (1) a concept model that predicts which domain concepts are present on a given instance, and (2) a distillation model that predicts the behaviour of a black-box model using the concept predictions. We validated ConceptDistil on a privately held real-world online retailer fraud detection dataset, and showed that ConceptDistil is able to optimize both tasks, bringing concept-explaianbility to any black-box model.

We present ConceptDistil, a concept-based knowledge distillation method that provides both data and model explanations. ConceptDistil uses a surrogate neural network that approximates the predictions of a black-box classifier while simultaneously producing concept explanations. As our method operates independently from the black-box model, it is model-agnostic and does not compromise the classification task performance. We validated ConceptDistil on a privately held real-world online retailer fraud detection dataset, and showed that ConceptDistil can learn both the explainability and the distillation tasks.

We propose a method for concept explainability that does not compromise classification performances;

Our method is model-agnostic which makes it applicable to a wider range of domains;

Our method provides both data explanations and concept score contributions which correspond to model explanations.

Appendix
========

Predictive performance on real world financial crime dataset {#appendix:tabular_performance}
------------------------------------------------------------

We ran experiments using private real-world online retailer fraud detection dataset that contains approximately 5.5M transactions. This dataset is highly unbalanced, with only 2% of transactions being fraudulent. We used TPE as optimization algorithm, running 50 iterations for each model type. To evaluate the performance of models on the classification task (fraud task) we consider the True Positive Rate (TPR) (or *Recall*), at 5% FPR. This percentage of FPR (5%) were picked due the business constrains defined by online retailer, minimizing the probability to harm an legitimate customer. The fraud performance evaluations were done using the full test dataset (not sampled) since we don't want to focus solely on the subset of transactions which require good explanations (FP, FN, uncertain) but instead on the whole set of transaction upon which the fraud model would take an approve/decline/review decision. The results for the best model of each type are presented in Table \[tab:tabular\_results\].

  **Model**           **Recall at 5% FPR (%)**      **Out-of-the-box explanations**     
  ------------------ -------------------------- --------------------------------------- --
  LightGBM                     68.31                Feature Importance Attributions     
  FFNN                         65.55                               -                    
  Self-explainable             60.59             Concepts (through Concept bottleneck)  

  : Performance comparison between gradient-boosting-based, FFNN, and self-explainable models ([@balayan2020teaching; @Koh2020conceptbottleneck]).

\[tab:tabular\_results\]

As already mentioned in section \[sec:introduction\], there are several studies which show that Deep Learning models struggle to outperform gradient-boosting-based models in tabular data. In this specific fraud detection use-case, we corroborate this finding with LightGBM model [@ke2017lightgbm] achieving 68.31% recall at the 5% FPR level, while FFNN achieved 65.55% recall at 5% FPR. Additionally, we tested a concept self-explainable model [@balayan2020teaching; @Koh2020conceptbottleneck], which achieved worse performance than both the best LighGBM and the FFNN, achieving 60.59% recall at 5% FPR. In this work, we test our ConceptDistil approaches using the best LightGBM and FFNN models as the base classifiers.

Concept Golden Dataset {#appendix:golden_dataset}
----------------------

In section \[sec:experimental\_setup\] we briefly describe the golden dataset of concepts that we use in our experiments. This dataset was obtained by collecting concept annotations in a total of 2643 transactions. These were further split into 1934 for training the *Concept Teachers*, 203 for the optimization of the teachers' parameters using TPE, and 506 for test. This last set was the one used to evaluate the explainability performance of the *Concept Teachers* and all ConceptDistil models and baselines presented. Despite the requirement for having concept labels being a limitation in this work, we can see that the effort of concept labelling can be mitigated by resourcing to a weak supervision strategy as the one used. We reduce the labeling effort in more than 3 orders of magnitude when compared to the effort of fully labeling the sampled dataset of around 500k transactions. Additionally, and despite the effort drawbacks, we consider that concept explainability can benefit from being establishing with ground truth concepts obtained from expert knowledge instead of unsupervised methodologies which can more easily result in meaningless concept definitions.

Concept Teachers {#appendix:exp_setup}
----------------

As described in section \[sec:experimental\_setup\] our Weakly Supervised Concept Labeling consisted in learning a set of *Concept Teachers*, which generated concept labels for all the remaining unlabeled transactions in the full and sampled datasets. To train the *Concept Teachers* we used the golden dataset enriched not only with the fraud features available at run-time but also with some additional information, such as fraud rule triggers and manual decisions, which is only generated during or after the historical fraud decision. The algorithm used for the *Concept Teachers* was a Random Forest classifier and the hyperparameters were optimized using a Tree-structured Parzen Estimator (TPE) [@bergstra2011algorithms] algorithm with 200 trials from which 30 trails were used for a random initialization. As the optimization metric we used ROC AUC on the golden validation set. We then evaluated the *Concept Teachers* in the test set of our golden dataset obtaining a mean ROC AUC of 75.72% across all concepts. Despite not being central to our work, the *Concept Teachers* performance shows and interesting result when compared to the 78.12% performance of our concept explainabilty baseline. This 2.4 pp improvement originates solely from the using more data in $\mathbb{X}$ (the concept explainability baselin trained in 460k instances labeled by our *Concept Teachers*) which illustrates that our weak supervision approach is successful to some degree.

Concept Prevalences {#appendix:concept_prevs}
-------------------

To provide more context about the concepts considered in this work for the Fraud Detection, we show in Table \[tab:concept\_prevs\_table\] the list of concepts as well as their prevalences on the golden dataset (Concept Teachers training data plus the golden test set).

  **Concept**              **Global (%)**   **Legit (%)**   **Fraud (%)**
  ----------------------- ---------------- --------------- ---------------
  Good Customer History        24.45            27.85           14.29
  High Speed Orderning         11.33            8.49            19.84
  Suspicious Delivery          22.86            20.16           30.95
  Suspicious Device            11.73            8.75            20.63
  Suspicious Email             21.07            18.93           30.16
  Suspicious Items             18.49            17.24           22.22

  : Concepts for the Fraud Detection use-case and corresponding prevalences on the golden dataset. The table shows the global prevalence and the prevalences conditioned on fraud classification labels (legit and fraud).

\[tab:concept\_prevs\_table\]

Hyperparameter search {#appendix:hyperparams}
---------------------

For the hyperparameter optimization, we varied the number of $C$ layers between 3 and 5, the number of $M$ layers between $[3,7]$ and the number of $A$ layers between 1 and 4 as well as their respective layer dimensions between 2 and 2048. We let our loss trade-off parameter $\lambda$ vary between 0.2 and 0.8, and the learning rate vary between 0.0005 and 0.01. As regularization, we let all hidden layers dropout probabilities vary between 0 and 0.4, the l2 weight penalization parameter vary between 0 and 0.1 and let the optimizer select whether to use or not Batch Normalization [@ioffe2015batch].

### Effect of $\lambda$ {#appendix:hyperparams_lambda}

To understand the effect of the $\lambda$ parameter used in the loss function (equation \[eq:final\_loss\]), we ran an experiment where we test jointly learning ConceptDistil on best LightGBM model (\[appendix:tabular\_performance\]) with same hyperparameter space described in previous paragraph, with small difference for $\lambda$ that now vary between $[0,1]$. Instead of using TPE for hyperparameter optimization which may be biased due to its sequential optimization approach, we opt for random search. In total we trained around 400 models, and present the result on figure \[fig:lambda\_effect\].

![The effect of $\lambda$ parameter on Absolute fidelity (Knowledge Distillation task) and Mean ROC AUC (Explainability task). Bigger size points are on the Pareto frontier @pareto1906manuale.<span data-label="fig:lambda_effect"></span>](images/lambda_effect_study.png){width="70.00000%"}

By analysing the bottom left area of the figure \[fig:lambda\_effect\] (absolute fidelity higher than 85%), we observe that higher values of $\lambda$, close to 1, lead to higher performance on Knowledge Distillation task. On the contrary, the top left area (Mean ROC AUC higher than 70%) is mainly populated with models that have small lambda values (close to 0). Thus, these models have higher weight on Explainability task, leading to better Mean ROC AUC values, but sacrificing absolute fidelity. The models that have a $\lambda$, values closed to 0.5, seems to achieve the better trade-off between two tasks, and are dominant on the Pareto frontier.

Baselines {#appendix:baselines}
---------

We develop two baseline models that are specialized on two individual tasks (explainability and distillation).

**Concept Explainabilty Baseline.** Our first baseline is a model with the architecture of the concept model from ConceptDistil which, given an input $\vx$, predicts the probability $p(\vy_E|\vx)$. When evaluated on the golden test set, this model gives us an upper bound of the explainability performance. Additionally, the best model obtained for the concept model baseline is the one used for the ConceptDistil – 2-staged.

**Knowledge Distillation baseline** The second baseline model is a standard surrogate (dubbed as Distillation baseline) which, given an input $\vx$, predicts the output of the black-box classifier $\vy_{KD}$. This standard surrogate has a multi-layer FFNN architecture where the number of layers were varied between 4 and 9 and the layer dimensions were varied between 2 and 4096. When evaluated on the sampled test set, this standard surrogate model gives us an upper bound for the knowledge distillation task performance.

The hyperparameters of both baseline models were optimized using the same methodology and ranges described in appendix \[appendix:hyperparams\].

{'source': 'arXiv_out/2205'}