abstract: Interpretable machine learning tackles the important problem that humans cannot understand the behaviors of complex machine learning models and how these models arrive at a particular decision. Although many approaches have been proposed, a comprehensive understanding of the achievements and challenges is still lacking. We provide a survey covering existing techniques to increase the interpretability of machine learning models. We also discuss crucial issues that the community should consider in future work such as designing user-friendly explanations and developing comprehensive evaluation metrics to further push forward the area of interpretable machine learning.
author:
Mengnan Du, Ninghao Liu, Xia Hu

bibliography: bibs4cacm.bib
title: Techniques for Interpretable Machine Learning [^1]

# Introduction

Machine learning is progressing at an astounding rate, powered by complex models such as ensemble models and deep neural networks (DNNs). These models have a wide range of real-world applications, such as movie recommendations of Netflix, neural machine translation of Google, speech recognition of Amazon Alexa. Despite the successes, machine learning has its own limitations and drawbacks. The most significant one is the lack of transparency behind their behaviors, which leaves users with little understanding of how particular decisions are made by these models. Consider, for instance, an advanced self-driving car equipped with various machine learning algorithms doesn't brake or decelerate when confronting a stopped firetruck. This unexpected behavior may frustrate and confuse users, making them wonder why. Even worse, the wrong decisions could cause severe consequences if the car is driving at highway speeds and might finally crash the firetruck. The concerns about the black-box nature of complex models have hampered their further applications in our society, especially in those critical decision-making domains like self-driving cars.

*Interpretable machine learning* would be an effective tool to mitigate these problems. It gives machine learning models the ability to explain or to present their behaviors in understandable terms to humans , which is named interpretability or explainability and we use them interchangeably in this paper. Interpretability would be an indispensable part for machine learning models in order to better serve human beings and bring benefits to society. For end-users, explanation will increase their trust and encourage them to adopt machine learning systems. From the perspective of machine learning system developers and researchers, the provided explanation can help them better understand the problem, the data and why a model might fail, and eventually increase the system safety. Thus there is a growing interest among the academic and industrial community in interpreting machine learning models and gaining insights into their working mechanisms.

Interpretable machine learning techniques can generally be grouped into two categories: intrinsic interpretability and post-hoc interpretability, depending on the time when the interpretability is obtained . Intrinsic interpretability is achieved by constructing self-explanatory models which incorporate interpretability directly to their structures. The family of this category includes decision tree, rule-based model, linear model, attention model, etc. In contrast, the post-hoc one requires creating a second model to provide explanations for an existing model. The main difference between these two groups lies in the trade-off between model accuracy and explanation fidelity. Inherently interpretable models could provide accurate and undistorted explanation but may sacrifice prediction performance to some extent. The post-hoc ones are limited in their approximate nature while keeping the underlying model accuracy intact.

Based on the above categorization, we further differentiate two types of interpretability: global interpretability, and local interpretability. Global interpretability means that users can understand how the model works globally by inspecting the structures and parameters of a complex model, while local interpretability locally examines an individual prediction of a model, trying to figure out why the model makes the decision it makes. Using the DNN in Figure  as an example, global interpretability is achieved by understanding the representations captured by the neurons at an intermediate layer, while local interpretability is obtained by identifying the contributions of each feature in a specific input to the prediction made by DNN. These two types bring different benefits. Global interpretability could illuminate the inner working mechanisms of machine learning models and thus can increase their transparency. Local interpretability will help uncover the causal relations between a specific input and its corresponding model prediction. Those two help users trust a model and trust a prediction, respectively.

In this article, we first summarize current progress of three lines of research for interpretable machine learning: designing inherently interpretable models (including globally and locally), post-hoc global explanation, and post-hoc local explanation. We proceed by introducing applications and challenges of current techniques. Finally, we present limitations of current explanations and propose directions towards more human-friendly explanations.

# Intrinsic Interpretable Model

Intrinsic interpretability can be achieved by designing self-explanatory models which incorporate interpretability directly into the model structures. These constructed interpretable models either are globally interpretable or could provide explanations when they make individual predictions.

## Globally Interpretable Model

Globally interpretable models can be constructed in two ways: directly trained from data as usual but with interpretability constraints, and being extracted from a complex and opaque model.

### Adding Interpretability Constraints

The interpretability of a model could be promoted by incorporating interpretability constraints. Some representative examples include enforcing sparsity terms or imposing semantic monotonicity constraints in classification models . Here sparsity means that a model is encouraged to use relatively fewer features for prediction, while monotonicity enables the features to have monotonic relations with the prediction. Similarly, decision trees are pruned by replacing subtrees with leaves to encourage long and deep trees rather than wide and more balanced trees . These constraints make a model simpler and could increase the model's comprehensibility by users.

Besides, more semantically meaningful constraints could be added to a model to further improve interpretability. For instance, interpretable convolutional neural networks (CNN) add a regularization loss to higher convolutional layers of CNN to learn disentangled representations, resulting in filters that could detect semantically meaningful natural objects . Another work combines novel neural units, called capsules, to construct a capsule network . The activation vectors of an active capsule can represent various semantic-aware concepts like position and pose of a particular object. This nice property makes capsule network more comprehensible for humans.

However, there are often trade-offs between prediction accuracy and interpretability when constraints are directly incorporated into models. The more interpretable models may result in reduced prediction accuracy comparing the less interpretable ones.

### Interpretable Model Extraction

An alternative is to apply interpretable model extraction, also referred as mimic learning , which may not have to sacrifice the model performance too much. The motivation behind mimic learning is to approximate a complex model using an easily interpretable model such as a decision tree, rule-based model, or linear model. As long as the approximation is sufficiently close, the statistical properties of the complex model will be reflected in the interpretable model. Eventually, we obtain a model with comparable prediction performance, and the behavior of which is much easier to understand. For instance, the tree ensemble model is transformed into a single decision tree . Moreover, a DNN is utilized to train a decision tree which mimics the input-output function captured by the neural network so that the knowledge encoded in DNN is transferred to the decision tree . To avoid the overfitting of the decision tree, active learning is applied for training. These techniques convert the original model to a decision tree with better interpretability and maintain comparable predictive performance at the same time.

## Locally Interpretable Model

Locally interpretable models are usually achieved by designing more justified model architectures that could explain why a specific decision is made. Different from the globally interpretable models that offer a certain extent of transparency about what is going on inside a model, locally interpretable models provide users understandable rationale for a specific prediction.

A representative scheme is employing attention mechanism , which is widely utilized to explain predictions made by sequential models, e.g., Recurrent Neural Networks (RNNs). Attention mechanism is advantageous in that it gives users the ability to interpret which parts of the input are attended by the model through visualizing the attention weight matrix for individual predictions. Attention mechanism has been used to solve the problem of generating image caption . In this case, a CNN is adopted to encode an input image to a vector, and an RNN with attention mechanisms is utilized to generate descriptions. When generating each word, the model changes its attention to reflect the relevant parts of the image. The final visualization of the attention weights could tell human what the model is looking at when generating a word. Similarly, attention mechanism has been incorporated in machine translation . At decoding stage, the neural attention module added to neural machine translation (NMT) model assigns different weights to the hidden states of the decoder, which allows the decoder to selectively focus on different parts of the input sentence at each step of the output generation. Through visualizing the attention scores, users could understand how words in one language depend on words in another language for correct translation.

# Post-hoc Global Explanation

Machine learning models automatically learn useful patterns from a huge amount of training data and retain the learned knowledge into model structures and parameters. Post-hoc global explanation aims to provide a global understanding about what knowledge has been acquired by these pre-trained models, and illuminate the parameters or learned representations in an intuitive manner to humans. We classify existing models into two categories: traditional machine learning and deep learning pipelines (see Figure ), since we are capable of extracting some similar explanation paradigms from each category. We introduce below how to provide explanation for these two types of pipelines.

## Traditional ML Explanation

Traditional machine learning pipelines mostly rely on feature engineering, which transforms raw data into features that better represent the predictive task, as shown in Figure . The features are generally interpretable and the role of machine learning is to map the representation to output. We consider a simple yet effective explanation measure which is applicable to most of the models belonging to traditional pipeline, called *feature importance*, which indicates statistical contribution of each feature to the underlying model when making decisions.

### Model-agnostic Explanation

Model-agnostic feature importance is broadly applicable to various machine learning models. It treats a model as a black-box and does not inspect internal model parameters.

A representative approach is *Permutation Feature Importance* . The key idea is that the importance of a specific feature to the overall performance of a model can be determined by calculating how the model prediction accuracy deviates after permuting the values of that feature. More specifically, given a pre-trained model with $n$ features and a test set, the average prediction score of the model on the test set is $p$, which is also the baseline accuracy. We shuffle the values of a feature on the test set and compute the average prediction score of the model on the modified dataset. This process is iteratively performed for each feature and eventually $n$ prediction scores are obtained for $n$ features respectively. We then rank the importance of the $n$ features according to the reductions of their score comparing to baseline accuracy $p$. There are several advantages for this approach. First, we do not need to normalize the values of the hand-crafted features. Second, it can be generalized to nearly any machine learning models with hand-crafted features as input. Third, this strategy has been proved to be robust and efficient of implementation.

### Model-specific Explanation

There also exists explanation methods specifically designed for different models. Model-specific methods usually derive explanations by examining internal model structures and parameters. We introduce below how to provide feature importance for two families of machine learning models.

**Generalized linear models** GLM is constituted of a series of models which are linear combination of input features and model parameters followed by feeding to some transformation function (often nonlinear) . Examples of GLM includes linear regression, logistic regression, etc. The weights of a GLM directly reflect feature importance, so users can understand how the model works by checking their weights and visualizing them. However, the weights may not be reliable when different features are not appropriately normalized and vary in their scale of measurement. Besides, the interpretability of an explanation will decrease when the feature dimensions become too large, which may be beyond the comprehension ability of humans.

**Tree-based ensemble models** Tree-based ensemble models, such as gradient boosting machines, random forests and XGBoost , are typically inscrutable to humans. There are several ways to measure the contribution of each feature. The first approach is to calculate the accuracy gain when a feature is used in tree branches. The rationale behind is that without adding a new split to a branch for a feature, there may be some misclassified elements, while after adding the new branch, there are two branches and each one is more accurate. The second approach measures the feature coverage, i.e., calculating the relative quantity of observations related to a feature. The third approach is to count the number of times that a feature is used to split the data.

## DNN Representation Explanation

DNNs, in contrast to traditional models, not only discover the mapping from representation to output, but also learn *representations* from raw data , as illustrated in Figure . The learned deep representations are usually not human interpretable , hence the explanation for DNNs mainly focuses on understanding the representations captured by the neurons at intermediate layers of DNNs. In the following, we introduce explanation methods for two major categories of DNN, i.e., CNN and RNN.

### Explanation of CNN Representation

There has been a growing interest to understand the inscrutable representations at different layers of CNN. Among different strategies to understand CNN representations, the most effective and widely utilized one is through finding the preferred inputs for neurons at a specific layer. This is generally formulated in the *activation maximization* (AM) framework , which can be formulated as: $$\textbf{x}^* = \operatorname*{argmax}_{ \textbf{x} } \textbf{f}_{l} (\textbf{x}) - \mathcal{R}(\textbf{x}),$$ where $\textbf{f}_{l} (\textbf{x})$ is the activation value of a neuron at layer $l$ for input $\textbf{x}$, and $\mathcal{R}(\textbf{x})$ is a regularizer. Starting from random initialization, we optimize an image to maximally activate a neuron. Through iterative optimization, the derivatives of the neuron activation value with respect to the image is utilized to tweak the image. Eventually, the visualization of the generated image could tell what individual neuron is looking for in its receptive field. We can in fact do this for arbitrary neurons, ranging from neurons at the first layer all the way to the output neurons at the last layer, to understand what is encoded as representations at different layers.

While the framework is simple, getting it to work faces some challenges, among which the most significant one is the surprising artifact. The optimization process may produce unrealistic images containing noise and high-frequency patterns. Due to the large searching space for images, if without proper regularization, it is possible to produce images that satisfy the the optimization objective to activate the neuron but are still unrecognizable. To tackle this problem, the optimization should be constrained using natural image priors so as to produce synthetic images which resemble natural images. Some researchers heuristically propose hand-crafted priors, including total variation norm, $\alpha$-norm, Gaussian blur, etc. In addition, the optimization could be regularized using stronger natural image priors produced by a generative model, such as GAN or VAE, which maps codes in the latent space to the image spaces . Instead of directly optimizing the image, these methods optimize the latent space codes to find an image which can activate a given neuron. Experimental results have shown that the priors produced by generative models lead to significant improvements in visualization.

The visualization results provide several interesting observations about CNN representations. First, the network learns representations at several levels of abstraction, transiting from general to task-specific from the first layer to the last layer. Take the CNN trained with the ImageNet dataset for example. Lower-layer neurons detect small and simple patterns, such as object corners and textures. Mid-layer neurons detect object parts, such as faces, legs. Higher-layer neurons respond to whole objects or even scenes. Interestingly, the visualization of the last layer neurons illustrates that CNN exhibits a remarkable property to capture global structure, local details, and contexts of an object. Second, a neuron could respond to different images that are related to a semantic concept, revealing the multifaceted nature of neurons . For instance, a face detection neuron can fire in response to both human faces and animal faces. Note that this phenomenon is not confined to high layer neurons, all layers of neurons are multifaceted. The neurons at higher layers are more multifaceted than the ones at lower layers, indicating that higher-layer neurons become more invariant to large changes within a class of inputs, such as colors and poses. Third, CNN learns distributed code for objects . Objects can be described using part-based representations and these parts can be shared across different categories.

### Explanation of RNN Representation

Following numerous efforts to interpret CNN, uncovering the abstract knowledge encoded by RNN representations (including GRUs and LSTMs) has also attracted increasing interest in recent years. Language modeling, which targets to predict the next token given its previous tokens, is usually utilized to analyze the representations learned by RNN. The studies indicate that RNN indeed learns useful representations  .

First, some work examines the representations of the last hidden layer of RNN and study the function of different units at that layer, by analyzing the real input tokens that maximally activate a unit. The studies demonstrate that some units of RNN representations are able to capture complex language characteristics, e.g., syntax, semantics and long-term dependencies. For instance, a study analyzes the interpretability of RNN activation patterns using character-level language modeling . This work finds that although most of the neural units are hard to find particular meanings, there indeed exist certain dimensions in RNN hidden representations that are able to focus on specific language structures such as quotation marks, brackets, and line lengths in a text. In another work, a word-level language model is utilized to analyze the linguistic features encoded by individual hidden units of RNN . The visualizations illustrate that some units are mostly activated by certain semantic category, while some others could capture a particular syntactic class or dependency function. More interestingly, some hidden units could carry the activation values over to subsequent time steps, which explains why RNN can learn long-term dependencies and complex linguistic features.

Second, the research finds that RNN is able to learn hierarchical representations by inspecting representations at different hidden layers . This observation indicates that RNN representations bear some resemblance to their CNN counterpart. For instance, a bidirectional language model is constructed using a multi-layer LSTM . The analysis of representations at different layers of this model shows that the lower-layer representation captures context-independent syntactic information. In contrast, higher-layer LSTM representations encode context-dependent semantic information. The deep contextualized representations can disambiguate the meanings of words by utilizing their context, and thus could be employed to perform tasks which require context-aware understanding of words.

# Post-hoc Local Explanation

After understanding the model globally, we zoom in to the local behavior of the model and provide local explanations for individual predictions. Local explanations target to identify the contributions of each feature in the input towards a specific model prediction. As local methods usually attribute a model's decision to its input features, they are also called *attribution* methods. In this section, we first introduce model-agnostic attribution methods and then discuss attribution methods specific to DNN-based predictions.

## Model-agnostic Explanation

Model-agnostic methods allow explaining predictions of arbitrary machine learning models independent of the implementation. They provide a way to explain predictions by treating the models as black-boxes, where explanations could be generated even without access to the internal model parameters. They bring some risks at the same time, since we cannot guarantee that the explanation faithfully reflects the decision making process of a model.

### Local Approximation Based Explanation

Local approximation based explanation is based on the assumption that the machine learning predictions around the neighborhood of a given input can be approximated by an interpretable white-box model. The interpretable model does not have to work well globally, but it must approximate the black-box model well in a small neighborhood near the original input. Then the contribution score for each feature can be obtained by examining the parameters of the white-box model.

Some studies assume that the prediction around the neighborhood of an instance could be formulated as the linearly weighted combination of its input features . Attribution methods based on this principle first sample the feature space in the neighborhood of the instance to constitute an additional training set. A sparse linear model, such as Lasso, is then trained using the generated samples and labels. This approximation model works the same as a black-box model locally but is much easier to inspect. Finally, the prediction of the original model can be explained by examining the weights of this sparse linear model instead.

Sometimes, even the local behavior of a model may be extremely non-linear, linear explanations could lead to poor performance. Models which could characterize non-linear relationship are thus utilized as the local approximation. For instance, a local approximation based explanation framework can be constructed using if-then rules . Experiments on a series of tasks show that this framework is effective at capturing non-linear behaviors. More importantly, the produced rules are not confined merely to the instance being explained and often generalize to other instances.

### Perturbation Based Explanation

This line of work follows the philosophy that the contribution of a feature can be determined by measuring how prediction score changes when the feature is altered. It tries to answer the question: which parts of the input, if they were not seen by the model, would most change its prediction? Thus, the results may be called counterfactual explanations. The perturbation is performed across features sequentially to determine their contributions, and can be implemented in two ways: omission and occlusion. For omission, a feature is directly removed from the input, but this is impractical in practice since few models allow setting features as unknown. As for occlusion, the feature is replaced with a reference value, such as zero for word embeddings or specific gray value for image pixels. Nevertheless, occlusion raises a new concern that new evidence may be introduced and that can be used by the model as a side effect . For instance, if we occlude part of an image using green color and then we may provide undesirable evidence for the *grass* class. Thus we should be particularly cautious when selecting reference values to avoid introducing extra pieces of evidence.

## Model-specific Explanation

There are also explanation approaches exclusively designed for a specific type of model. Below we introduce DNN-specific methods, which treat the networks as white-boxes and explicitly utilize the interior structure to derive explanations. We divide them into three major categories: back-propagation based methods in a top-down manner; perturbation based methods in a bottom-up manner; investigation of deep representations in intermediate layers.

### Back-propagation

Back-propagation based methods calculate the gradient, or its variants, of a particular output with respect to the input using back-propagation to derive the contribution of features. In the simplest case, we can back-propagate the gradient . The underlying hypothesis is that larger gradient magnitude represents a more substantial relevance of a feature to a prediction. Other approaches back-propagate different forms of signals to the input, such as discarding negative gradient values at the back-propagation process , or back-propagating the relevance of the final prediction score to the input layer . These methods are integrated into a unified framework where all methods are reformulated as a modified gradient function . This unification enables comprehensive comparison between different methods and facilitates effective implementation under modern deep learning libraries, such as TensorFlow and PyTorch. Back-propagation based methods are efficient in terms of implementation, as they usually need a few forward and backward calculations. On the other hand, they are limited in their heuristic nature and may generate explanations of unsatisfactory quality, which are noisy and highlight some irrelevant features, as shown in Figure  (b).

### Mask Perturbation

Model-agnostic perturbation mentioned in the previous section could be computationally very expensive when handling an instance with high dimensions, since they need to sequentially perturb the input. In contrast, DNN-specific perturbation could be implemented efficiently through mask perturbation and gradient descent optimization. One representative work formulates the perturbation in an optimization framework to learn a perturbation mask, which explicitly preserves the contribution values of each feature . Note that this framework generally needs to impose various regularizations to the mask to produce meaningful explanation rather than surprising artifacts . Although the optimization based framework has drastically boosted the efficiency, generating an explanation still needs hundreds of forward and backward operations. To enable more computationally efficient implementation, a DNN model can be trained to predict the attribution mask . Once the mask neural network model is obtained, it only requires a single forward pass to yield attribution scores for an input.

### Investigation of Deep Representations

Either perturbation or back-propagation based explanations ignore the intermediate layers of the DNN that might contain rich information for interpretation. To bridge the gap, some studies explicitly utilize the *deep representations* of the input to perform attribution.

Based on the observation that deep CNN representations capture the high-level content of input images as well as their spatial arrangement, a guided feature inversion framework is proposed to provide local explanations . This framework inverts the representations at higher layers of CNN to a synthesized image, while simultaneously encodes the location information of the target object in a mask. Decomposition is another perspective to take advantage of deep DNN representations. For instance, through modeling the information flowing process of the hidden representation vectors in RNN models, the RNN prediction is decomposed into additive contribution of each word in the input text . The decomposition result could quantify the contribution of each individual word to a RNN prediction. These two explanation paradigms achieve promising results on a variety of DNN architectures, indicating that the intermediate information indeed contributes significantly to the attribution. Besides, deep representations serve as a strong regularizer, increasing the possibility that the explanations faithfully characterize the behaviors of DNN under normal operating conditions. Thus it reduces the risks of generating surprising artifacts and leads to more meaningful explanations.

# Applications

Interpretable machine learning has numerous applications. We introduce three representative ones: model validation, model debugging, and knowledge discovery.

## Model Validation

Explanations could help to examine whether a machine learning model has employed the true evidences instead of biases which widely exist among training data. A post-hoc attribution approach, for instance, analyzes three question answering models . The attribution heatmaps show that these models often ignore important part of the questions and rely on irrelevant words to make decisions. They further indicate that the weakness of the models is caused by the inadequacies of training data. Possible solutions to fix this problem include modifying training data or introducing inductive bias when training the model. More seriously, machine learning models may rely on gender and ethnic biases to make decisions . Interpretability could be exploited to identify whether models have utilized these biases to ensure models don't violate ethical and legal requirements.

## Model Debugging

Explanations also can be employed to debug and analyze the misbehavior of models when models give wrong and unexpected predictions. A representative example is adversarial learning . Recent work demonstrated that machine learning models, such as DNNs, can be guided into making erroneous predictions with high confidence, when processing accidentally or deliberately crafted inputs . However, these inputs are quite easy to be recognized by humans. In this case, explanation facilitates humans to identify the possible model deficiencies and analyze why these models may fail. More importantly, we may further take advantage of human knowledge to figure out possible solutions to promote the performances and reasonability of models.

## Knowledge Discovery

The derived explanations also allow humans to obtain new insights from machine learning model through comprehending their decision making process. With explanation, the area experts and the end-users could provide realistic feedbacks. Eventually, new science and new knowledge which are originally hidden in the data could be extracted. For instance, a rule-based interpretable model has been utilized to predict the mortality risk for patients with pneumonia . One of the rules from the model suggests that having asthma could lower a patient's risk of dying from pneumonia. It turns out to be true since patients with asthma were given more aggressive treatments which led to better outcomes.

# Research challenges

Despite recent progresses in interpretable machine learning, there are still some urgent challenges, especially on explanation method design as well as evaluation.

## Explanation Method Design

The first challenge is related to the method design, especially for post-hoc explanation. We argue that an explanation method should be restricted to truly reflect the model behavior under *normal operation conditions*. This criterion has two meanings. Firstly, the explanations should be faithful to the mechanism of the underlying machine learning model . Post-hoc explanation methods propose to approximate the behavior of models. Sometimes, the approximation is not sufficiently accurate, and the explanation may fail to precisely reflect the actual operation status of the original model. For instance, an explanation method may give an explanation that makes sense to humans, while actually, the machine learning model works in an entirely different way. Second, even when explanations are of high fidelity to the underlying models, they may fail to represent the model behavior under normal conditions. Model explanation and surprising artifacts are often two sides of the same coin. The explanation process could generate examples which are out of distribution from the statistics in the training dataset, including nonsensical inputs and adversarial examples , which are beyond the capability of current machine learning models. Without careful design, both global and local explanations may trigger the artifacts of machine learning models, rather than produce meaningful explanations.

## Explanation Method Evaluation

The second challenge involves the method evaluation. We introduce below the evaluation challenges for intrinsic explanation and post-hoc explanation.

The challenge for intrinsic explanation mainly lies in how to quantify the interpretability. There are broad sets of interpretable models which are designed according to distinct principles and have various forms of implementations. Take the recommender system as an example, both interpretable latent topic model and attention mechanism could provide some extent of interpretability. Nevertheless, how can we compare the interpretability between globally interpretable model and locally interpretable model? There is still no consensus on what interpretability means and how to measure the interpretability. Finale and Been propose three types of metrics: application-grounded metrics, human-grounded metrics, and functionally-grounded metrics . These metrics are complementary to each other and bring their own pros and cons regarding the degree of validity and the cost to perform evaluations. Adopting what metrics heavily depends on the tasks so as to make more informed evaluations.

For post-hoc explanation, comparing to evaluate its interpretability, it is equally important to assess the faithfulness of explanation to the original model, which is often omitted by existing literature. As mentioned before, generated explanations for a machine learning model are not always reasonable to humans. It is extremely hard to tell whether the unexpected explanation is caused by misbehavior of the model or limitation of the explanation method. Therefore, better metrics to measure the faithfulness of explanations are needed, in order to complement existing evaluation metrics. The degree of faithfulness can determine how confident we can trust a explanation. Nevertheless, the design of appropriate faithfulness metric remains an open problem and deserves further investigation.

# Discussion

We briefly introduce limitations of explanation methods that we have surveyed and then present explanation formats that might be more understandable and friendly to users.

## Limitations of Current Explanations

A major limitation of existing work on interpretable machine learning is that the explanations are designed based on the intuition of researchers rather than focusing on the demands of end-users. Current local explanations are usually given in the format of feature importance vectors, which are a complete causal attribution and a low-level explanation . This format would be satisfactory if the explanation audiences are developers and researchers, since they can utilize the statistic analysis of the feature importance distribution to debug the models. Nevertheless, this format is less friendly if the explanation receivers are lay-users of machine learning. It describes the full decision logic of a model, which contains huge amount of redundant information and will be overwhelming to users. The presentation formats could be further enhanced to better promote user satisfaction.

## Towards Human-friendly Explanations

Based on findings in social sciences and human behavioural studies , we provide some directions towards user-oriented explanations, which might be more satisfying to humans as a means of communication.

**Contrastive Explanations** They are also referred as differential explanations . They do not tell why a specific prediction was made, but rather explain why this prediction was made instead of another, so as to answer questions like *"Why $Q$ rather than $R$?"*. Here $Q$ is the fact which requires explanation, and $R$ is the comparing case, which could be a real one or virtual one. Consider, for instance, a user is declined mortgage. The user may compare with another real case and raise question "why didn't I get a mortgage when my neighborhood did?". On the other hand, the user may ask "Why was my mortgage rejected?". Here is an implicit contrast case, and actually the user is requesting explanation for a virtual case "How to get my mortgage loan approved?". Since it is compared to an event which has not happened, thus the desirable explanation here can also be called counterfactual explanation .

To provide contrastive explanations for a model prediction, similar strategy could be used for both above-mentioned comparisons. We first produce feature importance attribution for two instances: not-accepted case for the user, has-accepted case of a neighbor (or would-be-accepted case of the user), and then compare the two attribution vectors. Note that we could resort to adversarial perturbation to find the would-be-accepted case. Besides, it is recommended to provide a diverse set of reasons, i.e., to find multiple contrast cases, to make the explanation more informative. Ultimately, we generate explanations of the form "Your mortgage is rejected because your income is lower than your neighbor's, your credit history is not as strong as your neighbor's, etc" or "Your mortgage would be accepted if your income is raised from $x$ to $y$".

**Selective Explanations** Usually, users do not expect an explanation can cover the complete cause of a decision. Instead, they wish the explanation could convey the most important information that contributes to the decision . A sparse explanation, which includes a minimal set of features that help justify the prediction is preferred, although incompletely. Still use the mortgage case for example. One good explanation could be presenting users the top 2 reasons contributing to the decision, such as poor credit history, low income to debt ratio.

**Credible Explanations** Good explanation might be consistent with prior knowledge of general users . Suppose the generated top reasons for the mortgage case include marital status is single and education status is high school graduate, then it would be less trustable than an explanation outputting poor credit history and low income to debt ratio, since the latter two are more reasonable causes leading to rejection. Low credibility could be caused by the poor fidelity of explanation to the original model. On the other hand, the explanations maybe faithful, however, the machine learning model does not adopt correct evidences to make decisions.

**Conversational Explanations** Explanations might be delivered as a conversation between the explainer and explanation receivers . It means that we need to consider the social context, i.e., to whom an explanation is provided , in order to determine the content and formats of explanations. For instance, a preferred format is verbal explanation if it is explaining to lay-users.

Note that there are many other paths to user-friendly explanations. We refer interested readers to the survey by Miller  for a comprehensive list of directions. All the aforementioned directions serve an identical purpose that explanation should tell users why a decision was reached in a concise and friendly manner. More importantly, the explanation could inform users what could be possibly changed to receive a desired decision next time. Granted, there is still a long way to go to render explanations promote user's satisfaction. In future, researchers from different disciplines, including machine learning, human-computer interaction, and social science, are encouraged to closely cooperate to design really user-oriented and human-friendly explanations.

# Conclusions

Interpretable machine learning is an open and active field of research, with numerous interpretation approaches continuously emerging every year. We present a clear categorization and comprehensive overview of existing techniques for interpretable machine learning, aiming to help the community to better understand the capabilities and weaknesses of different interpretation approaches. Although techniques for interpretable machine learning are advancing quickly, some key challenges remain unsolved, and future solutions are needed to further promote the progress of this field.

[^1]:


abstract: The ability to generalize quickly from few observations is crucial for intelligent systems. In this paper we introduce APL, an algorithm that approximates probability distributions by remembering the most surprising observations it has encountered. These past observations are recalled from an external memory module and processed by a decoder network that can combine information from different memory slots to generalize beyond direct recall. We show this algorithm can perform as well as state of the art baselines on few-shot classification benchmarks with a smaller memory footprint. In addition, its memory compression allows it to scale to thousands of unknown labels. Finally, we introduce a meta-learning reasoning task which is more challenging than direct classification. In this setting, APL is able to generalize with fewer than one example per class via deductive reasoning.
author: Tiago Ramalho
Cogent Labs
`email@example.com`
Marta Garnelo
DeepMind
`email@example.com`
bibliography: iclr2019_conference.bib
title: Adaptive Posterior Learning:
  few-shot learning with a surprise-based
  memory module

# Introduction

Consider the following sequential decision problem: at every iteration of an episode we are provided with an image of a digit (e.g. MNIST) and an unknown symbol. Our goal is to output a digit $Y=X+S$ where $X$ is the value of the MNIST digit, and $S$ is a numerical value that is randomly assigned to the unknown symbol at the beginning of each episode. After seeing only a single instance of a symbol an intelligent system should not only be able to infer the value $S$ of the symbol but also to correctly generalize the operation associated with the symbol to any other digit in the remaining iterations of that episode.

Despite its simplicity, this task emphasizes three cognitive abilities that a generic learning algorithm should display: 1$.$ the algorithm can learn a behaviour and then flexibly apply it to a range of different tasks using only a few context observations at test time; 2$.$ the algorithm can memorize and quickly recall previous experiences for quick adaptation; and 3$.$ the algorithm can process these recalled memories in a non-trivial manner to carry out tasks that require reasoning.

The first point is commonly described as "learning to learn" or *meta-learning*, and represents a new way of looking at statistical inference . Traditional neural networks are trained to approximate arbitrary probability distributions with great accuracy by parametric adaptation via gradient descent . After training that probability distribution is fixed and neural networks can only generalize well when the testing distribution matches the training distribution . In contrast, meta-learning systems are trained to learn an *algorithm* that infers a function directly from the observations it receives at test time. This setup is more flexible than the traditional approach and generalizes better to unseen distributions as it incorporates new information even after the training phase is over. It also allows these models to improve their accuracy as they observe more data, unlike models which learn a fixed distribution.

The second requirement - being able to memorize and efficiently recall previous experience - is another active area of research. Storing information in a model proves especially challenging as we move beyond small toy-examples to tasks with higher dimensional data or real-world problems. Current methods often work around this by summarizing past experiences in one lower-dimensional representation  or using memory modules . While the former approach can produce good results, the representation and therefore the amount of information we can ultimately encode with such models will be of a fixed and thus limited size. Working with neural memory modules, on the other hand, presents its own challenges as learning to store and keep the right experiences is not trivial. In order to successfully carry out the task defined at the beginning of this paper a model should learn to capture information about a flexible and unbounded number of symbols observed in an episode without storing redundant information.

Finally, reasoning requires processing recalled experiences in order to apply the information they contain to the current data point being processed. In simple cases such as classification, it is enough to simply recall memories of similar data points and directly infer the current class by combining them using a weighted average or a simple kernel , which limits the models to performing interpolation. In the example mentioned above, more complex reasoning is necessary for human-level generalisation.

In this paper we introduce Approximate Posterior Learning (APL, pronounced like the fruit), a self-contained model and training procedure that address these challenges. APL learns to carry out few-shot approximation of new probability distributions and to store only as few context points as possible in order to carry out the current task. In addition it learns how to process recalled experiences to carry out tasks of varying degrees of complexity. This sequential algorithm was inspired by Bayesian posterior updating  in the sense that the output probability distribution is updated as more data is observed.

We demonstrate that APL can deliver accuracy comparable to other state-of-the-art algorithms in standard few-shot classification benchmarks while being more data efficient. We also show it can scale to a significantly larger number of classes while retaining good performance. Finally, we apply APL to the reasoning task introduced as motivation and verify that it can perform the strong generalization we desire.

The main contributions of this paper are:

-   A simple memory controller design which uses a surprise-based signal to write the most predictive items to memory. By not needing to learn what to write, we avoid costly backpropagation through memory which makes the setup easier and faster to train. This design also minimizes how much data is stored, making our method more memory efficient.

-   An integrated external and working memory architecture which can take advantage of the best of both worlds: scalability and sparse access provided by the working memory; and all-to-all attention and reasoning provided by a relational reasoning module.

-   A training setup which steers the system towards learning an algorithm which approximates the posterior without backpropagating through the whole sequence of data in an episode.

# Model

## Architecture

Our proposed model is composed of a number of parts: an **encoder** that generates a representation for the incoming query data; an external **memory store** which contains previously seen representation/ data pairings with writing managed by a **memory controller**; and a **decoder** that ingests the query representation as well as data from the memory store to generate a probability distribution over targets. We describe each of the parts in detail below[^1].

**Encoder** The encoder is a function which takes in arbitrary data $x_t$ and converts it to a representation $e_t$ of (usually) lower dimensionality. In all our experiments $x_t$ is an image, and we therefore choose a convolutional network architecture for the encoder. Architectural details of the encoder used for each of the experiments are provided in the appendix.

**Memory store** The external memory module is a database containing the stored experiences. Each of the columns corresponds to one of the attributes of the data. In the case of classification, for example, we would store two columns: the embedding ${e_m}$ and the true label ${y_m}$. Each of the rows contains the information for one data point. The memory module is queried by finding the k-nearest neighbors between a query and the data in a given column. The full row data for each of the neighbors is returned for later use. The distance metric used to calculate proximity between the points is an open choice, and here we always use euclidean distance.

Since we are not backpropagating through the memory, how do we ensure that the neighbors returned by the querying mechanism contain task-relevant information? We expect that class-discriminative embeddings produced by the encoder should cluster together in representation space, and therefore should be close in the sense of euclidean distance. While this is not mathematically necessary, in the following sections we will show that APL as proposed does work and retrieve the correct neighbors which means that in practice our intuition holds true.

**Memory controller** We use a simple memory controller which tries to minimize the amount of data points written to memory. Let us define surprise as the quantity associated with the prediction for label $y_t$ as $\mathbf{S} = -ln(y_t)$. Intuitively, this means that the higher the probability our model assigns to the true class, the less surprised it will be.

This suggests a way of storing the minimal amount of data points in memory which supports maximal classification accuracy. If a data point is 'surprising', it should be stored in memory; otherwise it can be safely discarded as the model can already classify it correctly.

How should the memory controller decide whether a point is 'surprising'? In this work we choose the simplest possible controller: if the surprise is greater than some hyperparameter $\sigma$, then that data should be stored in memory. For our experiments, we choose $\sigma \propto -\ln(N)$ where $N$ is the number of classes under classification which means that if the prediction confidence in the correct class is smaller than the probability assigned by a uniform prediction the value should be written to memory. In the appendix we show that after model training model performance is robust to variations in $\sigma$, as surprise becomes highly bimodal: a new data point tends to be either highly surprising (never seen something similar before) or not very surprising.

Conveniently, in the case of classification problems the commonly used cross-entropy loss reduces to our measure of surprise directly, and we therefore use the prediction loss as an input to the memory controller directly.

**Decoder** The decoder takes as input the query representation as well as all the data from the neighbors found in the external memory. We designed a relational feed-forward module with self attention which takes particular advantage of the external memory architecture. In addition we tested two other established decoder architectures: an unrolled relational working memory core and an unrolled LSTM. As all experiments have a classification loss at the end, all the decoders return a vector with logits for the $N$ classes under consideration. Full details of each architecture are provided in the Appendix.

-   **Relational self-attention feed-forward decoder.** The relational feed-forward module (see figure , left) processes each of the neighbors individually by comparing them with the query, and then does a cross-element comparison with a self-attention module before reducing the activations with an attention vector calculated from neighbor distances.

-   **Relational working memory decoder**   The relational working memory module (figure , right) takes in the concatenated neighbor embeddings and corresponding label embeddings as its initial memory state. The query is fed a number $N$ times as input to the relational memory core to unroll the computation.

-   **LSTM decoder** Finally we also test a vanilla LSTM decoder that takes in the query as the initial memory state and is fed each of the concatenated neighbor embeddings and corresponding label embeddings as its input each time step.

## Training setup

Since we are looking for a system which can update its beliefs in an online manner we need a training procedure that reflect this behaviour. We train the system over a sequence of *episodes* that are composed of sequences of pairs $(x_t, y_t)$. At the start of every episode the mapping $x_t \rightarrow y_t$ is shuffled in a deterministic manner (the exact details are task dependent and will be outlined in the experiments section). The data is then presented to the model sequentially in a random order. The model's memory is empty at the beginning of the episode.

At each time step, a batch of examples is shown to the model and a prediction is made. We then measure the instantaneous loss $L(\hat{y_t}, y_t)$ and perform a gradient update step on the network to minimize the loss on that batch alone. The loss is also fed to the memory controller for the network to decide whether to write to memory. In all the experiments below the task is to classify some quantity, therefore we use cross entropy loss throughout.

APL learns a sequential update algorithm, that is, it minimizes the expected loss over an episode consisting of a number of data elements presented sequentially. However we don't need to backpropagate through the sequence to learn the algorithm. Rather, the model's parameters are updated to minimize the cross-entropy loss independently at each time step. Therefore the only pressure to learn a sequential algorithm comes from the fact that episodes are kept small so that the decoder is encouraged to read the information coming from the queried neighbors instead of just learning to fit the current episode's label mapping in its weights after a few steps of gradient descent (which is what happens in the case of MAML ).

# Related work

Meta-learning as a research field covers a large number of areas. The concept of 'learning to learn' is not tied to a specific task and thus meta-learning algorithms have been successfully applied to a wide range of challenges like RL , program induction (Devlin et al., 2017) few-shot classification  and scene understanding .

Some meta learning models generate predictions in an autoregressive fashion by predicting the next target point from the entire prior sequence of consecutive observations . Algorithms of this kind have delivered state-of-the art results in a range of tasks such as supervised learning to classification. Nonetheless their reliance on the full context history in addition to their autoregressive nature hinders parallelization and hurts performance and scalability.

Another set of methods is based on the nearest neighbours approach . These methods use an encoder to find a suitable embedding and then perform a memory look up based on these representations. The result is a weighted average of the returned labels. As shown in  using a pure distance metric to compare neighbors results in worse performance than allowing a network to learn a comparison function. These kinds of methods thus suffer in comparison. Meta Networks  also use an external memory to enable learning from previous examples combined with a model featuring slow and fast weights to produce the output, enabling them to state-of-the-art performance in several benchmarks.

Conditional neural processes  summarize the data into a fixed representation by averaging over the outputs of an encoder. This representation is fed into a decoder together with a query to produce the output. These methods are more space and compute efficient but given the fixed and averaged representation may not scale to very large problems.

All of the above methods expect a fixed size context, thereby making life-long learning over large time horizons difficult. To enable this, an algorithm must learn to only write observations into memory when they provide additional predictive power. Memory augmented neural networks (MANN)  achieve this by learning a controller to write into a differentiable neural dictionary. However, this requires backpropagating through the entire sequence to learn, which makes credit assignment over long time sequences hard and is computationally expensive. The idea of using an external memory module has been explored in  and shown to produce good results. Compared to that work we introduce a simpler writing mechanism and the idea of a relational decoder to exploit the nearest neighbor structure.

# Experiments

## Few-shot Omniglot classification

The Omniglot dataset contains 1623 characters with 20 examples each. 1200 of the character classes are assigned to the train set while the remaining 423 are part of the test set. The examples are presented to the model sequentially in batches of 16 examples. For each episode, we choose $N$ classes and shuffle their labels. We then run an episode for a certain number of steps, which is decreased as the model's accuracy increases to encourage quick adaptation. This means that the model accuracy and number of memories written to memory is time dependent.

We follow an architecture similar to those used in previous work for the image encoder , which consists of four convolutional blocks with 3x3 convolutions, relu and batch normalization. We augment the number of classes in the training set by rotating each symbol 90, 180 and 270 degrees as in previous work . For this task, we found that all three decoder architectures perform similarly. A detailed comparison and all hyperparameters needed to reproduce this experiment are provided in the Appendix.

In Figure a we can see the behavior of the algorithm within a single episode. As it sees more examples its performance increases until it saturates at some point, when additional writes don't help anymore (assuming the exact same data piece won't be seen again, which is the regime we always assume here). In the simple case of 5-way Omniglot classification, fewer than 2 examples per class are sufficient to saturate performance.

In Figure b we demonstrate the evolution of the posterior distribution in 20-way classification for 3 different, fixed inputs. For the first step, where the memory is empty APL learns to output a uniform distribution ($p \simeq 1/N$ with N the number of classes under classification). As more examples are added to memory, its distribution refines until it sees an informative example for that class, at which point its prediction becomes very confident.

In Figure c we can see that different numbers of examples are written to memory for different classes, which demonstrates one of the advantages of this framework: **we only need to store as many examples as each class requires**, therefore if some classes may be more easily classified than others we can optimally use memory. In contrast, other models feed in a fixed context size per class which means they will either suffer in accuracy or use excessive memory.

|  | 5-way |  | 20-way |  | 423-way |  | 1000-way$^\dagger$ |  |
|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| 2-9 | 1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot |
| Matching nets | 98.1% | 98.9% | 93.8% | 98.5% | \- | \- | \- | \- |
| CNP | 95.3% | 98.5% | 89.9% | 96.8% | \- | \- | \- | \- |
| MANN | 82.2% | 94.9% | \- | \- | \- | \- | \- | \- |
| MAML | 98.7% | 99.9% | 95.8% | 98.9% | \- | \- | \- | \- |
| SNAIL | 99.07% | 99.78% | 97.64% | 99.35% | \- | \- | \- | \- |
| **APL** | 97.9% | 99.9% | 97.2% | 97.6% | 73.5% | 88.0% | 68.9% | 78.9% |

Omniglot test accuracies for fixed context sizes compared to other baselines. ($^\dagger$ For 1000-way classification rotated pseudoclasses are used.)

Despite the previous point, it is also worthwhile comparing model accuracy to existing baselines with a fixed context size. To this end, we pre-populate the memory of a trained model with 1 or 5 examples per class and calculate the accuracy over the test set. We emphasize that the model was not trained to do well in this fixed-context scenario, and yet for 1 and 5-shot classification we obtain performance comparable to state-of-the-art models without having extensively tuned hyperparameters. Furthermore we tested our model with a much higher number of classes: 423-way classification where where we can use the whole test set, and 1000-way where the test set is augmented with rotations of the characters as we do in training.

Finally, we test how the model fares on a completely new distribution, MNIST. For 1-shot, 10-way MNIST, APL trained on 20-way omniglot classification obtains 61% accuracy (compared to 72% cited by ). Testing our model in the sequential regime, we observe that it continues writing examples to memory as it sees surprising observations, which allows it to correct for the distribution shift. After writing 45 examples per class, it reaches an accuracy of 86% (there are 1000 examples per class in the MNIST test set, so it is not simply memorizing).

## ImageNet

We also applied our model to the full scale imagenet dataset. Unlike the above experiments there are no held out classes for testing, as the dataset was not conceived with held out classes in mind. Instead, we rely on shuffling the labels amongst the 1000 Imagenet classes and using the images from the test set for evaluation. This means the generalization results are slightly weaker than in the above sections, but they still provide important insights as to the scalability of our method to thousands of classes and applicability to harder scenarios.

As an encoder we use the pretrained Inception-ResNet-v2  due to computational constraints. For the fixed label case, this network reaches a top-1 accuracy of $80.4\%$. Training the encoder end-to-end might produce better results, an investigation which we leave to later work.

In the 20-way classification challenge, our method reaches $86.7\%$ top-1 accuracy (average accuracy after 50 iterations). Performance remains very high for 100-way ($72.9\%$ top-1 accuracy). The model's performance degrades somewhat ($52.6\%$ top-1 accuracy) for 1000-way classification, where all the classes are shuffled. This highlights that large scale meta-learning on real world datasets remains a challenge even when all the classes have been observed by the encoder, as in this case.

## Number analogy

The number analogy task challenges a meta-learning model to use logic reasoning to generalize with fewer than 1 example per possible class (Figure ). At each time step the network is shown two pieces of data, a number $X$ and a symbol $S$. It is asked to classify the result of $X+S$ based only on the current data and its previous experiments. We experiment with two levels of difficulty for this task: in the first, the number values are fixed and correspond to the MNIST digits, while there are 10 different symbols with unknown values in each episode; in the second, both digits and symbols have shuffled values. We sample the symbol values in the range $[-10, 10]$.

When querying the memory, we query $k$ neighbors via the number embeddings and $k$ neighbors via the symbol embeddings. This makes sure that any relevant information to the problem is available to the reasoning module. The rest of the training setup is identical to the Omniglot experiments, including the encoder network for the digits.

In the case where the numbers are fixed a human would need only see $10$ examples, one for each symbol, to be able to correctly generalize for all $100$ possible combinations. With one example per symbol, APL reaches 97.6% accuracy on the test set.

When both numbers and symbols are shuffled each episode, a logical deduction process must be performed to infer the correct symbols. Our model is able to generalize using 50 examples written to memory (Figure ) which is still fewer than seeing one of all 100 possible combinations.

In this complex case, once a symbol's meaning has been figured out it is no longer necessary to solve the system of equations for that unknown. It would be interesting to explore how a system could additionally store this information in memory for later reuse, a question we leave for later work.

# Conclusion

We introduced a self-contained system which can learn to approximate a probability distribution with as little data and as quickly as it can. This is achieved by putting together the training setup which encourages adaptation; an external memory which allows the system to recall past events; a writing system to adapt the memory to uncertain situations; and a working memory architecture which can efficiently compare items retrieved from memory to produce new predictions.

We showed that the model can:

-   Reach state of the art accuracy with a smaller memory footprint than other meta-learning models by efficiently choosing which data points to remember.

-   Scale to very large problem sizes thanks to the use of an external memory module with sparse access.

-   Perform fewer than 1-shot generalization thanks to relational reasoning across neighbors.

### Acknowledgments

The authors thank Elco Bakker, Alex Pritzel and David Raposo for insightful discussions; Paul Komarek, Adrià Puigdomènech for help with writing code; and Kevin McKee for revising the manuscript.

# Supplementary material

## Model architectures and training details

For all experiments below we use the same training setup. For each training episode we sample elements from $N$ classes and randomly shuffle them to create training batches. For every batch shown to the model, we do one step of gradient descent with the Adam optimizer. We anneal the learning rate from $10^{-4}$ to $10^{-5}$ with exponential decay over 1000 steps (decay rate $0.9$).

For all experiments, the query data is passed through an embedding network (described below in), and this vector is used to query an external memory module. The memory module contains multiple columns of data, one of which will be the key and the others will contain data associated with that key. The necessary columns for each experiment are outlined below. The memory size is chosen so that elements will never be overwritten in a single episode (i.e. memory size $>$ batch size $\times$ number of iterations).

The returned memory contents as well as the query are fed to one of the decoder architectures described in section .

### Omniglot

For omniglot we encode the images with a convolutional network composed of a single first convolution to map the image to 64 feature channels, followed by 12 convolutional blocks. Each block is made up of a step of Batch Normalization, followed by a ReLU activation and a convolutional layer with kernel size 3. Every three blocks the convolution contains a stride 2 to downsample the image. All layers have 64 features. Finally we flatten the activations to a 1D vector and pass it through a Layer Normalization function.

For all decoders we use a hidden dimensionality of $512$, take their final state and pass it through a linear layer to generate the final logits for classification, using a Cross Entropy loss.

### ImageNet

The encoder is a pretrained Inception-ResNet-v2 network  with the standard preprocessing as described in the paper. We use as embedding the pre-logit activations. All decoders use a hidden dimensionality of $1024$. After the decoder step we take their final state and pass it through a linear layer to generate the final logits for classification, using a Cross Entropy loss.

### Analogy task

The encoder for MNIST uses the same convolutional network as described in the Omniglot section. The symbols are one-hot vectors for the first set of experiments. The memory is queried for neighbors both of the digit embeddings as well as the symbols, and found neighbors are concatenated and fed to the decoder.

The decoder process is identical to Omniglot. However, the classification target is now the one-shot encoded version of the result of the computation $X + S$, where $X$ is the digit value and $S$ is the symbol value. As $S \in [-5, 5]$, we sum $5$ to all values to obtain valid one-hot encodings (which means there are 20 possible values in all).

## Decoder Architectures comparison

### Relational Self-attention feed forward module

Consider the set $\{e_t, e_{1...m}, l_{1...m}, d_{1...m}\}$, where $e_t$ is the encoded target, $e_{1...m}$ the encoded observations from the memory, $l_{1...m} = f(y_{1...m})$ are the labels processed by a simple embedding layer to project the classes into a higher dimensional space, and $d_{1...m}$ are the euclidean distances between $e_t$ and $e_{1...m}$. By concatenating all these vectors, we have a set of inputs to a relational block .

This tensor (of shape \[batch size, k, sum of all embeddings feature sizes\]) is fed to what we call a relational self-attentional block: first the tensor is passed through a multihead attention layer (Figure ), which compares all elements to each other and returns a new tensor of the same shape; then a shared nonlinear layer (ReLU, linear, layer norm) processes each element individually. The self-attentional blocks are repeated 8 times in a residual manner (the dimensionality of the tensor never changes).

Finally, we pass the distances between neighbors and query through a softmax layer to generate an attention vector which is multiplied with the activations tensor over the first axis (this has the effect of weighting closer memories more). The tensor is then summed over that first axis to obtain the final representation.

### Relational working memory

We use a Relational working memory core as described in , (figure , right). The memory is initialized with the concatenated vectors $\{e_{1...m}, l_{1...m}, d_{1...m}\}$. The query $e_t$ is fed a number $N=5$ times as input to the relational memory core to unroll the computation. The final memory state is passed through a linear layer to obtain the logits.

### LSTM

We use a standard LSTM module, with initial state equal to the query embedding, and at each time step we feed in the neighbor embedding as concatenated with the embedded label. The LSTM is rolled out for $k$ time steps (i.e. the number of neighbors). Its final output state is taken as input to the logits linear layer.

### Decoder architecture comparison

We compared all three decoder architectures for the classification case and found they perform equally well for the classification case, as shown in the figure below.

For the analogy task, we found the Relational self-attention feed forward module to work best, as outlined in the main text.

## Effect of threshold parameter on performance

How does the choice of parameter $\sigma$ affect the performance of APL? Empirically we have verified that for a large range of $\sigma$, memory size and accuracy are largely unchanged. This is due to the feedback loop between the number of items stored and classification accuracy: as more items are stored in memory, the more elements are correctly classified and not stored in memory. Therefore the memory storage mechanism is self-regulating, and the number of elements in memory ends up being largely flat.

In Fig.  we show the final memory size and average accuracy for the last 100 data points after showing APL 2000 unique data points for the case of 200-way classification. In this case the 'natural' (uniform predictions) $\sigma$ is around 5.2, which seems to be close to optimal for accuracy vs. elements in memory. We can increase the value somewhat but eventually the model can't write to memory any more and performance tanks. On the other side of the curve, for $\sigma = 0$ where we write everything, performance is slightly higher but at a roughly 8x memory cost.

## Relation to continual learning

While we study APL in the few-shot learning setting, the algorithm could also be used in the continual learning setup . We consider an experiment where each task consists of learning 10 new and previously unseen classes. For each task we present the models with 200 unique examples, and report the average accuracy for the last 100 examples seen. Examples are drawn from the test set of classes.

In the case of progressive networks, one gradient descent step is taken after each example. For each task, a new logits layer is added on top of a convolutional encoder (same architecuture as APL) pretrained on the omniglot training set. APL is run as described in the main text.

The results are summarized in figure : APL can perform as well or better than a progressive network on this kind of task without needing access to gradient information, as its memory store can provide the requisite information to update its predictions to the new task.

[^1]: Source code for the model is available at <https://github.com/cogentlabs/apl>.