Which explainable artificial intelligence techniques are there?
Explainable AI and model interpretability are hyper-active and hyper-hot areas of current research (think of holy grail, or something), which have been brought forward lately not least due to the (often tremendous) success of deep learning models in  various tasks, plus the necessity of algorithmic fairness &amp; accountability.
Here are some state of the art algorithms and approaches, together with implementations and frameworks.

Model-agnostic approaches

LIME: Local Interpretable Model-agnostic Explanations (paper, code, blog post, R port)

SHAP: A Unified Approach to Interpreting Model Predictions (paper, Python package, R package). GPU implementation for tree models by NVIDIA using RAPIDS - GPUTreeShap (paper, code, blog post)

Anchors: High-Precision Model-Agnostic Explanations (paper, authors' Python code, Java implementation)

Diverse Counterfactual Explanations (DiCE) by Microsoft (paper, code, blog post)

Black Box Auditing and Certifying and Removing Disparate Impact (authors' Python code)

FairML: Auditing Black-Box Predictive Models, by Cloudera Fast Forward Labs (blog post, paper, code)


SHAP seems to enjoy high popularity among practitioners; the method has firm theoretical foundations on co-operational game theory (Shapley values), and it has in a great degree integrated the LIME approach under a common framework. Although model-agnostic, specific &amp; efficient implementations are available for neural networks (DeepExplainer) and tree ensembles (TreeExplainer, paper).

Neural network approaches (mostly, but not exclusively, for computer vision models)

The Layer-wise Relevance Propagation (LRP) toolbox for neural networks (2015 paper @ PLoS ONE, 2016 paper @ JMLR, project page, code, TF Slim wrapper)

Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (paper, authors' Torch code, Tensorflow code, PyTorch code, yet another Pytorch implementation, Keras example notebook, Coursera Guided Project)

Axiom-based Grad-CAM (XGrad-CAM): Towards Accurate Visualization and Explanation of CNNs, a refinement of the existing Grad-CAM method (paper, code)

SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (paper, code, Google blog post)

TCAV: Testing with Concept Activation Vectors (ICML 2018 paper, Tensorflow code)

Integrated Gradients (paper, code, Tensorflow tutorial, independent implementations)

Network Dissection: Quantifying Interpretability of Deep Visual Representations, by MIT CSAIL (project page, Caffe code, PyTorch port)

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, by MIT CSAIL (project page, with links to paper &amp; code)

Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions (paper, code)

Transparecy-by-Design (TbD) networks (paper, code, demo)

Distilling a Neural Network Into a Soft Decision Tree, a 2017 paper by Geoff Hinton, with various independent PyTorch implementations

Understanding Deep Networks via Extremal Perturbations and Smooth Masks (paper), implemented in TorchRay (see below)

Understanding the Role of Individual Units in a Deep Neural Network (preprint, 2020 paper @ PNAS, code, project page)

GNNExplainer: Generating Explanations for Graph Neural Networks (paper, code)

Benchmarking Deep Learning Interpretability in Time Series Predictions (paper @ NeurIPS 2020, code utilizing Captum)

Concept Whitening for Interpretable Image Recognition (paper, preprint, code)



Libraries &amp; frameworks
As interpretability moves toward the mainstream, there are already frameworks and toolboxes that incorporate more than one of the algorithms and techniques mentioned and linked above; here is a partial list:

The ELI5 Python library (code, documentation)

DALEX - moDel Agnostic Language for Exploration and eXplanation (homepage, code, JMLR paper), part of the DrWhy.AI project

The What-If tool by Google, a feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code (project page, code, blog post)

The Language Interpretability Tool (LIT) by Google, a visual, interactive model-understanding tool for NLP models (project page, code, blog post)

Lucid, a collection of infrastructure and tools for research in neural network interpretability by Google (code; papers: Feature Visualization, The Building Blocks of Interpretability)

TorchRay by Facebook, a PyTorch package implementing several visualization methods for deep CNNs

iNNvestigate Neural Networks (code, JMLR paper)

tf-explain - interpretability methods as Tensorflow 2.0 callbacks (code, docs, blog post)

InterpretML by Microsoft (homepage, code still in alpha, paper)

Captum by Facebook AI - model interpetability for Pytorch (homepage, code, intro blog post)

Skater, by Oracle (code, docs)

Alibi, by SeldonIO (code, docs)

AI Explainability 360, commenced by IBM and moved to the Linux Foundation (homepage, code, docs, IBM Bluemix, blog post)

Ecco: explaining transformer-based NLP models using interactive visualizations (homepage, code, article).

Recipes for Machine Learning Interpretability in H2O Driverless AI (repo)



Reviews &amp; general papers

A Survey of Methods for Explaining Black Box Models (2018, ACM Computing Surveys)

Definitions, methods, and applications in interpretable machine learning (2019, PNAS)

Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead (2019, Nature Machine Intelligence, preprint)

Machine Learning Interpretability: A Survey on Methods and Metrics (2019, Electronics)

Principles and Practice of Explainable Machine Learning (2020, preprint)

Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges (keynote at 2020 ECML XKDD workshop by Christoph Molnar, video &amp; slides)

Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020, Information Fusion)

Counterfactual Explanations for Machine Learning: A Review (2020, preprint, critique by Judea Pearl)

Interpretability 2020, an applied research report by Cloudera Fast Forward, updated regularly

Interpreting Predictions of NLP Models (EMNLP 2020 tutorial)

Explainable NLP Datasets (site, preprint, highlights)

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges



eBooks (available online)

Interpretable Machine Learning, by Christoph Molnar, with R code available

Explanatory Model Analysis, by DALEX creators Przemyslaw Biecek and Tomasz Burzykowski, with both R &amp; Python code snippets

An Introduction to Machine Learning Interpretability (2nd ed. 2019), by H2O



Online courses &amp; tutorials

Machine Learning Explainability, Kaggle tutorial

Explainable AI: Scene Classification and GradCam Visualization, Coursera guided project

Explainable Machine Learning with LIME and H2O in R, Coursera guided project

Interpretability and Explainability in Machine Learning, Harvard COMPSCI 282BR



Other resources

explained.ai blog

A Twitter thread, linking to several interpretation tools available for R

A whole bunch of resources in the Awesome Machine Learning Interpetability repo