2020TheEL} classified methods of explanation as (a) decision-making algorithm-based explanations with the objective of debugging, (b) model-based explanations designed for compensating for differences in cognitive or computational agent models and associated information-processing capabilities, and (c) plan-based explanations for conveying a series of action plans.}. The details of these requirements are provided in the following subsections. 



\subsection{Owning an interpretable decision-making space $\Pi$: Requirement 1}
The user's interpretable decision-making space $\Pi$ is expressed as
\begin{equation}
\begin{split}
\label{spacedef}
&Given:\mathbb{A}^U,\ \tau^U  \\
&\Pi\ such\ that\ \mathbb{A}^U : \Pi \times \tau^U \not\rightarrow \phi,
\end{split}
\end{equation}
where $\mathbb{A}^U$ denotes the user's planning algorithm, and $\tau^U$ denotes a type of constraint. When a policy is derived using these and $\Pi$, and when some policy $\pi$ is determined without encountering an error, the decision-making space $\Pi$ is defined as interpretable to the user. 
This definition implies that $\Pi$ is composed of states and actions that are natural to humans, and the transition between states is consistent with the conventions that govern the real world (e.g., laws of physics)\footnote{Such an interpretable decision-making space can be acquired by learning, including physical identity and interaction. 
The existence of this interpretable decision-making space is related to the encoding of meaning in languages, and it can be explained by the idea of symbol emergence in robotics \cite{SER2015,SER2019}, which is a mechanism by which a symbol system is socially constructed and shared.}. 

The easiest approach to build a user-interpretable decision-making space is for humans to write the rules. 
However, as is well known, this approach has limitations in creating such systems and eventually faces the problem that humans cannot write a complete rules set determining a single correct action in every conceivable situation. 
Further, another approach has also been developed involving constructing a causal graph from certain information to the target state \cite{madumal2020distal}. 
This research is actively being pursued in a field called ``causal discovery''\cite{causaldiscovery2019}; however, it is considered difficult to model the vast number of factors in the world sequentially and achieve a framework that can comprehensively learn these causal relationships. For the construction of causal graphs, the factors of interest are specified by humans, and it is difficult to say that they constitute autonomous learning by the system.

An alternative to the above-mentioned methods is to build a decision-making space that can be interpreted by humans; this can be achieved by having the robot itself acquire a world model \cite{worldmodel}. 
The world model framework involves robots modeling the relationship between their own behavior and changes in the outside world through interactions with their environment. 
Information on changes in the outside world is obtained by observations by the robot itself, and therefore, the decision-making space of a robot can be understood by humans provided the system includes a mechanism for converting the observed information of the robot into a form that can be interpreted by humans. 
The world model framework is similar to the conventional symbolic reasoning framework, but does differ signficantly. 
It has the potential to build a decision-making space that humans can understand without direct human intervention through deep learning and advanced state representation learning methods \cite{SRLforControl2018}\footnote{Inference without action can also be modeled conceptually with a similar approach. 
For example, when considering the transition from the ``state of being infected by the influenza virus'' to a ``joint pain state,'' there is a transition from the ``state of being infected by the influenza virus'' to ``prostaglandin is generated'' because of the immune reaction. This substance then increases the body's responsiveness to pain and reaches the ``joint pain state.''}. 

One challenge to learning in such a real-world model is that actions need to be pre-defined. 
When modeling state changes accompanying the actual action of an agent, the world model can be constructed at the motor command level of the robot. 
In an environment where actions cannot be easily defined, a framework is required in which even selectable actions are learned autonomously\footnote{For example, methods such as action segmentation \cite{nakamura2017seg,nagano2019} can be used.} and meaning is assigned to actions based on differences between states before and after an action. 


\subsection{Estimation of the model of others: Requirement 2}
In the MRP, solving Eq. (\ref{eq:explanation}) and generating an explanation $\epsilon$ requires that $\mathbb{A}^R_U \approx \mathbb{A}^U$, $\Pi^R_U \approx \Pi^U$, and $\tau^R_U \approx \tau^U$ be satisfied. Here, if $\mathbb{A}^R_U \approx \mathbb{A}^U$ and $\tau^R_U \approx \tau^U$, then the problem is estimating $\Pi^R_U$.
\begin{equation}
\begin{split}
\label{eq:requirement2}
& Given: \mathbb{A}^{R} : \Pi^{R} \times \tau^{R} \rightarrow \pi^{R},~ \mathbb{A}^R_U \approx \mathbb{A}^U,~ \tau^R_U \approx \tau^U \\
& \Pi_{U}^{R}  \rightarrow \hat{\Pi}_{U}^{R} -q  ~~such~that~~ \mathbb{A}_{U}^{R} : \hat{\Pi}_{U}^{R} \times \tau_{U}^{R} \rightarrow \pi^{R}.
\end{split}
\end{equation}
Here, $q$ represents a query from the user. 
The estimation of $\mathbb{A}^U$ or $\tau^U$ can be considered a more general problem; however, a comprehensive estimation is believed to be a very difficult problem. Even if the decision-making space is identical to that of the world model held by the human, the planning algorithm does not indicate the reward function in decision-making. The size of the decision-making space that humans can consider when planning, the observable information, and other operators from the outside world are entangled and interdependently conditioned in a complex manner and thus difficult to determine via a computational mechanism.

Estimating the decision-making space and planning algorithms of humans plays a very important role in explanation generation. For example, it would seem obvious that a key is required to open a front door to a home, but this may not be obvious to a toddler whose parents always open the door for them. When instructing such an agent lacking contextual information to ``open the door,'' one could say, ``place this key in the keyhole, turn it to the left, and then pull the door.'' Human beings consider such factors daily, and this task captures the essential meaning of ``communication.'
\footnote{This example was related to a behavioral sequence; however, this applies to knowledge reasoning as well. For example, if someone did not understand the connection between joint pain and influenza, it might be better to explain that ``a fever occurred to get rid of the influenza virus, and the substance secreted in the body at this time causes joint pain.'' However, to do so, not only is the construction of a decision-making space using the information given by humans necessary, but the system must also autonomously and widely learn more detailed information and seemingly unrelated information. From this point of view, the autonomous acquisition of the abovementioned decision-making space is indispensable in explanation generation.}


\subsection{Estimation of the information needed for a user to estimate $\pi^R$: Requirement 3}
This requirement is the problem of identifying a method to find $\epsilon$ in Eq. (\ref{eq:explanation}). Generating explanations that promote human comprehension of robot policies in the decision-making space requires explanations that are easily accepted as (i) having some human characteristics and (ii) appropriate for specific individuals. These two points are discussed in the following sections.

\subsubsection{Good explanation from the perspective of cognitive science}
We consider explanations that are easily accepted by humans from the perspective of cognitive science. According to cognitive science, two approaches are understood to serve as indicators to determine the quality of an explanation, including likeliness and loveliness \cite{explanatory,lipton}. Likeliness is a probabilistically defined indicator of explanation quality. In the likeliness framework, a good explanation is defined one that maximizes the posterior probability $P(X|E_i)$ of an event $X$ that we wish to explain when a certain explanation $E_i$ is presented. Loveliness, in contrast is an indicator defined from an axiological perspective. 
In the framework of loveliness, studies have focused on simplicity and latent scope as indicators that determine the goodness of an explanation.
Simplicity is an index expressing the number of assumed causes, where one with a smaller number is preferred by humans \cite{lombrozo}. Latent scope was predicted to be introduced by a cause, and this was ultimately an unobserved event; a higher posterior probability $P(X|E_i)$ was estimated with a smaller number for the latent scope \cite{harrypotter,