\section{Introduction}

Explanation regeneration is the task of retrieving and combining two or more facts from an external knowledge source to reconstruct the explanation supporting a certain natural language hypothesis \cite{xie-etal-2020-worldtree,jansen2018worldtree}. As such, this task represents a crucial intermediate step for the development and evaluation of explainable Natural Language Inference (NLI) models \cite{wiegreffe2021teach,thayaparan2020survey}.
In particular, explanation regeneration on science questions has been identified as a suitable benchmark for complex multi-hop and abductive inference capabilities \cite{xie-etal-2020-worldtree,jansen2019textgraphs,jansen2018worldtree}. Scientific explanations, in fact, require the articulation and integration of commonsense and scientific knowledge for the construction of long explanatory reasoning chains \cite{clark2018think,jansen2016s}. Moreover, since the structure of scientific explanations cannot be derived from the decomposition of the questions, the task requires the encoding of complex abstractive mechanisms for the identification of relevant explanatory knowledge \cite{valentino2021unification,zhou2021encoding,thayaparan2020explanationlp}. 

To tackle these challenges, most of the existing approaches for explanation regeneration leverage the power of the cross-attention mechanism in Transformers \cite{devlin2019bert,vaswani2017attention}, training sequence classification models on the task of composing relevant explanatory chains supervised via human-annotated explanations \cite{cartuyvels-etal-2020-autoregressive,das2019chains,chia2019red,banerjee2019asu}. While Transformers achieve state-of-the-art performance, the adoption of cross-encoder architectures makes abductive inference intrinsically inefficient and not scalable to massive corpora. The problem of constructing scientific explanations efficiently, in fact, is still under-studied, with the state-of-the-art Transformer \cite{cartuyvels-etal-2020-autoregressive} requiring $\approx 9$ seconds per question on corpora of $< 10k$ facts. 

\begin{figure*}[t]
\centering
\includegraphics[width=\textwidth]{approach.pdf}
\caption{We propose a hybrid architecture, scalable to massive facts banks, that performs abductive inference autoregressively. At each time step $t$, we perform inference via Maximum Inner Product Search (MIPS) (1) combining dense and sparse encoders to compute relevance and explanatory power of candidate facts (2) and expand the explanation (3). The explanatory power of a given fact is estimated by analysing explanations for similar hypotheses (Explanations Corpus), while the relevance of a fact at time-step $t$ depends on the partial explanation constructed at time $t-1$.}
\label{fig:approach}
\end{figure*}


In this work, we are interested in developing new mechanisms to enable abductive inference at scale, maximising both accuracy and inference time in explanation regeneration. To this end, we investigate the construction of abductive solvers through scalable bi-encoder architectures \cite{reimers-gurevych-2019-sentence} to perform efficient multi-hop inference via Maximum Inner Product Search (MIPS) \cite{8733051}. Given the complexity of abductive inference in the scientific domain, however, the adoption of bi-encoders alone is expected to lead to a drastic drop in performance since the cross-attention mechanism in Transformers cannot be leveraged to learn meaningful compositions of explanatory chains.
To tackle the lack of cross-attention, we hypothesise that the orchestration of latent and explicit patterns emerging in natural language explanations can enable the design of the abstractive mechanisms necessary for accurate regeneration while preserving the scalability intrinsic in bi-encoders.

To validate this hypothesis, we present \textbf{SCAR} (for \underline{Sc}alable \underline{A}utoregressive Infe\underline{r}ence), a hybrid abductive solver that autoregressively combines a Transformer-based bi-encoder with a sparse model that leverages explicit patterns in corpora of scientific explanations.  
Specifically, SCAR integrates sparse and dense encoders to define a joint model of relevance and explanatory power and perform abductive inference in an iterative fashion, conditioning the probability of retrieving a fact at time-step $t$ on the partial explanation constructed at time-step $t-1$ (Fig. \ref{fig:approach}).
We performed an extensive evaluation on the WorldTree corpus \cite{xie-etal-2020-worldtree,jansen2019textgraphs,jansen2018worldtree}, presenting the following conclusions: 
\begin{enumerate}
    \item The proposed abductive solver achieves performance comparable with the state-of-the-art cross-encoder while being $\approx 50$ times faster.
    \item  We study the impact of the hybridisation on semantic drift, showing that SCAR is more robust in the construction of challenging explanations requiring long reasoning chains.
    \item We investigate the applicability of SCAR on science question answering without additional training, demonstrating improved accuracy and robustness when performing abductive inference iteratively.
    \item We perform a scalability analysis by gradually expanding the adopted facts bank, showing that SCAR can scale to corpora containing millions of facts.
\end{enumerate}

To the best of our knowledge, we are the first to propose a hybrid autoregressive solver for explanation regeneration in the scientific domain, and demonstrate its efficacy for abductive natural language inference at scale.

\section{Explanation Regeneration}

Given a scientific hypothesis $h$ (e.g., \emph{``Two sticks getting warm when rubbed together is an example of a force producing heat''}), the task of explanation regeneration consists in reconstructing the explanation supporting $h$, composing a sequence of atomic facts $E_{seq} = f_1,\ldots,f_n$ retrieved from external knowledge sources.

Explanation regeneration can be framed as a multi-hop abductive inference problem, where the goal is to construct the best explanation for a given natural language statement adopting multiple inference steps. To learn to perform abductive inference, a recent line of research relies on explanation-centred corpora, which are typically composed of two distinct knowledge sources \cite{jansen2019textgraphs}:
\begin{enumerate}
    \item A fact bank of individual \emph{commonsense and scientific sentences} including the knowledge necessary to construct explanations for scientific hypotheses.
    \item A corpus of \emph{explanations} consisting of true hypotheses and natural language explanations composed of sentences from the fact bank. 
\end{enumerate}


\section{Autoregressive Inference}
\label{sec:autoregressive_inference}
To model the multi-hop nature of scientific explanations, we propose a hybrid abductive solver that performs inference autoregressively (Fig. \ref{fig:approach}). Specifically, we model the probability of composing an explanation sequence $E_{seq} = f_1, \ldots, f_n$ for a certain hypothesis $h$ using the following formulation:

\begin{equation}
\small
P(E_{seq}|h) = \prod_{t=1}^{n}P(f_t|h,f_1, \ldots f_{t-1})
\end{equation}

\noindent where $n$ is the maximum number of inference steps to construct an explanation and $f_t$ represents a  fact retrieved at time-step $t$ from the facts bank. We implement the model recursively by updating the hypothesis $h$ at each time-step $t$, concatenating it with the partial explanation constructed at time step $t-1$:

\begin{equation}
\small
h_t = g(h, f_1, \ldots f_{t-1})
\end{equation}

\noindent where $g(\cdot)$ represents the string concatenation function.
The probability $P(f_t|h_t)$ is then approximated via an explanatory scoring function that jointly models relevance and explanatory power. Since similar scientific hypotheses require similar explanations \cite{valentino2021unification}, we define explanatory power as a measure to capture explicit patterns in the explanations,  quantifying the extent to which a given fact explains similar hypotheses in the corpus (additional details in Section \ref{sec:exp_power}). The explanatory scoring function is then defined as follows:
\begin{equation}
\small
es(f_t, h_t) = \lambda_1 \cdot r(f_t, h_t) + (1-\lambda_1) \cdot pw(f_t, h)
\label{eq:explanatoryscore}
\end{equation}

\noindent where $pw(\cdot)$ represents the explanatory power of $f_t$, while $r(\cdot)$ represents the relevance of $f_t$ at time step $t$. The probability of selecting a certain fact $f_t$, therefore, depends jointly on its explanatory power and relevance with respect to the partially constructed explanation. To leverage the complementary aspects of dense and sparse representations, we implement $r(\cdot)$ through a hybrid model. Specifically, given a sparse $s(\cdot)$ and a dense $d(\cdot)$ encoder, we compute $r(\cdot)$ as follows:
\begin{equation}
\small
r(f_t, h_t) = sim(s(f_t), s(h_t)) + sim(d(f_t), d(h_t))
\end{equation}

\noindent with $sim(\cdot)$ representing the cosine similarity between the embeddings. In our experiments, we adopt BM25 \cite{robertson2009probabilistic} as a sparse representation, while Sentence-BERT \cite{reimers-gurevych-2019-sentence} is adopted to train the dense encoder $d(\cdot)$ (see section \ref{sec:dense_encoder} for additional details).

\subsection{Explanatory Power}
\label{sec:exp_power}

Recent work have shown that \emph{explicit explanatory patterns} emerge in corpora of natural language explanations \cite{valentino2021unification} -- i.e., facts describing scientific laws with high \emph{explanatory power} (i.e., laws that explain a large set of phenomena such as gravity, or friction) are frequently reused to explain similar hypotheses. Therefore, these patterns can be leveraged to define a computational model of explanatory power for estimating the relevance of abstract scientific facts, providing a framework for efficient abductive inference in the scientific domain.

Given an unseen hypothesis $h$, a sentence encoder $s(\cdot)$, and a corpus of scientific explanations, the explanatory power of a generic fact $f_i$ can be estimated by analysing explanations for similar hypotheses in the corpus:

\begin{equation}
\small
pw(f_i, h) = \sum_{h_k \in kNN(h)}^K {sim(s(h),s(h_k))} \cdot \mathbbm{1}(f_i, h_k)
\label{eq:unification_score}
\end{equation}  

\begin{equation}
\small
    \mathbbm{1}(f_i,h_k) =
        \begin{cases}
            1 & \text{if } f_i \in E_{k}\\
            0 & \text{if } f_i \notin E_{k}
        \end{cases}
\end{equation}

\noindent where $kNN(h) = \{h_1, \ldots, h_K\}$ represents a list of hypotheses retrieved according to the similarity $sim(\cdot)$ between the embeddings $s(h)$ and $s(h_k)$, and $\mathbbm{1}(\cdot)$ is the indicator function verifying whether $f_i$ is part of the explanation $E_k$ for the hypothesis $h_k$. Specifically, the more a fact $f_i$ is reused for explaining hypotheses that are similar to $h$ in the corpus, the higher its explanatory power.

In this work, we hypothesise that this model can be integrated within a hybrid abductive solver based on dense and sparse encoders (Sec. \ref{sec:autoregressive_inference}), improving inference performance while preserving scalability. In our experiments, we adopt BM25 \cite{robertson2009probabilistic} to compute the explanatory power efficiently.




\subsection{Dense Encoder}
\label{sec:dense_encoder}
To learn a dense representation for efficient multi-hop inference, we fine-tune a Sentence-BERT model using a bi-encoder architecture \cite{reimers-gurevych-2019-sentence}. The bi-encoder adopts a siamese network to learn a joint embedding space for hypotheses and explanatory facts in the corpus. Following Sentence-BERT, we obtain fixed sized sentence embeddings by adding a mean-pooling operation to the output vectors of BERT \cite{devlin2019bert}. We employ a unique BERT model with shared parameters to learn a sentence encoder $d(\cdot)$ for both facts and hypotheses.

\paragraph{Training.} The bi-encoder is fine-tuned on inference chains extracted from the gold explanations in the WorldTree corpus \cite{jansen2019textgraphs}. To train the model autoregressively, we first serialize the gold explanations sorting the facts in decreasing order of BM25 similarity with the hypothesis. We adopt BM25 for the serialization since the facts with lower lexical similarity are typically the ones requiring multi-hop inference. Subsequently, given a training hypothesis $h$ and an explanation sequence $E_{seq} = f_1, \ldots, f_n$, we derive $n$ positive example tuples $(h_t, f_t)$, one for each fact $f_t \in E_{seq}$, using $h_t = g(h, f_1, \ldots, f_{t-1})$ as hypothesis. To make the model robust to distracting information, we construct a set of negative examples for each tuple $(h_t, f_t)$ retrieving the top most similar facts to $f_t$ that are not part of the gold explanation (using BM25). We found that the best results are obtained when considering $5$ negative examples for each positive tuple. Adopting this methodology, we are able to construct a training set containing $\approx 76K$ inference chains. We use the constructed training set and the siamese network to fine-tune the encoder via contrastive loss \cite{contrastive_loss}, which has been demonstrated to be effective for learning robust dense representations.

\paragraph{Inference.} At the cost of sacrificing the performance gain resulting from the cross-attention, the bi-encoder allows for efficient multi-hop inference through Maximum Inner Product Search (MIPS). To enable scalability, we construct an index of dense embeddings for the whole facts bank. To this end, we adopt the approximated inner product search index (IndexIVFFlat) available in FAISS \cite{8733051}. At inference time, we derive the vector of the test hypothesis at each time-step $t$ using the trained encoder,
and employ approximated MIPS to retrieve the similarity scores for the top candidate facts in the corpus. Subsequently, we combine the similarity scores with the sparse model to compute relevance and explanatory power (Equation \ref{eq:explanatoryscore}) and select the fact $f_t$ to expand the explanation at time-step $t$. After $t_{max}$ steps, we rank the remaining facts in the corpus considering hypothesis and partially constructed explanation.

\begin{table*}[h]
    \small
    \centering
    \ra{1}
    \begin{tabular}{p{5cm}p{8cm}c}
        \toprule
         \multirow{2}{*}{\textbf{Model}} & \multirow{2}{*}{\textbf{Approach Description}} &
         \multirow{2}{*}{\textbf{MAP}}\\\\
       \midrule
       \textbf{Dense Models (Cross-encoders)}\\
       \midrule \citet{cartuyvels-etal-2020-autoregressive} & Autoregressive BERT cross-encoder & \textbf{57.07}\\
       \citet{das2019chains} & BERT path-ranking + single fact ensemble & 56.25\\
       \citet{das2019chains} & BERT single fact & 55.74\\
       \citet{das2019chains} & BERT path-ranking & 53.13\\
       \citet{chia2019red} & BERT re-ranking with gold IR scores & 49.45\\
       \citet{banerjee2019asu} & BERT iterative re-ranking & 41.30\\
         \midrule
       \textbf{Sparse Models}\\
       \midrule
        \citet{valentino2021unification} & Unification-based Reconstruction & \textbf{50.83}\\
        \citet{chia2019red} & Iterative BM25 & 45.76 \\
        BM25 \cite{robertson2009probabilistic} & BM25 Relevance Score & 43.01\\
        TF-IDF & TF-IDF Relevance Score & 39.42\\
        \midrule
        \textbf{Hybrid Model (Bi-encoders)} \\
        \midrule
        \textbf{SCAR} & Scalable Autoregressive Inference & \textbf{56.22} \\
        \bottomrule
    \end{tabular}
    \caption{Results on the test-set and comparison with previous approaches. SCAR significantly outperforms all the sparse models and obtains comparable results with state-of-the-art cross-encoders.}
    \label{tab:compare_approaches_overall}
\end{table*}

\section{Empirical Evaluation}

We perform an extensive evaluation on the WorldTree corpus \cite{xie-etal-2020-worldtree,jansen2018worldtree} adopting the dataset released for the shared task on multi-hop explanation regeneration\footnote{\url{https://github.com/umanlp/tg2019task}} \cite{jansen2019textgraphs}, where a diverse set of sparse and dense models have been evaluated. 

WorldTree is a subset of the ARC corpus \cite{clark2018think} that consists of multiple-choice science questions annotated with natural language explanations supporting the correct answers. The explanations in WorldTree contain an average of six facts and as many as 16, requiring challenging multi-hop inference to be regenerated. The WorldTree corpus provides a held-out test-set consisting of 1,240 science questions with masked explanations where we run the main experiment and comparison with published approaches. On the other hand, we adopt the explanations in the training-set ($\approx 1,000$) for training the dense encoder (section \ref{sec:dense_encoder}) and computing the explanatory power for unseen hypotheses at inference time (section \ref{sec:exp_power}). 
To run our experiments, we first transform each question and correct answer pair into a hypothesis following the methodology described in \cite{demszky2018transforming}. 

We adopt \texttt{bert-base-uncased} \cite{devlin2019bert} as a dense encoder to perform a fair comparison with existing cross-encoders employing the same model. The best results on explanation regeneration are obtained when running SCAR for 4 inference steps (additional details in Section \ref{sec:ablation_study}). In line with the shared task, the performance of the system is evaluated through the Mean Average Precision (MAP) of the produced ranking of facts with respect to the gold explanations in the corpus. 

\subsection{Explanation Regeneration}

Table~\ref{tab:compare_approaches_overall} reports the results achieved by our best model on the explanation regeneration task together with a comparison with previously published approaches. Specifically, we compare our hybrid framework based on bi-encoders with a variety of sparse and dense retrieval models. 

Overall, we found that SCAR significantly outperforms all the considered sparse models (+5.39 MAP compared to \citet{valentino2021unification}), obtaining, at the same time, comparable results with the state-of-the-art cross-encoder (-0.85 MAP compared to \citet{cartuyvels-etal-2020-autoregressive}). 
The following paragraphs provide a detailed comparison with previous work. 


\paragraph{Dense Models.}

As illustrated in Table~\ref{tab:compare_approaches_overall}, all the considered dense models employ BERT \cite{devlin2019bert} as a cross-encoder architecture for abductive inference. The state-of-the-art model proposed by \citet{cartuyvels-etal-2020-autoregressive} adopts an autoregressive formulation similar to SCAR. However, the use of cross-encoders makes the model computationally expensive and intrinsically not scalable. Due to the complexity of cross-encoders, in fact, the authors have to employ a pre-filtering step based on TF-IDF and apply the model for re-ranking a small set of candidate facts at each iteration.  In contrast, we found that the use of a hybrid model allows SCAR to achieve comparable performance (-0.85 MAP), being 50 times faster without any pre-filtering step (Section \ref{sec:inference_time}).
The second-best dense approach employs an ensemble of two BERT models \cite{das2019chains}. A first BERT model is trained to predict the relevance of each fact individually given a certain hypothesis. A second BERT model is adopted to re-rank a set of two-hops inference chains constructed via TF-IDF. The use of two BERT models in parallel, however, makes the approach computationally exhaustive. We observe that SCAR can achieve similar performance with the use of a single BERT bi-encoder, outperforming each individual sub-component in the ensemble with a drastic improvement in efficiency (SCAR is 96.8 times and 167.4 times faster respectively, see Section \ref{sec:inference_time}).
The remaining dense models \cite{chia2019red,banerjee2019asu} adopt BERT-based cross-encoders to re-rank the list of candidate facts retrieved using sparse Information Retrieval (IR) techniques. As illustrated in Table~\ref{tab:compare_approaches_overall}, SCAR outperforms these approaches by a large margin (+6.77 and +14.92 MAP respectively).

\paragraph{Sparse Models.}

We compare SCAR with sparse models presented on the explanation regeneration task. We observe that SCAR significantly outperforms the Unification-based Reconstruction model proposed by \citet{valentino2021unification} (+5.39 MAP), which employs a similar model of explanatory power in combination with BM25, but without dense representation and autoregressive inference. These results confirm the contribution of the hybrid model together with the importance of modelling abductive inference in a iterative fashion. In addition, we compare SCAR with the model proposed by \citet{chia2019red} which adopts BM25 vectors to retrieve facts iteratively. We found that SCAR can dramatically improve the performance of this model by 10.46 MAP points. Finally, we measure the performance of standalone sparse baselines for a sanity check, showing that SCAR can significantly outperform BM25 and TFIDF on the abductive inference task (+13.21 and +16.8 MAP respectively), while preserving a similar scalability (Section \ref{sec:scalability}). 
\begin{table}[t]
    \small
    \centering
    \ra{1}
    \begin{tabular}{p{3cm}cc}
        \toprule
         \multirow{2}{*}{\textbf{Model}} & 
         \multirow{2}{*}{\textbf{MAP $\uparrow$}} &
         \multirow{2}{*}{\textbf{Time (s/q) $\downarrow$}}\\\\
        \midrule
        \textbf{Cross-encoders}\\
        \midrule
        Autoregressive BERT & \textbf{57.07} & 9.6 \\ 
        BERT single fact & 55.74 & 18.4 \\
        BERT path-ranking & 53.13 & 31.8\\
        \midrule
        \textbf{Proposed Model}\\
        \midrule
        SCAR & 56.22 (98.5\%) & \textbf{0.19} ($\times50.5$) \\
        \bottomrule
    \end{tabular}
    \caption{Detailed comparison with BERT cross-encoders on the test set in terms of Mean Average Precision (MAP) and inference time (seconds per question).}
    \label{tab:time_results}
\end{table}





\subsection{Inference Time}
\label{sec:inference_time}

We performed additional experiments to evaluate the efficiency of SCAR and contrast it with state-of-the-art cross-encoders. To this end, we run SCAR on 1 16GB Nvidia Tesla P100 GPU and compare the inference time with that of dense models executed on the same infrastructure \cite{cartuyvels-etal-2020-autoregressive}. Table~\ref{tab:time_results} reports MAP and execution time in terms of seconds per question. 
As evident from the table, we found that SCAR is 50.5 times faster than the state-of-the-art cross-encoder \cite{cartuyvels-etal-2020-autoregressive}, while achieving 98.5\% of its performance. Moreover, when compared to the individual BERT models proposed by \citet{das2019chains}, SCAR is able to achieve better MAP score (+0.48 and +3.09), increasing even more the gap in terms of efficiency ($96.8$ and $167.4$ times faster). 


\begin{figure*}[t]
\centering
\subfloat[\label{fig:lexical_overlaps}]{\includegraphics[width=0.5\textwidth]{images/plot_expl_k.pdf}}
\hfill
\subfloat[\label{fig:map_exp_length}]{\includegraphics[width=0.5\textwidth]{images/plot_expl_size.pdf}}
\caption{(a) Impact of increasing the number of similar hypotheses $K$ to estimate the explanatory power (Equation \ref{eq:unification_score}). (b) Performance considering gold explanations with increasing number of facts. The models start with a comparable MAP score at 1, ending up with a wider gap in performance, confirming that the proposed hybrid model is more robust to semantic drift.
}
\end{figure*}



\begin{table}[t]
    \small
    \centering
    \ra{1}
    \begin{tabular}{p{3cm}ccc}
        \toprule
         \multirow{2}{*}{\textbf{Model}} & \multirow{2}{*}{\textbf{$t_{max}$}} &
         \multirow{2}{*}{\textbf{MAP}} &
         \multirow{2}{*}{\textbf{Time (s/q)}}\\\\
        \midrule
         Bi-encoder & 1 & 41.98 & 0.04 \\ 
         & 2 & \textbf{42.17} & 0.08 \\ 
         & 3 & 39.97 & 0.12 \\
         & 4 & 38.34 & 0.16 \\ 
         & 5 & 37.24 & 0.19 \\ 
         & 6 & 36.64 & 0.24 \\
        \midrule
         BM25 & 1 & 45.99 & 0.02 \\ 
         & 2 & 47.77 & 0.04 \\ 
         & 3 & \textbf{48.35} & 0.05 \\
         & 4 & 48.06 & 0.07\\
         & 5 &  47.97 & 0.09 \\
         & 6 & 47.66 & 0.11\\
        \midrule
        Bi-encoder + BM25 & 1 & 51.53 & 0.05 \\
         & 2 & 54.52 & 0.08 \\
         & 3 & 55.65 & 0.14 \\
         & 4 & 56.07 & 0.18 \\
         & 5 & \textbf{56.24} & 0.22 \\
         & 6 & 55.87 & 0.27 \\
        \midrule
        \midrule
        SCAR (Exp. power) & 1 & 57.10 & 0.06 \\
         & 2 & 59.20 & 0.10 \\
         & 3 & 59.73 & 0.15 \\
         & 4 & \underline{\textbf{60.28}} & 0.19 \\
         & 5 & 59.79 & 0.24 \\
         & 6 & 59.36 & 0.29 \\
        \bottomrule
    \end{tabular}
    \caption{Ablation study on the dev-set, where $t_{max}$ represents the maximum number of iterations adopted to regenerate the explanations, and $(s/q)$
    is the inference time in terms of seconds per question.}
    \label{tab:iterative_results}
\end{table}




\subsection{Ablation Studies}
\label{sec:ablation_study}

In order to understand how the different components of our approach complement each other, we carried out distinct ablation studies. The studies are performed on the dev-set since the explanations on the test-set are masked. 

Table~\ref{tab:iterative_results} presents the results on explanation regeneration for different ablations of SCAR adopting an increasing number of iterations $t$ for abductive inference. The results show how the performance improves as we combine sparse and dense models, with a decisive contribution coming from each individual sub-component. Specifically, considering the best results obtained in each case, we observe that SCAR achieves an improvement of 18.11 MAP when considering only the dense component (Bi-encoder) and 11.93 MAP when compared to the sparse model (BM25). Moreover, the ablation demonstrates the fundamental role of the explanatory power model in achieving the final performance, which leads to an improvement of 4.04 MAP over the hybrid Bi-encoder + BM25 relevance model (Equation \ref{eq:explanatoryscore}). 

Overall, we notice that performing abductive inference iteratively is beneficial to the performance across the different components. We observe that the improvement is more prominent when comparing $t_{max}=1$ (only using the hypothesis) with $t_{max}=2$ (using hypothesis and first fact), highlighting the central significance of the first retrieved fact to support the complete regeneration process.  Except for the Bi-encoder, the experiments demonstrate a slight improvement when adding more iterations to the process, obtaining the best results for SCAR using a total of 4 inference steps.

We notice that the best performing component in terms of inference time is BM25. The integration with the dense model, in fact, slightly increases the inference time yet leading to a decisive improvement in terms of MAP score. Even with the overhead caused by the Bi-encoder, however, SCAR can still perform abductive inference in less than half a second per question, a feature that demonstrates the scalability of the approach with respect to the number of iterations.


Finally, we evaluate the impact of the explanatory power model by considering a larger set of training hypotheses  (Figure~\ref{fig:lexical_overlaps}). To this end, we compare the performance across different configurations with increasing values of $K$ in Equation \ref{eq:unification_score}. The results demonstrate the positive impact of the explanatory power model on abductive inference, with a rapid increase of MAP peaking at $K=80$. After reaching this value, we observe that considering additional hypotheses has little impact on the model's performance. 

\subsection{Semantic Drift}

Recent work has shown that the regeneration of natural language explanations is particularly challenging for multi-hop inference models as it can lead to a phenomenon known as \emph{semantic drift} -- i.e., the composition of spurious inference chains caused by the tendency of drifting away from the original context in the hypothesis \cite{khashabi2019capabilities,xie-etal-2020-worldtree,jansen2019textgraphs,thayaparan2020survey}. In general, the larger the size of the explanation, the higher the probability of semantic drift.
Therefore, it is particularly important to evaluate and compare the robustness of multi-hop inference models on hypotheses requiring long explanations.
To this end, we present a study of semantic drift, comparing the performance of different ablations of SCAR on hypotheses with a varying number of facts in the gold explanations. The results of the study are reported in Figure~\ref{fig:map_exp_length}.

Overall, we observe a degradation in performance for all the considered models that becomes more prominent as the explanations increase in size. Such a degradation is likely due to semantic drift. However, the results suggest that SCAR exhibits more stable performance on long explanations ($\geq$ 6 facts) when compared to its individual sub-components. In particular, the plotted results in Figure \ref{fig:map_exp_length}  clearly show that, while all the models start with comparable MAP scores on explanations containing a single fact, the gap in performance gradually increases with the size of the explanations, with SCAR obtaining an improvement of $13.46$ MAP over BM25 + Bi-encoder on explanations containing more than 10 facts. 

These results confirm the hypotheses that dense and sparse models possess complementary features for explanation regeneration and that the proposed hybridisation has a decisive impact on improving abductive inference for scientific hypotheses in the most challenging setting.





\subsection{Question Answering}

\begin{table}[t]
    \small
    \centering
    \ra{1}
    \begin{tabular}{p{2.5cm}cccc}
        \toprule
         \multirow{2}{*}{\textbf{Model}} & 
         \multirow{2}{*}{\textbf{t = 1}} &
         \multirow{2}{*}{\textbf{t = 2}} &
         \multirow{2}{*}{\textbf{t = 3}} &
         \multirow{2}{*}{\textbf{t = 4}}
         \\\\
        \midrule
        Random & 25.00 & 25.00 & 25.00 & 25.00\\
        \midrule
        BM25 & \textbf{48.23} &  39.82 & 35.84 & 33.18\\
        Bi-encoder & \textbf{54.42} & 52.21 & 50.88 & 50.00\\
        Bi-encoder + BM25 & \textbf{59.29} & 52.21 & 47.79 & 44.69\\ 
        \midrule
        SCAR & 60.62 & 60.62 & \underline{\textbf{61.06}} & 57.96\\
        \bottomrule
    \end{tabular}
    \caption{Accuracy in question answering using the models as abductive inference solvers without additional training.}
    \label{tab:qa_results}
\end{table}

 Since the construction of spurious inference chains can lead to wrong answer prediction, semantic drift often influences the downstream capabilities of answering the question. Therefore, we additionally evaluate the performance of SCAR on the multiple-choice question answering task, employing the model as an abductive inference solver without additional training.  Specifically, given a multiple-choice science question, we employ SCAR to construct an explanation for each candidate answer, and derive the relative candidate answer score by summing up the explanatory score of each fact in the explanation (Equation \ref{eq:explanatoryscore}). Subsequently, we consider the answer with the highest-scoring explanation as the correct one. Table~\ref{tab:qa_results} shows the results achieved adopting different iterations $t$ for abductive inference.

Similarly to the results on explanation regeneration, this experiment confirms the interplay between dense and sparse models in improving the performance and robustness on downstream question answering. Specifically, we observe that, while the performance of different ablations decreases rapidly with an increasing number of inference steps, the performance of SCAR are more stable, reaching a peak at $t = 3$. These results confirm the robustness of SCAR in multi-hop inference together with its resilience to semantic drift.

\subsection{Scalability}
\label{sec:scalability}

Finally, we measure the scalability of SCAR on large facts banks containing milions of sentences. To perform this analysis, we gradually expand the set of facts in the WorldTree corpus by randomly extracting sentences from GenericsKB\footnote{\url{https://allenai.org/data/genericskb}} \cite{bhakthavatsalam2020genericskb}, a curated facts bank of commonsense and scientific knowledge. To evaluate scalability, we compare the inference time of SCAR with that of standalone BM25. The results of this experiment, reported in Figure \ref{fig:scalability_comparison}, demonstrate that SCAR scales similarly to BM25. Even considering the overhead caused by the Bi-encoder model, in fact, SCAR is still able to perform abductive inference in less than 1 second per question on corpora containing 1 million facts.

\begin{figure}[t]
\centering
\includegraphics[width=\columnwidth]{images/scalable_plot.pdf}
\caption{Graph showing the scalability of SCAR to corpora containing a million facts. We observe a trend similar to BM25, with the inference time of SCAR remaining below one second per question.}
\label{fig:scalability_comparison}
\end{figure}

\section{Related Work}

Multi-hop inference is the task of combining two or more pieces of evidence to solve a particular reasoning problem. This task is often used to evaluate explainable inference models since the constructed chains of reasoning can be interpreted as an explanation for the final predictions \cite{wiegreffe2021teach,thayaparan2020survey}.

Given the importance of multi-hop inference for explainability, there is a recent focus on datasets and resources providing annotated explanations to support and evaluate the construction of valid inference chains \cite{xie-etal-2020-worldtree,jhamtani2020learning,Khot_Clark_Guerquin_Jansen_Sabharwal_2020,ferreira2020natural,khashabi2018looking,yang2018hotpotqa,mihaylov2018can,jansen2018worldtree,welbl2018constructing}. 
Most of the existing datasets for multi-hop inference, however, contain explanations composed of up to two sentences or paragraphs, limiting the possibility to assess the robustness of the systems on long reasoning chains. Consequently, most of the existing multi-hop models are evaluated on two-hops inference tasks \cite{li2021hopretriever,tu2020select,yadav2020unsupervised,fang2020hierarchical,asai2019learning,banerjee2019careful,yadav2019quick,thayaparan2019identifying}.

On the other hand, explanation regeneration on science questions is designed to evaluate the construction of long explanatory chains (with explanations containing up to 16 facts), in a setting where the structure of the inference cannot be derived from a direct decomposition of the questions \cite{xie-etal-2020-worldtree,jansen2019textgraphs,jansen2018worldtree}.
To deal with the difficulty of the task, state-of-the-art models leverage the cross-attention mechanism in Transformers \cite{vaswani2017attention}, learning to compose relevant explanatory chains supervised via human-annotated explanations \cite{cartuyvels-etal-2020-autoregressive,das2019chains,chia2019red,banerjee2019asu}.

The autoregressive formulation proposed in this paper is similar to the one introduced by \citet{cartuyvels-etal-2020-autoregressive}, which, however, perform iterative inference though a cross-encoder architecture based on BERT \cite{devlin2019bert}. Differently from this work, we present a hybrid architecture based on bi-encoders \cite{reimers-gurevych-2019-sentence} with the aim of maximising both accuracy and inference time in explanation regeneration. Our model of explanatory power is based on the work done by \citet{valentino2021unification}, which shows how to leverage explicit patterns in corpora of scientific explanations for multi-hop abductive inference. In this paper, we build upon previous work by demonstrating that models based on explicit patterns can be effectively combined with dense representation to achieve nearly state-of-the-art performance while preserving scalability. 

Our framework is related to recent work on dense retrieval for knowledge-intensive NLP tasks, which focuses on the design of scalable architectures with Maximum Inner Product Search (MIPS) based on Transformers \cite{xiong2021answering,zhao-etal-2021-multi-step,lin-etal-2021-differentiable,karpukhin-etal-2020-dense,NEURIPS2020_6b493230,dhingra2019differentiable}. Our multi-hop inference model is similar to \citet{lin-etal-2021-differentiable} and \citet{xiong2021answering} which adopt bi-encoders for open-ended commonsense reasoning and open-domain question answering. However, to the best of our knowledge, we are the first to integrate bi-encoders in a hybrid architecture for scalable abductive inference in the scientific domain. 

\section{Conclusion}

In this work, we focused on the problem of performing abductive natural language inference at scale, aiming at maximising both accuracy and inference time in scientific explanation regeneration. 

We proposed SCAR, a hybrid architecture, scalable to massive facts banks, that performs abductive inference autoregressively, conditioning the probability of retrieving a fact at time-step $t$ on the partial explanation constructed at time-step $t-1$. SCAR combines a Transformer-based bi-encoder with a sparse model that leverages explicit patterns in corpora of scientific explanations, defining a joint model of relevance and explanatory power for abductive inference.

An extensive evaluation demonstrated that the proposed abductive solver achieves performance comparable with that of state-of-the-art cross-encoders while being $\approx 50$ times faster. Moreover, we investigated the impact of the hybridisation on semantic drift and science question answering, showing that SCAR is more robust in the construction of challenging explanations requiring long reasoning chains.

This work demonstrated the effectiveness of hybrid models in enabling explainable natural language inference at scale, opening the way for future research at the intersection of latent and explicit models for explanation generation. As a future work, we plan to investigate the integration of relevance and explanatory power in a end-to-end differentiable architecture, and explore the applicability of the framework on additional commonsense and scientific reasoning tasks. 

