\section{Conclusion}
In this paper, we have firstly proved that existing mainstream GCN-based models have difficulty in well capturing the complicated non-linearity of graph data. Then, in order to better capture the complicated and non-linear distributions of nodes, we have proposed a novel GraphAIR framework that explicitly models the neighborhood interaction in addition to the neighborhood aggregation scheme. By employing residual learning strategy, we disentangle learning the neighborhood interaction from the neighborhood aggregation, which makes the optimization easier. The proposed GraphAIR is compatible with most existing graph convolutional models and it can provide a plug-and-play module for the neighborhood interaction. Finally, GraphAIR based on well-known models including GCN, GraphSAGE, and GAT have been thoroughly investigated through empirical evaluation. Extensive experiments on benchmark tasks including node classification and link prediction demonstrate the effectiveness of our model.

\section{Evaluation}
We extensively evaluate our proposed GraphAIR model on the node classification task and link prediction using five public datasets. Besides, we also conduct ablation studies on the neighborhood interaction module. For readers of interest, we include comparison of training time and all details of the experimental configurations in the supplementary material.

\subsection{Datasets}

We use five widely-used datasets to evaluate model performance on both transductive learning and inductive learning scenarios. Specifically, three citation networks (Cora, Citeseer, Pubmed) are used for tranductive node classification and link prediction, one knowledge graph (NELL) is used for transductive node classification, and one multi-graph molecular network (PPI) is for inductive node classification. We exactly follow the setup in \cite{Yang:2016ts,Kipf:2016tc,Kipf:2016ul,Velickovic:2018we}. The statistics of datasets used throughout the experiments are summarized in Table \ref{tab:datasets}.

\textbf{Citation networks.}
We build undirected citation networks from three datasets, where documents and citations are treated as nodes and edges respectively. We treat the bag-of-words of each document as the feature vector. Our goal is to predict the class of each document. Only twenty labels per class are used for training.

\textbf{Knowledge graph.}
The dataset collected from the knowledge base of Never Ending Language Learning (NELL) contains entities, relations, and text description. For every triplet \((e_1, r, e_2)\), where \(e_1\) and \(e_2\) are entities and \(r\) is the relationship between them, \(r\) will be assigned with two separate nodes \(r_1\) and \(r_2\). Then, we add two edges between \((e_1, r_1)\) and \((e_2, r_2)\). For the knowledge graph, we conduct the entity classification. Similarly, we use bag-of-words as feature vectors. Only one label per class is used for training.

\textbf{Molecular network.}
We use the PPI (protein-protein interaction) network that consists of twenty-four (24) graphs corresponding to different human tissues. Each node contains fifty (50) features composed of positional gene sets, motif gene sets, and immunological signatures. We select twenty (20) graphs as the training set, two (2) for validation, and two (2) for testing.

\begin{table}
	\centering
	\caption{Dataset statistics.}
	\resizebox{\columnwidth}{!} {
	\begin{tabular}{cccccccccc}
		\toprule
		Dataset & Cora & Citeseer & Pubmed & NELL & PPI \\
		\midrule
		Task  & \multicolumn{4}{c}{Transductive} & Inductive \\
		Type  & \multicolumn{3}{c}{Citation network} & Knowledge graph & Molecular \\
		\# Vertices & 2,708 & 3,327 & 19,717 & 65,755 & 56,944 \\
		\# Edges & 5,429 & 4,732 & 44,338 & 266,144 & 818,716 \\
		\# Classes & 7     & 6     & 3     & 210   & 121 \\
		\# Features & 1,433 & 3,703 & 500   & 5,414 & 50 \\
		\# Training nodes & 140   & 120   & 60    & 210   & 44,906 \\
		\# Test nodes & 1,000 & 1,000 & 1,000 & 1,000 & 5,524 \\
		\# Validation nodes & 500   & 500   & 500   & 500  & 6,514 \\
		\bottomrule
	\end{tabular}
	}
	\label{tab:datasets}
\end{table}

\subsection{Experiments on Node Classification}

\subsubsection{Baseline Methods}
We comprehensively compare our method with various traditional random-walk-based algorithms and state-of-the-art GCN-based methods. We closely follow the experimental setting of previous work; the performance of those baselines is reported as in their original papers.\footnote{In experiments, we found that the results reported in \citet{Hamilton:2017tp} after ten epochs did not converge to the best values. For a fair comparison with other models, we reuse its official implementation and report the results of the baselines after 200 epochs.}

\textbf{Transductive node classification.}
In the transductive setting, the baselines include skip-gram-based network embedding method DeepWalk \cite{Perozzi:2014ib}, graph convolutional networks with higher-order Chebyshev filters (Planetoid) \cite{Yang:2016ts}, graph convolution with one-hop neighbors (GCN) \cite{Kipf:2016tc}, and graph attention networks (GAT) \cite{Velickovic:2018we}. In addition, we further compare the performance of the proposed model with the recently proposed simplified graph convolutional networks (SGCs) \cite{Wu:2019vz} which removes redundant non-linear activations. Also, we modify graph isomorphic networks (GINs) \cite{Xu:2019ty} which utilize non-linear MLPs as the aggregation function for the node classification task. Note that since GIN was originally proposed for graph classification, we apply two GIN convolutional layers and remove the graph-level readout function for the transductive node classification task.

\textbf{Inductive node classification.}
For inductive node classification, we mainly compare GraphAIR with inductive graph convolutional networks (GraphSAGE) \cite{Hamilton:2017tp} and graph attention networks (GAT) \cite{Velickovic:2018we}. Note that GraphSAGE provides several variants of neighborhood aggregators: SAGE-GCN concatenates the features of the neighborhoods and the central node, SAGE-mean takes the average over neighborhood feature vectors, SAGE-LSTM combines neighborhood features by using a LSTM model, and SAGE-pool uses an element-wise max-pooling operator to aggregate the neighborhood information nonlinearly.

\subsubsection{Experimental Configurations}
We employ our GraphAIR framework on top of three representative models, including GCN, GraphSAGE, and GAT, which is denoted by AIR-GCN, AIR-SAGE, and AIR-GAT, respectively. Particularly, while GraphSAGE proposes several variants for neighborhood aggregation, among them only SAGE-mean satisfies the coefficient normalization in Eq. (\ref{eq:node-interaction}). Therefore, we select SAGE-mean as the base model for GraphAIR. For a fair comparison, we closely follow the same hyper-parameters setting as the underlying graph convolutional model, such as learning rate, dropout rate, weight decay factor, hidden dimensions, etc. Considering GIN is originally proposed for graph-level classification, the hidden dimensions are set to the same as GCN. In the experiment, we only tune the weights of three loss functions by grid search, where $\lambda_i \in [0.1, 0.2, \dots, 1.5], \forall i \in \{1,2,3\}$. For the transductive setting, we use the features of all data but only the labels of the training set are used for training. For the inductive setting, we train our model without the validation data and testing data. In addition, we report the average accuracy of 20 measurements.

\subsubsection{Results and Analysis}

\begin{table*}
	\small
	\centering
	\caption{Accuracies of node classification with the best performance highlighted in bold.}
	\resizebox{\textwidth}{!}{
	\subfloat[Transductive]{
	\begin{tabular}{ccccc}
		\toprule
		Method & Cora & Citeseer & Pubmed & NELL \\
		\midrule
		DeepWalk & 67.2\% & 43.2\% & 65.3\% & 58.1\% \\
		Planetoid & 75.7\% & 64.7\% & 77.2\% & 61.9\% \\
		GIN+0 & 78.3\% & 62.9\% & 78.0\% & 65.5\% \\
		GIN+\(\epsilon\) & 76.6\% & 63.8\% & 75.5\% & 63.5\% \\
		SGC   & 81.0\% \(\pm\) 0.0\% & 71.9\% \(\pm\) 0.1\% & 78.9\% \(\pm\) 0.0\% & 72.8\% \\
		GAT   & 83.0\% \(\pm\) 0.7\% & 72.5\% \(\pm\) 0.7\% & 79.0\% \(\pm\) 0.3\% & -- \\
		\bottomrule
	\end{tabular}
	\label{tab:transductive-results}
	}
	\quad
	\subfloat[Inductive]{
	\begin{tabular}{cc}
		\toprule
		Method & PPI \\
		\midrule
		Random & 39.6\% \\
		SAGE-GCN & 55.6\% \\
		SAGE-mean & 64.5\% \\
		SAGE-LSTM & 66.8\% \\
		SAGE-pool & 73.8\% \\
		GAT & 97.3\% \(\pm\) 0.2\% \\
		\midrule
		AIR-SAGE-mean & 66.3\% \(\pm\) 0.1\%\\
		AIR-GAT & \textbf{98.6\% \(\pm\) 0.2\%} \\
		\bottomrule
	\end{tabular}
	\label{tab:inductive-results}
	}
	}
\end{table*}

\textbf{Transductive.}
We summarize the results of transductive node classification in Table \ref{tab:transductive-results}. Note that even we apply the sparse version implementation of GAT, it requires more than 64G memory on NELL dataset. Thus, the performance of GAT and AIR-GAT is not reported. From the tables, it is seen that GraphAIR achieves state-of-the-art performance over all datasets, which demonstrates the effectiveness of the proposed GraphAIR framework. SGC acquires comparable results to that of GCN, which corresponds to our conclusion in Proposition 1 that existing GCNs are not able to learn the nonlinearity of graph data sufficiently. For our proposed AIR-GCN, it outperforms its base model GCN by margins of 3.2\%, 2.6\%, 1.0\%, and 2.5\%. The same trends hold for AIR-GAT with its base model GAT as well. To sum up, the improvements demonstrate the effectiveness of modeling the non-linear distributions of nodes.

In addition, another important observation is that, both AIR-GAT and AIR-GCN outperform the complex non-linear opponents such as GIN. Although MLPs are able to asymptotically approximate any complicated and non-linear functions theoretically, they tend to converge to undesired local minima in practice \cite{Rojas:1996wf}.
The experimental results prove the rationality of \emph{explicitly} introducing neighborhood interaction.

\textbf{Inductive.}
The results of inductive learning are shown in Table \ref{tab:inductive-results}. AIR-SAGE-mean outperforms its base model SAGE-mean by 1.8\%. Besides, we can clearly observe that AIR-GAT achieves the best performance. It is worth noting that the previous state-of-the-art method has already reached pretty high performance and the proposed AIR-GAT still acquires the improvement of 1.3\% over the vanilla GAT. Besides, it is suggested that the proposed GraphAIR framework is also generalizable for multiple graphs.

\subsection{Experiments on Link Prediction}

In order to further verify our proposed framework is general for other graph representation learning tasks, we conduct experiments on link prediction additionally. We choose citation networks as benchmark datasets and compare against various state-of-the-art methods, including graph autoencoders (GAE) \cite{Kipf:2016ul} and variational graph autoencoders (VGAE) \cite{Kipf:2016ul}, as well as other baseline algorithms, including SC \cite{Tang:2011cx} and DeepWalk \cite{Perozzi:2014ib}. We employ our GraphAIR framework on the basis of GAE, which constructs the graph autoencoder with GCNs. The resulting model is denoted by AIR-GAE.

We report the performance in terms of area under the ROC curve (AUC) based on the performance of 20 runs. The mean performance and standard error are presented in Table \ref{tab:link-prediction}. It is shown from the table that the proposed AIR-GAE outperforms its vanilla opponents GAE and VGAE, which once again verifies the necessity to incorporate the neighborhood interaction to neighborhood aggregation. Please note that previous state-of-the-art methods have already obtained high enough performance on the Pubmed dataset and our method AIR-GAE pushes the boundary with absolute improvements of 2.8\%, achieving 99.2\% in terms of AUC. Also, it can be observed that the proposed method obtains much more obvious improvements, compared with the performance of node classification. We suspect that this is primarily because models for the link prediction task usually employ pairwise decoders for calculating the probability of the link between two nodes. For example, GAE and VGAE assume the probability that there exists an edge between two nodes is proportional to the dot product of the embeddings of these two nodes. Therefore, our approach, which explicitly models the neighborhood interaction through the multiplication of the embeddings of two nodes, is inherently related to the link prediction task and obtains more improvements.

\begin{table}
	\small
	\centering
	\caption{AUC of link prediction in citation networks with the best performance highlighted in bold.}
	\label{tab:link-prediction}
	\resizebox{\columnwidth}{!}{
	\begin{tabular}{ccccccc}
	\toprule
	Method   & Cora  & Citeseer & Pubmed \\
	\midrule
	SC & 84.6\% \(\pm{}\) 0.01\% & 80.5\% \(\pm{}\) 0.01\% & 84.2\% \(\pm{}\) 0.02\% \\
	DeepWalk & 83.1\% \(\pm{}\) 0.01\% & 80.5\% \(\pm{}\) 0.02\% & 84.4\% \(\pm{}\) 0.00\% \\
	\midrule
	GAE & 91.0\% \(\pm{}\) 0.02\% & 89.5\% \(\pm{}\) 0.04\% & 96.4\% \(\pm{}\) 0.00\% \\
	VGAE & 91.4\% \(\pm{}\) 0.01\% & 90.8\% \(\pm{}\) 0.02\% & 94.4\% \(\pm{}\) 0.02\% \\
	\midrule
	AIR-GAE & \textbf{95.4\% \(\pm{}\) 0.01\%} & \textbf{95.0\% \(\pm{}\) 0.01\%}  & \textbf{99.2\% \(\pm{}\) 0.02\%}  \\
	\bottomrule
	\end{tabular}
	}
\end{table}

\subsection{Ablation Studies on the Neighborhood Interaction Module}

As we analyzed in Section 3.3, the number of parameters in GraphAIR is almost two times than that of the underlying graph convolutional model. In this section, we conduct ablation studies to answer the following questions:
\begin{itemize}
	\item \textbf{Q1}: How much improvement has the proposed neighborhood interaction module brought?
	\item \textbf{Q2}: Does the disentangled residual learning strategy bring sufficient improvements?
\end{itemize}

To answer Q1 and verify the effectiveness of GraphAIR is introduced by the proposed neighborhood interaction module rather than the larger number of parameters in the model, we remove the neighborhood interaction module of AIR-GCN. Then, the resulting model has exactly the same parameters as AIR-GCN. As there are almost double parameters than vanilla GCN in the resulting model, we denote the resulting model as DP-GCN (Double-Parameter GCN).

To answer Q2, we employ only one branch of graph convolutional networks consisting \((K+1)\) layers to produce the output representations. To obtain neighborhood interaction \(\bm{h}^\text{ir}\), we directly make use of the self-interaction strategy described in Eq. (9) instead of Eq. (11). The resulting model is termed as self-IR-GCN.

For a fair comparison, other experimental configurations are kept the same as AIR-GCN. The results of node classification are presented in Table \ref{tab:ablation-study}. It is seen from the table that the proposed AIR-GCN achieves the best performance and outperforms DP-GCN and self-IR-GCN. For Q1, we can observe that DP-GCN only obtains slightly better accuracy on Cora and Citeseer and almost the same performance as the vanilla GCN on Pubmed. It can be verified that the neighborhood interaction module mainly contributes to the performance improvement of the proposed AIR-GCN model. For Q2, it is seen that the performance of self-IR-GCN only gets slightly improved on three datasets, which demonstrates the rationality of modeling neighborhood interaction. However, disengaging the neighborhood interaction from neighborhood aggregation can bring more improvements.

\begin{table}
	\small
	\centering
	\caption{Accuracies of node classification in the ablation study.}
	\resizebox{\columnwidth}{!}{
	\begin{tabular}{ccccc}
		\toprule
		Method & Cora & Citeseer & Pubmed \\
		\midrule
		GCN   & 81.5\% & 70.3\% & 79.0\% \\
		DP-GCN & 82.3\% \(\pm\) 0.1\% & 71.0\% \(\pm\) 0.1\% & 79.0\% \(\pm\) 0.2\% \\
		self-IR-GCN & 82.6\% \(\pm\) 0.0\% & 70.8\% \(\pm\) 0.2\% & 79.2\% \(\pm\) 0.1\% \\
		AIR-GCN & {84.7\% \(\pm\) 0.1\%} & {72.9\% \(\pm\) 0.1\%} & {80.0\% \(\pm\) 0.1\%} \\
		\bottomrule
	\end{tabular}
	}
	\label{tab:ablation-study}
\end{table}

\section{Introduction}
Graph representation learning aims to transform nodes on the graph into low-dimensional dense vectors whilst still preserving the attribute features of nodes and structure features of graphs. These node embeddings can then be fed into downstream machine learning algorithms to facilitate graph analytical tasks, such as node classification \cite{Kipf:2016tc,Velickovic:2018we}, link prediction \cite{Chen:2018vh}, and community detection \cite{Chen:2019vl}.

In recent years, there has been a surge of research interest in utilizing neural networks to handle graph-structured data. Among them, graph convolutional networks (GCNs) have been shown effective in graph representation learning. They can model complex attribute features and structure features of graphs and achieve the state-of-the-art performance on various tasks. The core of graph convolution is that nodes learn their representations by aggregating features from their neighbors, i.e. the ``neighborhood aggregation'' scheme. Recently, some graph convolutional models, which primarily differ in the neighborhood aggregation strategies, have been proposed \cite{Kipf:2016tc,Velickovic:2018we,Zhuang:2018ks,Xu:2018vn}. For example, GCN \cite{Kipf:2016tc} can be seen as the approximation of aggregation on the first-order neighbors; GraphSAGE \cite{Hamilton:2017tp} designs several aggregators for inductive learning, where unlabeled data does not appear in the training process; GAT \cite{Velickovic:2018we} introduces the attention mechanism to model influence of neighbors with learnable parameters.

From a historical perspective, machine learning research has gone through a long process of development, with one clear trend from simple and linear models to complex and non-linear models. For example, limitations of the linear support vector machine (SVM) motivated the development of non-linear and more expressive kernel-based SVM classifiers \cite{Boser:1992uo}.
Besides, similar trends can be observed in the realm of image processing as real-world data distribution is usually rather complex. For example, simple and linear image filters \cite{Harris:1988tk} are gradually superseded by non-linear convolutional neural networks (CNNs) \cite{LeCun:1989wh}.
Driven by the significance of modeling complex and non-linear distributions of data, a question arises: \textit{are existing GCNs capable enough to model the complex and non-linear distributions of graphs?} We find that most previous graph convolutional models (e.g., GCN and GAT) are usually shallow with only one or two non-linear activation function layers, which may restrict the model from well capturing the complicated non-linearity of graph data.

In this paper, we first theoretically prove that the effect of non-linear activation functions in GCNs is to introduce the interaction terms of neighborhood features. We then show that coefficients of the neighborhood interacting terms are relatively small in current GCN-based models. To this end, we present a general framework named GraphAIR (\underline{A}ggregation and \underline{I}nte\underline{R}action). The key idea behind our approach is to \textit{explicitly model the neighborhood interaction in addition to neighborhood aggregation, which can better capture the complex and non-linear node features}.
As illustrated in Figure \ref{fig:node-interaction}, GraphAIR consists of two parts, i.e. \emph{aggregation} and \emph{interaction}. The aggregation module constructs node representations by combining features from neighborhoods; the interaction module explicitly models neighborhood interactions through multiplication.

\begin{figure}
	\centering
		\includegraphics[width=\columnwidth]{figures/node-interactions.pdf}
	\caption{
	A graphical illustration of the proposed GraphAIR model. The aggregation module sums up the neighborhood features; the interaction module models the pair-wise feature interaction among the neighborhoods.}
	\label{fig:node-interaction}
\end{figure}

Nevertheless, several challenges exist in modeling the neighborhood interaction. Firstly, different nodes may have various numbers of adjacent neighbors, leading to different numbers of interaction pairs among neighbors. Thereby, defining a universal neighborhood interaction operator which is able to handle arbitrary numbers of interaction pairs is challenging. Secondly, it is preferable to propose a general plug-and-play interaction module instead of designing model-specific neighborhood interaction strategies for different GCN-based models.

To tackle the aforementioned challenges, we derive that the neighborhood interaction can be easily obtained through the multiplication of node embeddings.
As a result, both of the neighborhood aggregation module and the neighborhood interaction module can be implemented by most existing graph convolutional layers.

In a nutshell, the main contributions of this paper are three-fold. Firstly, to best of our knowledge, it is the first work to explicitly model neighborhood interaction for capturing non-linearity of graph-structured data. Secondly, the proposed GraphAIR can easily integrate off-the-shelf graph convolutional models, which shows favorable generality. Thirdly, extensive experiments conducted on benchmark tasks of node classification and link prediction show that GraphAIR achieves the state-of-the-art performance.

\section{The Proposed Method: GraphAIR}

In this section, we firstly formulate the model of neighborhood interaction and then describe how the parameters of GraphAIR model can be learned. Finally, we summarize the overall model architecture and analyze the computational complexity.

\subsection{Modeling the Neighborhood Interaction with Residual Functions}

As discussed in Section \ref{sec:aggregators}, the node representation resulting from the neighborhood aggregation scheme is less likely to well capture complicated non-linearity of graphs because they learn the neighborhood interaction implicitly and inefficiently. In this section, we describe the embedding generation algorithm of GraphAIR, which aims to incorporate the neighborhood interaction into node representations. To begin with, a natural idea to model the quadratic terms of neighborhood interaction is formulated as:
\begin{equation}
	\bm{h}_i^\text{ir} = \sum_{j\in\mathcal{N}_i} \sum_{k\in\mathcal{N}_i} \beta_{jk} \bm{n}_j \odot \bm{n}_k,
\end{equation}
where \(\bm{h}_i^\text{ir}\) is the neighborhood \underline{i}nte\underline{r}action representation of node \(i\), \(\beta_{jk}\) denotes the coefficient of the quadratic term, and \(\odot\) is the element-wise multiplication operator. However, it is infeasible to learn \(\beta_{jk}\) in our case. For each node \(i\), there are \(O(|\mathcal{N}_i|^2)\) coefficients to estimate, which exposes the risk of overfitting. To alleviate this problem, we simply assign \(\beta_{jk}\) as the product of importance weights \(e_{ij}\) and \(e_{ik}\). The simplification is reasonable with the following aspects. For node \(i\), if \(e_{ij}\) and \(e_{ik}\) are large, then the neighbor nodes \(j\) and \(k\) should be considered as important factors for the representation of node \(i\). Compared to other interacting terms, the interaction between node \(j\) and \(k\) are likely to provide more relevant information about node \(i\). Consequently, \(\beta_{jk}\) should be large.
In contrast, if \(e_{ij}\) and \(e_{ik}\) are small, neighbor nodes \(j\) and \(k\) may have a slight impact on node \(i\). Thus the interacting coefficient should be small as well. Formally, we arrive at:
\begin{equation}
	\begin{aligned}
		\bm{h}_i^\text{ir} & = \left(\sum_{j \in \mathcal{N}_i} e_{ij} \bm{n}_j\right) \odot \left(\sum_{k \in \mathcal{N}_i} e_{ik} \bm{n}_k\right) \\
		& = \left(\sum_{j \in \mathcal{N}_i} e_{ij} \bm{h}_j \bm{W}\right) \odot \left(\sum_{k \in \mathcal{N}_i} e_{ik} \bm{h}_k \bm{W}\right) = \bm{h}_i^\text{agg} \odot \bm{h}_i^\text{agg},
	\end{aligned}
	\label{eq:preliminary-interaction}
\end{equation}
where \(\bm{h}_i^\text{agg} = \sum_{j \in \mathcal{N}_i} e_{ij} \bm{h}_j \bm{W}\) denotes the representation resulting from neighborhood aggregation.

In order to introduce more non-linearity to our model, we apply non-linear activation function on the two representations resulting from neighborhood aggregation and neighborhood interaction respectively. Besides, to combine these two representations, we add them using a skip connection:
\begin{equation}
	\bm{h}_i^\text{air} = \sigma(\bm{h}_i^\text{agg}) + \sigma(\bm{h}_i^\text{ir}).
	\label{eq:preliminary-residual}
\end{equation}
However, although we adopt a skip connection here, we argue that we still cannot benefit from residual learning, where both of the suboptimal representation and the residual function are implemented by \emph{different non-linear layers}. As formulated in Eqs. (\ref{eq:preliminary-interaction},\ref{eq:preliminary-residual}), the two representations resulting from neighborhood aggregation and interaction are based on the \emph{same} weight matrix \(\bm{W}\), which means the variations of the two representations during the back-propagation process are highly correlated. According to \citet{Bengio:2013bu}, it is important to disentangle the factors of variation to the representations as only a few factors tend to change at a time. Therefore,
to make use of residual learning which can ease the optimization, we introduce another weight matrix \(\bm{W}^\prime \in \mathbb{R}^{d_k \times d_{k + 1}}\) to disentangle learning the neighborhood interaction from neighborhood aggregation. Formally, instead of Eq. (\ref{eq:preliminary-interaction}), we use the following equation to learn the neighborhood interaction in our model:
\begin{equation}
	\bm{h}_i^\text{ir} = \left(\sum_{j \in \mathcal{N}_i} e_{ij} \bm{h}_j \bm{W}\right) \odot \left(\sum_{k \in \mathcal{N}_i} e_{ik} \bm{h}_k \bm{W}^\prime\right) = \bm{h}_i^\text{agg} \odot \bm{\bar{h}}_i^\text{agg},
	\label{eq:node-interaction}
\end{equation}
where the first term \(\bm{h}_i^\text{agg} = \sum_{j \in \mathcal{N}_i} e_{ij} \bm{h}_j \bm{W}\) denotes the representation resulting from neighborhood aggregation and the second term \(\bm{\bar{h}}_i^\text{agg} = \sum_{k \in \mathcal{N}_i} e_{ik} \bm{n}_k \bm{W}^\prime\) provides the other half node representation for multiplication in the interaction process. \(\bm{h}_i^\text{agg}\) is the input representation to the residual module and \(\bm{W}^\prime\) is the learnable weight of the residual function. Note that both terms \(\bm{h}_i^\text{agg}\) and \(\bm{\bar{h}}_i^\text{agg}\) can be implemented by existing graph convolutional layers. Thus the proposed GraphAIR framework is compatible with most existing GCN-based models and it provides a plug-and-play module for the neighborhood interaction.

\subsection{Learning the Parameters of GraphAIR}

In this section, we introduce how to learn the parameters under the GraphAIR framework. As we aim to propose a general approach for graph representation learning, we can apply different kinds of graph-based loss function, such as the proximity ranking loss in link prediction tasks and the cross-entropy loss in node classification tasks. Without loss of generality, we take the task of node classification as an example.

To compute the probability that each node belongs to a certain class, existing GCN-based models usually employ one additional graph convolutional layer with a softmax classifier for prediction. Then, the output representation \(\bm{z}_i\) is formulated as:
\begin{equation}
	\bm{z}_i = g\left(\bm{h}_i^{(k)}\right) = \operatorname{softmax} \left( \sum_{j\in\mathcal{N}_{i}}e_{ij} \bm{h}_j^{(k)} \bm{W}^{(k + 1)} \right),
	\label{eq:output}
\end{equation}
where \(g(\cdot)\) is the prediction function, \(\bm{W}^{(k + 1)} \in \mathbb{R}^{d_k \times |\mathcal{Y}|}\), and \(|\mathcal{Y}|\) is the number of classes. Then, the loss of node classification can be calculated as
\(\mathcal{L} = \frac{1}{n}\sum_{i = 1}^n \mathcal{L}_\text{clf}(\bm{z}_i, \bm{y}_i)\)
where \(\bm{y}_i\) is the true label for node \(i\) and \(\mathcal{L}_\text{clf}\) is the cross-entropy loss.

To obtain more accurate node embeddings \(\bm{h}_i^\text{agg}\) and \(\bm{\bar{h}}_i^\text{agg}\), we apply two auxiliary classifiers on \(\bm{h}_i^\text{agg}\) and \(\bm{\bar{h}}_i^\text{agg}\). Subsequently, the resulting representation \(\bm{h}_i^\text{ir}\) for the neighborhood interaction will be more precise as well. Then, as formulated in Eq. (\ref{eq:output}), we apply one additional graph convolutional layer on each of \(\bm{h}_i^\text{air}\), \(\bm{h}_i^\text{agg}\), and \(\bm{\bar{h}}_i^\text{agg}\) to attain \(\bm{z}_i^\text{air}\), \(\bm{z}_i^\text{agg}\), and \(\bm{\bar{z}}_i^\text{agg}\). Eventually, the overall objective function is the weighted sum of the three losses:
\begin{equation}
	\begin{aligned}
		\mathcal{L}_\text{total} &= \frac{1}{n}\sum_{i = 1}^n \left[ \lambda_1 \mathcal{L}_\text{clf}(\bm{z}_i^\text{air}, \bm{y}_i) + \lambda_2 \mathcal{L}_\text{clf}(\bm{z}_i^\text{agg}, \bm{y}_i) + \lambda_3 \mathcal{L}_\text{clf}(\bm{\bar{z}}_i^\text{agg}, \bm{y}_i) \right] \\
		&= \lambda_1 \mathcal{L}^\text{air} + \lambda_2 \mathcal{L}^\text{agg} + \lambda_3 \mathcal{\bar{L}}^\text{agg}, \\
	\end{aligned}
\end{equation}
where \(\lambda_1\), \(\lambda_2\), and \(\lambda_3\) are hyperparameters controlling weights of the three loss functions. For training, we minimize the total loss \(\mathcal{L}_\text{total}\), while for inference, we only use \(\bm{z}_i^\text{air}\), since \(\bm{z}_i^\text{agg}\) and \(\bm{\bar{z}}_i^\text{air}\) are to ensure \(\bm{h}_i^\text{air}\) is accurate enough.

\subsection{Model Architecture and Complexity Analysis}

We suppose there are \((K + 1)\) layers in the underlying graph convolutional model, where the last layer is employed for node classification. For GraphAIR, we employ two separate and symmetric branches, each of which consists of \(K\) graph convolutional layers to obtain \(\bm{h}_i^\text{agg}\) and \(\bm{\bar{h}}_i^\text{agg}\). Then, considering \(\bm{h}_i^\text{agg}\) and \(\bm{\bar{h}}_i^\text{agg}\) have aggregated enough information from neighborhoods, here we conduct the neighborhood interaction only once by multiplying \(\bm{h}_i^\text{agg}\) and \(\bm{\bar{h}}_i^\text{agg}\) for the sake of efficiency. Additionally, we employ three graph convolutional layers followed by softmax activation functions on \(\bm{h}_i^\text{air}\), \(\bm{h}_i^\text{agg}\), and \(\bm{\bar{h}}_i^\text{agg}\). In summary, there will be \((2K + 3)\) layers in GraphAIR.

Each layer in GraphAIR has the same space and time complexity as the underlying model and the additional computation cost of GraphAIR is mainly introduced by the multiplication process for the neighborhood interaction. For the neighborhood interaction in Eq. (\ref{eq:node-interaction}), the cost is \(O(nd)\) where \(d\) is the embedding dimension. For each layer of the existing graph convolutional model such as GCN and GAT, it takes \(O(n^2d)\) time to proceed Eq. (\ref{eq:summarization-aggregator}). Therefore, the additional computation cost of neighborhood interaction is insignificant. That is to say, our proposed approach is as asymptotically efficient as the underlying graph convolutional model.

\section{Background and Preliminaries}

In this section, we firstly introduce the notations used throughout the paper and then summarize some of the most common GCN models. Last, we briefly introduce residual learning which we employ in our model.

\subsection{Notations}

Let \(\mathcal{G} = (\bm{A}, \bm{X})\) be an undirected graph with \(n\) nodes, where \(\bm{A} \in \mathbb{R}^{n \times n}\) is the adjacency matrix, \(\bm{X} \in \mathbb{R}^{n \times m}\) is the feature attribute matrix, and \(\bm{x}_i \in \mathbb{R}^{1 \times m}\) denotes the attribute of node \(i\). Please kindly note that in this paper we primarily focus on undirected graphs, but our proposed method can be easily generalized to work with weighted or directed graphs.

\subsection{Aggregators in Graph Convolutional Models}
\label{sec:aggregators}

As mentioned above, existing GCNs mainly differ in the neighborhood aggregation functions. The representative graph convolutional model such as GCN \cite{Kipf:2016tc} and GAT \cite{Velickovic:2018we} can be formulated as:
\begin{align}
	\bm{n}_i^{(k)} &= \bm{h}_i^{(k)} \bm{W}^{(k)}, \label{eq:weighted-representation}\\
	\bm{h}_i^{(k + 1)} &= \sigma\left( \sum_{j\in\mathcal{N}_i} e_{ij} \bm{n}_j^{(k)} \right), \label{eq:summarization-aggregator}
\end{align}
where \(\bm{h}_i^{(k)} \in \mathbb{R}^{d_k}\) is the embedding of the \(i\)\textsuperscript{th} node resulting from the \(k\)\textsuperscript{th} graph convolutional layer, \(\bm{W}^{(k)} \in \mathbb{R}^{d_k \times d_{k + 1}}\) is a learnable weight matrix, \(e_{ij}\) is a scalar which indicates the importance of node \(j\)'s features to node \(i\), and \(\sum_{j} e_{ij} = 1\). \(\sigma(\cdot)\) is the activation function, e.g., \(\operatorname{ReLU}(\cdot) = \max(0, \cdot)\) and \(\mathcal{N}_i\) is the set containing the first-order neighbors of node \(i\) as well as node \(i\) itself. To obtain the node embedding, a linear transformation is first conducted to project features to a new feature subspace. Then, the node embedding can be updated by weighted summation over the projected features of its neighbors, followed by a non-linear activation function.

Different models adopt different strategies to design the aggregators. For GCN, it uses a predefined weight matrix \(\hat{\bm{A}} = \tilde{\bm{D}}^{-\frac{1}{2}}\tilde{\bm{A}}\tilde{\bm{D}}^{-\frac{1}{2}}\) for summarization, where \(\tilde{\bm{A}} = \bm{A} + \bm{I}\) is the adjacency matrix with self-loops and \(\tilde{D}_{ii} = \sum_j \tilde{A}_{ij}\). Here, entry \(a_{ij}\) of \(\hat{\bm{A}}\) is a predefined weight factor for weighted summarization over neighborhoods, i.e. \(e_{ij}\) in Eq.(\ref{eq:summarization-aggregator}). Unlike GCN, GAT makes use of the attention mechanism to explicitly learn \(e_{ij}\) as follows:
\begin{equation}
	\begin{aligned}
		\alpha_{ij} &= g(\bm{n}_i, \bm{n}_j), \\
		e_{ij} &= \operatorname{softmax}(\alpha_{ij}) = \frac{\exp{(\alpha_{ij})}}{\sum_{k \in \mathcal{N}_i} \exp{(\alpha_{ik})}},
	\end{aligned}
\end{equation}
where \(g: \mathbb{R}^{d} \times \mathbb{R}^{d} \rightarrow \mathbb{R}\) is a self-attention function, which can be simply implemented as a feed-forward neural network.

\textbf{The implicit and insufficient neighborhood interaction involved in existing GCNs.}
It is seen from Eq. (\ref{eq:summarization-aggregator}) that without the activation function, the node representation would depend linearly on the neighborhood features. Then, although mainstream models adopt non-linear activation functions, which is able to introduce the neighborhood interaction implicitly as a side effect, they still face challenges in learning the neighborhood interaction sufficiently. We take the sigmoid function \(s(t) = \frac{1}{1+e^{-t}}\) as an example and approximate it with Taylor polynomials. Note that mainstream GCN-based models use piecewise non-saturating activation functions, such as \(\operatorname{ReLU}\) and \(\operatorname{LeakyReLU}(x)=\max(0.01x, x)\). These functions suppress negative values yet are still linear for positive values. Here we analyze the sigmoid function as it brings more non-linearity. Since the elements in the node embeddings are small\footnote{Most existing graph convolutional models, including GCN, GraphSAGE, and GAT normalize the input and initialize the weights using Glorot initialization \cite{Glorot:2010uc}.}, the high-order interacting terms among the neighborhoods are small as well. Then, we just analyze the coefficients of high-order interacting terms, which is claimed in the following proposition.
\begin{proposition}
	When applying the sigmoid function \(s(t)\) on the result of the linear combination as formulated in Eq. (\ref{eq:summarization-aggregator}), the equivalent coefficient of high-order interacting terms of the neighborhood embeddings is at most \(\frac{1}{48}\).
	\label{prop:implicit-interaction}
\end{proposition}

\begin{proof}
	The sigmoid function \(s(t)\) can be approximated as Taylor polynomials at \(t_0 = 0\):
	\begin{equation}
		s(t) \approx \sum_{p=0}^P \frac{s^{(p)}(0)}{p!} t^{p} = \frac{1}{2} + \frac{1}{4}t - \frac{1}{48}t^3 + \dots + \frac{s^{(P)}(0)}{P!} t^P,
	\end{equation}
	where \(P\) is the degree of the polynomial. The approximation error can be bounded using the Lagrange form of the remainder:
	\begin{equation}
		\begin{aligned}
			|R_p(t)| \leq & \frac{|t|^{P + 1}}{(P + 1)!}M_p, \quad \\
			& \text{where } |s^{P+1}(\theta)| \leq M_p, \quad \theta \in (-t, t).
		\end{aligned}
	\end{equation}
	Since the coefficient of the quadratic term is zero, we set \(P = 2\) and analyze the contribution of high-order interacting terms. Then, replacing \(t\) with \(\sum_{j \in \mathcal{N}_i} e_{ij}\bm{n}_j^{(k)}\), Eq. (\ref{eq:summarization-aggregator}) can be written as follows:
	\begin{equation}
		\begin{aligned}
		& \bm{h}_i^{(k + 1)} = \frac{1}{2} + \frac{1}{4} \left( \sum_{j \in \mathcal{N}_i} e_{ij}\bm{n}_i^{(k)} \right) \\
		& + M\left(\sum_{j \in \mathcal{N}_i}\sum_{k \in \mathcal{N}_i}\sum_{l \in \mathcal{N}_i} e_{ij}e_{ik}e_{il}\bm{n}_j^{(k)} \cdot \bm{n}_k^{(k)} \cdot \bm{n}_l^{(k)} \right),
		\end{aligned}
	\end{equation}
	where \(M\) is the bound of the remainder whose absolute value is at most \(\frac{1}{48}\), which concludes the proof. Detailed proof is given in Appendix C in the supplementary material.
\end{proof}

\begin{remark}
	Proposition \ref{prop:implicit-interaction} states that the effect of non-linear activation functions in GCNs is to introduce the interaction terms of neighborhood features. Importantly, the coefficients of the neighborhood interacting terms in current GCN-based models are relatively small, leading to a negligible contribution to node representations. As existing GCNs are usually shallow with only one or two non-linear layers to avoid oversmoothing and overfitting \cite{Li:2018wc}, non-linearity of graph data cannot be learned sufficiently.
\end{remark}

\subsection{Residual Learning}

In this paper, we employ residual learning to combine the neighborhood aggregation and interaction. Residual learning \cite{He:2016ib} is a widely-used building block for deep learning. Suppose \(h(\bm{x})\) is the true and desired mapping and \(\bm{x}\) is the suboptimal representation which serves as the input feature to the residual module. Residual learning can be formulated as:
\begin{equation}
	h(\bm{x}) = f(\bm{x}) + \bm{x},
\end{equation}
where \(f(\cdot)\) is a residual function. Practically, we can apply a few non-linear layers to obtain the suboptimal representation \(\bm{x}\) and some other non-linear layers to implement the residual function \(f\). The essence of residual learning lies in the skip connection, through which the earlier representations are able to flow to later layers. The skip connection enables more direct reuse of the suboptimal representation and improves the information flow during forward and backward propagation \cite{He:2016ib}, which makes the network easier to be optimized. Many approaches \cite{He:2016ib,Li:2018ta} have shown that residual learning helps break away from the local optimum and improving the performance.

\section{Related Work}

There have been a lot of attempts in recent literature to employ neural networks for graph representation learning. Among them, graph convolutional neural networks (GCNs) receive a lot of research interests. GCN-based models generally follow the neighborhood aggregation scheme. To be specific, the model passes the input signals from neighborhoods through filters to aggregate information. Many approaches design different strategies to aggregate information from nodes' neighborhood. According to different strategies, these models can be roughly grouped into two categories, i.e. spectral-based approaches and spatial-based approaches.

One the one hand, spectral methods depend on the Laplacian eigenbasis to define parameterized filters. The first work \cite{Bruna:2014vg} introduce convolutional operations in the Fourier domain by computing the eigendecomposition of the graph Laplacian, which results in potentially heavy computational burden. Following its work, \citet{Defferrard:2016vo} propose to approximate filters using Chebyshev expansion of the graph Laplacian. Then, graph convolutional neural networks (GCNs) \cite{Kipf:2016tc} have been widely applied for graph representation learning. The core of GCNs is the neighborhood aggregation scheme which generates node embedding by combining information from neighborhoods. Since GCN only captures local information, DGCN \cite{Zhuang:2018ks} then proposes to construct an information matrix to encode global consistency.

On the other hand, the spatial approaches directly operate on spatially close neighbors. To enable parameter sharing of filters across neighbors of different sizes, \citet{Duvenaud:2015ww} first propose to learn weight matrices for different node degrees. MoNet \cite{Monti:2017ih} proposes a spatial-domain model to provide a unified convolutional network on graphs. To compute node representations in an inductive manner, GraphSAGE \cite{Hamilton:2017tp} samples fixed-size neighborhoods of nodes and performs aggregation over them. Similarly, \citet{Gao:2018dj} select a fixed number of neighbors and enable the use of conventional convolutional operations on Euclidean spaces. Recently, GAT \cite{Velickovic:2018we} introduces attention mechanisms to graph neural networks, which computes hidden representations by attending over neighbors with a self-attention strategy.

Recently, some methods are proposed to focus on linearity and non-linearity of graphs respectively. On the one hand, simplified graph convolutional networks (SGCs) \cite{Wu:2019vz} try to reduce the complexity and eliminate redundant computation of GCN by successively removing non-linear activation functions. SGC makes assumptions that non-linearity between GCN layers is not critical to the model performance and the majority of the benefit is brought by the neighborhood aggregation scheme. While being more computationally efficient, SGC achieves comparable empirical performance to vanilla GCN.

There are other methods arguing that modeling non-linear distributions of node features can bring improvements. For example, GraphSAGE-LSTM \cite{Hamilton:2017tp} employs the long-short-term memory (LSTM) module to learn the complex relationships between the nodes. Empirically, GraphSAGE-LSTM outperforms other aggregation functions such as GraphSAGE-mean and GraphSAGE-GCN. Graph isomorphic networks (GIN) \cite{Xu:2019ty} apply multilayer perceptrons (MLPs) in each graph convolutional layer, which is able to model complex non-linearity of graphs. Although theoretically it is well known that MLPs are universal approximators \cite{Hornik:1991te}, there is no formal theorem giving instructions on how to asymptotically approximate the desired function (\citeauthor{Patterson:1996ug} \citeyear{Patterson:1996ug}, p.~182; \citeauthor{Fausett:1994tq} \citeyear{Fausett:1994tq}, p.~328). Different from GraphSAGE-LSTM and GIN, to best of our knowledge, our work is the first to point out that most existing GCNs may not well capture non-linearity of graph data and we demonstrate the effectiveness of explicitly modeling non-linearity of graphs.


\section{Introduction}
A massive star with a mass larger than $8-10$  $\mathrm{M}_{\odot}$
can end its life in a core-collapse supernova (CCSN) explosion
\cite{Colgate:1966ax}.
During this process, a huge amount of  energy
 is released of which almost 99\% is
in the form of neutrinos of all flavors. Although an infinitesimal fraction
of the released energy is in the form of electromagnetic radiation, only $\sim$ 0.01\%,
it is already enough to allow a CCSN to  outshine its
host galaxy.

Neutrinos can
experience flavor oscillations
as they propagate.
Neutrino oscillations  can, at least in principle, notably impact the physics of CCSNe.
Firstly,  it can affect the heavy elements nucleosynthesis
occurring in the SN environment by modifying
the $n/p$ ratio through the weak reactions
  $\bar\nu_e + p \rightleftharpoons n + e^+$ and
  $\nu_e + n \rightleftharpoons p + e^-$ \cite{Qian:1996xt}.
 Neutrino oscillations can  modify  the
$\nu_e$ and $\bar\nu_e$ spectra which in turn modifies
the rate of these reactions.

 Moreover,  the SN dynamics can also
 be affected by neutrino oscillations.
 Although the mechanism of the  explosion is not well-understood yet,
 a popular mechanism proposed in
the late 1980s is the so-called \textit{delayed explosion mechanism}
in which the explosion is aided by absorbing a fraction of the energy of
 neutrinos emitted from the SN core
\cite{Bethe:1984ux, Janka:2012wk, Burrows:2012ew}.
Likewise, neutrino oscillations can  modify  the neutrino energy deposition
by changing  $\nu_e$ and $\bar\nu_e$ spectra.

Finally, to confront theory with observations,
predictions of the neutrino fluxes and spectra are crucial for future observations
 of CCSNe neutrino signals and the measurement of the neutrino diffuse background
\cite{Gava:2009pj, Horiuchi:2008jz, Beacom:2010kk,
   Mirizzi:2015eza,Horiuchi:2017qja, Suwa:2019svl}.

The phenomenon of neutrino oscillations in dense neutrino media,
such as the one expected in the SN environment,
 is remarkably
different from the one in vacuum and matter.
Neutrinos
can experience collective flavor evolution
in a dense neutrino gas
owing to the fact that
the coherent forward scattering
 by the background neutrino gas can now play a role in neutrino evolution

 because of the
 large neutrino number densities \cite{Pastor:2002we,duan:2006an, duan:2006jv, duan:2010bg,
  Chakraborty:2016yeg}.

 Most of the initial understanding of neutrino
 flavor evolution in the SN environment
 was based on the stationary spherically symmetric
 neutrino bulb model  \cite{duan:2006an} in which neutrinos are emitted isotropically
 from the surface of the neutrinosphere.
  The salient feature of the results obtained
   in this model
  is the existence
 of the flavor swapping phenomenon in which $\nu_e$ ($\bar\nu_e$) exchanges
 its spectra with $\nu_x$ ($\bar\nu_x$) for a range of neutrino
 energies  \cite{duan:2006jv, duan:2007sh, dasgupta:2009mg, duan:2010bg,
 Galais:2011gh, Duan:2007bt}.
  This is indeed a consequence of  collective neutrino
 oscillations in the SN environment.

 However, neutrinos could also undergo the so-called \textit{fast} flavor conversions
on  scales  as short as a few cm
in the densest regions of the SN core
\cite{Sawyer:2005jk, Sawyer:2015dsa,
 Chakraborty:2016lct, Izaguirre:2016gsx,  Wu:2017qpc,
  Capozzi:2017gqd, Richers:2019grc,
  Dasgupta:2016dbv, Abbar:2017pkh, Abbar:2018beu, Capozzi:2018clo,
 Martin:2019gxb, Capozzi:2019lso, Doring:2019axc, Chakraborty:2019wxe, Johns:2019izj}.
Unlike the traditional  collective modes which occur on scales determined
by  the neutrino vacuum frequency $\omega = \Delta m^2/2E $
($\sim  \mathcal O  (1)$ km for a $10$ MeV neutrino and atmospheric mass splitting),
 fast modes  occur
 on scales $\sim G_{\mathrm{F}}^{-1} n_\nu ^{-1}$
 with $n_\nu$ and $G_{\mathrm{F}}$ being the neutrino number
 density and the Fermi coupling constant, respectively.
 It is believed that a necessary condition for the occurrence of fast modes is
   the presence of crossing(s) in the angular distribution of
 electron lepton number carried by neutrinos (ELN)
 \cite{Izaguirre:2016gsx, Capozzi:2017gqd,
 Dasgupta:2016dbv, Abbar:2017pkh}.
 It has been shown that in the presence of ELN crossings,
 fast modes can arise due to the merging of
 two non-collective modes \cite{Capozzi:2019lso} and  $G_{\rm{F}} n_\nu$ can play the role of $\omega$
which in turn allows for the existence of flavor conversion
modes on relatively short scales
 \cite{Abbar:2017pkh}.

 Fast modes, if exist,
 can remarkably  influence the physics
of CCSNe. On the one hand, they can lead to
neutrino flavor conversions
within the SN zones that have long been thought to be the realm
of scattering processes (occurring on scales $\sim G_{\rm{F}}^{-2} E^{-2} n_{\rm{B}} ^{-1}$
with $n_{\rm{B}}$ being the baryon number density)
\cite{Cirigliano:2017hmk, Capozzi:2018clo}.
On the other hand, they
can result in flavor conversions  close to the
surface of the proto-neutron star (PNS) where it can
be more influential. This is important
 since (if fast modes are absent) calculations have shown so far that significant neutrino flavor conversions are not likely to occur close to the surface
of the PNS, in spite of the existence of flavor
instabilities therein
\cite{Duan:2014gfa,Chakraborty:2015tfa,Abbar:2015mca,Abbar:2015fwa, Dasgupta:2015iia}.
In fact, the unstable modes can turn stable before growing significantly due to the rapid variations of
the physical conditions during the neutrino propagation
\cite{Chakraborty:2015tfa, Hansen:2019iop}.
However, fast modes can change this picture by
occurring on short enough scales and therefore not being bothered
by the rapid variations of the physical conditions.

Although such ELN crossings were not considered in the
  bulb model calculations (neutrinos were assumed to be emitted isotropically
 from the surface of a single sharp neutrinosphere), they are speculated
 to exist in realistic SN models. This simply arises from
 the fact that
 $\nu_e$ and $\bar\nu_e$ decouple at different radii. Thus, one might simply
 expect that ELN crossings should be unavoidable
 in the SN environment \cite{Sawyer:2015dsa, Chakraborty:2016lct}.

Nevertheless,
one-dimensional SN simulations have not shown  such ELN crossings
during the early stages of  CCSNe within the
shock region \cite{Tamborra:2017ubu, Shalgar:2019kzy} (note,
however, that  they
might still exist in the pre-shock SN region \cite{Morinaga:2019wsv}).
 Although the angular distributions of $\bar\nu_e$'s are
 normally more peaked  than
 that of $\nu_e$'s in the forward direction, the large difference between
 the number densities of $\nu_e$ and $\bar{\nu}_e$
 hinders the occurrence of ELN crossings.
 However, this story can be changed in multidimensional (multi-D)
 SN models. Indeed, recent multi-D SN
 simulations have shown that the
 neutrino distributions can be highly asymmetric  in the
 presence  of lepton-emission self-sustained asymmetry (LESA)
 \cite{Tamborra:2014aua, Vartanyan:2019ssu, Sugiura:2019xuv, Walk:2019ier, Glas:2018vcs,
 OConnor:2018tuw, Janka:2016fox, Tamborra:2014hga, Nagakura:2019evv}.
 Such  asymmetric neutrino distributions   can significantly    help increasing the chance of
 the occurrence of ELN crossings by providing SN zones
 with smaller difference  between $n_{\bar\nu_e}$
 and $n_{\nu_e}$ \cite{Abbar:2018shq}.

 However, most of the state-of-the-art multi-D SN
simulations do not provide such detailed angular information of
 neutrinos  due to
the simplifications made in the neutrino transport.
SN simulations in which full  neutrino
 angular distributions are available have  become accessible
just  recently \cite{Sumiyoshi:2012za, Sumiyoshi:2014qua, Nagakura:2017mnp}.
The first investigation of the occurrence of ELN crossings in multi-D SN models was reported in Ref.~\cite{Abbar:2018shq}
in which the Boltzmann equations were solved for a number of
fixed SN matter profiles and
a number of ELN crossings and the associated fast modes were found in both 2D and 3D calculations of an
$11.2\mathrm{M}_{\odot}$ progenitor model.
Subsequently, the results of a self-consistent 2D
SN simulation (solving the Boltzmann  equations and hydrodynamics simultaneously) were reported
in which ELN crossings were not observed for the selected few spatial points \cite{Azari:2019jvr}.
Although both of the  computations were performed for an
$11.2\mathrm{M}_{\odot}$ progenitor model, the employed equations of state (EOS)
were different. This can have profound consequences for
the occurrence of ELN crossings, as will be discussed in this paper.

In this study, we explore
the neutrino angular distributions
obtained by solving
 the Boltzmann  equations for several fixed SN matter profiles
  which are representative snapshots
 taken from  multi-D SN simulations.
Following our previous study \cite{Abbar:2018shq},
we present a more detailed investigation of
the ELN crossings in an $11.2\mathrm{M}_{\odot}$ progenitor  model.
Furthermore, we present and analyse our results of the calculations of
 a $27\mathrm{M}_{\odot}$ progenitor models
 in which no ELN crossings within/above the neutrino decoupling
 region  were observed (Sec.~\ref{sec:ELNcrossings}).
We also discuss the possibility of the
existence of a class of ELN crossings in very deep regions
 well inside the PNS
 (Sec.~\ref{sec:PNS})\footnote{Refs.~\cite{Nagakura:2019sig, DelfanAzari:2019tez}
 appeared while this manuscript was in the last stages of its preparation.
In Ref.~\cite{DelfanAzari:2019tez}, the authors  report similar  occurrence of  deep ELN crossings
inside the PNS.}.
By performing linear stability  analysis (Sec.~\ref{sec:linear}),
we show that  fast modes  associated with the ELN crossings
can lead to flavor conversion rates as large as  several
e-folds per nanosecond (Sec.~\ref{sec:ELNcrossings}).

\section{Linear stability analysis.---} \label{sec:linear}

 The state of a neutrino traveling with momentum $\mathbf{p}$
  can be specified by its flavor density matrix $\varrho_{\mathbf{p}}(t, \bf{x})$ \cite{Sigl:1992fn}
  at each time $t$ and point $\textbf{x}$,
and its flavor evolution in the absence of the collision term is governed by the
Liouville-Von Neumann mean-field equation of motion
\cite{Sigl:1992fn,Strack:2005ux,Cardall:2007zw,Volpe:2013jgr, Vlasenko:2013fja
\begin{equation}
i (\partial_t + \mathbf{v} \cdot \nabla) \varrho_{\mathbf{p}} = [\rm{H}_{\mathbf{p}},\varrho_{\mathbf{p}}],
\label{Eq:EOM}
\end{equation}
with $ \mathbf{v} $ being the neutrino velocity
which in the spherical coordinate can be defined as  $ \mathbf{v} =
(\sin\theta_\nu \cos\phi_\nu, \sin\theta_\nu \sin\phi_\nu, \cos\theta_\nu)$
 for a neutrino with emission angles ($\theta_\nu$, $\phi_\nu$).
 Also, $\mathrm{H}_{\mathbf{p}} = \mathrm{H_{vac}} + \mathrm{H_{mat}} + \mathrm{H}_{\nu \nu, \mathbf{p}}$
 is the total  Hamiltonian
where, in the two-flavour scenario,
\begin{align}
\rm{H_{vac}} &\approx \frac{\eta\omega}{2}
\left[ {\begin{array}{cc}
1   &   0  \\
0  & -1 \\
\end{array} } \right] \\
 \rm{H_{mat}}  &= \frac{\lambda}{2}
\left[ {\begin{array}{cc}
1   &   0  \\
0  & -1 \\
\end{array} } \right]
\end{align}
with $\eta = +1  (-1)$ for the inverted (normal) mass ordering
 and $\lambda = \sqrt2 G_{\mathrm{F}} n_e$ with $n_e$ being the electron number density \cite{Wolfenstein:1977ue,Mikheev:1986gs},
where it is assumed that $\theta_{\rm{v}} \ll 1 $
and strong matter currents are absent.
Finally,
\begin{equation}
\begin{split}
 \mathrm{H}_{\nu \nu, \mathbf{p}} = \sqrt2 G_{\rm{F}}  \int  \frac{\mathrm{d}^3\mathbf{p'}}{(2\pi)^3}
  \big( 1- \mathbf{v} \cdot \mathbf{v'}  \big)
   \big(
 \varrho_{\mathbf{p'}}
 -
 \bar{\varrho}_{\mathbf{p'}} \big),
   \end{split}
 \end{equation}
 is the contribution from neutrino-neutrino refraction \cite{Fuller:1987aa,Notzold:1988kx,Pantaleone:1992xh}.
 Here, the flavor density matrices can be written as \cite{Banerjee:2011fj}
 \begin{align}
  \varrho = \frac{f_{\nu_e} + f_{\nu_x}}{2} \mathbb{1}
  + \frac{f_{\nu_e} - f_{\nu_x}}{2} \begin{bmatrix}
    s & S \\ S^* & -s \end{bmatrix},
\end{align}
where $S$ and $s$  are some complex and real quantities, respectively,
and  $f_{\nu}$ is the (prior to oscillation)  neutrino occupation numbers,
so that  the neutrino number  and flux densities are
 \begin{equation}
\begin{split}
n_\nu &= \int\!\frac{\mathrm{d}^3 p}{(2\pi)^3} f_\nu(\mathbf{p}),\\
\mathbf{j}_\nu &= \int\!\frac{\mathrm{d}^3 p}{(2\pi)^3}
f_\nu(\mathbf{p}) \mathbf{v},\\
\end{split}
 \end{equation}
respectively. Likewise, $\bar\nu$ quantities can be defined with respect to
$f_{\bar\nu_e}$ and $f_{\bar\nu_x}$.

 In the presence of fast modes, Eq.~\eqref{Eq:EOM} turns out to be approximately blind to
the neutrino energy (at least in the linear regime).
 Thus, one
can set $\omega=0$ and drop the energy dependency of $\varrho$
and integrate over the neutrino energy. It then proves to be useful
to define the neutrino electron lepton number (ELN) as  \cite{Izaguirre:2016gsx}
\begin{equation}
  G_\mathbf{v} =
  \sqrt2 G_{\mathrm{F}}
  \int_0^\infty \frac{E_\nu^2 \mathrm{d} E_\nu}{(2\pi)^3}
        [f_{\nu_e}(\mathbf{p})- f_{\bar\nu_e}(\mathbf{p})],
 \label{Eq:G}
\end{equation}
assuming $f_{\nu_x}(\mathbf{p}) = f_{\bar{\nu}_x}(\mathbf{p})$.

One can then linearise Eq.~\eqref{Eq:EOM} by keeping the
terms of order $\mathcal{O}(|S_\mathbf{v}|)$ or larger
\cite{Banerjee:2011fj, Vaananen:2013qja, Izaguirre:2016gsx},
\begin{equation}
 i (\partial_t + \mathbf{v} \cdot \bm{\nabla}) S_\mathbf{v}
 = (\epsilon_0 + \mathbf{v} \cdot
 \boldsymbol{\epsilon} ) S_\mathbf{v}
 - \int\!\mathrm{d}\Gamma_{\mathbf{v}'} (1 - \mathbf{v} \cdot \mathbf{v}')
 G_{\mathbf{v}'} S_{\mathbf{v}'},
 \label{Eq:linear}
 \end{equation}
 where $\mathrm{d}\Gamma_{\mathbf{v}'}$ is the differential solid angle in the
 direction of $\mathbf{v}'$,
 $\epsilon_0 = \lambda+\int\!\mathrm{d}\Gamma_{\mathbf{v}'} G_{\mathbf{v}'}$,
 and $\bm{\epsilon} = \int\!\mathrm{d}\Gamma_{\mathbf{v}'}
 G_{\mathbf{v}'} \mathbf{v}'$.

Because Eq.~\eqref{Eq:linear} is linear in $S_\mathbf{v}$,
it has the normal solutions of the form
\begin{equation}
S_\mathbf{v}  = Q_\mathbf{v}
 e^{ -i\Omega t  + i{\textbf{K}} \cdot {\textbf{x}}}.
 \label{Eq:exp}
 \end{equation}
 Collective neutrino oscillations can then lead to significant
 flavor conversions if there exist  solutions with complex
 $\Omega$ or/and $\mathbf{K}$.

Furthermore,
if the neutrino and antineutrino distributions
 possess axial symmetry,
 it is convenient  to  integrate
neutrino quantities over $\phi_\nu$ and
work with  $\phi_\nu$-integrated quantities:
\begin{equation}
f_\nu(\theta_\nu)=\int\frac{E_\nu^2\mathrm{d}E_\nu \mathrm{d}\phi_\nu}{(2\pi)^3} f_\nu(\mathbf{p})
\label{eq:f}
 \end{equation}
and
\begin{equation}
G(\theta_\nu)=\int_0^{2\pi}\mathrm{d}\phi_\nu G_\mathbf{v}
\label{eq:G}
\end{equation}

To study fast modes in the SN environment,
we then assumed a homogenous neutrino gas at each SN zone
for which we adopted the local angular distributions from the
SN model. This could be justified by the fact that fast
modes are expected to occur on scales much
smaller than that of the SN core. Thus,  one can safely
ignore the global geometry of the core and focus on the
local physics of the problem. However, this approximation is
valid as long as the neutrino instabilities
occur on scales well inside the considered SN zone.

It should be noted that a significant amount of backward traveling neutrinos
within the neutrino decoupling region
 leads to the existence of   spatial fast instabilities even without ELN crossings
\cite{Izaguirre:2016gsx,  Capozzi:2017gqd, Abbar:2017pkh}.
However, to have growing instabilities,
the existence of  temporal instability  with complex $\Omega$
 is necessary \cite{sturrock1958kinematics, Capozzi:2017gqd}.
Therefore,  one can focus on the temporal modes
having in mind that
the spatial instabilities are only important to determine the absolute or convective
nature of the growing instabilities \cite{Yi:2019hrp}.

\section{Occurence of ELN CROSSINGS IN MULTI-D SN MODELS}\label{sec:ELNcrossings}

\begin{figure*}[tbh!]
\centering
\begin{center}
\includegraphics*[width=1.\textwidth, trim= 10 30 10 10,clip]{Fig_3Dcolor_200ms.pdf}
\end{center}
\caption{
The Mollweide projection of $\alpha$
(upper panels) and the electron fraction $Y_e$ (lower panels) of the
3D  $11.2\mathrm{M}_{\odot}$
progenitor model calculations   in the  $t= 200$ ms snapshot  at $r=54.5$, 95.5 and 150 km, respectively.
Crosses indicate the ELN crossings.
Note that the color scales may
not be the same for different panels. At this time,
neutrinos decouple from matter at radii $\simeq 50-70$ km,
highly depending on their flavors and energies.}
\label{fig:3D11_200}
\end{figure*}

\begin{figure*}[tbh!]
\centering
\begin{center}
\includegraphics*[width=1.\textwidth, trim= 10 30 10 10,clip]{Fig_3Dcolor_Ye.pdf}
\end{center}
\caption{
The Mollweide projection of the  electron fraction $Y_e$ of the
   3D $11.2\mathrm{M}_{\odot}$
progenitor model  calculations in the $t= 200$ ms snapshot  at $r=11.3$, 17.2 and 23.6 km, respectively.
The pattern observed in Fig.~\ref{fig:3D11_200} first appears  very deep
inside the PNS.
Note that the color scales may
not be the same for different panels.
  }
\label{fig:3D11_Ye}
\end{figure*}

The neutrino distributions in this study were obtained by the calculations of  neutrino
transport for the fixed backgrounds (density, temperature and electron fraction)
 of 2D/3D matter profiles in the supernova core.
 The time evolution of the neutrino distributions was  followed
 until  a stationary state for the  neutrino quantities was reached.

The
Boltzmann equation  was solved directly in the full phase space
 to obtain the neutrino energy and
angle distributions.

The multi-angle multi-energy neutrino transport in 2D/3D space
was carried out by using  the $S_n$ method
and the time evolution was handled by the time implicit method.
Further details of our numerical method can be found in Refs.~\cite{Sumiyoshi:2012za, Sumiyoshi:2014qua}.

In the neutrino transport, three neutrino species, namely, $\nu_e$, $\bar{\nu}_e$ and $\nu_x$ were handled where
$\nu_x$ represents $\nu_\mu$ and $\nu_\tau$ and their anti-particles.
The microphysics used in the evaluation of the neutrino transport
is the same as the one in  Refs.~\cite{Sumiyoshi:2014qua, Abbar:2018shq} where
the neutrino reaction rates for emission, absorption, scattering and pair processes were taken
mostly from Ref.~\cite{Bruenn:1985en} and its extensions.
  In addition, the Lattimer \& Swesty EOS
 \cite{Lattimer:1991nc}  was used to
be consistent with the original supernova simulations.

We adopted multi-D supernova profiles
of the two progenitor models of an  $11.2\mathrm{M}_{\odot}$ (both 2D and 3D)
and a $27\mathrm{M}_{\odot}$ (only 3D) \cite{Sumiyoshi:2014qua, Takiwaki:2011db,Takiwaki:2013cqa}.
The  spatial resolutions of the Boltzmann calculations were set to be (256, 64, 1)
 and (256, 64, 32) for the numbers of spatial zones $(N_r, N_\Theta, N_\Phi)$
 in spherical coordinates
   for the 2D and 3D models, respectively, where
   a maximum radius of 2613 km
 from the original simulations was reached.
 For the neutrino   momentum space,
 a resolution of (14, 6, 12) was employed for $(N_{E_\nu}, N_{\theta_\nu}, N_{\phi_\nu})$
 in  both 2D and 3D calculations.
 In addition, to examine the  angular convergence of  the calculations,
 we performed a similar neutrino transport calculations
 with a higher resolution of $N_{\theta_\nu} = 36$
  for the snapshots of the 2D  $11.2\mathrm{M}_{\odot}$  progenitor model
 at $t=150$ and 200 ms.

\begin{figure*}[tbh!]
\centering
\begin{center}
\includegraphics*[width=1.\textwidth, trim= 10 30 10 10,clip]{Fig_3Dcolor_27M.pdf}
\end{center}
\caption{
The Mollweide projection of $\alpha$ of the
3D  $27\mathrm{M}_{\odot}$
progenitor model calculations in the  $t= 200$ ms snapshot  at $r=54.5$, 95.5 and 150 km, respectively.
Note that the color scales are
not  the same for different panels.
  }
\label{fig:3D27}
\end{figure*}

For the  $11.2\mathrm{M}_{\odot}$ progenitor model, three representative snapshots
 at $t=100$, 150 and 200 ms  from the original 2D and 3D simulations
 were selected
 for which  the Boltzmann  equations
were solved. As  mentioned in Ref.~\cite{Abbar:2018shq},
ELN crossings within/above the decoupling region were
 only observed in the snapshot at $t=200$ ms
in  the 2D calculations
 whereas
 they were found
in all of the time snapshots in the 3D calculations.
For instance, in the snapshot at $t=200$ ms,
the ELN crossings above the neutrinosphere appear first at
$r \simeq 46$ km within a relatively narrow region in
the southern hemisphere (Fig.~\ref{fig:3D11_200}).
As the radius increases, the crossings zone expands initially
 because the neutrino distributions get more
 peaked in the forward direction. However, it
 gets narrower again at larger radii and disappears at $r \sim 200$ km.
As will be discussed later, this disappearance of the ELN crossings could be an
 artificial result of the limited angular resolution of the neutrino transport calculations.

 As pointed out in Ref.~\cite{Abbar:2018shq}, the ELN crossings zones appear to be correlated with
the zones where the $\nu_e$-$\bar{\nu}_e$ asymmetry parameter,
\begin{equation*}
\alpha = \frac{n_{\bar\nu_e}}{n_{\nu_e}} ,
\end{equation*}
is close to 1.
This, indeed, is not coincidental and  can be
 understood as follows.
 In the SN environment,
 the angular distribution of $\bar{\nu}_e$ is normally more peaked in the forward direction
than that of $\nu_e$ because they decouple at smaller radii.
One then might be tempted to assume that

the occurrence of ELN crossings  in  CCSNe is inevitable.
However,
if the $\nu_e$-$\bar{\nu}_e$ asymmetry is large,
the occurrence of ELN crossings could be remarkably  suppressed.
In fact, a larger $\nu_e$-$\bar{\nu}_e$ asymmetry means a more
separated $\nu_e$ and $\bar{\nu}_e$ angular distributions
and therefore, less chance for the occurrence of ELN crossings.
Thus, ELN crossings are more
likely to occur within the zones with small $\nu_e$-$\bar{\nu}_e$ asymmetries
($\alpha$ close to 1).

\begin{figure*}[tbh!]
\centering
\begin{center}
\includegraphics*[width=1.\textwidth,trim = 10 10 10 10,clip]{Fig_Ylm.pdf}
\end{center}
\caption{
Radial evolution of the multipole components of
$\alpha$ Eq.~\eqref{Eq:Ylm} for the $t=200$ ms snapshots.
 In  2D and 3D $11.2\mathrm{M}_{\odot}$ progenitor models, strong dipole and quadrupole components in $\alpha$ exist; while  in the  $27\mathrm{M}_{\odot}$ one, these  are only found inside the PNS.
  }
\label{fig:Ylm}
\end{figure*}

\begin{figure*}[tbh!]
\centering
\begin{center}
\includegraphics*[width=1.\textwidth, trim= 5 10 10 10,clip]{Fig_2D_G_36.pdf}
\end{center}
\caption{Angular distributions of neutrinos $f_\nu$ (left),
ELN (middle), as  functions of the angular variable $\theta_{\nu}$,
and the corresponding eigenvalues $\Omega_\mathrm{i}$ (right panels) Eqs.(\ref{Eq:linear}-
\ref{Eq:exp}), as functions of the real wave number $K$.  The results correspond to fast neutrino oscillation
 modes propagating in the radial direction,
at the four different radii $r= 63.7, 93.2, 107.7$ and $135$ km.
The angular distributions are extracted from the spatial point
with $\cos\Theta = 0.99$ in the 2D model in the $t=200$ ms snapshot.
  The zoomed-up subplot indicates the shallow crossing at  $\cos\theta_\nu \simeq -0.56$
  in the calculation with $N_{\theta_{\nu}} = 36$.
  A similar shallow crossing exists in the calculations with $N_{\theta_{\nu}} = 6$ but
  the  low angular resolution does not allow for a definite recognition of it.
  At this time, the neutrinospheres of different flavors are located
 at radius $\sim 50-70$ km (see Fig.~4 of Ref.~\cite{Abbar:2018shq}).
  }
\label{fig:G}
\end{figure*}

\begin{figure}[tbh!]
\centering
\begin{center}
\includegraphics*[width=.4\textwidth, trim= 5 15 10 10,clip]{Fig_Phi.pdf}
\end{center}
\caption{
Neutrino angular distributions
as functions of $\phi_\nu$
for $\nu_e$ (solid) and $\bar\nu_e$ (dashed) and at
different $\cos\theta_\nu$ values (Eq.~(\ref{Eq:ang})). The results are
obtained from the 2D calculations of the $t= 200$ ms $11.2\mathrm{M}_{\odot}$
progenitor model,
for the spatial zone at $r=61.8$ km and $\cos\Theta=0.98$.
The  asymmetry in $\phi_\nu$ turns out to be  small for
the spatial zones with ELN crossings.}
\label{fig:phi}
\end{figure}

The pattern in $\alpha$ (and the neutrino distributions) is obviously  (anti)correlated  with
a similar pattern in the electron fraction $Y_e$ (Fig.~\ref{fig:3D11_200}).
 Fig.~\ref{fig:3D11_Ye} shows that it first appears very deep inside the PNS.
 As pointed out in Ref.~\cite{Tamborra:2014aua}, in the case of LESA
convectional flows in the PNS could generate asymmetries in the neutrino fluxes.
 In our 3D models, the pattern in $Y_e$
 shows up first at $r \sim 11$ km and becomes more distinct at larger radii.

 The  pattern  in $Y_e$ can then be associated with
a similar pattern in $\alpha$ because more (less) $\nu_e$'s ($\bar{\nu}_e$'s)
 are emitted  where $Y_e$ is larger.

 For the  $27\mathrm{M}_{\odot}$ progenitor model, three representative snapshots
 at $t=150$, 200 and 250 ms  from the original 3D simulations were selected.

 In contrast to the case of the  $11.2\mathrm{M}_{\odot}$ progenitor model,
 no ELN crossings above the neutrinosphere  were found in
 this case.
 A few representative
 Mollweide projections of  $\alpha$  at different radii
 (corresponding to the ones in Fig.~\ref{fig:3D11_200} for the
  $11.2\mathrm{M}_{\odot}$ progenitor model)
  are plotted in Fig.~\ref{fig:3D27}.
 Although it shows some slight  spatial variations,
the value of $\alpha$ within/above the neutrino decoupling region
is always  much smaller than one in this model which in turn
provides a little chance for the occurrence of ELN crossings.

 During the early stages of a CCSN, (the average of) $\alpha$ tends to be
 relatively small. This
  can seriously  hinder
the occurrence of ELN crossings in the SN environment.
In particular, no ELN crossings within the shock region have been found
 in 1D SN models so far \cite{Tamborra:2017ubu, Shalgar:2019kzy}
 except in the pre-shock SN region \cite{Morinaga:2019wsv}.
 However, the situation can be different  in multi-D SN models
 where the neutrino distributions can be spatially asymmetric due to multi-D hydrodynamics.
  In fact, although the average value of $\alpha$
 is thought to be similar in 1D and multi-D SN models,
 the existence of multipole structures in $\alpha$ in the latter can
 allow for regions with large $\alpha$'s which increases
 the chance for the occurrence of ELN crossings. Fig.~\ref{fig:Ylm} presents
 the $l=0,1,2$
  multipole components of the spherical harmonics decomposition
 of  $\alpha$, defined as
 \begin{equation}
 \alpha_l = \bigg(   \sum_{m=-l}^l \left| \int \frac{\mathrm{d}\Omega}{\sqrt{4\pi}} \ Y^*_{lm}(\Theta,\Phi) \alpha(\Theta,\Phi) \right|^2     \bigg)^{1/2}.
 \label{Eq:Ylm}
 \end{equation}
 As one can see, both the
  2D and the 3D $11.2\mathrm{M}_{\odot}$ progenitor models show
strong dipole and quadrupole components in $\alpha$. On the other hand
  these are weak in the $27\mathrm{M}_{\odot}$ progenitor model,
except for a small region inside the PNS where ELN crossings exist
  (see Sec.~\ref{sec:PNS}).
 This  provides  an explanation  on why there is no ELN crossings within/above the neutrino decoupling
  region for this model.
 Furthermore, while in the 2D calculations the dipole component is dominant in all of the snapshots,
 it is the quadrupole  term that is stronger, at least at smaller radii, in all
 of the 3D snapshots.

 \

A few representative $\phi_\nu$-integrated angular distributions of  $\nu_e$, $\bar{\nu}_e$
and the corresponding ELN (Eqs.~\eqref{eq:f} and \eqref{eq:G})
 are presented in Fig.~\ref{fig:G}
for the  $t=200$ ms  2D $11.2\mathrm{M}_{\odot}$ progenitor model.
The angular distributions with $N_{\theta_{\nu}} = 6$ and  $N_{\theta_{\nu}} = 36$ are in
a relatively good agreement. The only exception is
at larger radii, where
the calculations with smaller number of angle bins
 fails (expectedly) to capture the angular
structures at small emission angles.
In fact,
as the radius gets larger,
the ELN crossings get narrower
because the neutrino angular distributions
 become more forward peaked. Therefore, higher angular resolution
is needed to capture them.
This implies that the disappearance of the ELN crossings at larger radii
could be an artefact of the  low angular resolution of the
neutrino  transport calculations.
The average neutrino quantities are in much better
 agreement between the two calculations with different resolutions,
  the values of $n_\nu$ and $\alpha$ differing
at most by $2-3\ \%$.

 Moreover, we noticed that
 there are cases where the ELN distributions exhibit
 two crossings. This phenomenon  seemingly
 occurs at larger radii where the neutrino
 angular distributions are highly peaked
 in the forward direction. As  shown in
 Fig.~\ref{fig:G}, at r = 135 km there is a first shallow crossing at negative values of $\cos\theta_\nu$ and a second one at  $\cos\theta_\nu \simeq 1$.

Besides the analysis of the neutrino angular distributions,
 we  performed a linear stability analysis and solved Eq.~\eqref{Eq:linear}
to find the growth rates of the unstable modes
(Eq.~\eqref{Eq:exp})
at each spatial zone.
 The calculations assume axial symmetry for
the neutrino gas.
 In fact,
the neutrino angular distributions
\begin{equation}\label{Eq:ang}
f_\nu(\theta_\nu,\phi_\nu)=\int\frac{E_\nu^2\mathrm{d}E_\nu} {(2\pi)^3} f_\nu(\mathbf{p}),
\end{equation}
are highly symmetric in $\phi_\nu$
 at all of the spatial zones for  which  ELN crossings were observed, as illustrated by
Fig.~\ref{fig:phi}.
Overall, the deviation from axial symmetry
in the zones with ELN crossings
 turns out to be smaller than a few percent
in both 2D and 3D models.
However,  one should keep in mind that this
can be changed for rotating models where the angular
distributions can be quite asymmetric in $\phi_\nu$ \cite{Harada:2018ubo}.
As mentioned earlier,
the temporal instability
is necessary to have
 growing perturbations.
 The complex eigenvalues associated with the temporal
fast modes are plotted in Fig.~\ref{fig:G}.
The corresponding exponential growth rates
can be as large as several e-folds per nanoseconds and
tend to be larger for the wider ELN crossings.

Our results suggest that
even the more accessible calculations with
low angular resolution might provide
 a quite reliable estimate of the presence of ELN crossings and correspondingly fast modes.
 This is corroborated by the fact that
 we did not find false ELN crossings
 in the calculations with lower angular resolution and the associated instability growth
 rates provided a reasonable estimate of the results
based on higher angular resolution. Obviously, one cannot exclude that
in a dynamic self-consistent CCSN simulation,
 the prompt convection might be more suppressed
 if the angular resolution is too low \cite{Nagakura:2017mnp}.

\begin{figure*}[tbh!]
\centering
\begin{center}
\includegraphics*[width=1.\textwidth, trim= 10 30 10 10,clip]{Fig_PNS_crossings_1.pdf}
\end{center}
\caption{
The Mollweide projection of $\alpha$
of the 3D calculations  at $r=23.6$ km  in different time snapshots.
The few crosses indicate ELN crossings inside the PNS. Note
that the crossings inside the PNS are not as abundant as
the ones within the decoupling region and they only exist within the zones where $\alpha$ is extremely close to one.
  }
\label{fig:PNS}
\end{figure*}

\begin{figure*}[tbh!]
\centering
\begin{center}
\includegraphics*[width=1.\textwidth, trim= 10 15 10 10,clip]{Fig_PNS_crossings_f.pdf}
\end{center}
\caption{ Angular distributions, as functions of $\theta_{\nu}$,
for $\nu_e$, $\bar\nu_e$ and $\nu_x$
(left), ELN (middle) and the
corresponding eigenvalues $\Omega_\mathrm{i}$  (right panel),
as a function of the real wave number $K$.  Interpolated values are also shown (dashed lines).
 The results are for the fast
 modes propagating in the radial direction,
 at the spatial zone with
$r=26.4$, $ \cos\Theta = 0.40$ and $\Phi = 0.72\pi$
deep inside the PNS, for which an ELN crossing exists.
The calculations are for 3D $11.2\mathrm{M}_{\odot}$
progenitor model and the $t= 200$ ms snapshot.
Note that the neutrino angular distributions are very close to each
other and highly  non-degenerate, with $n_{\bar\nu_e}/n_{\nu_e} = 1.001$
and $ n_{\nu_x}/ n_{\nu_e} = 0.972$. }
\label{fig:PNS-fnu}
\end{figure*}

\begin{figure*}[tbh!]
\centering
\begin{center}
\includegraphics*[width=1.\textwidth, trim= 10 30 10 10,clip]{Fig_PNS_crossings.pdf}
\end{center}
\caption{
The Mollweide projection of the electron fraction $Y_e$,
the electron neutrino chemical potential $\mu_{\nu_e}$
and the temperature $T$
of the 3D $11.2\mathrm{M}_{\odot}$ progenitor model calculations  at $r=23.6$ km
in the $t=150$ ms snapshot corresponding to the very left panel
in Fig.~\ref{fig:PNS}.
$\mu_{\nu_e}$
and $T$ are both in MeV.
 }
\label{fig:mu_nu}
\end{figure*}

\begin{figure}[tbh!]
\centering
\begin{center}
\includegraphics*[width=.48\textwidth, trim= 5 0 10 5,clip]{Fig_PNS_crossings_f2.pdf}
\end{center}
\caption{
The neutrino energy distributions
 at the spatial point corresponding to the one in Fig.~\ref{fig:PNS-fnu}.}
\label{fig:eng}
\end{figure}

\section{ELN crossings  inside the PNS } \label{sec:PNS}
\noindent
 Apart from the ELN crossings  occurring
 within/above the decoupling region of neutrinos, there  is another
 class of ELN crossings that can appear in deep regions inside
 the PNS. In our calculations, such ELN crossings
exist  at radii in the range $20-28$ km.

  For the 2D $11.2\mathrm{M}_{\odot}$ progenitor model, only two ELN crossings at $\sim 25$ km
 exist in the snapshot at $t=100$ ms, while
 they appear in all 3D snapshots.
 As for the  $27\mathrm{M}_{\odot}$  progenitor
 model, the ELN deep crossings  are present
  in the $t=150$ and 200 ms snapshots  (Fig.~\ref{fig:PNS}).
Note that there exist no ELN crossings within the decoupling region
 for this progenitor model.

 Since the neutrino angular distributions are
 significantly  isotropic inside the PNS, the deep crossings can
 only  exist if $\alpha$ is extremely  close to one, i.e.,
 the $\nu_e$ and $\bar{\nu}_e$ angular distributions are very close to each other
 (Fig.~\ref{fig:PNS-fnu}). Note that if $\alpha$ has values both below and
 above 1, it is very likely to have this sort of ELN crossings
 provided that the  spatial  resolution is high enough.

 With this in mind, it is safe to say that the key question here is whether such hot spots (as in Fig.~\ref{fig:PNS}) do actually
 show up in  more realistic SN simulations.

 It should be also noted that such ELN crossings do not necessarily occur
 at $\cos\theta_\nu \simeq 1$.

 The reason for having  SN zones for which $\alpha \simeq 1$ can be
 understood as follows. As shown in
 Fig.~\ref{fig:mu_nu},
 the neutrino gas is very non-degenerate ($\mu_{\nu}/T \simeq 0$) at the ELN crossing points
 inside the PNS.
   In fact, the chemical potentials of
$\nu_e$ and $\bar{\nu}_e$ become similar at these points, i.e., $\mu_{\nu_e}
\simeq \mu_{\bar{\nu}_e} \simeq 0$ ($\mu_{\nu_e} = \mu_{\mathrm{p}} - \mu_{\mathrm{n}} + \mu_{e}$).
This happens because  $\mu_{\mathrm{p}} - \mu_{\mathrm{n}} $
can  almost cancel $\mu_{e}$ at these SN zones where $Y_e$ is minimum\footnote{
We would like to thank Shoichi Yamada for valuable conversation about this
possiblity.}.
 Thus, a  correlation exists between $Y_e$ and $\mu_{\nu_e}$.
In addition, the temperature has its maximum value at these points.
As mentioned in Ref.~\cite{Janka:2016fox},
this (anti)correlation between $Y_e$ and $T$ can be explained by noting that
in spite of the asymmetric neutrino distributions,
the density and pressure tend to maintain their spherical symmetry.
This simply arises from the sphericality of the gravitational potential
which mainly governs the variations of these quantities. Therefore,
one should expect higher temperatures within the zones where
$Y_e$ is lower to keep the combination of  thermal  and
 electron degeneracy pressures constant ($p = p(\rho,T,Y_e)$).

 Although the existence  of the  deep ELN crossings
 surprisingly allows for the occurrence  of fast modes
 inside the PNS,
there are two essential points that should be kept in mind.
 Firstly,  one
 can indeed observe  large flavor conversion rates
  for all the SN  zones where
 $\alpha$ is extremely close to one,  even for those the
 ELN crossings are absent.
 For example,  in an isotropic homogeneous
 monoenergetic neutrino gas initially consisting  of  $\nu_e$ and $\bar{\nu}_e$ (bipolar model),
 for $|\alpha-1| \ll 1$, the exponential growth rate  governing flavor evolution
 is \cite{Duan:2007fw, mangano:2014zda}
  \begin{equation}
\Omega_\mathrm{i} \approx  \sqrt{2\mu\omega},
 \end{equation}
for
 \begin{equation}
 \frac{2}{(1+\sqrt{\alpha})^2} < \mu/\omega < \frac{2}{(1-\sqrt{\alpha})^2},
 \end{equation}

 for
 $\eta=1$\footnote{In the case of slow modes, the  exponential growth rate is
  $\sim  \sqrt {\omega \mu}$. However, unless $\alpha$ is extremely close to one,
  $\mu \sim \omega$
  which means $\Omega_\mathrm{i} \sim \omega$. One can
  observe a similar instability for $\eta=-1$ by breaking the axial/spatial symmetry.}.
 This implies that  even in the bipolar model (in the absence of ELN crossings),
 there can exist unstable
 slow modes  on scales  $\propto G_{\mathrm{F}}^{-1/2} n_\nu ^{-1/2}$
   at spatial zones where $\alpha \simeq 1$.

  Such scales  can be $\lesssim 1$ m
  (compare with fast modes occuring  on scales $\sim G_{\mathrm{F}}^{-1} n_\nu ^{-1} \lesssim  10$ cm)
  for these SN zones inside the PNS.
This is     much shorter than the collisional scales ($\lesssim  \mathcal O  (10^2)$ m) therein.
Therefore, having large neutrino flavor conversion rates
for non-degenerate SN zones is not unique to fast modes.

 Secondly, the neutrino distributions are significantly similar for all neutrino species within these zones
meaning that not only $n_{\nu_e} \simeq n_{\bar{\nu}_e} \simeq n_{\nu_x}$ (Fig.~\ref{fig:PNS-fnu}),
but also the energy distributions
 \begin{equation}
\mathcal{F}_\nu(E_\nu)=\int \mathrm{d}\Gamma_{\mathbf{v}} E_\nu^2  f_\nu(\mathbf{p}),
 \end{equation}
 are very similar for
different neutrino species  (Fig.~\ref{fig:eng}).
Consequently,   neutrino oscillations should not have significant impact
on the flavor content of the neutrino gas.

\section{Discussion and conclusion}

To assess the possibility of the occurrence of ELN crossings and the
associated fast neutrino flavor conversion
modes in the SN environment,
we have studied neutrino angular distributions obtained by
solving the Boltzmann equations for  fixed  matter profiles of some
representative  snapshots   during the post-bounce phase of CCSNe
in 2D and 3D  for  an $11.2\mathrm{M}_{\odot}$ and a $27\mathrm{M}_{\odot}$ (only 3D)
progenitor  models.

For the $11.2\mathrm{M}_{\odot}$  progenitor model,
ELN crossings were observed in both 2D and 3D models.
It turns out that
they tend to occur within the SN zones where the
 $\nu_e-\bar{\nu}_e$ asymmetry parameter, $\alpha$,
 is very close to one,  as in Ref.\cite{Abbar:2018shq}.
 Unlike the 1D SN modes in which $\alpha$ is
 thought to be globally small during the early stages
 of a CCSN,
  zones with large $\alpha$'s could exist in multi-D SN models in the presence of
 a spatially asymmetric neutrino distributions due to multi-D hydrodynamics.
 The reason is that in spite of having similar average values of
 $\alpha$ to 1D models, multi-D SN models can feature
 larger $\alpha$'s in  zones where $\alpha$ is significantly larger  than its averaged value.

 The pattern in $\alpha$ is  (anti)correlated with a similar pattern
 in $Y_e$ (and $T$) that appears very deep inside the PNS and is thought to
 be caused by the convectional flows therein.
 This is closely related to LESA
  \cite{Tamborra:2014aua, Vartanyan:2019ssu, Sugiura:2019xuv, Walk:2019ier, Glas:2018vcs,
 OConnor:2018tuw, Janka:2016fox, Tamborra:2014hga, Nagakura:2019evv}
 in which strong dipole  structure exists in the neutrino distributions
 which can be preceded by strong higher multipole structures at earlier times.
 Our study suggests that if the multipole structures (either  dipole at
 later times or higher multipoles at earlier times) are strong enough,
 they can lead to the existence of zones with significantly large $\alpha$'s
 and a good chance for the occurrence of ELN crossings.
  However,     at this point in time it is not very clear how the fast
 modes propagate from these zones to other regions.

 Note also that such asymmetric structures
 in $\alpha$ do not necessarily need to come from LESA.
 In general any significant asymmetry in the neutrino distributions,
 self-sustained with relatively constant direction (as in LESA) or not,  can
 increase the chance for the occurrence of ELN crossings in the SN
 environment.

 To see how increasing the neutrino angular resolution
can affect the occurrence of ELN crossings, we
performed  calculations with higher
angular resolution with $N_{\theta_\nu}=36$ in 2D
for our $11.2\mathrm{M}_{\odot}$  progenitor model.
It turns out that not only the calculations with
lower resolution do not produce any false
ELN crossings, but also they can provide a relatively
good estimate of the flavor conversion  rates in a more
reliable calculation with higher angular resolution.
This is intriguing since the calculations with higher
resolution are barely accessible in multi-D SN models.

 For the $27\mathrm{M}_{\odot}$  progenitor model,
 no ELN crossings within the neutrino decoupling region were found.
 Although there exists a  quadrupole structure in $\alpha$,
 it is still too weak to result in the existence of regions
 with  large $\alpha$'s. This comes as no surprise
 since the evolution of the multipole structures in neutrino
 emission   depends sensitively on a number factors such as
 the mass of the progenitor,
 the treatment of  neutrino transport and even the employed EOS
 (see, e.g., Fig.~5 in Ref.~\cite{Janka:2016fox}).

  We would like to point out that, no
 ELN crossings were found
  in a  recent self-consistent 2D SN simulations
 using the Furusawa’s EOS \cite{Furusawa:2011wh}, as opposed to the Lattimer \& Swesty EOS
 used in our calculations.  This result is not inconsistent with
 the present study nor with Ref.~\cite{Abbar:2018shq}, since
 convective  motion, crucial to the existence
 and evolution of neutrino distribution asymmetries,
 are  much weaker in the simulations with Furusawa’s EOS \cite{Nagakura:2017mnp}.
 Therefore, one may expect to observe weaker  distribution
 asymmetries in the simulation with the latter EOS.

Apart from the ELN crossings appearing within/above the neutrino decoupling region,
 our results show that there exist a class of ELN crossings which can occur
in  deep regions, well inside the PNS.
We observed this sort of ELN crossings in both of
the  progenitor models we investigated.
 Because the neutrino angular distributions
are extremely isotropic below the neutrinosphere, these ELN crossings can exist only
within the zones where
$\alpha \simeq 1$.

Needless to say, if ELN crossings and their corresponding  fast modes
 actually occur in the SN environment and lead to significant flavor conversion, they can have
profound consequences
for the SN physics.

\section*{Acknowledgments}

S.A. and M.C.V.  acknowledge support from ”Physique
fondamentale et ondes gravitationelles” (PhysFOG) of the Observatoire de Paris.
H.D. is supported by US DOE NP grant No. DE-SC0017803 at UNM.
This work is supported by
Grant-in-Aid for Scientific Research
(26104006, 15K05093, 19K03837)
and
Grant-in-Aid for Scientific Research on Innovative areas
"Gravitational wave physics and astronomy:Genesis"
(17H06357, 17H06365)
from the Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan
For providing high performance computing resources,
Computing Research Center, KEK,
JLDG on SINET4 of NII,
Research Center for Nuclear Physics, Osaka University,
Yukawa Institute of Theoretical Physics, Kyoto University,
Nagoya University,
Hokkaido University,
and
Information Technology Center, University of Tokyo are acknowledged.
This work was partly supported by
research programs at K-computer of the RIKEN AICS,
HPCI Strategic Program of Japanese MEXT,
"Priority Issue on Post-K computer" (Elucidation of the Fundamental Laws and Evolution of the Universe)
and
Joint Institute for Computational Fundamental Sciences (JICFus).

\bibliographystyle{elsarticle-num