\section{Introduction}
The question of knowing under what conditions  quasi-metric  spaces are Baire spaces, has been treated by several authors (the definition of quasi-metric space  will be reminded later). There exist some positive results in this direction which can be found for example in  \cite{Kl} and \cite{FG} (see also \cite{Cob}). However, it turns out that in the case of biBanach asymmetric normed spaces, the result is positive if and only if  the topology itself induces a topology of a norm. Indeed, the main result of this note is precisely to show  that if an asymmetric normed space is not isomorphic to its associated normed space, then it is not a Baire space (Theorem \ref{Baire}). Consequently, in asymmetric normed spaces, the conditions given in \cite[Theorem ~2.1]{FG} implicitly implies that these spaces are isomorphic to their associated normed spaces (in fact, we think that the condition of quasi-regularity in \cite{FG} implies that the index of symmetry introduced in \cite{Ba-Fl}, is strictly bigger than zero and so the asymmetric norm is equivalent to the associated norm). We do not know if the result of this paper extends to the general case of quasi-metric spaces (see Open quastion 1). However, we prove using our main result, that the semi-Lipschitz free space  over any quasi-metric space  is a Baire space if and only if the quasi-metric of the space  is equivalent to  its symmetrized metric. The concept of semi-Lipschitz free space over a quasi-metric space was introduced recently by A. Daniilidis, J. M. Sepulcre and F. Venegas in \cite{D-all}, in  the same way as the classical Lipschitz free space of Godefroy-Kalton in \cite{GK} (see also Section \ref{Index-Baire}). It follows easily from the results of this note that if the semi-Lipschitz-free  over a bicomplete quasi-metric space is a Baire space then it will be the same for the quasi-metric space. The converse of this fact remains an open question (see Open question 2, at the end of the paper). The tool used to establish our results is based on the use of the index of symmetry introduced recently in \cite{Ba-Fl}.
\section{Notation and Definitions}
In this section, we will recall the classical notions that will be used subsequently. For literature on quasi-hemi-metric, asymmetric normed spaces, semi-Lipschitz functions  and their applications, we refer to \cite{Cob}, where the developement of Functional Analysis on these spaces is detailed. We refer also to the various studies developed in \cite{GRS, GRS1}, \cite{CobM}, \cite{AFG1, AFG2}, \cite{RS} and \cite{DJV}. Quasi-metric spaces and asymmetric norms have recently attracted a lot of interest in modern mathematics, they arise naturally when considering non-reversible Finsler manifolds \cite{CJ, DJV, PT}. For an introduction and recent study of asymmetric free spaces (or semi-Lipschitz free spaces), we refer to the recent paper \cite{D-all}.
\begin{definition} A quasi-hemi-metric space is a pair $(X, d)$, where $X \neq \emptyset$  and
$$d : X \times X \to  [0, +\infty)$$
is a function, called quasi-hemi-metric (or quasi-hemi-distance), satisfying:

$(i)$ $\forall x, y, z \in X: d(x, y) \leq  d(x, z) + d(z, y)$ (triangular inequality);

$(ii)$ $\forall x, y \in X: x = y  \iff  [d(x, y) = 0 \textnormal{ and } d(y,x)=0]$.

A quasi-metric space is a pair $(X, d)$ satisfying $(i)$ and the following condition $(ii')$: $\forall x, y \in X: x = y  \iff  d(x, y) = 0$.

\end{definition}
We define respectively the open and closed balls  centered at $x$ with radious $r\geq 0$ as follows
$$B(x,r):=\lbrace y\in X: d(x,y)<r\rbrace.$$
$$B[x,r]:=\lbrace y\in X: d(x,y)\leq r\rbrace.$$
In a similar way, we define asymmetric norms on real linear spaces as follows.
\begin{definition} Let $X$ be a real linear space. We say that $\|\cdot|: X \to \R^+$ is an asymmetric norm on $X$
if  the following properties hold.
\begin{enumerate}[$(i)$]
    \item For every $\lambda \geq 0$ and every $x \in X$, $\|\lambda x|= \lambda \|x|$.
    \item For every $x, y \in X$, $\|x+y|\leq \|x|+\|y|$.
    \item For every $x \in X:  \|x|=\|-x|=0 \iff x=0$.
\end{enumerate}
\end{definition}
An asymmetric norm on a real linear space $(X,\|\cdot|)$ naturally induced a quasi-hemi-metric as follows : $$d(x,y)=\|y-x|, \hspace{3mm}  \forall x, y \in X.$$
Notice that a sequence $(x_n)$ converges to $x$ if $d(x,x_n)=\|x_n-x|\to 0$ but in general we dont have that $\|x-x_n|\to 0$.

Every quasi-hemi-metric space $(X, d)$ has an associated metric space $(X,d_s)$ where $d_s(x,y):=\max \lbrace d(x,y),d(y,x) \rbrace$ for every $x,y \in X$. An asymmetric normed space $(X,\|\cdot|)$ has also an associated normed space $(X,\|\cdot\|_s)$, where $\|x\|_s:=\max(\|x|,\|-x|)$ for all $x\in X$.
\begin{definition} A quasi-hemi-metric space $(X, d)$ is said to be bicomplete if its associated metric space is complete. An asymmetric normed space is said to be a biBanach space if its associated normed space is a Banach space.
\end{definition}
\vskip5mm
\paragraph{\bf The space of continuous linear functionals.} Let $(X,\|\cdot|)$ be an asymmetric normed spaces  and $(\R,\|\cdot|_\R)$ be the asymmetric normed real line equipped with the asymmetric norm $\|t|_{\R}=\max\lbrace t,0\rbrace$. A linear functional $p: (X,\|\cdot|)\to (\R,\|\cdot|_{\R})$ is called bounded if there exists $C\geq 0$ such that $$p(x)\leq C\|x|, \hspace{2mm} \forall x\in X.$$
In this case, we denote $\|p|_{\flat}:=\sup_{\|x|\leq 1}p(x)$. We denote $X^{\flat}$ the convex cone of all bounded functionals  from $(X,\|\cdot|_X)$ into $ (\R,\|\cdot|_{\R})$.
It is known (see \cite[Proposition 2.1.2]{Cob}) that a linear functional $p$ is bounded if and only if it is continuous, which in turn is equivalent to being continuous at $0$. It is easy to see that $p\in X^{\flat}$, if and only if $p: (X,\|\cdot|)\to (\R,|\cdot|)$ is upper semicontinuous.   The constant $\|p|_{\flat}$ can be calculated also by the formula (see \cite[Proposition 2.1.3]{Cob})
$$\|p|_{\flat}=\sup_{\|x|= 1} \|p(x)|_{\R}.$$
 The topological dual of the associated normed space $X_s:=(X,\|\cdot\|_s)$ of $X$  is denoted $X^*$ and is equipped with the usual dual norm denoted $\|p\|_{*}=\sup_{\|x\|_s\leq 1} \langle p, x\rangle$, for all $p\in X^*$.  Note, from \cite[Theorem 2.2.2]{Cob}, that the convex cone $X^{\flat}$ is not trivial, that is, $X^{\flat}\neq \lbrace 0 \rbrace$ whenever $X\neq\lbrace 0 \rbrace$. We always have that
$$X^{\flat} \subset X^* \textnormal{ and }  \|p\|_{*}\leq \|p|_{\flat}, \textnormal{ for all } p\in X^{\flat}.$$
Thus, $X^{\flat}$ is a convex cone included in $(X^*,\|\cdot\|_{*})$ but is not a vector space in general. We say that $(X,\|\cdot|_X)$ and $(Y,\|\cdot|_Y)$ are isomorphic and we use the notation $(X,\|\cdot|_X) \simeq(Y,\|\cdot|_Y)$, if there exists a bijective linear operator $T: (X,\|\cdot|_X)\to (Y,\|\cdot|_Y)$ such that $T$ and $T^{-1}$ are bounded.

\paragraph{\bf The space of semi-Lipschitz functions.} Let  $(X, d)$ be a quasi-hemi-metric space equipped with a distinguished point $x_0$ (called a base point of $X$) and $(\R,\|\cdot|_\R)$ be the asymmetric normed real line equipped with the asymetric norm $\|t|_{\R}=\max\lbrace t,0\rbrace$. A function $f : (X, d)\to (\R,\|\cdot|_{\R})$ is said to be semi-Lipschitz if, there exists a positive real number  $C_f\geq 0$ such that
$$\|f(x)-f(y)|_{\R}=\max \lbrace f(x)-f(y), 0\rbrace \leq C_f d(x,y), \hspace{3mm} \forall x, y\in X.$$
In general $-f$ is not semi-Lipschitz function, when $f$ is.  The convex cone of all semi-Lipschitz functions that vanish at $x_0$ will be denoted by $\textnormal{SLip}_0(X)$.
We denote $$\|f|_L:=\sup_{d(x,y)>0 } \frac{\|f(x)-f(y)|_{\R}}{d(x,y)}; \hspace{3mm} \forall f\in \textnormal{SLip}_0(X),$$

\section{The main result} \label{Index-Baire}

 We define the index of symmetry of a  quasi-hemi-metric space $(X, d)$ in the same way as that introduced in \cite{Ba-Fl}  for asymmetric normed spaces:
$$c(X):=\inf_{d(y,x)>0} \frac{d(x,y)}{d(y,x)} \in [0,1].$$
The conjugate of $c(X)$ is defined by $\overline{c}(X):=\sup_{d(y,x)>0} \frac{d(x,y)}{d(y,x)}$.
In the case of asymmetric normed space, using the quasi-metric $d(x,y)=\|y-x|$, we see that $c(X)=\inf_{\|x|=1} \|-x|$.

This index measures the degree of symmetry of the quasi-hemi-metric $d$, it was firstly introduced and studied in the assymetric normed space in \cite{Ba-Fl}, where a topological classification of asymmetric normed spaces has been given according to this index. Similarly to the framework of asymmetric normes spaces, we have the following proposition.

\begin{proposition} \label{index} Let $(X, d)$ be a quasi-hemi-metric space. Then, the following assertions hold.

$(i)$ $c(X) \in [0,1]$.

$(ii)$ If $c(X)>0$, then $\overline{c}(X)=\frac{1}{c(X)}\in[1,+\infty[$. If  moreover, $(X, d)$ is a quasi-metric space (that is $d(x,y)=0$ iff $x=y$) then, this formula extends to the case when $c(X)=0$, with $\overline{c}(X)=+\infty$.

$(iii)$ $d$ is a metric if and only if $c(X)=1$.

\end{proposition}
\begin{proof} $(i)$ It is clear that $c(X)\geq 0$. Suppose by contradiction that $c(X)>1$. Then, for every $x, y\in X$ such that $d(y,x)>0$ we have that $d(x,y)>d(y,x)>0$. Thus, since $d(x,y)>0$, we also have that $d(y,x)>d(x,y)$ which is a contradiction.  Thus,  $c(X) \in [0,1]$. To see $(ii)$, we observe by definition that $c(X)d(y,x)\leq d(x,y)$ for all $x, y\in X$ (even if $c(X)=0$). If, $c(X)>0$, then we get that $\overline{c}(X)\leq \frac{1}{c(X)}<+\infty$. On the other hand, we have that $d(x,y)\leq \overline{c}(X)d(y,x)$ for all $x,y \in X$. This implies that $\frac{1}{\overline{c}(X)}\leq c(X)$. Hence, we have that $\overline{c}(X)=\frac{1}{c(X)}\in[1,+\infty[$. If $c(X)=0$, then, there exists a pair of sequences $(a_n,b_n)\in X\times X$ such that $d(b_n,a_n)>0$ and $\frac{d(a_n,b_n)}{d(b_n,a_n)}\to 0$. Since $(X, d)$ is a quasi-metric space and $d(b_n,a_n)>0$, then $d(a_n,b_n)>0$. Thus, $\frac{d(b_n,a_n)}{d(a_n,b_n)}\to +\infty$ and so $\overline{c}(X)=+\infty=\frac{1}{c(X)}$. For part $(iii)$, suppose that $c(X)=1$. In this case, we have that $d(x,y)\geq d(y,x)$ for all $x,y\in X$, which implies that $d(x,y)=d(y,x)$ for all $x,y\in X$. Hence $d$ is a metric. The converse is trivial.
\end{proof}
Recently, we proved in \cite{Ba-Fl}, the following result that we will use. The difficult part in this theorem is $(iii) \Longrightarrow (i)$.

\begin{theorem} \label{Ba-Fl} \textnormal{(\cite[Corollary 3]{Ba-Fl})} Let $(X, \|\cdot|)$ be an asymmetric normed space. The following assertions are equivalent.

$(i)$ $c(X)>0$.

$(ii)$ $ (X, \|\cdot|) $ is isomorphic to its associated normed space.

$(iii)$ $X^{\flat}$ is a vector space.

\end{theorem}

\vskip5mm

Now, we give our main result.

\begin{theorem} \label{Baire}  Let $(X,\|\cdot|)$ be an asymmetric normed  space. Suppose that $c(X)=0$. Then, $(X,\|\cdot|)$ is not a Baire space. If moreover, we assume that $(X,\|\cdot|)$ is  biBanach, then the converse is also. Consequently, a biBanach asymmetric normed space is a Baire space if and only if it is isomorphic to its associated normed space.
\end{theorem}

\begin{proof} Suppose that $c(X)=0$, it follows using Theorem \ref{Ba-Fl}, that $X^{\flat}$ is not a vector space. Thus, there exists a linear functional $p\in  X^{\flat}$ such that $-p\not \in X^{\flat}$. Since $p\in X^{\flat}$, then $p$ is upper semicontinuous from $(X,\|\cdot|)$ into $(\R,|\cdot|)$. For each $n\in \N$, let $F_n:=\lbrace x \in X: -p(x)\leq n \rbrace$. Then $F_n$ is a closed subset of $(X,\|\cdot|)$ (since $-p$ is lower semicontinuous) and we have that $X=\cup_{n\in \N}F_n$. Suppose by contradiction that $(X,\|\cdot|)$ is a Baire space. Then, there exists $n_0\in \N$ such that  $F_{n_0}$ has a non empty interior. Let $\varepsilon >0$ and $x_0\in X$ such that  $B_{\|\cdot|}(x_0,\varepsilon) \subset F_{n_0}$. For each $h\in B_{\|\cdot|}(0,1)$, we have that $x_0+\varepsilon h\in B_{\|\cdot|}(x_0,\varepsilon)$. Thus, $$-p(x_0+\varepsilon h)\leq n_0.$$
In other words, for each $h\in B_{\|\cdot|}(0,1)$ we have that $-p(h)\leq \frac{n_0+p(x_0)}{\varepsilon}$. This implies that $\|-p|_{\flat}\leq \frac{n_0+p(x_0)}{\varepsilon}<+\infty$  which contradict the fact that $-p\not \in X^{\flat}$.

Now, to see the second part, if $c(X)>0$, then we have that (see \cite[Proposition 3.]{Ba-Fl}):
\begin{eqnarray*} \label{eq2}
c(X)\|x\|_s\leq \|x|\leq \|x\|_s, \hspace{2mm} \forall x\in X.
\end{eqnarray*}
 Thus, as $(X, \|\cdot\|_s)$ is a Banach space, it is a Baire space and so  $(X, \|\cdot|)$ is also a Baire space, since its topology is equivalent to the norm topology by the above inequality. The equivalence between the condition $c(X)>0$ and the fact that $(X,\|\cdot|)$ is isomorphic to its associated normed space, is trivial from the above inequalities.
\end{proof}
\begin{example} \label{ex}   Let $X= l^{\infty}(\N^*)$ equipped with the asymmetric norm $\|\cdot|_{\infty}$ defined by
$$\|x|_{\infty}=\sup_{n\in \N^*}\|x_n|_{\frac 1 n}\leq \|x\|_{\infty},$$
where for each $t\in \R$ and each $n\in \N^*$, $\|t|_{\frac 1 n}=t$  if $t\geq 0$ and $\|t|_{\frac 1 n}=-\frac{t}{n}$ if $t\leq 0$. Then, for each $n\in \N^*$, we have $\|e_n|_{\infty}=1$ and $\|-e_n|_{\infty}=\frac 1 n$, where $(e_n)$ is the canonical basis of $c_0(\N^*)$. It follows that $c(l^{\infty}(\N^*))=0$ and so by the above theorem, $( l^{\infty}(\N^*), \|\cdot|_{\infty})$ is not a Baire space.
\end{example}
\begin{remark} The above theorem extends easily to asymmetric normed cone with the same proof, replacing "biBanach" by "bicomplete" (we refer to \cite{D-all, Cob} for more informations about asymmetric normed cones and their duals).
\end{remark}
Notice that it is easy to see that for any quasi-hemi-metric space $(X,d)$, we have that $ c(X)d_s\leq d\leq d_s$ ($c(X)$ my be equal to zero). Thus,  if $c(X)>0$ and $(X,d)$ is bicomplete (equivalently $(X,d_s)$ is a complete metric space), then $(X,d)$ is a Baire space as $d$ is equivalent to the complete metric $d_s$. We dont know if the converse remains true. We have the following open question.
\vskip5mm
\noindent{\bf Open question 1.} Does the result of Theorem \ref{Baire} extends to any quasi-hemi-metric space? More precisely, if $(X,d)$ is any quasi-hemi-metric space, does $c(X)=0$ implies that $(X,d)$ is not a Baire space?
\vskip5mm
We are going to prove in Theorem \ref{A-free} bellow, that the condition $c(X)=0$ in quasi-hemi-metric space,  is however always equivalent to the fact that the semi-Lipschitz free space $\mathcal{F}_a(X)$ over $(X, d)$ (introduced recently in \cite{D-all}) is not a Baire space. We recall below the notion of semi-Lipschitz free space and we refer to \cite{D-all} for more details.
\vskip5mm
 \paragraph{\bf The semi-Lipschitz free space.}  The semi-Lipschitz free space over a quasi-hemi-metric space $X_d:=(X,d)$ was recently introduced and studied in \cite{D-all}. Its construction is analogous to the classical Lipschitz free space over a metric space introduced by Godefroy-Kalton in \cite{GK}.  Notice that the cone $\textnormal{SLip}_0 (X_d)$ in our paper corresponds to the cone $\textnormal{SLip}_0 (X_{\overline{d}})$ in the paper \cite{D-all}, where $\overline{d}$ denotes the conjugate of $d$ defined by $\overline{d}(x,y)=d(y,x)$ for all $x, y\in X$.  For every $x\in X$ we consider the corresponding evaluation mapping
\begin{eqnarray*}
\delta_x : \textnormal{SLip}_0(X_{\overline{d}})&\to& \R\\
                      f &\mapsto& f(x)
\end{eqnarray*}
It is shown in \cite[Proposition 3.1]{D-all} that for each $x \in X$, both the evaluation
functional $\delta_x$ and  its opposite $-\delta_x$ belong to the dual cone of $\textnormal{SLip}_0(X_{\overline{d}})$. The semi-Lipschitz free space over $(X, d)$, denoted by $\mathcal{F}_a(X)$, is defined as the (unique) bicompletion of the asymmetric normed space $(\textnormal{span} \lbrace\delta_x : x\in X \rbrace, \|\cdot|^*)$, where $\|\cdot|^*$ is the restriction of the norm of the dual cone of $\textnormal{SLip}_0(X_{\overline{d}})$ (corresponding to $\|\cdot|_{\flat}$ in our notation). It is proved in \cite[Theorem 3.5]{D-all} that  the
dual cone of the asymmetric normed space $\mathcal{F}_a(X)$ (that is $\mathcal{F}_a(X)^{\flat}$, also denoted by $\mathcal{F}_a(X)^{*}$  in \cite{D-all})  is isometrically isomorphic to $\textnormal{SLip}_0(X_{\overline{d}})$ and  that (see \cite[Proposition 3.3]{D-all})
\begin{eqnarray*}
\delta : (X,d) &\to& (\mathcal{F}_a(X), \|\cdot|^*)\\
                      x &\mapsto& \delta_x
\end{eqnarray*}
is an isometry onto its image.  Therefore, for any $x, y \in X$, we have
$$\overline{d}(x,y)=d(y,x)=\|\delta_x-\delta_{y}|^*.$$
\begin{proposition} \label{Comp-index} Let $(X,d)$ be  a quasi-hemi-metric space and $\mathcal{F}_a(X)$ be the semi-Lipschitz free space over $(X, d)$. Then, we have that $c(\mathcal{F}_a(X))\leq c(X)$. Moreover,  we have that

$(i)$ $c(X)=0$ if and only if $c(\mathcal{F}_a(X))=0$.

$(ii)$ $c(X)=1$ if and only if $c(\mathcal{F}_a(X))=1$.
\end{proposition}
\begin{proof} By the definition of the index of symmetry, we have that
\begin{eqnarray*}
c(\mathcal{F}_a(X)) &:=&\inf_{Q\in \mathcal{F}_a(X); \|Q|^*>0}\frac{\|-Q|^*}{\|Q|^*}\\
                                 &\leq& \inf_{x, y\in X; \|\delta_x -\delta_{y}|^*>0}\frac{\|\delta_{y} -\delta_{x}|^*}{\|\delta_x -\delta_{y}|^*}\\
                                 &=& \inf_{d(y,x)>0}\frac{d(x,y)}{d(y,x)}\\
                                 &=& c(X).
\end{eqnarray*}
Clearly $c(X)=0$ implies that $c(\mathcal{F}_a(X))=0$. To see the converse,  suppose that $c(\mathcal{F}_a(X))=0$. By applying  Theorem \ref{Ba-Fl} to the asymmetric normed space $\mathcal{F}_a(X)$, we get that $\mathcal{F}_a(X)^{\flat}$ is not a vector space. Thus, there exists $f\in \mathcal{F}_a(X)^{\flat}=\textnormal{SLip}_0(X_{\overline{d}})$ such that $-f\not \in \mathcal{F}_a(X)^{\flat}=\textnormal{SLip}_0(X_{\overline{d}})$. Thus, there exists a pair of sequences $((a_n),(b_n))\subset X\times X$ such that for all $n\in \N\setminus \lbrace 0 \rbrace$, we have
\begin{eqnarray*}
nd(a_n,b_n) < f(a_n)-f(b_n) \leq \|f|_Ld(b_n,a_n).
\end{eqnarray*}
This implies that:  $\forall n\in \N\setminus \lbrace 0 \rbrace$, $d(b_n,a_n)>0$ and $\frac{d(a_n,b_n)}{d(b_n,a_n)}<\frac{\|f|_L}{n}\to0$. This shows that $c(X)=0$.

For the second part,  if  $c(\mathcal{F}_a(X))=1$, then clearly $c(X)=1$ since the index of symmetry belongs to the segment $[0,1]$. Now, suppose that $c(X)=1$, then by part $(iii)$ of Proposition \ref{index}, we have that $(X,d)$ is a metrci space. It follows that $\textnormal{SLip}_0(X_{\overline{d}})=\textnormal{Lip}_0(X_{d})$ is the classical Banach space of real-valued Lipschitz functions on $(X,d)$ that vanish at $x_0$ and so $\mathcal{F}_a(X)$ coincides with the classical Lipschitz-free space which is a Banach space. Thus $c(\mathcal{F}_a(X))=1$ by part $(iii)$ of Proposition \ref{index}.
\end{proof}
We dont know if $c(\mathcal{F}_a(X))= c(X)$, but it probably seems true.
\begin{theorem} \label{A-free} Let $(X,d)$ be  a quasi-hemi-metric space and $\mathcal{F}_a(X)$ be the semi-Lipschitz free space over $(X, d)$.  Then, $c(X)=0$ if and only if $(\mathcal{F}_a(X),\|\cdot|^*)$ is not a Baire space.
\end{theorem}
\begin{proof} The proof follows from Proposition \ref{Comp-index} and Theorem \ref{Baire}, noting that  $(\mathcal{F}_a(X),\|\cdot|^*)$ is a biBanach space by construction.
\end{proof}
From the above result, we can reformulate the remark just before the Open question 1, as follows.

\begin{proposition} Suppose that $(X,d)$ is a bicomplete quasi-hemi-metric space such that  $(\mathcal{F}_a(X),\|\cdot|^*)$  is a Baire space. Then $(X,d)$ is also a Baire space.
\end{proposition}
We dont know if the converse in the above proposition is true in general.
\vskip5mm
\noindent{\bf Open question 2.} Let $(X,d)$ be a quasi-hemi-metric space. Suppose that $(X,d)$ is a Baire space, does $(\mathcal{F}_a(X),\|\cdot|^*)$  is also a Baire space?

\vskip5mm
It  is clear that Open question 1 and Open question 2, are in fact equivalent.
\vskip5mm
\section*{Declaration} The author declare that there is no conflict of interest.
\section*{Acknowledgment}
This research has been conducted within the FP2M federation (CNRS FR 2036) and  SAMM Laboratory of the University Paris Panthéon-Sorbonne.

\bibliographystyle{amsplain}


\section{Introduction}\label{sec:introduction}

Graph convolutional neural networks (GCNNs) have shown particular promise in predicting properties of small molecules.
These properties can be biological activity against protein targets~\cite{yang_comprehensive_2020, sakai_prediction_2021, nguyen_meta-learning_2020}, more general pharmacokinetics and physicochemical properties~\cite{montanari_modeling_2019, peng_enhanced_2020, feinberg_improvement_2020} or toxicity~\cite{ma_deep_2020}.
Using accurate \textit{in silico} predictions of such properties to prioritize the synthesis of compounds can lead to tremendous time and cost savings during drug discovery projects.

However, the opaque nature of artificial neural networks has long been a stumbling block for wider adoption.
In particular, the difficulty in extracting a straightforward rationalization for a molecular prediction in terms of atomic or fragment contributions limits the adoption of machine learning models among medicinal chemists.

What makes a good rationalization?
In the case of explaining molecular properties, the answer is often site attribution.
A chemist, when presented with an image of a molecule, may infer properties like solubility and melting point from a few specific atoms and fragments rather than the molecule as a whole.
Explaining the predictions in terms of atomic contributions helps build trust in deep learning for molecular properties.

\section{Our Contribution}\label{sec:our-contribution}

In a GCNN, particular combinations of dimensions of the learned representations can be visually mapped onto the molecule after any convolution.
Existing attribution methods for GCNNs make use of that fact.
If these dimensions are correlated or if a particular prediction makes use of very many dimensions, the resulting explanations may be confusing and overwhelming for the end user.
To mitigate these problems, we introduce two new methods to improve the interpretability of GCNN predictions: Batch Representation Orthonormalization (BRO) and Gini regularization.
Both are available in the Pytorch Geometric library~\cite{fey_fast_2019}: \url{https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#functional}.
An overview of our approach is shown in Figure~\ref{fig:overview}.

\begin{figure*}[t]
    \vskip 0.2in
    \begin{center}
        \centerline{\includegraphics[trim=0 100 0 100, clip, width=\textwidth]{figures/overview.pdf}}
        \caption{
        \textbf{Batch Representation Orthnormalization and Gini Regularization}
        From left to right: the input molecule passes through a number of graph convolutional operators, generating new graph embeddings.
        $n$ embeddings of arbitrary dimension (in this figure: 128) are generated, where $n$ is the number of nodes or atoms in the graph.
        During training, a regularization loss will be computed on the final embedding, $\mathbf{H}$, according to the BRO regularizer in Equation~\ref{eq:BRO}.
        The contrast between the cosine similarities among all pairs of node representations $h_j$ for a model trained without (Baseline) and with the BRO regularizer is shown: note that in the BRO case, many more overlaps are pushed to zero.
        The node representations are then aggregated by a global average pooling layer.
        A final linear layer transforms these aggregations into predictions.
        The multi-task model depicted in the figure has 10 outputs.
        The effects of constraining the weights of this layer according to the Gini regularizer (Equation~\ref{eq:gini}) is shown on the right.
        }
        \label{fig:overview}
    \end{center}
    \vskip -0.2in
\end{figure*}

\subsection{Batch Representation Orthonormalization (BRO)}

Our work is loosely inspired by linear combination of atomic orbitals (LCAO) theory~\cite{albright_orbital_2013}.
In LCAO theory, a complete set of molecular orbitals $\psi$ is built from the superposition of an orthonormal basis set of atomic electronic wavefunctions: $\psi_j = \sum_i^N c_{ij}\phi_i$ where $\phi_i$ are the electronic atomic basis functions.
The molecular orbitals must also be mutually orthonormal--that is, the coefficients $c_{ij}$ must be normalized so that $\psi_j^* \psi_j = 1$ and $\psi_j^*\psi_{j'} = 0$.

Pictorially, one can think of LCAO as building up the molecular orbitals by sequentially making bonding and anti-bonding combinations of symmetry-related atoms until the full molecular orbital set is constructed.
GCNNs can also be conceptualized as building up a full molecular picture from atomic sites.
Instead of combining symmetry-related sites, GCNNs combine the information from increasingly distant atomic neighbors on the molecule.
This correspondence is not only superficial: the final molecular orbitals obtained by this symmetry mixing procedure can be described by the eigenvectors of the adjacency matrix.
The contrast of these two principles is depicted in Figure~\ref{fig:LCAOvsGNN}.

\begin{figure}[t]
    \vskip 0.2in
    \begin{center}
        \centerline{\includegraphics[trim=40 80 320 80, clip, width=\columnwidth]{figures/LCAO.pdf}}
        \caption{
        On the left we illustrate the LCAO principle for a hypothetical four-hydrogen molecule ($\sigma$-bonding only).
        The colors represent the sign of the atomic orbital: neighboring atomic sites with different signs are said to be anti-bonding, with correspondingly higher energy.
        Step 0 shows the topology of the molecule.
        Step 1 shows combining atomic orbitals which are symmetry-related in bonding and anti-bonding combinations.
        Step 2 shows a combination of the intermediate orbitals from Step 1 into the final molecular orbitals.
        The atomic contributions in the final molecular orbitals are sized according to the eigenvectors of the adjacency matrix, and they are ranked according to eigenvalue: most bonding (bottom) to most anti-bonding (top).
        The right side of the image shows an analogous process for GCNNs, receiving information from increasingly distant (represented by a lighter shade) neighbors for each convolutional step.
        This happens simultaneously for each node.
        }
        \label{fig:LCAOvsGNN}
    \end{center}
    \vskip -0.2in
\end{figure}

In the LCAO picture, orthonormality is guaranteed by construction.
The representations $\mathbf{H}$ created by a graph convolution ($h_{ij}$ for the $i$th atom and $j$th molecular embedding; $h_j$ is a vector that can be mapped directly onto the molecule) have no such guarantee.
In this work, we introduce an orthonormality constraint in the training process to partially recover this property.
For a molecule with representation $\mathbf{H}$, the molecular regularization loss is defined by:

\begin{equation}
    \label{eq:BRO}
    \mathcal{L}_{\textrm{BRO}}^\mathrm{mol} = \frac{\lambda}{2
    } \ || \mathbf{HH}^T - \mathbf{I}||_2
\end{equation}

Where $||\cdot||_2$ indicates the vector 2-norm and $\mathbf{I}$ is the identity matrix.
$\lambda$ is a hyperparameter.
Since the forward pass of a GCNN will typically use mini-batching, the regularization loss must be aggregated over all graphs in the batch.
Computing so many normalizations is computationally taxing, so in this work we limit BRO application to the outputs of the final graph convolution.
We observe that training typically takes four times as long.
The effects of this constraint on the node embeddings for a single molecule are shown in the center of Figure~\ref{fig:overview}.

\subsection{Gini Regularization}

Multitask neural networks (MTNNs) and multitask graph convolutional neural networks (MT-GCNNs) are of special interest for modeling molecular properties.
Often, a user is interested in multiple properties of a molecule.
Since many physicochemical properties are related, it makes intuitive sense that internal representations could be reused for different tasks.

A typical MT-GCNN architecture will follow the graph convolutional layers with a global graph aggregation layer, which aggregates the node-level outputs into graph-level outputs for graph property prediction.
A sequential, fully-connected neural network then makes graph level predictions based on the graph-level outputs from the global aggregation layer.
The simplest example of this architecture, which we will use in this paper, is a global average pooling (GAP) layer followed by a single fully-connected layer.

We wish to constrain the weights of the final fully-connected layer to be sparse.
We reason that this should have the two-fold effect of reducing the number of node representations $h_j$ that are relevant to a specific prediction and revealing which representations are shared among tasks.
The usual $\ell_1$ or $\ell_2$ regularization is not appropriate here, as penalizing the magnitude of the weights directly damages the performance of regression metrics.

Instead, we use a regularizer inspired by the Gini coefficient~\cite{gini_variabilita_1912}.\footnote{We first documented this approach in our workshop paper ~\citealt{henderson_gini_2020}.}
The Gini coefficient was invented in the context of economics as a straightforward way to compare income or wealth inequality across different countries~\cite{dorfman_formula_1979}.
A high Gini coefficient implies a inequal wealth distribution, while a low one indicates a more equitable distribution.

For our model, we want the representations to have high inequality: each prediction should be dominated by as few representations as possible.
We select each row $w_i$ of the weights of the final fully-connected layer which is responsible for predicting task $i$ from the $n$ outputs of the GAP layer and compute:

\begin{equation}
    \mathcal{L}_\textrm{Gini}^i = \sum_j^n \sum_{j'}^n \frac{
    | w_{ij} - w_{ij'} |
    } {
    2 (n^2 - n) \bar{w_i}
    }
    \label{eq:gini}
\end{equation}

where $j$ ranges over all weights in the row.
$\bar{w_i}$ is the mean weight value for the row.
The Gini coefficient $\mathcal{L}^i_\textrm{Gini}$ ranges from zero to one: zero if all $w$ are equal and one if one $w_i$ is non-zero and the rest zero.
Since weights of a linear transform are not necessarily restricted to be non-negative, we will always use $|w|$ rather than $w$ for computing the Gini regularization during training.

We take the mean $g = \frac{1}{n}\sum_i^n \mathcal{L}_\textrm{Gini}^i$ over all tasks.
The training loss becomes $L / g^m$ where $L$ is the multi-task regression loss, and $m$ is a hyperparameter to tune the effect of the Gini regularization.
The effects of the Gini regularization on a trained model are shown on the right side of Figure~\ref{fig:overview}.

\section{Related Work}\label{sec:related-work}

Our work is inspired by convolutional neural network (CNN) approaches.
In particular, the work of~\citealt{zhang_interpretable_2018} seeks to align individual convolutional filters to particular concepts, constraining categorical and spatial entropy.
Like our approach, this method requires no annotations.
Forcing orthonormality in the learned representations is conceptually similar to disentanglement or finding a more interpretable ``basis set'': this approach is explored for CNNs in~\citealt{zhou_interpreting_2019}.

Orthogonality or orthormality is often used as a constraint in training neural networks, however this is usually applied to the weights of learnable parameters~\cite{huang_orthogonal_2018}.
Recurrent neural networks in particular stand to benefit from orthonormalization as a means to address exploding or vanishing gradients~\cite{arjovsky_unitary_2016, jing_tunable_2017, vorontsov_orthogonality_2017}.
The exact form of the BRO layer was derived from ~\citealt{xie_all_2017}, although here again applied to learnable parameters rather than learned representations.

In the context of GCNNs, methods such as Graph Information Bottleneck~\cite{wu_graph_2020}, Deep Graph Infomax~\cite{velickovic_deep_2018}, and Infograph~\cite{sun_infograph_2019} seek to constrain representations according to information-theoretic principles.

Many attribution methods for neural networks have been proposed~\cite{zhou_learning_2016, selvaraju_grad-cam_2017, shrikumar_learning_2017, smilkov_smoothgrad_2017, sundararajan_axiomatic_2017} and later adapted to GCNNs ~\cite{xie_interpreting_2019, ying_gnnexplainer_2019, pope_explainability_2019, baldassarre_explainability_2019}.
In this work we focus on Class Activation Maps (CAM)~\cite{zhou_learning_2016}.
CAM is a simple yet powerful method connecting network outputs and the learned node representations.
In GCNNs where the node feature aggregation is done by a GAP layer, CAM corresponds to the scalar product of individual node features by the weights of the output layer:

\begin{equation}
    \textrm{Attribution}(\textrm{atom}_i) = w_j^\mathsf{T}  \cdot h_{i}
    \label{eq:cam}
\end{equation}

where $h_i$ corresponds to the learned feature vector for atom $i$ and $w_j$ corresponds to the row of the output weight matrix of the network corresponding to the $j$th task.
If in the single-task case, $w_j = \mathbf{W}$.
We expect our contributions to specifically help with CAM-like attribution methods: the Gini constraint reduces the number of non-zero elements in $\mathbf{W}$ and the BRO orthonormalization ensures that the different dimensions of the learned atom representations $\mathbf{H}$ are independent from each other.

\section{Experiments and Results}\label{sec:experiments-and-results}

We wish to show the effect of BRO and Gini regularization on model prediction explainability.
To illustrate this, we show the effect of our regularizers on attribution maps on benchmark datasets for which an exact attribution score can be calculated.

Next, we train new models on proprietary assay data to predict physicochemical endpoints.
We generate attribution maps from models trained with and without our constraints and survey experts on which attribution they prefer.

In all experiments, no hyperparmeter tuning was performed: the BRO and Gini regularizers are added to the baseline models with the their respective hyperparameters fixed ($\lambda=0.001$ for BRO regularization and $m=5.0$ for Gini regularization).
We chose values for parameters $\lambda$ and $m$ qualitatively: we sought the largest values for each that did not degrade evaluation metrics too much (see Section~\ref{subsec:model-performance}).

\subsection{Attribution Benchmarks}\label{subsec:attribution-benchmarks}

Recently,~\citealt{sanchez-lengeling_evaluating_2020} proposed three artificial tasks to evaluate attribution methods on molecular GCNNs.
These simple tasks have an associated ground truth explanation that can be used to score any attribution method's output.
For instance, if we trained a model to predict whether a molecule contains a benzene ring or not, a perfect attribution would highlight only the atoms in the benzene ring and no others.
We followed their protocol and metrics to benchmark the effect of our proposed regularizers on the attribution performance.

We focus on CAM and CAM-derived attribution methods, including GradCAM~\cite{selvaraju_grad-cam_2017} either on all convolutional layers or only the last one.
This is because the authors found CAM to systematically outperform other attribution methods.
Since our regularization method has not been implemented yet on edge features, we focus on the benchmark graph architectures that use node features exclusively (graph convolutional network (GCN) and graph attention network (GAT) in~\citealt{sanchez-lengeling_evaluating_2020}).
We also introduce a variation on CAM that we call TopRep.
TopRep applies the CAM Equation~\ref{eq:cam} using only the weight of the output layer that has the highest magnitude: that is, $w_j$ of Equation~\ref{eq:cam} retains the entry with the largest absolute value, and all others are set to zero.

\subsubsection{Dataset and Network Architecture}

\citealt{sanchez-lengeling_evaluating_2020} present three attribution tasks: ``Benzene'', ``Amine-Ether-Benzene'' and ``CrippenLogP.''
The Benzene task is as described above.
The Amine-Ether-Benzene task tests if each of the Amine, Ether, and Benzene fragments are present in the molecule (logical AND).
Both of these attribution tasks are scored using AUROC~\cite{bradley_use_1997}.
The underlying classification task is also scored with AUROC.
The CrippenLogP task is a regression task, with both attribution and regression scored with Pearson correlation.
We discuss this task in detail at the end of this section.

For the GCN and GAT models, we use the hyperparemeters given in Sanchez-Lengeling and modify their code only insofar as necessary to add the BRO and Gini regularizers.
Eighty trials of each combination are run.
Our modifications to the benchmarking code are available at \url{https://github.com/bayer-science-for-a-better-life/graph-attribution}.

\subsubsection{Results}

Figure~\ref{fig:attribution_boxplots} show the detailed distributions of attribution scores for the Benzene, Amine-Ether-Benzene, and CrippenLogP tasks ($n=80$ for every model/task combination).
For many combinations, BRO or Gini regularization markedly improve the attribution AUROC score.
Often, the combination of BRO and Gini yields better results than either regularizer alone.
This effect can be most strongly seen in the Benzene GAT combination (bottom row).
This bolsters our reasoning that it takes both sparsity \textit{and} orthogonal respresentations to generate a good attribution.
While there are some task/model combinations for which none of our constraints improve attribution, no combination's attribution AUROC score suffers significantly from the Gini+BRO regularizers.

TopRep also gives competitive results, even for the baseline configuration.
This is surprising in the baseline case, because TopRep is implicitly throwing out a lot of information if the output weight matrix has not been made sparse through, for example, Gini regularization.

Table~\ref{tab:CAM_benchmark} summarizes our results on all three benchmark tasks, including CrippenLogP.

\begin{figure}
    \centering
    \begin{subfigure}{\linewidth}
        \centering
        \includegraphics[trim=40 85 340 83, clip, width=1.0\linewidth]{figures/boxplot_benzene.pdf}
    \end{subfigure}
    \begin{subfigure}{\linewidth}
        \centering
        \includegraphics[trim=40 85 340 113, clip, width=1.0\linewidth]{figures/boxplot_logic10.pdf}
    \end{subfigure}
    \begin{subfigure}{\linewidth}
        \centering
        \includegraphics[trim=40 90 340 113, clip, width=1.0\linewidth]{figures/boxplot_crippen.pdf}
    \end{subfigure}
    \caption{
    Boxplots of attribution AUROC scores for all tasks.
    Both GCN and GAT models were considered.

    }
    \label{fig:attribution_boxplots}
\end{figure}

\begin{table}[h]
    \scriptsize
    \centering
    \caption{Performance of the attribution methods on the three benchmark tasks (mean for 80 trials per combination).
    For classification tasks, we report the attribution AUROC score and for regression the Pearson correlation.
    Best value in each column in bold.
    }
    \label{tab:CAM_benchmark}
    \resizebox{\columnwidth}{!}{
    \begin{tabular}{llrrrrrr}
        \toprule
        &       & \multicolumn{2}{c}{Benzene} & \multicolumn{2}{c}{AmEthBenz} & \multicolumn{2}{c}{CrippenLogP} \\
        \multicolumn{1}{l}{Attr.} & Const. & GCN   & GAT   & GCN   & GAT   & GCN   & GAT \\
        \midrule
        Random & None  & 0.614  & 0.613  & \cellcolor[rgb]{ .914,  .949,  .886} 0.500 & {\cellcolor[rgb]{ .898,  .941,  .867}} 0.495 & {\cellcolor[rgb]{ 1,  .996,  .992}} -0.086 & -0.093 \\
        & BRO   & 0.613  & 0.613  & \cellcolor[rgb]{ .894,  .937,  .863} 0.507 & \cellcolor[rgb]{ .878,  .933,  .843} 0.503 & -0.092 & -0.093 \\
        & Gini  & \cellcolor[rgb]{ .996,  1,  1} 0.614 & 0.613  & \cellcolor[rgb]{ .922,  .957,  .898} 0.498 & \cellcolor[rgb]{ .886,  .937,  .855} 0.500 & -0.091 & \cellcolor[rgb]{ 1,  .996,  .992} -0.088 \\
        & both  & 0.612  & 0.615  & \cellcolor[rgb]{ .918,  .953,  .89} 0.499 & \cellcolor[rgb]{ .886,  .937,  .855} 0.500 & -0.091 & \cellcolor[rgb]{ 1,  1,  .996} -0.091 \\
        \midrule
        Grad & None  & \cellcolor[rgb]{ .875,  .906,  .961} 0.678 & \cellcolor[rgb]{ .89,  .918,  .969} 0.670 & \cellcolor[rgb]{ .988,  .996,  .984} 0.475 & \cellcolor[rgb]{ .941,  .965,  .922} 0.476 & \cellcolor[rgb]{ .961,  .698,  .518} 0.130 & \cellcolor[rgb]{ .969,  .757,  .612} 0.071 \\
        CAM & BRO   & \cellcolor[rgb]{ .882,  .914,  .965} 0.673 & \cellcolor[rgb]{ .91,  .933,  .973} 0.659 & 0.471  & \cellcolor[rgb]{ .886,  .937,  .855} 0.500 & \cellcolor[rgb]{ .957,  .682,  .494} 0.141 & \cellcolor[rgb]{ .973,  .792,  .667} 0.048 \\
        (last) & Gini  & \cellcolor[rgb]{ .796,  .847,  .937} 0.718 & \cellcolor[rgb]{ .835,  .878,  .949} 0.697 & \cellcolor[rgb]{ .851,  .918,  .812} 0.520 & \cellcolor[rgb]{ .808,  .89,  .753} 0.536 & \cellcolor[rgb]{ .949,  .635,  .42} 0.176 & \cellcolor[rgb]{ .973,  .796,  .675} 0.045 \\
        & both  & \cellcolor[rgb]{ .8,  .851,  .937} 0.715 & \cellcolor[rgb]{ .8,  .851,  .937} 0.716 & \cellcolor[rgb]{ .8,  .886,  .741} 0.537 & \cellcolor[rgb]{ .882,  .933,  .847} 0.503 & \cellcolor[rgb]{ .949,  .627,  .404} 0.182 & \cellcolor[rgb]{ .973,  .8,  .682} 0.041 \\
        \midrule
        Grad & None  & \cellcolor[rgb]{ .847,  .886,  .953} 0.692 & \cellcolor[rgb]{ .973,  .98,  .992} 0.628 & \cellcolor[rgb]{ .949,  .973,  .933} 0.488 & 0.448  & \cellcolor[rgb]{ .949,  .627,  .408} 0.181 & \cellcolor[rgb]{ .976,  .827,  .729} 0.023 \\
        CAM & BRO   & \cellcolor[rgb]{ .851,  .886,  .953} 0.691 & \cellcolor[rgb]{ .965,  .973,  .988} 0.632 & \cellcolor[rgb]{ .969,  .984,  .961} 0.482 & \cellcolor[rgb]{ .937,  .965,  .918} 0.477 & \cellcolor[rgb]{ .949,  .608,  .376} 0.194 & \cellcolor[rgb]{ .98,  .859,  .773} 0.003 \\
        (all) & Gini  & \cellcolor[rgb]{ .769,  .824,  .929} 0.733 & \cellcolor[rgb]{ .855,  .894,  .957} 0.687 & \cellcolor[rgb]{ .843,  .91,  .796} 0.523 & \cellcolor[rgb]{ .878,  .929,  .843} 0.505 & \cellcolor[rgb]{ .949,  .612,  .384} 0.192 & \cellcolor[rgb]{ .984,  .867,  .784} -0.002 \\
        & both  & \cellcolor[rgb]{ .769,  .827,  .929} 0.732 & \cellcolor[rgb]{ .835,  .875,  .949} 0.698 & \cellcolor[rgb]{ .824,  .898,  .769} 0.530 & \cellcolor[rgb]{ .945,  .969,  .925} 0.474 & \cellcolor[rgb]{ .945,  .6,  .365} 0.201 & \cellcolor[rgb]{ .988,  .894,  .835} -0.022 \\
        \midrule
        CAM   & None  & \cellcolor[rgb]{ .275,  .451,  .773} \textcolor[rgb]{ 1,  1,  1}{0.989} & \cellcolor[rgb]{ .278,  .459,  .773} \textcolor[rgb]{ 1,  1,  1}{0.984} & \cellcolor[rgb]{ .537,  .737,  .404} \textcolor[rgb]{ 1,  1,  1}{0.621} & \cellcolor[rgb]{ .467,  .694,  .314} \textcolor[rgb]{ 1,  1,  1}{0.691} & \cellcolor[rgb]{ .933,  .506,  .216} \textcolor[rgb]{ 1,  1,  1}{0.268} & \cellcolor[rgb]{ .937,  .533,  .259} \textcolor[rgb]{ 1,  1,  1}{0.221} \\
        & BRO   & \cellcolor[rgb]{ .275,  .451,  .773} \textcolor[rgb]{ 1,  1,  1}{0.989} & \cellcolor[rgb]{ .29,  .467,  .776} \textcolor[rgb]{ 1,  1,  1}{0.977} & \cellcolor[rgb]{ .545,  .741,  .412} \textcolor[rgb]{ 1,  1,  1}{0.619} & \cellcolor[rgb]{ .447,  .682,  .29} \textcolor[rgb]{ 1,  1,  1}{0.701} & \cellcolor[rgb]{ .933,  .498,  .204} \textcolor[rgb]{ 1,  1,  1}{0.274} & \cellcolor[rgb]{ .937,  .537,  .263} \textcolor[rgb]{ 1,  1,  1}{0.219} \\
        & Gini  & \cellcolor[rgb]{ .271,  .451,  .773} \textcolor[rgb]{ 1,  1,  1}{0.990} & \cellcolor[rgb]{ .267,  .447,  .769} \textcolor[rgb]{ 1,  1,  1}{\textbf{0.989}} & \cellcolor[rgb]{ .439,  .678,  .278} \textcolor[rgb]{ 1,  1,  1}{\textbf{0.653}} & \cellcolor[rgb]{ .439,  .678,  .278} \textcolor[rgb]{ 1,  1,  1}{\textbf{0.703}} & \cellcolor[rgb]{ .933,  .494,  .196} \textcolor[rgb]{ 1,  1,  1}{0.278} & \cellcolor[rgb]{ .941,  .553,  .29} \textcolor[rgb]{ 1,  1,  1}{0.208} \\
        & both  & \cellcolor[rgb]{ .271,  .451,  .773} \textcolor[rgb]{ 1,  1,  1}{0.990} & \cellcolor[rgb]{ .275,  .451,  .773} \textcolor[rgb]{ 1,  1,  1}{\textbf{0.989}} & \cellcolor[rgb]{ .463,  .69,  .306} \textcolor[rgb]{ 1,  1,  1}{0.646} & \cellcolor[rgb]{ .451,  .686,  .294} \textcolor[rgb]{ 1,  1,  1}{0.699} & \cellcolor[rgb]{ .929,  .49,  .192} \textcolor[rgb]{ 1,  1,  1}{\textbf{0.279}} & \cellcolor[rgb]{ .929,  .49,  .192} \textcolor[rgb]{ 1,  1,  1}{\textbf{0.248}} \\
        \midrule
        TopRep & None  & \cellcolor[rgb]{ .267,  .447,  .769} \textcolor[rgb]{ 1,  1,  1}{\textbf{0.991}} & \cellcolor[rgb]{ .286,  .463,  .776} \textcolor[rgb]{ 1,  1,  1}{0.979} & \cellcolor[rgb]{ .737,  .851,  .663} 0.556 & \cellcolor[rgb]{ .678,  .816,  .584} 0.595 & \cellcolor[rgb]{ .945,  .592,  .353} 0.205 & \cellcolor[rgb]{ .953,  .635,  .42} 0.153 \\
        & BRO   & \cellcolor[rgb]{ .286,  .463,  .776} \textcolor[rgb]{ 1,  1,  1}{0.982} & \cellcolor[rgb]{ .314,  .482,  .784} \textcolor[rgb]{ 1,  1,  1}{0.967} & \cellcolor[rgb]{ .957,  .976,  .945} 0.486 & \cellcolor[rgb]{ .627,  .788,  .518} 0.619 & \cellcolor[rgb]{ .945,  .58,  .337} 0.214 & \cellcolor[rgb]{ .957,  .667,  .471} 0.131 \\
        & Gini  & \cellcolor[rgb]{ .282,  .459,  .776} \textcolor[rgb]{ 1,  1,  1}{0.984} & \cellcolor[rgb]{ .271,  .451,  .773} \textcolor[rgb]{ 1,  1,  1}{0.988} & \cellcolor[rgb]{ .882,  .933,  .847} 0.510 & \cellcolor[rgb]{ .761,  .863,  .69} 0.558 & \cellcolor[rgb]{ .945,  .588,  .349} 0.208 & \cellcolor[rgb]{ .953,  .643,  .431} 0.148 \\
        & both  & \cellcolor[rgb]{ .271,  .451,  .773} \textcolor[rgb]{ 1,  1,  1}{\textbf{0.991}} & \cellcolor[rgb]{ .271,  .451,  .773} \textcolor[rgb]{ 1,  1,  1}{0.987} & \cellcolor[rgb]{ .741,  .855,  .667} 0.555 & \cellcolor[rgb]{ .682,  .82,  .592} 0.593 & \cellcolor[rgb]{ .945,  .596,  .357} 0.204 & \cellcolor[rgb]{ .949,  .631,  .416} 0.155 \\
        \bottomrule
    \end{tabular}%
    }
    \label{tab:addlabel}%
\end{table}%

We notice that the CAM-related attribution methods work poorly for the regression task CrippenLogP.
This was already observed in~\citealt{sanchez-lengeling_evaluating_2020}, where only the integrated gradients approach~\cite{sundararajan_axiomatic_2017} seemed to perform slightly better than other attribution methods.

We interpret this finding in the construction of the benchmark task itself.
The authors used a dataset with experimentally determined solubility~\cite{delaney_esol_2004} and consider as ground truth explanation the atom contribution for the water-octanol partition coefficient as calculated by the Crippen LogP empirical model~\cite{wildman_prediction_1999}.
Although logP and solubility are correlated, we assume that logP is not always a good explanation for a more complex endpoint like solubility, and therefore it should not be surprising if the built model's explanations do not align very well with the logP atom contributions.
A better task might be to build a model for logP and use the Crippen LogP attributions as ground truth explanations.
This experiment was not performed here to stay in the framework of the benchmark.

\subsection{Physicochemical endpoints}\label{subsec:physicochemical-tasks}

The results on the constructed benchmark dataset in the previous section encouraged us to evaluate our methods against expert opinion: a much costlier undertaking.

It is impossible to reliably and quickly calculate properties like solubility or binding affinity $ab\ initio$ from the chemical structure of a compound, let alone attribute the prediction to specific sites or functional groups.
When screening drug candidates, a medicinal chemist may synthesize many similar molecules with the desired activity while simultaneously trying to minimize or maximize some other property.

Thus, being able to guess which modifications may alter these properties is extremely helpful.
Here, we seek to mimic the chemists' intuition, and see if the BRO and Gini constraints may produce models that generate attributions that more closely match their instincts.

\subsubsection{Dataset}

The data used for building the multitask GCNN model is a dataset of measured physicochemical properties for small molecules from 10 different assays.
Extensive discussion of the preparation and characteristics of the dataset can be found in our previous work~\citealt{montanari_modeling_2019}.
There are a total of 537,443 compounds with assays covering properties such as solubility, lipophilicity (logD at two different pHs), melting point, human serum albumin binding and membrane affinity; 79\% of the compounds are measured for only one endpoint, 11\% for two, 9\% for 3, and 1\% four or more.

All ten endpoints along with their frequencies and the correlation between pairs is shown in Figure~\ref{fig:gini_correlations}.
We also show the cosine similarities of the rows of the trained output weights corresponding to each endpoint.
The similarities of both the baseline and Gini-constrained model mimic the underlying correlation between the measurements, but the Gini-constrained version is sparser.
This may imply it has learned a more specific relationship between the endpoints than merely the data distribution.

Seven cross-validation folds were generated using $k$-means clustering on the ECFC6 fingerprints~\cite{rogers_extended-connectivity_2010}.

\begin{figure}
    \vskip 0.2in
    \begin{center}
        \centerline{\includegraphics[trim=40 80 250 80, clip, width=\columnwidth]{figures/correlations.pdf}}
        \caption{
        The first row shows the Pearson correlation between each pair of endpoints, along with a table giving a description and count of each.
        The second row shows the cosine similarity of the rows of an output weight matrix for an unconstrained and Gini-constrained model.
        }
        \label{fig:gini_correlations}
    \end{center}
    \vskip -0.2in
\end{figure}

\subsubsection{Network Architecture}

The model, initially built in Tensorflow with the DeepChem library ~\cite{ramsundar_deep_2019} was converted to Pytorch Geometric for easier access and customization~\cite{fey_fast_2019}.
We tried as much as possible to follow the DeepChem graph convolution mechanism which is an implementation of the~\citealt{duvenaud_convolutional_2015} algorithm.

Input node features are computed with DeepChem and consist of 75 atomic properties: number of radical electrons, whether the atom is aromatic, and one-hot encoding on each of the atom type, degree, formal charge, and hybridization.
Two convolutional layers are then applied, then a GAP layer going from individual node features to a global graph encoding.
A hyperbolic tangent function is applied to the graph encoding before the final linear layer.

Our models were trained using the ADAM optimizer~\cite{kingma_adam_2017} for forty epochs with exponential learning rate decay with a base of 0.97 decaying every 1000 steps.
No hyperparemeter search is performed: models with either or both of BRO and Gini regularization use the same hyperparameters as the baseline models.
Model code is available in the Supplementary Material.

\subsubsection{Survey of Medicinal Chemists}

To test our methods in a realistic setting, we prepared a survey to gather opinions from human experts.
We collected public compounds with reported measured solubility~\cite{sorkun_aqsoldb_2019}, logD~\cite{alelyunas_high_2010, low_optimised_2016} or melting point~\cite{williams_melting_2015}.
These compounds were chosen for their problematic properties: they were either very insoluble, having a high logD, or a low melting point.

We then used the multitask GCN built on physicochemical data (in both the constrained and non constrained version) to predict the solubility, logD or melting point of those molecules.
Because the data is collected from public sources, potentially many different assays were used to measure the different endpoints of interest.
Therefore, an exact match of the predictive model (built on proprietary established assay data) with the publicly reported values is not expected.
Since we focused on problematic molecules, we compared predictions and reported experimental values qualitatively only.
For example, insoluble molecules with a predicted solubility over 30 mg/L were discarded.
In general, molecules with grossly inaccurate predictions by the models were excluded from the survey.
In total, 10 molecules were kept for solubility, 8 for logD and 6 for melting point.
Attribution methods were then applied on each predicted molecule: either CAM (our baseline, using the unconstrained model), CAM$_{\mathrm{BRO+Gini}}$ (CAM applied to the Gini-sparsified and BRO-disentangled model), TopRep$_{\mathrm{BRO+Gini}}$ (CAM using only the top weight from the constrained model output), or a random attribution map (any node representation generated by the constrained model, selected randomly).

The users always had the possibility to skip a question or select a negative answer (``No answer convinces me'').
The attribution maps were plotted on the molecule structure using the RDKit's SimilarityMap functionality~\cite{landrum_rdkit_2006}.
An example question as shown to the medicinal chemists is shown in Figure~\ref{fig:medchemq}.
15 medicinal chemists answered the 24 questions and the results are shown in Figure~\ref{fig:survey}.
The participants were all employed at Bayer on two different sites in Germany, and were at different stages of their career.
They had no previous explicit experience with such specific molecules or tasks although the molecules are known drugs and the tasks are tasks they reason about on a daily basis.

\begin{figure}[ht]
    \vskip 0.2in
    \begin{center}
        \centerline{\includegraphics[trim=340 170 340 200, clip, width=\columnwidth]{figures/medchemq.pdf}}
        \caption{
        An example question from the survey of medicinal chemists: ``Which atoms drive the low solubility of the molecule? (Green: lower solubility / Magenta: Higher solubility.''
        This shows one of the query molecules for solubility taken from~\cite{sorkun_aqsoldb_2019}  with various site attributions for solubility (LOO task).
        Answers were randomly ordered for each question, blind to both us and the medicinal chemists.
        In this case we have a) CAM$_{\mathrm{BRO+Gini}}$ b) random c) CAM (baseline) d) TopRep$_{\mathrm{BRO+Gini}}$.
        }
        \label{fig:medchemq}
    \end{center}
    \vskip -0.2in
\end{figure}

\begin{figure}
    \vskip 0.2in
    \begin{center}
        \centerline{\includegraphics[trim=40 80 320 80, clip, width=\columnwidth]{figures/survey_results.pdf}}
        \caption{
        Results of survey of medicinal chemists.
        Not all participants answered all questions.
        The plot shows the distribution of responses for each option for each endpoint.
        For example: for logD questions, a median of 7 respondents (ranging from 0 to 10) picked CAM$_{\mathrm{BRO+Gini}}$, with 50\% of the logD questions receiving between 4 to 8 votes for CAM$_{\mathrm{BRO+Gini}}$.
        }
        \label{fig:survey}
    \end{center}
    \vskip -0.2in
\end{figure}

CAM$_{\mathrm{BRO+Gini}}$ and TopRep$_{\mathrm{BRO+Gini}}$ are the most voted attribution methods among the 24 questions, with CAM$_{\mathrm{BRO+Gini}}$ being favored for logD and TopRep$_{\mathrm{BRO+Gini}}$ for solubility and melting point.
Across all endpoints, they are both picked more often than would be expected at random (binomial test, p-value=0.01 and p-value $<$ 0.001 respectively).
Surprisingly, Baseline CAM cannot be distinguished from the Random attribution result (binomial test, p-value=0.07).

This real-life experiment shows that, with a predictive model that is in practice useful for medicinal chemists, the regularization constraints (Gini and BRO) lead to significantly preferred attribution maps.
We also note that the logD endpoint is the easiest for chemists to rationalize and led to the most imbalanced votes in favor of CAM$_{\mathrm{BRO+Gini}}$ (coefficient of variation $\sigma/\mu=0.95$).
Solubility and melting point are less straightforward, and, accordingly, chemists tend to disagree more in their voting ($\sigma/\mu=0.86$ and 0.59 respectively).

All survey questions with attribution maps, including aggregated responses and answer key, are available in the Supplementary Material.

\subsection{Model Performance}\label{subsec:model-performance}

The Gini and BRO regularizations have a slight negative impact on the classification or regression metrics of the models.
The impact on the AUROC scores for the ``Benzene'' and ``Amine-Ether-Benzene'' classification tasks and the Pearson correlation for the ``CrippenLogP'' regression task described in Section~\ref{subsec:attribution-benchmarks} are illustrated in Figure~\ref{fig:auroc}.

As with the Attribution AUROC, it is nearly impossible to improve the classification AUROC of the Benzene task: all models are extremely close to perfect classification.
Even in this regime, it is clear that the BRO layer negatively effects performance.
In the Amine-Ether-Benzene task, the differences are not significant.
This may be because both BRO and Gini regularization were designed with multitask models in mind, and the Amine-Ether-Benzene is really a multitask problem in disguise.
Finally, adding the BRO and Gini constraints to the CrippenLogP task does not drastically alter model performance, despite doing little to improve interpretability.

\begin{figure}
    \vskip 0.2in
    \begin{center}
        \centerline{\includegraphics[trim=35 225 390 80, clip, width=\columnwidth]{figures/auroc.pdf}}
        \caption{
        Effect of regularization techniques on the evaluation metric for the Benzene, Amine-Ether-Benzene, and CrippenLogP tasks.
        }
        \label{fig:auroc}
    \end{center}
    \vskip -0.2in
\end{figure}

The various endpoints of the multitask models trained on physicochemical data described in Section~\ref{subsec:physicochemical-tasks} are affected unevenly.
The cross-validation $R^2$ scores are shown in Table~\ref{tab:admet_r2}.
The solubility endpoints LOP and LOX are strongly affected.
These are the two least represented assays in the dataset, with 2,334 and 7,392 out of 537,443 compounds, respectively.

\begin{table}[htb]
  \centering
  \small
  \caption{Effect of BRO and Gini regularizers on predictive performance for all physicochemical endpoints.
  Mean of scaffold-split cross-validation $R^2$ score and standard deviation in parenthesis.}
  \label{tab:admet_r2}
    \resizebox{\columnwidth}{!}{
    \begin{tabular}{lrrrr}
    \toprule
          & Baseline & BRO   & Gini  & BRO + Gini \\
    \midrule
    LOD   & \cellcolor[rgb]{ .647,  .647,  .647} 0.85 (0.03) & \cellcolor[rgb]{ .682,  .686,  .255} 0.82 (0.03) & \cellcolor[rgb]{ .212,  .686,  .733} \textcolor[rgb]{ 1,  1,  1}{0.83 (0.03)} & \cellcolor[rgb]{ .29,  .6,  .29} \textcolor[rgb]{ 1,  1,  1}{0.81 (0.04)} \\
    LOA   & \cellcolor[rgb]{ .667,  .667,  .667} 0.82 (0.04) & \cellcolor[rgb]{ .694,  .698,  .282} 0.80 (0.04) & \cellcolor[rgb]{ .227,  .69,  .737} \textcolor[rgb]{ 1,  1,  1}{0.82 (0.04)} & \cellcolor[rgb]{ .318,  .616,  .318} \textcolor[rgb]{ 1,  1,  1}{0.79 (0.04)} \\
    LOM   & \cellcolor[rgb]{ .835,  .835,  .835} 0.56 (0.13) & \cellcolor[rgb]{ .863,  .863,  .671} 0.53 (0.13) & \cellcolor[rgb]{ .659,  .863,  .886} 0.53 (0.12) & \cellcolor[rgb]{ .71,  .835,  .71} 0.51 (0.13) \\
    LOH   & \cellcolor[rgb]{ .875,  .875,  .875} 0.50 (0.07) & \cellcolor[rgb]{ .89,  .894,  .741} 0.48 (0.09) & \cellcolor[rgb]{ .749,  .902,  .918} 0.47 (0.09) & \cellcolor[rgb]{ .808,  .89,  .808} 0.44 (0.10) \\
    LMP   & \cellcolor[rgb]{ .914,  .914,  .914} 0.44 (0.08) & \cellcolor[rgb]{ .91,  .91,  .784} 0.45 (0.08) & \cellcolor[rgb]{ .792,  .918,  .929} 0.44 (0.08) & \cellcolor[rgb]{ .82,  .898,  .82} 0.43 (0.09) \\
    LON   & \cellcolor[rgb]{ .867,  .867,  .867} 0.51 (0.09) & \cellcolor[rgb]{ .89,  .894,  .741} 0.48 (0.09) & \cellcolor[rgb]{ .718,  .886,  .906} 0.49 (0.08) & \cellcolor[rgb]{ .792,  .882,  .792} 0.45 (0.10) \\
    LOO   & \cellcolor[rgb]{ .875,  .875,  .875} 0.50 (0.20) & \cellcolor[rgb]{ .902,  .906,  .773} 0.46 (0.24) & \cellcolor[rgb]{ .749,  .902,  .918} 0.47 (0.20) & \cellcolor[rgb]{ .792,  .882,  .792} 0.45 (0.23) \\
    LOP   & \cellcolor[rgb]{ .894,  .894,  .894} 0.47 (0.15) & \cellcolor[rgb]{ .918,  .918,  .8} 0.44 (0.19) & 0.30 (0.40) & \cellcolor[rgb]{ .945,  .969,  .945} 0.34 (0.34) \\
    LOQ   & \cellcolor[rgb]{ .835,  .835,  .835} 0.56 (0.11) & \cellcolor[rgb]{ .867,  .867,  .686} 0.52 (0.14) & \cellcolor[rgb]{ .643,  .859,  .882} 0.54 (0.11) & \cellcolor[rgb]{ .722,  .843,  .722} 0.50 (0.15) \\
    LOX   & \cellcolor[rgb]{ .859,  .859,  .859} 0.52 (0.06) & \cellcolor[rgb]{ .89,  .894,  .741} 0.48 (0.07) & \cellcolor[rgb]{ .792,  .918,  .929} 0.44 (0.09) & \cellcolor[rgb]{ .82,  .898,  .82} 0.43 (0.09) \\
    \bottomrule
    \end{tabular}%
    }
\end{table}%

It is possible that hyperparameter tuning could recover these performance losses.
Anecdotally, we note that adding additional graph convolutional layers seems to help.
This can, however, dilute the explanations which become further ``de-localized'' with each convolution.

\section{Conclusion}

We describe and implement two new regularization techniques: Batch Representation Orthonormalization and Gini regularization.
We show qualitatively that these techniques disentangle node representations and force models to make predictions using fewer of them.

We demonstrate that models trained with these constraints generate better site attributions on a benchmark dataset for many attribution methods.
Further, we show that for models trained on assay data of interest to medicinal chemists, human experts significantly prefer attribution maps generated from models trained with the constraints introduced in this paper.

While we give an exact definition for the BRO constraint in Equation~\ref{eq:BRO}, the choice of the vector 2-norm is arbitrary.
The novelty of our approach is normalizing the node representations per graph, and it is likely that other norms may yield better results in other circumstances.
Similarly, while the Gini coefficient has desirable properties for this application, other measures of dispersion could also be explored.

Finally, our methods could be extended to graph architectures which exploit edge features.
Future work may explore the separate or joint orthonormalization of node and edge features.

\section*{Acknowledgements}

We would like to thank the Bayer medicinal chemists based in Berlin and Wuppertal for their essential feedback on our work.
We would also like to thank Andreas Goeller, Marco Bertolini, Jorge Kageyama, and Tuan Le for their helpful input.
Funding in direct support of this work: Bayer AG Life Science Collaboration (``Explainable AI'').