\section{Introduction}


\begin{figure}[t]
    \begin{subfigure}[b]{0.2\textwidth}
        \centering
        \includegraphics[scale=0.25]{kg_chain.pdf}
    \caption{Chain structure.}
    \label{fig:KG_chain}
    \end{subfigure}
    \begin{subfigure}[b]{0.276\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{kg_ring.pdf}
        \caption{Ring structure.}
    \label{fig:KG_ring}
    \end{subfigure}
    \centering
    \begin{subfigure}[b]{0.425\textwidth}
        \centering
        \includegraphics[width=1\linewidth]{kg_tree.pdf}
        \caption{Hierarchy structure.}
    \label{fig:KG_hie}
    \end{subfigure}
    \caption{An illustration of three types of KG structures. (a) shows the most common chain structure in KGs, which can usually be directly modelled in Euclidean space (e.g., TransE, TransH, etc.). (b) is a ring structure of KGs that can be captured in hypersphere space~\cite{cao2022geometry}. (c) stands for hierarchical structure which is usually encoded in hyperbolic or spherical spaces~\cite{chami2020low,sun2020knowledge}. }
    \label{fig:illu_KG_struc}
\end{figure}


Knowledge Graphs are a type of multi-relational graphs that store the factual knowledge in real-world. Nodes in KGs represent real-world entities (e.g., names, events and products) and edges represent relationships between the entities. 
Normally, a KG can be efficiently stored as knowledge triples, where each triple consists of two entities and one factual relation between them (i.e., \textit{<head entity, relation, tail entity>}). 
For example, in the triple \textit{<RNA virus, subclass, COVID-19}>, \textit{RNA virus} and \textit{COVID-19} are real-world entities and \textit{subclass} represents the relation between \textit{RNA virus} and \textit{COVID-19}.
Over the recent years, rapid growth has been witnessed in building large-scale KGs, such as YAGO~\cite{suchanek2007yago}, Wikidata~\cite{vrandevcic2014wikidata}, Freebase~\cite{bollacker2008freebase} and DBepedia~\cite{auer2007dbpedia}. Due to their effectiveness in storing and representing factual knowledge, they have been successfully applied in question answering~\cite{saxena2020improving,yasunaga2021qa}, recommendation system~\cite{zhou2020improving,sun2020multi}, information retrieval~\cite{wise2020covid,gaur2022iseeq} and other domain-specific applications~\cite{long2020integrated,li2020real}.
Despite KGs are effective in representing structured factual information, 
they are difficult to manipulate due to the large-scale and complicated graph structure. 
Therefore, how to effectively and efficiently extract and 
leverage useful information in large-scale KGs for downstream tasks, such as link prediction~\cite{sun2019rotate,zhang2019quaternion,chami2020low} and entity classification~\cite{yu2022jaket,xie2016representation,ji2015knowledge}, is a tough task. To tackle this challenging task, the \textbf{Knowledge Graph Embedding} (KGE) technique was proposed, and has been receiving a lot of attention in machine learning community~\cite{bordes2013translating,sun2019rotate,chami2020low,10.1145/3447772,liang2021cross,meng2021mixture}. The essential idea of KGE is to learn to embed entities and relations of a KG into a low-dimensional space (i.e. vectorial embeddings), where the embeddings
are required to preserve the semantic meaning and relational structure of the original KG. The learned embeddings of entities and relations can then be leveraged to solve downstream applications, such as KG completion~\cite{zhang2021drug,abboud2020boxe,zhang2020few,bordes2013translating, wang2014knowledge}, question answering~\cite{yasunaga2021qa,zheng2021knowledge,dong2015question,xu2016question,liang2021profiling}, information extraction~\cite{fei2021enriching,zhang2021fine,hoffmann2011knowledge,wu2010open} and entity classification~\cite{li2021robust,shen2012linden}. 

Many KGE techniques have been proposed to learn the embeddings of entities and relations in KGs~\cite{sun2019rotate,wang2014knowledge,lin2015learning,ji2016knowledge,xiao2015transa}.
Some KGE methods propose to learn KG embeddings by preserving \emph{relational patterns} between entities in KGs. For example, in order to capture the transformation relationships between entities, TransE~\cite{bordes2013translating} was proposed to embed KGs into \textit{Euclidean Space} and represent relations between entities as translation vectors between entity embeddings in the vector space. Moreover, in order to preserve and infer other relational patterns including symmetry, antisymmetry, inversion and composition in KGs, RotatE~\cite{sun2019rotate} was proposed to map KGs into \textit{Complex Vector Space}, where relations are represented as rotations between entities.

Another line of KGE methods propose to learn KG embeddings by preserving \emph{structural patterns} of KGs. This line of works was motivated by the fact that large-scale KGs usually contain many complex and compound structures. 
For example, in Figure~\ref{fig:illu_KG_struc}, we provide an illustration of three typical types of structure patterns in KGs, namely chain structure, ring structure and hierarchy structure.
In order to effectively capture the hierarchy structures in KGs, ATTH~\cite{balazevic2019multi} was proposed to embed KGs into  \textit{Hyperbolic Space} with trainable curvatures, where richer transformations can be used to separate nodes than Euclidean space~\cite{nickel2017poincare}, while capturing logical patterns simultaneously. 

In addition, some KGE methods also try to embed KGs in other mathematical spaces to model some desirable properties in KGs. For example, KG2E~\cite{he2015learning} is the first ``density-based” embedding technique, which learns embeddings of KGs in \textit{Probability Space} with
Gaussian distribution, to model the uncertainties of entities and relations. Moreover, TorusE~\cite{ebisu2018toruse} chooses a compact \textit{Lie Group} as its embedding manifold to deal with the regularisation problems, and ModulE~\cite{chai2022module} also introduce \textit{Group} theory to model both entities and relations as group element, which can accommodate and outperform most of the existing KGE models.

From the perspective of representation space, we found that above KGE methods mostly learn embeddings in different mathematical spaces, e.g., Euclidean space, Hyperbolic space and Probability space, to capture different relational and structural patterns in KGs. Indeed, different mathematical spaces have their unique strengths, which are beneficial to capture different patterns and properties in KGs. 
{Therefore, we argue that representation space plays a significant role in KGE methods, as it determines the patterns and properties of KGs that can be captured and preserved by KG embeddings.} In addition to the KGE domain, some studies~\cite{murphy2012machine,peng2021hyperbolic,burnaev2021manifold} also demonstrate the importance of mathematical space in traditional machine learning.

Some surveys have been devoted to discussing traditional machine learning models from the perspective of mathematical space~\cite{murphy2012machine,peng2021hyperbolic}.
However, there is no yet a systematic review of KGE methods from the perspective of mathematical space. 
Existing surveys about KGE methods focus either on the encoding model or the applications of KGE methods. For example, 
Wang et al.~\cite{wang2017knowledge} classify KGE methods based on their embedding functions and categorise them into three folds: translation-based models, semantic matching models and additional information-based models. Ji et al.~\cite{ji2021survey} provide a full-scaled view to introduce KGE from four aspects:
{representation learning}, scoring function, encoding models and auxiliary information. 
Lu et al.~\cite{lu2020utilizing} survey KGE methods with a concentration on utilising textual information.


\begin{figure}[t]
    \centering
    \resizebox{0.95\textwidth}{!}{
        \includegraphics[scale=0.5]{KGE_represent.pdf}
    }
    \vspace{-1em}
    \caption{Three perspectives and corresponding instances for introducing representation spaces in knowledge graph embedding: (a) Algebraic Structure. (b) Geometric Structure.  (c) Analytical Structure.}
    \label{fig:rep_in_kge}
\end{figure}


\textit{Accordingly, in this paper we aim at providing a comprehensive survey on representation spaces for knowledge graph embedding techniques,
summarising different properties of representation spaces as well as providing guidance for building KGE methods. }
In order to have a better understanding of KGE methods from a novel spatial view, inspired by the fundamental mathematical space system, we build a systematic, comprehensive and multi-angle categorisation to classify existing KGE methods based on their representation spaces. 
Specifically, we propose to classify existing KGE methods into three categories, namely \textbf{Algebraic Structure}, \textbf{Geometric Structure} and \textbf{Analytical Structure}. 
Figure \ref{fig:rep_in_kge} provides an overview of our classification framework and some representative KGE methods that belong to each category (the detailed version can be found in Section~\ref{sec:spaces_in_KGE}). 
In this survey, we will introduce the definitions and properties of the above three mathematical structures and introduce some representative KGE methods that belong to these categories in detail. Moreover, we will summarise the experimental results of different KGE methods and provide some suggestions and guidance for building more expressive and powerful KGE methods. Furthermore, we will point out new trends and further directions of KGE methods from the perspective of representation space. 

{To the best of our knowledge, we are the first survey to summarise KGE models by establishing a comprehensive mathematical spatial architecture.} To sum up, the contributions of our work can be summarised as follows:
\begin{itemize}
    \item This is the first paper that comprehensively surveys the relationships between mathematical spaces and KGE techniques. Particularly, we summarise properties of different mathematical spaces used in KGE methods, 
    so as to clearly understand their mathematical properties for different KGE approaches.
    \item We categorise existing KGE models according to their representation spaces, while providing detailed descriptions and comparisons of these works from the perspective of mathematical spaces. 
    \item We provide ideas of space selection for the KGE task based on our analysis on the essential properties of different spaces, which could help researchers and practitioners better understand the space characteristics, and provide guidance for building their KGE models (including loss function, optimisations, etc.). 
    \item 
    We put forward some suggestions and future directions for the KGE tasks by showing some unique properties in different mathematical spaces/structures. These properties can be inspired and generalised to other scenarios such as natural language processing, transformer, etc., not only for the KGE task.
\end{itemize}


The remainder of this article is organised as follows. 
Section~\ref{sec:preliminaries} introduces notations and the rigorous definitions of fundamental mathematical spaces, as well as the relationships between them. This section will provide some preliminary knowledge about various representation spaces, and build connection between these spaces and the three key components of KGE models (i.e. embedding mapping, score function and representation training). 
Since the fundamental mathematical spaces could not cover diverse spaces used in existing KGE methods, 
we develop a systematic and comprehensive framework to categorise KGE methods from the perspective of representation space. To highlight the excellent effect that different mathematical features could give to KGE, section~\ref{sec:spaces_in_KGE} introduces the proposed classification category, properties of different spaces, as well as
summarise how spatial advantages work in KGE models. Subsequently, Section~\ref{sec:applications} introduces some spatially related KG downstream tasks. Through the results, the advantages of mathematical space in particular scenes and which features are critical to the tasks are well summarised.
Finally, we present our conclusion and future work in Section~\ref{sec:futuredirec}, in which we summarise the respective strengths of three different mathematical structures and the reasons behind them, which will help inspire us to construct state-of-the-art algorithms in more fields, no limited to KGE. 




\section{Preliminaries}
\label{sec:preliminaries}
In this section, we introduce the notations used throughout the paper, provide two lines of preliminary knowledge related to our survey: knowledge graph embedding (KGE) and fundamental mathematical spaces.
Specifically, we introduce our notations in subsection~\ref{subsec:notationsAndBackground}, provide an overview of knowledge graph embedding methods in subsection~\ref{subsec:kge} and briefly introduce some basic mathematical spaces and their relationships in subsection~\ref{subsec:relationship}. 


\subsection{Notations and Mathematical Background}
\label{subsec:notationsAndBackground}

The definitions of some mathematical terminologies and symbols that appear in the text are shown in Table \ref{tab:symbol_tem}. We denote mathematical spaces
by blackboard bold characters (e.g., $\mathbb{S}$ denotes topological space). Particularly, we use $\mathbb{R}$ and $\mathbb{C}$ to denote the field of real numbers and the field of complex numbers, respectively. We represent scalars with normal characters (e.g., $x \in \mathbb{R}$ denotes a real scalar), while vectors and matrices are denoted by the bold lowercase characters (e.g., $\mathbf{z} \in \mathbb{R}^n$ denotes a real vector) and the bold uppercase characters (e.g., $\mathbf{X} \in \mathbb{R}^{m \times n}$ denotes a real-valued matrix), respectively.

We represent a knowledge graph as $\mathcal{G} = \{\mathcal{E}, \mathcal{R}, \mathcal{T}\}$, where $\mathcal{E}$ denotes the set of entities (nodes), $\mathcal{R}$ denotes the set of relations (the types of edges) and $\mathcal{T}$ represents the relational facts (edges) in the knowledge graph. Facts observed in $\mathcal{G}$ are stored as a collection of triples: $\mathcal{T} =\{ (h, r, t) \}$, where each triple consists of a head entity $h \in \mathcal{E}$, a tail entity $t \in \mathcal{E}$, and a relation $r \in \mathcal{R}$ between them, e.g., \textit{<Beijing, isCapitalOf, China}> represents the fact that Beijing is the capital of China. We also use lower case and bold character to denote the embeddings of entities and relations. Specifically, for a fact triple $(h, r, t)$, we represent the embeddings of head entity, tail entity and the relation between them as $\mathbf{h}$, $\mathbf{t}$ and $\mathbf{r}$, respectively. 


\begin{table*}[!t]
    \centering
    \footnotesize
    \renewcommand{\arraystretch}{1.2}
    \caption{Descriptions of symbols and terminologies.}
    \begin{tabular}{L{3cm}L{8.8cm}<{}}
        \toprule
        \textbf{Symbol or Terminology}               & \multicolumn{1}{c}{\textbf{Definition}}                                               \\ \midrule
         $\varnothing $                               & The set containing no elements is called an empty set $\varnothing $.     \\ \midrule
        $\mathbb{R}^n$,$\mathbb{C}^n$...             & Commonly, $\mathbb{R}^n$ represents $n$-dimensional (Real) Vector Space, $\mathbb{C}^n$ denotes $n$-dimensional Complex Vector Space. Other spaces' symbols will be explained in additional detail below.                                                                                                  \\ \midrule
        $\mathbb{S}$,   $\tau$              & A set $\mathbb{S}$  is a finite or infinite collection of objects in which order has no significance, and multiplicity is generally also ignored. And the set $\tau$ is an \textit{open set} if every point in $\tau$ has a neighbourhood lying in the set.                           \\ \midrule
        \textit{Intersection}($\cap$), union($\cup$) & The intersection of two sets $\mathcal{A}$ and $\mathcal{B}$ is the set of elements common to $\mathcal{A}$ and $\mathcal{B}$($\mathcal{A}\cap\mathcal{B}$). The union of two sets $\mathcal{A}$ and $\mathcal{B}$ is the set obtained by combining the members of each($\mathcal{A}\cup\mathcal{B}$). \\ \midrule
        \textit{Field}                               & A field is any set of elements that satisfies the field axioms for both addition and multiplication and is a commutative division algebra.                                                                                                                                                             \\ \midrule
        \textit{Complete(ness)}                      & A space is a complete metric space in which every Cauchy sequence is convergent.                                                                                                                                                                                                                       \\ \midrule
        \textit{Complex conjugate}                   & The complex conjugate of a complex number $z=a+bi$ is defined to be $\overline{z}=a-bi$.                                                                                                                                                                                                               \\ \midrule
        \textit{Homeomorphism}                       & A homeomorphism, also called a continuous transformation, is an equivalence relation and one-to-one correspondence between points in two geometric figures or topological spaces that is continuous in both directions.                                                                                \\
        \bottomrule
    \end{tabular}
    \label{tab:symbol_tem}
\end{table*}


\subsection{Knowledge Graph Embedding}
\label{subsec:kge}
\emph{Given a knowledge graph $\mathcal{G}$, the goal of KGE is to learn a mapping function $f$, which projects the entities and relations in $\mathcal{G}$ into a dense and low-dimensional space. The learned embeddings are expected to preserve the structural and attribute information of the original knowledge graph as much as possible, such that they can be leveraged to effectively and efficiently infer the relationships between entities. }

Normally, the paradigm of learning knowledge graph embeddings consists of three components, namely \textit{embedding mapping}, \textit{score function} and \textit{representation training}. The \textit{embedding mapping component} projects the entities and relations to a low-dimensional space and represents them as embedding vectors. A variety of mathematical spaces have be used to define the embeddings of entities and relations. For example, TransE~\cite{bordes2013translating} proposes to learn embeddings in Euclidean space to model the transformation of entities, while RotatE~\cite{sun2019rotate} proposes to embed KGs into complex vector space to model symmetry/antisymmetry of relations. 
Additionally, many other mathematical spaces have also been leveraged in KGEs, such as probability space~\cite{xiao2015transg,he2015learning}, hyperbolic space~\cite{balazevic2019multi,pan2021hyperbolic} and spherical space~\cite{xiao2015one,dong2021hypersphere,cao2022geometry}.
The main focus of this survey is to provide an thorough review of KGE methods from the perspective of embedding space, summarising different properties of different embedding spaces and providing guidance for building KGE methods.

The \textit{score function} is another key component of KGE methods. Score function, denoted as $s(\cdot)$, is a function defined on the mapped embedding space, which is used to measure the plausibility that a triple holds. Specifically, the score function is defined on the embedding vectors of a triple, i.e., $s(\mathbf{h}, \mathbf{r}, \mathbf{t})$, which is supposed to assign higher scores to positive triples (real facts) while assigning lower scores to negative triples (false facts). Based on different representation spaces, different score functions have been proposed to measure the plausibility of triples~\cite{bordes2013translating,sun2019rotate,xiao2015one}. For example, in Euclidean representation space, the translation-based score function $s(\mathbf{h}, \mathbf{r}, \mathbf{t}) =- || \mathbf{h} + \mathbf{r} - \mathbf{t}||_{1/2}$~\cite{bordes2013translating} is widely used to measure the confidence that a triple is positive. Based on the different properties of representation spaces, different score functions can be defined. Please refer to Section~\ref{sec:spaces_in_KGE} for detailed discussion of different score functions.

The \textit{representation training} component of KGE methods aims to learn the entity and relation embeddings by maximising the scores of positive triples while minimising the scores of negative triples. Since it is infeasible to obtain the precise positive and negative triples in a knowledge graph, one convention in the field of KGE is to regard observed triples, i.e., $\mathcal{T}$, as positive examples while sampling unobserved triples, i.e., $\mathcal{T}^{-}$, as negative examples~\cite{bordes2013translating,wang2014knowledge}. The negative triples can be generated by randomly replacing the head entity or the tail entity in an observed triple with a random entity sampled from the entity set, i.e., 
\begin{align}
    \mathcal{T}^{-} = & \{ (h^{-}, r, t) \mid (h,r,t) \in \mathcal{T} \wedge (h^{-}, r, t) \notin \mathcal{T} \wedge h^{-} \in \mathcal{E} \} \cup \notag \\
    & \{(h, r, t^{-}) \mid (h,r,t) \in \mathcal{T} \wedge (h, r, t^{-}) \notin \mathcal{T} \wedge t^{-} \in \mathcal{E} \}. 
\end{align}


Given the positive and negative triples (i.e. $\mathcal{T}$ and $\mathcal{T}^{-}$), various objective functions can be used to train representations of KGE models. For example, the \textit{margin-based ranking loss}~\cite{bordes2013translating,wang2014knowledge,lin2015learning,ji2015knowledge} is a widely adopted objective function for training representations of entities and relations. It is defined as: 
\begin{align}
    \mathcal{L}_{margin} = \sum_{(h,r,t) \in \mathcal{T}} \sum_{(h^{-}, r, t^{-}) \in \mathcal{T}^{-}} \max \left(0, \gamma - s(\mathbf{h}, \mathbf{r}, \mathbf{t}) + s(\mathbf{h^{-}, \mathbf{r}, \mathbf{t}^{-}}) \right),
\end{align}
where $\gamma$ is a margin hyperparameter. The margin-based ranking loss assumes that observed triples are more valid than unobserved triples, and therefore favours higher scores of observed triples $\mathcal{T}$ than unobserved triples $\mathcal{T}^{-}$. Another widely used objective function for learning knowledge graph embeddings is the \textit{cross-entropy loss}~\cite{trouillon2016complex,dettmers2018convolutional,lacroix2018canonical,sun2019rotate}, which is defined as:
\begin{align}
    \mathcal{L}_{CE} = \sum_{(h,r,t) \in \mathcal{T} \cup \mathcal{T}^{-}} \log (1 + \exp(-y_{hrt} \cdot s(\mathbf{h}, \mathbf{r}, \mathbf{t}))),
\end{align}
where $y_{hrt} \in \{ -1, 1 \}$ is the label of a triple $(h,r,t)$, and $y_{hrt}=1$ indicates that the triple is a positive example while $y_{hrt}=-1$ indicates that the triple is a negative example. After defining the objective function, the knowledge graph embeddings are learned by minimising the objective function via stochastic optimisation, where a small batch of positive examples and negative examples are sampled for optimisation at each training iteration. It is worth noting that most KGE models~\cite{bordes2013translating,socher2013reasoning,yang2014embedding,trouillon2016complex,nickel2016holographic} have additional constraints that the norm of knowledge graph embeddings is less than or equal to $1$. Such constraints can prevent the model to trivially minimise the objective function by simply increasing the norm of the embedding vectors~\cite{bordes2013translating}. 


\subsection{Fundamental Mathematical Spaces}
\label{subsec:relationship}

In general, various types of fundamental mathematical spaces have been widely used for representation learning in computer science. However, there is no easy way to elaborate all these spaces due to the complex inclusion and overlapping of these spaces. Here we first introduce some basic spaces through algebraic definitions. We start by introducing the topological space, which is the most basic mathematical space. 
\textit{Topological Space} is a kind of mathematical structure defined on a set, from which the concept of \textit{convergence, connectivity, continuity} etc., are introduced~\cite{rudin1973functional}.
A special case of topological space is the \textit{Metric Space}, where the \textit{distance} between any elements in the set is introduced in the topological space. The \textit{Vector Space}, also referred to as \textit{Linear Space}~\cite{rudin1973functional}, is another fundamental space that are widely used in representation learning. The vector space is a set that defines both \textit{addition} and \textit{multiplication} satisfying eight operation conditions (see~\S~\ref{subsec:vector_space} for details).
On the basis of vector space, 
\textit{Normed Vector Space} is defined, which additionally introduces the concept of \textit{norm} to define the \textit{length} between elements in the set. Particularly, \textit{Inner Product Space} is a special normed vector space, 
which additionally introduce \textit{inner product} to define the concept of \textit{angle} between elements in the set.
\textit{Euclidean Space} is a finite dimensional inner product space which is widely used nowadays~\cite{lang2012algebra,rudin1973functional}.

The inclusion relationship of the above spaces can be summarised by: $\left \{ \textit{Inner Product Vector Space} \right \}\subset \left \{ \textit{Normed Vector Space} \right \}\subset\left \{ \textit{Metric Space} \right \} \subset \left \{ \textit{Topological Space} \right \}$. Going from the left to the right, each category is carried by the next one. In inner product vector space, we can use \textit{inner product} to express both the \textit{length} and \textit{angle} of vectors, since the \textit{inner product} induces a \textit{norm}. But in a normed vector space, we could just measure the distance between two points through space metric. Then in metric space and topological space, there is less basic concept that can be measured. In another word, the ``capacity'' of spaces is getting weaker.
To sum up, the \textit{inner product} induces \textit{norm}, \textit{norm} induces distance, and distance induces topology, therefore: \textit{every inner product space is a normed vector space, every normed vector space is a metric space and every metric space is a topological space.} It should be noticed that linear space is an algebraic structure while topological space is a topological structure. So they are not juxtaposed since they are considered in different categories. The relationships between different mathematical spaces are presented in Figure~\ref{fig:space_relationships}. In what follows, we provide rigorous definitions of above mathematical spaces, which also be summarised in Table~\ref{tab:basic_space}.


\begin{figure}[t]
    \centering
    \includegraphics[scale=0.5]{relation_of_basic_spaces.pdf}
    \caption{The relationships between different mathematical spaces. The basic spatial relationship is $\left \{ \textit{Inner Product Vector Space} \right \}\subset \left \{ \textit{Normed Vector Space} \right \}\subset\left \{ \textit{Metric Space} \right \} \subset \left \{ \textit{Topological Space} \right \}$. In addition, we tease out more detailed relations, such as the \textit{complete} case of Inner Product Space is Hilbert Space.}
    \label{fig:space_relationships}
\end{figure}


\subsubsection{Topological Space}
\label{subsec:vectortopo_space}


\begin{definition}
A \textit{topological space} is a set $\mathbb{S}$ in which a collection of subsets $\tau$  (called \textit{open sets}, see Table \ref{tab:symbol_tem}) is specified, with the following properties~\cite{rudin1973functional}: $\mathbb{S}$ is open, $\varnothing $ is open, the intersection (see Table \ref{tab:symbol_tem}) of any two open sets is open, and the union of every collection of open sets is open. Such a collection $\tau$ is called a \textit{topology} on $\mathbb{S}$. When clarity seems to demand it, the topological space corresponding to the topology $\tau$ will be written as $(\mathbb{S}, \tau)$ rather than $\mathbb{S}$. Metric space, uniform space~\cite{treves2016topological,lee2010introduction} are examples of topological space.
\end{definition}


\subsubsection{Vector Space (Linear Space)}
\label{subsec:vector_space}

\begin{definition}
Let $\Phi$ stands for either $\mathbb{R}$ or $\mathbb{C}$, i.e., real field or complex filed (See Table \ref{tab:symbol_tem}). A \textit{scalar} is a member of the \textit{scalar field} $\Phi$. A \textit{vector space} over $\Phi$ is a set $\mathbb{X}$, whose elements are called vectors, and in which two operations, \textit{addition} and \textit{scalar multiplication}, are defined, with the following 
algebraic properties~\cite{rudin1973functional}:
\begin{itemize}
    \item[(a)] 
          For any two vectors $\mathbf{x}$ and $\mathbf{y}$ in $\mathbb{X}$, 
          the addition between them is represented as $\mathbf{x}+\mathbf{y}$, with the following properties:
          \begin{equation}
              \mathbf{x} + \mathbf{y} = \mathbf{y} + \mathbf{x}  \text{ and }  \mathbf{x} + (\mathbf{y} + \mathbf{z}) = (\mathbf{x} + \mathbf{y}) + \mathbf{z} .
          \end{equation}

          The vector space $\mathbb{X}$ contains a unique vector $\mathbf{0}$ (the \textit{zero vector} or \textit{origin} of $\mathbb{X}$) such that $\mathbf{x} + \mathbf{0} = \mathbf{x}$ for every $\mathbf{x} \in \mathbb{X}$, and
          each $\mathbf{x} \in \mathbb{X}$ corresponds a unique vector $-\mathbf{x}$ such that $\mathbf{x} + (-\mathbf{x}) = \mathbf{0}$.


    \item[(b)] 
          For any scalar $\alpha, \beta \in \Phi$ and vector $\mathbf{x} \in \mathbb{X}$, the scalar multiplication between them is denoted as $\alpha \mathbf{x}$, with the following properties: 
          \begin{equation}
              \alpha (\mathbf{x} + \mathbf{y}) = \alpha \mathbf{x} + \alpha \mathbf{y}, (\alpha + \beta) \mathbf{x} = \alpha \mathbf{x} + \beta \mathbf{x} .
          \end{equation}
          
          Like the zero vector in vector space, the zero element of the scalar field can also be defined in a similar way, which is denoted as $0$. 
\end{itemize}

Particularly, a \textit{real vector space} is the one for which $\Phi = \mathbb{R}$, while a \textit{complex vector space} is the one for which $\Phi = \mathbb{C}$. In the following, any statements about vector spaces in which the scalar field is not explicitly specified means that they can be applied to both real vector space and complex space.
\end{definition}


\subsubsection{Normed Space}

\begin{definition}
A vector space $\mathbb{X}$ is said to be a \textit{normed space} if
each vector $\mathbf{x} \in \mathbb{X}$ is associated with a nonnegative real number $\left \| \mathbf{x} \right \|$, called the \textit{norm} of $\mathbf{x}$~\cite{rudin1973functional}, in such a way that: 
\begin{itemize}
    \item[(a)] $\left \| \mathbf{x} + \mathbf{y} \right \|\leq \left \| \mathbf{x} \right \| + \left \| \mathbf{y} \right \|$ for all $\mathbf{x}$ and $\mathbf{y}$ in $\mathbb{X}$,
    \item[(b)] $\left \| \alpha \mathbf{x} \right \| = \left | \alpha \right | \left \| \mathbf{x} \right \|$ if $\mathbf{x} \in \mathbb{X}$ and $\alpha$ is a scalar,
    \item[(c)] $\left \|  \mathbf{x} \right \| > 0$ if $\mathbf{x}\neq \mathbf{0}$.
\end{itemize}
\quad The word ``norm'' is also used to denote the \textit{function} that maps $\mathbf{x}$ to $\left \| \mathbf{x}  \right \|$.
Every normed space can be regarded as a particular metric space, in which the distance $d(\mathbf{x},\mathbf{y})$ between $\mathbf{x}$ and $\mathbf{y}$ is defined as $\left \| \mathbf{x} - \mathbf{y} \right \|$. 
The properties of the distance function $d$ in metric space are: 
\begin{itemize}
    \item[(a)] $0\leq d(\mathbf{x},\mathbf{y})< \infty $ for all $\mathbf{x}$ and $\mathbf{y}$,
    \item[(b)] $d(\mathbf{x},\mathbf{y}) = 0$ if and only if $\mathbf{x} = \mathbf{y}$,
    \item[(c)] $d(\mathbf{x},\mathbf{y}) = d(\mathbf{y},\mathbf{x})$ for all $\mathbf{x}$ and $\mathbf{y}$,
    \item[(d)] $d(\mathbf{x},\mathbf{z}) \leq d(\mathbf{x},\mathbf{y}) + d(\mathbf{y},\mathbf{z})$ 
    for all $\mathbf{x},\mathbf{y},\mathbf{z}.$
\end{itemize}
\quad A \textit{Banach Space} is a special normed space which is \textit{complete} (see Table \ref{tab:symbol_tem}) in the metric defined by its norm, which means that every Cauchy sequence is required to converge. {(A Cauchy sequence is a sequence whose elements become arbitrarily close to each other as the sequence progresses~\cite{lang2012algebra}.)}
\end{definition}


\begin{table*}[!t]
    \centering
    \footnotesize
    \renewcommand{\arraystretch}{1.2}
    \caption{Descriptions of four basic spaces in mathematics: Vector Space, Normed Space, Inner Product Space and Topological Space. It is important to note that they are listed here for a general introduction, not because they are juxtaposed. The specific relationships between these spaces can found in~\S\ref{subsec:relationship}.}
    \label{tab:basic_space}
    \vspace{-0.5em}
    \begin{tabular}{p{3cm}p{5cm}p{3cm}}
        \toprule
        \textbf{Space}             & \textbf{Description}                                                                                        & \textbf{Property}                         \\
        \midrule
        \term{Topological Space}   & a geometrical space in which \textbf{closeness} is defined.                                                 & \makecell[l]{$\bullet$ \textit{open set}} \\
        \midrule
        \term{Vector Space}        & a set of vectors in which two operations, \textbf{addition} and \textbf{scalar multiplication} are defined. & \makecell[l]{$\bullet$ \textit{addition}  \\ $\bullet$ \textit{multiplication}}                               \\
        \midrule
        \term{Normed Space}        & a vector space over the real or complex numbers, on which a \textbf{norm} is defined.                       & \makecell[l]{$\bullet$ \textit{addition}  \\$\bullet$ \textit{multiplication}\\ $\bullet$ \textit{norm}: $\left \| x \right \|$}                       \\
        \midrule
        \term{Inner Product Space} & a vector space with a binary operation called an \textbf{inner product}.                                    & \makecell[l]{$\bullet$ \textit{addition}  \\ $\bullet$ \textit{multiplication} \\$\bullet$ \textit{inner product}:$\left \langle x,x \right \rangle$}  \\
        \midrule
        \term{Euclidean Space} & a commonly used \textbf{finite} dimensional inner product space over real numbers.                                    & \makecell[l]{$\bullet$ \textit{addition}  \\ $\bullet$ \textit{multiplication} \\$\bullet$ \textit{inner product}:$\left \langle x,x \right \rangle$} \\
        \midrule
        \term{Manifold} & a topological space which is \textbf{locally Euclidean}.                                  & \makecell[l]{$\bullet$ each point has \\a certain neighbourhood} \\
        \bottomrule
    \end{tabular}
\end{table*}


\subsubsection{Inner Product Space}

\begin{definition}
A complex vector space $\mathbb{H}$ is called an \textit{inner product space} or \textit{unitary space} if
each ordered pair of vectors $\mathbf{x}$ and $\mathbf{y}$ in $\mathbb{H}$ is associated with a complex number $\langle\mathbf{x}, \mathbf{y}\rangle$, called the \textit{inner product} or \textit{scalar product} of $\mathbf{x}$ and $\mathbf{y}$, such that the following rules hold~\cite{rudin1973functional}:
\begin{itemize}
    \item[(a)] $\langle\mathbf{y}, \mathbf{x}\rangle = \overline{\langle\mathbf{x}, \mathbf{y}\rangle}$ (The bar denotes complex conjugation (see Table \ref{tab:symbol_tem})),
    
    \item[(b)] $\langle\mathbf{x} + \mathbf{y}, \mathbf{z}\rangle = \langle\mathbf{x}, \mathbf{z}\rangle + \langle\mathbf{y}, \mathbf{z}\rangle$,
    
    \item[(c)] $\langle\alpha \mathbf{x}, \mathbf{y}\rangle $= $\alpha\langle\mathbf{x}, \mathbf{y}\rangle$ if $\mathbf{x} \in \mathbb{H}, \mathbf{y} \in \mathbb{H}, \alpha \in \mathbb{C}$,
    
    \item[(d)] $\langle\mathbf{x}, \mathbf{x}\rangle \geq 0$ for all $\mathbf{x} \in \mathbb{H}$,
    \item[(e)] $\langle\mathbf{x}, \mathbf{x}\rangle = 0$ only if $\mathbf{x} = 0$.
\end{itemize}
Particularly, if the normed space is complete, it is called a \textit{Hilbert Space}.
\end{definition}


\subsubsection{Euclidean Space}

\begin{definition}
\textit{Euclidean space} is the basic space of geometry, intended to represent physical space~\cite{ball1960short}. Generally, Euclidean space refers to Euclidean vector space, which is a finite dimensional inner product space over real numbers. Based on the algebraic definition of Euclidean space, a plane or solid (i.e. Euclidean geometry) can be well represented by lines and points. 
\end{definition}


\subsubsection{Manifold}

\begin{definition}
A topological space $\mathbb{M}$ is said to be a \textit{manifold} or \textit{locally Euclidean of dimension n} if every point of $\mathbb{M}$ has a neighbourhood in $\mathbb{M}$ that is homeomorphic (see Table \ref{tab:symbol_tem}) to an open subset of $n$-dimensional Euclidean space~\cite{lee2010introduction}. Manifolds constitute a generalisation of objects and the concept of a manifold is central to many parts of geometry since it allows complicated structures, such as sphere, curved surface, etc. to be described in terms of well-understood topological properties of simpler spaces.
\end{definition}




\section{Representation Spaces in Knowledge Graph Embedding}
\label{sec:spaces_in_KGE}

Since a KG usually consists of many complicated structures (e.g., 1-to-N, N-to-N, and hierarchical relationships), 
researchers have proposed to embed KGs in different representation spaces in order to better preserve such complicated structural information~\cite{xiao2015one,ebisu2018toruse,zhang2019quaternion,guo2021bique,chami2020low}. Indeed, different representation spaces have their unique structures and properties, as we show in Section~\ref{sec:preliminaries}.
However, in addition to the fundamental mathematical spaces introduced in Section \ref{sec:preliminaries}, there are more spaces that provide better properties for KGE. For example, in hyperbolic space, the region and length increase exponentially with the radius, which provides more available space for embedding task~\cite{balazevic2019multi,chami2020low,nickel2017poincare}. Moreover, in Lie group, embedding vectors will never diverge unlimitedly and therefore regularisation of embedding vectors is no longer required for effective learning~\cite{ebisu2018toruse}. 
As a result, KGE methods built on different representation spaces are able to capture and preserve different structural and attribution information in original KGs. However, there is neither a systematic review of KG embedding methods from the perspective of representation spaces, nor any literature showing how to properly choose representation space given particular KGE tasks. In this paper, we aim to fill this gap by summarising KGE methods based on the structures and properties of their mathematical representation spaces. 

Note that in Section~\ref{sec:preliminaries}, we introduced some algebraic definitions of some basic spaces. Based on them, some geometric perspectives, such as Euclidean geometry and hyperbolic geometry, can also be introduced accordingly. At the same time, we notice that there are various kinds of mathematical spaces in KGE, which play a significant role and belong to different mathematical structures. 
The diverse spaces in KGE often have complicated relationships. For example, manifolds and Euclidean geometry have inclusion relations since there are overlapping structures between them. In addition, some spaces are not juxtaposed such as spherical space and probability space because they originate from different mathematical structures, which cannot be discussed in the same category.

As a result, in order to better understand the influence of different mathematical representation spaces on KGE methods, we build a systematic, comprehensive, multi-angle categorisation to classify the special spaces and categorise KGE models more accurately based on three mathematical structures, namely  
\textbf{Algebraic Structure}, \textbf{Geometric Structure} and \textbf{Analytical Structure}.
Most KGE models fall under these three structures, which proves the rationality of our categorisation, as shown in Figure \ref{fig:tree_KGE}. It is worth noting that we will add additional definitions in Section~\ref{sec:spaces_in_KGE} for KGE spaces that are not mentioned in Section~\ref{sec:preliminaries}.

In this section, 
we will first describe the definitions and properties of the above three mathematical structures, after which we will provide some representative KGE methods that learn embedding in the corresponding mathematical structure, as well as summarise how spatial advantages work in KGE models. 


\begin{figure*}
    \footnotesize
    \resizebox{1\textwidth}{!}{
    \begin{forest}
        for tree={
        forked edges,
        grow'=0,
        draw,
        rounded corners,
        node options={align=center,},
        text width=4cm,
        s sep=12pt,
        calign=child edge, calign child=(n_children()+1)/2,
        l sep = 10 pt,
        l=40pt
        },
        [Knowledge Graph Embedding, part0,fill=m1,draw=black
        [Geometric Structure , part0,fill=m2,draw=black
        [Euclidean Geometry,part0,fill=m3,draw=black
        [Cartesian Coordinate,part0,fill=m4,draw=black
        [TransE~\cite{bordes2013translating}; RotatE~\cite{sun2019rotate};\\ PairRE~\cite{chao2021pairre}; InterHT~\cite{wang2022interht};\\ TripleRE~\cite{yu2022triplere}; TranS~\cite{zhang2022trans};\\ CompoundE~\cite{ge2022compounde};HopfE~\cite{bastos2021hopfe}; 
        ,fill=m5,draw=white,
        draw=black,align=left,
        inner xsep=1.5mm,text width=4.2cm]
        ]
        [Polar Coordinate,part0,fill=m4,draw=black
        [HAKE~\cite{zhang2020learning}; H$^2$E~\cite{wang2021knowledge};\\ HBE~\cite{pan2021hyperbolic},fill=m5,draw=white,rounded corners=3pt,align=left,
        draw=black,text width=4.2cm,inner xsep=1.5mm]
        ]
        [Spherical Coordinate,part0,fill=m4,draw=black
        [STKE~\cite{wang2022stke},fill=m5,draw=white,rounded corners=3pt,text centered,text width=4.3cm,draw=black]
        ]
        ]
        [Hyperbolic Geometry,part0,fill=m3,draw=black
        [MuRP~\cite{balazevic2019multi}; ATTH~\cite{chami2020low}; \\ HBE~\cite{pan2021hyperbolic}; HyperKA~\cite{sun2020knowledge};\\
        UltraE~\cite{xiong2022ultrahyperbolic}; RotL~\cite{wang2021hyperbolic};\\HyperHKGE~\cite{pan2021hyperbolic}; H$^2$E~\cite{wang2021knowledge};\\
        GIE~\cite{cao2022geometry}; MuRMP~\cite{wang2021mixed},fill=m5,draw=white,rounded corners=3pt,align=left,inner xsep=0pt,draw=black,text width=4cm
        ]
        ]
        [Spherical Geometry,part0,fill=m3,draw=black
        [ManifoldE~\cite{xiao2015one};TransC~\cite{lv2018differentiating};\\ GIE~\cite{cao2022geometry}; MuRS~\cite{wang2021mixed}\\
        HyperspherE~\cite{dong2021hypersphere};,fill=m5,draw=white,rounded corners=3pt,align=left,inner xsep=0pt,draw=black,text width=4cm]
        ]
        ]
        [Algebraic Structure, part0,fill=m2,draw=black
        [Group , part0,fill=m3,draw=black
            [TorusE~\cite{ebisu2018toruse}; DihEdral~\cite{xu2019relation};\\ NagE~\cite{yang2020nage}; ModulE~\cite{chai2022module};\\ KGLG~\cite{ebisu2019generalized}; DensE~\cite{lu2020dense};\\
            GrpKG~\cite{yang2021knowledge},fill=m5,draw=white,rounded corners=3pt,align=left,inner xsep=0pt,draw=black,text width=4cm]
        ]
        [Ring, part0,fill=m3,draw=black
            [M$\ddot{o}$biusE~\cite{chen2021mobiuse},fill=m5,draw=white,rounded corners=3pt,align=left,inner xsep=0pt,draw=black,text width=4cm]
        ]
        [Vector Space, part0,fill=m3,draw=black
        [Real Vector Space,part0,fill=m4,draw=black
        [TransE $\&$ its extensions~\cite{bordes2013translating,wang2014knowledge};\\ RESCAL~\cite{nickel2011three}; DisMult~\cite{yang2014embedding};\\ MQuatE~\cite{yu2021mquade}; StructurE~\cite{zhang2022structural};\\ TimE~\cite{zhang2021knowledge}; LineaRE~\cite{peng2020lineare};\\ ReflectE~\cite{zhang2022knowledge}; ExpressE~\cite{pavlovic2022expressive}, fill=m5,draw=white,rounded corners=3pt,align=left,inner xsep=1.5mm,draw=black,text width=4.2cm
        ]
        ]
        [Complex Vector Space,part0,fill=m4,draw=black
    [ComplEx~\cite{trouillon2016complex}; RotatE~\cite{sun2019rotate};\\ QuatE~\cite{zhang2019quaternion}; QuatDE~\cite{gao2021quatde};\\ BiQUE~\cite{guo2021bique}; DualE~\cite{cao2021dual};\\DualQuatE~\cite{gao2021dual}; CORE~\cite{ge2022core};\\ HA-RotatE~\cite{wang2021hierarchical},fill=m5,draw=white,rounded corners=3pt,align=left,inner xsep=1.5mm,draw=black,text width=4.2cm]
        ]
        ]
        ]
        [Analytical Structure,part0,fill=m2,draw=black
        [Probability Space,part0,fill=m3,draw=black
        [TransG~\cite{xiao2015transg}; KG2E~\cite{he2015learning},fill=m5,draw=white,draw=black]
        ]
        [Euclidean Space,part0,fill=m3,draw=black
        [FieldE~\cite{nayyeri2021knowledge}; TANGO~\cite{han2021temporal},fill=m5,draw=white,draw=black]
        ]
        ]
        ]
    \end{forest}}
    \caption{The systematic categorisation of KGE models based on three mathematical perspectives.}
    \label{fig:tree_KGE}
\end{figure*}


\subsection{Algebraic Structure}
An algebraic structure is a nonempty set on which one or more finite operations satisfying the axiom are defined~\cite{greub2012linear,wilkinson2013linear}. Some 
representative algebraic structures include \textit{vector space, group} and \textit{ring}.
For example, the vector space $\mathbb{X}$ is an algebraic structure that 
involves many common \textit{binary operations}, such as addition, subtraction and multiplication. 
From the algebraic point of view, most models in machine learning and even knowledge graph embedding involve algebraic operations, such as m$\ddot{o}$buis embedding~\cite{balazevic2019multi,ungar2001hyperbolic,chen2021mobiuse}, group embedding~\cite{ebisu2018toruse,lu2020dense} and so on, which all belong to the methods in the category of algebraic structure.


\subsubsection{Vector Space} 
~\\
The \textbf{Vector Space} is the most widely adopted mathematical space in the field of machine learning, whose definition is provided in Section~\ref{subsec:vector_space}. Since there are two  
convenient algebraic operations defined in vector space: \textit{vector addition} and \textit{scalar multiplication}, it is also widely used by many KGE methods. These methods utilise these operations to project both entities and relations into the same vector space, with the objective of preserving the relational interactions among entities in the vector representation space. As discussed in Section~\ref{subsec:vector_space}, the vector space can be classified as \textit{real vector space} and \textit{complex vector space} based on the scalar field. Following this classification criteria, we classify vector space based KGE methods as real vector space based KGE methods and complex vector space based KGE methods. In what follows, we will describe these two types of KGE methods in detail. 


\paragraph{Real Vector Space.}
One representative KGE methods based on real vector space is TransE~\cite{bordes2013translating}. Given a knowledge graph, it projects all entities and relations into a low-dimensional real vector space $\mathbb{R}^k$. 
Specifically, it directly represents the head entity $h$, the relation $r$ and the tail entity $t$ in a fact triple as embedding vectors $\mathbf{h}$, $\mathbf{r}$ $\mathbf{t}$ $\in \mathbb{R}^k$, respectively. 
If the triple $(h,r,t)$ holds, the embedding of the tail entity $\mathbf{t}$ should be as much as close to the head entity embedding $\mathbf{h}$ 
translated by the relation embedding $\mathbf{r}$, which conforms to the principal: $\mathbf{h} + \mathbf{r} \approx \mathbf{t}$. {It is very simple and effective to directly express the relations between entities by \textit{addition} operation.}
Therefore, the score function in TransE is defined as:
\begin{align}
s(\mathbf{h},\mathbf{r},\mathbf{t}) = - || \mathbf{h} + \mathbf{r} - \mathbf{t}||_{1/2}.    
\end{align}


However, despite 
TransE is simple and effective, it performs poorly when dealing with multi-relation (e.g., 1-to-$N$, $N$-to-1, $N$-to-$N$) data~\cite{toutanova2015observed}. 
For example, suppose that a relation $r_1$ is a $1$-to-$N$ relation, then for a head entity $h_1$, it may have relation $r_1$ with two different tail entities $t_1$ and $t_2$. TransE would enforce these two tail entities to have approximately the same embedding, i.e., $\mathbf{t}_1 \approx \mathbf{t}_2$, which is inaccurate since $t_1$ and $t_2$ are two different entities. The same analysis also applies to $N$-to-$1$ and $N$-to-$N$ relations.
To overcome the problems of TransE in modelling muti-relation data, TransH~\cite{wang2014knowledge} projects the head entity $h$ and tail entity $t$ into the hyperplane where the relationship $r$ resides. Specifically, TransH assumes that each relation embedding $\mathbf{r}$ lies in a different relation-specific hyperplane $\mathbf{w}_r$. In order to measure the plausibility that a triple $(h, r, t)$ holds, the head entity embedding $\mathbf{h}$ and tail entity embedding $\mathbf{t}$ are first projected into the relation-specific hyperplane $\mathbf{w}_r$:
\begin{align}
    \mathbf{h}_{\perp} = \mathbf{h} - \mathbf{w}_{r}^\top \mathbf{h} \mathbf{w}_{r}, 
    \mathbf{t}_{\perp} = \mathbf{t} - \mathbf{w}_{r}^\top \mathbf{t} \mathbf{w}_{r}, 
\end{align}
where $\mathbf{h}_{\perp}$ and $\mathbf{t}_{\perp}$ represent the projected head and tail entity embeddings, respectively. Therefore, the score function of a triple $(h, r,t)$ is defined as:
\begin{align}
    s(\mathbf{h},\mathbf{r},\mathbf{t}) = - ||\mathbf{h}_{\perp} + \mathbf{r} - \mathbf{t}_{\perp} ||_2^2 .
\end{align}

By projecting the entity embedding to relation-specific hyperplane, TransH allows entities to have different embeddings in different relations, which 
ensures that the embedding of $t$ is different even if the head entity or the relation are the same. {From TransH, we can see that \textit{projection} plays a key role, and projection is a very common operation in vector space that is used to establish various connections (See Figure \ref{fig:tensor_project})}.

Despite TransH can effectively handle multi-relations in KGs, it still assumes that the relation embeddings and entity embeddings belong to the same embedding space,
which will limit diversity. However, an entity may have multiple aspects and different relations may focus on different aspects of entities. To address this problem, TransR~\cite{lin2015learning} proposes to embed entities in an entity space $\mathbb{R}^k$ ($\mathbf{h}, \mathbf{t} \in \mathbb{R}^k$), and embed relations in a different relation space $\mathbb{R}^d$ ($\mathbf{r} \in \mathbb{R}^d$, $k$ and $d$ are not necessarily identical).  
The entity embedding and relation embedding are correlated by the relation-specific mapping matrix ($\mathbf{M}_r\in \mathbb{M}^{k \times d}$), which projects entity embedding from entity space to relation space. Therefore, the score function in TransR is defined as:
\begin{align}
    s(\mathbf{h},\mathbf{r},\mathbf{t}) = - ||\mathbf{M}_r \mathbf{h} + \mathbf{r} - \mathbf{M}_r \mathbf{t} ||_2^2 .
\end{align}

RESCAL~\cite{nickel2011three} models the semantic interactions between entities by using bilinear operations with score function:  
\begin{align}
    s(\mathbf{h},\mathbf{r},\mathbf{t}) =  \mathbf{h}^\top \mathbf{M}_r\mathbf{t}  =  \bigg(\sum_i\sum_j[\mathbf{M}_r]_{ij}\cdot\mathbf{h}_i\cdot\mathbf{t}_j \bigg), 
\end{align}
where each relation is defined as a matrix $\mathbf{M}_r\in \mathbb{M}^{d \times d}$, and ${[\mathbf{M}_r]}_{ij}$ denotes the i-th row and j-th column of the matrix $\mathbf{M}_r$. Similar linear models such as DisMult~\cite{yang2014embedding} and HolE~\cite{nickel2016holographic} have proven their great performance on downstream tasks. {As a result, with simple and efficient linear operations, the plausibility of facts can be measured by matching latent semantics of entities and relations (See Figure \ref{fig:semantic_match}).}


Recently, there are some KGE models that learn embeddings in the real vector space as well.    
For example, ReflectE~\cite{zhang2022knowledge} uses reflection transformation to map properties and entities by Householder matrix. LineaRE~\cite{peng2020lineare} interprets a relation as a linear function of entities to capture connectivity patterns. TimE~\cite{zhang2021knowledge} projects entities into non linear time domain to obtain better diversity distribution. It could be found that many mathematical operations of real vector space are worth exploring and applying to KGE.



\paragraph{Complex Vector Space.}
Compared with real vector embedding, complex vector embedding (or complex embedding for brevity) can handle a large variety of binary relations~\cite{trouillon2016complex}, such as symmetric and antisymmetric. 
Complex embedding methods for KGs, which embed entities and relations in the \textbf{Complex Vector Space} (i.e., $\mathbf{h}, \mathbf{r}, \mathbf{t} \in \mathbb{C}^k$), have also been widely studied.
ComplEx~\cite{trouillon2016complex} is the first model to use complex embedding in KGE. 
Specifically, it defines a score function with the Hermitian dot product in complex vector space:
\begin{align}
    s(\mathbf{h},\mathbf{r},\mathbf{t}) = Re \big( \mathbf{h}^\top diag(\mathbf{r})\bar{\mathbf{t}} \big) = Re \bigg(\sum_i\mathbf{r}_i\cdot\mathbf{h}_i\cdot\mathbf{t}_j \bigg), 
\end{align}
where $\bar{\mathbf{t}}$ is the conjugate (see Table \ref{tab:symbol_tem}) of $\mathbf{t}$, $Re(\cdot)$ 
denotes the operation to obtain the real part of a complex number. 
Since the score function is not symmetric any more, the facts
with antisymmetric relations can receive different scores depending on the ordering of entities. Thus ComplEx can effectively capture antisymmetric relations while retaining the efficiency benefits of the dot product. 

Motivated by the Euler's rule $e^{i\theta} = cos\theta + sin\theta$, RotatE~\cite{sun2019rotate} maps entities and relations into the complex vector space and defines each relation as a rotation from the source entity to the target entity with the principal $\mathbf{t} = \mathbf{h} \circ \mathbf{r}$ (where $\circ$ denotes the Hadamard product, i.e., element-wise product). Different entities can be directly modelled through the angular transformation (See Figure ~\ref{fig:angle_trans}) so as to capture some patterns including symmetry, 
antisymmetry, inversion, and composition. 
Compared with ComplEx, QuatE~\cite{zhang2019quaternion} takes advantage of quaternion representations to enable richer and more expressive semantic matching between head and tail entities with the use of Hamilton product ($\otimes$, i.e., quaternion multiplication), where a quaternion $Q$ is defined as $Q = a+b\textbf{i}+c\textbf{j}+d\textbf{k}$ and (\textbf{i, j, k}) are imaginary units satisfying the Hamilton's rule: $\textbf{i}^2 = \textbf{j}^2 = \textbf{k}^2 = \textbf{ijk} = -1$.
BiQUE~\cite{guo2021bique} extends the quaternion system to a more powerful algebraic system called biquaternion $q = (w_r+w_i\textbf{I})+(x_r+x_i\textbf{I})+(y_r+y_i\textbf{I})+(z_r+z_i\textbf{I})$, where $w_r,x_r,y_r,z_r,w_i,x_i,y_i,z_i\in \mathbb{R}$. Also by utilising the Hamilton product of biquaternions, BiQUE imbues itself with a strong geometric interpretation (i.e., the Euclidean/hyperbolic rotation). {In addition to the fact that quaternions have more degree of freedom in the four dimensional space. It is worth mentioning that the interpolation between two quaternions is extremely easy, which helps to establish rotations (See Figure ~\ref{fig:quat_rota}).}
Recently, DualE~\cite{cao2021dual} employs a novel framework which can embrace both translation and rotation operations in the dual quaternion space($Q_{dual} = a + \epsilon b$, where $a$ and $b$ are quaternions while $\epsilon$ is a dual unit). DualQuatE~\cite{gao2021dual} combines the idea of DualE and QuatE. HA-RotatE~\cite{wang2021hierarchical} and CORE~\cite{ge2022core} inherit the structure of RotatE and expand the ability of KGs embedding. 

\begin{figure*}[!t]
    \begin{subfigure}[b]{0.37\textwidth}
        \centering
        \includegraphics[viewport= 250 140 580 450,clip,page=1,width=\linewidth]{alg_operations.pdf}
        \caption{Tensor Projection in Real Vector Space.}
        \label{fig:tensor_project}
    \end{subfigure}
    \begin{subfigure}[b]{0.37\textwidth}
        \centering
        \includegraphics[viewport= 265 140 600 450,clip,page=2,width=\linewidth]{alg_operations.pdf}
        \caption{Similarity Matching in Real Vector Space.}
        \label{fig:semantic_match}
    \end{subfigure}

    \begin{subfigure}[b]{0.37\textwidth}
        \includegraphics[viewport= 265 160 565 450,clip,page=3,width=1\linewidth]{alg_operations.pdf}
        \caption{Angle Transformation in Complex Vector Space.}
        \label{fig:angle_trans}
    \end{subfigure}
    \begin{subfigure}[b]{0.37\textwidth}
        \includegraphics[viewport= 265 160 565 450,clip,page=5,width=1\linewidth]{alg_operations.pdf}
        \caption{3-Dimensional Rotation in Quaternion.}
        \label{fig:quat_rota}
    \end{subfigure}
    \caption{An Illustration of some kind of algebraic operations which can be applied for knowledge graph embedding from algebraic perspective.}
    \label{fig:Illustration_algebric}
\end{figure*}



\subsubsection{Group}
~\\
A \textbf{\textit{group}}~\cite{bourbaki1989commutative,lang2012algebra,rotman2000first} is an algebraic structure composed of a set and an operation, which is ubiquitous in all fields inside and outside mathematics. For example, symmetry groups describe the symmetries of geometry~\cite{robinson1938representations}, and lie groups are used in particle physics~\cite{georgi1974unity}. Because of its unique abstract algebra properties, groups are also widely used in machine learning. In this section, we will start with the definition of groups and then describe the KGE models 
that leverage groups as embedding space. 

\begin{definition}
A (binary) \textbf{\textit{operation}} on a set $\mathbb{G}$ is a function:
\begin{equation*}
    \ast : \mathbb{G} \times \mathbb{G} \rightarrow \mathbb{G}.
\end{equation*}
\end{definition}

\begin{definition}
A \textbf{\textit{group}} is a set $\mathbb{G}$ equipped with an operation $\ast$ and a special element $e\in \mathbb{G}$, called the \textbf{\textit{identity}}, such that: 
\begin{itemize}
    \item[(a)] (Associativity): For every $a,b,c\in \mathbb{G}$, $(a\ast b) \ast c= a\ast (b\ast c)$;
    \item[(b)] (Existence of identity): $e\ast a = a$ for all $a\in \mathbb{G}$;
    \item[(c)] (Existence of inverses): for every $a\in \mathbb{G}$, there is $a'\in \mathbb{G}$ with $a'\ast a = e$.
\end{itemize}
\end{definition}

To tackle TransE’s regularisation problem, TorusE~\cite{ebisu2018toruse}
proposes to embed KGEs into a special algebraic structure---Torus. A torus is an Abelian Lie group, which is derived from the vector space through the nature projection $\pi : \mathbb{R}^n \rightarrow T^n, x\mapsto\left [ x \right ]$ ($T^n$ denotes quotient space~\cite{ebisu2018toruse}). With the help of torus, the model never diverge unlimitedly. 

DihEdral~\cite{xu2019relation} is the first attempt to employ finite non-Abelian group in KG embedding to account for relation compositions. 
Since the elements in a dihedral group are well constructed by rotation and reflection operations, and the multiplication between elements can be Abelian or non-Abelian, DihEdral is capable to capture all desired properties: (skew-)symmetry, inversion and (non-) Abelian composition. DensE~\cite{lu2020dense} decomposes a relation operator into a SO(3) group-based (SO(3): Special Orthogonal Group in 3 dimensions) rotation as well as a scaling transformation. NagE~\cite{yang2020nage} proves for the first time that the group algebraic structure is significant for designing relational embedding models. \textit{Specifically, the definition of group can naturally satisfy the basic properties (e.g., inversion, composition) of knowledge graphs}, which means the  group-based models should have 
the great potential to deal with KGE tasks. Other recent models based on group structure such as ModulE ~\cite{chai2022module} consider both entity and relation as group element so as to achieve state-of-the-art performance. 

\subsubsection{Ring}
~\\
In mathematics, \textbf{\textit{rings}}~\cite{bourbaki1989commutative,lang2012algebra,rotman2000first} are algebraic structures that generalised fields. Commutative rings are one of the main branches of ring theory. Examples include the set of integers with addition and multiplication, and the set of polynomials with the same operations. 
Ring theory was later proved useful in geometry and analysis ~\cite{faith2012algebra}. 
In this section, we will first introduce the definition of ring, and then discuss the KGE models that use ring as embedding space.

\begin{definition}
A \textbf{\textit{ring}} is a set $\mathbb{S}$ equipped with two binary operations: $+$ (\textit{addition}) and $\cdot$ (\textit{multiplication}) which satisfy the following axioms,
\begin{itemize}
    \item[(a)]$\mathbb{S}$ is an Abelian group under addition, meaning that: 
          \begin{itemize}
              \item[$\bullet$] (Associative): $(a+b)+c = a+(b+c)$ for all $a,b,c\in R$;
              \item[$\bullet$] (Commutative): $a+b = b+a$ for all $a,b\in \mathbb{S}$;
              \item[$\bullet$] (Additive identity): There is an element $0$ in $\mathbb{S}$ such that $a+0=a$ for all $a\in \mathbb{S}$;
              \item[$\bullet$] (Additive inverse): For each $a$ in $\mathbb{S}$ there exists $-a$ such that $a+(-a) = 0$.
          \end{itemize}
    \item[(b)] $\mathbb{S}$ is a monoid under multiplication, meaning that:
          \begin{itemize}
              \item[$\bullet$] (Associative): $(a\cdot b)\cdot c = a\cdot(b\cdot c)$ for all $a,b,c\in \mathbb{S}$;
              \item[$\bullet$] (Multiplicative identity): There is an element 1 in $\mathbb{S}$ such that $a\cdot 1 = a $ and $1\cdot a = a$ for all $a\in \mathbb{S}$.
          \end{itemize}
    \item[(c)] Multiplication is distributive with respect to addition, meaning that:
          \begin{itemize}
              \item[$\bullet$] (Left distributivity): $a\cdot (b+c) = (a\cdot b)+(a\cdot c)$ for all $a,b,c\in \mathbb{S}$;
              \item[$\bullet$] (Right distributivity): $(b+c)\cdot a = (b\cdot a)+(c\cdot a)$ for all $a,b,c\in \mathbb{S}$.
          \end{itemize}
\end{itemize}
\end{definition}

M$\ddot{o}$biusE~\cite{chen2021mobiuse} extends KGE to manifold-based embedding, in which the entities and relations are embedded to the surface of M$\ddot{o}$bius ring. With the scoring function ($\textit{dist} (\textbf{h}\oplus\textbf{r},\textbf{t})$, where $\oplus$ and \textit{dist} represent addition and distance function specially defined on M$\ddot{o}$bis ring, respectively), M$\ddot{o}$biusE has much more expressiveness and flexibility than TorusE~\cite{ebisu2018toruse} due to the extra properties on ring. As M$\ddot{o}$biusE subsumes TorusE, it naturally inherits all the desired properties of TorusE including symmetric/antisymmetry, inversion, and composition. \textit{It is worth mentioning that the M$\ddot{o}$bius band is a non-oriented surface and the concept of clockwise and counterclockwise cannot be clearly defined, which may be helpful for certain tasks that are closely related to orientation.}

\subsection{Geometric Structure}
A geometric structure~\cite{goldman1988varieties} on a manifold is a complete Riemannian metric which is locally homogeneous (i.e., any two points have isometric neighbourhoods). Although there are few detailed definitions of geometric structures, here we
focus more on the knowledge graph embedding models built on different geometric models/spaces and analyse them in detail from three geometric perspectives: Euclidean Geometry, Hyperbolic Geometry and Spherical Geometry. We summarise the operations of these three geometries in Table \ref{table:summaryofgeometry}.

\begin{table*}[!t]
    \centering
    \footnotesize
    \renewcommand{\arraystretch}{1}
    \setlength\tabcolsep{6pt}
    \caption{Summary of operations in Euclidean, Hyperboloid and Spherical $(K<0)$~\cite{nayyeri2021knowledge,lou2020differentiating,weber2018curvature}.}
    \resizebox*{!}{0.38\columnwidth}{
    \begin{tabular}{p{1.9cm}p{3.2cm}p{4.2cm}p{3.6cm}}
        \toprule
                                     & \textbf{Euclidean}                            & \textbf{Hyperboloid}                                                                                                                                              & \textbf{Spherical}                                                                               \\
        \midrule
        Manifold $\mathcal{M} $      & $\mathbb{R}^n$                                & $\mathbb{H}^n_K = \left \{ x\in \mathbb{R}^{n+1}:\left \langle x,x \right \rangle = \frac{1}{K} \right \}$                                                        & $\mathbb{S}^n_K = \left \{ x\in \mathbb{R}^{n+1}:\left \langle x,x \right \rangle = 1 \right \}$ \\ \midrule
        Distance $d(x,y)$            & $\left \langle \sqrt{x-y,x-y} \right \rangle$ & $\frac{1}{\left | K \right |}cosh^{-1}(K\left \langle x,y \right \rangle)$                                                                                        & $cos^{-1}(\left \langle x,y \right \rangle)$                                                     \\ \midrule
        Exponential map $exp^K_x(v)$ & x+v                                           & $cosh(\sqrt{\left | K \right |}\left \| v \right \|)x+v\frac{sinh(\sqrt{\left | K \right |}\left \| v \right \|)}{\sqrt{\left | K \right |}\left \| v \right \|}$ & $cos(\left \| v \right \|)x+sin(\left \| v \right \|)\frac{v}{\left \| v \right \|}$             \\ \midrule
        Curvature $\mathfrak{C} $    & 0                                             & $<0    $                                                                                                                                                          & $>0 $                                                                                            \\ \midrule
        Sum of angles $\mathfrak{A}$ & $\pi$                                         & $<\pi$                                                                                                                                                            & $>\pi$                                                                                           \\ \midrule
        Illustration                 & \begin{minipage}[c]{0.2\columnwidth}
            \centering
            \raisebox{-.5\height}{\includegraphics[viewport= 200 150 700 450,clip,page=1,width=\linewidth]{summary_geo_op.pdf}}
        \end{minipage}                    & \begin{minipage}[c]{0.2\columnwidth}
            \centering
            \raisebox{-.5\height}{\includegraphics[viewport= 150 150 650 450,clip,page=2,width=\linewidth]{summary_geo_op.pdf}}
        \end{minipage}                                                                                                                                        & \begin{minipage}[c]{0.2\columnwidth}
            \centering
            \raisebox{-.5\height}{\includegraphics[viewport= 200 150 700 450,clip,page=3,width=\linewidth]{summary_geo_op.pdf}}
        \end{minipage}                                                                       \\
        \bottomrule
    \end{tabular}}
    \label{table:summaryofgeometry}

\end{table*}

\subsubsection{Euclidean Geometry}
~\\
Euclidean geometry is the study of geometrical shapes (plane and solid) and figures based on different axioms and theorems. It is basically introduced for flat surfaces or plane surfaces. This part of geometry was employed by the Greek mathematician \textit{Euclid}, who has also described it in his book, \textit{Elements}~\cite{fitzpatrick2007euclid}. Geometry is derived from the Greek words `geo' which means earth and `metrein' which means `to measure'. 

Euclidean geometry deals with things like points, lines, angles, squares, triangles, and other shapes. Based on different coordinate systems, in this part, we will divide KGE models built in Euclidean geometry into four part: \textit{Cartesian Coordinate, Polar Coordinate} and \textit{Spherical Coordinate}. It is worth noting that: Euclidean geometry under geometric structure and vector space under algebraic structure are very similar and have some 
common concepts. 
This is because geometric features and algebraic features are usually closely related. But in this section we will emphasise the advantages of KGE models from the geometric perspective.


\paragraph{Cartesian Coordinate.}
Most of the translation-based models are based on the common Cartesian coordinates. For example, TransE follows the principle: $\mathbf{h} + \mathbf{r} \approx \mathbf{t}$, i.e., the vector of head entity add the relation vector is equal to the tail vector through a translation. These three vectors are connected from head to tail, forming a closed path in Cartesian coordinates.
RotatE shows that it's ingenious to define each relation as a rotation from the source entity to the target entity. For example, two relations $\mathbf{r}_1$ and $\mathbf{r}_2$ are inverse if and only if their embeddings are conjugate: $\mathbf{r}_1 = \bar{\mathbf{r}}_2$ ($\mathbf{r}_1:(cos(\theta_1),sin(\theta_1)),\mathbf{r}_2:(cos(\theta_2),sin(\theta_2))$), which means they are symmetric about the real axis (i.e., $sin(\theta_1) = -sin(\theta_2)$). All relation patterns can be clearly illustrated by geometric transformation in Cartesian coordinates. 
Moreover, QuatE~\cite{zhang2019quaternion} extends complex space to quaternion where there are two planes of rotations.
Inspired by RotatE, Tang et al.~\cite{tang2020orthogonal} extend RotatE from 2D complex domain to high dimensional space with orthogonal transformations, which preserves the ability of modelling different patterns while achieving better performance.
However, it remains challenging for RotatE to handle complex relations. To mitigate this problem, PairRE~\cite{chao2021pairre} uses paired vectors for each relation $\mathbf{r} = \left [ \mathbf{r}^H,\mathbf{r}^T \right ]$, where the value of $\mathbf{r}^H$ and $\mathbf{r}^T$ can be changed 
to fit the complex relations. 
Through the scoring function, relation vectors can project entities to arbitrary positions inside a unit circle lying on the Cartesian coordinates. 
Further analysis also proves that PairRE can capture subrelation in addition to those that RotatE can model~\cite{sun2019rotate}. 

Other Cartesian coordinate based KGE models have also emerged recently. For example, TripleRE \cite{yu2022triplere} inherits PairRE’s projection part and builds translation part by its own way. InterHT~\cite{wang2022interht} merges the information from tail entity to head entity representation. TranS~\cite{zhang2022trans} is proposed to efficiently capture single relations. CompoundE~\cite{ge2022compounde} successfully embeds KGs by leveraging three fundamental Euclidean geometric operations. The above geometric models based on Cartesian coordinates are illustrated in Figure~\ref{fig:Illustration_in_car}. \textit{In conclusion, \textit{translation, rotation, reflection} and \textit{scaling} are the commonly used geometric transformations in KGE, which have been proved to have excellent properties.} 

\begin{figure*}[!t]
\begin{subfigure}[b]{0.33\textwidth}
    \centering
    \includegraphics[viewport= 250 150 600 450,clip,page=1,width=\linewidth]{geo_illu.pdf}
    \caption{TransE}
    \label{fig:car_transe}
\end{subfigure}
\begin{subfigure}[b]{0.33\textwidth}
    \centering
    \includegraphics[viewport= 280 200 650 450,clip,page=2,width=1.2\linewidth,scale=1.5]{geo_illu.pdf}
    \caption{RotatE}
    \label{fig:car_rotate}
\end{subfigure}    
\begin{subfigure}[b]{0.3\textwidth}
    \centering
    \includegraphics[viewport= 250 150 600 450,clip,page=3,width=\linewidth]{geo_illu.pdf}
    \caption{QuatE}
    \label{fig:car_quate}
\end{subfigure}    

\begin{subfigure}[b]{0.32\textwidth}
    \centering
    \includegraphics[viewport= 250 165 600 450,clip,page=4,width=\linewidth]{geo_illu.pdf}
    \caption{PairRE}
    \label{fig:car_pairre}
\end{subfigure}
\begin{subfigure}[b]{0.32\textwidth}
    \centering
    \includegraphics[viewport= 250 165 600 450,clip,page=5,width=\linewidth]{geo_illu.pdf}
    \caption{CompoundE}
    \label{fig:car_compounde}
\end{subfigure}    
\begin{subfigure}[b]{0.32\textwidth}
    \centering
    \includegraphics[viewport= 250 165 600 450,clip,page=6,width=\linewidth]{geo_illu.pdf}
    \caption{TranS}
    \label{fig:car_trans}
\end{subfigure}    
\caption{Some geometric illustrations of KGE models in Cartesian Coordinate. ``$\leftrightarrow$`` illustrates the distance (scoring) function.}
\label{fig:Illustration_in_car}
\end{figure*}



\paragraph{Polar Coordinate.}
In order to naturally reflect hierarchy of KGs, HAKE~\cite{zhang2020learning} models entities and relations in the polar coordinate system. The radial coordinate aims to model entities ($\mathbf{h}_m,\mathbf{t}_m,\mathbf{r}_m \in \mathbb{R}^k$) at different level, and the angular coordinate aims to distinguish entities ($\mathbf{h}_p,\mathbf{t}_p,\mathbf{r}_p \in \left [ 0,2\pi \right )$ ) at the same level of hierarchy. Combining the modules and phase information, HAKE significantly outperforms the SOTA hierarchy-based models while also completely capturing relational patterns.
Although H$^2$E~\cite{wang2021knowledge}'s representation space belongs to hyperbolic geometric, it also uses modulus and phase information to embed entities--- which named Hyperbolic Polar Embedding. Modulus embedding models the inter-level hierarchy and Phase Embedding models the intra-level hierarchy.
Recently, HBE~\cite{pan2021hyperbolic} is also a hyperbolic geometry-based KGE model but with polar coordinate system by Mobius multiplication and Mobius addition in extended Poincare ball.


\paragraph{Spherical Coordinate.}
{
STKE~\cite{wang2022stke} represents entities and relations in a spherical coordinate system, but for temporal knowledge graph. Each entity contains the radial part $r$, the azimuth part $\theta$ and the polar part $\varphi$. With the help of spherical coordinate, STKE treats temporal changes as scaling and rotation of entity embeddings, which can dynamically distinguish different time-constrained entities.}

\subsubsection{Hyperbolic Geometry}
~\\
MuRP~\cite{balazevic2019multi} is proposed to embed hierarchical multi-relational data in the Pointcar$\acute{e}$ ball of hyperbolic space, where the Pointcar$\acute{e}$ ball is defined as a $d$-dimensional manifold with the form: $\mathbb{B}^d_c  = \left \{ x\in \mathbb{R}^d:c\left \| x \right \| ^2<1, c>0\right \}$. Compared with Euclidean space, Hyperbolic surface can be seen to have more space to represent entities by considering a hypersphere with increasing radius~\cite{sala2018representation}, 
so that MuRP outperforms Euclidean KGE models and achieves competitive performance, especially in hierarchical datasets.
Instead of learning in a hyperbolic space with fixed curvature as in MuRP, ATTH~\cite{chami2020low} leverages expressive hyperbolic isometries to simultaneously capture logical patterns and hierarchies. For each relation (e.g., rotation, reflection), it learns a specific absolute curvature $c_r$ to avoid precision errors. 
Based on ATTH, models in different conditions are also proposed (i.e., ATTE, ROTE/H, REFE/H).
As also mentioned in section about polar coordinate, HBE~\cite{pan2021hyperbolic} employs an extended Pointcar$\acute{e}$ ball to capture hierarchical structures using polar coordinate system (Pointcar$\acute{e}$ disk) to solve the boundary constraints which might happen in conventional Pointcar$\acute{e}$ ball.
Sun et. al.~\cite{sun2020knowledge} also represent KG embedding in a hyperbolic space by HyperKA , but firstly incorporate hyperbolic translation embedding with graph neural network (GNN).
However, with the constant emergence of hyperbolic models, Kai Wang et. al.~\cite{wang2021hyperbolic} could not help asking: \textit{Is Hyperbolic Geometry necessary?} Hyperbolic space is more computationally expensive, so does it outweigh the cost? To tackle this issue, RotL and Rot2L are proposed to simplify the hyperbolic operation in RotH. 
By defining Flexible Addition, RotL can reduce the computation complexity of RotH~\cite{chami2020low} and save over $50\%$ training time. However, it should not be ignored that knowledge graphs have multiple mixed relations, and excessive focus on hierarchical information often neglects capturing other information. Hyperbolic hierarchical transformations are introduced in HypKGE~\cite{zheng2022hyperbolic} to extract hierarchies. MuRMP~\cite{wang2021mixed} utilises the mix-curvature model combined with GNN to better capture intrinsic heterogeneous structure in the KGs. UltraE~\cite{xiong2022ultrahyperbolic} considers an ultrahyperbolic manifold to overcome the non-hierarchical embedding problems.


\subsubsection{Spherical Geometry}
~\\
Before introducing the spherical geometry models, in order to avoid the confusion of the concepts of spherical coordinate system and spherical geometry, we first make a specific distinction between them. The spherical coordinate system mentioned above can be consider as a tool to describe points' positions naturally. And the basic concepts behind it are points and straight lines based on the Euclidean geometry. However, in spherical geometry~\cite{harris1998spherical}, the basic concepts are point and great circles where any two lines meet in two points, and there are also no parallel lines.

TransC~\cite{lv2018differentiating}, in order to differentiate instances and concepts, encodes each concept in knowledge graph as a sphere (e.g. concept $c_i$ is encoded as a sphere $s_i(\textbf{p}_i,m_i)$, where $\textbf{p}_i$ denotes the centre, $m_i$ denotes the radii) and each instance as a vector in the same semantic space. For example, for distinguishing the relations between concepts and sub-concepts (i.e. $\textit{subClassOf}$ ), TransC construct several possible positions (e.g. inclusion, intersection, separation)  between two concept spheres for different conditions.
Comparing with TransC, HypersphereE~\cite{dong2021hypersphere} extends the sphere into hypersphere so as to not neglect the uncertainty of instances.
Another special model named ManifoldE~\cite{xiao2015one}, which expands point-wised embedding to manifold-based embedding. Instead of adopting previous translational-based principle $\textbf{h}+\textbf{r} = \textbf{t}$, ManifoldE employs manifold-based principle $\mathbf{M}(\textbf{h,r,t}) = D_r^2$ ($\mathbf{M}$ is the manifold function) in sphere and hyperplane respectively. In Sphere condition, entity which is always considered as a point in classic models is now extended to a whole high dimensional sphere. Through this mean, ManifoldE avoids much noise so as to best distinguish true facts. {The smoothness of the sphere surface makes the embedding very flexible, and there can be countless mappings from the centre of the circle to the surface. }
As we know that KGs usually contain rich types of structure such as hierarchical and cyclic typed structures. Embedding KGs in single curvature space, such as Euclidean or hyperbolic space, overlooks the intrinsic heterogeneous structures of KGs, and therefore cannot accurately capture their structures. Hence, M$^2$GNN, as a mixed-curvature model, is designed to address this issue. The scoring function is defined as $\phi_{\mathbb{P}^{d_o,d_h,d_s}_{K_o,K_h,K_s}(e_h,r,e_t)}$, where $\mathbb{P}$ denotes the mixed-curvature space, $d_o,d_h,d_s$ are the dimensions of  of the component spaces of Euclidean, hyperbolic, and spherical, $K_{o/h/s}$ are the corresponding curvature of spaces.


\subsection{Analytic Structure}
An analytical structure is usually thought of a structure of having a measure. For instance, metric (or distance) is well defined in Euclidean space so that we can integrate, differentiate and other analytical operations. Similarly, in the probability space, probabilistic measure is defined, so it can consider as a analytical structure. In this section, we will deeply dig into the KGE models by dividing them into two main spaces: Probability Space and Euclidean Space.

\subsubsection{Probability Space}
~\\
To the best of our knowledge, KG2E~\cite{he2015learning} is the first density-based KGE model which represents each entity/relation by a multi-dimensional Gaussian distribution $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma} )$ in probability space, where the mean vector $\boldsymbol{\mu}$ indicates its position and the covariance metric $\boldsymbol{\Sigma}$ indicates the corresponding (un)certainty which impacts on others. In addition, two similar methods based on KL-divergence and expected likelihood, are proposed to inspect the difference of asymmetric and symmetric respectively.
Previous models have formally consider the issue of multiple relation semantics in KGs. However, the traditional translational-based models always assign only one vector for one vector, ignoring the fact that a relation may have multiple meanings. Thus, TransG~\cite{xiao2015transg} is proposed by leveraging a Bayesian non-parametric infinite mixture model to handle multiple relation semantics by generating multiple translation components for a relation. In TransG, entities are generated by a certain Gaussian distribution, where the $m-$th component translation vector of relation $r$ is represented as: 
$\boldsymbol{\mu}_{r,m} = \mathbf{t} - \mathbf{h} \sim \mathcal{N}(\boldsymbol{\mu}_t - \boldsymbol{\mu}_h,(\sigma_h^2+\sigma_t^2)E)$. 
Through this process, TransG would automatically select the best match between h, t and r. Other probability-based models such as DBKGE~\cite{liao2021learning} infers dynamic embedding for temporal KGs by Gaussian representation. {Above techniques prove that Gaussian embedding can not only obtain unstructured information, but also capture extra uncertain information~\cite{xu2021understanding}.}

\subsubsection{Euclidean Space}
~\\
In order to improve the structure preservation capabilities of KGE models, Nayyeri et.al~\cite{nayyeri2021knowledge} propose a novel KGE model named FieldE which employs ordinary differential equations (ODEs) for embedding KGs into a Euclidean space. Each entity is represented by a vector in $\mathbb{R}^n$ denoted by $\textbf{e}(t)$ and each relation is represented as a vector field $f_{\theta_{r}}$ on a Riemannian Manifold, where $\textbf{e}(t)$ lies on a trajectory (continuous) on the manifold $\mathcal{M}$ solving the ODE: $\frac{d\textbf{e}(t)}{dt} = f_{\theta_{r}}(\textbf{e}(t))$. Hence, FieldE could capture the continuity of changes in the embedding space and describe the underlying geometry by nature. ODE method can also be used in temporal KGs, in which TANGO~\cite{han2021temporal} is proposed to learn continuous-time representations of entities and relations dynamically by Neural ODE method~\cite{chen2018neural}. {In addition to continuous analysis, other analytical perspectives such as derivability, differentiability, integrability are also worth exploring in KGEs.}


\section{Downstream Tasks of Knowledge Graph Embedding}
\label{sec:applications}

After introducing a systematic review of existing KGE models from the perspective of mathematical structure, this section focuses on KGE-based downstream tasks.
We highlight some important and popular applications which are usually employed to evaluate the performance of embedding models. After summarising and comparing the performance of some KGE models, we analyse the advantages and disadvantages of KGE models from different spatial perspectives.

In what follows, we first describe the process of \textbf{link prediction}, a fundamental task in KGE domain, 
and focus on one popular task: \textbf{Hierarchy Acquisition} in the link prediction scenario. In addition, we analyse and discuss the task of \textbf{Pattern Inference}. The strengths of mathematical features to the above tasks are summarised.



\begin{table*}[!tb]
    \caption{The link prediction results on the WN18RR and FB15k-237 datasets, which are classified by its geometric space. The best scores of 32-dimensional models are in \textbf{bold}, the best average scores are \textbf{coloured}. TransE, ComplEx, QuatE, RotatE, MuRE, MuRMP results are taken from ~\cite{xiong2022ultrahyperbolic}. TuckER, RefE, RotE, AttE, MuRP, RefH, RotH, AttH, Rot2L results are taken from ~\cite{wang2021hyperbolic}. MuRMP is a mix-curvature model. UltraE is an ultrahyperbolic based model, in which its best results (when $q=6$) are chosen. $^*$ denotes: DFieldE ($\mathbb{S}$) and MuRS results are not obtained in the case of low dimensions (d = 32), but in high dimensions. Hence the average score of spherical-based model is not comparable. $^\triangledown$ denotes: GIE is a geometric interactive model and its result is also in high dimensional conditions, which is not added to the average calculation.}
    \small
    \centering
    \label{tab:2}
    \begin{tabular}{llcccccc}
        \toprule
        \multirow{2}*{\textbf{Type}} & \multirow{2}*{\textbf{Method}} & \multicolumn{3}{c}{\textbf{FB15K237}} & \multicolumn{3}{c}{\textbf{WN18RR}}  
        \\
        \cline{3-8}
        ~                            & ~                               & \textbf{MRR}                           & \textbf{Hits@10}                    & \textbf{Hits@1}         & \textbf{MRR}            & \textbf{Hits@10}        & \textbf{Hits@1}         \\
        \hline
        \multirow{13}*{\makecell[l]{Euclidean-based                                                                                                                                                                                                            \\ Models}} 
        & TransE & 0.295 & 0.466 & 0.210 & 0.366 & 0.515 & 0.274\\
        & ComplEx & 0.287 & 0.456 & 0.203 & 0.421 & 0.476 & 0.391\\
        & QuatE & 0.293 & 0.460 & 0.212 & 0.421 & 0.467 & 0.396\\
        & RotatE & 0.290 & 0.458 & 0.208 & 0.387 & 0.491 & 0.330\\
        & TuckER & 0.306 & 0.475 & 0.223 & 0.428 & 0.474 & 0.401 \\
        & MuRE & 0.313 & 0.489 & 0.226 & 0.458 & 0.525 & 0.421  \\
        & HAKE & 0.296 & 0.463 & 0.212 & 0.416 & 0.467 & 0.389 \\
        & RefE  & 0.302 & 0.474 & 0.216 & 0.455 & 0.521 & 0.419 \\
        & RotE & 0.307  & 0.482 & 0.220 & 0.463 & 0.529 & 0.426 \\
        & AttE & 0.311 & 0.488 & 0.223 & 0.456 & 0.526 & 0.419 \\
        & Rot2L & 0.326 & 0.503 & 0.237 & 0.475 & 0.554 & 0.434\\
        & EucHKGE & 0.330 & 0.510 & 0.240 & 0.475 & 0.556 & 0.432\\
        \cline{2-8}
        & \textit{Avg\_score}& 0.305 & 0.477 & 0.219 & 0.435 & 0.508 & 0.394 \\
        \hline
        \multirow{7}*{\makecell[l]{Hyperbolic-based                                                                                                                                                                                                           \\ Models}} & MuRP & 0.323 & 0.501 & 0.235 & 0.465 & 0.544 & 0.420 \\
        & RefH & 0.312 & 0.489 & 0.224 & 0.447& 0.518 & 0.408   \\
        & RotH & 0.314 & 0.497 & 0.223 & 0.472 & 0.553& 0.428  \\
        & AttH & 0.324 & 0.501 & 0.236 & 0.466& 0.551 & 0.419  \\
        & HypHKGE & 0.330 & 0.510 & 0.240 & 0.475 & 0.556 & 0.432 \\
        &DFieldE ($\mathbb{P}$) & 0.330 & 0.510 & \textbf{0.250} & 0.480 & \textbf{0.570} & \textbf{0.440}\\
        \cline{2-8}
        ~                            & \cellcolor{m4}\textit{Avg\_score}             & \cellcolor{m4}0.322                & \cellcolor{m4}0.501             & \cellcolor{m4}0.235 & \cellcolor{m4}0.468 & \cellcolor{m4}0.549 & \cellcolor{m4}0.425 \\
        \hline
        \multirow{2}*{\makecell[l]{Spherical-based                                                                                                                                                                                                            \\Models}} 
        & DFieldE ($\mathbb{S}$)                         & 0.360                          & 0.550                               & 0.270         & -        & -           & -          \\
        ~                            & MuRS & 0.338 & 0.525 & 0.249 & 0.454 & 0.550 & 0.432  \\
        \cline{2-8}
        ~ & \textit{Avg\_score$^*$}             & 0.349 & 0.538 & 0.260 & 0.454 & 0.550 & 0.432\\
        \hline
        \multirow{3}*{\makecell[l]{Mixed            Models}} & MuRMP                         & 0.319          & 0.502                             & 0.232           & 0.470       & 0.547            & 0.426           \\
        ~ & UltraE & \textbf{0.338} & \textbf{0.514} & 0.247 & \textbf{0.483} & 0.555 & 0.425\\
        ~                            & GIE$^\triangledown$ & 0.362 & 0.552 & 0.271 & 0.491 & 0.575 & 0.452  \\
        \cline{2-8}
        ~ & \cellcolor{m1}\textit{Avg\_score} & \cellcolor{m1}0.329  & \cellcolor{m1}0.508 & \cellcolor{m1}0.240                  & \cellcolor{m1}0.477   & \cellcolor{m1}0.551   &\cellcolor{m1}0.426                   \\
    \bottomrule        
    \end{tabular}
\end{table*}

\subsection{Link Prediction}
{
Link prediction aims to predict the existence of edges (triples) in knowledge graphs, which is a fundamental task since many existing KGs have missing facts or incorrect edges~\cite{nickel2015review}. 
In particular, the task of link prediction is often formulated as predicting missing entity in an incomplete fact triple, i.e., predicting head entity $h$ in $(?, r, t)$, or tail entity $t$ in $(h, r, ?)$, where $(?, r, t)$ and $(h, r, ?)$ denote fact triples with missing entities. For example, given a triple \textit{<?, subclass, COVID-19}>, the goal is to predict the superclass of COVID-19. 
Therefore, link prediction is also referred to as knowledge graph completion~\cite{nickel2015review}, entity prediction~\cite{lin2015modeling} or entity ranking~\cite{bordes2014semantic}. 
}


Given learned entity and relation embeddings with KGE methods, link prediction is carried out through a ranking procedure. Specifically, in order to predict the tail entity of an incomplete fact triple $(h, r, ?)$, we take each entity $t'$ in the KG as an candidate answer and calculate the plausibility of the triple $(h,r,t')$, which is achieved by calculating the score function of the employed KGE method. For example, for KGE methods that learn KG embeddings in Euclidean space, the translation-based score function $s(\mathbf{h}, \mathbf{r}, \mathbf{t}) = -||\mathbf{h} + \mathbf{r} - \mathbf{t}||_{1/2}$ is often used to assign scores for triples. In contrast, for KGE methods that learn embeddings in complex space, the rotation-based score function $s(\mathbf{h}, \mathbf{r}, \mathbf{t}) = -||\mathbf{h} \circ \mathbf{r} - \mathbf{t}||^2$ is often utilised. 
After obtaining the scores of candidate answers, we can rank those candidate entities in the descending order of their scores, and select the highest ranked entity as the prediction result. A similar procedure can also be used to predict the missing head entity $h$ in $(?, r, t)$. For evaluation, a common practice is to record the ranks of correct answers in previous ranked list and leverage those ranks to calculate the evaluation metrics. 


\subsection{Hierarchy Acquisition}
Nowadays, more and more KGE models not only aim to obtain SOTA in link prediction task, but also pay special attention to whether they can capture hierarchy properties. 
{One main reason why these works focus on capturing hierarchy structures} is that: \textbf{Multi-relational knowledge graphs often exhibit multiple simultaneous hierarchies}~\cite{dettmers2018convolutional,toutanova2015observed}. However, conventional models (e.g., TransE, RotatE ) merely emphasis on capturing these hierarchies. Therefore, we focus on KGE models from the view of mathematical space, 
{aiming to find} the most appropriate space to capture multi-layered information. {In this} part, we first introduce what hierarchy exactly is, and then draw our conclusion by comparing the performance of KGE models in different spaces on hierarchy-contained datasets. 

\paragraph{What is Hierarchy?} Semantic hierarchy is a ubiquitous property in knowledge graphs. Some relations can induce various hierarchical structure. For instance, \textit{chair} is at a higher lever than \textit{armchair, fighting\_chair} under the relation \textit{hypernym}, and \textit{armchair, fighting\_chair} are parent nodes to their part: \textit{backrest, leg} with the relation \textit{has\_part}. Such hierarchies can be treated as "tree-like" structures intuitively.


\paragraph{Results.} We summarise the performance of KGE models in link prediction task on two hierarchical datasets: WN18RR \cite{dettmers2018convolutional} and FB15K-237 \cite{toutanova2015observed}, where
the curvature metric $\xi_G$ (The lower the metric $\xi_\mathcal{G}$ is, the more hierarchical the knowledge graph is~\cite{gu2018learning}.) of two datasets are -2.54 and -0.65, respectively. 
We summarise the results of KGE models with an embedding dimension of $32$ in Table~\ref{tab:2}. In the table, KGE models are categorised into four groups based on their embedding space: Euclidean-based models, Hyperbolic-based models, Spherical-based models, and Mixed models which learn KG embeddings in multiple mathematical spaces. 
When comparing the results of different groups of KGE models (i.e., the \textit{Avg\_score} of different groups), we found that 
Hyperbolic models significantly outperform Euclidean baselines on WN18RR and FB15K-237 
(coloured in orange). 
The average scores of the Spherical models are not comparable since the model results are obtained in different conditions. 
Therefore, benefited from \textbf{Hyperbolic Space}, hyperbolic models often perform well in rich-hierarchy datasets since they can better capture multi-level 
information. 
From a mathematical point of view, hyperbolic space has the 
natural advantage of capturing hierarchies. 
Nevertheless, this does not mean that hyperbolic-based models are necessarily superior to other spaces, since some Euclidean-based models can also achieve high performance through ingenious tricks, such as ReflectE~\cite{zhang2022knowledge}, CompoundE~\cite{ge2022compounde}.
In addition, the results of the mixed models (coloured in purple) which learn embeddings in multiple spaces are more outstanding, which means that better performance can be achieved by using multiple geometry simultaneously with a suitable hybrid method. 





\subsection{Patterns inference}

Another popular task is about exploring the pattern inference capability since large-scale KGs always exhibit various types of relationships. As described in ~\cite{bordes2013irreflexive}, a excellent model should be able to learn all combinations of these properties: (a) \textbf{symmetry} (e.g., marriage, is\_similar\_to). (b) \textbf{antisymmetry} (e.g., father\_of). (c) \textbf{inversion} (e.g., hypernym and hyponym). (d) \textbf{composition} (e.g., my mother's father is my grandpa.).
However, looking back at the development of previous KGE models, it was a difficult process to build a model that could capture all of the above attributes simultaneously. In this section, we will mainly analyse the advantages of models focusing on inferring relational patterns, and summarise the key factors of capturing those properties.\\
~\\
First, we give these four important patterns' formal definitions as belows~\cite{sun2019rotate}:
\begin{definition}
    A relation ${\mathtt{r}}$ is \textbf{symmetric} if $\forall x, y$
    \begin{equation*}
        \mathbf{r}(x, y) \Rightarrow \mathbf{r}(y, x).
    \end{equation*}
    A clause with such form is a \textbf{symmetry} pattern.
\end{definition}

\begin{definition}
    A relation $\mathbf{r}$ is \textbf{antisymmetric} if $\forall x, y$
    \begin{equation*}
        \mathbf{r}(x, y) \Rightarrow \neg \mathbf{r}(y,x).
    \end{equation*}
    A clause with such form is a \textbf{antisymmetry} pattern.
\end{definition}

\begin{definition}
    Relation $\mathbf{r}_1$ is \textbf{inverse} to relation $\mathbf{r}_2$ if $\forall x, y$
    \begin{equation*}
        \mathbf{r}_2 (x, y) \Rightarrow \mathbf{r}_1 (y, x).
    \end{equation*}
    A clause with such form is a \textbf{inversion} pattern.
\end{definition}

\begin{definition}
    Relation $\mathbf{r}_1$ is \textbf{composed} of relation $\mathbf{r}_2$ and relation $\mathbf{r}_3$ if $\forall x, y, z$
    \begin{equation*}
        \mathbf{r}_2 (x, y) \wedge \mathbf{r}_3 (y, z) \Rightarrow \mathbf{r}_1 (x, z).
    \end{equation*}
    A clause with such form is a \textbf{composition} pattern.
\end{definition}

Next, we analyse specific operations in different spaces to explain how existing models can infer and model those patterns. Table.\ref{tab:pattern_inf} shows the patterns inference capabilities of some main KGE works, and their corresponding spaces and operators.

\paragraph{Addition.} The KGE models with addition operation as its core are often found among distance-based models such as SE~\cite{bordes2011learning}, TransE~\cite{bordes2013translating} and TransE's variants. The characteristic of addition is that it is easy to establish the connection between vectors, and the complexity is very low.
In SE~\cite{bordes2011learning}, the scoring function is too sketchy to capture any of these patterns; By defining $h + r \approx t$, TransE~\cite{bordes2013translating} makes good use of vector addition to establish the relationship between relations and entities, and can capture three patterns except symmetry.
Since then, a large number of translation-based models(e.g., TransH~\cite{wang2014knowledge}, TransR~\cite{lin2015learning}, etc.) still retain addition operation but carry out special operation $f_r(\cdot)$ (mostly matrix multiplication based on r). What remains unchanged is that the core of operation is still Addition, that is, to a large extent, addition corresponds to the concept of ``translation''.
However, with the added projection parameters, these models are unable to encode inverse and composition. Although they have made progress in dealing with complex relations, the Trans's variants are a step back in terms of modelling patterns --- a drawback of the simplicity of addition.

\paragraph{Product.} The term ``product'' refers to the results of one or more multiplications. Most existing SOTA models use product operation, such as ComplEx~\cite{trouillon2016complex} and RotatE~\cite{sun2019rotate}. Products in different spaces have different properties to help the KGE models infer more patterns. Here we mainly divide the product operations into the following groups and analyse them in detail.

\begin{itemize}
    \item \textit{Dot Product.} The dot product may be defined algebraically or geometrically. With algebraic definition, inner product is a way to multiply vectors together, with the result of this multiplication being a scalar. With geometric definition, the notions of length and angles can be defined by means of the dot product. In KGE, the fundamental purpose of product is to expect to establish complex relations between relations and entities through multiplication rather than simple addition. For example, the scoring function in DistMult~\cite{yang2014embedding} is defined as $f_r(h,t)=\textbf{h}^{\top}diag(\textbf{r})\textbf{t}=\sum_i\left [ \textbf{r} \right ]_i\cdot\left [ \textbf{h} \right ]_i\cdot\left [ \textbf{t} \right ]_i$, we can see that each score in the summation is a direct multiplication of $\textbf{h}_i\textbf{r}_i\textbf{t}_i$; similar in ComplEx~\cite{trouillon2016complex}, the scoring function is extended to complex space. Note that they are both RESCAL~\cite{nickel2011three}'s extensions, and RESCAL was originally built for implicit semantic matching by factorisation, which is one of the latent advantages of dot products.
    \item \textit{Hadmard Product.} Hadamard Product (also known as element-wise product) is a binary operation by which the elements corresponding to same row and columns of given vectors/matrices are multiplied together to form a new vector/matrix. The difference between dot product and Hadamard product operationally is the aggregation by summation. The dot product of two vectors gives only a scalar number while the Hadamard product of two vectors gives a complete vector, which preserves a large amount of transformed information of KGs. As in RotatE~\cite{sun2019rotate}, through the principle $\textbf{t} = \textbf{h} \circ \textbf{r}$, \textbf{r} becomes a element-wise rotation from the head entity to the tail entity; In HAKE~\cite{zhang2020learning}, each $\textbf{r}_i$ is regarded as a scaling transformation between two moduli; Cross interaction operations are also applied by utilising Hadamard product in CrossE~\cite{zhang2019interaction}. Similarly uses of Hadamard product also appear in ComplEx~\cite{trouillon2016complex}, PairRE~\cite{chao2021pairre}, etc. In general, the Hadamard product is closely related to the concept of ``rotation''. Besides, the Hadamard product appears in lossy compression algorithms such as JPEG, can also be used for describing NN as LSTM~\cite{hochreiter1997long}, GRU~\cite{cho2014learning} or enhancing, suppressing or masking image regions.
    \item \textit{Other Products.} Other subsequent product operations are mostly extended based on the existing advantages of Dot/Hadamard product. Zhang et.al~\cite{zhang2019quaternion} believed that latent inter-dependencies between all components are aptly captured with Hamilton product, encouraging a more compact interaction between entities and relations; What's more advanced than RotatE is that DualE~\cite{cao2021dual} adopts quaternion inner product operation to model both translation and rotation, which improves the capability of inferring three important patterns.
\end{itemize}

\paragraph{Other Operations.} 
Other operations are mostly those with exclusive rules under some special conditions, but show strengths in some specific tasks (e.g., multi-relation tasks). For example, simple matching in SME~\cite{bordes2014semantic}/RESCAL~\cite{nickel2011three}, although it only uses simple linear/bilinear matching, it opens a new field for semantic matching KGE model; Circular correlation in HolE~\cite{nickel2016holographic}, which makes compressions of pairwise tensor product to enhance its efficiency; Orthogonal Transform~\cite{nickel2016holographic} that aims to Unleash the original potential of RotatE into higher dimensions; Mobius addition and attention in ATTH~\cite{chami2020low} are utilised to represent relations as parameterised geometric operations that directly map to logical properties.


\begin{table*}[!t]
    \caption{The pattern inference abilities of several models by using different operators in certain spaces. SE, TransE, TransR, DisMult, ComplEx and RotatE results are taken from ~\cite{sun2019rotate}. Other results come from origin papers. $\mathbb{R}$ represents the Euclidean space. $\mathbb{R}_p$ represents the Euclidean space with polar coordinate system~\cite{zhang2020learning}. $\mathbb{C}$ represents the complex vector space. $\mathbb{G}$ represents the group. $\mathbb{H}$ represents the hyperbolic space. $\mathbb{UH}$ represents the ultra-hyperbolic space. $\mathbb{Q}$ represents the quaternion vector space. $\mathbb{BQ}$ and $\mathbb{DQ}$ represents the biquaternion vector space and dual quaternion vector space, respectively, both of them are special case of $\mathbb{Q}$.}
    \label{tab:pattern_inf}
    \begin{center}
    \resizebox*{!}{0.55\columnwidth}{
        \begin{tabular}{lcccccc}
            \toprule
            \multirow{2}{*}{\textbf{Method}} & \multirow{2}{*}{\textbf{Manifold}}
                                             & \multirow{2}{*}{\textbf{Operator}}
                                             & \multicolumn{4}{c}{\textbf{Relation Patterns}}                                                                                                                  \\
                                             &                                                 &                                                                  & $Sym$  & $Asym$           & $Inv$  & $Comp$ \\ \hline
            SE                               & $\mathbb{R}$                                       & Addition($+$)                                                    & \ding{55} & \ding{55}           & \ding{55} & \ding{55} \\ \hline
            TransE                           & $\mathbb{R}$                                       & Addition($+$)                                                    & \ding{55} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            TransR                           & $\mathbb{R}$                                       & Addition($+$)                                                    & \ding{51} & \ding{51}           & \ding{55} & \ding{55} \\ \hline
            DistMult                         & $\mathbb{R}$                                       & Inner Product($\left \langle \cdot  \right \rangle$)             & \ding{51} & \ding{55}           & \ding{55} & \ding{55} \\ \hline
            HolE                          & $\mathbb{R}$                                       & Circular Coorelation($\star$)                                    & \ding{51} & \ding{51}           & \ding{51} & \ding{55} \\ \hline
            HAKE                             & $\mathbb{R}_{p}$                                & Hadmard Product($\circ$)                                         & \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            PairRE                           & $\mathbb{R}$                                       & Hadmard Product($\circ$)                                         & \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            ComplEx                          & $\mathbb{C}$                              & Hadmard Product($\circ$)                                         & \ding{51} & \ding{51}           & \ding{51} & \ding{55} \\ \hline
            RotatE                           & $\mathbb{C}$                              & Hadmard Product($\circ$)                                         & \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            ExpressivE                           & $\mathbb{R}$                                       & Hadmard Product($\circ$)                                         & \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            OTE                              & $\mathbb{R}$                                       & Orthogonal Transform($\phi$)                                     & \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            QuatE                            & $\mathbb{Q}$                           & Harmilton Product($\otimes$)                                     & \ding{51} & \ding{51}           & \ding{51} & \ding{55} \\ \hline
            BiQUE                            & $\mathbb{BQ}$                               & Harmilton Product($\otimes$)                                     & \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            DualE                            & $\mathbb{DQ}$                            & Dual Quaternion Product($\left \langle \otimes  \right \rangle$) & \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            Rotate3D & $\mathbb{R}$ & Rotational Product($\odot$)& \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            ATTH                             & $\mathbb{H}$                                      & M$\ddot{o}$bius addition($\oplus^c$)                  & \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \hline
            DihEdral                         & $\mathbb{G}$                                           & Matrix Product($\odot^*$)                                          & \ding{51} & \ding{51} & \ding{51} & \ding{51} \\ \hline
            UltraE & $\mathbb{UH}$ & Manh.-like Distance($d_\mathbb{U}(\cdot)$) & \ding{51} & \ding{51}           & \ding{51} & \ding{51} \\ \bottomrule
            
        \end{tabular}}
    \end{center}

\end{table*}



\subsection{Knowledge Infusion To Enhance Other Domain Applications}
Apart from the aforementioned applications that are appropriate for KG embeddings, there are other wider fields where KG representation learning could be infused to enhance.
Knowledge graph based questions answering (KGQA) is a fundamental, but still challenging task. It recognises the user's question input in order to obtain the accurate answers composed of KG entities. Existing methods include semantic parsing models~\cite{yih2015semantic,liang2016neural,lan2020query}, reinforcement learning~\cite{xiong2017deeppath,qiu2020stepwise}, and so on. 
Knowledge reasoning is a process of using known knowledge to infer new knowledge. Researchers always use machine learning methods~\cite{wang2021kepler,zhang2019ernie,wang2019kgat} to infer potential relations between entity pairs and identify erroneous knowledge based on existing data automatically. There are lots of other external applications based on knowledge graph embedding are still worth exploring, such as Recommendation System~\cite{bellini2017auto,wang2019multi,zhao2017meta}, Information Retrieval~\cite{raviv2016document,ensan2017document,liu2015latent} and other specific domain (e.g. Cyber Security, Biomedicine, etc.).



\section{Future Directions}
\label{sec:futuredirec}
In this section, we summarise the some core advantages of mathematical structures and propose several noteworthy future directions, which we hope to inspire the readers to construct more flexible and adaptable KGE algorithms in the future.

\subsection{Algebraic Operation}
The ability of knowledge representation is often directly related to the properties of model's algebraic operations. The algebraic advantages can sometimes compensate for the model's shortcomings, which allow the model to handle complex and non-intuitive relationships, such as 1-N, N-1 and N-M relations. One future direction is to construct a powerful operation that is able to deal with multiple tasks simultaneously. Recent works such as  CompoundE~\cite{ge2022compounde} have started using combination operations to better handle complex relation types in different conditions.
Another promising method is to build a unified algebraic KGE framework~\cite{hayashi2017equivalence,liu2017analogical} for better understanding the nature of algebraic operations. Nevertheless, algebraically inspired KGE models still have many perspectives to ponder.

\subsection{Geometric Embedding}
Several current KGE methods are inspired by some useful geometric properties. For instance, ManifoldE~\cite{xiao2015one} utilise spherical embedding to extend the entity representation from one point to a sphere. ATTH~\cite{chami2020low} provides a hyperbolic embedding for capturing hierarchies based on hyperbolic isometries. However, most knowledge graphs embrace complicated structures which result in that these existing methods perform poorly when modelling the knowledge graph with hybrid structures~\cite{cao2022geometry}. Hence, it is crucial to establish a more comprehensive geometric embedding methods. Some works provide mixture-curvature embedding~\cite{wang2021mixed} or geometry interactive~\cite{cao2022geometry} KGE to enhance the ability of geometric embedding. Another meaningful research direction is how to build geometric knowledge learning models with low energy, since complex geometric embedding always needs more training time, especially on large-scale KGs. Wang et al.~\cite{wang2021hyperbolic} provides a fewer parameters ``RotH-like~\cite{chami2020low}'' model. However, it is still a worthwhile direction to build a comprehensive geometric embedding model while keeping low energy consumption.

\subsection{Analytical Optimisation}
Analytical properties of knowledge representation are of great importance but poorly studied. Any of these properties, such as convergence, stability, complexity and derivability etc., will greatly affect the model performance on downstream tasks. For example, the embedding space has to be differentiable, or the model could not be trained by gradient descent. Several works focus on solving certain analytical properties in KGE. TorusE~\cite{ebisu2018toruse} started by analysing convergence, by choosing a compact Lie group as an embedding space, the model never diverge unlimited and regularisation is no longer required. In addition, it is worth paying special attention to build some highly efficient KGE model through optimisation analytically. Peng et al.~\cite{peng2021highly} adopt segmented embeddings divide the entity representation space into multiple independent sub-spaces explicitly.
Nevertheless, there still has a long way to go to deal with analytical optimisations, so as to construct more reliable KGE models in the future.


\subsection{Others}
It is worth noting that the above three mathematical perspectives are not independent but complementary to each other. Therefore, one possible research direction is to build  multi-mathematical perspective models. FieldE~\cite{nayyeri2021knowledge} employs the neural ODE that represents relations as trajectories connecting neighbouring nodes in the graph, which can continuously (analytical view) represent the underlying geometries including Euclidean, Poincare Ball, Hyperboloid and Spherical (geometric view). Moreover, it is of great significance to note the above summarised implications are \textbf{not limited to the KGE domain, but indeed can be extended to any domains}. Recent works strongly bears this out. Hyperbolic space can be utilised in generative models~\cite{ding2021deep} geometrically, neural order differential equations (NODE) are considered as another breakthrough in deep learning with strong abilities in supervised learning,
sequential diffusion~\cite{jiang2021learning,kang2021stable,thorpe2021grand++}, etc. 
It is essential to exploit the properties of mathematical space to explore more possibilities not only in KGE but also in other fields.




\section{Conclusion}
Over the recent years, the rapid growth of Knowledge Graph Embedding has been witnessed. As we have seen in our analysis of the current state of the art, most of the existing works tend to  utilise the properties of different spaces to build knowledge representations. Hence, we present a comprehensive review of the state-of-the-art KGE techniques from the perspective of representation spaces, and discuss the existing KGE methods according to three mathematical perspectives: (1) Algebraic perspective, (2) Geometric perspective, and (3) Analytical perspective. In particular, we first introduce the basic concept of mathematical spaces and the relationship between them. We then introduce and compare the strengths of KGE techniques from different mathematical perspectives in terms of embedding space, scoring function, optimisation, etc. At the same time, some unique characteristics of specific spaces are summarised through the experimental results. We also discuss some future directions such as efficient geometric embedding, rational optimisation methods. We believe that it is vital to analysis the model from the mathematical point of view, to determine more precisely what causes the possible failures in each method and find the appropriate mathematics tools to solve them. This article takes the field of KGE as an example to show the powerful role of mathematical spaces and their properties, thereby inspiring more strong mathematical modelling in more fields. 




\bibliographystyle{acm}
