\section{Introduction}
Algorithms in Reinforcement Learning (RL) control with function approximation and discontinuous action selection strategies are empirically known to exhibit complex phenomena  such as i.) instability, ii.) policy oscillation and chattering, iii.) multiple attractors, and iv.) worst policy convergence \citep{bertsekas1996neuro, bertsekas2011approximate, de2000existence, young2020understanding}. This holds true even in the simplest of cases such as Q-learning and SARSA(0) with linear function  approximation and $\epsilon$-greedy exploration \citep{gordon1996chattering, gordon2000reinforcement}. Accordingly, a  mathematical framework to explain such behaviors has been a long standing open question \citep[Problem~1]{sutton1999open}. In the related but simpler prediction problem, analyses have relied on the Stochastic Approximation (SA) and Ordinary Differential Equation (ODE) viewpoint. The same approach in control settings, however, has had limited success due to its heavy dependence on the continuity of the underlying vector field \citep[Section~2.1]{borkar2009stochastic}, \citep[Section~4.1]{benaim1999dynamics}, \citep[Section~1.5]{benveniste2012adaptive}. Specifically, all  results that apply to these settings  require stringent smoothness properties on policy improvement  \citep{perkins2002convergent, melo2008analysis, chen2019finite, zou2019finite}. In this work, we propose a novel framework based on Stochastic Recursive Inclusions (SRIs) and Differential Inclusions (DIs) to directly work with discontinuous driving functions.

To elaborate on the need for a novel framework, note that RL schemes  have been viewed as SA algorithms so far, i.e., as update rules in $\mathbb{R}^d$ having the form
\begin{equation}
    \label{e:SA.update.rule}
    \theta_{n + 1} = \theta_n + \alpha_n[h(\theta_n) + M_{n + 1}], \qquad n \geq 0,
\end{equation}
where $h$ is a suitable driving function, $\alpha_n$ is the stepsize, and $M_{n + 1}$ is noise. Further, when $h$ is Lipschitz continuous, the ODE method has been used to show that the limiting dynamics of \eqref{e:SA.update.rule} is completely governed by the differential equation $\dot{\theta}(t) = h(\theta(t))$ \citep{benaim1999dynamics, borkar2009stochastic}. Approximate RL algorithms in general can also be cast as \eqref{e:SA.update.rule}. However, if the resulting $h$ functions turn out to be discontinuous, as is the case with approximate Q-learning and SARSA(0) under $\epsilon$-greedy exploration, one is unable to apply the ODE method. Our proposed framework circumvents this by, in a sense, jointly considering all possible local update directions as a set and then viewing the resultant dynamics from a DI perspective. We remark that this has been a standard approach to handle a discontinuous driving function in classical control theory.

Formally, a DI \citep{aubin2012differential, smirnov2022introduction} is a relation of the form
\begin{equation}
    \label{e:DI}
    \dot{\theta}(t) \in h(\theta(t)),
\end{equation}
where $h$ is a \emph{set-valued} map which associates with any point $\theta \in \mathbb{R}^d$ a set $h(\theta) \subseteq \mathbb{R}^d.$ An ODE is a special case of a DI where $h(\theta)$ is a \textit{singleton}. If $h$ is \textit{Marchaud}, i.e., smooth in a set-valued sense (see Appendix~\ref{s:Proof.Main.Result} for the precise definition), the DI in \eqref{e:DI} is guaranteed to have at least one continuous solution for every initial point  \cite[Lemma~5.3]{borkar2009stochastic}. However, unlike in ODEs, solutions of a DI need not be unique. An SRI \cite{benaim2005stochastic, benaim2006stochastic, borkar2009stochastic}, on the other hand, is an update rule in $\mathbb{R}^d$ of the form
\begin{equation}
\label{e:stochastic.recursive.inclusion}
    \theta_{n + 1} = \theta_n + \alpha_n[y_n + M_{n + 1}], \qquad n \geq 0,
\end{equation}
where $y_n \in h(\theta_n) \subseteq \mathbb{R}^d$ for some set-valued function $h,$ while $\alpha_n$ and $M_{n + 1}$ are as before. Clearly, any SA scheme is also an SRI. If $h$ is Marchaud and satisfies some stability conditions and the sequences $(\alpha_n)$ and $(M_{n + 1})$ satisfy the standard Robbins-Monro and martingale-difference conditions, respectively, then \cite{borkar2009stochastic} and \cite{ramaswamy2017generalization} (see Theorem~\ref{thm:SRI.stability.convergence} for the details) showed that the sequence $(\theta_n)$ is stable, i.e., almost surely bounded, and converges to the suitable invariant sets of the DI in \eqref{e:DI}.

Our proposed approach, a key contribution of this work,  is to view approximate RL algorithms as SRIs and then appeal to the above result for understanding their limiting dynamics. Applying this perspective, our main result states that the asymptotic behaviors of linear  Q-learning and SARSA(0) under $\epsilon$-greedy exploration are completely governed by their respective limiting DIs. The limiting \textit{deterministic} DI thus provides the first pathway to systematically identify and explain the whole range of limiting phenomena that a general approximate method can exhibit, answering the question posed by \cite{sutton1999open}. As illustrations of this approach, we show in Section \ref{s:Numerical.Examples} how aspects of an algorithm's limiting DI govern whether it possesses unique or multiple attractors, and whether it is susceptible to chattering in policy or parameter space. We are also able to identify a hitherto unseen `sliding mode' phenomenon. Separately, we demonstrate that distinct algorithms for the same RL problem can have qualitatively different limiting DIs and, accordingly, different limiting behaviors.

\textbf{Related Work}: We provide here a brief survey of the literature on approximate RL algorithms.

Several works identify and report a variety of complex behaviors for approximate algorithms. These include the classic example of instability for Q-learning with linear function approximation \citep{baird1995residual} and the phenomenon of chattering in SARSA(0) \citep{gordon1996chattering} and its approach to a bounded region \citep{gordon2000reinforcement}. \citet{de2000existence} show how approximate value iteration may not possess any fixed points, while \citet{bertsekas2011approximate} argues that approximate policy iteration schemes may generally be prone to policy oscillations, chattering, and convergence to poor solutions. More recently, \citet{young2020understanding} show experimentally how a range of approximate RL algorithms involving value estimation and greedification exhibit pathological behaviors such as policy oscillation, multiple fixed points, and consistent convergence to the `worst' policy. Our framework, along with illustrations in Section~\ref{s:Numerical.Examples}, helps explain many of these limiting phenomena in a rigorous fashion.

On the theoretical front, while global convergence results exist for Q-learning and SARSA for the tabular (no approximation) setting (see e.g., \citep{bertsekas1996neuro}), analysis with function approximation has been much more challenging. The difficulty in RL control primarily stems from the changing nature of policies (i.e., sampling distributions) which renders the iterations nonlinear (even with linear function approximation).
One prominent stream of work focuses on extensions of linear SA theory to analyze nonlinear SA updates such as the ones in Q-learning with linear function approximation \citep{melo2008analysis, carvalho2020new,chen2019finite} and nonlinear (neural) function approximation \citep{fan2020theoretical,xu2020finite}. However, all these works {\em hold the behavior policy fixed} to mitigate the effect of changing sampling distributions. Another notable work is that of \citet{lee2020unified}, which uses ideas from switched system theory to analyze nonlinear ODEs. This is perhaps similar in spirit to our endeavor as switched systems are also useful for analyzing discontinuous dynamics.

Another set of analyses apply SA techniques to study SARSA(0) with incrementally changing policies  \citep{perkins2002convergent, perkins2002existence, melo2008analysis, zhang2022chattering,zou2019finite}.
However, a restrictive assumption used here is that the policy improvement operator is Lipschitz-continuous, e.g., softmax,  which ensures the limiting ODE is `very smooth.' Note that $\epsilon$-greedy policies do not fall in this class, which we are able to handle with the DI framework.

\section{Setup and Main Result}
\label{s:Setup.Main.Result}
We begin this section by recalling the RL problem of finding the optimal policy with linear function approximation. Thereafter, we discuss a generic learning scheme for solving this problem that includes as special cases the linear\footnote{Henceforth, we will use linear Q-learning (resp. linear SARSA(0)) to refer to Q-learning (resp. SARSA(0)) with linear function approximation.} Q-learning and SARSA(0) algorithms with $\epsilon$-greedy exploration. Finally, we state our main result that describes the asymptotic behavior of this generic algorithm and, hence, of these approximate methods.

Let $\Delta(U)$ denote the set of probability measures on a set $U.$ At the heart of RL, we have a Markov Decision Process (MDP) represented by the tuple $(\mathcal{S}, \mathcal{A}, \gamma, \mathbb{P}, r),$ where $\mathcal{S}$ denotes a finite state space, $\mathcal{A}$ represents a finite action space equipped with a total order, $\gamma$ is the discount factor, and $\mathbb{P}: \mathcal{S} \times \mathcal{A} \to \Delta(\mathcal{S})$ and $r: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$ are deterministic functions such that $\mathbb{P}(s, a)(s') \equiv \mathbb{P}(s'|s, a)$ specifies the probability of moving from state $s$ to $s'$ under action $a,$ while $r(s, a, s')$ is the one-step reward obtained in this transition. One of the main goals in RL control  is to  find $Q_* \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}|},$ the optimal Q-value function, associated with this MDP.

In practice, however, it is often the case that the state and action spaces are large. In such situations, to reduce the search space dimension,  it is common to try and find an {\em approximation} to $Q_*,$ instead of $Q_*$ itself. In this work, we focus on linear function approximation. That is, we presume we are given a feature matrix $\Phi \in \mathbb{R}^{|\mathcal{S}||\mathcal{A}| \times d}$ and the goal is to find a $\theta_* \in \mathbb{R}^d$ such that $Q_* \approx \Phi \theta_*.$

Two algorithms to find such a $\theta_*$ are linear Q-learning and SARSA(0) with $\epsilon$-greedy exploration. While these are widely used in practice, a rigorous understanding of their limiting behavior is missing in the literature and this is what our main result addresses. To enable a joint investigation, we first propose a generic learning scheme that includes the above two methods as special cases.

We need a few notations for stating this generic scheme. Let $\phi^T(s, a),$ where $^T$ implies transpose, denote the $(s, a)$-th row of $\Phi.$ Further, assume that $\Phi$ and $r$ satisfy the following condition.
{
\renewcommand{\theenumi}{$\mathcal{B}_\arabic{enumi}$}
\begin{enumerate}[leftmargin=*]
    \item \label{a:phi.reward} $\Phi$ has full column rank. Further, there exist constants $K_r, K_\phi  \geq 0$ such that $\|\phi(s, a)\| \leq K_\phi$ and $|r(s, a, s')| \leq K_r$ for all $s, s' \in \mathcal{S}$ and $a \in \mathcal{A}.$
\end{enumerate}
}
Next, let $\epsilon \in (0, 1]$ and $\pi^\epsilon_n: \mathcal{S} \to \Delta(\mathcal{A})$ the $\epsilon$-greedy policy at time $n \geq 0.$ That is, if $\theta_n$ is the estimate of our generic algorithm at time $n,$ then let
\begin{equation}
    \pi^\epsilon_n(a'|s) =
    %
    \begin{cases}
    %
    1 - \epsilon (1 - 1/|\mathcal{A}|)
    %
    & \text{ if } a' = \underset{a}{\arg \max} \, \phi^T (s, a) \theta_n, \\[1ex]

    \epsilon/|\mathcal{A}| & \text{ otherwise. }
    %
    \end{cases}
\end{equation}
We presume that $\arg \max$ breaks ties between actions that have the same value of $\phi^T(s, a) \theta_n$ using the total order on $\mathcal{A}.$

Since the state and action spaces are finite, the number of $\epsilon$-greedy policies is also finite.
Suppose these policies satisfy the following condition.

{
\renewcommand{\theenumi}{$\mathcal{B}_\arabic{enumi}$}
\begin{enumerate}[leftmargin=*]
    \setcounter{enumi}{1}

    \item The Markov chains induced by each of the finitely many $\epsilon$-greedy policies are ergodic and, hence, have their own unique stationary distributions.
\end{enumerate}
}

Finally, let $\epsilon' \in [0, 1].$  Then, given some initial estimate $\theta_0 \in \mathbb{R}^d,$ the exact update rule of our generic algorithm at time $n \geq 0$ is
\begin{equation}
\label{e:generic.update}
\begin{aligned}
    \theta_{n + 1} = {} & \theta_n + \alpha_n \delta_n \phi(s_n, a_n)  \\
    %
    \delta_n = {} & r(s_n, a_n, s'_n) + \gamma \phi^T(s'_{n}, a_n')\theta_n - \phi^T(s_n, a_n) \theta_n,
\end{aligned}
\end{equation}
where $\alpha_n$ is the stepsize, $s_n$ is the current state at time $n$ which we presume\footnote{This  assumption is standard \cite{dalal2018finite, liu2015finite, sutton2008convergent, tsitsiklis1997analysis} and is needed to ensure that the update rule only has martingale noise. Otherwise, the update rule will additionally have Markovian noise, the analysis of which is beyond the scope of this paper. Also, in practice, experience replay is used to mimic sampling from the stationary distribution.} is independently sampled from the stationary distribution corresponding to the Markov chain\footnote{At any time $t \geq 0,$ this Markov chain moves from state $s$ to $s'$ with probability $\sum_{a}\pi^\epsilon_n(a|s) \mathbb{P}(s'|s, a).$} induced by $\pi^\epsilon_n,$ $a_n \sim \pi^\epsilon_n(\cdot|s_n),$ while $s_n' \sim \mathbb{P}(\cdot| s_n, a_n)$ and $a_n' \sim \pi^{\epsilon'}_n(\cdot|s_n').$ Clearly, \eqref{e:generic.update} with $\epsilon' = 0$ and $\epsilon' = \epsilon$ corresponds to the linear Q-learning and SARSA(0) algorithms with $\epsilon$-greedy exploration, respectively.

With regards to the stepsize sequence $(\alpha_n)_{n \geq 0},$ we presume that it satisfies the standard Robbins-Monro condition, i.e.,
{
\renewcommand{\theenumi}{$\mathcal{B}_\arabic{enumi}$}
\begin{enumerate}[leftmargin=*]
    \setcounter{enumi}{2}

    \item \label{a:stepsize} $\sum_{n \geq 0} \alpha_n = \infty$ and $\sum_{n \geq 0} \alpha_n^2 < \infty.$

\end{enumerate}
}

We need a few more notations before we can state our main result. For $\mathbf{a} \equiv (\mathbf{a}(s))_s \in \mathcal{A}^{\mathcal{S}},$ let the associated \textit{greedy region} $\mathcal{P}_{\mathbf{a}} := \{\theta \in \mathbb{R}^d: \forall s \in \mathcal{S}, \mathbf{a}(s) = \arg\max_a
\phi^T(s, a) \theta\},$ where $\arg \max$ breaks ties using the total order. In other words, at each $\theta \in \mathcal{P}_\mathbf{a},$ $\mathbf{a}$ is the greedy policy corresponding to the  Q-value funtion $\Phi \theta.$ Indeed, for some $\mathbf{a},$ it is possible that $\mathcal{P}_\mathbf{a} = \emptyset.$ Nonetheless, $\{\mathcal{P}_\mathbf{a} : \mathbf{a} \in \mathcal{A}^{\mathcal{S}}\}$ partitions $\mathbb{R}^d,$ i.e., for any $\theta \in \mathbb{R}^d,$ there exists a unique $\mathbf{a}$ such that $\theta \in \mathcal{P}_\mathbf{a}.$ Separately, let $\pi^\epsilon_\mathbf{a}$ (resp. $\pi^{\epsilon'}_\mathbf{a}$)
be the $\epsilon$-greedy (resp. $\epsilon'$-greedy) policy associated with the policy $\mathbf{a}.$
It is easy to see that $\pi^\epsilon_n = \pi^\epsilon_\mathbf{a}$ and $\pi^{\epsilon'}_n = \pi^{\epsilon'}_\mathbf{a}$ whenever $\theta_n \in \mathcal{P}_\mathbf{a}.$

Next, let $d^{\epsilon}_\mathbf{a}$ denote the stationary distribution associated with the Markov chain induced by $\pi^\epsilon_\mathbf{a}.$ Further, let
\begin{equation}
\label{e:b_ba.defn}
    b_{\mathbf{a}} = \mathbb{E}[\phi(s_n, a_n) r(s_n, a_n, s_n')] =  \Phi^T D^{\epsilon}_{\mathbf{a}} \mathbf{r}
\end{equation}
and
\begin{equation}
\label{e:A_ba.defn}
    A_\mathbf{a} = \mathbb{E}[\phi(s_n, a_n) \phi^T(s_n, a_n) - \gamma \phi(s_n, a_n) \phi^T(s_n', a_n')] =  \Phi^T D^{\epsilon}_{\mathbf{a}} (\mathbb{I} - \gamma P^{\epsilon'}_\mathbf{a}) \Phi,
\end{equation}
where $D^{\epsilon}_\mathbf{a}$ is the diagonal matrix of size $|\mathcal{S}||\mathcal{A}| \times |\mathcal{S}||\mathcal{A}|$  whose $(s, a)$-th diagonal entry is $d^{\epsilon}_\mathbf{a}(s) \pi^\epsilon_\mathbf{a}(a|s),$ $\mathbf{r}$ is the $|\mathcal{S}||\mathcal{A}|$-dimensional vector whose $(s, a)$-th coordinate is $    \mathbf{r}(s, a) = \sum_{s' \in \mathcal{S}} \mathbb{P}(s'|s, a) r(s, a, s'),$  while $P^{\epsilon'}_\mathbf{a}$ is the matrix of size $|\mathcal{S}||\mathcal{A}| \times |\mathcal{S}||\mathcal{A}|$ such that $P^{\epsilon'}_\mathbf{a}((s, a), (s', a')) = \mathbb{P}(s'|s, a) \pi^{\epsilon'}_\mathbf{a}(a'| s').$ Assume that the matrices $A_\mathbf{a}, \mathbf{a} \in \mathcal{A}^\mathcal{S},$  satisfy the following condition.
{
\renewcommand{\theenumi}{$\mathcal{B}_\arabic{enumi}$}
\begin{enumerate}[leftmargin=*]
    \setcounter{enumi}{3}

    \item \label{a:PD.matrix} For all $\mathbf{a} \in \mathcal{A}^\mathcal{S},$ $A_\mathbf{a}$ is positive definite\footnote{The matrix $A_\mathbf{a}$ need not be symmetric.}, i.e., $\theta^T A_\mathbf{a} \theta > 0 \, \forall \theta \in \mathbb{R}^d.$
\end{enumerate}
}

Finally, for $\theta \in \mathbb{R}^d,$ let $f: \mathbb{R}^d \to \mathbb{R}^d$ and the set-valued $h: \mathbb{R}^d \to 2^{\mathbb{R}^d}$ be given by
\begin{equation}
\label{e:base.driving.function}
    f(\theta) := \sum_{\mathbf{a} \in \mathcal{A}^{\mathcal{S}}} (b_\mathbf{a} - A_\mathbf{a} \theta) \mathds{1} [\theta \in \mathcal{P}_\mathbf{a}] \qquad \text{ and } \qquad h(\theta) = \bigcap_{\delta > 0} \overline{\textnormal{co}} (f(B(\theta, \delta))),
\end{equation}
where $\mathds{1}$ denotes the indicator function, $\overline{\textnormal{co}}$ is the convex closure, and $B(\theta, \delta)$ is the open ball of radius $\delta$ centered at $\theta.$ As we show later in the proof outline for Theorem~\ref{thm:main.result}, $f(\theta)$ denotes our generic algorithm's expected update direction at time $n$ if $\theta_n = \theta,$ while $h(\theta)$ is the collection of all possible local update directions in the vicinity of $\theta.$ Now, our main result basically states that the limiting behavior of \eqref{e:generic.update} is completely governed by the DI
\begin{equation}
\label{e:lim.DI}
    \dot{\theta}(t) \in h(\theta(t));
\end{equation}
the convex closure in the definition of $h$ ensures the above DI admits continuous solutions.

In relation to \eqref{e:lim.DI}, we will say a set $\Gamma \subset \mathbb{R}^d$ is \textit{invariant} if, for every $x \in \Gamma,$ there is \textit{some} solution trajectory $(\theta(t))_{t \in (-\infty, \infty)}$ of \eqref{e:lim.DI} with $\theta(0) = x$ that lies entirely in $\Gamma.$
An invariant set $\Gamma$ is additionally said to be
\textit{internally chain transitive} if it is compact and, for $x, y \in \Gamma,$ $\epsilon > 0,$ and $T > 0,$ there exist $m \geq 1$ and points $x_0 = x, x_1, \ldots, x_{m - 1}, x_m = y$ in $\Gamma$ such that one of the solution trajectories of \eqref{e:lim.DI} initiated at $x_i$ meets the $\epsilon$-neighborhood of $x_{i + 1}$ for $0 \leq i < m$ after a time that is equal or larger than $T.$ Such characterizations are useful to restrict the possible choices for limiting sets of an SRI.

\begin{theorem}[\textbf{Main Theorem}: SRI-DI connection]
\label{thm:main.result}
Suppose $\mathcal{B}_1, \ldots, \mathcal{B}_4$ hold. Then, almost surely, the iterates of \eqref{e:generic.update} are stable, i.e., $\|\theta_n\| < \infty,$ and converge to a (possibly sample path dependent) closed connected internally chain transitive invariant set of the differential inclusion in \eqref{e:lim.DI}.
\end{theorem}
\begin{proof}[Proof Outline]
The statement follows by verifying all the technical assumptions of   Theorem~\ref{thm:SRI.stability.convergence}, originally proved in \citep{borkar2009stochastic, ramaswamy2017generalization}, that discusses the stability and convergence of SRIs and their connections to DIs. The full details are given in the Appendix~\ref{s:Proof.Main.Result}. In the discussion below, we only show how the  generic algorithm in \eqref{e:generic.update} can be viewed as a SRI whose limiting DI is \eqref{e:lim.DI}.

For $n \geq 0,$ let $\mathcal{F}_n := \sigma(\theta_0, s_0, a_0, s'_0,\ldots, s_{n - 1}, a_{n - 1}, s'_{n - 1}, a'_{n - 1}),$ $y_n = \mathbb{E}[\delta_n \phi(s_n, a_n)|\mathcal{F}_n],$ and $M_{n + 1} = \delta_n \phi(s_n, a_n) - y_n.$ Separately, let $M_0 = 0.$ Since $\theta_n \in \mathcal{F}_n,$ i.e., $\theta_n$ is measurable with respect to $\mathcal{F}_n,$ we have
\begin{equation}
    y_n = f(\theta_n) = \sum_{\mathbf{a} \in \mathcal{A}^{\mathcal{S}}}  \left(b_{\mathbf{a}} - A_{\mathbf{a}} \theta_n\right) \mathds{1}[\theta_n \in \mathcal{P}_{\mathbf{a}}] \in h(\theta_n).
\end{equation}
Accordingly, the update rule in \eqref{e:generic.update} can now be rewritten as $\theta_{n + 1} = \theta_n + \alpha_n [y_n + M_{n + 1}], n \geq 0,$ which is of the desired form.
\end{proof}

\begin{corollary}[Isolated Equilibria]
\label{cor:isolated.points}
If the only internally chain transitive invariant sets for the DI in \eqref{e:lim.DI} are isolated equilibrium points, then $(\theta_n)$ almost surely converges to a (possibly sample path dependent) equilibrium point.
\end{corollary}

\begin{remark}
The almost sure convergence shows that the limiting DI in \eqref{e:lim.DI} captures \textit{all} possible asymptotic behaviors of our generic scheme given in \eqref{e:generic.update}.
\end{remark}

\begin{remark}
If the DI in \eqref{e:lim.DI} satisfies the condition in
Corollary~\ref{cor:isolated.points}, then our generic scheme given in \eqref{e:generic.update} will not exhibit parameter chattering.
\end{remark}

\begin{remark}
Our formal proof for stability builds upon the result in \citep{ramaswamy2017generalization}. As in \citep{gordon2000reinforcement}, the positive definiteness of the $A_\mathbf{a}$ matrices (Assumption~\ref{a:PD.matrix}) is what enables this result. For SARSA(0), this assumption can be verified as in \cite{gordon2000reinforcement}.
\end{remark}

\input{illustrations}

\input{conclusion}

\nocite{*}
\bibliographystyle{apalike}

\section{Conclusion and Future Directions}
\label{s:Conclusions.Future.Directions}
We have introduced the novel toolset of SRIs and DIs to aid the understanding of algorithmic dynamics of approximate RL algorithms. The recipe is conceptually simple -- for a stochastic iteration, write down its update in expectation, view it as a DI, and use insights from the DI to infer the limiting behavior of the original iteration. We believe that the primary value of this work is in this bridge that has been uncovered to this hitherto unexplored toolset in the context of RL algorithms. This toolset is already capable of providing principled explanations of a range of algorithmic dynamics that have been well observed and documented.

We may have merely scratched the surface with regard to wielding the DI approach to its fullest potential. Our framework paves the way for a more comprehensive understanding of existing approximate-RL algorithms as well as the design of new schemes with `well-behaved' DIs, hopefully resulting in value beyond just  analysis to the realm of synthesis.

On a somber note, the insights we have uncovered about algorithms using arguably the simplest possible (linear) function approximation casts doubt on their utility in more complicated, nonlinear approximation architectures. It is plausible that there are many more failure modes in these settings than the range observed in the linear case, as also seen by  \citet{young2020understanding} in neural approximation. Even with linear function approximation, we have shown in this paper that (a) one could converge to multiple attractors with disparate qualities, (b) one could oscillate between multiple policies' greedy regions, making it difficult to devise simple stopping criteria (c) even if one converges always to a unique point within a greedy partition, the induced greedy policy could be arbitrarily bad -- these spell doom for the practitioner unless carefully addressed and investigated. Separately, we acknowledge that our theory addresses only linear function approximation and more work is needed to extend it to nonlinear approximation architectures.

Yet another promising direction for future research, which is not taken up in this paper, is that of the quality of the limiting iterates (assuming there is some reasonable form of convergence). Again, \citet['Worst-case example']{young2020understanding} and \citet{bertsekas2011approximate} point out that approximate policy iteration schemes could very well converge to the `worst possible outcome', which, perhaps, with suitable design considerations based on the DI connection, can be avoided in new algorithms to come. For instance, a whole new class of algorithms can arise by assigning arbitrary values of $\epsilon, \epsilon'$ in the generic update \eqref{e:generic.update} with favourable properties. On a slightly different but related note, our work also reinforces the fact that merely ensuring stability of an incremental RL algorithm's iterates is by no means sufficient to guarantee good performance -- discontinuous policy updates often swamp out gains from stability by inducing complex and varied convergence modes.

\section{Illustration via Numerical Examples}
\label{s:Numerical.Examples}
We illustrate in this section how our main result (Theorem \ref{thm:main.result}) can be used to explain the limiting dynamics of the Q-learning and SARSA(0) algorithms with linear function approximation and $\epsilon$-greedy exploration. We study these algorithms applied to a range of 2-state 2-action MDPs, whose rewards, probability transition matrices and state-action features have been generated via random search (each MDP's details, such as probability
transition function, reward function, discount factor, and the algorithm's exploration parameter $\epsilon$, are provided in Appendix \ref{s:appendix.MDP.details}). We shall see that even with 2-dimensional linear function approximation, the algorithms' trajectories exhibit complex and non-trivial phenomena, and show how our results explain their causes via the associated DIs.

\subsection{Case study: The range of dynamics for Q-learning with only 2 greedy policies}
\label{ss:QL_2_greedy_regions}

This section explores some representative MDPs which, under linear approximation of their action-value functions in $\mathbb{R}^2$, exhibit only 2 greedy policies but behave very differently under Q-learning with function approximation. Note that we take up only the Q-learning update rule in the interest of space, though similar analyses and conclusions will hold for the SARSA(0) update rule with linear function approximation, depending on its appropriate DI.

\textbf{Two self-consistent greedy regions -- Multiple attractors}.
The MDP and features $\Phi$ for this section induce 2 greedy regions, i.e.,  $\mathbb{R}^2$ is partitioned into 2 greedy regions (see definition of $\mathcal{P}_\mathbf{a}$ in Section~\ref{s:Setup.Main.Result})  $\mathcal{P}_{\mathbf{a}_1}$ and $\mathcal{P}_{\mathbf{a}_2}$ via greedy policies $\mathbf{a}_1$ and $\mathbf{a}_2$, respectively.

Let us call the point $A_{\mathbf{a}_i}^{-1} b_{\mathbf{a}_i}$ a {\em landmark} for the partition $\mathcal{P}_{\mathbf{a}_i}$, $i = 1, 2$, where the $A_{\mathbf{a}}$ and $b_{\mathbf{a}}$ are as in \eqref{e:b_ba.defn},\eqref{e:A_ba.defn} (all matrices our examples will be positive-definite, hence invertible). The landmark $A_{\mathbf{a}_i}^{-1} b_{\mathbf{a}_i}$ is the (unique) point to which the ODE $\dot{\theta}(t) = b_{\mathbf{a}_{i}} - A_{\mathbf{a}_{i}} \theta$ would converge. Put another way, it is the point to which the Q-learning algorithm would converge were it to always sample actions $a_n$ in \eqref{e:generic.update} according to the $\epsilon$-greedy policy based on $\mathbf{a}_i$.

The greedy regions in this example are both {\em self-consistent}. By this, we mean that each contains its own landmark: $A_{\mathbf{a}_i}^{-1} b_{\mathbf{a}_i} \in \mathcal{P}_{\mathbf{a}_i}$ for $i = 1, 2$. The landmark s, along with their respective color-shaded greedy regions, are shown as diamonds in Fig. \ref{fig:2PartitionBothSelfConsistent_A}. Note that since each greedy region must be a cone and there are only 2 regions, each must be a halfspace with the origin $0$ on its boundary.

Fig. \ref{fig:2PartitionBothSelfConsistent_B} shows two sample paths of the iterates $\theta_n$ of Q-learning (update \eqref{e:generic.update} and $\epsilon' = 0$), with $\epsilon=0.1$ greedy exploration, $\gamma=0.75$ discount factor, decaying  stepsizes $\alpha_n = \Theta(1/n)$ and an iteration count of $20,000$, against the backdrop of the partition diagram. The starting positions, marked with black dots, were chosen arbitrarily. %
Note that segments of a trajectory are often parallel to each other, because the number of directions to move from an iterate is finite in a finite MDP.

Though the trajectories may traverse both greedy regions, they eventually approach one of the two landmark s (this observation remains true after several repeated trials). How can this be explained via our theoretical framework?

To answer this, let us look at the DI $\dot{\theta}(t) \in h(\theta(t))$ induced by Q-learning for this MDP, obtained via \eqref{e:base.driving.function}. Specifically, $h(\theta)$ equals the singleton set $\left\{ b_{\mathbf{a}_{1}} - A_{\mathbf{a}_{1}} \theta \right\}$ for $\theta$ lying in the interior of the blue region (say $\mathcal{P}_{\mathbf{a}_1}$), the singleton set $\left\{ b_{\mathbf{a}_{2}} - A_{\mathbf{a}_{2}} \theta \right\}$ in the interior of the red region (say $\mathcal{P}_{\mathbf{a}_2}$), and the connected segment $\textnormal{co}\left( \left\{ b_{\mathbf{a}_{1}} - A_{\mathbf{a}_{1}} \theta, b_{\mathbf{a}_{2}} - A_{\mathbf{a}_{2}} \theta \right\} \right)$ whenever $\theta$ is on the (1-dimensional) boundary of the two regions. In this DI, each point in the interior of a greedy region suffers a drift `towards' the region's landmark. Points on the boundary, however, can be subjected to {\em any} drift in the convex hull of the drifts induced by each partition's affine dynamics at that point.

Theorem \ref{thm:main.result} says that with probability 1, the iterates must converge to an invariant and internally chain transitive set of the DI. Recall that an invariant set $I$ means that $\forall \theta \in I$ there is some trajectory $\tilde{\theta}(t)$, $-\infty < t < \infty$ lying {\em entirely} in $I$ and passing through $\theta$ at time $0$. Roughly, for $I$ to be internally chain transitive, for any pair of points $\theta_{\text{start}}, \theta_{\text{end}} \in I$, a closeness tolerance $\eta$ and a travel time $T$, we must be able to find a solution of the DI and intermediate waypoints in $I$ such that the solution starts close to  $\theta_{\text{start}}$, ends close to $\theta_{\text{end}}$, and reaches close to each successive waypoint after time $T$.

Examining this DI leads to the conclusion that it possesses only 3 invariant, internally chain transitive {\em singleton} sets -- the 2 limiting dynamics points, plus a point $\theta^\circ$ on the boundary whose convex hull contains the $0$ vector\footnote{The coordinates of this (2-dimensional) point can be analytically obtained by solving 2 equations expressing that: (1) $\theta^\circ$ lies on the boundary of the greedy regions, (2) the vectors $b_{\mathbf{a}_{\text{red}}} - A_{\mathbf{a}_{\text{red}}} \theta^\circ$ and $b_{\mathbf{a}_{\text{blue}}} - A_{\mathbf{a}_{\text{blue}}} \theta^\circ$ point in opposite directions.} -- for the following reasons. These 3 singletons are certainly invariant and internally chain transitive sets, because for each there is a solution of the DI that always stays at that point for all $-\infty < t < \infty$ by virtue of zero drift. It also follows that there can be no other invariant set for the DI, because (a) any solution of the DI that, at any time, is in the interior of a region continues within that region towards its respective landmark, and (b) any solution that, at any time, is on the boundary but not at $\theta^\circ$ must suffer a drift that eventually pushes it into some greedy region's interior or along the boundary to $\theta^\circ$. We do not give a formal proof that these are the only invariant, internally chain transitive sets of the DI; the reader is directed to standard references on DIs \citep{aubin2012differential, smirnov2022introduction} and discontinuous dynamical systems \citep{cortes2008discontinuous} for rigorous arguments.

Thus, according to our result, the stochastic iterates of Q-learning must eventually approach one of these 3 isolated points. We additionally remark that the point $\theta^\circ$ on the boundary is an unstable equilibrium point, because any neighborhood around it contains points (in the interiors of $\mathcal{P}_{\mathbf{a}_1}$ and $\mathcal{P}_{\mathbf{a}_2}$) from where the DI's solutions will escape towards the landmark s. If the martingale difference noise $(M_n)$ is sufficiently `rich', i.e., it does not put all its probability mass in a subspace of $\mathbb{R}^2$, then random chance will cause the iterates to eventually escape this unstable equilibrium point. This condition will hold if all state-actions' features span $\mathbb{R}^2$, which they indeed do.

Our analysis also brings out the fact that asymptotically (as $n \to \infty$), there are no policy oscillations (iterates jumping regions infinitely often) nor parameter oscillations (iterates that `bounce between' two regions at a positive distance) that can occur, because the iterates must eventually stabilize around one of the two landmark s in the interior of the greedy regions.

\textbf{No self-consistent greedy regions -- Sliding mode attractor}.
We consider a different MDP and feature set that induce 2 greedy regions, each of which has its landmark\, in the other region. Two trajectories of Q-learning, along with the partition structure, are shown in  Fig. \ref{fig:2PartitionNoSelfConsistent} with the same parameters and stepsizes as before, and appear to move eventually towards the boundary between the two regions.

Reasoning about the DI and its solutions for this case paints a qualitatively different picture than that of the previous example. As before, for each point $\theta$ in the interior of any greedy region, there is a singleton, nonzero drift $h(\theta) = \{b - A\theta \}$, based on the $b, A$ of the region containing $\theta$. However, this drift is directed toward the {\em other} region. It follows that any trajectory of the DI initialized at a region's interior point $\theta$ will reach the boundary in finite time, which rules out interior points from belonging to any internally chain transitive sets.

The only possible invariant, internally chain transitive sets, then, are to be found on the 1-dimensional boundary between the 2 regions. A moment's thought convinces us that the only such set is the singleton comprising the unique point $\theta^\star$ on the boundary for which both dynamics' drifts point in opposite directions (equivalently, $0$ is in their convex hull). This point is strikingly different than the point $\theta^\circ$ of the previous example in that is a globally {\em stable} equilibrium point -- indeed, any deviation from this point pushes the DI's trajectory {\em towards} it rather than {\em away from} it. Moreover, observe that any trajectory of the DI started from an interior point of a region first reaches the boundary in finite time, due to a positive `velocity' towards the boundary. Thereafter, it `slides' along the boundary to reach $\theta^\star$, i.e., $\theta(t)$ always belongs to the boundary $\forall t \geq t_0$ for some time $t_0$. Such a trajectory is termed a {\em sliding mode} and has been extensively studied in control theory  (see e.g., \cite{utkin2013sliding, cortes2008discontinuous}). We remark that this kind of sliding mode attractor $\theta^\star$ is perhaps in the spirit of what \citet{bertsekas2011approximate} mentions as ``... the limit of optimistic policy iteration tends to be on a common boundary of several subsets of the greedy partition and may not meaningfully represent a cost approximation of any of the corresponding policies".

To our knowledge, the existence of sliding modes and associated equilibria has not been previously demonstrated for approximate RL algorithms. Also, our main result (Theorem \ref{thm:main.result}) guarantees that asymptotically, the stochastic iterates $\theta_n$ will not oscillate or chatter, although policy oscillations are possible (the iterates may cross the boundary infinitely often on their way to $\theta^\star$). This does not contradict previous claims about (parameter) chattering being observed in Q-learning or SARSA(0) \citep{young2020understanding, gordon1996chattering}, because we work with decaying Robbins-Monro type stepsizes, whereas previous work considers constant ($\Omega(1)$) stepsize schedules. We also remark that although our main result tells us only about the asymptotic nature of the iterates, more can perhaps be teased out about the nature of the trajectories via the connection between SREs and DIs; in fact, we conjecture that the existence of a sliding mode solution in the DI must imply that policy chattering is bound to take place almost surely.

\textbf{Only one self-consistent greedy region -- Unique attractor}.
We consider an MDP that, with linear Q-function approximation in $d=2$, induces two greedy regions with exactly one of them being self-consistent. The DI for this setting has the property that points in the red region will experience a (vector) drift, towards the boundary, which is bounded away from $0$ in norm. As a result, any trajectory that starts in the red region will, in finite time, hit the boundary. The driving (set-valued) function $h(\theta)$ for each boundary point can be shown to have all its constituent drift vectors `pointing into' the blue region, so any solution from the boundary must instantaneously leave it. Continuing from here, the trajectory moves towards the blue landmark, which can be shown to be the sole invariant, internally chain transitive set of the DI. It is thus clear, from Theorem \ref{thm:main.result}, that all the original stochastic iterates of the algorithm must eventually approach this point. This is a situation in which no policy or parameter chattering occurs. Also, as observed by \citet{young2020understanding}, it is possible that this (single) landmark\, point to which every trajectory converges may represent a very poor quality approximation for the optimal value function.

\begin{figure}[h]
\centering
\begin{subfigure}{0.4\textwidth}
\includegraphics[width=0.9\linewidth]{example_TwoPartitionMDP_BothSelfConsistent.png}
\caption{Greedy regions}
\label{fig:2PartitionBothSelfConsistent_A}
\end{subfigure}
\begin{subfigure}{0.4\textwidth}
\includegraphics[width=0.9\linewidth]{example_TwoPartitionMDP_BothSelfConsistent_dynamics.png}
\caption{Trajectories under Q-learning}
\label{fig:2PartitionBothSelfConsistent_B}
\end{subfigure}
\caption{{\footnotesize An MDP where function approximation in $\mathbb{R}^2$ gives rise to 2 greedy regions, both of which are self-consistent. (Left) The shaded areas represent the greedy regions, with the (self-consistent) landmark s for each partition's dynamics marked with diamonds of the respective colors. (Right) Two sample trajectories of Q-learning, with a $\Theta(1/n)$ stepsize schedule, overlaid on the partition diagram with their starting iterates marked with black dots. The iterates are observed to converge to one of the self-consistent landmark s (behavior). The differential inclusion here has only the self-consistent landmark s as its asymptotically stable attractors (cause).}}
\label{fig:2PartitionBothSelfConsistent}
\end{figure}

We discuss other examples such as the chattering phenomenon observed in \citet{gordon1996chattering, young2020understanding} in the Appendix.

\begin{figure}[h]
\centering
\begin{subfigure}{0.4\textwidth}
\includegraphics[width=0.9\linewidth]{example_TwoPartitionMDP_BothNotConsistent_dynamics.png}
\caption{}
\label{fig:2PartitionNoSelfConsistent}
\end{subfigure}
\begin{subfigure}{0.4\textwidth}
\includegraphics[width=0.9\linewidth]{example_TwoPartitionMDP_BothPointToOnePartition_dynamics.png}
\caption{}
\label{fig:2PartitionOneSelfConsistent}
\end{subfigure}
\caption{{\footnotesize (Left) An MDP where function approximation in $\mathbb{R}^2$ gives rise to 2 greedy regions, neither of which is self-consistent. Two sample trajectories of the iterates of Q-learning, with a $\Theta(1/n)$ stepsize schedule, are overlaid on the partition diagram with their starting iterates marked with black dots. The iterates converge to a unique point on the partition boundary (behavior) The differential inclusion here has a single `sliding mode' attractor on the boundary (cause). (Right) An MDP where function approximation in $\mathbb{R}^2$ gives rise to 2 greedy regions, exactly one of which which is self-consistent. Two sample trajectories of the iterates of Q-learning, with a $\Theta(1/n)$ stepsize schedule, are overlaid on the partition diagram with their starting iterates marked with black dots. The iterates always converge to the unique self-consistent point in one of the partitions (behavior). The differential inclusion here has a unique attractor (cause).}}
\label{fig:2PartitionNoAndOneSelfConsistent}
\end{figure}

\subsection{Case study: Disparate behavior for Q-learning and SARSA(0) with identical feature sets}

We show in this section that Q-learning and SARSA(0), both with the same linear Q-function approximation architecture and $\epsilon$-exploration rate, can exhibit qualitatively different limiting behavior. The cause for this differing behavior will be again shown to lie in the associated differential inclusions (DIs). Figures \ref{fig:Disparity_QL_SARSA_QL} and \ref{fig:Disparity_QL_SARSA_SARSA} each show two stochastic trajectories of Q-learning and SARSA(0) with linear function approximation, overlaid on the (identical) greedy region partition diagram for the underlying 2-state 2-action MDP, along with the (non-identical) landmark\, points for the greedy regions induced by each algorithm. Recall that as per \eqref{e:generic.update}, the only way these algorithms differ is in the fact that the second action $a_n'$ is chosen at time $n$ from an $\epsilon'$-greedy policy, with $\epsilon' = 0$ for Q-learning and $\epsilon'=\epsilon$ for SARSA(0). We have also ensured that both algorithms induce stable matrices $A_{\mathbf{a}}$, thus their iterates cannot diverge.

We see that the configuration of landmark\,  points for each partition's dynamics in the DI is different in each case. This makes the green and red greedy regions self-consistent (i.e., they contain their own limiting dynamics) for Q-learning, whereas they are both non self-consistent for the SARSA(0) DI. This disparity in structure explains why trajectories for Q-learning are seen to approach the (stable) red and blue points, and those for SARSA(0) approach a `sliding mode' attractor on the boundary of the red and green greedy regions.

An observation in a similar spirit as ours has been made previously, albeit at a high level, about approximate policy iteration by \citet{bertsekas2011approximate}, that ``... the choice of the iterative policy evaluation method ... does not seem crucial for the quality of the final policy obtained (as long as the methods converge)", leading to `` ... an open question whether (this notable failure of approximate policy
iteration) is due to the inherent difficulty of the
tetris problem, or whether it can be attributed to inherent pathologies of approximate policy iteration, such
as oscillations/chattering between relatively poor policies, or weaknesses of the cost function approximation
philosophy." Q-learning and SARSA(0) differ only in their approach to policy evaluation via the sampling of the action $a_n'$. However, our DI framework leads us to believe that the complex dynamical possibilities of these algorithms may primarily be a result of the rich vector fields or DI structures that they induce rather than the MDP (e.g., tetris) itself being difficult.

\begin{figure}[h]
\centering
\begin{subfigure}{0.4\textwidth}
\includegraphics[width=0.9\linewidth]{example_QL_SARSA_disparity_QLstructure_dynamics.png}
\caption{Trajectories under Q-learning}
\label{fig:Disparity_QL_SARSA_QL}
\end{subfigure}
\begin{subfigure}{0.4\textwidth}
\includegraphics[width=0.9\linewidth]{example_QL_SARSA_disparity_SARSAstructure_dynamics.png}
\caption{Trajectories under SARSA(0)}
\label{fig:Disparity_QL_SARSA_SARSA}
\end{subfigure}
\caption{{\footnotesize Example of an MDP for which Q-learning and SARSA(0) algorithms induce qualitatively different limiting structure. (Left) Two sample trajectories of the iterates of Q-learning, with a $\Theta(1/n)$ stepsize schedule, are overlaid on the partition structure and landmark point structure induced by Q-learning. In the landmark point structure induced by Q-learning, there is no `sliding mode' point on the boundary of the green and red partitions, thus the trajectories move towards the red landmark in this case. (Right) Two sample trajectories of the iterates of SARSA(0), with a $\Theta(1/n)$ stepsize schedule and the same initial positions as for Q-learning, are overlaid on the partition diagram of the MDP. In the landmark~ structure induced by SARSA(0), there is a `sliding mode' equilibrium point on the boundary of the green and red regions towards which both trajectories move (see inset).}}
\label{fig:Disparity_QL_SARSA}
\end{figure}


\section{\label{sec_intro}Introduction}

	Devices exploiting the laws of quantum mechanics promise to deliver the next step-change in technology.
	In particular, quantum sensing exploiting multi-particle quantum entanglement could have a substantial impact in our daily lives, as it features better scaling of precision compared to classical methods~\cite{Pezze_Rev_2018, Monz_2011,berrada_integrated_2013,hosten_measurement_2016}.
	Several classes of multi-particle entangled states have been proposed and realized, among them squeezed states \cite{Meyer_experimental_2001, riedel_atom_2010, Wu_retrieval_2020}, Dicke states \cite{barontini_deterministic_2015, Lucke_detecting_2014}, and GHZ or ``Schr\"odinger's cat'' states \cite{leibfried_toward_2004,Monz_2011, omran_generation_2019}. These have been realized in a variety of systems, and among them, interfaces of atoms with optical cavities has delivered so far some of the best performance in terms of metrological gain \cite{haas_entangled_2014,barontini_deterministic_2015,hosten_measurement_2016,Gietka_Supersolid_2019,Torre_Dissipative_2013,Victoria_Probing_2019,Hamilton_atom_2015,Sabulsky_multi_2022}.

	The main issue with entangled states is that they are extremely fragile, and this is particularly marked for symmetric entangled states that require all the atoms in the ensemble to be indistinguishable. This is because it is very difficult to maintain symmetry and indistinguishably in presence of decoherence and particle loss. Asymmetric entangled states are instead more resilient, but they are usually considered less-performing and also difficult to realize in a controlled fashion. In this work, we discuss a novel architecture to optimally generate metrologically useful multi-particle entangled states using atoms trapped in a cavity. In our system, single atoms are trapped in optical tweezers and their interaction is mediated by the field in the cavity. We show that with the addition of two ``transverse" control beams, it is possible to adiabatically generate asymmetric entangled states that are optimised for a certain measurement protocol. We concentrate in particular on the measurement of an external magnetic field and a magnetic field gradient. We compare the performance of the states produced with classical states and ideal maximally-entangled Dicke states. We find that in the regime of parameters explored, our states always outperform classical states and even outperform the ideal Dicke states when measuring a magnetic field gradient. We also show that our protocol can be easily scaled up.
	\begin{figure}
		\centering
		\includegraphics[width=0.55\textwidth]{Experimental_proposal_2}
		\caption{Pictorial representation (not to scale) of the system: individual atoms (black spheres) are trapped in optical tweezers (pink beam). These atoms are interacting with a single mode of a Fabry-Pérot cavity (blue field). The optical tweezers can be generated using a microscope objective. The right of the image shows a representation of a simple 3-level atomic system and the light fields coupling to the transitions. The two control fields (green fields) $\Omega_1$ and $\Omega_2$ couple to two separate halfs of the atoms during the adiabatic creation of the entangled states.}
		\label{fig:cartoon}
	\end{figure}

	The article is organized as follows: in Section~\ref{sec:thesystem} we describe the experimental system and we discuss a realistic implementation of it, in Section~\ref{sec_entangled} we introduce the adiabatic protocol to generate optimally-entangled states for the measurement of an external magnetic field and a magnetic field gradient.
	In section~\ref{sec_sensing}, we quantify the performance of our entangled states and compare their precision with classical states and entangled Dicke states. Section~\ref{conclusion} is finally devoted to the conclusions.

	\section{The System}
	\label{sec:thesystem}

	Our system is depicted in Fig.~\ref{fig:cartoon} and consists of $N$ individual three-level atoms trapped with optical tweezers inside the mode of an optical cavity.  We divide the atoms into two halves, and each half is individually addressed by dedicated light beams with Rabi frequencies $\Omega_1(t)$ and $\Omega_2(t)e^{i \phi}$ respectively. The cavity and each of the control beams are tuned to be resonant with the atomic transitions as shown in Fig.~\ref{fig:cartoon}. Such a configuration realizes a $\Lambda$-type system for each atom, in which the two ground states act as qubit states. The cavity is used to mediate the interaction between the individually trapped atoms. Trapping atoms in tweezers allows us to suppress detrimental atom-atom collisions.

	Our protocol for the generation of multiparticle entanglement consists in finding the optimal evolution for the control light fields $\Omega_1(t)$ and $\Omega_2(t)e^{i \phi(t)}$ such that the Classical Fisher information ($\mathcal{F}$) for sensing a certain parameter $\theta$ is maximum. Detailed derivation of $\mathcal{F}$ for our system is given in~\ref{app:fish_info}. As the discussion here is specific to a feasible measurement protocol, we use Classical Fisher information instead of Quantum Fisher information. Generally, it is hard to find measurement protocols which saturates the bounds given by Quantum Fisher information, hence we avoid it here. For detailed discussion on Quantum and Classical Fisher information see reference~\cite{Pezze_Rev_2018}.
	The Classical Fisher information is related to the measurement uncertainty of an an unknown quantity $\theta$ via the following identity,
	\begin{equation}
	\Delta \theta \ge \sqrt{\frac{1}{M \mathcal{F}}},
	\end{equation}
	where $M$ are the number of measurements.
	For large $M$, the above identity is known to be saturated by Maximum-Likelihood method~\cite{Braunstein_1992}.

	The evolution of the system density matrix is governed by the following master equation,
	\begin{equation}
	\label{eqn:master_eqn}
	\begin{split}
	&\frac{d\hat{\rho}}{dt} = - i [\hat{H}_f, \hat{\rho}(t)] - \sum_{A} \sum_{j = 0,1}  \frac{\Gamma_j}{2} \left(\hat{\sigma}^\dagger_{A,j} \hat{\sigma}_{A,j} \hat{\rho} - 2\hat{\sigma}_{A,j}\hat{\rho} \hat{\sigma}^\dagger_{A,j} + \hat{\rho}\hat{\sigma}^\dagger_{A,j} \hat{\sigma}_{A,j} \right) \\
	& -  \frac{\kappa}{2} \left(\hat{a}^{\dagger} \hat{a} \hat{\rho} - 2\hat{a}\hat{\rho} \hat{a}^{\dagger}  + \hat{\rho} \hat{a}^{\dagger} \hat{a}  \right),
	\end{split}
	\end{equation}
	where the Hamiltonian $\hat{H}_f$ takes into account the coupling to the cavity and a classical light field. $A$ denotes a particular atom, $\hat{\sigma}_{A,j} = \ket{j_{A}}\bra{e_{A}}$, $\Gamma_0$ and $\Gamma_1$ are decay rates from level $\ket{e}$ to levels $\ket{0}$ and $\ket{1}$ respectively, $\hat{a}$ is the annihilation operator for the cavity field, and $\kappa$ is the cavity field decay rate. The explicit form of $\hat{H}_f$ is given in the~\ref{app:calculations}.

	\begin{figure}
		\centering
		\includegraphics[width=0.9\textwidth]{pulses_all.pdf}
		\caption{Evolution of the light fields for (a) two atom system for case-I (see section~\ref{sec_sensing}), (b) two atom system for case-II.
		}
		\label{fig:probability_evolution}
	\end{figure}

	To illustrate a realistic implementation, we concentrate on Rubidium 87 atoms, which have been the workhorse of cold atoms experiments due to their ease of cooling to ultracold temperatures.
	As qubit states we choose the $|F,m_F\rangle = |1, 1\rangle$ and $|2, 0\rangle$ hyperfine levels of the $5^{2}\text{S}_{1/2}$ ground state, such that $\ket{0} = |1, 1\rangle$ and $\ket{1} = |2, 0\rangle$. To implement our protocol, it is preferable to minimize off-resonant excitations by separating the transition between the two qubit states from those with other Zeeman states using an external magnetic field. At the same time, it is important to avoid excessive noise coming from magnetic field fluctuations. We found that at a magnetic field of $B_0 = 654.39$ Gauss (see \ref{app:zeeman}), the energy difference between the two qubit states has a low sensitivity to magnetic field noise. Indeed the change in transition frequency between these two levels is $\Delta f \sim  10^{-3}$ Hz for a magnetic field fluctuation of $1$~mG. At the same time, this magnetic field separates the Zeeman states by $\sim 100$~MHz and is easy to produce in an experiment. It is worth noticing that states whose energy difference has low sensitivity for magnetic field noise can also be found for lower magnetic fields~\cite{Szmuk_2015} \footnote{The stable and high magnetic field is only required during the generation of the entangled states. Once these are created, the magnetic field can be adiabatically switched off and a measurement of some other unknown magnetic field can be performed.}. In addition, the energy difference between the levels $|0\rangle$ and $|1\rangle$ does not depend on the intensity of the trapping light as the AC-stark shift for the levels  of $5^{2}\text{S}_{1/2}$ state is independent of $F$ and $m_F$ if linearly polarized light is used for trapping~\cite{Arora_state_2012,le_kien_dynamical_2013}. This will make a quantum superposition of these states insensitive to fluctuations in the intensity of the tweezer trap. The above properties make the chosen levels ideal for the storage of quantum information. It should be possible to extend the coherence time of these states for $> 1$ second if the magnetic field fluctuations are $<100$~mG, which is possible with current technology~\cite{merkel_magnetic_2018,farolfi_design_2019}.

	Concerning the cavity, we choose a common configuration (see e.g. \cite{Welte_photonmediated_2018}) with length 1 mm and radius of curvature of the cavity mirrors 10 mm. This results in $g_0 \sim 4$ MHz, where $\hbar g_0  = \mu_{1,e}\sqrt{\hbar \omega_c/(2  \epsilon_0)}$ is the interaction strength between the cavity field and a single atom~\cite{steck_quantum_2011}, $\omega_c$ is the cavity resonance frequency, and $\mu_{1,e}$ is the transition dipole moment between the $\ket{1}$ and $\ket{e}$ levels \footnote{Here we consider the excited state $\ket{e}$ to be  the eigenstate closest in composition to $(F,m_F) = (1, 0)$ level of the $5^{2}\text{P}_{1/2}$ electronic state (This state is part of the D1 spectrum in $^{87}$Rb~\cite{Steck_Rb_87}). After diagonalizing the hyperfine and Zeeman Hamiltonian at a magnetic field of $B_0 = 654.39$ Gauss, the eigenstate closest to the state $(1, 0)$ is not pure and has a 10$\%$ mixture from other states. We take this into consideration when calculating $\mu_{1,e}$, which lets us accurately calculate $g_0$.}. Considering a reflectivity of $99.9999~\%$~\cite{Cole_16,Maldaner_20} for the cavity mirrors, we obtain a cavity loss rate of $\kappa/\hbar \sim 0.1$~MHz. Our cavity parameters should allow for up to 1000 spatially separated atoms with nearly-identical atom-cavity coupling  ($\ge 0.9 g_0$ for all the atoms), that can be accurately positioned using the optical tweezers~\cite{Endres1024,barredo_synthetic_2018}. The position stability of the atoms can be achieved by cooling the atoms to their vibrational ground state in the tweezer trap. This will localize the atoms to much smaller distance ($\sim$~30 nm) than the wavelength of the light. This can be achieved using techniques demonstrated in reference~\cite{Kaufman_cooling_2012}.

	\section{Entangled state creation}
	\label{sec_entangled}
	The goal of our protocol is to find the optimal pulse sequence that maximizes $\mathcal{F}$ for two cases. Where case-I is all  atoms sensing a common field, and case-II is the two halves sensing the difference between two fields. At the same time, to minimize decoherence we must avoid populating the state $\ket{e}$. Therefore we consider only quasi-adiabatic pulse sequences~\cite{wu_fast_2017}. The maximization is done using generalized simulated annealing~\cite{TSALLIS_1996}. The optimization variables include one value of the total time $T$ and 10 values each of $\Omega_1(t)$, $\Omega_2(t)$, and $\phi(t)$. The 10 values are equally spaced between $T$. The values between these points are then found using a smooth cubic interpolating function. During the optimization, the points are bounded in range (0,1) MHz for the amplitudes and (0, 2$\pi$) for the phase difference. Unless stated otherwise, we use the parameters $2\Gamma_0 = 2\Gamma_1 = 5.75$~MHz, $g_0 = 4$~MHz, and $\kappa = 0.1$~MHz. To simulate conditions close to the experimental one, we consider that (i) the change in $\Omega_1(t)$ and $\Omega_2(t)$ is step-wise, with each step corresponding to $\sim 1~\mu$s. This is achieved by digitizing the interpolation function found during the optimization. (ii) The control field resolution is 8-bit such that $\Omega$ can take values only in steps of $\Omega_\text{max}/2^8$. (iii) As detuning from excited state, $\Delta$ and the two-photon detuning, $\delta$ (see \ref{app:calculations} for definitions) are noisy variables, they are randomly sampled during each time step from a Gaussian distribution with a mean of zero. The standard deviation for the above Gaussians for $\Delta$ and $\delta$ are 1~MHz and 10~kHz, respectively. For the cavity, we consider only excitations up to $n = 1$. We observe that including higher photon number states does not change $\mathcal{F}$ because during the evolution of the system the cavity is only virtually excited.

	To simulate Eqn.~\ref{eqn:master_eqn} we use the python numerical package Qutip~\cite{qutip2}. As an example, in  Fig.~\ref{fig:probability_evolution} we show the evolution of $\Omega_1(t)$, $\Omega_2(t)$ and $\Phi(t)$ in a two-atom system for case-I and case-II.

	In general, we find that the states produced at the end of the evolution have the form
	\begin{equation}
	\label{eqn:state}
	\hat{\rho}_\text{DL} \approx p\ket{DL_N^{(m)}}\bra{DL_N^{(m)}}+(1-p)\hat{\rho}_\text{loss},
	\end{equation}
	where
	\begin{equation}
	\ket{DL_N^{(m)}} = \left(^{N}_{m}\right)^{-1/2}\sum_k C_k \ket{0^{\otimes (N-m)}1^{\otimes m}}_k,
	\end{equation}
	and $p$ is the purity with which $\ket{DL_N^{(m)}}$ is created.
	$\{k\}$ is the set of all distinct permutations of the spins.
	In these states the $m$ excitations (occupations of state $\ket{1}$) are delocalized over all the qubits.
	Interestingly, these states are close to the maximally entangled Dicke states, that can be obtained when all the $C_k$ coefficients are equal.

	To create the state shown by Eqn.~\ref{eqn:state} we start with the state $\ket{0^{\otimes (N-m)}1^{\otimes m}}$, where the $m$ excitations are localized on the last $m$ qubits. Such a state can be easily achieved for arbitrary $m$ from a fully polarized state (all atoms in $\ket{0}$) if the two parts formed by first $N-m$ atoms and last $m$ atoms are separately addressable, which is also the minimal requirement to create the state in Eqn.~\ref{eqn:state}. Here one of the transverse beam shown in Fig.~\ref{fig:cartoon} addresses $m$ qubits and the other addresses $N-m$ qubits\footnote{This is the simplest case, in the tweezer setup all atoms can also be addressed individually.}. Once we have such a non-entangled state, the protocol shown in Fig.~\ref{fig:probability_evolution} then slowly delocalizes these $m$ excitations over all the qubits, creating an entangled state in the process. The system parameters we use are close to the ones required for two atoms undergoing quantum Zeno dynamics and adiabatic processes in a dressed state basis~\cite{wu_fast_2017}. Hence our protocol with off-resonant pulses should follow similar processes. The excitations hop from one atom to another via virtual excitations of the photons in the cavity. These processes help in keeping the system in the basis spanned by states $\ket{0^{\otimes (N-m)}1^{\otimes m}}_k$ as the total number of excitations are not changed. However the system is not perfect and in addition to this delocalization, there are incoherent processes that result in probability $(1-p)$ to occupy states where the excitations are different from $m$. The random nature of these processes mixes the target state $\ket{DL_N^{(m)}}$ with these other unwanted sates, the unwanted states are collectively denoted by $\hat{\rho}_\text{loss}$. These unwanted process are proportional to $\Gamma$ and $\kappa$ and modeled into the simulations by Lindblad master equation as shown in Eqn.~\ref{eqn:master_eqn}.

	\section{Quantum-enhanced Metrology}
	\label{sec_sensing}
	\begin{figure}
		\centering
		\includegraphics[width=0.9\textwidth]{CFI_simple_plus_twopart.pdf}
		\caption{Maximum Fisher information ($\mathcal{F}_\mathrm{max}$) as a function of atom number for (a) case-I and (b) case-II. Dotted green line shows the quantum shot-noise limit, dashed black line is for Dicke states, blue triangles are for states we create. For (a) the blue solid line is a $N + a N^2$ fit to the blue triangles. For $N>6$, we use sequence optimized for $N = 6$.}
		\label{fig:sensitivity_vs_atno}
	\end{figure}
	In this section, we evaluate the metrological gain of the Dicke-like states~\cite{Toth_07} produced with our protocol for case I and II.
	A change in magnetic field results in a change in the energy difference between the levels $\ket{0}$ and $\ket{1}$, which can be detected by measuring the change in the phase difference between these levels. Quantitatively, the effect of this field on the atoms can be represented by the following unitary matrix,
	\begin{equation}
	\hat{U}_\text{ph} = e^{- i \hat{H}_\text{ph} },
	\end{equation}
	where $\hat{H}_\text{ph} = \sum_{A = 1}^{N} \theta_A \hat{\sigma}^A_z/2$ is the Hamiltonian generating the phase difference,  $\hat{\sigma}^A_z$ is the Pauli-$z$ operator for atom $A$, and  $\theta_A = f(B_A,t)$ is the angle that is the function of the magnetic field strength ($B_A$) for atom $A$ and the time of interaction ($t$).

	For case I), after the state in Eqn.~\ref{eqn:state} is created, we first perform a $\pi/2$ rotation of each qubit individually,
	$\hat{U}_{\pi/2}\hat{\rho}_\text{DL}\hat{U}_{\pi/2}^{\dagger}$ and then we let it evolve under the action of $\hat{U}_\text{ph}$.
	After a certain time $t$, $\theta(B,t)$ is imprinted on the state, and we then rotate it back with another $\pi/2$ pulse. After this stage a measurement of the atomic populations is performed.
	As measurements can be performed on each atom separately in our system, each measurement will result in a projection onto the diagonal basis $\ket{b_j} = \ket{i_0,i_1...,i_N}$, where $i$ takes the values 0 and 1, and $N$ is the number of atoms. In Fig.~\ref{fig:sensitivity_vs_atno}(a) we report the Maximum Fisher information (maximized over $\theta$) as a function of the atom number for this case~\footnote{{Our computational resources allow us to perform the simulations only till N = 10. For $N>10$, the requirements for RAM and processor speeds are outside the available computational resources.}}. Here $\theta_A = \theta$ for all $A$.
	In Fig.~\ref{fig:sensitivity_vs_atno}(a), we also compare the Fisher information of our states with those of the fully symmetric Dicke states and the coherent states $\ket{1_\text{all}} =  \ket{1_1,1_2...,1_N}$. The non-entangled state gives a quantum shot-noise limit of $\mathcal{F}_\mathrm{max} = N$. We find that the states generated by our protocol perform better than the shot-noise limit and hence are useful for quantum metrology. A $N+a N^2$ fit to $\mathcal{F}_\mathrm{max}$ gives the $N^2$ component $a = 0.34\pm0.01$. The Dicke state's Fisher information scales instead as $\sim N + 0.5 N^2$~\cite{T_th_2014}. The reduction in $\mathcal{F}_\mathrm{max}$ for our states with respect to the ideal Dicke states is expected due to the presence of decoherence during the entanglement protocol. We also note that the optimal time evolution of fields found for two atoms is also optimal for $N>2$, removing the need to find an optimal evolution each time $N$ is changed. Assuming a best case scenario, extrapolating $\mathcal{F}_\mathrm{max}$ for our states with $N = 1000$ will lead to 25 dB gain in precision compared to classical systems. We note that the optimal time required to create such states scales linearly with $N$.

	It is well known that the presence of correlated dephasing (common field fluctuations) greatly reduces the utility of a class of entangled states~\cite{Monz_14_2011} that are optimal for estimating a parameter which is common to all atoms. Hence, in presence of common noise, it is advantageous to create entangled states which are insensitive to such correlated dephasing~\cite{Dorner_2012}. The Dicke-like states ($\ket{DL_N^{(N/2)}}$) are such states, because of equal number of atoms in the state $\ket{0}$ and $\ket{1}$ for all of its components, all common phase changes are canceled out~\cite{Dorner_2012}. This makes them well suited to measure differences in fields or field gradients, which is the case-II). For case-II) the change in phase is such that $\theta_A = \theta/2$ for $A = 1$ to $N/2$ and $\theta_A = -\theta/2$ for $A = N/2+1$ to $N$, where $\theta$ is the phase difference introduced by a field difference $\Delta B$.

	For case II) the Fisher information for a classical state is $\mathcal{F}_\mathrm{max} = N/4$, because each half of the atoms will detect the local field with $\mathcal{F}_\mathrm{max} = N/2$. Since we are interested in the difference of these local fields, $\mathcal{F}_\mathrm{max}$ decreases further by a factor of 2. From Fig.~\ref{fig:sensitivity_vs_atno}(b) we can see that the states we create (optimized for each $N$ separately) perform better than the quantum shot-noise limit also in this case. Notably, for case-II) our states outperform also the maximally entangled Dicke states in the region of parameters that we have explored.

	The Fisher information for our Dicke-like states depends on the value of $\theta$. The scaling $\sim N + 0.5 N^2$ for the Dicke states is proven to hold for small $\theta$ and high $N$~\cite{T_th_2014}. We investigate the Fisher information of our scheme as a function of $\theta$, the results of which are shown in Fig.~\ref{fig:sensitivity_vs_angle}. We see that $\mathcal{F}$ oscillates as a function of $\theta$. These oscillations become less prominent as the number of atoms $N$ increases, as can be seen after comparing $\mathcal{F}$ for 4-atoms and 8-atoms from Fig.~\ref{fig:sensitivity_vs_angle}.
	\begin{figure}
		\centering
		\includegraphics[width=0.9\textwidth]{FI_vs_theta_all.pdf}
		\caption{Fisher information as a function of the angle $\theta$. Dashed red line is for 4 atoms and solid blue line is for 8 atoms. (a) Case-I and (b) case-II.}
		\label{fig:sensitivity_vs_angle}
	\end{figure}

	For our system, the maximum Fisher information is determined by the single atom cooperativity $\eta  = 4g_0^2/(\Gamma \kappa)$. Ideally one would want the photon-atom coupling $g_0$ to be considerably greater than the damping rates $\Gamma$ and $\kappa$ that introduce decohrence during the entangling operation.
	One may decide to use atomic levels with small $\Gamma$; however this will also result in smaller $g_0$, which is not desirable. Also, the mode volume of the cavity can be reduced to increase the value of $g_0$ to as high as 215~MHz~\cite{colombe_strong_2007}. However, there are two disadvantages of this: the number of atoms which can fit in the cavity will reduce and the value of $\kappa$ will go up. Explicit calculations using the parameters form~\cite{colombe_strong_2007} showed no improvements in the Fisher information for our protocol.

	With better optical coatings for the cavity mirrors, we expect the value of $\kappa$ to reduce further and hence an improvement in the maximum Fisher information. In Fig.~\ref{fig:fid_vs_kappa} we compare the Fisher information for various values of $\kappa$ for case I) and case II), where we optimized the light pulses for each change in parameters. The results are shown in Fig.~\ref{fig:fid_vs_kappa}. As expected, the maximum Fisher information increases as $\kappa$ is reduced. The reduction flattens out after a $\kappa_\text{min}$. For case I) it occurs at $\kappa_\text{min} \approx 10^{-2}$ MHz and for case II) at $\kappa_\text{min} \approx 0.1$ MHz.

	\begin{figure}
		\centering
		\includegraphics[width=0.5\textwidth]{FI_vs_kappa.pdf}
		\caption{Fisher information as the function of $\kappa$. Red circles are for case-I and blue triangles for case-II. Here $\Delta = 0$, $\delta = 0$, $2\Gamma_0 = 2\Gamma_1 = 5.75$~MHz, $g_0 = 4$~MHz, and $N = 4$.}
		\label{fig:fid_vs_kappa}
	\end{figure}

	\section{Conclusion}
	\label{conclusion}
	We have proposed a scheme for generation of multi-particle entanglement using an ensemble of singly-trapped atoms in an optical cavity. By using two external control fields, we can dynamically generate optimal entangled states according to the measurements one needs to perform. We have simulated a realistic system, accounting also for decoherence effects. We have performed these simulations for up to 10 atoms. Sources of entanglement infidelity were identified, and possible improvements in technology were suggested. We have shown how these entangled states can be used for performing quantum-enhanced metrology. We find that our proposed states perform better than the quantum shot-noise limit in measuring magnetic fields and magnetic field gradients. The protocol presented here is quite generic and can be used for other quantum sensing applications. For example, our states could be used to make better optical clocks. They also can be used to measure electric fields if one of the ground state is excited to a Rydberg state.

    \ack
		The author was supported by the Leverhulme Trust Research Project Grant UltraQuTe (Grant No. RGP-2018-266) during this work. The author thanks Dr. Giovanni Barontini for feedback and guidance on this work.


\section{Introduction}
\label{sec:intro}

Cosmic voids are promising cosmological probes for testing the dark energy problem and alternative gravity theories.
Since they are vast underdense regions of space, they encode key information about the geometry and expansion history of the Universe.
Moreover, the potential of voids has increased recently with the development of modern spectroscopic surveys, which are covering a volume and redshift range without precedents.

There are two primary statistics in void studies: (i) the void size function, that describes the abundance of voids \citep{svdw,abundance_jennings}, and (ii) the void-galaxy cross-correlation function \citep{clues2,rsd_cai}, that characterises the density and peculiar velocity fields around them.
Both statistics are affected by distortions in the observed spatial distribution of galaxies, which translate into anisotropic patterns: (i) the redshift-space distortions \citep[RSD]{rsd_kaiser}, a dynamical effect, and (ii) the \citet[AP]{ap} distortions, a geometrical effect.
Both effects encode fundamental information about the cosmological parameters \citep{aprsd_correa, aprsd_hamaus2020, rsd_hawken2, aprsd_nadathur2020}.

However, there has been an obstacle that has prevented the use of voids as reliable cosmological probes.
Our standard picture of distortions is incomplete.
Traditionally, we have focused only on the spatial distribution of galaxies around voids.
The truth is that the RSD and AP effects also affect intrinsic void properties, such as their number, their size and their spatial distribution.
These systematics generate additional anisotropic patterns on observations that lead to biased cosmological constraints if they are not taken into account properly.

The purpose of this article is to compile the main results of various works in order to lay the theoretical foundations for designing reliable and unbiased cosmological tests based on the void size function and the void-galaxy cross-correlation function.

\section{Redshift-space effects in voids}
\label{sec:zeffects}

This section is based on the results obtained in \cite{zvoids_correa} using the so-called spherical void finder \citep{clues3}.
Galaxies can be considered as particles under the mapping between real and redshift space.
They are totally conserved in this process, only their position changes.
Voids, by contrast, are extensive regions: some may get lost, others may be created artificially.
Nevertheless, we found that voids above the shot-noise level are almost conserved under this mapping, the loss of voids decreases as larger sizes are considered.
Therefore, it is statistically valid to assume \textit{void number conservation}.

Redshift-space voids are systematically bigger than their real-space counterparts.
This \textit{expansion effect} (hereafter t-RSD) is a consequence of the RSD induced by tracer dynamics at scales around the void radius.
The volume of voids is also affected by the AP effect.
This is because the discrepancies between the fiducial and true cosmologies generate deviations between the void dimensions along ($\parallel$) and perpendicular ($\perp$) to the line of sight (hereafter LOS).
This is the \textit{Alcock-Paczynski volume effect}.
An analytical description is possible from dynamical (Eq.~\ref{eq:velocity}) and cosmological (Eq.~\ref{eq:ap_coord}) considerations, from which a linear relation is found between the real and redshift-space radii: $R_{\rm v}^{\rm rs}$ and $R_{\rm v}^{\rm zs}$, respectively:
\begin{equation}
    R_{\rm v}^{\rm zs} = q_{\rm AP} ~ q_{\rm RSD} ~ R_{\rm v}^{\rm rs},
    \label{eq:q_ap_rsd}
\end{equation}
where
\begin{equation}
    q_{\rm RSD} = 1 - \frac{1}{6} \frac{f(z)}{b} \Delta_{\rm v, g}^{\rm NL}, \\
    q_{\rm AP} = \sqrt[3]{(q_{\rm AP}^\perp)^2 q_{\rm AP}^\parallel},
    \label{eq:q_factors}
\end{equation}
where in turn, $q_{\rm AP}^\perp = D_{\rm M}^{\rm fid}(z)/D_{\rm M}^{\rm true}(z)$ and $q_{\rm AP}^\parallel = H_{\rm true}(z)/H_{\rm fid}(z)$.
In $q_{\rm RSD}$, $\Delta_{\rm v, g}^{\rm NL}$ is the average density contrast threshold defined for void identification, $f(z)$ the logarithmic growth rate of density perturbations, and $b$ the bias parameter relating the galaxy and matter density fields.
In $q_{\rm AP}$, $D_{\rm M}$ denotes the comoving angular diameter distance, and $H$ the Hubble parameter.
The indices ``fid" and ``true" refer to fiducial and true cosmology quantities, respectively.
In both cases, $z$ is the mean redshift of the void.

Finally, redshift-space void centres are systematically shifted along the LOS.
This is the \textit{off-centring effect} (hereafter v-RSD), a consequence of a new class of RSD induced by the global dynamics of the region containing the void.
An analytical description can be obtained from an RSD displacement applied to the void centre:
\begin{equation}
    s_{\rm v \perp} = r_{\rm v \perp}, \\
    s_{\rm v \parallel} = r_{\rm v \parallel} + v_{\rm v \parallel} \frac{1 + z}{H(z)}.
	\label{eq:vrsd}
\end{equation}
Here, $\mathbf{r}_{\rm v} = (r_{\rm v \perp}, r_{\rm v \parallel})$, $\mathbf{s}_{\rm v} = (s_{\rm v \perp}, s_{\rm v \parallel})$ and $\mathbf{v}_{\rm v} = (v_{\rm v \perp}, v_{\rm v \parallel})$ denote the real-space position vector of the centre, its redshift-space counterpart, and the void net velocity \citep{lambas_sparkling_2016}, respectively.

\section{Void size function}
\label{sec:vsf}

The void size function quantifies the comoving number density of cosmic voids as a function of their effective radii.
It is analogous to the mass function of dark matter haloes.
Therefore, it can be modelled using the excursion set formalism combined with the spherical evolution of density perturbations \citep{svdw}.

Following this approach, \cite{abundance_jennings} derived a \textit{volume conserving model}, which assumes that the total volume occupied by cosmic voids is conserved in the transition from linear to non-linear regime:
\begin{equation}
    \frac{dn_{\rm v}}{d\mathrm{ln}R_{\rm v}^{\rm rs}} =
    \frac{f_{\mathrm{ln}\sigma}(\sigma)}{4/3 \pi (R_{\rm v}^{\rm rs})^3} \frac{d\mathrm{ln}\sigma^{-1}}{d\mathrm{ln}R_{\rm v, L}^{\rm rs}}
    \frac{d \mathrm{ln} R_{\rm v, L}^{\rm rs}}{d \mathrm{ln} R_{\rm v}^{\rm rs}}.
    \label{eq:vsf_vdn}
\end{equation}
Here, $\sigma$ is the square root of the mass variance (dependent on the matter power spectrum), whereas $f_{\mathrm{ln}\sigma}$ is the fraction of the Universe occupied by voids:
\begin{equation}
    f_{\mathrm{ln}\sigma} = 2 \sum_{j=1}^{\infty} j\pi \chi^2 \mathrm{sin}(j\pi \mathcal{D}) \mathrm{exp} \left[ - \frac{(j\pi \chi)^2}{2} \right],
    \label{eq:flnsig}
\end{equation}
where in turn, $\chi = \mathcal{D}\sigma / |\Delta_{\rm v}^{\rm L}|$ and $\mathcal{D} = |\Delta_{\rm v}^{\rm L}| / (|\Delta_{\rm v}^{\rm L}| + \Delta_{\rm c}^{\rm L})$.
$\Delta_{\rm c}^{\rm L}$ and $\Delta_{\rm v}^{\rm L}$ represent the two barriers needed in the excursion set to take into account both the void-in-cloud and void-in-void modes.
The former is expected to vary within $1.06 < \Delta_{\rm c}^{\rm L} < 1.686$, the moments of turn-around and collapse, respectively.
The latter is expected to be the moment of shell-crossing in the expansion process.
Finally, $R_{\rm v, L}^{\rm rs}$ is the linear radius predicted by theory.
It can be related to its non-linear counterpart by a constant factor: $R_{\rm v}^{\rm rs} = \gamma R_{\rm v, L}^{\rm rs}$, with $\gamma = (1 + \Delta_{\rm v}^{\rm NL})^{-1/3}$.

The shell-crossing moment for voids identified from the underlying total matter distribution corresponds to the non-linear value $\Delta_{\rm v}^{\rm NL} = -0.8$, or equivalently, to the linearly extrapolated value $\Delta_{\rm v}^{\rm L} = -2.71$.
However, in practice, we can only define voids from the observed galaxy distribution.
In this case, it is not trivial to relate this event to a specific density threshold.
This can be overcome by the procedure suggested by \cite{abundance_bias_contarini}.
First, we define an observational density threshold as low as possible: $\Delta_{\rm v, g}^{\rm NL}$ (usually less than $-0.7$).
Then, we relate this value to the one corresponding to the total matter density field by means of the bias parameter: $\Delta_{\rm v}^{\rm NL} = \Delta_{\rm v, g}^{\rm NL}/b$.
Finally, we derive its corresponding linear value with the fitting formula provided by \cite{bernardeau}: $\Delta_{\rm v}^{\rm L} = C[1-(1+\Delta_{\rm v}^{\rm NL})^{-1/C}]$, with $C=1.594$.

The last aspect that we need to consider is to relate the redshift and real-space void radii by means of the t-RSD and AP volume effects (Eq.~\ref{eq:q_ap_rsd}).
These are the theoretical foundations for designing a cosmological test with the void size function.

\section{Void-galaxy cross-correlation function}
\label{sec:vgcf}

\begin{figure}[!t]
\centering
\includegraphics[width=85mm]{boss.pdf}
\caption{
Plane-of-sky and line-of-sight projections of the void-galaxy cross-correlation function measured from BOSS DR12 data and from the simulated MultiDark Patchy mocks.
}
\label{fig:boss}
\end{figure}

In \cite{aprsd_correa}, we provided an empirical fitting expression for the real-space correlation function suitable for voids above the shot-noise level:
\begin{equation}
  \xi(r) =
    \begin{cases}
      Ar - 1 & \text{if} ~ r < r_\mathrm{wall}\\
      -\xi_0 \left[ \left( \frac{r}{r_0} \right)^{-3} + \left( \frac{r}{r_0} \right)^{-\alpha} \right] & \text{if} ~ r \geq r_\mathrm{wall},
    \end{cases}
  \label{eq:dens_model}
\end{equation}
where $r$ denotes distance to the void centre, and $r_{\rm wall}$ is a characteristic scale separating their inner and outer parts.
This model has three free parameters: $\xi_0$, $r_0$ and $\alpha$ (the slope A can be derived from $\Delta_{\rm v, g}^{\rm NL}$).

To study the distortion effects, it is more convenient to represent the correlation function as a 2D contour map with axes perpendicular and along the LOS: $\xi(s_\perp,s_\parallel)$.
The following expressions relate the real and redshift-space separation vectors between a void-galaxy pair: $\mathbf{r} = (r_\perp, r_\parallel)$ and $\mathbf{s} = (s_\perp, s_\parallel)$, via their relative velocity: $\mathbf{v} = (v_\perp, v_\parallel)$:
\begin{equation}
    s_\perp = r_\perp, \\
    s_\parallel = r_\parallel + v_\parallel \frac{1+z}{H(z)}.
	\label{eq:rsd_coord}
\end{equation}

The t-RSD effect can be modelled with the so-called \textit{Gaussian streaming model} \citep{clues2}:
\begin{equation}
    \resizebox{0.9\hsize}{!}{
    $1 + \xi(s_\perp, s_\parallel) =
    \int_{-\infty}^{\infty} \frac{1 + \xi(r)}{\sqrt{2 \pi}\sigma_{\rm v}}
    \mathrm{exp} \left[- \frac{(v_\parallel - v(r)\frac{r_\parallel}{r})^2}{2\sigma_{\rm v}^2} \right]
    \mathrm{d}v_\parallel$,
    }
    \label{eq:gsm}
\end{equation}
where $v(r)$ is a profile characterising the velocity field $\mathbf{v}$, and $\sigma_{\rm v}$ the velocity dispersion, taken as a free parameter.
The former can be derived analytically in the linear regime:
\begin{equation}
    v(r) = - \frac{H(z)}{(1+z)} \frac{f(z)}{b} \frac{1}{r^2} \int_0^r \xi(r') r'^2 dr'.
	\label{eq:velocity}
\end{equation}

Actually, the distance between a void-galaxy pair must be inferred from an angle $\theta$ on the plane of the sky (hereafter POS) and a redshift separation $\zeta$ along the LOS.
This involves the following cosmological equations:
\begin{equation}
    s_\perp = D_{\rm M}(z) \theta, \\
    s_\parallel = \frac{c}{H(z)} \zeta,
	\label{eq:ap_coord}
\end{equation}
where $c$ denotes the speed of light.
We consider in this way the AP effect.

Now, we will follow the methodology that we developed in \cite{aprsd_correa} by considering the POS and LOS projections of the correlation function.
They are obtained respectively by projecting it towards the $\perp$-axis in a given redshift range, and towards the $\parallel$-axis in a given angular range.
It is important to consider that correlations are measured via a binning scheme, where several scales are mixed.
A generic bin is a cylindrical shell oriented along the LOS with the following dimensions: $s_\perp^{\rm int}$ (internal radius), $s_\perp^{\rm ext}$ (external radius), $s_\parallel^{\rm low}$ (lower height) and $s_\parallel^{\rm up}$ (upper height).
Then, the correlation estimator for such a bin is given by:
\begin{equation}
    \xi_{\rm bin} = 2
    \frac{ \int_{s_\parallel^{\rm low}}^{s_\parallel^{\rm up}} ds_\parallel \int_{s_\perp^{\rm int}}^{s_\perp^{\rm ext}} s_\perp [1 + \xi(s_\perp,s_\parallel)] d s_\perp }
    { ((s_\perp^{\rm ext})^2-(s_\perp^{\rm int})^2) (s_\parallel^{\rm up}-s_\parallel^{\rm low}) } - 1.
    \label{eq:scales}
\end{equation}
The projected correlations are special cases with the following limits: (i) $s_\parallel^{\rm low} \rightarrow 0$ and $s_\parallel^{\rm up} \rightarrow \mathrm{PR_\parallel}$ for the POS correlation, and (ii) $s_\perp^{\rm int} \rightarrow 0$ and $s_\perp^{\rm ext} \rightarrow \mathrm{PR_\perp}$ for the LOS correlation.
Here, $\mathrm{PR_\perp}$ and $\mathrm{PR_\parallel}$ are the corresponding projection ranges.

This model still needs to consider the v-RSD effect.
This is work in progress.
It can be modelled by incorporating information about the void net velocity distribution into Eq.~(\ref{eq:gsm}).
But there is more.
In \cite{zvoids2_correa}, we found that there is an extra source of anisotropic patterns in the correlation function: the intrinsic ellipsoidal nature of voids.
This is the \textit{ellipticity effect} (e-RSD).
Currently, we are working in an extension of the spherical void finder that allows to define voids as free-shape underdense regions of maximal volume (Paz et al, in prep.).
These are the theoretical foundations for designing a cosmological test with the correlation function.

By way of illustration\footnote{More details can be found at Dr. Correa's PhD thesis \url{https://rdu.unc.edu.ar/handle/11086/21041}.}, Fig.~\ref{fig:boss} shows the POS and LOS projections of the correlation function measured from the spectroscopic survey BOSS DR12 \citep{boss}.
We also show the results obtained from the simulated MultiDark Patchy mock surveys \citep{patchy_kitaura}.
The consistency between both data sets is a promising result.
The error bars were calculated from the covariance matrix obtained from these mocks.
In both cases, we selected a void sample with sizes between $30 \leq R_\mathrm{v}^{\rm zs}/\mathrm{Mpc} \leq 35$.
Correlations were treated directly in terms of the angular and redshift coordinates, $\theta$ and $\zeta$, hence it is not necessary to assume a fiducial cosmology for the posterior analysis.
We took the following projection ranges: $\mathrm{PR}_\perp = 0.0232$ and $\mathrm{PR}_\parallel = 0.0131$, chosen to be approximately equal to $30~\mathrm{Mpc}$ in the fiducial cosmology.
This allows us to capture effectively the anisotropic patterns in both directions.
The LOS projection is much more sensitive to the distortion effects than the POS projection.
Hence, the large differences between the two confirm the fact that it is possible to detect and analyse all the redshift-space effects addressed in this work with high precision.

\section{Conclusions}
\label{sec:conclusions}

Cosmic voids are promising cosmological probes.
However, it is mandatory a full description of all the redshift-space effects that affect observational measurements in order to obtain unbiased cosmological constraints.
Void systematics can be summarised as follows: (i) the AP effect, (ii) the expansion effect (t-RSD), (iii) the off-centring effect (v-RSD), and (iv) the ellipticity effect (e-RSD).
We made a description in a nutshell of these effects and laid the theoretical foundations for designing reliable cosmological tests with the void size function and the void-galaxy cross-correlation function.

\bibliographystyle{baaa}
\small
\input{articulo.bbl}

\end{document}