\section{Introduction}
The ever growing use of gestures in advanced technologies, such as augmented or virtual reality environments, requires more and more understanding of the different levels of representation of gestures, from meanings to motion characterized by causal physical and biological phenomena. This is even more true for skilled and expressive gestures, or for communicative gestures such as sign language gestures, both involving high level semiotic and cognitive representations, and requiring extreme rapidity, accuracy, and physical engagement with the environment.

In this paper we highlight the tight connection between the high-level and low-level representations involved in signed languages. Signed languages are fully-formed languages that serve as a means of communication between deaf people. They are characterized by meanings: they have their own rules of compositional elements, grammatical structure, and prosody; but they also include multimodal components that are put into action by movements.
They are indeed by essence multi-channel, in the sense that several modalities are implicated when performing motion: body, hands, facial expression, gaze direction, acting independently but participating all together to convey meaningful and discriminative information. In signed language storytelling for example, facial expressions may be used to qualify actions, emotions, and enhance meaning.

We focus on data-driven models, which are based on observations of real signed language gestures, using captured motion or videos. Motion capture allows us to find relevant representations that encode the main spatio-temporal characteristics of gestures. In the same way, analyzing videos may lead to annotations where significant labels indicate the morpho-syntactic nature of elements composing gestures, and may constitute a starting point for determining phonetic structures. By combining both pieces of information, motion capture data and videos, we may also extract accurate low and high level features that help to understand sign language gestures. We believe that data-driven methods, incorporating constraints extracted from observations, significantly improve the quality and the credibility of the synthesized motion. To go beyond, we propose this synthesis-by-analysis method, corrected by a perceptual evaluation loop, to model the underlying mechanisms of signed language gesture production.\\

In the remainder of the paper, we propose a guideline aiming at characterizing the role of sensory-motor information for signed language understanding and production, based on the motor theory of sign language perception. We then provide a general methodology for analyzing, synthesizing, and evaluating signed language gestures, where different sensory data are used to extract linguistic features and infer motor programs, and to determine the action to perform in a global action-perception loop. The different concepts and models are illustrated by related works, both from the points of view of signed language linguistics and movement science communities.

After describing related works in the next section, we propose sign language production and perception models underlying the motor theory of sign language perception. A methodology is then proposed to highlight how this theory may be exploited in both theoretical sign language research and motion sciences.

\section{Related Works}

There are two main approaches in modeling and producing sign language gestures, that are addressed differently in the different research communities: the first one, addressed by the signed language linguists, concerns the formation of the meaning from observations; the second one, addressed by motion science researchers, is related to motion generation and recognition from high-level sign descriptions. Most of the time, these two approaches are considered separately, as the two research communities do not share the same tools and methods. \\

Linguistic researchers work on signed languages from some observation of natural utterances, most often through video data: they build theories describing the mapping between these observations and linguistic components (phonetics, phonological structures, etc.).
The resulting models are still widely debated in the sign language community, and usually, motion characterization is not seen as a prime objective for elaborating phonological model \cite{Hulst2001} or phonetic model \cite{JohnsonLiddell10}.  In order to validate their observations and analysis, they need better knowledge of movement properties: kinematic invariants within signs and between signs, physical constraints, etc. Invariant laws in movements are discussed in \cite{Gibet03}. \\

Movement researchers on the other hand (bio-mechanicians, neuroscientists, computer animators, or roboticians) try to build simulation models that imitate real movements. Their approach consists, from high-level descriptions (planning), of specifying a sequence of actions as a procedural program.
They need to acquire better knowledge of the rules governing the system behavior, such as syntactic rules or parameterization of the sign components according to the discourse context. The next problem consists of interpreting these rules using specific computer languages (from scripting languages to procedural or reactive languages), and traducing them into sensory-motor processes underlying the physical system that produce movement.

Most of the works in this area focus on the expressivity of the high-level computer languages, using descriptive or procedural languages, for example the XML-based specification language called SiGML \cite{Kennaway2003} which is connected to  the HamNoSys \cite{Prillwitz1989} notation system, and interpreted into signed language gestures using classical animation techniques. A more exhaustive overview of existing systems using virtual signers technology can be found in \cite{Gibet11}. For these kinds of applications involving signed language analysis, recognition, translation, and generation, the nature of the performed gestures themselves is particularly challenging.

Alternatively, data-driven animation methods can be substituted for these pure synthesis methods. In this case the motions of a real signer are captured  with different combinations of motion capture techniques.
Though these methods significantly improve the quality and credibility of animations, there are nonetheless several challenges to the  reuse of motion capture data in the production of sign languages. Some of them
are related to the spatialization of the content, but also to the rapidity and precision required in motion performances, and to the dynamic aspects of movements. All these factors are responsible for phonological inflection processes. Incorrectly manipulated, they may lead to imperfections in the performed signs (problems in timing variations or synchronization between channels) that can alter the semantic content of the sentence.
A detailed discussion on the important factors for the design of virtual signers in regard to the animation problems is proposed in  \cite{Courty10}. \\

Little has been done so far to determine the role of sensory-motor activity for the understanding (perception and production) of signed languages. The idea that semantic knowledge is embodied into sensory-motor systems has given rise to many studies, bringing together researchers from domains as different as cognitive neuroscience and linguistics, but most of these works concern spoken languages.
This interaction between language and action are based on different claims such as:
\begin{itemize}
\item{imagining and acting share the same neural substrate \cite{Goldman06}};
\item{language makes use in large part of brain structures akin to those used to support perception and action \cite{Arbib08}.}
\end{itemize}

Among these recent research interests, some researchers share the idea that motor production is necessarily involved in the recognition of sensory (audio, visual, etc.) encoded actions; this idea echoes what is called the \emph{motor theory of speech perception} which holds that the listener recognizes speech by activating the motor programs that would produce sounds like those that are being heard \cite{Liberman67}. Within this theory, sensory data are auditory or visual clues (mouth opening), and the motor actions are vocal gestures (movements of the vocal tract, tongue, lips, etc.).

This theory can be easily transposed to sign languages, and we will call it the \emph{Motor Theory of Sign Language Perception}. In this case too, the linguistic information is embodied into sensory-motor processes, where sensory data may be visual clues (iconic gestures, classifiers), or perception of action (contact between several body parts, velocity or acceleration characteristics, etc.).

\section{The Motor Theory of Sign Language Perception}

All the evidence briefly reported in the previous section tends to show that perception and production of language utterances are closely related. It remains to describe or model this relationship. At the light of this evidence, the motor theory of speech perception, which states that what we perceive is nothing but the movement of the articulatory system (body movements), suggests that part of conceptual and language structures are encoded at motor program levels, e.g. as a sequence of motor actions allowing to produce the desired sensory (or perceptive) effect.

Similarly to the motor theory of speech perception, the motor theory of sign language perception that we promote in this paper claims that what we perceive is the movement of body articulators, and that the encoding and decoding of linguistic information should be partly addressed at motor program level characterizing the movement intention. Furthermore, if we accept the idea that the motor program level  is where the linguistic cues are encoded, then the motor theory of perception leads to consider that we can infer motor programs from observed sensory cues only (motor act). We call this inference an inversion process since its purpose is to deduce the cause from the consequence (sensory observation).\\

Therefore, if we go further in the modeling of these concepts, we assume that the motor theory of sign language perception is based on two inversion mechanisms, one for sign language production, and the other one for sign language perception. These mechanisms will be used as part of encoding and decoding processes of linguistic units. By linguistic units we mean here phonetic and phonological elements specific to sign languages.\\

The first inversion process for sign language production, also called encoding process, is represented in Figure~\ref{fig:sensorymotor2}. It is a closed-loop system, where the signer uses sensory information to produce the desired actions corresponding to a specific motor program. The signer performing gestures perceives the environment through many sensory cues: he can view his interlocutor, and also the entities positioned in the signing space (spatial targets); he may also capture auditive, tactile (perception of touch), proprioceptive (perception of muscles and articulations), and kinesthetic clues (perception of velocity, acceleration, etc.) from its own body movements. These sensory cues are then inverted to provide motor commands that modify the current action applied to the musculor-skeleton system. When producing sign language gestures, the linguistic information is also exploited to generate a sign language utterance which is then translated into a motor program.

In the context of sign language synthesis, the motor programs may be represented by a sequence of goals, as for example key postures of the hand, or targets in hand motion, or facial expression targets. These targets are then interpreted into continuous motion, through an inverse kinematics or dynamics solver \cite{Gibet1994}, \cite{Lebourque99}, \cite{GibetM03}.\\

\begin{figure}[tb]
\centering{}
\includegraphics[width=0.8\textwidth]{Sensorymotor2}
\caption{Sign language production: encoding from motor program and linguistic information}
\label{fig:sensorymotor2}
\end{figure}

The second inversion process used for gesture perception, also called decoding process, is represented in Figure~\ref{fig:sensorymotor3}. From the observation of a signer, it consists in extracting multi-sensorial cues, and then to simultaneously infer motor programs (allowing to reproduce the detected sensory cues), and extract linguistic information.

Our approach to sign language perception can be divided into two kinds of analysis studies. The first one consists of a linguistic analysis that tries to extract phonetic or phonological features from the observation of signers. The second one consists in finding invariants or motor schemes in the data, above which one can build linguistic knowledge.

This last approach, inspired from the neuroscience community, may exploit statistical tools in order to extract some regular features or schemes embodied to motion data.

\begin{figure}[tb]
\centering{}
\includegraphics[width=0.8\textwidth]{Sensorymotor3}
\caption{Sign language perception: decoding for inferring motor program and extracting linguistic information}
\label{fig:sensorymotor3}
\end{figure}

\section{Methodology: Sign Language Production and Perception}

In practice, production and perception are closely linked in a language communication context. In order to study jointly both mechanisms, we propose a general and experimental methodology based on an analysis (perception) / synthesis (production) approach, depicted in Figure~\ref{fig:methodology}. It contains the following three building blocks.

\begin{itemize}
\item{i) The Analysis block refers to the perception or decoding aspect of the methodology. It uses observed information from simultaneously captured motion data and videos. It is based on hypothesis related to the linguistically encoded structure of signs, and the motor programs underlying the performed gestures.
In practice, given the different nature of information that should be encoded (symbolic and numerical), it is more efficient to process and store data in two different structures, namely a semantic database for linguistic annotations, and a raw database for motion capture data;}

\item{ii) The Synthesis block covers the production or encoding aspect of the methodology. It is composed of a sensory-motor animation system which uses both a scripting process expressing a new utterance and  the corresponding motor program that uses pre-recorded motion chunks. Moreover, a 3D rendering engine allows to visualize the virtual signer performing the signs;}

\item{iii) The Evaluation block makes possible the evaluation of the analysis hypothesis, at the light of the synthesized gestures. Deaf experts or sign language signers may indeed qualify the different performances (quality of the gestures, realism, understandability), and propose some changes of the models and sub-segment structures including motor program schemes. We conjecture that during evaluation, based on their own sensory-motor inversion loop, experts or signers are implicitly able to validate or invalidate the synthesized motor performance and subsequently the hypothesis that have been made for the elaboration of the motor programs.}\\
\end{itemize}

\begin{figure}[tb]
\centering{}
\includegraphics[width=1.0\textwidth]{methodology1}
\caption{Analysis, synthesis, and evaluation methodology}
\label{fig:methodology}
\end{figure}

This analysis-by-synthesis methodology requires to bring together researchers from different communities.
Preliminary work has been undertaken on the basis of the collected data within the project \emph{SignCom} \cite{signcom}. Some models and results underlying the former methodology are presented below, in the context of analysis of French sign language corpora, and data-driven synthesis. The use of  3D avatars driven by semantic and raw motion databases also allows us to go further the restrictions of videos, and to evaluate the feasibility and understandability of the models.

\paragraph{\bf Corpus and database}

 The observational data are composed of 50 minutes of sign language motion captured data which gather data recorded with 43 body markers, 41 facial markers, and 12 hand markers, and videos of the same sequences recorded with one camera. Some of the challenges posed by the corpus creation and the capture of heterogenous data flows are detailed in \cite{DuarteGibet2010b} and \cite{DuarteGibet2010}. It should be noted that the choice of the corpus (choice of the thematics, limited vocabulary, choice of lexical and non-lexical signs, motion forms, etc.) may potentially influence the analysis and synthesis processes.

\paragraph{\bf Analysis} The previous corpus has been analyzed and indexed by sign language experts: we separated the linguistic indexing from the raw motion indexing.

 \begin{itemize}
\item {\bf The linguistic indexing} is provided by annotations performed by sign language linguists associated to deaf people.
Signs are generally decomposed into various components, such as location, handshape, and movement as proposed by Stokoe \cite{Stokoe1960}.  Since then, other linguists have expanded and modified Stokoe's decompositional system, introducing wrist orientation, syllabic patterning, etc. \cite{JohnsonLiddell10}. However, signed languages are not restricted to conveying meaning via the configuration and motion of the hand, but instead involve the simultaneous use of both manual and non-manual components. The manual components of signed language include hand configuration, orientation, and placement or movement, expressed in the signing space (the physical three-dimensional space in which the signs are performed). Non-manual components consist of the posture of the upper torso, head orientation, facial expression, and gaze direction.\\

Following this structural description of signs, we annotate the selected corpus, identifying each sign type found in the video data with a unique gloss so that each token of a single type can be easily compared, and segmenting the different tiers composing the signs by exploiting grammatical and phonological models \cite{JohnsonLiddell10}. The structure of the annotation scheme is characterized by:
\begin{itemize}
\item{a spatial structure, defined by several tiers and a structural organization by gathering several channels;}
\item{a temporal structure, resulting from manual and semi-automatic segmentation, allowing transitions / strokes labelling;}
\item{a manual labeling with elements and patterns borrowed from linguists; we have followed the phonetics model of Johnson and Liddell \cite{JohnsonLiddell10}.\\
}
\end{itemize}

This annotation scheme allows to match motion data and phonetic structure, as
shown in figure \ref{fig:annotation}, thus providing ways to index synchronously the motion to the
phonetic tiers.\\

\begin{figure}[tb]
\centering{}
\includegraphics[width=1.0\textwidth]{annotation}
\caption{A proposed annotation scheme, matching motion data and phonetic structure}
\label{fig:annotation}
\end{figure}

\item {\bf The motion indexing} is based on motion processing.
Sign language data have already been studied, following different approaches. We first identified phonological items, described as sequences of motion targets and handshape targets  \cite{Lebourque99}, and used motor control rules, as the ones described in \cite{Gibet03}, to produce realistic hand motion.

Using motion captured data from French sign language corpora, we have also developed specific analysis methods that have led to the extraction of low-level or high-level motor schemes. We first automatically segmented handshape sequences \cite{Heloir05}, or hand movements that may be correlated to motor control laws \cite{Gibet08}. Secondly, statistical analysis have been conducted to characterize the phasing between hand motion and handshape \cite{Heloir07}, or to categorize hand motion velocity profiles within signs or during transitions between signs \cite{Duarte2010} (controlled, ballistic, and inverse-ballistic movements).
Similar works have been carried out to show the temporal organization in Cued Speech production \cite{AttinaBCO04}.

\end{itemize}

We also implemented a two-levels indexed database (semantic and raw data) \cite{Awad09}. From such database, it will be possible to go further in the statistical analysis, and thus extract other invariants features and motor schemes, and to use them for re-synthesis. \\

\paragraph{\bf Synthesis} Conversely, using these enriched databases to produce new utterances from the corpus data remains challenging regarding the hypothesis derived from the analysis processes. Different factors may be encoded into the motor program driving the synthesis engine, such as the dynamics of the gestures (velocity profiles, etc.), the synchronization between the channels, or the coarticulation effects by using the sequence of pre-identified targets.

The multichannel animation system for producing utterances signed in French Sign Language (LSF) by a virtual character is detailed in \cite{Gibet11}.  \\

\paragraph{\bf Evaluation}
Concerning evaluation issues, the idea is not so much to evaluate the signing avatar, but to evaluate the different hypotheses related to the decoding of signs, from the observation of sign language performances, and to the corresponding encoding of signs within the synthesis system. With this analysis-by-synthesis approach it is possible to possibly refine the different hypothesis and to help understanding the coupled production-perception mechanisms.

Currently, the research community focuses on the usability of the avatar. The evaluation process can be divided into two processes: i) the evaluation of the acceptability of the avatar, which can be measured by human-likeness, coordination, fluidity, realism of the three-dimensional rendering; ii) the evaluation of the understandability of the avatar, which requires the recognition of signs by measuring the precision of the signs, the co-articulation effects, etc., and measuring the level of recognition of the sentences and the story. A preliminary evaluation has been performed in \cite{Gibet11}. Understanding, characterizing more thoroughly the production and perception of sign language in the context of a motor theory of perception is a natural and promising perspective that should be carried out in the near future.

\section{Conclusion}
This paper promotes a motor theory of sign language perception as a guideline for the understanding of linguistic encoding and decoding of sign language gestures. According to this theory, what we perceive is nothing but the movement of the body's articulators. In other words, this assumption states  that the linguistic knowledge is mapped onto sensory-motor processes. Such an a priori statement relies on two main hypothesis: firstly, we are able to infer motor data from sensory data through a sensory motor inversion process, and secondly, elements of linguistic information are somehow encoded into motor programs. A methodology straightforwardly derived  from these two hypothesis and based on a so-called analysis-by-synthesis loop is detailed. This loop, through a perceptive evaluation carried out by sign language experts, allows to validate or invalidate hypothesis on linguistic encoding at motor program levels.
Although much work remains to be done to validate the methodology and the motor theory of sign language perception itself, its feasibility and practicality has been demonstrated in the context of French sign language corpora analysis and data-driven synthesis.

It should be noted that the study of sign languages is a favorable field for validating motor theories of perception, since it is rather easy to infer the articulators' movements from sensory data (captured data and videos). However, this promising interdisciplinary research orientation requires the involvement of sign language linguists, deaf signers, neuroscientists and computer scientists.

\subsection*{Acknowledgments}
This work has been partially supported by the $SignCom$ project, an Audiovisual and Multimedia project by the French National Research Agency (ANR).

\bibliographystyle{unsrt}


\section{Introduction}

The non-perturbative gluon and ghost propagators may be obtained from their Schwinger-Dyson Equations (SDEs). They are interesting quantities that provide important information when studying confinement and dynamical chiral symmetry breaking in Quantum Chromodynamics (QCD). They are necessary inputs for studying bound states and amongst other things, can be used to obtain non-perturbative predictions for the running coupling $\alpha_s$. A number of early studies~\cite{Brown:1988bm,Brown:1988bn,Atkinson:1997tu,Bloch:2003yu,Alkofer:2003jr} found various singularities in the vanishing $p^2$ limit for the non-perturbative propagator dressings. However, more recent results from a variety of methods have found the dressings to be finite~\cite{Cucchieri:2007md,Aguilar:2008xm,Fischer:2008uz,Bogolubsky:2009dc,Binosi:2009qm,Cucchieri:2011ga,Cucchieri:2011um}.

In a recent study \cite{Pennington:2011xs}, the gluon and ghost propagator dressings, in the absence of quarks, were studied. The SDEs were solved self-consistently using a range of approximations for the vertices. Emphasis was given to how well these quantities are determined -- particularly in the physical region around 0.1 and 1 GeV. It turns out that the vertices are indeed very important if numerically precise results are desired.

Landau gauge is used, since for this kind of problem it is the most appealing theoretically. The non-perturbative corrections to the ghost-gluon vertex are simplest in Landau gauge and statements regarding confinement are typically formulated in Landau gauge also. Furthermore, there are several lattice results to which we may compare.

The dressing functions that we solve for are related directly to the propagators, which for the gluon reads,
\begin{equation}
D_{\mu\nu}(p)= \frac{\ensuremath{\mathcal{G}\ell}(p^2)}{p^2}\Bigl(g_{\mu\nu}-\frac{p_\mu p_\nu}{p^2}\Bigr)
\label{eq_gluonprop}
\end{equation}
and similarly for the ghost,
\begin{equation}
D(p)=-\frac{\ensuremath{\mathcal{G}h}(p^2)}{p^2}
\end{equation}
where $D_{\mu\nu}(p)$ is the full gluon propagator in Landau gauge and $D(p)$ is the ghost propagator. The dressing functions $\ensuremath{\mathcal{G}\ell}$ and $\ensuremath{\mathcal{G}h}$ contain all of the non-perturbative physics of these two Green's functions. The simple transverse dressing of the gluon in Eq.~(\ref{eq_gluonprop}) is due to the Slavnov-Taylor identity for the propagator and is an important feature of the gauge invariance of the theory.

First we present the solutions of the ghost equation that are obtainable using a fixed gluon input. These solutions then provide a natural starting point for investigating the coupled system where simultaneous solutions of both propagator dressings are found. During the course of this work it became apparent that many simple vertices do not admit self-consistent solutions. We test a range of vertices with a range of solutions. A preferred system motivated by theoretical arguments is then selected. We conclude by comparing to the extant lattice data.

\section{The Ghost Equation}

Using a fixed gluon input we may solve the Ghost SDE alone and investigate its sensitivity to a range of input vertices. This is useful because this type of SDE is very simple to solve, and it teaches us what to expect when solving the more complicated, coupled equations self-consistently. In particular, the choice of ghost-gluon vertex applied here does have an effect, and understanding this is useful when solving the coupled system of gluons and ghosts

In order to study the ghost equation in isolation we are required to provide a gluon input, we use the following model~\cite{Aguilar:2010gm},
\begin{equation}
\ensuremath{\mathcal{G}\ell}(p^{2})=\frac{p^{2}}{m^{2}+p^{2}\left(1+\frac{11}{12\pi}\frac{N_{c}g^{2}}{4\pi}\mathrm{Log}\left(\frac{p^{2}+ m^2}{\mu^2}\right)\right)^{\frac{13}{22}}}\,.
\label{eq_gluonmodel}
\end{equation}
In this form the gluon contains a mass term $m$, which is typically $\mathcal{O}(\Lambda_{\mathrm{QCD}})$ in lattice and other SDE studies. The remaining terms arise in order to reproduce the one-loop behaviour. This is similar to many contemporary representations of the gluon dressing function, and a closely related form has recently been applied to a phenomenological study of hadron physics observables~\cite{Qin:2011dd,Qin:2011xq}. This form reproduces the leading resummed logarithm one finds from perturbation theory at large momenta.

\begin{figure}[tbh]
  \begin{center}
  \includegraphics[width=0.2\textwidth]{./Fig3}
  \end{center}
  \caption{The ghost-gluon vertex indicating the momentum definition we adopt, the outgoing ghost momentum is $q$ and the gluon momentum is $k$.}
  \label{fig_gccdiag}
\end{figure}

The next step is to specify the ghost-gluon vertex which in principle is all that is required in order to solve the ghost equation given a fixed gluon input. The full ghost-gluon vertex has the form,
\begin{equation}
\Gamma_\mu=igf^{abc}\left(\alpha(k,p,q) q_\mu + \beta(k,p,q) k_\mu \right)
\end{equation}
where $\alpha(k,p,q)$ and $\beta(k,p,q)$ are the non-perturbative dressing functions and the momenta are defined in Fig.~\ref{fig_gccdiag}. Initially we make use of the bare vertex in place of the fully dressed one, where $\alpha=1$ and $\beta=0$. This is motivated by Taylor's theorem~\cite{Taylor:1971ff} which constrains the sum of the functions to unity at vanishing incoming ghost momentum: $p=0$. When solving for the gluon, the following extended vertices will be used,
\begin{equation}
\Gamma_{\mu}^{(1)}=igf^{abc}\left(q_{\mu} - k_{\mu}\frac{k.q}{k^{2}}\mathcal{F}_\mathrm{IR}(k,p,q)\right),
\label{eq_gcctransv}
\end{equation}
\begin{equation}
\Gamma_{\mu}^{(2)}=igf^{abc}\left(q_{\mu} - p_{\mu}\frac{k.q}{k^{2}}\mathcal{N}_{\mathrm{IR}}\mathcal{F}_{\mathrm{IR}}(k,p,q)\right),
\label{eq_gcctt}
\end{equation}
where $\mathcal{F}_{\mathrm{IR}}=1$ for small momenta and $\mathcal{F}_{\mathrm{IR}}=0$ for large momenta. $\mathcal{N}_{\mathrm{IR}}$ is a non-perturbative normalisation parameter.

The ghost-gluon vertex in Eq.~(\ref{eq_gcctransv}) is defined to be transverse in the small-$k$ limit, and has been used to solve these equations previously~\cite{Fischer:2008uz}. In the ghost equation, only the $q_\mu$ term survives and hence this gives identical results to the bare vertex when solving for the ghost alone. This is not the case for $\Gamma_{\mu}^{(2)}$, which is a minor modification that has been chosen in order to satisfy the Taylor condition. The additional term now vanishes when $p\to 0$, but contributes in both equations.

The final requirement is then to specify the renormalisation procedure. In the numerical method adopted here, it is most straightforward to use the momentum subtraction scheme, where the equations are subtracted from themselves at an arbitrary value of the propagator momentum. There are two choices that influence which solutions may be obtained. A subtraction at $p^2=0$ may produce a singular ghost dressing function. However a subtraction at some perturbative momentum can be applied and in some ways is better since the dressings are already well known in that region. The Schwinger-Dyson equations tell us how these evolve from the known results at large momenta into the non-perturbative region. The alternative is to make an assumption about the ghost dressing at zero momentum.

The renormalised ghost equation with a cutoff regularisation reads,
\begin{align}
\ensuremath{\mathcal{G}h}^{-1}(p^2)&=\tilde{Z_3}(\mu^2,\Lambda^2) + \Pi_{gc}(p^2,\mu^2)\\
             &=\ensuremath{\mathcal{G}h}(\mu^2) + \Pi_{gc}(p^2,\mu^2) + \Pi_{gc}(\mu^2,\mu^2)
\end{align}
in the second line we eliminate the renormalisation constant $\tilde{Z}_3(\mu^2,\Lambda^2)$. The function $\Pi_{gc}(p^2,\mu^2)$ is the ghost self-energy diagram, the details of which are given in~\cite{Pennington:2011xs}. This is the equation that is solved numerically.
To check the dependence and range of solutions available, we vary the value of $1/\ensuremath{\mathcal{G}h}(0)$. The solutions are given in Fig.~\ref{fig_mixedghostsols}. We observe two classes of solutions. With $\ensuremath{\mathcal{G}h}(0)\gtrsim2$ then the Ghost dressings in the perturbative region are practically indistinguishable. In the small $p^2$ region any value in the range $\infty>\ensuremath{\mathcal{G}h}(0)>2$ produces the same perturbative solution. Reducing $\ensuremath{\mathcal{G}h}(0)$ there is then some critical value where the solutions in the perturbative region also begin to fall. The precise value of $\ensuremath{\mathcal{G}h}(0)$ for which this occurs is dependent upon the couplings, the vertex and the gluon. All of the solutions are monotonic and decrease with increasing $p^2$. By specifying the value at $p^2=0$ then all other points are implicitly given by the equation.

\begin{figure}[tbh]
  \begin{center}
  \includegraphics[width=0.45\textwidth]{./Fig5}
  \end{center}
  \caption{Examples of ghost solutions on a log-log plot subtracting at zero momentum.  The solid curves display different ghost dressings $\ensuremath{\mathcal{G}h}(p^{2})$,
while the dashed curve is the gluon dressing $\ensuremath{\mathcal{G}\ell}(p^{2})$. Only the specified subtraction value $\ensuremath{\mathcal{G}h}(0)$ is varied between the solutions which can be read from the plot. The $\ensuremath{\mathcal{G}h}(p^{2}=0)$ value is fixed and the ghost equation solved with the depicted gluon until the ghost inputs and outputs are self-consistent.  The units of $p^{2}$ are arbitrary since we have not fixed the coupling to the physical value, but may be considered to be ${\cal O}\,(1$ GeV$^2$).}
  \label{fig_mixedghostsols}
\end{figure}

We may then switch to the other viewpoint of subtracting in the perturbative region and choose the typical perturbative condition $\ensuremath{\mathcal{G}h}(\mu^2)=1$. In doing this, none of the solutions subtracted at zero in Fig.~\ref{fig_mixedghostsols} are obtained. The ghost dressing rises from unity at $p^2=\mu^2$ to a finite value at zero momentum. It is possible to subtract at zero and fine tune to this value and obtain the same solution, although this is a special case.

Specifying $\ensuremath{\mathcal{G}h}(\mu^2)=1$ as one would do perturbatively leads to the interesting consequence that only one of these many Ghost dressings is obtained. This is distinct from singular solution and importantly no renormalisation group running connects this solution to the singular one. Since the expectation is that asymptotically free QCD at large momentum transfers is accurately described by the perturbative solution, this is the solution we favour. This effect is depicted in Fig.~\ref{fig_ghostlin}, where the solid curve is the unique solution found by imposing $\ensuremath{\mathcal{G}h}(\mu^2)=1$ whilst the remaining curves are those subtracted at zero from Fig.~\ref{fig_mixedghostsols}.

Hence we carry forward this arrangement when solving the gluon equation. It is worth noting that this same effect has been tested for gluons that vanish more or less quickly and the same qualitative features arise. Similar effects have been observed in other studies~\cite{Boucaud:2008ky,Pene:2009iq,RodriguezQuintero:2010wy,Watson:2010cn}.

\begin{figure}[tbh]
  \begin{center}
  \includegraphics[width=0.45\textwidth]{./Fig6}
  \end{center}
  \caption{The dotted curves correspond to the zero momentum subtracted solutions and the colours match Fig.~5. The dashed curve is the gluon input, $\ensuremath{\mathcal{G}\ell}(p^2)$, and the solid curve is the physically relevant solution of the ghost equation, $\ensuremath{\mathcal{G}h}(p^2)$. The units of $p^2$ are arbitrary.}
  \label{fig_ghostlin}
\end{figure}

 \begin{figure}[tbh]
   \begin{center}
   \includegraphics[width=0.45\textwidth]{./Fig7}
   \end{center}
   \caption{A comparison of two solutions normalised using the perturbative condition. The (red) dotted curve is the ghost solution, $Gh(p^2)$, with a bare vertex or transverse vertex. The (blue) dashed curve is the ghost dressing obtained using $\Gamma^{(2)}_\mu$. The (black) solid curve is the fixed gluon dressing input $\ensuremath{\mathcal{G}\ell}(p^2)$ from the model. The units of $p^2$ are arbitrary internal units.}
   \label{fig_ghosttay}
 \end{figure}

Finally we compare the result using $\Gamma_{\mu}^{(1)}$, which is identical to the bare vertex in the ghost equation, to the solution found using $\Gamma_{\mu}^{(2)}$. This comparison is shown in Fig.~\ref{fig_ghosttay} where a large difference is apparent between the two results in the non-perturbative region.

\section{The coupled Gluon and Ghost system}

The coupled system is naturally more complicated, although we already have most of the ingredients that are needed. We choose to work using only the one-loop graphs since the two-loop graphs are a considerable complication numerically. It is possible that these terms are important and in order to obtain numerically precise results it is likely that they are essential. The gluon equation has the form
\begin{align}
\ensuremath{\mathcal{G}\ell}^{-1}(p^2)&=Z_3(\mu^2,\Lambda^2) + \Pi_{2c}(p^2,\mu^2) + \Pi_{2g}(p^2,\mu^2)\\
             &=\ensuremath{\mathcal{G}\ell}^{-1}(\mu^2)+\Pi_{2c}(p^2,\mu^2)-\Pi_{2c}(\mu^2,\mu^2) + \Pi_{2g}(p^2,\mu^2) -\Pi_{2g}(\mu^2,\mu^2)\label{eq_gluonsub}
\end{align}
where in the second line we have subtracted to remove the renormalisation constant, $\Pi_{2g}$ is the gluon loop contribution to the vacuum polarisation, and $\Pi_{2c}$ arises due to the ghost loop. The ghost-gluon vertices we have already specified are sufficient for $\Pi_{2c}$. The final requirement is to specify the triple-gluon vertex. To illustrate the uncertainty we select three different vertices and check the sensitivity in the solutions.

The first we apply is a simple form that reproduces the resummed leading logarithmic running found at one-loop order in perturbation theory~\cite{Bloch:2003yu},
\begin{align}
\Gamma^{(A)}_{\mu\nu\rho}(k,p,q) = \frac{\ensuremath{\mathcal{G}h}(p^2)\ensuremath{\mathcal{G}h}(q^2)}{\ensuremath{\mathcal{G}\ell}(p^2)\ensuremath{\mathcal{G}\ell}(q^2)}\Gamma^{(0)}_{\mu\nu\rho}(k,p,q)
\end{align}
this is proportional to the bare Lorentz structure $\Gamma^{(0)}_{\mu\nu\rho}(k,p,q)$ and $p$ and $q$ are chosen to be the internally contracted legs under the loop integration. What one immediately notices about this form is that it is not symmetric under interchange of legs. Since this vertex is expected to be symmetric to all orders and since the Ward-Slavnov-Taylor Identity (WSTI) is also symmetric under the interchange of legs, a symmetric form is desirable. Hence we also apply,
\begin{align}
\Gamma^{(B)}_{\mu\nu\rho}(k,p,q)&=\Gamma^{(0)}_{\mu\nu\rho}(k,p,q)\,
\frac{1}{3}\left(\frac{\ensuremath{\mathcal{G}h}(k^{2})}{\ensuremath{\mathcal{G}\ell}(k^{2})}+\frac{\ensuremath{\mathcal{G}h}(p^{2})}{\ensuremath{\mathcal{G}\ell}(p^{2})}+\frac{\ensuremath{\mathcal{G}h}(q^{2})}{\ensuremath{\mathcal{G}\ell}(q^{2})}\right)
\end{align}
where the ghost-over-gluon form is suggested by the WSTI, and for simplicity this is chosen to be proportional to the bare Lorentz structure. The final dressing of the triple-gluon vertex we apply is a solution of the WSTI using a bare ghost-gluon scattering kernel,
\begin{align}
\Gamma^{(C)}_{\mu\nu\rho}(k,p,q)&=\frac{1}{2}\left(\frac{\ensuremath{\mathcal{G}h}(q^{2})}{\ensuremath{\mathcal{G}\ell}(p^{2})}+\frac{\ensuremath{\mathcal{G}h}(q^{2})}{\ensuremath{\mathcal{G}\ell}(k^{2})}\right)g_{\mu\nu}(k-p)_{\rho}\nonumber\\
                                &+\frac{1}{2}\left(\frac{\ensuremath{\mathcal{G}h}(k^{2})}{\ensuremath{\mathcal{G}\ell}(q^{2})}+\frac{\ensuremath{\mathcal{G}h}(k^{2})}{\ensuremath{\mathcal{G}\ell}(p^{2})}\right)g_{\nu\rho}(p-q)_{\mu}\nonumber\\
                                &+\frac{1}{2}\left(\frac{\ensuremath{\mathcal{G}h}(p^{2})}{\ensuremath{\mathcal{G}\ell}(k^{2})}+\frac{\ensuremath{\mathcal{G}h}(p^{2})}{\ensuremath{\mathcal{G}\ell}(q^{2})}\right)g_{\rho\mu}(q-k)_{\nu}\; ,
\label{eq_gggwsti}
\end{align}
now each term of the bare structure receives its own non-perturbative dressing and these have a symmetric structure. More elaborate forms are also possible and eventually something more will be required in order to obtain a triple-gluon vertex that has a consistent ghost-gluon scattering kernel with the ghost-gluon vertex as prescribed by the WSTI.

\begin{figure}[tbh]
  \begin{center}
  \includegraphics[width=0.45\textwidth]{./Fig9}
  \end{center}
  \caption{The range of solutions obtained using the possible vertex combinations. The label $(i,j)$ refers to the vertices used in obtaining the solutions corresponding to $\Gamma^{(i)}_{\mu}$ for the ghost-gluon vertex and $\Gamma^{(j)}_{\mu\nu\rho}$ for the triple-gluon vertex. The missing curve corresponds to the $\Gamma^{(1)}_{\mu}$ and $\Gamma^{(B)}_{\mu\nu\rho}$, self-consistent solutions were not obtained there. The parameters are not varied between the solutions, only the vertices. The units of $p^{2}$ are arbitrary since we have not fixed the coupling to a physical value.}
  \label{fig_vertices}
\end{figure}

The next step is to implement these vertices numerically in the ghost and gluon SDEs and to solve them self-consistently. This is done numerically in the momentum subtraction scheme, subtracting both equations at the renormalisation point $\mu$. The equations are solved iteratively and are fairly insensitive to the initial function. The model gluon in Eq.~(\ref{eq_gluonmodel}) may be used and $\ensuremath{\mathcal{G}h}(p^2)=1$ is adequate for the ghost. Convergence is achieved rapidly using a Newton-Raphson method, although a natural iterative procedure is sufficient.

In Fig.~\ref{fig_vertices} we see a broad range of gluon dressings given the different vertex inputs. The position of the peak of the gluon propagator moves around considerably. This is important because it is related to the effective gluon mass and physical quantities depend on this, as has been shown in~\cite{Qin:2011dd,Qin:2011xq}. This plot tells us that just using any old vertex is insufficient and one should appeal to all possible constraints when choosing this input.

These differences in the propagator dressings arise from differences in the vacuum polarisation integrals, which come directly from the vertices. It is informative to look inside the contributions for the gluon from each loop to disentangle the range of vertex contributions. The simplest triple-gluon vertex $\Gamma^{(A)}_{\mu\nu\rho}$ is actually quite close to the completely bare triple-gluon vertex in the small $p^2$ region, whilst the dressed symmetric triple-gluon vertices give quite different results.

\begin{figure}[!hbt]
  \begin{center}
  \includegraphics[width=0.45\textwidth]{./Fig10a}
  \includegraphics[width=0.45\textwidth]{./Fig10b}
  \end{center}
  \caption{Gluon polarization functions, $p^{2}\Pi_{2c}(p^{2})$ (positive values) and $p^{2}\Pi_{2g}(p^{2})$ (negative values). Left: IR region, Right: UV region. In the UV the curves are indistinguishable on this scale. In the IR, the ghost-loop curves for multiple solutions lie on top of each other. The bare ghost-gluon vertex, not shown here, always gives a vanishing IR contribution given an IR vanishing gluon dressing and an IR finite ghost dressing. The functions $p^{2}\Pi_{2g}(p^{2},\mu^{2})$ are labelled according to the input vertices with the contributions from $\Gamma_{\mu}^{(i)}$ and $\Gamma_{\mu\nu\rho}^{(j)}$ labelled as $(i,j)$ in the plot. The units of $p^{2}$ are arbitrary
  }
  \label{fig_pols}
\end{figure}

\noindent In order to see this we take the subtracted contribution from each loop integration and since they typically behave as $1/p^2$ for small $p^2$, we multiply them by $p^2$ in order to get a better view of the differences. This we show in Fig.~\ref{fig_pols}, where the quantities plotted are $p^2\left(\Pi_{i}(p^2,\mu^2)-\Pi_{i}(\mu^2,\mu^2)\right)$, from Eq.~\ref{eq_gluonsub}. In the right panel we see the large $p^2$ region, which is essentially determined by perturbation theory, where all contributions are the same. Differences arise in the left panel of Fig.~\ref{fig_pols} where we zoom in on the small $p^2$ region.

The negative curves are the contributions from the triple-gluon vertices. The plotted functions $p^{2}\Pi_{2g}(p^{2},\mu^{2})$ are labelled according to the input vertices with the contributions from $\Gamma_{\mu}^{(i)}$ and $\Gamma_{\mu\nu\rho}^{(j)}$ labelled as $(i,j)$ in the plot. It is evident that $\Gamma^{(A)}_{\mu\nu\rho}$ leads to a vanishing contribution when plotted in this way whilst the sensibly dressed symmetric vertices do not, they are finite and negative. The gluon dressing function is not expected to change sign and is positive in the perturbative region, so it is necessary that a positive contribution also arises to cancel that from the gluon loop. In the one-loop-only system with these triple-gluon vertices, a positive contribution must arise from the ghost-loop terms. This is indeed the case since we have self-consistent solutions for the curves that are plotted. The combination $(1,B)$ is missing since for this set of parameters it did not satisfy the
 condition. It is then not possible to find self-consistent solutions. The same condition is also why we need non-trivial ghost-gluon vertices, since the bare vertex gives a vanishing contribution when plotted in this way. This leads directly to a change of sign in the gluon dressing and no stable set of solutions can be found.

The fact that a symmetric triple-gluon dressing results in a negative contribution is very important, since it leads to the condition that the ghost loop, and hence the ghost-gluon vertex, need not be transverse alone and can work together with the other loops, as they do in the perturbative region, in order to produce a transverse gluon dressing.

\section{Comparison with Lattice QCD}

Primarily we have been motivated by theoretical issues encountered in solving the Schwinger-Dyson equations for the gluon and ghost propagators. However lattice studies exist for these quantities and the results are complementary. The possible issues that result from a applying a finite grid to Euclidean spacetime are quite different to those that we may induce here by truncation, so a comparison is a useful independent cross-check. Importantly there are lattice computations that are precise and in the pure gauge sector.

An early lattice result favoured a finite, massive solution of the gluon~\cite{Mandula:1987rh}. Following this many recent lattice studies now exist using very large lattices~\cite{Cucchieri:2007md,Cucchieri:2011ga,Cucchieri:2011um,Bicudo:2010wi}. We compare our calculations to \cite{Bogolubsky:2009dc}, which provides results in Landau gauge for both dressing functions. The qualitative behaviour is the same, with a finite ghost dressing function and a vanishing gluon propagator dressing function.

\begin{figure}[thb]
  \begin{center}
  \includegraphics[width=0.45\textwidth]{./Fig11a}
  \includegraphics[width=0.45\textwidth]{./Fig11b}
  \end{center}
  \caption{Self-consistent solutions tuned to lattice solutions, showing the dressing functions. The solid curves depict a smooth fit to the lattice data. The heavier region is where the functions are represented by lattice data and the feint region represents the natural extrapolation. The broken curves are the tuned solutions. Left: The blue curve corresponds to the gluon dressing function $\ensuremath{\mathcal{G}\ell}(p^{2})$, this vanishes as $p^{2}\to0$, the red curve corresponds to the ghost dressing function $\ensuremath{\mathcal{G}h}(p^{2})$. Right: The blue curve corresponds to $\ensuremath{\mathcal{G}\ell}(p^{2})/p^{2}$, this is $\sim10$ as $p^{2}\to0$, the red curve corresponds to the ghost dressing function again.}
  \label{fig_latdf}
\end{figure}

In Fig.~\ref{fig_latdf} we show the solutions of the equations obtained self-consistently with three different sets of parameters that reproduce different features of the lattice data. In obtaining these solutions we use the preferred set of vertices, $\Gamma^{(C)}_{\mu\nu\rho}$ from Eq.~(\ref{eq_gggwsti}) and the ghost-gluon vertex given in Eq.~(\ref{eq_gcctt}). We then tune the parameters to obtain a reasonable representation of the lattice data. We are not able to find a close fit over the whole momentum region. All three sets of parameters fail to reproduce the magnitude of the peak of the gluon propagator dressing function shown in the left panel of Fig.~\ref{fig_latdf}. It is possible that this is due to the effects of the two-loop graphs. However to be sure one would need to  perform the full dressed two-loop integrals.

Another difference is the value of the coupling in the perturbative region. In all three sets of solutions, the lattice appears to have a larger value of the coupling. This leads to a slower increase in the SDE solutions between the peak of the gluon propagator and the subtraction point. A similar effect is seen in the ghost dressings in this region. In the non-perturbative region it is possible to find similar values of the dressings simultaneously: The dashed solution is the closest we show and the effect is most visible in the right panel of Fig.~\ref{fig_latdf}.

\section{Conclusion}

We solved the Schwinger-Dyson Equations of QCD in the Landau gauge in the absence of quarks. In solving the ghost equation we found the singular ghost is excluded when a subtraction point in the perturbative region is selected. This was true for all vertices that we tested and for a range of gluon inputs although only one was shown. This agrees with earlier findings~\cite{Boucaud:2008ky}.

\begin{figure}[tbh]
 \begin{center}
  \includegraphics[width=0.45\textwidth]{Fig12}
  \end{center}
  \caption{The running coupling $\alpha_{T}(p^{2})$ is shown as the black solid curve. The running coupling $\alpha_{EC}(p^{2})$ is shown as the green solid curve. The gluon dressing $\ensuremath{\mathcal{G}\ell}(p^{2})$ is the dashed blue curve and the dotted red curve is the ghost dressing function $\ensuremath{\mathcal{G}h}(p^{2})$. The units of $p^{2}$ are now very close to GeV$^{2}$.}
  \label{fig_rc}
\end{figure}

When solving the gluon equation we found that non-trivial vertex dressings were essential in that simple vertices did not lead to self-consistent solutions. It was found that symmetric triple-gluon vertex dressings lead generically to a non-zero contribution of the gluon-loop diagram in the non-perturbative region. The main implication of this is that the ghost-loop term is not required to be transverse alone and that both terms can contribute to the transversality of the gluon propagator, as they do in the perturbative region. This combined with Taylor's theorem led us to test a simple modification of $\Gamma^{(1)}_{\mu}$, a vertex already known to give self-consistent solutions with $\Gamma^{(2)}_{\mu}$ in~Eq.~(\ref{eq_gcctt}). This modification protects Taylor's theorem whilst sacrificing the transversality of that individual loop. It is interesting to note that another ghost-gluon vertex dressing presented at this workshop leads to similar effects in the ghost equation~\cite{Boucaud:2011eh}.

A range of solutions were found by selecting a conservative set of parameters and testing the effects of using the different vertices. A wide variation was found in the physical region leading us to the suggestion that more advanced vertices are required in order to obtain numerically precise results. Proposals for new vertices in a related method specifically investigating the mass generation mechanism have been suggested~\cite{Papavassiliou:2011ty,Aguilar:2011xe}.

We may use the method we present here to produce a running coupling by considering the ghost-gluon vertex renormalisation. There are two definitions we apply, first using the standard Taylor definition which we denote $\alpha_T$ and a second form, $\alpha_{EC}$ related to the Taylor form, but removes the mass term from the gluon dressing~\cite{Aguilar:2009nf}. These curves and a set of solutions fixed using physically meaningful parameters are given in Fig.~\ref{fig_rc}. The effective gluon mass we found from this gluon propagator was $m_g=400$ MeV.

These equations produced self-consistent solutions of the ghost and gluon propagator dressings that were in qualitative agreement with lattice QCD results. Quantitative differences from this method could arise in either the vertex dressings or two-loop graphs. Regardless of precision, qualitatively all results are the same and they point to a finite gluon propagator in the non-perturbative region, corresponding to a dynamically generated gluon mass as proposed by Cornwall~\cite{Cornwall:1981zr}.

\section*{Acknowledgements}
The Institute for Particle Physics Phenomenology at Durham University, UK, its staff and students are gratefully aknowledged for providing an ideal working environment for much of this study.  DJW thanks  Jefferson Laboratory for hospitality while finalising this work. This paper has in part been authored by Jefferson Science Associates, LLC under U.\ S.\ DOE Contract No. DE-AC05-06OR23177.
This work was also supported by the U.\ S.\ Department of Energy, Office of Nuclear Physics, Contract No.~DE-AC02-06CH11357.

\bibliographystyle{jhep}


\section{Introduction}

Text-to-Speech (TTS) systems have received a lot of attention in recent years due to the great advances by deep learning, which have allowed for the popularization of voice applications such as virtual assistants. Most TTS systems were tailored from a single speaker voice, but there is current interest in synthesizing voices for new speakers, not seen during training, employing only a few seconds of speech samples. This approach is called zero-shot multi-speaker TTS (ZS-TTS) as in \citep{arik2018neural, jia2018transfer, cooper2020zero, choi2020attentron}.

ZS-TTS was first proposed~\cite{arik2018neural} by extending the DeepVoice~3~\citep{deepvoice3}. Also, \cite{jia2018transfer} explored Tacotron 2 \citep{tacotron2} using external embeddings extracted from a trained speaker encoder using a generalized end-to-end loss (GE2E)~\citep{ge2e}, resulting in a model that can generate speech, resembling the target speaker. Similarly, \cite{cooper2020zero} explored Tacotron 2 with different speaker embeddings methods. The authors showed that LDE~\citep{cai2018exploring} embeddings improved the similarity and synthesized a more natural speech for novel speakers when compared to X-vector \citep{snyder2018x} embeddings. The authors in \cite{cooper2020zero} also showed that training a gender-dependent model improves the similarity for unseen speakers.

In this context, a major issue is the similarity gap between observed and unobserved speakers during training. In an attempt to reduce this gap, Attentron~\cite{choi2020attentron} proposed a fine-grained encoder with an attention mechanism for extracting detailed styles from various reference samples and a coarse-grained encoder. As a result of using several reference samples instead of one, they achieved a better similarity for unseen speakers.

Despite the recent results, zero-shot multi-speaker TTS remains an open problem in particular concerning the difference in the quality of seen and unseen speakers. Also, current approaches rely heavily on Tacotron 2, while there is potential to improve results with the use of flow-based methods~\citep{kingma2016improved}. In this context, FlowTron~\cite{valle2020flowtron} allowed for the manipulation of multiple aspects of speech, such as pitch, tone, speech rate, cadence, and accent. Also, \cite{kim2020glow} proposed GlowTTS reaching similar quality to Tacotron 2 but with an increase in speed of 15.7 times while permitting speech velocity manipulation.

In this paper, we propose a novel method, Speaker Conditional GlowTTS (SC-GlowTTS), for zero-shot learning of unseen speakers. Our model relies on GlowTTS~\cite{kim2020glow} for the part that converts input characters to spectrograms. SC-GlowTTS uses an external speaker encoder based on Angular Prototypical loss \citep{chung2020in}, to learn speaker embedding vectors, and adapts the HiFi-GAN  \citep{kong2020hifi} vocoder to convert the output spectrograms to the waveform.
Our contribution is as follows:
\begin{itemize}

    \item A novel zero-shot multi-speaker TTS approach that achieves state-of-the-art results with just 11 speakers in the training set;
    \item An architecture that enables high quality and faster than real-time speech synthesis in the zero-shot multi-speaker TTS setting;

    \item Adjusting a GAN-based vocoder for the spectrograms predicted by the TTS model on the training dataset, in order to significantly improve the similarity and speech quality for new speakers.
\end{itemize}

The audio samples for each of our experiments are available on the demo web-site\footnote{\url{https://edresson.github.io/SC-GlowTTS/}}. In addition, for reproducibility the implementation is available at the Coqui TTS\footnote{\url{https://github.com/coqui-ai/TTS}}, and checkpoints of all experiments are available at the Github repository\footnote{\url{https://github.com/Edresson/SC-GlowTTS}}.

\section{Speaker Conditional GlowTTS Model}\label{sec:TTSModel}

Speaker Conditional Glow-TTS (SC-GlowTTS) builds upon GlowTTS, but includes several novel modifications. In addition to the GlowTTS's transformer-based encoder network, we explore a residual dilated convolutional network \citep{vainer2020speedyspeech} and gated convolutional network \citep{dauphin2017language}; to our knowledge, used for the first time in this context. Our convolutional residual encoder is based on~\citep{vainer2020speedyspeech}, however we used the Mish \citep{misra2019mish} instead of ReLU activation function. On the other hand, our gated convolutional network \citep{dauphin2017language} consists of 9 convolutional blocks and each block includes a dropout layer, a 1D convolution, and a layer normalization \citep{ba2016layer}. We use kernel size 5, dilation rate 1, and 192 channels in all convolutional layers. A flow-based decoder is used with the same architecture and configuration as the GlowTTS model. However, to transform it into a zero-shot TTS model, we include speaker embeddings in the affine coupling layers on all 12 decoder blocks.
We also used the FastSpeech's duration predictor network \citep{ren2019fastspeech} to predict character durations. To capture different speech characteristics of different speakers, we added speaker embeddings to the input of the duration predictor. Finally, the HiFi-GAN \citep{kong2020hifi} is used as a vocoder.

The SC-GlowTTS model, during inference, is illustrated in Figure~\ref{fig:generaltop}, where $(\concat)$ indicates concatenation. During training, the model uses the Monotonic Alignment Search (MAS)~\citep{kim2020glow}, where the decoder's objective is to condition the mel spectrogram and an input speaker embedding in a $P_Z$ prior distribution. The purpose of MAS is to align the $P_Z$ prior distribution with the encoder's output. During inference, MAS is not used, instead, the $P_Z$ prior distribution and alignment are predicted by the text encoder and the duration predictor network. Finally, a latent variable $Z$ is sampled from the prior distribution $P_Z$. Using the inverted decoder and the speaker embeddings, a mel spectrogram is synthesized in parallel, transforming the latent variable $Z$ via the flow-based decoder.

 \begin{figure}[]
  \scriptsize
  \centering
 \includegraphics[width=0.45\textwidth]{images/GlowTTS-Zeroshot-TTS-atual.pdf}
  \caption{Speaker Conditional GlowTTS General Architecture.}
  \label{fig:generaltop}
\end{figure}

For brevity, we denominate the SC-GlowTTS model with the transformer, residual convolution, and gated convolution based encoders as SC-GlowTTS-Trans, SC-GlowTTS-Res and SC-GlowTTS-Gated model, respectively.

\section{Experiments} \label{sec:method}

\subsection{Speaker Encoder} \label{sec:SE}

Our speaker encoder is a stack of 3 LSTM layers with a linear output layer, similar to \citep{ge2e}. We use 768 LSTM units and 256 units for the linear layer. For training, we used audios sampled at 16 kHz and extracted mel spectograms using a 1024ms window using the Fast Fourier Transform (FFT), with a hop length of 256 and 1024 FFT components, from which we retain only 80 mel coefficients. Optimization was carried out using the Angular Prototypical \citep{chung2020in} loss function different than the original work. The optimizer RAdam \citep{liu2019variance} was used during 320k steps using 64 speakers per batch, with 10 samples of each speaker and a learning rate of $10^{-4}$.

\subsection{Zero-Shot Multi-Speaker Tacotron 2}\label{sec:tacozeroshot}

We compare our approach with Tacotron 2. Following the proposal of  \cite{jia2018transfer}, \cite{paul2020speaker} and \cite{cooper2020zero} we use local sensitive attention \citep{tacotron2}. We concatenate the speaker embeddings to the input of the attention module as in \cite{jia2018transfer,cooper2020zero}, given that the latter showed this was adequate for a gender-independent Tacotron model. To alleviate possible issues in the attention module we use Double Decoder Consistency (DDC) \citep{erogolDDC} with gradual training \citep{erogolgradual} and guided attention \citep{dctts}. In Tacotron, the number of output frames per decoder iteration is called the reduction rate (R) \citep{erogolgradual, tacotron2}. The idea of the DDC is to combine two decoders with different reduction factors. One decoder (coarse) works with a higher R and another decoder (fine) works with a smaller R value. Gradual training simply starts training with a larger R and decreases it during the training. In our experiments, we use $R=7$ for the coarse decoder, while for the fine decoder we used gradual training, starting from $R=7$ and decreasing it as follows: $R=5$ at step 10k; $R=3$ at step 25k; $R=2$ at step 70k.

\subsection{Audio datasets}\label{sec:method:base}
Our speaker encoder was trained with all partitions of the LibriSpeech dataset \citep{librispeech}, the English version of Common Voice \citep{ardila2019common}, the VCTK and VoxCeleb (v1 and v2) datasets \citep{chung2018voxceleb2}, totaling approximately 25k speakers

Our zero-shot multi-speaker TTS model is trained using VCTK \citep{veaux2016superseded} dataset, an English language dataset containing 44 hours of speech and 109 speakers, sampled at 48KHz. Each speaker pronounces approximately 400 sentences. Pre-processing was carried out in order to remove long periods of silence. We applied voice activity detection (VAD) using Webrtcvad toolkit\footnote{https://github.com/wiseman/py-webrtcvad}. We have divided the VCTK dataset into: train, validation (containing the same speakers as the train set) and test. For the test set, we selected 11 speakers not present in the validation or training set; following the proposal by \cite{jia2018transfer}, we selected 1 representative from each accent totaling 7F/4M (speakers 225, 234, 238, 245, 248, 261, 294, 302, 326, 335 and 347). For the HiFi-GAN \citep{kong2020hifi} vocoder initial training uses \textit{train-clean-100} and \textit{train-clean-360} partitions of the LibriTTS \citep{zen2019libritts} dataset.

\subsection{Experimental setup}\label{sec:method:experiments}

We carried out four training experiments:

\begin{itemize}
    \item \textbf{Experiment 1:} Tacotron zero-shot model, described in Section \ref{sec:tacozeroshot}, trained for 210k steps.
     \item \textbf{Experiment 2:} SC-GlowTTS-Trans model trained for 150k steps.
    \item \textbf{Experiment 3:}  SC-GlowTTS-Res model trained for 150k steps.
    \item \textbf{Experiment 4:} SC-GlowTTS-Gated model trained for 150k steps.
\end{itemize}

In all experiments, we used RAdam \citep{liu2019variance} with batch size 128, an initial learning rate of $10^{-3}$, and Noam's learning rate schedule~\citep{vaswani2017attention} with 4000 warmup steps. We use the same configuration to extract the mel spectrograms from the speaker encoder, detailed in the Section \ref{sec:SE} but with 22khz sampling rate. We use the VCTK dataset in all our experiments using the training, validation, and test partitions as specified in Section \ref{sec:method:base} and we use the validation set to choose the best checkpoint for each experiment comparing the loss value.

In all experiments, we choose to use phonemes as input instead of text. Specifically, we used the Phonemizer tool\footnote{\url{https://github.com/bootphon/phonemizer/}}, which supports several languages. In addition, we add a blank token between each of the phonemes in the input sentence for the GlowTTS-based models, as suggested by the original work\footnote{\url{https://github.com/jaywalnut310/glow-tts}} \cite{kim2020glow}.

HiFi-GAN v2 model was used as a vocoder, due to its effective speed/quality trade-off. As a starting point, we used the model provided by the authors trained for 500k steps with the LJ Speech \citep{ito2017lj} dataset. We first trained the HiFi-GAN model for 75k steps with the LibriTTS dataset. Afterward, the model is adjusted for other 190k steps using the VCTK dataset, using the training and validation partitions as specified in Section \ref{sec:method:base}.

\cite{kong2020hifi} showed that adjusting the HiFi-GAN model with the spectrogram of the TTS model, improves quality for a single speaker. However, it remains an open question whether it improves; (i) the quality in multi-speaker, (ii) speech similarity for unseen speakers in ZS-TTS settings. To answer this, our TTS models synthesize each of the sentences in the training and validation splits of VCTK dataset. We enabled teacher forcing to keep the alignments between predicted spectrogram frames and the input phonemes. For SC-GlowTTS we use the MAS to align the decoder output with the encoder output. Using these spectrograms extracted from each model, we fine-tuned the checkpoint initially trained with the LibriTTS dataset, for an additional 190k steps, producing the fine-tuned HiFi-GAN (HiFi-GAN-FT).

\section{Results and  Discussion} \label{sec:results}

In this paper, the synthesized speech quality is evaluated using mean opinion score (MOS) study, following \citep{mos}. MOS scores were obtained with rigorous crowdsourcing \citep{dcMOS}.
For the MOS calculation, 15 professional collaborators per audio were invited from a total of 68 unique contributors (35F/33M). To compare the similarity between the synthesized voice and the original speaker, we calculate the Speaker Encoder Cosine Similarity (SECS). The SECS consists of calculating the cosine similarity between the embeddings of two audios extracted from the speaker encoder. It ranges from -1 to 1, and a larger value indicates a stronger similarity \citep{cooper2020zero}.  Following \cite{choi2020attentron}, we compute SECS using the speaker encoder of the Resemblyzer \citep{ge2e,Jemine2019Master} package; thus, allowing comparison with those studies.
\textcolor{black}{We also report the MOS similarity (Sim-MOS) following the work of \cite{jia2018transfer} and \citep{choi2020attentron}.}

We also compare the run-time of each model by calculating the Real Time Factor (RTF) on a CPU and GPU. For speed tests we used a machine with an NVIDIA GeForce GTX Titan V GPU, an Intel (R) Xeon (R) CPU E5-2603 v4 @ 1.70GHz processor with 6 CPU cores and 15 Gb of RAM. The training was carried out on an NVIDIA V100 GPU. Also, RTF was calculated considering the full synthesis run, from the input phonemes to the output waveform. We synthesize 15 different sentences as in \cite{peng2020non} 10 times for each of the 11 speakers of the VCTK test set and calculated the average.

As a reference sample for the extraction of speaker embeddings, we use the fifth sentence of the VCTK (i.e, speakerID\_005.txt), since all test speakers uttered it and because it is a long sentence (20 words). In this way, all speakers are presented in the zero-shot multi-speaker TTS model by a reference sample with the same number of words and speech content

For the calculation of MOS and SECS we randomly drew 55 sentences from the \textit{test-clean} subset of the LibriTTS, considering only sentences with more than 20 words. We randomly select five sentences for each of the 11 test speakers, ensuring that all 55 test sentences are synthesized and that all the test speakers are considered. As ground truth, we select 5 audios randomly for each of the 11 test speakers (55 in total), only audios with more than 20 words are studied.

On the other hand, for the SECS ground truth, we compared the 55 audios chosen at random (explained above) with the reference audios used to synthesize the sentences (fifth sentence of the VCTK dataset for each of the test speakers).

\begin{table*}[]
\centering
\caption{Real Time Factor, MOS and Sim-MOS with 95\% confidence intervals and the SECS for all our experiments.}
\label{tab:rtfsecs}
{\small
\begin{tabular}{l|l|l|l|l|l}
\hline
\textbf{Experiment - Model}               & \textbf{Vocoder}        & \textbf{RTF (CPU - GPU)} &  \textbf{SECS} & \textbf{MOS} & \textbf{Sim-MOS}\\\hline
Ground Truth &-- & -- & 0.9236   & 4.12  $\pm$ 0.06 &  4.127 $\pm$ 0.06 \\\hline
\multirow{1}{*}{Attentron ZS \cite{choi2020attentron}} &   WaveRNN         &  --  &     (0.731)   &  (3.86 $\pm$ 0.05)  &  (3.30  $\pm$ 0.06) \\\cline{2-6} \hline
\multirow{2}{*}{1 - Tacotron 2}
                               & HiFi-GAN         &  0.5782 - 0.2485 &   0.7589       &    3.57 $\pm$ 0.08 &  3.867 $\pm$ 0.08 \\\cline{2-6}

                               & HiFi-GAN-FT     &   -     &  0.7791      &   3.74   $\pm$ 0.08 &  3.951 $\pm$ 0.07\\\hline

\multirow{2}{*}{2 - SC-GlowTTS-Trans}
                               & HiFi-GAN         &   0.3612 - 0.1557      & 0.7641           &   3.65 $\pm$ 0.07 & 3.905 $\pm$ 0.07\\\cline{2-6}

                               & HiFi-GAN-FT      &  -       &  \textbf{0.8046}         &   3.78 $\pm$ 0.07 &  \textbf{3.999 $\pm$ 0.07} \\\hline

\multirow{2}{*}{3 - SC-GlowTTS-Res}
                               & HiFi-GAN         &  0.3597 - 0.1545     & 0.7440         &       3.45 $\pm$ 0.09 & 3.828 $\pm$ 0.08 \\\cline{2-6}
                               & HiFi-GAN-FT      &  -      & 0.7969   &   3.70   $\pm$  0.07    &   3.916 $\pm$ 0.07 \\\hline
\multirow{2}{*}{4 - SC-GlowTTS-Gated}
                               & HiFi-GAN         &  0.3474 - 0.1437            & 0.7432          &       3.55 $\pm$  0.08  & 3.852 $\pm$ 0.08 \\\cline{2-6}
                               & HiFi-GAN-FT      &  -       & 0.7849     &   \textbf{3.82  $\pm$   0.07} &   3.952 $\pm$ 0.07  \\\hline

\end{tabular}
}
\end{table*}

Table \ref{tab:rtfsecs} shows the RTF in CPU and GPU, MOS with 95\% confidence intervals and SECS for all of our experiments. Speed tests show that the fastest model on both CPU and GPU is SC-GlowTTS-Gated, followed by the SC-GlowTTS-Res model. The SC-GlowTTS-Trans model is the slowest of the SC-GlowTTS family, however, still much faster than Tacotron 2. Despite this, with the integration with the HiFi-GAN vocoder all models are real time in both CPU and GPU.

SECS score of the ground truth reached 0.9222 because it compares the sample used as a reference with other real speech samples of the same speaker. This value is intended to show an upper bound for SECS, i.e., a model that perfectly "copies" the voice of a target speaker.

The best SECS for synthesis with the HiFi-GAN vocoder (without fine-tuning) was obtained by the SC-GlowTTS-Trans model (experiment 2), followed by Tacotron 2 (experiment 1). The SC-GlowTTS-Res model (experiment 3) achieved the third-best SECS being only better than SC-GlowTTS-Gated (experiment 2). Using the HiFi-GAN-FT, the SC-GlowTTS-Trans model also obtained the best SECS, followed by the SC-GlowTTS-Res model. The SC-GlowTTS-Gated model reached the third-best SECS being only better than the Tacotron 2 model. We found that the fine-tuning of the HiFi-GAN vocoder in the spectrograms extracted from the TTS models significantly improves SECS for the new speakers. The SECS increased from 0.7589 to 0.7791, 0.7641 to 0.8046, 0.7440 to 0.7969 and 0.7432 to 0.7849, respectively, for the models Tacotron 2, SC-GlowTTS-Trans, SC-GlowTTS-Res and SC-GlowTTS-Gated.

\textcolor{black}{For Sim-MOS the results are similar to those of SECS. However, there are some differences which can be explained by the overlapping of the Sim-MOS confidence intervals between the experiments. Improvement with the use of HiFi-GAN-FT can also be seen in all experiments.}

Finally, we compare our results with those presented by the Attentron model. In \cite{choi2020attentron}, the authors reported SECS values, also calculated by the speaker encoder that we use. Although the authors use only 8 speakers (4F/4M) for the test and we use 11 speakers, we believe the comparison is fair and leaves no selection criteria undefined. The Attentron model in zero-shot mode reached a SECS of only 0.731. Such a model uses multiple samples to synthesize speech instead of just one, performing few-shot TTS with 8 reference samples. This approach achieves a SECS of 0.788, slightly lower than our best SECS, 0.8046. Despite the advantage of the few-shot approach, our model still achieves a higher SECS than Attentron. \textcolor{black}{The authors also reported the Sim-MOS reaching 4.83 $\pm$ 0.02 for ground truth speech, zero-shot mode Attentron reaches 3.30 $\pm$ 0.06 and few-shot mode 3.57 $\pm $0.05. Considering these values, our best model was superior at 3,999 $\pm$ 0.07. Furthermore, the results of our model are closer to the ground truth being only 0.128 smaller while in \cite{choi2020attentron} the best experiment's difference is 1.26.
}

For the MOS, ground truth speech reached 4.12. The SC-GlowTTS-Gated model with the HiFi-GAN-FT vocoder was the closest, reaching a MOS of 3.82. Moreover, as in SECS, where the HiFi-GAN-FT vocoder improved speech similarity, the best MOS was achieved using the same vocoder. With the adjustment of the HiFi-GAN vocoder in the spectrograms extracted from the TTS model, the MOS for new speakers increased significantly from 3.57 to 3.74, 3.65 to 3.78, 3.45 to 3.70, 3.55 to 3.82, respectively, for all models Tacotron 2, SC-GlowTTS-Trans, SC-GlowTTS-Res and SC-GlowTTS-Gated. Our MOS values are on par with the other state-of-the-art ZS-TTS models such as \citep{cooper2020zero, choi2020attentron}.

\section{SC-GlowTTS performance with few speakers}

To emulate a scenario with few speakers, we reflect our test set by selecting a subset of the VCTK dataset training set. This new training set consists of 11 speakers, 7F/4M. We selected 1 representative for each accent, except for the ``New Zealand'' accent that has only one speaker and it is in our test set, so we added an ``American'' speaker instead, the chosen speakers were 229, 249, 293, 313, 301, 374, 304, 316, 251, 297 and 323. \textcolor{black}{From this new training set, we have selected random samples to use as a validation set. As a test set, we use the same one defined in Section \ref{sec:method:base}.}

We use the SC-GlowTTS-Trans model and train it with the LJSpeech~\citep{ito2017lj} dataset for 290k steps. This pre-training in a single-speaker dataset was carried out to prime the encoder of the model in a larger vocabulary. We fine-tuned the SC-GlowTTS-Trans model in the new training set with only 11 speakers for 70k steps and using the validation set, we selected the best checkpoint as step 66k. In addition, using the HiFi-GAN model trained in the LibriTTS dataset for 75k steps, we adjusted for other 95k steps using the same technique. \textcolor{black}{This new experiment resulted in a SECS of 0.7707, a MOS of 3.71 $\pm$ 0.07 and Sim-MOS of 3.93 $\pm$ 0.08. These results are compatible with SECS of 0.7791, MOS 3.74 and Sim-MOS 3.951 $\pm$ 0.07 achieved by Tacotron 2, which used a much larger set of 98 speakers.} Therefore, our SC-GlowTTS-Trans model converges with a 9.8 times smaller dataset, with comparable performance to Tacotron 2. We believe that this is an important step forward especially for ZS-TTS in low-resource languages.

\section{Zero-Shot Voice Conversion}

As in the original GlowTTS \citep{kim2020glow} model, we do not provide any information about the speaker’s identity to the model encoder, so the distribution predicted by the encoder is forced to be independent of the speaker identities. Therefore, like GlowTTS, SC-GlowTTS can convert voices using only the model’s decoder. However, in our work, we condition SC-GlowTTS with external speaker embeddings. It enables our model to resemble the voice for speakers not seen in the training by performing a zero-shot voice conversion. Samples of the zero-shot voice conversion are present on the demo page\footnote{\url{https://edresson.github.io/SC-GlowTTS/}}.

\section{Conclusions and future work} \label{sec:conc}

In this work, we present a novel method, SC-GlowTTS, achieving state-of-the-art ZS-TTS results. We explored three different encoders for the SC-GlowTTS model and showed that a transformer-based encoder gave the best similarity for speakers not seen in the training. Our SC-GlowTTS models are superior to Tacotron 2. Also, when combined with an external speaker encoder, SC-GlowTTS models can perform ZS-TTS with only 11 speakers in the training set. Finally, we found that the adjustment of the HiFi-GAN vocoder in the spectrograms predicted by the TTS model in the training and validation set can significantly improve the similarity and the quality of the synthesized speech (MOS) for speakers not seen in the training. \textcolor{black}{As future work, following the work of \citep{choi2020attentron}, we intend to extend the SC-GlowTTS as a few-shot approach.}

\section{Acknowledgements}
This study was financed in part by the Coordena\c{c}\~{a}o de Aperfei\c{c}oamento de Pessoal de N\'{i}vel Superior -- Brasil (CAPES) -- Finance Code 001.  Also, we would like to thank the CyberLabs\footnote{\url{https://cyberlabs.ai/}} for financial support for this paper and DefinedCrowd for making industrial-level MOS testing so easily available. Finally, we would like to thank all contributors to the Coqui TTS repository\footnote{\url{https://github.com/coqui-ai/TTS}}, this work was only possible thanks to the commitment of all.

\bibliographystyle{IEEEtran}