\section{Introduction}
\paragraph{}
The discoveries of quantum mechanics and general relativity have
caused two of the greatest revolutions in twentieth-century physics,
and their reconciliation remains one of its most important pieces of
unfinished business. One could expect that this reconciliation would
entail a modification of one or both of these very successful
theories, and hope that it would cast light on the transition between
classical and quantum physics. Indeed, the only candidate for a
consistent quantum theory of gravity is string
theory \cite{gswbook}, which is
essentially non-local and hence modifies general relativity at short
distances. It has also been suggested that the usual formulation of
quantum mechanics and quantum field theory might require
modifications in a consistent quantum theory of gravity.
Specifically, studies of field theory in topologically non-trivial
space-times such as black hole backgrounds have indicated that
information loss across an event horizon requires the introduction of
mixed quantum-mechanical states, and the possibility, forbidden by
conventional quantum mechanics or $S$-matrix theory, that pure states
may evolve into mixed states.
\paragraph{}
It has been proposed by Hawking \cite{hawk} that a full quantum theory of
gravity might be formulated only in terms of density matrices
describing in general mixed states, and that the transitions between
initial and final density matrices might not be factorizable as
products of $S$-matrix elements and their hermitian conjugates:
\begin{equation}
           \rho_{out} = \nd{S}
           \rho_{in} \qquad : \qquad  \nd{S}
           \ne   S^{\dagger} S
\label{I}
\end{equation}

\noindent
Evidence for this suggestion was
inferred from studies of
topologically non-trivial solutions of Einstein's equations in
Euclidean space-times. Two of us (J.E. and D.V.N.) then suggested
together with J.S. Hagelin and M. Srednicki \cite{ehns}
that the evolution of
quantum-mechanical systems over time-scales that are long compared
with the Planck time should be described by a modified Liouville
equation:
\begin{equation}
      \partial _t \rho =i[\rho, H] + \nd{\delta H} \rho
\label{II}
\end{equation}

\noindent
A modification of the Liouville equation of the type (\ref{II})
is characteristic of open quantum-mechanical
systems \cite{open}, and represents in
our interpretation the intrinsic coupling of a microscopic system to
space-time foam.
\paragraph{}
We derived upper bounds of the order
of $1 GeV/ M_{Planck}$ on the magnitude of matrix elements of the
$\nd{\delta H }$ term in hadrons from the consistency with conventional
quantum mechanics of measurements of the $K^0 - {\overline K}^{0}$
system \cite{ehns,emnnextpaper}
and long-baseline
neutron interferometry \cite{ehns}. Subsequently, the
same two of us
together with S. Mohanty \cite{emhn}
demonstrated that the non-quantum-mechanical
effects of the $\nd{\delta H}$ would be enhanced in macroscopic
systems such as SQUIDs, and could lead to the collapse of the wave
function of a macroscopic object. An operationally similar
modification of the Liouville equation was
proposed independently
on completely
phenomenological grounds by Ghirardi, Rimini and Weber
\cite{ghirardi}, and the
required values of their model parameters were entirely consistent
with our upper bounds and order-of-magnitude estimates.
\paragraph{}
The study of topologically non-trivial space-times, event
horizons and singularities in string theory was opened up by
Witten's realization that suitable Wick rotations of a cosmological
string theory could be interpreted as black holes in Minkowski or
Euclidean space. We then embarked on a series of studies of quantum
coherence in such stringy black hole backgrounds, with the explicit
motivation of checking the conjectured breakdowns (\ref{I}) and
(\ref{II}) of conventional quantum field theory and quantum mechanics.
We found that quantum coherence was maintained in
the scattering of
light particles on a black hole background by an infinite set of
local $W_{1+\infty}$-symmetries \cite{emn1,emn2}, which
link light asymptotic
states to massive string states, and whose associated conserved
$W$-charges could encode all initial-state
information. We believe
that quantum coherence can
be maintained in this situation only in
an intrinsically
non-local theory such as string with its infinite set of gauge
symmetries, and that local field theories are doomed to failure in
this respect because they only have a finite set of hair. We have
shown that the infinite set of stringy $W$-hair is sufficient to label
all the black hole states, and thereby quench all the entropy
$S = M^2$ of a spherically-symmetric black hole in four
dimensions \cite{emn4}. We
have also argued \cite{emn3}
that black hole decay is a purely quantum-mechanical
higher-genus effect that does not require a thermal description in
terms of Hawking radiation, and that all the $W$-charges are in
principle measurable via scattering experiments or Aharonov-Bohm
phases for massive string
states \cite{emnw}. We have also given a stringy
desription of space-time foam in terms of a plasma of defects on the
world-sheet, which is intimately related to a Hall
conductor \cite{emndua}.
\paragraph{}
The maintenance of quantum coherence in the scattering of light
particles off a stringy black hole does $not$, however, mean that
conventional quantum mechanics and quantum field theory are
sacrosanct. The essential problem is that although all the $W$-charges
are {\it in  principle} measurable, this would entail observations
of the massive string states that are linked to the light states by
the $W$-symmetries, and such observations are not $in~ practice$
possible in realistic laboratory
experiments. We argue in this paper that a
modification of the Liouville equation of the type (\ref{II}) is
indeed $essential$ for observable systems
containing only light particles. Its
origin is exactly the $W$-linkage between the
light and heavy string states, which means that the former must be
regarded as an $open~ system$ coupled to the
unobserved heavy
degrees of
freedom. A modification of the form (\ref{II}) would be forbidden for
any {\it exactly  marginal} deformation of the underlying
string theory. However, it
is known from explicit examples that $in~ non-trivial ~space-time~
backgrounds$ such exactly marginal deformations involve massive string
states. Therefore, any deformation of the effective theory of light
particles in which the massive modes are unobserved will $necessarily$
not be exactly marginal, and will hence be associated with a
contribution to $\nd{\delta H}$ (\ref{II}). The unitarity of the
effective field theory of the light degrees of freedom and
Zamolodchikov's c-theorem \cite{zam}, as
proved by one of the authors (N.E.M.)
and Miramontes \cite{mavmir}, guarantee
that any contribution to $\nd{\delta H}$
can only
increase the entropy of the light-particle system. Topologically
non-trivial space-time backgrounds appear $intrinsically$ in the
stringy realization of space-time foam, so we
conclude that  a $\nd{\delta H }$ term
is $inevitable$ in string theory. We argue, moreover,
that the dominant such modifications are just those associated with
the known $S$-wave deformations of spherically-symmetric black holes.
A primitive order-of-magnitude estimate indicates that their
contribution to $\nd{\delta H }$ might be suppressed by just one
inverse power of $M_{Planck}$, and hence be
close to the present
experimental upper limits \cite{ehns,emnnextpaper}, and
in the ball-park needed to explain the
collapse of the wave function for a classical system.

\section{Renormalization Group Flow as Quantum Friction}
\paragraph{}
    Let us consider a general dynamical system whose state
is described by a density matrix $\rho\{q^i,p_i\}$, where the $q^i$
are generalized coordinates, and the $p_i$ are their associated
conjugate
momenta. In conventional quantum mechanics, the $q^i$
include conventional space-time coordinates, which in
string theory become parameters of background target spaces
corresponding to $\sigma$-models on the world-sheet. Thus the $q^i$
can be regarded as couplings in a space of possible $\sigma$-models.
We consider the evolution of $\rho$ with respect to a renormalization
group flow variable $t = \ln \Lambda$, where $\Lambda$ is some
covariant cutoff. We will identify $t$ with a conventional time
variable, which may or may not be considered as a string Liouville
mode. On dimensional grounds, the target time is then
measured in units of Planck length. This in turn
implies
that
even marginal changes in the renormalisation group scale
produce appreciable time-variations in target-space.
It is clear that renormalisability
of the system implies
\begin{equation}
\frac{d\rho}{dt} = 0 = \frac{\partial  \rho}{\partial  q^i}
 \dot q^i   +
\frac{\partial  \rho}{\partial  p_i}\dot p_i
\label{III}
\end{equation}

\noindent where $ (.\dot ..)$ denotes $\partial _t (...)$.
\paragraph{}
It is known \cite{ems}
that in the Wilson renormalization scheme the
renormalization flow of the coordinates ($\sigma$-model couplings)
$q^i$ is given in the neighbourhood of a fixed point $S_0$ by
the gradient of Zamolodchikov's $c$-function $\Phi (q^i, S_0$):
\begin{equation}
             \beta ^i \equiv \dot q^i
    =G_{ij}(S_0)\frac{\partial \Phi (q^i, S_0)}
{\partial q^i }
\label{IV}
\end{equation}

\noindent
where $G_{ij}$ is the metric in the space of coupling constants:
\begin{equation}
          G_{ij} \equiv 2|z|^2 < O_i(z,{\overline z}) O_j(0, 0) >
\label{V}
\end{equation}

\noindent with $<...>$ denoting an
average with respect to the deformed
$\sigma$-model action $I^* + \int d^2z q^i O_i $, where
$I^*$ is the
fixed-point conformal field theory action,
and the $O_i$ are a complete set of (renormalised)
vertex operators.
\paragraph{}
    In view of equation (\ref{IV}), we can regard $\Phi$ as a Lagrange
function, to which we can associate a Hamiltonian in the usual way:
\begin{equation}
         \Phi = \int dt
       [ \dot q^i\cdot p_i - H(q^i,p_i)  ]
\label{VI}
\end{equation}

\noindent
Again in the usual way, it follows that
\begin{equation}
\dot q^i =\frac{\partial H}{\partial p_i}
\label{VIIa}
\end{equation}

\noindent and
\begin{equation}
\dot p_i = -\frac{\partial  H}{\partial q^i} - G_{ij}\beta^j
\label{VIIb}
\end{equation}

\noindent The
second term in equation (8) is a friction term characteristic of
open dynamical systems \cite{open}, which
will be at the root of our subsequent
modification of quantum mechanics
\footnote{In the Wilson renormalisation scheme (\ref{IV}) the
friction is {\it linear }, which simplifies many of the
computations. However in practice, or in certain cases where
this scheme is not applicable \cite{ems}, one can work in schemes
where (\ref{IV}) is valid but
$G_{ij}=G_{ij}(q)$, in which case one is faced with a
{\it non-linear } friction problem. For most of our discussion
we can stay in the general case.}.
Substituting the expressions
(\ref{VIIa}),(\ref{VIIb})
into the time derivative equation (\ref{III}), we find the following
equation
\begin{equation}
   \dot \rho = i[\rho, H] + G_{ij}\beta ^j
   \frac{\partial \rho}{\partial p_i}
\label{VIII}
\end{equation}
for the classical evolution of the density matrix
\footnote{A similar
equation has been considered by Kogan \cite{kog}, but
his formalism is incorrect
because he
considers
total time derivatives in places
where one should consider partial ones.
In addition, he
identifies the Liouville field with the
target time coordinate, which clearly does not apply to
the two-dimensional black hole
string theory \cite{aben,polch}. We stress
again that from our point of view the world-sheet cutoff
is identified with the target time (evolution parameter)
independently of any (possible) identification of the
Liouville
field with time.}. When proposing any
such modification of the traditional evolution equation for the
density matrix, it is important to check that the total probability
$P = \int dp_i dq^i tr \rho ( p_i, q^i )$ is conserved. In our
case (\ref{VIII}), it is easy to check that
\begin{equation}
      \partial _t P     = \int dp_l dq^l tr
\frac{\partial }{\partial p_i }(G_{ij}(q)\beta ^j (q) \rho (p,q))
\label{IX}
\end{equation}

\noindent which vanishes if there are no contributions from the boundary of
phase space, as would occur if it has no boundaries, or if $\rho$
vanishes there.
\paragraph{}
    The quantum version of this classical equation is obtained by
first replacing
\begin{equation}
             \{,\} \rightarrow  \frac{1}{i} [,]
\label{X}
\end{equation}
The quantum version of the second term in (\ref{VIII}) is obtained by
recalling Euler's equation (\ref{VIIa}), whose quantum version is
\begin{equation}
        \frac{\partial H}{\partial p_i}
        = - i [q^i,H]
\label{XI}
\end{equation}

\noindent from which it follows that
\begin{equation}
    \frac{\partial F(H)}{\partial  p_i} = -i [q^i,F(H)]
\label{XII}
\end{equation}

\noindent for any function $F$ of the Hamiltonian, including in particular the
density matrix $\rho$. We therefore arrive at the folowing evolution
equation for the quantum density matrix:
\begin{equation}
  \dot \rho = i[\rho, H] -i G_{ij} [\rho , q^i]\beta ^j
\label{XIII}
\end{equation}

\noindent
We will not enter here into a discussion of the appropriate quantum
ordering of the factors in the second term in equation (\ref{XIII}),
which represents the quantum friction inherent to our open dynamical
system.
\paragraph{}
An additional remark we would like
to make concerns
the connection of the gradient flow relation (\ref{IV})
with the problem of a supersymmetric quantum mechanical
particle moving in a constant magnetic field ( more
specifically in a Kahler potential
$\Phi (q)$  \cite{vaf}). Indeed, it can be shown \cite{vaf}
that whenever
relation
(\ref{IV}) is valid, the evolution of $q^i$ in
$t=ln\Lambda$
can be described by a super-quantum mechanics
Hamiltonian  $H=\frac{1}{2}(Q + Q^{\dagger})$,
with supercharge $Q=\psi ^i (\nabla _i - \beta_i)$
and its hermitian
conjugate $Q^{\dagger}$, where the
$\psi ^i $ are
Grassmann
variables satisfying canonical commutation relations. The
charges are nilpotent : $Q^2=(Q^{\dagger})^{2}=0$ if
and only if
the {\it covariant} $\beta _i $ are curl-free, i.e. if relation
(\ref{IV}) is satisfied. In this case the metric in coupling
constant space can be viewed as a Kahler metric, whose
potential coincides with the flow-function $\Phi (q)$ \cite{vaf}.
This analogy
makes clearer the physical interpretation
of the renormalisation group flow as a
friction problem related
to the quantum
motion of
a particle in a magnetic field \cite{cald}.
\paragraph{}
    Our previous demonstration that the total probability $P$ is
conserved carries over directly from the classical case (\ref{VIII}) to
the quantum case (\ref{XIII}). However, entropy is $S = -
tr\rho\ln\rho$ is not necessarily conserved: we see easily that
\begin{equation}
     \dot S     = - trGij[\rho,q^i]\beta^j ln\rho
\label{XIV}
\end{equation}

\noindent which does not in general vanish if $\beta^i \ne   0$. Our next task
is to demonstrate that indeed $\beta_i \equiv G_{ij}\beta ^j
\ne 0$, and in so doing we will
be able to argue that the entropy $S$ increases monotonically, as one
would expect on general physical grounds.

\section{Light Particle Operators are not Exactly Marginal}
\paragraph{}
    Since string theory is based on conformal field theory, one might
naively expect that the renormalization coefficients $\beta^i$
introduced above all vanish, and hence that the quantum friction
term in equation (\ref{XIII}) vanishes identically. This is indeed the
case in a flat space-time background, but is not true for operators
that create light particles in non-trivial backgrounds. The prototype
for this phenomenon is the operator creating a massless particle in a
two-dimensional string black hole background, which can also be
interpreted as a spherically-symmetric four-dimensional black hole
background. We first present a heuristic argument that $\beta ^i \ne
0$ in this case, which is followed by a more complete treatment
in section 5.
\paragraph{}
    We recall that the black hole solution possesses a
$W_{1+\infty}$-symmetry that is global on the world-sheet, and includes
the Virasoro algebra. It becomes a local symmetry
when elevated
in target space,
providing an infinite-dimensional extension of the general coordinate
transformations of general relativity. This $W$-symmetry is
responsible, in our interpretation, for the existence of the
$S$-matrix for scattering massless particles, confusingly
called ``tachyons'', in this black hole background, and hence
for the maintenance of quantum coherence in this scattering process.
The feature of these $W$-symmetries that is crucial for the present
discussion is that they link together states with different masses.
In particular, they relate massless ``tachyon'' states to massive
string levels. One could therefore suspect that the conformal
subalgebra of the $W$-symmetry associated with the Virasoro algebra
might also combine massless and massive states in an essential way, and
we shall see in a moment that such is indeed the case. Therefore,
the $\beta$-functions for operators contructed out of light
particle fields alone will not have the full $W$-symmetry, and
hence not be conformal in general, and hence have some $\beta ^i
\ne 0$. But physical laboratory experiments are conducted with light
particles, corresponding to states that are massless in the
approximation
which we discuss here, and do not measure any properties of the
massive string states. Therefore the relevant vertex operators are
precisely those that involve only massless states, and hence have
non-vanishing $\beta$-functions. Hence the $effective$ quantum
mechanics of observable particles has the $extra$ quantum friction term
in (\ref{XIII}), even though the density matrix of the $full$ string
theory would be described by the $usual$ Liouville equation without
such a term. The light observable particles constitute an open
quantum-mechanical system coupled by $W$-symmetry to unobserved
massive states.
\paragraph{}
    Examples of the above assertion that in non-trivial backgrounds
exactly marginal deformations generally involve massive string
states have been provided in ref. \cite{ChLyk}. In two-dimensional
flat-space Liouville theory, the following ``tachyon'' vertex operator is
exactly marginal:
\begin{equation}
             \phi^{c,-c}_{-1/2,0,0} =(g_{++}g_{--})^{-\frac{1}{2}}
F(\frac{1}{2} ; \frac{1}{2} ; 1 ; \frac{g_{+-}g_{-+}}{g_{++}g_{--}})
\label{XV}
\end{equation}

\noindent where $g_{ab}$, $a,b =+,-$ represent the components
of a generic $SL(2,R)$ element \cite{witt}.
However, the corresponding exactly marginal operator in a
two-dimensional Minkowski space-time black hole, described by an
SL(2,R)/O(1,1) Wess-Zumino (WZ)
coset model, is
\begin{equation}
          L^1_0 {\overline L_0}^1
          = \phi^{c,-c}_{-1/2,0,0}+ i (\psi ^{++}-
\psi^{--}) + \dots
\label{XVI}
\end{equation}

\noindent
where
\begin{equation}
  \psi^{\pm\pm}
  \equiv : ({\overline J}^{\pm} )^N (J^{\pm})^N
  (g_{\pm\pm})^{j+m-N} :
\label{XVII}
\end{equation}

\noindent with
$J^{\pm} \equiv (k-2) (g_{\pm\mp}\partial _z g_{\pm\pm} -
g_{\pm\pm}\partial _z g_{\pm\mp} )$, and
${\overline J}^{\pm} \equiv (k-2) (g_{\mp\pm}\partial _{{\bar z}}
g_{\pm\pm} - g_{\pm\pm}\partial _{{\bar z}} g_{\mp\pm})$,
with $k$ the WZ model level parameter \cite{witt}.
The combination $\psi^{++} - \psi^{--}$ generates a level-one
massive string mode, and the dots in equation (\ref{XVI})
represent
operators that generate higher-level massive string states. An
analogous exactly marginal opearator is
\begin{equation}
          L^2_0 {\overline L_0}^2   = \psi^{++} + \psi^{--} + \psi^{-+}
+\psi^{+-} + \dots
\label{XVIII}
\end{equation}

\noindent
which also involves in an essential way operators for massive string
modes. For later use we note that the coupling constant,
$\alpha $,
corresponding to this deformation of the coset model
is responsible for a global rescaling of the target
space-metric \cite{ChLyk}, and therefore to a global
constant shift by $\alpha $ of the dilaton field. Thus it
produces shifts in the black hole mass \cite{witt}.
\paragraph{}
We now remark that in two-dimensional string theory
all these massive
states are in fact discrete states of the type
generated by the $W_{1+\infty}$ algebra which maintains quantum
coherence \cite{emn1,ms}. Each
of these discrete solitonic states can be represented
as a singular gauge configuration \cite{klepol},
whose conserved $W$-charges can be
measured in principle by generalized Aharonov-Bohm phase
effects \cite{emnw}.
However, laboratory measurements of light microscopic objects such as
the $K^0 - {\overline K}^0$ system or interfering neutron beams do not
measure such effects, so do not observe the massive states and are
restricted to the light (massless) parts of the exactly marginal
operators (\ref{XVI},\ref{XVIII}). These by themselves are not exactly
marginal, hence the corresponding light-field $\beta$-functions do not
vanish, and therefore make non-zero contributions to the quantum
friction term in equation (\ref{XIII}). We shall give order
of maginitude estimates of these effects in the next section.
\paragraph{}
    As we have discussed in ref.\cite{emndua}
the sum over
quantum configurations in the string path integral includes a sum
over many vortices and spikes on the world-sheet:
\begin{equation}
Z=\int D{\tilde X} exp(-\beta S_{eff}({\tilde X}) )
\label{act}
\end{equation}

\noindent where ${\tilde X} \equiv   \beta^{\frac{1}{2}}X$,
$\beta $ is a `temperature ' variable  characterising
the topological defects on the world-sheet,
and
\begin{eqnarray}
\nonumber
\beta S_{eff} &=& \int d^2 z [ 2\partial {\tilde X}
{\overline \partial } {\tilde X} +  \frac{1}{4\pi }
[ \gamma _v\epsilon ^{\frac{\alpha}{2}-2}
(2 \sqrt{|g(z)|})^{1-\frac{\alpha}{4}}: cos (\sqrt{2\pi \alpha }
[{\tilde X}(z) + {\tilde X}({\bar z})]):   \\
&+&  (\gamma _v, \alpha,
{\tilde X}(z) + {\tilde X}({\bar z}) )
\rightarrow (
\gamma _m, \alpha ', {\tilde X}(z) - {\tilde X}({\bar z}))]]
\label{XIX}
\end{eqnarray}

\noindent Here $\gamma_{v,m}$ are the fugacities for vortices and spikes
respectively, and
\begin{equation}
 \alpha \equiv 2\pi \beta q_v^2  \qquad \alpha ' \equiv
\frac{e^2}{2\pi \beta}
\label{anom}
\end{equation}

\noindent are related to the conformal
dimensions $\Delta_{v,m}$ of the vortex and
spike creation operators respectively, namely
\begin{eqnarray}
\nonumber
\alpha =4 \Delta _v \qquad   \alpha ' =4 \Delta _m \\
      \Delta _m =\frac{(eq_v)^2 }{16 \Delta _v}
\label{conf}
\end{eqnarray}

\noindent In the
low-temperature phase relevant in the present-day Universe,
this path-integral is dominated by a plasma of world-sheet
spikes corresponding
to microscopic Minkowski-space black holes, which constitute a
stringy realization of space-time foam \cite{emndua}.
Therefore, many
configurations with non-zero values of the $\beta^i$ contribute
importantly to the string path integral, and one cannot $a~ priori$
expect the modification (\ref{XIII}) of the quantum Liouville equation
to be negligible.
\paragraph{}
The reader might worry that although individual configurations
contribute to the quantum fiction term in (\ref{XIII}), some as yet
unknown symmetry principle might cause their total net contribution to
vanish. An immediate intuitive counterargument to this suggestion
is the comment that no-one has ever seen a macroscopic body speed
up as a result of friction!
Thus a cancellation would be surprising in the light of our
physical picture of
the renormalisation
group flow (of unitary theories)
as
the motion of a (supersymmetric) particle in
coupling constant space, under the influence of an external
magnetic field \cite{vaf,ems}.
Indeed, as we shall now explain, the
unitarity of the truncated effective light-particle field theory
guarantees that all quantum friction terms tend to increase the
entropy, in accord with everyday experience, and hence cannot cancel in
the path-integral sum.
\paragraph{}
In our case,
it is easy to see that
the renormalization group flow is
irreversible, as has also been argued in ref. \cite{emndua}
on the basis of
the isomorphism of the space-time foam theory of vortices and spikes
with a Hall conductor. The rate of change of the Lagrange function
$\Phi$ (6) is given by
\begin{equation}
     \dot \Phi     =  \beta^i \frac{\partial \Phi }
     {\partial q^i}
\label{XX}
\end{equation}

\noindent which can be rewritten using equation (\ref{IV})
\begin{equation}
    \dot \Phi     = -  \beta^i  G_{ij} \beta^j
\label{XXI}
\end{equation}

\noindent Since the low-energy effective field theory of the light degrees of
freedom should be unitary by itself, the metric $G^{ij}$ in the space
of coupling constants must be positive definite \cite{zam}.
Hence the Lagrange
function decreases monotonically, corresponding to a monotonic
change in the effective central charge. This in turn corresponds to
a monotonic increase in the entropy, as can be seen explicitly from
the expression (\ref{XIV}) for
the rate of change of the quantum entropy.
The commutator factor in (\ref{XIII}) can be rewritten as
\begin{equation}
        -i[ \rho ( H ), q^i ] =
\frac{\partial \rho}{\partial p_i} =
\frac{\partial \rho}{\partial H} \dot q^i
=\frac{\partial \rho}{\partial H} \beta^i
\label{XXII}
\end{equation}

\noindent Substituting this expression into equation (\ref{XIV}), we find
\begin{equation}
 \dot S = Tr \beta^i G_{ij} \beta ^j
\frac{\partial \rho}{\partial H} ln\rho
\label{XXIII}
\end{equation}

\noindent In systems that exchange energy with their environment,
as is our case, the density matrix is actually given by
\begin{equation}
      \rho =   e^{ \beta(F-H)}
\label{fren}
\end{equation}

\noindent where $F$ is the free energy of the system. In the
particular case of strings, $F$ is identified
with the effective action $\Phi $ \cite{mavmir}.
Thus, taking into account the facts
that $ln \rho < 0$,
and that for unitary $\sigma$-models
$\beta ^i G_{ij} \beta ^j = 2|z|^4 <\Theta (z,{\bar z}),
\Theta (0)>$ is positive definite \cite{zam}, one observes that
the quantum entropy does indeed increase monotonically
thanks to the unitarity of the effective low-energy theory. This
simple and general argument excludes the possibility of a sneaky
cancellation between different contributions to the quantum friction.
\paragraph{}
    We conclude this section by noting an essential difference
between wormhole calculus and the type of quantum gravitational
physics that we are discussing in this paper. Wormholes connect
different parts of the target space-time via throats. Classical
gauge symmetries that are carried by particles falling into one end
of the wormhole are carried out by particles expelled at the other end,
and there is no sign of information loss. Wormholes can give rise to
non-local effects in space-time, which may be physically dubious for
other reasons, but do not cause obvious problems for conventional
quantum field theory or quantum mechanics. We do not address here
the question whether
string theory admits such wormholes. The defects on the world-sheet
that we discuss here correspond to Minkowski space black holes without
throats, where, as discussed above,
information is transferred monotonically by
$W$-symmetries to unobserved massive string modes, resulting in the
above-mentioned monotonic increase in the apparent entropy of the
observable light degrees of freedom.

\section{Simple Models of Quantum Friction Effects}
\paragraph{}
    Before making an order-of-magnitude estimate of quantum friction
effects in the low-energy effective field theory derived from string,
we first remind the reader of the standard formalism for microscopic
systems interacting with ``environmental'' oscillator modes. This
enables us to make contact with the previous literature on the
transition between the quantum behaviour of microscopic systems and
the classical behaviour of macroscopic systems. The standard
``environmental'' formalism assumes weak perturbations around the
equilibrium state of the microscopic system, which is justified in
our case by the suppression of Planck-mass excitations at low
energies. The effective lagrangian for a single harmonic oscillator
coupled to ``environmental'' oscillators representing the infinite set
of massive string modes is
\begin{equation}
    L=\frac{1}{2}M\dot q^2 -V(q) + \frac{1}{2} \sum_j
(m_j \dot x_j^2 - m_j \omega _j^2 x_j^2 ) -
\sum_j F_j(q)x_j - \sum_j \frac{F_j(q)^2}{2m_j \omega_j^2}
\label{tn}
\end{equation}

\noindent where the last two terms represent the interactions. The density
matrix of the system (\ref{tn}) can be expressed as a path integral in
the usual way. The reduced density matrix describing the evolution
of the
primary microscopic harmonic oscillator
from $q=q_i$ at time $t=0$ to $q=q_f$ at time $T$
is then given by
\begin{equation}
K(q_i,q_f, T  ) =K_0(T) \int _{q(0)=q_i}^{q( T  )=q_f}
Dq(\tau) e^{-S_{eff}[q(\tau)]/\hbar }
\label{trt}
\end{equation}

\noindent where the prefactor
$K_0(T)$
involves only the environmental oscillator frequencies \cite{cald},
and can be absorbed in the normalisation of the
uncoupled case.
The Euclidean
effective action $S_{eff}$
is given by
\begin{equation}
  S_{eff}[q(\tau)] =\int _0^{T} d\tau
  (\frac{1}{2}M\dot q^2 + V(q))
+ \frac{1}{2} \int _{-\infty}^{+\infty} d\tau ' \int _0^{T}  d\tau
\alpha (\tau - \tau ') (q(\tau ) - q(\tau ') )^2
\label{trtI}
\end{equation}

\noindent where
\begin{equation}
   \alpha (\tau -\tau ') \equiv \sum _n
   \frac{C_n^2}{4m_n \omega _n} e^{-\omega _n (\tau -\tau ')}
\label{trtII}
\end{equation}

\noindent In deriving these results, we have simplified matters by assuming
linearity in the interactions: $F_j(q) = q C_j$,
corresponding to a
Wilson renormalisation scheme
in string theory \cite{ems}. In this case it can be shown that
$\alpha$ is given by the asymptotic form of the so-called
Drude model \cite{cald}
\begin{equation}
    \alpha (\tau - \tau ') = \eta \frac{1}{(\tau -\tau ')^2}
\label{drude}
\end{equation}

\noindent where $\eta$  is the ``environmental'' friction
coefficient, given by
\begin{equation}
      \eta   \equiv \sum_j \frac{C_j^2}{m_j\omega _j^2 } \delta (\omega
- \omega _j)
\label{frcc}
\end{equation}

\noindent We see from Eq. (\ref{trtII}) that the strengths of the dissipation
terms are suppressed for heavy ``environmental'' oscillators, a feature
that we might expect to carry over to more massive modes in string
theory. The ``environmental'' dissipation effects (\ref{trtII}) are
suppressed linearly in the large masses or frequencies.
If
this result is
carried over
to string theory (with the correspondence of the oscillator
frequencies $\omega _j$
in (\ref{frcc}) to the string mass levels),
the
magnitude of the stringy quantum friction
effects could be comparable to the upper bounds established in
refs. \cite{ehns,emnnextpaper} for
the neutral kaon and neutron systems.

\section{Order-of-Magnitude Estimate in String Theory}
\paragraph{}
In
previous sections we have seen that in two-dimensional
string theory the exactly marginal operators that
turn on backgrounds for the light degrees of freedom
in the presence of a black-hole
contain necessarily massive string states which
are discrete solitonic
states, that are not measured in laboratory
experiments. To estimate the orders of magnitude of the quantum-
gravitational friction term in (\ref{XIII}) and
of the rate of change of the
entropy (\ref{XXIII}), we need
to discuss the magnitude of the renormalization
functions $\beta^i$. In general, one has
\begin{equation}
   \beta ^i = \sum _{N=1}^{\infty} C^i_{i_1...i_N}g^{i_1}...g^{i_N}
\label{beta}
\end{equation}

\noindent where the
coefficients $C^{i}_{i_1...i_N}$
are not
totally symmetric among covariant and contravariant
indices. The usual on-shell string N-point amplitudes
coincide with the expansion coefficients
$\gamma _{i_1...i_N}=<\int d^2z_1V_{i_1}...\int d^2z_NV_{i_N}>$
of the
covariant $\beta _i \equiv G_{ij}\beta ^j$ \cite{mavmir2,brust}.
These coefficients
are totally symmetric
as follows from the flow equation (\ref{IV}).
Factorization
implies that all higher-order coefficients are obtainable from 3-point
functions \cite{brust}. If
one chooses an appropriate Wilson renormalization scheme,
all contact terms are eliminated from these 3-point functions if the
boundaries of moduli space are treated correctly \cite{distdoy}, as
we assume for the
coset black hole model, in which case the metric tensor $G^ij$ in
coupling constant space
is flat with its non-zero entries $O (1)$ \footnote{
The existence of such a Wilson scheme, and hence the
vanishing of such contact terms, is subtle in certain
cases
such as supersymmetric
strings \cite{green}, requiring a
careful definition
of correlation functions with colliding
punctures
\cite{distdoy}. Thus it might not be easy
for this scheme
to be implemented
in practice
in the black hole case,
whose singularity is associated with a
twisted supersymmetric fixed point \cite{eguchi}.
If a conventional renormalisation scheme
were adopted in this case \cite{verl}, it would
give non-linear friction. However,
we believe that physical observables
would not be affected, since the corresponding
conformal field theory is well-defined
even at the singularity \cite{witt}.}.
\paragraph{}
For exactly marginal operators with both light- and
heavy-state terms: $V_i  =  V_i^{(L)}    +  V_i^{(H)} $, such as
$L^1_0 {\overline L}^1_0$ (17), we have
\begin{equation}
  0 =\beta^i = G^{ij}(S_0)\sum_{N=1}^{\infty}
   g^{i_1}...g^{i_N} <\Pi_{n=1}^{N}(V_{i_n}^{(L)}+ V_{i_n}^{(H)})>
\label{thrf}
\end{equation}

\noindent Therefore, defining the light-state renormalization coefficients
\begin{equation}
     {\hat \beta}^i = \sum_{N=1}^{\infty}
      G^{ij}(S_0) g^{i_1}...g^{i_N}
<\Pi_{n=1}^{N}
 V_{i_n}^{(L)}>
\label{thrfiv}
\end{equation}

\noindent we find
\begin{equation}
   {\hat \beta}^i = - \sum _{N=1}^{\infty}
    \sum_{M=1}^{N}
   g^{i_1}...g^{i_N} <\Pi_{m=1}^{M}
    V_{i_m}^{(M)}\Pi_{n=1}^{N-M} V_{i_n}^{(L)}>
\label{thrsix}
\end{equation}

\noindent The effective $\beta^i$ appearing in sections 2 and 3 are in fact the
${\hat \beta}^i$ (\ref{thrfiv}) in
the light-field theory, and we now revert to the hatless
notation of those earlier sections. As we have seen, these
effective $\beta ^i$
are non-vanishing to
the extent that the mixed light-heavy N-point functions in (36) are
non-zero.
\paragraph{}
To estimate these, we use the Euclidean counterpart
of the $SL(2,R)/O(1,1)$  WZ coset model \cite{witt}, which can
be mapped \cite{BerKut} into a
$c =1$ matrix model with a modified cosmological constant on the
world-sheet \footnote{Although Euclidean and Minkowski coset models
are not physically equivalent in this formalism
\cite{emn3,distl},
the Euclidean version
will be sufficient
for our
order-of-magnitude estimates if
we assume that the
analytic
continuation in target space needed to produce
Minkowski black holes
does not affect
the scaling arguments we are using below.}: in their notation
\begin{equation}
            2\pi\mu \beta {\overline \beta }
 exp (-2\frac{\phi}{\alpha_{+}}) \equiv  \mu  V_{1,1}^{(-)}
\label{twentyone}
\end{equation}

\noindent where $\beta$,${\overline \beta }$ are chiral bosons
used along with the free field $\phi $ and another
chiral boson $\gamma $
to parametrise $SL(2,R)$ \cite{BerKut}.
The scale $\mu$ is related to the mass of the black hole by
$\mu  =  M_{BH} / M_{Planck}$, and the massive string states are
represented in general by
\begin{equation}
      V_{r_1r_2}^{(\pm)} =[\partial ^{r_1r_2}X + ...]
e^{i\frac{r_1 - r_2}{\sqrt{2}} \pm \beta _{r_1r_2}^{\pm}\phi }
\label{twentifive}
\end{equation}

\noindent The $(-)$ states are often discarded in usual Liouville
theory \cite{polch}, but
play a key role here, as we shall see. In particular, following
ref. \cite{BerKut}, we have the following
representation for the WZ model representing a two-dimensional
black hole \cite{witt}
\begin{equation}
     S_{WZ} = S_{c=1,\mu=0} + \mu V_{1,1}^{(-)}
\label{twentysix}
\end{equation}

\noindent The (+) states are massive string modes, so the correlation functions
of interest to us can be expressed as
\begin{equation}
  \sum_{l=0}^{+\infty} \frac{1}{l!}\mu^l
  <\Pi _{i=1}^{N'}V^{(+)}_{r_ir_{i+1}}
\Pi _{j=1}^{M} V_j^{0} \Pi _{m=1}^{l} V_{r_{m-1}r_{m}}^{(-)}>_1
\label{twentyseven}
\end{equation}

\noindent where
the superscript (0)
denotes a massless ``tachyon'' state, and the
subscript 1 on the v.e.v. denotes the fact that it is calculated in
a $c$=1 matrix model with $\mu  = 0 $.
\paragraph{}
    We now seek to understand which correlation
functions are in general non-vanishing, and how they scale
with $\mu$. The first important observation in this regard is that
correlators of $V^{(-)}$
operators alone vanish: only correlators with
at least one $V^{(+)}$ operator are non-zero. This means that the black
hole dynamics necessarily turns on physical massive string states, as
we had seen previously from the point of view of the $W$-symmetry
linking massless and massive modes \cite{emn1,emnw}. The
second comment is that the
non-zero correlators contain in general logarithmic factors that
violate the expected power-law scaling with $\mu$. Specifically,
there are factors of $(ln \mu )^{\pm 1}$
for insertions of $V^{(\pm)}$
operators respectively, associated with divergences close to the
Fermi level and representing physical pole conditions in amplitudes
\cite{grosnewm,difkut}. Thus we
have the following general scaling laws:
\begin{equation}
<\Pi _{i=1}^{2N'}V^{(+)}_{r_{i-1}r_{i}}
\Pi _{j=1}^{M} V_j^{0} \Pi _{m=1}^{2N} V_{r_{m-1}r_{m}}^{(-)}>_1
\propto \bigg \{ \begin{array}{cc}
\mu^s (ln\mu )^{N'-N} \times F(\{ k \}) ~~for N'-N > 0 \\
0 \qquad otherwise        \end{array}
\label{twentyeight}
\end{equation}

\noindent where $F(\{ k \} )$ is a kinematical factor, and
the power $s$ is determined by naive kinematical scaling
considerations. It is related to
the Liouville-energy
non-conservation which arises
from the reality of the coupling
of the
Liouville field in the expression for the
vertex operators \cite{pol}. For
resonant amplitudes, $s=0$. For
general $s$
one can only compute the
amplitudes by analytic continuation from the
integer $s$ case
\cite{goul}. Such a procedure is assumed in the following.
\paragraph{}
    We see in the general scaling law
(\ref{twentyeight})
logarithmic factors in the non-zero correlators that
appear to
vanish as $ln \mu \rightarrow  0$, i.e.,
as $\mu \rightarrow  1$. Therefore we are led to
conclude that the
contributions to the quantum-gravitational
friction are suppressed for
configurations with $\mu \simeq 1 $. Formally, the
expression (\ref{twentyeight})
is ill-defined for $\mu  \rightarrow 0$, i.e.,
$ln \mu \rightarrow -\infty $. This
we interpret as a reflection of the well-known
ill-definition of the $c$ = 1 matrix model
itself when $\mu  = 0$. Analytic continuation is not
safe in this regime, since
if
one
formally evaluates the amplitudes
in the limit $\mu \rightarrow 0$ by analytically
continuing to $s=integer$ $>$ $N'-N$ in (\ref{twentyeight}), then
the amplitudes vanish. We interpret these subtleties
as follows:
as
discussed in ref. \cite{emndua},
we believe that the appropriate
string vacuum is a space-time foam consisting of a plasma of spikes on
the world-sheet corresponding to microscopic Minkowski-space black
holes. The end-state of black hole decay is a microscopic black hole
that is indistinguishable from this foam, so the latter should
be
subtracted and serve as a regulator for black hole physics. This
would therefore be dominated in our application by configurations with
$\mu  \ge 1 $ in the gravitational path integral
analogue of the simple model (32).
\paragraph{}
  Incorporating the estimate (\ref{twentyeight})
into
the expression (\ref{thrsix}) for the
effective light-field renormalization coefficients $\beta^i$, and then
using the path integral (20,21) to perform the sum over microscopic
Minkowski black holes in the space-time foam, which is dominated by
configurations with $\mu  =  M_{BH} / M_{Planck} \ge 1 $
as discussed
above, we estimate that the coefficients of the quantum-gravitational
friction terms in (\ref{XIII},\ref{XXIII})
are $O (1)$ in Planck units. Thus their effects
are suppressed by dimensional powers of $M_{Planck}$, and the largest
could be $O ( M_{Planck}^{-1})$ as in
the simple model (32). Their detailed
evaluation requires more knowledge of the effective low-energy theory
derived from string, and goes beyond the scope of this paper.
\newpage
\section{ Collapse of the wave-function}.
\paragraph{}
It has been argued \cite{emhn}
that the microscopic entropy increase
offered by quantum gravity
effects may cause the collapse
of macroscopic wave functions and their
transition from quantum to classical behaviour.
We now discuss whether quantum quantum-gravitational
friction (\ref{XIII}) and the rate of entropy increase
(\ref{XXIII})
could have this effect. We start from the simple
model for friction described in section 4. In the
string case, the renormalized $\sigma$-model
couplings play the r\^oles of coordinates
$q$, and $\tau$ is the renormalization group scale
parameter, identified with target time.
Since the $q$'s are almost but not exactly marginal
in the truncated effective light-mode field theory,
the dominant contributions to the path-integral
in the asymptotic Drude model (\ref{drude})
come from the limit
where $\tau \rightarrow \tau '$.
In this case eq (\ref{trtI}) may be written as
\begin{eqnarray}
\nonumber     \rho (q_i,q_f,  T  ) / \rho_S (q_i,q_f,  T  ) \simeq
e^{ - \eta  \int _0^{T} d\tau \int_{\tau-\epsilon}^{\tau + \epsilon}
d\tau '
\frac{(q(\tau) - q(\tau '))^2}{(\tau - \tau ')^2}} \simeq \\
e^{-\eta \int_0^{T} d\tau \int_{\tau ' \simeq \tau }
d\tau '
 \beta ^i G_{ij}(S_0) \beta ^j } \simeq
e^{ - DT ({\bf q_i}  - {\bf  q_f} )^2 + \dots }
\label{asym}
\end{eqnarray}

\noindent where $D$ is a small constant,
proportional to the sum of the
squares of the
effective anomalous dimensions of the renormalised
couplings $q^i$.
We
reinterpret
(31) as representing the overlap, after an elapsed time interval
$T$, between quantum systems localized at
different values ${\bf q}={\bf q_i}, {\bf q_f}$ of
the coordinates, and
the subscript ``S'' denotes quantities evaluated
in
Schroedinger
quantum mechanics.
Equation (\ref{asym})
exhibits a quadratic dependence on the $\beta $-functions
in this simple mechanical model for friction.  It
captures
all the essential features of the complicated
string case, in which the light-mode effective $\beta$-functions
are suppressed by inverse powers of $M_{Planck}$, as we
discussed in the previous section.
\paragraph{}
The vanishing of off-diagonal terms in the density matrix
leads in general to the collapse of the wave-function.
This effect is usually negligible for microscopic
light-mode systems - except possibly for special cases
such as the $K^{0}-{\overline K}^{0}$ system discussed
elsewhere \cite{ehns,emnnextpaper} - but it is enhanced
for macroscopic systems. This was first derived in ref.
\cite{emhn} in the context of a wormhole model, but the result
is more general and applies here. We consider a macroscopic
body containing many particles with coordinates $q^i, i=1,2,...N$.
Decomposing these into a centre-of-mass coordinate $Q$ and relative
coordinates $\Delta q^i$, $(i=1,2,...N)$, writing
the full density matrix as $\rho _q = \rho _Q \rho _{\Delta q}$,
and assuming friction terms $-Dt(q^i -q'^i)^2 $ $(i=1,2,...N)$,
the equation for the centre-of-mass motion is
\begin{equation}
 \dot{\rho}_Q \simeq i[\rho _Q , H(Q)]\rho _{\Delta q} +
i [\rho _{\Delta q}, H(\Delta q)]\rho _Q - \sum_{i=1}^{N}
D(q^i - q'^i)^2\rho (q^i,q'^i)
\label{centre}
\end{equation}

\noindent Tracing over the relative coordinates $\Delta q^i$, we find
\begin{equation}
\dot{\rho}_Q \simeq i[\rho _Q, H(Q)] - ND(Q-Q')^2\rho _Q
\label{macr}
\end{equation}

\noindent and hence
\begin{equation}
    \rho (Q',Q,t) / \rho _S(Q',Q, t) \simeq
e^{-NDt(Q' - Q)^2 }
\label{final}
\end{equation}

\noindent where we see an enhanced suppression for bodies
containing many particles.
As an illustration, if $D \simeq m^6_{proton}/M^3_{Planck}$
as in ref. \cite{emhn}, then the locations of bodies with
$N \ge 10^{24}$ constituent particles (cf Avogadro's
number)
are fixed to within
a Bohr radius within about $10^{-7}$ sec. A similar mechanism
for wave function collapse was introduced phenomenologically
in ref. \cite{ghirardi} without reference to any microscopic
model.
\paragraph{}
A final comment we would like to make concerns other
possible
effects of the space-time foam, which could also
be enhanced
macroscopically. As we have argued in \cite{emndua}, the space-time
foam is represented as a plasma of world-sheet spikes, and one
should examine the motion of a single light string mode in this
plasma which
is not yet
feasible.
We notice, however, that taking into account the no-net-force
condition among the spikes of the plasma
\cite{emndua}, we can define a `centre-of-mass' coordinate
for the black hole moduli and examine the motion of the
tachyon in a single coset model, representing in some sense
this `collective' effect. In such a case, following \cite{emhn},
one would have an enhanced
macroscopic non-quantum mechanical
effect if there  were
microscopic violations of the no-net-force
condition, leading to a motion of the centre of mass of the
black-hole moduli. Such effects are left for future investigations.
However one should keep in mind that they might also
lead to ``observable'' modifications of quantum mechanics.

\section{Discussion}
\paragraph{}
    The following remarks may help the reader to understand
intuitively the relation of this work to other studies. As we have
emphasized previously \cite{emnw}, the
scattering of light particles off a black hole
resembles the Callan-Rubakov \cite{cr}
description of the scattering of charged
particles off a grand unified monopole. There, conservation
laws enforce selection rules on the scattered particles, and
here similar selection rules ensure that quantum coherence is conserved
by the full light/heavy particle system. The situation we have
discussed in this paper is more akin to the 't Hooft
\cite{thooft} analysis of
instanton effects in the Standard Model. In that case, scattering light
particles have a very small probability of ``meeting'' an instanton,
but then it generates new physics. In our case, light particles again
have a very small probability of ``meeting'' a microscopic or virtual
black hole, but then it also generates new physics. Since the heaavy
degrees of freedom are necessarily ignored in a light-particle
scattering experiment, information and hence coherence are lost in
this case.
\paragraph{}
    We have laid out in this paper the conceptual basis for the
modification of laboratory quantum mechanics in string theory. We
re-emphasize that the full string theory is perfectly
quantum-mechanical, but that experiments cannot see this unless they
measure all the massive string states, which are linked to light
states by an infinite set of $W$-symmetries. Observing these massive
modes is an impossibility in realistic laboratory experiments.
Therefore, one must work with an effective theory of the light degrees
of freedom as an open system, with the corresponding modification
(\ref{XIII})
of the Liouville equation for the density matrix. A simple extension
of this argument will
lead to a non-factorizing $\nd{S}$-matrix description
of the transition between initial and final density matrices in
quantum field theory. The quantum-gravitational friction that we have
identified in this paper appears able to explain qualitatively the
transition from quantum-mechanical behaviour of microscopic systems to
the classical behaviour of macroscopic systems containing Avogadro's
number of elementary particles.
\paragraph{}
\noindent {\Large {\bf Acknowledgements }} \\

The work of D.V.N.
is partially supported by DOE grant DE-FG05-91-ER-40633 and
by a grant from Conoco Inc.


\section{Introduction}

Generating an image from a text prompt or caption, also known as text-to-image, is a challenging problem. Image pixels are considered continuous values across the image, while the text is represented as discrete values. The difficulty in mixing continuous and discrete representation has hampered the progress in this field. However, the recent improvement in multimodal research, most notably DALL-E \cite{dalle} led to a dramatic improvement in the quality in text-to-image generation. In the DALLE-E architecture, the image is encoded into discrete tokens with discrete variational autoencoder dVAE \cite{dvae}. The relationship between the discrete image tokens and discrete text tokens can be modelled by a transformer \cite{transformer}. The text and image tokens are concatenated, and the transformer is trained to predict each of them in an autoregressive manner.

DALL-E does not include the image results of people, but results of subsequent models adopting similar transformer architecture \cite{xmcgan} \cite{cogview} show poor performance in generating people images. The people can have blurry faces, distorted bodies, out of proportion bodies and missing or additional limbs. We hypothesise that the text's inherent ambiguity partly makes it difficult for the model to learn the mapping to the human body's high degree of freedom. This challenge includes describing body shape, size, pose, clothing details, position in the images, camera view and the number of people, all difficult to describe in words fully. Thus, we propose to add human pose as an additional input constraint to improve the correctness and fidelity of people being generated. To our knowledge, our paper is the first to use both text and human pose in the generative image model. We can see this as enforcing disentanglement of content and style of image \cite{styletransfer} where the content is the pose, and the text depicts the style. Furthermore, motivated by the inadequacy of existing metrics to measure image errors specific to people, we devised a People Count Error (PCE) metric to measure the false positive rate when generating images of multiple people. Therefore, we can empirically and qualitatively show that the disentanglement improves the fidelity of people generated with a significantly reduced number of false positive or erroneous body parts.

\vspace{-1em}
\begin{figure}[!htb]
\centering
\includegraphics[width=8cm]{images/deepfashion/teaser.png}
\caption{Our text-and-pose-to-image model takes in text prompt and the target pose as inputs to generate photo-realistic people images. Our method supports partial pose view, multiple people, all at different scales. }
\label{fig:teaser}
\end{figure}

The current method of representing pose for transformer is to convert skeleton image into discrete image token with dVAE. This is the method adopted by VQGAN \cite{vqgan} which frames pose-to-image as an image-to-image problem. However, this image pose encoding has shortcomings, primarily high computational complexity, which increases quadratically with image resolution, limiting the output resolution of images at around $256 \times 256$. Therefore, we have devised a novel, efficient and accurate pose representation suitable for a transformer - \emph{Keypoint Pose Encoding} (KPE). Instead of using the high dimensional skeleton image containing redundant information, we use body joint keypoints for pose representation. The low dimensional representation is invariant to changing the target image resolution or domain, e.g. from the natural landscape to synthetic objects. It is also 10x more memory efficient and increases computational inference speed by over 73\%. Finally, we also demonstrate that KPE can generate video by using pose information to offer consistency in people's appearance from frame to frame. This is infeasible with existing text-to-image models. The advantages of our proposed method against other methods are summarised in Table \ref{table:compare_method}. In summary, our contributions are:
\begin{enumerate}
    \item \textbf{Pose and Text Guided multi-person Image Generation} We are the first to use both text and pose guided models to generate images of multiple people. In addition, our method can be used for image sequence generation.
    \item \textbf{Novel Keypoint Pose Encoding (KPE)} To enable the \emph{tokenisation} of a human pose representation that is computationally efficient and invariant to changes in target images such as the resolution.
    \item \textbf{Introduction of a person centric metric} A new metric to measure the false error rate of generated humans in multi-person images in generated images. Helping to illustrate that the disentanglement of pose and text leads to better higher quality results with reduced false positive humans.
\end{enumerate}

\section{Related Work}
\subsection{Text-to-Image Generation}
In GAN-based text-to-image models, early works such as StackGAN\cite{stackgan}, AttenGAN\cite{attngan}, or XMC-GAN\cite{xmcgan} are forms of conditional GAN \cite{cgan} where the text sequence is projected to an embedding vector as a conditioning feature, i.e. the text is modelled as continuous variables. More recently, researchers turned their attention to the transformer architecture that has proven to excel in natural language processing (NLP) tasks. Instead of using a single embedding vector for the entire word sequence, it is encoded into a sequence of discrete text tokens. Each token is then projected into the transformer's embedding space. The transformer is trained to predict the text tokens autoregressively in sequence.

However, there are challenges in applying the transformer to computer vision tasks. The high dimensionality of image pixels makes it computationally expensive as the transformer computational complexity is $O(N^2)$. Moreover, image pixels are considered continuous real values and do not fit into the discrete token needed by the transformer. DALL-E~\cite{dalle} addressed these problems by using VQ-VAE \cite{vqvae} which is a variant of dVAE to encode the image into discrete tokens. Hence, both text and image are represented as discrete tokens, allowing the adoption of generic transformers to predict the image tokens from text tokens. The resulting image is generated by feeding the image tokens to the decoder part of dVAE. This method forms the basis of many modern transformer-based text-to-image models such as CogView \cite{cogview} and N\"{U}WA \cite{nuwa}. We employ a similar method to tokenise the text in our models but condition our transformer on a further additional modality, i.e., pose, to enrich the image quality and precision.

\subsection{Pose Guided Image Generation}
There are a few ways of representation pose in the context of image generation; including, body keypoint heatmap \cite{Ma2017}\cite{Siarohin2018}\cite{Yang2020}, segmentation maps \cite{gaugan} and a skeleton image \cite{pix2pix}\cite{Zhu2019}. These representations can be generalised as a 2D spatial tensor $\mathbb{R}^{H\times W \times C}$ where $H$ and $W$ are the image height and width. $C$ is the number of image colour channels or classes for segmentation and heatmap. In other words, pose-to-image is framed as image-to-image where an input image is a form of 2D spatial tensor representing pose, and the output image is the person image. However, these representations have many redundancies; for example, a segmentation map includes every body pixel and background. The same goes for skeleton images, dominated by background pixels that do not carry any useful information about the pose. A skeleton image and the keypoint heatmap are derived from Cartesian coordinates of body joints key points that contain all the pose information. The reason for using the 2D spatial tensor for pose representation is that the spatial information is required for convolutional layers in GANs \cite{Tompson2014}, but this is no longer a pre-requisite for transformer. Despite this, recent generative transformer models \cite{vqgan,nuwa} continue using an image for pose representation by encoding image into discrete image tokens. The image tokenisation can take be very slow, \cite{dalle_mini} attempts to reduce the training time by pre-encoding the images into tokens, but this prohibits the use of augmentation onto the images and poses during training. Thus, several papers \cite{skeletor} \cite{Li2021} came to realise the shortcoming of using skeleton images and started to use keypoint for pose estimation regression. However, we are the first to use pose keypoint to guide the image generation.

\newcommand{\textcolor{green}{\checkmark}}{\textcolor{green}{\checkmark}}
\newcommand{\textcolor{orange}{\checkmark}}{\textcolor{orange}{\checkmark}}
\newcommand{\textcolor{red}{\ding{55}}}{\textcolor{red}{\ding{55}}}

\begin{center}
\vspace{-5mm}
\begin{table}
\centering
\begin{tabular}{ C{2.5cm} | C{1.6cm}  C{1.6cm}  C{1.6cm}  C{2cm} C{2cm} }
 \toprule
  Models & Text Guidance & Pose Guidance & Generate Image Sequence & Fast Pose Encoding & Invariant Pose Representation \\[0.5ex]
\midrule
 GANs \cite{vqgan} \cite{Ma2017}  \cite{pix2pix}  & \textcolor{red}{\ding{55}} &\textcolor{green}{\checkmark} &  \textcolor{red}{\ding{55}} & \textcolor{red}{\ding{55}} & \textcolor{red}{\ding{55}} \\
\midrule
 \cite{dalle} \cite{xmcgan}  \cite{cogview}  &  \textcolor{green}{\checkmark}  & \textcolor{red}{\ding{55}} & \textcolor{red}{\ding{55}} &\textcolor{red}{\ding{55}} & \textcolor{red}{\ding{55}} \\

\midrule
 N\"{U}WA \cite{nuwa} & \textcolor{orange}{\checkmark} \textcolor{orange}{*} & \textcolor{orange}{\checkmark} \textcolor{orange}{*}  & \textcolor{green}{\checkmark} & \textcolor{red}{\ding{55}} & \textcolor{red}{\ding{55}} \\

  \midrule
 KPE (Proposed Method) & \textcolor{green}{\checkmark}  &  \textcolor{green}{\checkmark} &   \textcolor{green}{\checkmark} & \textcolor{green}{\checkmark} & \textcolor{green}{\checkmark}  \\ [1ex]
 \bottomrule
\end{tabular}
\caption{Comparing features with other person image generation models. Our method is the only one that support both text and pose input. *N\"{U}WA uses only text or pose but not both at the same time. Our pose encoding method is fast, memory efficient and invariant to changes in image data (e.g., resolution, domain) or image encoder.}
\label{table:compare_method}
\vspace{-7mm}
\end{table}
\end{center}

As shown in Table \ref{table:compare_method}, our proposed method is the only model that uses both text and pose to guide the generation of people in images. While N\"{U}WA supports text guidance and pose guidance, they do not co-exist in the same model. Furthermore, N\"{U}WA is a 3D transformer that is more computationally expensive to run than our model that is based on a conventional 2D transformer.

\section{Method}

Figure \ref{fig:Overview} shows the overall architecture of our pose guided text-to-image model. The first stage is to convert the text, pose keypoints and image into tokens with their respective encoders. Then the tokens are projected into embedding before adding positional encoding. We use learnable positional encoding for text tokens, and axial positional encoding \cite{axial} for image tokens due to its 2D structure. We do not use positional encoding for keypoint tokens as they are equally important for all positions of the image tokens. The transformer is made of 12 transformer encoder blocks adopted from \cite{dallepytorch}, and there is no transformer decoder block. We will now describe the details of token encoders.

\begin{figure}[!htb]
\centering
\includegraphics[width=0.5\linewidth]{images/models/transformer5.png}
\caption{Architectural diagram of our pose-guided text-to-image generation model. The text, pose keypoints and image are encoded into tokens and go into an encoder-only transformer.}
\label{fig:Overview}
\end{figure}

\subsection{Text Encoder}
Like DALL-E~\cite{dalle}, we use the BPE (Byte Pair Encoding) tokeniser \cite{bpe} for text tokenisation, the encoder breaks the word into subwords, and they are assigned discrete identifier numbers which become the text tokens. The text tokens are then projected into embedding  $\mathbb{R}^{t \times d}$, where $t$ is the fixed text token input length and $d$ is the transformer dimension. The BPE tokeniser is pretrained with the vocabulary of the dataset.

\subsection{Image Encoder}
In our model, we use VQ-VAE \cite{vqvae} from VQGAN \cite{vqgan} for image encoding and decoding. Its GAN training pipeline has produced better image quality than the dVAE used by DALL-E. The encoder $E$ first converts the continuous image  $x \in \mathbb{R}^{H\times W \times C}$ with height $H$, width $W$ and colour channel $C$ into code  $\hat{z} = E(x) \in \mathbb{R}^{h\times w \times n_z}$ with $h$ and $w$ the spatial dimension and $n_z$ the dimension of the code. Then, each of the codes is quantised \textbf{q}(.) to its closest discrete codebook entry $z_k$ using equation:
\begin{align}
    z_q = \textbf{q}(\hat{z}) := \left(\arg \min_{z_k \in Z} \|\hat{z}_{ij} - z_k \|\right) \in \mathbb{R}^{h\times w \times n_z}
    \label{eq:vqgan_quantise}
\end{align}
where $Z = \{z_k\}^K_{k=1} \in \mathbb{R}^{n_z}$ is the discrete codebook.
The decoder G reconstructs the discrete code into image $\hat{x}$:
\begin{align}
    \hat{x} = G(z_q) = G(\textbf{q}(E(x)))
\end{align}
The model and codebook can be trained end-to-end using the loss function:
\begin{align}
    L_{VQ}(E, G, Z) = \|x-\hat{x} \|^2 & + \|sg[E(x)-z_q] \|^2_2  \\
                                       & + \|sg[z_q]-E(x) \|^2_2
\end{align}
where $sg[.]$ denotes stop-gradient operation.

Overall, the image is tokenised into $h \times w $ grid of discrete image tokens. Although $h$ and $w$ are hyperparameters, they have a linear correlation with image resolution $H$ and $W$ to maintain the same quality of image texture details. Therefore, increasing image resolution will lead to more extended image token lengths, resulting in a quadratic increase in computational complexity. The image encoder is pretrained with the target images like a text encoder. The discrete image tokens are then projected into transformer embedding $\mathbb{R}^{h \times w \times d}$.

\subsection{Keypoint Pose Encoder (KPE)}
KPE is our method for pose representation. It converts detected pose positions for multiple people into keypoint tokens and then encodes them into a keypoint embedding.

A single 2D keypoint is defined as a tuple of $(x, y, v)$ where $x$ and $y$ are the normalised Cartesian coordinates in [0, 1] and $v$ is the visibility score in $[0, 1]$. We denote multi-person 2D keypoint format as $(x, y, v)_{i,j}$ where $i$ is the person index, $j$ is the keypoint index from 0 to $N-1$ where $N$ is the total number of keypoint defined. Different pose estimation models use different keypoint schemes, but the common keypoints are the nose, neck, shoulder, elbow, wrist, hip, knee, ankle, eye, ear, big toe, and heel.

\begin{figure}[!htb]
\vspace{-5mm}
\centering
\includegraphics[width=0.8\linewidth]{images/models/keypoint.png}
\caption{Block diagram showing KPE encoding multi-person pose to keypoint tokens by using a maximum of three people as an example. If the image contains fewer people, the tokens will be padded with zeros. The tokens are flattened and projected into keypoint embedding within the transformer. The skeleton image is for illustration purposes as we use the keypoints directly from the pose estimation model's outputs.}
\label{fig:kpe}
\end{figure}

Figure \ref{fig:kpe} shows the process of KPE. The process is similar to the method in Vision Transformer (ViT)\cite{vit} of grouping image pixels into image patches, except that we group keypoints into keypoint tokens. Each keypoint token corresponds to a skeleton joint; in this illustration, keypoint token $0$ is designated for the right eye and keypoint token $1$ is for the left wrist. The right eye keypoint for all the people in the image are stacked together and moved into token $0$. In this example, we define the system to support up to four people, and if there are fewer than the maximum number of people, they will be padded with zeros in the keypoints. The same goes for keypoints that are not visible. The resulting keypoint tokens will have a length of $N$. In other words, the number of fixed with the keypoint definition does not change with the number of people in the image or changes in image resolution. One thing worth noting is that keypoint token is not discrete; it is a continuous value within the range [0, 1].

The next step is to ensure that the pose embedding has the same dimension as the transformer embedding. We propose two methods; the first one is to pad the keypoint tokens with zeroes to match the transformer dimension. This method is the fastest as it does not require any arithmetic computation. It can accommodate many people up to constraint within $3 M <= d$ where $M$ is the maximum number of people, and $d$ is the transformer dimension.  We tested this method to be working faster. However, for the generality of the unbounded number of people, we use a linear layer to project the keypoint tokens into keypoint embedding. Overall, the KPE converts multi-person keypoints $ \mathbb{R}^{M \times N \times 3}$  into embedding $ \mathbb{R}^{N \times d}$.

\subsection{Training}
To train, the text tokens $T$, keypoint tokens $K$ and image tokens $I$ are concatenated to be fed into the transformer. The transformer output has the same length as the input, and it aims to generate the same tokens as the input tokens. As the text tokens and image tokens are discrete, prediction of them become a multiclass classification problem, and we use the cross entropy loss $\mathcal{L}_{ce}$ as is common with transformer training. However, the keypoint tokens are continuous values, and we use an $L2$ loss $\mathcal{L}_{L2}$ on the keypoint embedding. Therefore, the overall loss function is:
\begin{align}
    \mathcal{L} = \mathcal{L}_{ce}(T) + \lambda_{I}\mathcal{L}_{ce}(I) +  \lambda_{K}\mathcal{L}_{L2}(K)
    \label{eq:kpe}
\end{align}
$\lambda_{I}$ and $\lambda_{K} $ are constants. Higher value encourages more accurate people image and pose respectively, and their values are discussed in Section~\ref{sec:implemantion}.

\subsection{Inference of A Pose Guided Image}
The image token input is left blank at inference, only providing values for the text and keypoint inputs. After a forward pass, we sample the first image token at the transformer output. If we select the image token with the highest probability, the model will generate the same image every time and become deterministic. Therefore, to introduce variation in generating images, we select several tokens with the highest probability and sample them uniformly. The size of the sampling pool is a hyperparameter; setting it too low can give more variation but may also introduce more errors in the image. The sampled token is used in the input image token at the same position and used as a condition together with text and keypoint tokens to predict the next image token. The process then repeats for subsequent image tokens for the given text and pose inputs.

\subsection{Generating Image Sequence}
While our model is not a fully fledged video generation model that trains using video samples like N\"{U}WA, we can use a trained model to generate a sequence of images. This is difficult with DALL-E or similar text-to-image models as the person will appear in random poses across video frames. While pose-to-image models like VQGAN could use pose sequences to ensure consistent poses, they suffer from an inconsistent appearance like clothing colour. We can constrain the appearance by using the same text caption and reducing the sampling pool of image tokens during inference.  To anchor the top of the image result with the previous frame, we use the image tokens from initial image rows from the first video frame as image input for subsequent frames,

\section{Experiments}
In this paper, we implement our KPE model and train it on a multi-person dataset derived from the DeepFashion \cite{deepfashion} dataset. In addition, we implemented two baseline models for comparison. DALL-E\cite{dalle} as an ablation study to understand the effect of using pose guidance on generated image quality. Moreover, we implement a DALL-E\cite{dalle}+VQGAN\cite{vqgan} model to compare the performance of KPE against VQGAN's method of using skeleton images for pose representation. We highlight the advantages of KPE in Section \ref{sec:dalle_vqgan}.

\subsection{DeepFashion Dataset}
We use DeepFashion's fashion synthesis benchmark dataset for the experiments. The original dataset contains 78.5K images of a single person and a brief description of the gender, clothing colour, and type. However, most of the samples in this dataset contain a single person standing in the centre with a neutral pose. We believe that text-to-image models will produce satisfactory results even without pose guidance. This motivated us to derive a multiscale and multi-person dataset from the original images. Therefore, we randomly sampled the single-person images, randomly resized them by 10\%, cropped them, and concatenated them into a single 256 $\times$ 256 image. As a result, the new images will have between 1 to 3 people in various locations, sizes and poses. This adds about 49.5K of images for 2 and 3 people, bringing the total training dataset to 177.5K. In contrast to some generative approaches, we create a train/test split to ensure that the models are not exposed to the reserved test dataset in the training process. Therefore, all image results in the paper were generated using the test set. The test dataset contains 446, 299 and 255 samples for 1, 2 and 3 people. Then we use the OpenPose \cite{openpose} pose estimation approach to obtain the multi-person pose keypoints. We will release code to enable the generation of the multi-person training and test data and splits.

\subsection{Evaluation Metrics}
We evaluate the performance of our approach with a range of quantitative metrics in addition to qualitative results. To evaluate the \emph{realness} of the images, we use the perceptually trained Inception Score (\textbf{IS})\cite{is}, in line with literature image generative models \cite{Ma2017}\cite{Zhu2019}\cite{dalle}\cite{nuwa}\cite{cogview}. However, as studied by \cite{is_note}, IS that uses \cite{inception} pretrained on Imagenet \cite{imagenet} does not work well for data that does not fall into Imagenet's 1000 classes. Unfortunately, Imagenet does not include a human category despite its many classes of clothing, e.g., t-shirts, jeans, sweatshirts, miniskirts etc., that may have a human in it. Therefore, images with perfect clothing but unrealistic looking people may still get high IS. This is worse for the multi-person dataset, spreading the Inception model's prediction to multiple clothing classes, lowering IS.

To overcome this limitation, we propose a new evaluation metric that uses a pose estimation model such as OpenPose \cite{openpose} that was trained on the multi-person dataset. We call this metric People Count Error (\textbf{PCE}). Given an image of people $x$,  $gt()$ indicates ground truth function that returns the labelled number of people in the image, and $h()$ is the function that returns the number of people detected by the pose estimation algorithm; therefore, PCE is defined as:
\begin{align}
    PCE(x) =
    \begin{cases}
    1, & \text{if } h(x)\ne gt(x)\\
    0, & \text{otherwise}
    \end{cases}
\end{align}

Generated images can have false positive errors, such as having additional detected joints or arms in an impossible pose. As most pose estimation approaches such as OpenPose have explicitly learnt the human body anatomy, it would assign the additional body parts in doubt to a new person, increasing the people count and flagging PCE as 1. Unlike IS, which requires a large amount of data, PCE applies to a single image, making it usable to find an error in an individual image. Visual examples of results that PCE would return as 1 are shown in the results in Fig \ref{fig:error}.

A further metric to measure the accuracy of the generated images is Object Keypoint Similarity (\textbf{OKS}) from the MSCOCO keypoint challenge \cite{oks}. OKS measures the $L2$ distance between the ground truth joints labels and the estimation from the generated image, giving higher weights to keypoints perceptually more important like shoulders, knees, hips to eyes, nose, and ears. To evaluate the pixel accuracy against the held out test data, we perform Structural Similarity (\textbf{SSIM}) and, as stated in~\cite{Ma2017} masking out the background. In addition, we crop away the excessive background as we found this can dominate SSIM when people appear small within the image.

\subsection{Implementation details}
\label{sec:implemantion}
We adopt a two-stage training process like DALL-E. The first stage trains VQ-VAE using the VQGAN pipeline on the target images to encode $256 \times 256$ images into $16 \times 16 = 256$ image tokens where each token can assume 8192 possibilities. We use an open-source implementation of DALL-E \cite{dallepytorch} in which the transformer dimension $d$ 512, with 8 heads and a depth of 12 encoder blocks. The text token length is set as 256, and the input text tokens will be truncated if they exceed this length. We train using the loss function in Equation \ref{eq:kpe} using OpenPose's BODY\_25 \cite{body25} pose format. Therefore, the keypoint token length is 25, corresponding to the 25 keypoints.

The DALL-E+VQGAN model is also a text-and-pose guided model. The difference with our KPE model is that it uses VQGAN's pose representation method of using VQ-VAE to encode skeleton images into pose image tokens. Like VQGAN, we reuse the VQ-VAE pretrained on target images. The DALL-E+VQGAN's loss function is:
\begin{align}
    \mathcal{L} = \mathcal{L}_{ce}(T) + \lambda_{I}\mathcal{L}_{ce}(I) +  \lambda_{K}\mathcal{L}_{ce}(P)
\end{align}
where $P$ are the pose image tokens.
We use the same VQ-VAE, training configuration and hyperparameters for all the models. For loss constants, We use $\lambda_{I}$=7 from \cite{dallepytorch}. We tried 1 and 10 for $\lambda_{K}$ but did not notice much difference in the results. Eventually, we select $\lambda_{K}$=10 to have the same order of magnitude as $\lambda_{I}$. For the optimiser, we use Adam \cite{adam}, with initial learning rate of 0.0001, $\beta_{1}$=0.9, $\beta_{2}$=0.999. The learning rate is reduced by half if the loss has plateaued for 12 epochs until it reaches 1e-6. We use a batch size of 10 and train for 100 epochs on an RTX5000 GPU with 16GB GPU memory.

Due to nondeterministic nature of image generation, we compute 5 images per sample in the test dataset and obtain the mean value of the metrics for the qualitative result. As stated before, we sample image tokens from the top 0.1\% highest probability or 8 out of 8192 tokens. This sampling improves the image quality and consistency of metrics values.

\section{Results}

Table \ref{table:deepfashion} shows that KPE tops all the evaluation metrics. The high IS indicates that KPE can generate realistic looking people. Figure \ref{fig:deepfashion_1} show examples showing a variety of genders, the number of people, multiscale person, poses and partial pose with missing keypoints. Given the OKS score of 0.97, which indicate a highly accurate pose, the high mask-SSIM score suggests the generated images have gender and clothing appearance matching the text description.

\begin{figure}[!htb]
\centering
    \begin{subfigure}{0.9\textwidth}
    \centering
    \includegraphics[width=10cm]{images/deepfashion/982.png}
    \caption{Our model can generate image showing fine grained details including fabric texture and fingers.}
    \end{subfigure}

    \begin{subfigure}{0.9\textwidth}
    \centering
    \includegraphics[width=10cm]{images/deepfashion/366.png}
    \caption{KPE model could generate multi-person with multiple scales. This example shows it works with partial pose where the knee keypoints are missing.}
    \end{subfigure}

    \begin{subfigure}{1.0\textwidth}
    \centering
    \includegraphics[width=11cm]{images/deepfashion/451.png}
    \caption{A variety of different poses by three people. Each of the people matches text description in gender and clothing appearance}
    \end{subfigure}

\caption{KPE model can generate photorealistic people with an accurate pose. This figure shows the pose illustration, ground truth, and three generated samples.}
\label{fig:deepfashion_1}
\end{figure}

\subsection{Comparison with DALL-E+VQGAN} \label{sec:dalle_vqgan}
Qualitatively, both KPE and DALL-E+VQGAN produce high-quality people images with an accurate pose. From Table \ref{table:deepfashion} against the baselines DALL-E and DALL-E+VQGAN, we can see that its OKS score matches KPE and is only marginally behind in IS and mask-SSIM, but PCE is twice the error rate of KPE. Apart from generating improved images, there are several advantages of using KPE that makes it an overall superior method:
\begin{itemize}[topsep=0pt]
    \item \textbf{Less memory.} The keypoint token length is an order of magnitude smaller than the pose image token hence requiring less computational memory. In our experiment, the image token length is 256 while there are only 25 keypoint tokens, making it at least 90\% more memory efficient.
    \item \textbf{Faster to run.} The reduction of token number reduces computational complexity, which is in $O(N^2)$ for transformer. Furthermore, encoding pose image using VQ-VAE is computationally expensive and removing this step can contribute to huge speed improvement. As a result, the KPE model's speed is 73\% faster than DALL-E+VQGAN, there is no speed penalty compared to DALL-E.
    \item \textbf{Invariant pose representation.} The same VQ-VAE is used to encode both the pose and target images. However, as VQ-VAE is normally pretrained on natural images, thus the trained VQ-VAE may not perform well on synthetic skeleton images.  In contrast, KPE relies only on the keypoint information and is invariant to the image nor VQ-VAE.
    \item \textbf{Scalable.} Increasing target image resolution or quality will require an increase in image token length hence more memory and slower to run. Since KPE is invariant to the image, the pose processing will not increase computational resources as the image resolution increases. This makes our method easier to scale to higher image resolution.
\end{itemize}

\begin{center}
\begin{table}
\centering
\begin{tabular}{c | c | c| c}
\toprule
  Pose Method & DALL-E & DALL-E+VQGAN & KPE (Our Method) \\ [0.5ex]
\midrule
 Number of pose tokens 	$\downarrow$& - & 256 & \textbf{25} \\
 Relative inference speed $\uparrow$& \textbf{1.73$\times$} & 1.0$\times$ & \textbf{1.73$\times$} \\
 \midrule
 IS 	$\uparrow$ & 2.912 & 3.027 &  \textbf{3.034} \\
 PCE ($\times 10^{-3}$) & 8.2 & 1.2 & \textbf{0.6} \\
  \midrule
 OKS (1 person) $\uparrow$& 0.691 & \textbf{0.976} & \textbf{0.976} \\
 OKS (2 people)$\uparrow$ & 0.555 & \textbf{0.970} & \textbf{0.970} \\
 OKS (3 people)$\uparrow$ & 0.484 & \textbf{0.965}  & 0.964 \\
 OKS (all) $\uparrow$& 0.598 & \textbf{0.970}  & \textbf{0.970} \\
 \midrule
 Mask-SSIM (1 person) $\uparrow$& 0.244 & 0.373 &  \textbf{0.384}\\
 Mask-SSIM (2 people) $\uparrow$& 0.302 & 0.457  & \textbf{0.461} \\
 Mask-SSIM (3 peoples) $\uparrow$& 0.260 & \textbf{0.456}  & 0.454 \\
 Mask-SSIM (all) $\uparrow$& 0.265 & 0.420  & \textbf{0.424} \\[0.5ex]

\bottomrule
\end{tabular}
\caption{Evaluation of different models on DeepFashion multi-person dataset. Our method KPE achieves highest scores in all metrics.}
\label{table:deepfashion}
\end{table}
\end{center}

\vspace{-2em}
\subsection{Ablation Study} \label{sec:error}
\begin{figure}
\centering
    \begin{subfigure}{0.19\textwidth}
    \includegraphics[width=1\textwidth]{images/error/no_pose_286_2.png}
    \caption{}
    \label{fig:error_a}
    \end{subfigure}
    \begin{subfigure}{0.19\textwidth}
    \includegraphics[width=1\textwidth]{images/error/no_pose_192_3.png}
    \caption{}
    \label{fig:error_b}
    \end{subfigure}
    \begin{subfigure}{0.19\textwidth}
    \includegraphics[width=1\textwidth]{images/error/no_pose_950_1.png}
    \caption{}
    \label{fig:error_c}
    \end{subfigure}
    \begin{subfigure}{0.19\textwidth}
    \includegraphics[width=1\textwidth]{images/error/no_pose_15_3.png}
    \caption{}
    \label{fig:error_d}
    \end{subfigure}
    \begin{subfigure}{0.19\textwidth}
    \includegraphics[width=1\textwidth]{images/error/no_pose_687_3.png}
    \caption{}
    \label{fig:error_e}
    \end{subfigure}

    \caption{Top row are erroneous images generated using DALL-E, and the bottom row shows their keypoints obtained from the images. PCE can capture image errors by comparing the generated and intended number of people.}
\label{fig:error}
\end{figure}
We did an ablation study comparing KPE against DALL-E, a text-to-image model without pose guidance. Looking at Table~\ref{table:deepfashion}, apart from having lower IS than the other two pose-guided models, it has the highest PCE rate at 8.2 $\times 10^{-3}$, which is over 13 times higher than KPE. Figure \ref{fig:error} shows an example of images that contains errors and how we can spot the error by using PCE. Figure \ref{fig:error_a} contains two realistically looking people but with an additional long arm floating in the centre of the image. The floating arm is assigned to the third person, causing PCE=1 as $h(x)\ne gt(x)$. Although measuring the discrepancy in people's count does not catch every error, it allows a standardised metric across a wide range of examples without the need for a manual visual inspection on each. Figure \ref{fig:error_b} to  \ref{fig:error_d} show examples of additional body parts where PCE=1, some of which can be difficult to see initially, like the additional face in Figure \ref{fig:error_d}. Also, we found that DALL-E sometimes generates fewer or more people than the text description. When it happens, it tends to contain some standalone body parts like Figure \ref{fig:error_a} and Figure \ref{fig:error_e}. The PCE can pick up the error in Figure \ref{fig:error_e} despite OpenPose failing to detect the incomplete person in the centre.

\subsection{Generation of an Image Sequence}
\begin{figure}
\includegraphics[width=1\textwidth]{images/video/video2.png}
\caption{A series of images generated by conditioning on the text and sequence of poses. The top rows of image tokens covering the head are also fixed as conditions to generate subsequent image tokens.}
\label{fig:video_1}
\end{figure}

Figure \ref{fig:video_1} shows how we can animate the image sequence by feeding text input and pose sequence to our KPE model. The text input constraint provides appearance consistency from image to image. Thus, enabling the creation of consistent and highly accurate image sequences driven by an external source, such as a webcam (as was done in this case).

\section{Limitations}
The DeepFashion dataset is hugely imbalanced, where men form only a tiny fraction of the dataset, and the rest are long-haired white females. Therefore, a pose-only guided model trained on the dataset is more likely to generate female images. To understand the effect of the gender bias, we generated images of various poses using the exact text prompt  ``a man wore blue shirt'' as shown in Figure \ref{fig:limitation}.

\begin{figure}
\centering
    \begin{subfigure}{0.24\textwidth}
    \includegraphics[width=1\textwidth]{images/limitation/image_121_man_22.png}
    \caption{}
    \label{fig:limitation_a}
    \end{subfigure}
    \begin{subfigure}{0.24\textwidth}
    \includegraphics[width=1\textwidth]{images/limitation/image_68_man_4.png}
    \caption{}
    \label{fig:limitation_b}
    \end{subfigure}
    \begin{subfigure}{0.24\textwidth}
    \includegraphics[width=1\textwidth]{images/limitation/image_115_man_2.png}
    \caption{}
    \label{fig:limitation_c}
    \end{subfigure}
    \begin{subfigure}{0.24\textwidth}
    \includegraphics[width=1\textwidth]{images/limitation/image_86_man_4.png}
    \caption{}
    \label{fig:limitation_d}
    \end{subfigure}
    \caption{Text of ``a man wore blue shirt'' was used to generate images from pose that is more masculine (a) towards more feminine pose in (d).}
\label{fig:limitation}
\end{figure}

We can see in Figure \ref{fig:limitation_a} and \ref{fig:limitation_b} that despite the gender bias in the dataset, our model can generate convincing men from neutral poses. However, when presented with poses that are exclusive to females in the dataset, the generated images (Figure \ref{fig:limitation_c} and \ref{fig:limitation_d}) lean toward feminine appearance. This limitation implies that pose is not entirely disentangled from gender, and the model learned the gender bias from poses containing information about body proportion. Given this insight, in future, we will collect and apply our approach to a more balanced dataset in the future to address this bias in gender and ethnicity.

\FloatBarrier
\section{Conclusions}
We have proposed the first pose+text to image generative model creating photorealistic multi-person images. We methodologically show that adding pose as guidance improves the image quality over SOTA text only guided generative image model.  We also created a novel multi-person pose encoding method suitable for modern transformer architecture, \textbf{KPE}. Not only does KPE generates a better image and produce a more accurate pose than existing methods, but it is also computationally efficient and does not add inference speed overhead. This allows for easy integration into transformer models. To measure the performance of generative images of people, we propose a suitable metric \textbf{PCE} to detect the false positive occurrence of generated people, overall creating SOTA performance both quantitatively and qualitatively.

\clearpage
\bibliographystyle{unsrt}