\section{Coherent, summed,  and incoherent cross sections}

\textbf{\textit{Summed cross section off the deuteron}}.
Applying the formalism of Ref.~\cite{Guzey:2018tlk} to $J/\psi$ photoproduction on the deuteron
and using completeness of final nuclear states,
one obtains the following expression for the summed (coherent+incoherent) $\gamma +d \to J/\psi+X$ differential cross section
\begin{equation}
\frac{d\sigma_d^{\rm sum}(t)}{dt}=\frac{\chi^2}{4\pi} \int d^2 {\vec b} \int d^2 {\vec b^{\prime}} e^{i \vec{q} \cdot
({\vec b}-\vec{b^{\prime}})} \langle 0|\Gamma_d^{\dagger}(\vec{b^{\prime}}) \Gamma_d(\vec{b}) |0 \rangle \,,
\label{eq:cs_sum}
\end{equation}
where $\chi$ is proportional to the $\gamma - J/\psi$ transition amplitude.
In Eq.~(\ref{eq:cs_sum}),
$\Gamma_d({\vec b})$ is the scattering amplitude on the deuteron in impact parameter space;
$\vec{b}$ and ${\vec q}$ are two-dimensional vectors of the impact parameter and the momentum transfer, respectively;
the latter is taken to be transverse, $t=-|{\vec q}|^2$.
The matrix element $\langle 0| \dots |0\rangle$
stands for the integration with the deuteron ground-state wave function $\Psi_d(\vec{r})$ squared
\begin{equation}
\langle 0| \dots |0\rangle=\int d^3 \vec{r}\, |\Psi_d(\vec{r})|^2 \,,
\label{eq:wf}
\end{equation}
where $\vec{r}$ is the relative distance between the proton and the neutron in the deuteron; averaging over
the deuteron polarization is implied.
In our analysis, we used the Paris deuteron wave function~\cite{Lacombe:1981eg}.

The nuclear scattering amplitude is expressed in terms of the nucleon amplitudes $\Gamma_N$ and $\Gamma_Y$ as follows
(we do not distinguish protons and neutrons)
\begin{eqnarray}
\Gamma_d(\vec{b},\vec{r})&=&\Gamma_N(\vec{b}-\frac{\vec r}{2})+\Gamma_N(\vec{b}+\frac{\vec r}{2})-\Gamma_N(\vec{b}-\frac{\vec r}{2})\Gamma_N(\vec{b}+\frac{\vec r}{2}) \nonumber\\
&+&\Gamma_Y(\vec{b}-\frac{\vec r}{2})+\Gamma_Y(\vec{b}+\frac{\vec r}{2})-\Gamma_N(\vec{b}-\frac{\vec r}{2})\Gamma_Y(\vec{b}+\frac{\vec r}{2}) \nonumber\\
&-& \Gamma_Y(\vec{b}-\frac{\vec r}{2})\Gamma_N(\vec{b}+\frac{\vec r}{2})
-\Gamma_Y(\vec{b}-\frac{\vec r}{2})\Gamma_Y(\vec{b}+\frac{\vec r}{2}) \,,
\label{eq:Gamma_d}
\end{eqnarray}
where $\Gamma_N$ and $\Gamma_Y$ correspond to the $\gamma +N \to J/\psi+N$ (elastic production) and
$\gamma +N \to J/\psi+Y$ (nucleon dissociation) processes, respectively.

Substituting Eq.~(\ref{eq:Gamma_d}) in Eq.~(\ref{eq:cs_sum}), keeping the terms even in $\Gamma_Y$,
 and neglecting the small contribution ${\cal O}(\Gamma_Y^2)$

 to the proton-neutron interference terms (both in IA and the shadowing correction), one obtains for the summed cross section
\begin{eqnarray}
\frac{d\sigma_d^{\rm sum}(t)}{dt} &=&
2 \frac{d\sigma_{\rm el}(t)}{dt}\left(1+F_d(4 t)\right)+2 \frac{d\sigma_{pd}(t)}{dt} \nonumber\\
&-&8\frac{d\sigma_{el}(t=0)}{dt} e^{-(B_{el}+B)t/2} \nonumber\\
 & \times&  \frac{d\sigma_{\rm diff}/dt(t=0)}{\sigma_{\rm tot}}  \int dt^{\prime} e^{-B t^{\prime}} F_d(4 t^{\prime})\,,
\label{eq:cs_sum2}
\end{eqnarray}
where $d\sigma_{\rm el}(t)/dt$ is the $\gamma + p \to J/\psi+p$ elastic production cross section on the proton,
$d\sigma_{\rm el}(t=0)/dt$ is its value at $t=0$, and $B_{\rm el}=4.3 \pm 0.2$ GeV$^{-2}$ is the slope of its $t$ dependence~\cite{Alexa:2013xxa};
$d\sigma_{pd}/dt$ is the cross section of nucleon-dissociative $J/\psi$ photoproduction $\gamma +N \to J/\psi+Y$;
$F_d(t)$ is the deuteron charge form factor. Note that the deuteron quadrupole form factor does not contribute to the summed cross section, see Ref.~\cite{Franco:1969qj}.

In Eq.~(\ref{eq:cs_sum2}), the first line is the impulse approximation (IA) and the second and third lines is the shadowing correction.
A characteristic feature of the summed cross section is the $4t$-argument of the deuteron form factor,
which makes the slope of the $t$ dependence at small $t$ to be two times as large as that for the coherent cross section on the deuteron in IA.

In LTA, the shadowing correction for $J/\psi$ photoproduction is expressed in terms of the  gluon contribution to the $d\sigma_{\rm diff}(t=0)/dt$ differential cross section of inclusive diffraction in deep inelastic scattering (DIS) on the proton
 $\gamma^{\ast} + p \to X +p$, the slope of its $t$ dependence $B \approx 6$ GeV$^{-2}$~\cite{H1:2006uea},
 and the gluon contribution to the  $\sigma_{\rm tot}$ total photon-proton cross section $\gamma^{\ast}+p \to X$, see details in~\cite{Frankfurt:2011cs}.
 This allows us to conveniently express the shadowing correction in Eq.~(\ref{eq:cs_sum2}) in terms
 of the shadowing correction to the gluon distribution in the deuteron $g_d(x,Q^2)$,
  \begin{eqnarray}
 S_g(x)&=&\frac{g_d(x,Q^2)}{2 g_N(x,Q^2)} \nonumber\\
 &=&1-\frac{d\sigma_{\rm diff}/dt(t=0)}{\sigma_{\rm tot}}
\int d t^{\prime} e^{-Bt^{\prime}} F_d(4 t^{\prime}) \,,
\label{eq:S_g}
\end{eqnarray}
where $g_N(x,Q^2)$ is the free nucleon gluon distribution. Note that all involved cross sections implicitly depend on energy (Bjorken $x$).

Equation~(\ref{eq:S_g}) allows us to rewrite the expression for the summed cross section in the following final
compact form
\begin{eqnarray}
\frac{d\sigma_d^{\rm sum}(t)}{dt}&=&2 \frac{d\sigma_{\rm el}(t)}{dt}\left(1+F_d(4 t)\right)+2 \frac{d\sigma_{pd}(t)}{dt} \nonumber\\
&-&8\frac{d\sigma_{\rm el}(t=0)}{dt} (1-S_g(x)) e^{-(B_{\rm el}+B) t/2} \,.
\label{eq:cs_sum3}
\end{eqnarray}

Since the deuteron is the lightest nucleus and
the STAR kinematics corresponds to $W=25$ GeV~\cite{STAR:2021wwq},
$x=M_{J/\psi}^2/W^2=0.015$, and $Q^2 \approx M_{J/\psi}^2/4 \approx 2.4-3$ GeV$^2$ ($M_{J/\psi}$ is the mass of $J/\psi$ and $Q$ is a
generic value of the resolution scale probed in $J/\psi$ photoproduction), the gluon shadowing correction in this case
is very small, $S_g(x)=0.99$. i.e., on the order of 1\%~\cite{Frankfurt:2011cs}.

\textbf{\textit{LTA predictions for $t$ dependence of $d+Au$ UPC cross section and comparison to STAR data}}.
It is customary to present UPC cross sections as a function of the rapidity of the produced final state, in the considered case,
the $J/\psi$ rapidity $y$. In the STAR notation, the UPC cross section and the cross section at the photon level are
related via the photon flux $\Phi_{T, \gamma}$
\begin{equation}
\frac{d^2 \sigma^{d+Au \to J/\psi+X}}{dt dy}=\Phi_{T, \gamma} \frac{d^2 \sigma^{\gamma+d \to J/\psi+X}}{dt dy} \,,
\label{eq:upc_star}
\end{equation}
where $t =-|\vec{p}_T|^2$ and $\vec{p}_T$ is the $J/\psi$ transverse momentum.
Since the STAR results are given for $d^2 \sigma^{\gamma+d \to J/\psi+X}/(dt dy)$, our predictions can be readily
compared to the data. This is presented in Fig.~\ref{fig:JPsi_RHIC}, which shows predictions of
Eq.~(\ref{eq:cs_sum3}) for the
summed $\gamma +d \to J/\psi+X$ cross section as a function of $|t|$ (blue dot-dashed curve) and the STAR data~\cite{STAR:2021wwq}
for the total cross section (filled circles). The error bars represent statistical and systematic errors added in quadrature.
 A comparison of the LTA prediction to the filled circles shows that while the theory
describes well the data within experimental errors at small $|t|$, it significantly overestimates the data for $|t| > 0.2$ GeV$^2$.

This discrepancy cannot be attributed to the gluon nuclear shadowing since it affects the summed cross section only at the
level of a few percent. In fact, with $1-2$\% precision, Eq.~(\ref{eq:cs_sum2}) for $|t| > 0.2$ GeV$^2$ can be
written in the following approximate simple form
\begin{equation}
\frac{d\sigma_d^{\rm sum}(t)}{dt} \approx 2 \frac{d\sigma_{\rm el}(t)}{dt}+2 \frac{d\sigma_{pd}(t)}{dt} \,,
\label{eq:cs_sum2_app}
\end{equation}
thus emphasizing that the gluon nuclear shadowing or details of the nuclear structure (deuteron wave function)
do not play a role here.

\begin{figure}[t]
\begin{center}
\epsfig{file=JPsi_RHIC_prc_new2.eps,scale=1.}
\caption{The summed $\gamma +d \to J/\psi+X$ differential cross section as a function of $|t|$: the LTA predictions (blue dot-dashed curve) and the STAR data~\cite{STAR:2021wwq} for the total cross section.
The experimental statistical and systematic errors have been added in quadrature.}
\label{fig:JPsi_RHIC}
\end{center}
\end{figure}

At this point it is important to discuss the input for the elastic and proton-dissociative cross sections
of $J/\psi$ photoproduction on the proton. In our analysis, we used the low-energy (LE) H1 parametrization based on the data covering the $25 \ {\rm GeV} < W < 80$ GeV interval~\cite{Alexa:2013xxa}.
In particular, we combined results of the fit for the $W$ dependence in Table 3 of Ref.~\cite{Alexa:2013xxa}
with that for the $t$ dependence in Table 2. The resulting parameterization reads
\begin{eqnarray}
\frac{d\sigma_{el}(t)}{dt} &=& N_{el} \left(\frac{W}{90}\right)^{\delta_{el}}  b_{el} e^{- b_{el}|t|} \,, \nonumber\\
\frac{d\sigma_{pd}(t)}{dt} &=& N_{pd} \left(\frac{W}{90}\right)^{\delta_{pd}}  \frac{f_{pd}(t)}{\int dt^{\prime}  f_{pd}(t^{\prime})} \,,\label{eq:sigma_el}
\end{eqnarray}
where $N_{el}=81 \pm 3$ nb, $\delta_{el}=0.67 \pm 0.03$, and
$b_{el}=4.3 \pm 0.02$ GeV$^{-2}$ for the elastic case; $N_{pd}=62 \pm 12$ nb,  $\delta_{pd}=0.42 \pm 0.05$, and
 $f_{pd}(t)=1/(1+b_{pd}|t|/n)^n$ with $n=3.58$ and $b_{pd}=1.6 \pm 0.2$ GeV$^{-2}$ for the proton-dissociative case.

Note that the H1 analysis included the proton-dissociative masses in the range $m_p < M_Y < 10$ GeV~\cite{Alexa:2013xxa}.
When comparing our calculations to the RHIC UPC data,
one has to keep in mind that in selection of UPC events, the STAR detector covers a wide range of rapidities vetoing
events with dissociation.  This includes $|y| <1$ covered by the Time Projection Chamber (TPC), $3.4 < |y| < 5$ covered by
two Beam-Beam Counters (BBCs), and Zero Degree Calorimeters (ZDCs) tagging forward neutrons.
To
explore this possibility, we set  $d\sigma_{pd}/dt=0$ in Eq.~(\ref{eq:cs_sum3}).
Note that since $d\sigma_{pd}/dt \gg d\sigma_{\rm el}/dt$ for sufficiently large $|t|$, the discrepancy mentioned
above should originate from modeling of $d\sigma_{pd}/dt$.
The resulting prediction is given by the red solid curve in Fig.~\ref{fig:JPsi_RHIC2} and leads to a very good agreement with the
STAR data over the entire range of $t$. For comparison, we also show our result from Fig.~\ref{fig:JPsi_RHIC} by the blue
dot-dashed curve.

\begin{figure}[t]
\begin{center}
\epsfig{file=JPsi_RHIC_prc2_new.eps,scale=1.}
\caption{
The summed $\gamma +d \to J/\psi+X$ differential cross section as a function of $|t|$: the LTA predictions
with the $d\sigma_{pd}/dt=0$ scenario (red solid curve) vs.~the STAR data~\cite{STAR:2021wwq} for the total cross section (filled
circles).
The data for the neutron-tagged cross section are shown by open circles.
Our result from Fig.~\ref{fig:JPsi_RHIC} is given by the blue
dot-dashed curve.
}
\label{fig:JPsi_RHIC2}
\end{center}
\end{figure}

Note that the H1 measurement of proton-dissociative $J/\psi$ photoproduction~\cite{Alexa:2013xxa} is the only collider
data in the relevant range of $W=25$ GeV; the earlier ZEUS data~\cite{ZEUS:2002vvv} correspond to higher $W$,
$80 \ {\rm GeV} < W < 120$ GeV.

The comparison of the LTA results to the RHIC data presented in Fig.~\ref{fig:JPsi_RHIC2} should be taken with some degree of caution.
First, the RHIC data corresponds to the rapidity range $|y| < 1$, where the UPC cross section of $J/\psi$ photoproduction
at RHIC energies has a rapid $y$ dependence, see, e.g.~\cite{Strikman:2005ze,Frankfurt:2006tp}. Hence, it is important to
model and take into account corrections for the acceptance smeared in $y$.
Second, to validate our hypothesis of the absence of the nucleon-dissociative contribution, it is important to model
the acceptance including the ZDCs and the distribution over the momentum of the recoiled nucleon.
This will also enable one to utilize the ``n-tagged data" (open circles in Fig.~\ref{fig:JPsi_RHIC2}),
which should help to clarify the role of the nucleon-dissociative term.

To explain the latter point, we notice that the incoherent cross section is given by the difference between the summed and
coherent cross sections, which in IA reads
\begin{eqnarray}
&&\frac{d\sigma_d^{\rm inc}(t)}{dt} =\frac{d\sigma_d^{\rm sum}(t)}{dt}-\frac{d\sigma_d^{\rm coh}(t)}{dt} \nonumber\\
&&=
2 \frac{d\sigma_{\rm el}(t)}{dt}\left(1+F_d(4 t)-2|F_d(t)|^2\right)+2 \frac{d\sigma_{pd}(t)}{dt} \,.
\label{eq:cs_incoh}
\end{eqnarray}
Thus, our hypothesis of the absence of a noticeable proton-dissociative contribution seems to be supported
by ``n-tagged data" for $|t| < 0.4$ GeV$^2$ since the difference between the solid and open circles in Fig.~\ref{fig:JPsi_RHIC2}
can be explained
by the nuclear coherent contribution. At the same time, the ``n-tagged data" with $|t| > 0.4$ GeV$^2$ look rather puzzling;
their analysis
requires detailed modeling.

We discussed in the Introduction that the STAR data on $J/\psi$ photoproduction in $d+Au$ UPCs is described well with the CGC framework with nucleon shape
fluctuations~\cite{Mantysaari:2019jhh}. We also explained above that the effects of leading twist gluon nuclear shadowing and
 gluon saturation as well as details of the deuteron structure are largely unimportant for $|t | \geq 0.2$ GeV and
 the summed cross section on the deuteron should be reproduced sufficiently accurately by approximate Eq.~(\ref{eq:cs_sum2_app}).
 Therefore, the difference in predictions of the LTA and CGC approaches
 could come from different extrapolations of the
 $d\sigma_{\rm el}(t)/dt$ and $d\sigma_{pd}(t)/dt$ cross sections on the proton from HERA to RHIC energies.

\textbf{\textit{Summary}}.
In this letter, we argue that the STAR data on $J/\psi$ photoproduction in
$d+Au$ UPCs is not sensitive to leading twist gluon nuclear shadowing,
gluon saturation, or details of the deuteron wave function, but instead primarily probes
elastic and probably
proton-dissociative $J/\psi$ photoproduction on quasi-free nucleons. Exploring the possibility that the latter
contribution is negligibly small, we obtain a very good description of the $t$ dependence of the STAR total cross section.
We point out that the difference between the LTA and CGC predictions
could come from different extrapolations of the
 elastic and proton-dissociative $J/\psi$ photoproduction cross sections on the proton from HERA to RHIC energies.

{\textbf{\textit{Acknowledgements}}.
The research of M.S.~was supported by the US Department of Energy Office of Science, Office of Nuclear Physics under
Award No.~DE-FG02-93ER40771. M.S.~thanks Theory Division of CERN for
hospitality while this work was done.


\section{Introduction}
\label{sec:intro}
Digital entertainment industry is growing multifold with ease of internet access and numerous options for on-demand streaming platforms such as Amazon Prime Video, Netflix, Hulu etc. These providers increase their viewership by enabling content in local languages.
Translation of subtitles across languages is a preferred cost effective industry practice to maximize content reach.
Subtitles are translated using bilingual (and sometimes multilingual) translators. They watch the content and use a source subtitle in one language to translate it to another language. Low translation quality and high man-power cost which grows significantly with
scarcity of target language resources are some problems with bilingual translators.
Low translation quality can cause increased usage drop-off and hurt content viewership for audience of target language.
Hence, translation quality estimation (QE) is one crucial step in the process.
Currently, a second translator evaluates the quality making evaluation as expensive as generating the translation itself.

Automated QE has been studied through the lens of binary classification between acceptable and unacceptable translation or scoring (or rating) to assign a score of translation acceptability within a given range.
However, binary classification ignores ``loosely" translated samples that often occur due to human judgment like paraphrasing, under-translation or over-translation.
Translators often rephrase sentences using contextual information from the video that is not available in the source sentence.
For the scoring approach, gathering large enough sample of reliable human validated data to train a supervised system is very expensive, time-consuming and does not scale to new languages.

In this work, we propose an automated QE system DeepSubQE~reducing both cost and time in subtitle translation while assuring quality.
To overcome the problem with conventional binary approaches, we introduced a third category of translation called Loose translations.
Our system takes a pair of sentences as input; one in source language and one in target language and classifies it in one of three categories --- \textit{Good translation}, \textit{Loose translation} or \textit{Bad translation}. This paper makes the following contributions:
\begin{itemize}
    \item We develop a novel system that can estimate quality of translations generated by either humans or MT systems with application to video subtitling.
    \item We demonstrate achieving good generalization for subtitling QE by augmenting data with various strategies including signals from learners that themselves fail to generalize as well for the task.

    \item We present a formulation that can handle paraphrasing and other contextually acceptable non-literal translations through appropriate synthesis of \textit{Loose Translations}.

\end{itemize}

The paper is divided in following sections. Section \ref{sec:prev-work} describes the different existing methods used for QE. Section \ref{sec:approach} discusses our approach to the problem, followed by section \ref{section:datasetgeneration} which explains the details of the dataset which we generated for training. Section \ref{sec:model-arch-train} describes the model architecture and training environment. Section \ref{sec:experiments} presents the experiments, results and observations. Section \ref{sec:conclusion} concludes our findings and presents future endeavours.

\section{Related work}
\label{sec:prev-work}
QE is important for evaluation of machine translation (MT) systems.
Automatic evaluation of machine translation systems is a well studied topic.
Metrics like BLeU \cite{Papineni2001BleuAM} and METEOR \cite{Banerjee2005METEORAA} are some industry-wide accepted metrics to evaluate a translation where a reference text is available.
Specifically, for each candidate translation (that is machine generated for MT systems) a reference generated by human translator is necessary for computing the metric.
However, we are interested in the setting where no reference is available.

Alternatively, HTER \cite{Snover2006ASO} is a metric used to estimate the post-edits required to improve an MT generated candidate translation to the level of human translation.
Numerous models have been proposed to predict HTER for a given MT output \cite{DBLP:conf/coling/BlatzFFGGKSU04, Specia2009EstimatingTS}.
State of the art for predicting HTER is given by a two-level neural Predictor-Estimator (PE) model \cite{DBLP:conf/wmt/KimLN17}.
PE finds the closest matching target token for each token in source and uses such matches to find the aggregate quality of translation for sentence pairs.

\begin{figure}[h]
    \includegraphics[width=\columnwidth]{resources/WMT-model-output-on-trainer-dataset.pdf}
    \caption{HTER score distribution of PE Model on our dataset computed using OpenKiwi toolkit \cite{DBLP:conf/acl/KeplerTTVM19}, see in color.}
    \label{fig:wmt-output-on-dataset}
\end{figure}

However, HTER metric is trained on data generated from MT output and hence is biased to error patterns generated by such systems.
But errors from subtitle files have a very different distribution and its patterns are not aligned with those that HTER captures.
For example, subtitles sometimes have complete mistranslations due to alignment errors that lie outside the space of MT outputs that HTER models are trained on.
Figure \ref{fig:wmt-output-on-dataset} shows lack of separation of HTER predictions from PE model on our three class data of Good, Loose and Bad.
Hence, HTER is unsuitable for our task of evaluating subtitle quality.

Our requirement is a method that evaluates quality subtitle data that could either be human or MT generated.
The set of translations span complete mistranslations (due to issues like alignment errors), loose translations (from additional contextual information and paraphrasing) and good translations (literal translations with complete overlap of meaning).
Apart from mistranslations, errors could also arise from drift in translations, captioning of non-spoken content, etc (see~\cite{DBLP:journals/corr/abs-1909-05362} for a survey).
None of the existing methods directly apply to our problem and we work on tailoring one accordingly to our use case.
We define a three way classification to account for the three classes of translation output.
Using signals from multiple diverse methods we gather data the represents our notion of classes.
A neural network is trained on this data that classifies a given pair of subtitle blocks to one of the three classes.

\section{Approach}
\label{sec:approach}

We define the QE problem given a subtitle file $A$ in source language and its translation in target language $B$.
Quality is measured on individual pairs of translation text blocks $(A_i$, $B_i)$ that are matched by timestamps.
Each text block could contain more than one sentence spoken by multiple speakers and captions for non-spoken content (such as whispers, laughs, loudly, etc).
The problem is to assign the translation $\hat{y} \gets f(A_i, B_i)$ to one of three categories $\hat{y} \in \{\text{Good, Loose, Bad}\}$.
A translation is \textit{Good}  if it is a perfect or near-perfect translation retaining all meaning from source and reads fluently.
It is \textit{Loose} if it is paraphrased or contains some contextual information not available in source text.
Translations of colloquial phrases and idioms also lie in this category.
\textit{Bad} translations are those in which the sentence pair have no overlap of meaning and the target is disconnected from the context in the video.

\begin{table*}[htbp]
  \centering
  \caption{Data distribution from different sources (in \%)}
    \begin{tabular}{r|c|c|c|c|c|c}
      & \textbf{Statistical} & \multirow{2}[0]{*}{\textbf{NMT}} & \textbf{Added} & \textbf{Scrambled} & \textbf{Drifted } & \textbf{Randomly} \\
      & \textbf{Classification} &   & \textbf{Captions} & \textbf{Text} & \textbf{Aligned} & \textbf{ Aligned} \\ \hline
        French & 18.83 & 33.76 & 6.58 & 6.58 & 17.13 & 17.13 \\
        German & 17.26 & 32.14 & 7.07 & 7.07 & 18.23 & 18.23 \\
        Italian & 16.44 & 32.95 & 6.57 & 6.57 & 18.74 & 18.74 \\
        Portuguese & 16.47 & 33.09 & 6.59 & 6.59 & 18.63 & 18.63 \\
        Spanish & 17.82 & 29.15 & 7.01 & 7.01 & 19.50 & 19.50 \\
    \end{tabular}%
  \label{tab:datasource-distribution}%
\end{table*}

Gathering sufficient human labelled data to train a supervised system is both expensive and time-consuming.
Lack of suitable publicly available subtitle data for this task motivated us to reuse large volumes of unlabelled subtitles.
We use signals like timestamp alignment and overlap statistics between source and target along with MT output for short sentences with common phrases for synthesizing samples from the three classes to learn the QE classifier.
Our augmentation methods also use statistical classifiers of lower capacity trained on external data.
The diversity and quality of data generated is critical for the QE classifier to learn good classification boundaries and generalize sufficiently well to unseen data.
We show from experiments on subtitles and other parallel corpora that by fitting a QE classifier of sufficiently high capacity to data generated as described here we get good generalization for our task.
Further details of our data augmentation methods are described in Section~\ref{section:datasetgeneration}.

We experiment with multiple neural network architectures for the classifier beginning with simpler ones with only CNNs and LSTMs.
A hybrid architecture of Bidirectional LSTM (BiLSTM) \cite{Graves2005FramewisePC} followed by CNN outperformed them.
Input to each model is a pair of sentences in two languages that outputs the probability of it belonging to three three classes.
We limit the length of each sentence to 25 tokens, lowercase each word and include punctuation and numbers.
Section~\ref{fig:modelarch} describes the model in detail.

\begin{figure*}[htbp]
    \includegraphics[width=\linewidth]{resources/MeaningRetentionv2-model.png}
    \caption{Visualization of DeepSubQE\ model's architecture.}
    \label{fig:modelarch}
\end{figure*}

\section{Data augmentation}
\label{section:datasetgeneration}

We begin with 30k video subtitle files in English with timestamp aligned translations into five languages (French, German, Italian, Portuguese, Spanish).
These translation text blocks are unlabeled and there is no information on which of three three QE classes they belong to.
We use various features measuring statistics of word and semantic overlap  between the source and the target as signals to label them.
These features are used with two different methods to assign two distinct scores to each sample indicating the likelihood of it being a Good or a Bad sample.
Samples where both scores are in strong agreement are given the corresponding class label of Good or Bad.
Others where both scores are aligned but only one of them is a strong indicator with high magnitude are marked as Loose.
The rest of the sample with no agreement among the scores are discarded.

\textbf{Bag-of-words model} (BOW). The first is a two parameter model that uses aligned pretrained embeddings~\cite{DBLP:journals/corr/abs-1710-04087} to score the sentence pair.
The embeddings are used create a cosine similarity matrix $S$ of size $(N \times M)$ for a source and target sentences with $N$ and $M$ words respectively.
A score $s_{\textrm{src}}$ ($s_{\textrm{tgt}}$) is computed for the source (target) by thresholding at $\theta_1$ and taking a max over the columns (rows) and averaging over the rows (columns). Correspondingly,
\begin{align}
    s_{\textrm{src}} &= \frac{1}{N} \sum_{i=1}^{N} \max_{j\in[M]}  \tau_{\theta_1}( S_{ij} ) ~\textrm{and}\\
    s_{\textrm{tgt}} &= \frac{1}{M} \sum_{i=1}^{M} \max_{j\in[N]}  \tau_{\theta_1}( S_{ij} ),
\end{align}
where $\tau_{\theta_1}(~\cdot ~)$ is element-wise thresholding at $\theta_1$.
The intuition is to aggregate similarity scores from the most relevant words of target for each word in the source and vice-versa.
The model assigns a score $s_{\textrm{BOW}} = \textrm{minimum}(s_{\textrm{src}}, s_{\textrm{tgt}})$.
The parameters are chosen by tuning on validation data of positives from NMT (implemented using ~\cite{DBLP:journals/corr/abs-1712-05690}) output and negatives from misaligned subtitles with no learning.
All samples labeled through this method are from subtitles which helps the final QE model to learn common patterns in video subtitles.

\textbf{Random Forest Classifier} (RFC).
This model uses features similar to BOW model with MUSE embeddings to train a random forest classifier on EuroParl dataset~\cite{Tiedemann2012ParallelDT}. Translations in EuroParl are augmented with errors such as incorrect word substitution and random sentence alignment to generate incorrect translations.
The model benefits from Europarl's paraphrasings that help it learn beyond literal translations.
We used about $600$k samples for each language to train and tuned parameters by cross-validation on a validation set.
RFC model assigns a probability score $s_{\textrm{RFC}}$ to each input sentence pair.

The two scores, $s_{\textrm{BOW}}$ and  $s_{\textrm{RFC}}$, both of which are in range $[0,1]$ are now used to label data to assign a label $\hat{y}$ following,

\begin{gather}
    \hat{y} =
    \begin{cases}
        \text{Bad} & \textrm{if}~ s_{\textrm{BOW}} \leq \delta_1 \wedge s_{\textrm{RFC}} \leq \delta_1, \\
        \text{Loose} & \textrm{if}~s_{\textrm{RFC}} \leq \delta_3 \wedge s_{\textrm{RFC}} \geq \delta_2, \\
        \text{Good} & \textrm{if}~s_{\textrm{BOW}} \geq \delta_4 \wedge s_{\textrm{RFC}} \geq \delta_4 \\
    \end{cases}
    \label{eq:weak-classifier-final-calc}
\end{gather}
where the thresholds are manually set to $(\delta_1,\delta_2,\delta_3,\delta_4)=(0.25,0.4,0.7,0.8)$.
This labels about $69\%$ of all samples into three classes.
All others samples with scores that do not fall in the specified ranges are filtered out due to disagreement among the models.
We refer to this set as \textit{Statistical Classification}.
Performance and training details of BOW and RFC are reported in Appendix \ref{appendix:binary-classifiers}.
We further augment this data with ~\textit{NMT} on short sentences with frequent phrases like greetings with the Good label.
More samples for the Loose category are generated by adding captions to the source (like whispers, sighs, etc) referred as \textit{Added Captions} or changing word order in the target to degrade fluency called \textit{Scrambled Text} samples.
This constitutes all the positives (Good and Loose) while we synthesize negatives (Bad) in two ways; we randomly choose a target for each source for easy negatives (called \textit{Random Aligned}) and  to choose a target from a temporally close block for hard negatives (called \textit{Drifted alignment}).
Label distribution of the final data is reported in Table~\ref{tab:datalabel-distribution}.

\begin{table}[htbp]
  \centering
  \caption{Dataset label distribution (in \%)}
    \begin{tabular}{r|c|c|c}
      & \textbf{Bad} & \textbf{Good} & \textbf{Loose} \\ \hline
    French & 39.50 & 34.17 & 26.33 \\
    German & 38.65 & 33.06 & 28.29 \\
    Italian & 39.40 & 34.33 & 26.27 \\
    Portuguese & 39.54 & 34.10 & 26.36 \\
    Spanish & 42.05 & 29.92 & 28.03 \\
    \end{tabular}%
  \label{tab:datalabel-distribution}
\end{table}%

\section{Model architecture}
\label{sec:model-arch-train}
State-of-the-art monolingual information retrieval systems \cite{Huang2013LearningDS, Shen2014LearningSR} use a hybrid architecture of RNN followed by a convolution network to extract semantic and syntactic features of text respectively.
We extended their idea to build a network with two monolingual encoders for source and target each to extract semantic features followed by a CNN for syntactic features.
Refer to figure~\ref{fig:modelarch} for visualization of the network's architecture.
Input to the model are 300 dimensional embeddings from pretrained FastText \cite{DBLP:journals/tacl/BojanowskiGJM17} for each token.
We used two BiLSTMs for each encoder with the outputs of both LSTM concatenated.
They were then sequentially fed to two convolution modules.
CNN output was passed through a fully connected layer before making a three class prediction.
We used ReLU activation with dropout and Batch Normalization.
We used Adam optimizer \cite{Kingma2014AdamAM} in all our experiments with a batch size of 8192. We chose a learning rate of $10^{-3}$ scheduled to drop by a factor of $10$ twice whenever the rate of training loss drop was less than $10^{-3}$ before stopping training.
Table \ref{tab:model-performance} shows the size of the dataset used for training and testing.

\section{Experiments}
\label{sec:experiments}
\begin{table*}[htbp]
  \centering
  \caption{Model accuracy on train and tests sets.}
    \begin{tabular}{r|r|r|r|r|r|r|r}
      & \multicolumn{2}{c|}{\textbf{Train}} & \multicolumn{5}{c}{\textbf{Test}} \\ \cline{2-8}
      & \textbf{\# Samples} & \textbf{Accuracy} & \textbf{\# Samples} & \textbf{Accuracy} & \textbf{Precision} & \textbf{Recall} & \textbf{F-Score} \\ \hline
        French & 4.23M & 93.91 & 0.83M & 91.49 & 91.04 & 90.42 & 90.63 \\
        German & 12.92M & 95.18 & 2.53M & 93.90 & 93.68 & 93.29 & 93.42 \\
        Italian & 3.74M & 94.41 & 0.73M & 92.12 & 91.64 & 91.00 & 91.24 \\
        Portuguese & 15.43M & 94.20 & 3.03M & 92.73 & 92.50 & 91.59 & 91.89 \\
        Spanish & 18.24M & 93.14 & 3.58M & 91.45 & 90.90 & 90.39 & 90.42 \\
    \end{tabular}%
  \label{tab:model-performance}%
\end{table*}%

Table~\ref{tab:model-performance} shows the model comparison across various measures. We observe that the model performs similarly with accuracy of above $91\%$ for all five languages.
The model also has similar performance across sentences of various lengths as shown in figure~\ref{fig:modelperformance-target-length} with longer sentences doing slightly better than shorter ones.
This is possibly because shorter ones when paraphrased are harder to detect than longer ones.

\begin{figure}[h]
    \includegraphics[width=\columnwidth]{resources/mr2-label-performance.png}
    \caption{Model accuracy for each label, plot in color.}
    \label{fig:modelperformance-label}
\end{figure}

\begin{figure}[h]
    \includegraphics[width=\columnwidth]{resources/target-sentence-length-performance-distribution.pdf}
    \caption{Model performance across target sentences of different lengths.}
    \label{fig:modelperformance-target-length}
\end{figure}

\subsection{Miss rate on parallel corpora}
We took a set of high quality subtitles that were translated and validated independently by two sets of distinct human translators.
A list of parallel sentences were extracted from them using their timestamp alignment information.
These translations that went through two rounds of human audit can safely be assumed to contain no Bad translations.
Since data consists only positives constituting of Loose and Good, we use miss rate or false negative rate (FNR) as the performance metric for this experiment.
We ran a similar test on public EuroParl data for reference that also has only positives.
Corresponding FNR numbers are listed in Table~\ref{tab:human-validation-results} along with number of parallel sentences in the test set.

\begin{table}[h]
    \centering
    \caption{Miss rate on parallel corpora.}
    \resizebox{\columnwidth}{!}{%
    \begin{tabular}{r|r|r|r|r}
         & \multicolumn{2}{c|}{\textbf{High-quality subs}} & \multicolumn{2}{c}{\textbf{EuroParl}} \\ \cline{2-5}
         & \textbf{\# sentences} & \textbf{FNR} & \textbf{\# sentences} & \textbf{FNR} \\ \hline
        French & 2.4k & 13.69 & 888k & 2.69 \\
        German & 9.9k & 13.94 & 919k & 2.55 \\
        Italian & 2.4k & 10.72 & 830k & 3.40 \\
        Portuguese & 21.3k & 12.30 & 891k & 4.01 \\
        Spanish & 25.2k & 12.31 & 888k & 2.87 \\
    \end{tabular}%
    }
    \label{tab:human-validation-results}%
\end{table}

The FNR is low for all language pairs and most false negatives we identified were contextual translations that should have been marked by the model as Loose but were mistaken as Bad.
For example --- in English-German, the phrase ``\textit{Jesus}" was rewritten to ``\textit{Meine Güte.}" which literally translates to \textit{my goodness}.
Such cases were under 14\% showing a good performance on Bad vs rest which is more critical than the Loose vs Good boundary.
Further, our subtitles data seem to have a higher sample of such contextually correct translations that are not a literal match.
Such paraphrasing is one area that we could improve upon in future work.

\subsection{Classification vs scoring}
\label{sec:class-ranking-diff}
As briefly discussed in Section~\ref{sec:intro}, QE can also be formulated as a scoring problem.
We chose the classification route using cross entropy loss for our model.
We compare an alternative that employs scoring based formulation
extending the ordinal regression objective from \cite{Liu2018ACD} that is defined as,
\begin{gather}
    \ell = \min(0, \hat{y} - \gamma_{y})^2 + \max(0, \hat{y} - \gamma_{y}')^2 \label{eq:loss-func}
\end{gather}
where $\hat{y}$ is the predicted score, $y$ is the label, $\gamma_{y}$ and $\gamma_{y}'$ are lower bound and upper bound thresholds respectively for~$y$. The values $(\gamma_{y},\gamma_{y}')$ for each label are set to $(0,0.35)$ for class Bad, $(0.35,0.65)$ for Loose and $(0.65,1)$ for Good.

\begin{table}[htbp]
    \centering
    \caption{Accuracy comparison of classification and scoring losses.}
    \begin{tabular}{r|r|r}
         & \textbf{Classification} & \textbf{Scoring} \\ \hline
        French & \textbf{91.49} & 87.46 \\
        German & \textbf{93.90} & 90.81 \\
        Italian & \textbf{92.12} & 88.61 \\
        Portuguese & \textbf{92.73} & 88.46 \\
        Spanish & \textbf{91.45} & 87.25 \\
    \end{tabular}%
    \label{tab:class-ranking-diff}%
\end{table}%

The model assigns Good samples a score higher than those to Loose which should have been higher than those assigned to Bad.
We changed the final fully-connected layer of the model to give only one output followed by a sigmoid to bound the score in the range~$[0, 1]$.
Table \ref{tab:class-ranking-diff} shows the test accuracy of both losses. We can see that the classification loss outperformed the scoring loss by approximately $4\%$ for each language.

\begin{table*}[htbp]
    \centering
    \caption{Test Accuracy for various model architectures.}
    \begin{tabular}{r|c|c|c|c|c|c}

     & \multirow{2}[0]{*}{\textbf{Baseline}} & \multirow{2}[0]{*}{\textbf{LSTM}} & \textbf{LASER} & \multirow{2}[0]{*}{\textbf{CNN}} & \textbf{LASER} & \multirow{2}[0]{*}{\textbf{DeepSubQE}} \\
      & & & \textbf{FC} & & \textbf{CNN} & \\ \hline

    French     &  68.20 &   67.25 &     62.90 &  88.75 &      88.72 &    \textbf{91.49} \\
    German     &  70.46 &   68.63 &     61.66 &  90.88 &      90.29 &    \textbf{93.90} \\
    Italian    &  66.96 &   66.80 &     60.70 &  89.63 &      89.36 &    \textbf{92.12} \\
    Portuguese &  68.40 &   70.89 &     62.19 &  89.93 &      88.24 &    \textbf{92.73} \\
    Spanish    &  70.73 &   68.86 &     61.81 &  88.50 &      87.22 &    \textbf{91.45} \\
    \end{tabular}
    \label{tab:diff-arch-performance}
\end{table*}

    \subsection{Comparison of architectures}
\label{sec:arch-compare}
We found that the existing convolution networks \cite{Liu2018MultilingualST} that try to classify bilingual dataset are not able to learn the linguistic nuances of text and fail in many cases. However, recurrent networks that are good with parsing variable length inputs with temporal dependencies complement them. We compared our hybrid network with an LSTM network, a CNN model and models using LASER sentence embeddings \cite{DBLP:journals/tacl/ArtetxeS19}.
For LSTM network, we concatenated the output of both BiLSTMs and fed to a fully connected layer. For CNN model, we had three convolution modules followed by a fully connected layer. We use the 1024-dimension language agnostic sentence embeddings from LASER.
Table~\ref{tab:diff-arch-performance} compares accuracy of various models on test data.
For baseline, we used the equation \ref{eq:weak-classifier-final-calc} to generate labels for test data.
An only-LSTM network works just about as well as the baseline but only-CNN network brings significant gains.
The convolutional layer is possibly evaluating semantic retention better ~\cite{DBLP:conf/wmt/KimLN17}.
LASER FC is a classifier trained with LASER embeddings fed into a fully connected layer that performs worse than baseline.
LASER CNN is a CNN on top of LASER embeddings and performs about as well as only-CNN.
Proposed hybrid model, however, outperformed all other networks including the CNN by $3\%$.
The LSTM when used in combination with the CNN is consistently improving prediction accuracies across languages.
One notable observation from models trained using LASER embeddings was that LASER models took about 10 epochs on average to meet our stopping criterion while DeepSubQE~model took around 34 epochs on average.

\begin{figure}[h]
    \includegraphics[width=\columnwidth]{resources/translation-embeddings-representation.pdf}
    \caption{Visualization from t-SNE of last layer of DeepSubQE.}
    \label{fig:translation-embedding-plot}
\end{figure}

In figure \ref{fig:translation-embedding-plot}, we present the t-SNE visualization \cite{Maaten2008VisualizingDU} of the last layer of our hybrid network for English-German pair.
The plot shows that there is one cluster for Good translations and one for Bad while there are two for Loose that are spread out spatially. This is because Loose translations can be close to either Good and Bad.

\section{Conclusion}
\label{sec:conclusion}

We studied the problem of translation quality estimation in video subtitles without any reference texts.
We show, empirically, how training data can be synthesized for a three-way classification into Good, Loose and Bad translations.
The model decision can then be integrated into the subtitle quality improvement process with Good being acceptable translations, Loose possibly requiring human post-edits and Bad needing complete rewrite.

The current work only uses subtitle block level translations to make a decision and ignores temporal aspect.
Temporal structure can bring significant information to make a better judgment particularly on Loose translations.
Also, training one model per language pair requires considerable operational load.
A multilingual model can reduce this load while helping resource starved languages.
Exploiting temporal information and learning a common space for multiple languages are future directions we are considering for this work.