\section{Acknowledgements}
We thank researchers at IBM IRL, IIT Kharagpur, Vishal Gupta and Dr. Sudeshna Sarkar for helpful discussions in this project.
\section{Introduction}
\label{sec:intro}
Text Simplification (TS) deals with transforming the original text into simplified variants to increase its readability and understandability. TS is an important task in computational linguistics, and has numerous use-cases in fields of education technology, targeted content creation, language learning, where producing variants of the text with varying degree of simplicity is desired. TS systems are typically designed  to simplify from two different linguistic aspects: (a) \textbf{Lexical} aspect, by replacing complex words in the input with simpler synonyms \cite{devlin1998use,candido2009supporting,yatskar2010sake,biran2011putting,glavavs2015simplifying}, and  (b) \textbf{Syntactic} aspect, by altering the inherent hierarchical structure of the sentences \cite{chandrasekar1997automatic,canning1999syntactic,siddharthan2006syntactic,filippova2008dependency,brouwers2014syntactic}. From the perspective of sentence construction, sentence simplification can be thought to be a form of  \textit{text-transformation} that involves three major types of operations such as (a) \textit{splitting} \cite{siddharthan2006syntactic,petersen2007text,narayan2014hybrid} (b) \textit{deletion/compression} \cite{knight2002summarization,clarke2006models,filippova2008dependency,rush2015neural,filippova2015sentence}, and (c) \textit{paraphrasing}  \cite{specia2010translating,coster2011simple,wubben2012sentence,wang2016text,nisioi2017exploring}.

Most of the current TS systems require large-scale parallel corpora for training (except for systems like \newcite{glavavs2015simplifying} that  performs only lexical-simplification), which is a major impediment in scaling to newer languages, use-cases, domains and output styles for which such large-scale parallel data do not exist. In fact, one of the popular corpus for TS in English language, \textit{i.e.,} the Wikipedia-SimpleWikipedia aligned dataset has been prone to noise (mis-aligned instances) and inadequacy (i.e., instances having non-simplified targets) \cite{xu2015problems,vstajner2015deeper}, leading to noisy supervised models \cite{wubben2012sentence}. While creation of better datasets (such as, \textit{Newsela} by \newcite{xu2015problems}) can always help, we explore the unsupervised learning paradigm which can potentially work with unlabeled datasets that are cheaper and easier to obtain.

At the heart of the TS problem is the need for preservation of language semantics with the goal of improving readability. From a neural-learning perspective, this entails a specially designed \emph{auto-encoder}, which not only is capable of reconstructing the original input but also can additionally introduce variations so that the auto-encoded output is a simplified version of the input. Intuitively, both of these can be learned by looking at the structure and language patterns of a large amount of non-aligned complex and simple sentences (which are much cheaper to obtain compared to aligned parallel data). These motivations form the basis of our work.

Our approach relies only on two unlabeled text corpora - one representing relatively simpler sentences than the other (which we call complex). The crux of the (\emph{\textbf{unsupervised}}) auto-encoding framework is a shared encoder and a pair of attention-based decoders (one for each type of corpus). The encoder attempts to produce semantics-preserving representations which can be acted upon by the respective decoders (simple or complex) to generate the appropriate text output they are designed for. The framework is crucially supported by two kinds of losses: (1) \emph{\textbf{adversarial loss}} - to distinguish between the real or fake attention context vectors for the simple decoder, and (2) \emph{\textbf{diversification loss}} - to distinguish between attention context vectors of the simple decoder and the complex decoder. The first loss ensures that only the aspects of semantics that are necessary for simplification are passed to the simple decoder in the form of the attention context vectors. The second loss, on the other hand, facilitates passing different semantic aspects to the different decoders through their respective context vectors. Also we employ \emph{\textbf{denoising}} in the auto-encoding setup for enabling syntactic transformations.

The framework is trained using unlabeled text collected from Wikipedia (complex) and Simple Wikipedia (simple). It attempts to perform simplification both lexically and syntactically unlike prevalent systems which mostly target them separately. We demonstrate the competitiveness of our unsupervised framework alongside supervised skylines through both automatic evaluation metrics and human evaluation studies. We also outperform another unsupervised baseline \cite{artetxe2018iclr}, first proposed for neural machine translation. Further, we demonstrate that by leveraging a small amount of labeled parallel data, performance can be improved further. Our code and a new dataset containing partitioned unlabeled sets of simple and complex sentences is publicly available\footnote{https://github.com/subramanyamdvss/UnsupNTS}.

\section{Related Work}
\label{sec:related}
Text Simplification has often been discussed from psychological and linguistic standpoints \cite{lallier80eval,mcnamara1996good,linderholm2000effects}. A heuristic-based system was first introduced by \newcite{chandrasekar1997automatic} which induces rules for simplification automatically extracted from annotated corpora. \newcite{canning1999syntactic} proposed a modular system that uses NLP tools such as morphological analyzer, POS tagger plus heuristics to simplify the text both lexically and syntactically. Most of these systems \cite{siddharthan2014survey} are separately targeted towards lexical and syntactic simplification and are limited to splitting and/or truncating sentences.
For paraphrasing based simplification, data-driven approaches were proposed like phrase-based SMT \cite{specia2010translating,vstajner2015deeper} or their variants \cite{coster2011simple,xu2016optimizing}, that combine heuristic and optimization strategies for better TS. Recently proposed TS systems are based on neural \emph{seq2seq} architecture \cite{bahdanau2014neural} which is modified for TS specific operations \cite{wang2016text,nisioi2017exploring}. While these systems produce state of the art results on the popular Wikipedia dataset \cite{coster2011simple}, they may not be generalizable because of the noise and bias in the dataset \cite{xu2015problems} and overfitting.
Towards this, \newcite{stajner2018} showed that improved datasets and minor model changes (such as using reduced vocabulary and enabling copy mechanism) help obtain reasonable performance for both in-domain and cross-domain TS.

In the unsupervised paradigm, \newcite{paetzold2016unsupervised} proposed an unsupervised lexical simplification technique that replaces complex words in the input with simpler synonyms, which are extracted and disambiguated using word embeddings. However, this work, unlike ours only addresses lexical simplification and cannot be trivially extended for other forms of simplification such as splitting and rephrasing. Other works related to style transfer \cite{zhang2018style,shen2017style,xu2018unpaired} typically look into the problem of sentiment transformation and are not motivated by the linguistic aspects of TS, and hence not comparable to our work. As far as we know, ours is a first of its kind end-to-end solution for unsupervised TS. At this point, though supervised solutions perform better than unsupervised ones, we believe unsupervised techniques should be further explored since they hold greater potential with regards to scalability to various tasks.

\section{Model Description}
\label{sec:system}
\begin{figure*}[t]
\centering
\includegraphics[width=10cm,height=10.345cm,keepaspectratio]{images/diagram-acl.pdf}
\caption{System Architecture. Input sentences of any domain is encoded by $\boldsymbol{E}$, and decoded by $\boldsymbol{G_s}$, $\boldsymbol{G_d}$. Discriminator $\boldsymbol{D}$ and classifier $\boldsymbol{C}$ tune the attention vectors for simplification. $\mathcal{L}$ represents loss functions. The figure only reveals one layer in $\boldsymbol{E}$, $\boldsymbol{G_s}$ and $\boldsymbol{G_d}$ for simplicity. However, the model uses two layers of GRUs (Section \ref{sec:system}).}
\label{fig:workflow}
\end{figure*}
Our system is built based on the encode-attend-decode style architecture \cite{bahdanau2014neural} with both algorithmic and architectural changes applied to the standard model. An input sequence of word embeddings $X=\{x_1,x_2,\ldots,x_n\}$ (obtained after a standard look up operation on the embedding matrix), is passed through a shared encoder ($\boldsymbol{E}$), the output representation from which is fed to two decoders ($\boldsymbol{G_s}$, $\boldsymbol{G_d}$) with attention mechanism. $\boldsymbol{G_s}$ is meant to generate a simple sentence from the encoded representation, whereas $\boldsymbol{G_d}$ generates a complex sentence. A discriminator ($\boldsymbol{D}$) and a classifier ($\boldsymbol{C}$) are also employed adversarially to distinguish between the attention context vectors computed with respect to the two decoders. Figure \ref{fig:workflow} is illustrates our system. We describe the components below.

\subsection{Encode-Attend-Decode Model}
\label{sec:enc-dec}
Encoder $\boldsymbol{E}$ uses two layers of bi-directional GRUs \cite{cho2014learning}, and decoders $\boldsymbol{G_s}$, $\boldsymbol{G_d}$ have two layers of GRUs each. $\boldsymbol{E}$ extracts the hidden representations from an input sentence. The decoders output sentences, sequentially one word at a time. Each decoder-step involves using \textit{global attention} to create a context-vector (hidden representations weighted by attention weights) as an input for the next decoder-step. The attention mechanism enables the decoders to focus on different parts of the input sentence. For the input sentence $X$ with $n$ words, the encoder produces $n$ hidden representations, $H=\{h_1,h_2,\ldots,h_n\}$. The context vector extracted from $X$ by a decoder $\boldsymbol{G}$ for time-step $t$ is represented as,
\begin{equation}
A_t(X) = \sum_{i=1}^n a_{it}h_i
\label{eqn:attention}
\end{equation}
where, $a_{it}$ denotes attention weight for the hidden representation at the $i^{th}$ input position with respect to decoder-step $t$.
As there are two decoders, $A_{st}(X)$ and $A_{dt}(X)$ denote the context vectors computed from decoders $\boldsymbol{G_s}$ and $\boldsymbol{G_d}$ respectively for time-steps $t\in\{1\ldots m\}$, $m$ denoting the total number of decoding steps performed\footnote{For a particular $X$, $m$ can differ for the two decoders.}. The matrices $A_s(X)$ and $A_d(X)$ represent the sequence of respective context vectors from all time-steps.

\subsection{Discriminator and Classifier}
A discriminator $\boldsymbol{D}$ is employed to influence the way the decoder $\boldsymbol{G_s}$ will attend to the hidden representations, which has to be different for different types of inputs to the shared encoder $\boldsymbol{E}$ (simple vs complex). The input to $\boldsymbol{D}$ is the context vector sequence matrix $A_{s}$ pertaining to $\boldsymbol{G_s}$, and it produces a binary output, $\{1,0\}$, $1$ indicating the fact that the context vector sequence is close to a typical context vector sequence extracted from simple sentences seen in the dataset. $\boldsymbol{G_s}$ and $\boldsymbol{D}$ are indulged in an adversarial interplay through an adversarial loss function (see Section \ref{sec:advloss}), analogous to GANs \cite{goodfellow2014generative}, where the generator and discriminators, converge to a point where the distribution of the generations eventually resembles the distribution of the genuine samples. In our case, adversarial loss tunes the context vector sequence from a complex sentence by $\boldsymbol{G_s}$ to ultimately resemble the context vector sequence of simple sentences in the corpora. \emph{This ensures that the resultant context vector for $\boldsymbol{G_s}$ captures only the necessary language signals to decode a simple sentence}.

A classifier ($\boldsymbol{C}$) is introduced for diversification to ensure that the way decoder $\boldsymbol{G_s}$ attends to the hidden representations remains different from $\boldsymbol{G_d}$. It helps distinguish between simple and complex context vector sequences with respect to  $\boldsymbol{G_s}$ and $\boldsymbol{G_d}$ respectively. \emph{The classifier  diversifies the context vectors given as input to the different decoders. Intuitively, different linguistic signals are needed to decode a complex sentence \textit{vis-\'a-vis} a simple one}. Refer Section \ref{sec:clfloss} for more details.

Both  $\boldsymbol{D}$ and $\boldsymbol{C}$ use a CNN-based classifier analogous to \newcite{kim2014convolutional}. All layers are shared between $\boldsymbol{D}$ and $\boldsymbol{C}$ except the fully-connected layer preceeding the softmax function.

\subsection{Special Purpose Word-Embeddings}
Pre-trained word embeddings are often seen to have positive impact on  sequence-to-sequence frameworks \cite{cho2014properties,qi2018and}. However, traditional embeddings are not good at capturing relations like synonymy \cite{tissier2017dict2vec}, which are essential for simplification. For this, our word-embeddings are trained using the \textit{Dict2Vec} framework\footnote{https://github.com/tca19/dict2vec}. \textit{Dict2Vec} fine-tunes the embeddings through the help of an external lexicon containing weak and strong synonymy relations. The system is trained on our whole unlabeled datasets and with seed synonymy dictionaries provided by \newcite{tissier2017dict2vec}. Our encoder and decoders share the same word embeddings. Moreover, the embeddings at the input side are kept static but the decoder embeddings are updated as training progresses. Details about hyper-parameters are given in Section \ref{subsec:param}.

\section{Training Procedure}
\label{sec:training}

\def$\boldsymbol{X_s}$ {$\boldsymbol{X_s}$}
\def$\boldsymbol{X_d}$ {$\boldsymbol{X_d}$}
\def$\boldsymbol{\mathcal{S}}$ {$\boldsymbol{\mathcal{S}}$}
\def$\boldsymbol{\mathcal{S}_h}$ {$\boldsymbol{\mathcal{S}_h}$}
\def\D {$\boldsymbol{\mathcal{D}}$}
\def$\boldsymbol{\mathcal{D}_h}$ {$\boldsymbol{\mathcal{D}_h}$}
\def$\in$ {$\in$}
\def$\boldsymbol{E}$ {$\boldsymbol{E}$}
\def$\boldsymbol{G}$ {$\boldsymbol{G}$}
\def$\boldsymbol{D_s}$ {$\boldsymbol{D_s}$}
\def$\boldsymbol{D_d}$ {$\boldsymbol{D_d}$}
\def$\boldsymbol{C}$ {$\boldsymbol{C}$}
\def\boldsymbol{\theta_E}{\boldsymbol{\theta_E}}
\def\boldsymbol{\theta_G}{\boldsymbol{\theta_G}}
\def\boldsymbol{\theta_{G_s}}{\boldsymbol{\theta_{G_s}}}
\def\boldsymbol{\theta_{G_d}}{\boldsymbol{\theta_{G_d}}}
\def\boldsymbol{\theta_{D_s}}{\boldsymbol{\theta_{D_s}}}
\def\boldsymbol{\theta_{C}}{\boldsymbol{\theta_{C}}}
\def\boldsymbol{\theta_{D}}{\boldsymbol{\theta_{D}}}
\def\boldsymbol{\theta_{D_d}}{\boldsymbol{\theta_{D_d}}}
\def$\boldsymbol{E-G_s}${$\boldsymbol{E-G_s}$}
\def\boldsymbol{E-G_s}{\boldsymbol{E-G_s}}
\def\boldsymbol{E-G}{\boldsymbol{E-G}}
\def\boldsymbol{E-G}{\boldsymbol{E-G}}
\def$\boldsymbol{E-G_d}${$\boldsymbol{E-G_d}$}
\def\boldsymbol{E-G_d}{\boldsymbol{E-G_d}}
\def\mathcal{L_rec}{\mathcal{L_rec}}

Let $\boldsymbol{\mathcal{S}}$ and $\boldsymbol{\mathcal{D}}$ be sets of simple and complex sentences respectively from large scale  unlabeled repositories of simple and complex sentences. Let $X_s$ denote a sentence sampled from the set of simple sentences $\boldsymbol{\mathcal{S}}$ and $X_d$ be a sentence sampled from the set of complex sentences $\boldsymbol{\mathcal{D}}$.
Let $\boldsymbol{\theta_E}\ $ denote the parameters of $\boldsymbol{E}$ and $\boldsymbol{\theta_{G_s}}\ , \boldsymbol{\theta_{G_d}}\ $ denote the parameters of $\boldsymbol{G_s}$ and $\boldsymbol{G_d}$ respectively. Also, $\boldsymbol{\theta_{C}}\ $ and $\boldsymbol{\theta_{D}}\ $ are the parameters of the discriminator and the classifier modules.
Training the model involves optimization of the above parameters with respect to the following losses and \textit{denoising}, which are explained below.

\subsection{Reconstruction Loss}
\label{sec:reconst}
Reconstruction Loss is imposed on both $\boldsymbol{E-G_s}$\ and $\boldsymbol{E-G_d}$\ paths. $\boldsymbol{E-G_s}$\ is trained to reconstruct sentences from $\boldsymbol{\mathcal{S}}$\ and $\boldsymbol{E-G_d}$\ is trained to reconstruct sentences from \D.
Let $P_{\boldsymbol{E-G_s}}({X})$ and $P_{\boldsymbol{E-G_d}}(X)$ denote the reconstruction probabilities of an input sentence $X$ estimated by the $\boldsymbol{E-G_s}$\ and $\boldsymbol{E-G_d}$\ models respectively. Reconstruction loss for $\boldsymbol{E-G_s}$\ and $\boldsymbol{E-G_d}$\ , denoted by $\mathcal{L}_{rec}$ is computed as follows.
\begin{align}
\small
 \mathcal{L}_{rec}(\boldsymbol{\theta_E},\boldsymbol{\theta_{G_s}},\boldsymbol{\theta_{G_d}}) = -\mathbb{E}_{X_s\boldsymbol{ \sim  \mathcal{S}}}[\log{P_{\boldsymbol{E-G_s}}(X_s)}] - \nonumber \\
\small
 \mathbb{E}_{X_d\boldsymbol{ \sim  \mathcal{D}}}[\log{P_{\boldsymbol{E-G_d}}(X_d)}]
\label{eqn:reconst}
\end{align}
\subsection{Adversarial Loss}
\label{sec:advloss}
Adversarial Loss is imposed upon the context vectors for $\boldsymbol{G_s}$. The idea is that, context vectors extracted even for a complex input sentence by $\boldsymbol{G_s}$ should resemble the context vectors from a simple input sentence. The discriminator $\boldsymbol{D}$ is trained to distinguish the \textit{fake} (complex) context vectors from the \textit{real} (simple) context vectors. $\boldsymbol{E-G_s}$\ is trained to perplex the discriminator $\boldsymbol{D}$, and eventually, at convergence, learns to produce real-like (simple) context vectors from complex input sentences. In practice, we observe that adversarial loss indeed assists $\boldsymbol{E-G_s}$\ in simplification by encouraging sentence shortening. Let $A_s(.)$ be a sequence of context vectors as defined in Section \ref{sec:enc-dec}. Adversarial losses for $\boldsymbol{E-G_s}$\ , denoted by $\mathcal{L}_{adv,G_s}$ and for discriminator $\boldsymbol{D}$, denoted by $\mathcal{L}_{adv,D}$ are as follows.
\begin{align}
\small
\mathcal{L}_{adv,D}(\boldsymbol{\theta_{D}}) = -\mathbb{E}_{X_s\boldsymbol{ \sim  \mathcal{S}}}[\log{(\boldsymbol{D}(A_s (X_s)))}] - \nonumber\\
\small
\mathbb{E}_{X_d\boldsymbol{ \sim  \mathcal{D}}}[\log{(1- \boldsymbol{D}(A_s (X_d))}] \\
\small
\mathcal{L}_{adv,G_s}(\boldsymbol{\theta_E},\boldsymbol{\theta_{G_s}}) = -\mathbb{E}_{X_d\boldsymbol{ \sim  \mathcal{D}}}[\log{(\boldsymbol{D}(A_s(X_d)))}]
\label{eqn:advloss}
\end{align}
\subsection{Diversification Loss}
\label{sec:clfloss}
Diversification Loss is imposed by the classifier $\boldsymbol{C}$ on context vectors extracted by $\boldsymbol{G_d}$ from complex input sentences in contrast with context vectors extracted by $\boldsymbol{G_s}$ from simple input sentences. This helps $\boldsymbol{E-G_s}$\ to learn to generate simple context vectors distinguishable from complex context vectors. Let $A_s(.)$ and $A_d(.)$ be sequence of context vectors as defined in Section \ref{sec:enc-dec}. Losses for classifier C, denoted by $\mathcal{L}_{div,C}$ and for model $\boldsymbol{E-G_s}$\, denoted by $\mathcal{L}_{div,G_s}$ are computed as follows.
\begin{align}
\small
\mathcal{L}_{div,C}(\boldsymbol{\theta_{C}}) = -\mathbb{E}_{X_s\boldsymbol{ \sim  \mathcal{S}}}[\log{(\boldsymbol{C}(A_s(X_s)))}] - \nonumber \\
\small
\mathbb{E}_{X_d\boldsymbol{ \sim  \mathcal{D}}}[\log{(1-\boldsymbol{C}(A_d(X_d)))}]\\
\small
\mathcal{L}_{div,G_s}(\boldsymbol{\theta_E},\boldsymbol{\theta_{G_s}}) = -\mathbb{E}_{X_d\boldsymbol{ \sim  \mathcal{D}}}[\log{(\boldsymbol{C}(A_s(X_d)))}]
\label{eqn:clfloss}
\end{align}
\subsection{Denoising}
\label{sec:denoising}
Denoising has proven to be helpful to learn syntactic / structural transformation from the source side to the target side \cite{artetxe2018iclr}. Syntactic transformation often requires reordering the input, which the denoising procedure aims to capture. Denoising involves arbitrarily reordering the inputs and reconstructing the original (unperturbed) input from such reordered inputs. In our implementation, the source sentence is reordered by swapping bigrams in the input sentences. The following loss function are used in denoising. Let $P_{\boldsymbol{E-G_s}}(X|noise(X))$ and $P_{\boldsymbol{E-G_d}}(X|noise(X))$ denote the probabilities that a perturbed input $X$ can be reconstructed by $\boldsymbol{E-G_s}$ and $\boldsymbol{E-G_d}$ respectively. Denoising loss for models $\boldsymbol{E-G_s}$\ and $\boldsymbol{E-G_d}$\ , denoted by $\mathcal{L}_{denoi}(\boldsymbol{\theta_E},\boldsymbol{\theta_{G_s}}, \boldsymbol{\theta_{G_d}})$ is computed as follows.
\begin{align}
\footnotesize
 \mathcal{L}_{denoi} = -\mathbb{E}_{X_s\boldsymbol{ \sim  \mathcal{S}}}[\log{P_{\boldsymbol{E-G_s}}(X_s|noise(X_s)})] - \nonumber \\
\small
 \mathbb{E}_{X_d\boldsymbol{ \sim  \mathcal{D}}}[\log{P_{\boldsymbol{E-G_d}}(X_d|noise(X_d))}]
 \label{eqn:denoising}
\end{align}
\begin{algorithm}[t]
\caption{Unsupervised simplification algorithm using denoising, reconstruction, adversarial and diversification losses. }
\label{algo:algo1}
\textbf{Input:} simple dataset $\boldsymbol{\mathcal{{S}}}$, complex dataset $\boldsymbol{\mathcal{{D}}}$.
\begin{algorithmic}
\State \textbf{\textit{Initialization phase:}}
\Repeat
\State Update $\boldsymbol{\theta_E}$, $\boldsymbol{\theta_{G_s}}$, $\boldsymbol{\theta_{G_d}}$ using $\mathcal{L}_{denoi}$
\State Update $\boldsymbol{\theta_E}$, $\boldsymbol{\theta_{G_s}}$, $\boldsymbol{\theta_{G_d}}$ using $\mathcal{L}_{rec}$
\State Update $\boldsymbol{\theta_D}$, $\boldsymbol{\theta_C}$ using $\mathcal{L}_{adv,D}$ $\mathcal{L}_{div,C}$
\Until{specified number of steps are completed}
\State \textbf{\textit{Adversarial phase:}}
\Repeat
\State Update $\boldsymbol{\theta_E}$, $\boldsymbol{\theta_{G_s}}$, $\boldsymbol{\theta_{G_d}}$ using $\mathcal{L}_{denoi}$
\State Update $\boldsymbol{\theta_E},\boldsymbol{\theta_{G_s}},\boldsymbol{\theta_{G_d}}$ using $\mathcal{L}_{adv,G_s}$, $\mathcal{L}_{div,G_s}$, $\mathcal{L}_{rec}$
\State Update $\boldsymbol{\theta_D}$, $\boldsymbol{\theta_C}$ using $\mathcal{L}_{adv,D}$, $\mathcal{L}_{div,C}$
\Until{specified number of steps are completed}
\end{algorithmic}
\end{algorithm}

Figure \ref{fig:workflow} depicts the overall architecture and the losses described above; the training procedure is described in Algorithm \ref{algo:algo1}. The \textit{initialization phase} involves training the $\boldsymbol{E-G_s}$, $\boldsymbol{E-G_d}$~ using the reconstruction and denoising losses only. Next, training of $\boldsymbol{D}$ and $\boldsymbol{C}$\ happens using the respective adversarial or diversification losses. These losses are not used to update the decoders at this point. This gives the discriminator, classifier and decoders time to learn independent of each other. In the \textit{adversarial phase}, adversarial and diversification losses are introduced alongside denoising and reconstruction losses for fine-tuning the encoder and decoders. Algorithm \ref{algo:algo1} is intended to produce the following results: i) $\boldsymbol{E-G_s}$ should simplify its input (irrespective of whether it is simple or complex), and
ii) $\boldsymbol{E-G_d}$ should act as an auto-encoder in complex sentence domain. The discriminator and classifier enables preserving the appropriate aspects of semantics necessary for each of these pathways through proper modulation of the attention context vectors.

A key requirement for a model like ours is that the dataset used has to be partitioned into two sets, containing relatively simple and complex sentences. The rationale behind having two decoders is that while $\boldsymbol{G_s}$ will try to introduce simplified constructs (may be at the expense of loss of semantics), $\boldsymbol{G_d}$ will help preserve the semantics. The idea behind using the discriminator and classifier is to retain signals related to language simplicity from which $\boldsymbol{G_s}$ will construct simplified sentences. Finally, denoising will help tackle nuances related to syntactic transfer from complex to simple direction. We remind the readers that, TS, unlike machine translation, needs complex syntactic operations such as sentence splitting, rephrasing and paraphrasing, which can not be tackled by the losses and denoising alone. Employing additional explicit mechanisms to handle these in the pipeline is out of the scope of this paper since we seek a \textit{prima-facie} judgement of our architecture based on how much simplification knowledge can be gained just from the data.
\subsection{Training with Minimal Supervision}
\label{sec:semisup}
Our system, by design, is highly data-driven, and like any other sequence-to-sequence learning based system, can also leverage labeled data. We propose a semi-supervised variant of our system that could gain additional knowledge of simplification through the help of a small amount of labeled data (in the order of a few thousands). The system undergoes training following steps similar to Algorithm \ref{algo:algo1}, except that it adds another step of optimizing the \textit{cross entropy loss} for both the $\boldsymbol{E-G_s}$\ and $\boldsymbol{E-G_d}$\ pathways by using the reference texts available in the labeled dataset. This step is carried out in the adversarial phase along with other steps (See Algorithm \ref{algo:algo2}).

The cross-entropy loss is imposed on both $\boldsymbol{E-G_s}$\ and $\boldsymbol{E-G_d}$\ paths using parallel dataset (details mentioned in Section \ref{subsec:dataset}) denoted by $\Delta = (\boldsymbol{\mathcal{S}_p}, \boldsymbol{\mathcal{D}_p})$.
For a given parallel simplification sentence pair $(X_s,X_d)$,
let $P_{\boldsymbol{E-G_s}}({X_s}|{X_d})$ and $P_{\boldsymbol{E-G_d}}(X_d|X_s)$ denote the probabilities that $X_s$ is produced from $X_d$  by the $\boldsymbol{E-G_s}$\ and the reverse is produced by the $\boldsymbol{E-G_d}$\ respectively. Cross-Entropy loss for $\boldsymbol{E-G_s}$\ and $\boldsymbol{E-G_d}$\ denoted by $\mathcal{L}_{cross}(\boldsymbol{\theta_E},\boldsymbol{\theta_{G_s}},\boldsymbol{\theta_{G_d}})$ is computed as follows:
\begin{align}
\small
 \mathcal{L}_{cross} = -\mathbb{E}_{(X_s,X_d)\boldsymbol{ \sim  \Delta}}[\log{P_{\boldsymbol{E-G_s}}(X_s|X_d)}] - \nonumber \\
\small
 \mathbb{E}_{(X_s,X_d)\boldsymbol{ \sim  \Delta}}[\log{P_{\boldsymbol{E-G_d}}(X_d|X_s)}]
\label{eqn:reconst}
\end{align}

\begin{algorithm}[h]
\caption{Semi-supervised simplification algorithm using denoising, reconstruction, adversarial and diversification losses followed by cross-entropy loss using parallel data. }
\label{algo:algo2}
\textbf{Input:} simple dataset $\boldsymbol{\mathcal{{S}}}$, complex dataset $\boldsymbol{\mathcal{{D}}}$, parallel dataset $\Delta = (\boldsymbol{\mathcal{S}_p}, \boldsymbol{\mathcal{D}_p})$
\begin{algorithmic}
\State \textbf{\textit{Initialization phase:}}
\Repeat
\State Update $\boldsymbol{\theta_E}$, $\boldsymbol{\theta_{G_s}}$, $\boldsymbol{\theta_{G_d}}$ using $\mathcal{L}_{denoi}$
\State Update $\boldsymbol{\theta_E}$, $\boldsymbol{\theta_{G_s}}$, $\boldsymbol{\theta_{G_d}}$ using $\mathcal{L}_{rec}$
\State Update $\boldsymbol{\theta_D}$, $\boldsymbol{\theta_C}$ using $\mathcal{L}_{adv,D}$ $\mathcal{L}_{div,C}$
\Until{specified number of steps are completed}
\State \textbf{\textit{Adversarial phase:}}
\Repeat
\State Update $\boldsymbol{\theta_E}$, $\boldsymbol{\theta_{G_s}}$, $\boldsymbol{\theta_{G_d}}$ using $\mathcal{L}_{denoi}$
\State Update $\boldsymbol{\theta_E},\boldsymbol{\theta_{G_s}},\boldsymbol{\theta_{G_d}}$ using $\mathcal{L}_{adv,G_s}$, $\mathcal{L}_{div,G_s}$, $\mathcal{L}_{rec}$
\State Update $\boldsymbol{\theta_D}$, $\boldsymbol{\theta_C}$ using $\mathcal{L}_{adv,D}$, $\mathcal{L}_{div,C}$
\State Update $\boldsymbol{\theta_E}$, $\boldsymbol{\theta_{G_s}}$ using $\mathcal{L}_{cross}$
\State Update $\boldsymbol{\theta_E}$, $\boldsymbol{\theta_{G_d}}$ using $\mathcal{L}_{cross}$
\Until{specified number of steps are completed}
\end{algorithmic}
\end{algorithm}

\section{Experiment Setup}
\label{sec:exp}
In this section we describe the dataset, architectural choices, and model hyperparameters. The implementation of the experimental setup is publicly available\footnote{https://github.com/subramanyamdvss/UnsupNTS}.
\begin{table}[t]
\centering
\footnotesize
\begin{tabular}{lllll}
    \toprule
    \textbf{Category} & \textbf{\#Sents} & \textbf{Avg.} & \textbf{Avg.} & \textbf{FE-} \\
     & & \textbf{Words} & \textbf{FE} & \textbf{Range} \\
    \midrule
    Simple  & 720k & 18.23 & 76.67 & 74.9-79.16\\
    Complex & 720k & 35.03 & 7.26 & 5.66-9.93\\
    \bottomrule
\end{tabular}
\caption{Statistics showing number of sentences, average words per sentence, and average FE score, FE score limits for complex and simple datasets used for training.}
\label{tab:datastat}
\end{table}
\subsection{Dataset}
\label{subsec:dataset}
For training our system, we created an unlabeled dataset of simple and complex sentences by partitioning the standard \textit{en-wikipedia} dump. Since partitioning requires a metric for measuring text simpleness we categorize sentences based on their readability scores. For this we use the Flesch Readability Ease (henceforth abbreviated as FE) \cite{flesch1948new}. Sentences with lower FE values (up to 10) are categorized as complex and sentences with FE values greater than 70 are categorized as simple\footnote{FE has its shortcomings to fully judge simpleness, but we nevertheless employ it in the absence of stronger metrics}. The FE bounds are decided through trial and error through manual inspection of the categorized sentences. Table \ref{tab:datastat} shows dataset statistics. Even though the dataset was created with some level of human mediation, the manual effort is insignificant compared to that needed to create a parallel  corpus.

To train the system with minimal supervision (Section \ref{sec:semisup}), we extract $10,000$ pairs of sentences from various datasets such as Wikipedia-SimpleWikipedia dataset introduced in \newcite{hwang2015aligning} and the Split-Rephrase dataset by \newcite{narayan2017split}\footnote{https://github.com/shashiongithub/Split-and-Rephrase}. The Wikipedia-SimpleWikipedia was filtered following \newcite{nisioi2017exploring} and $4000$ examples were randomly picked from the filtered set. From the Split-Rephrase dataset, examples containing one compound/complex sentence at the source side and two simple sentences at the target side were selected and $6000$ examples were randomly picked from the selected set. The Split-Rephrase dataset is used to promote sentence splitting in the proposed system.

To select and evaluate our models, we use the test and development sets\footnote{We acknowledge that other recent datasets such as Newsela could have been used for development and evaluation. We could not get access to the dataset unfortunately.} released by \cite{xu2016optimizing}. The test set ($359$ sentences) and development set ($2000$ sentences) have 8 simplified reference sentences for each source sentence.

\subsection{Hyperparameter Settings}
\label{subsec:param}
For all the variants, we use a hidden state of size 600 and word-embedding size of 300. Classifier $C$ and discriminator $D$ use convolutional layers with filters sizes from $1$ to $5$. 128 filters of each size are used in the CNN-layers. Other training related hyper parameters include learning rate of 0.00012 for $\boldsymbol{\theta_E,\theta_{G_s},\theta_{G_d}}$ , 0.0005 for $\boldsymbol{\theta_D,\theta_C}$ and batch size of 36. For learning the word-embedding using \textit{Dict2Vec} training, the window size is set to 5. Our experiments used at most 13 GB of GPU memory. The Initialization phase and Adversarial phase took 6000 and 8000 steps in batches respectively for both \textsc{UNTS} and \textsc{UNTS}+10K systems.

\subsection{Evaluation Metrics}
For automatic evaluation of our system on the test data, we used four metrics, (a) SARI (b)  BLEU (c) FE Difference (d) Word Difference, which are briefly explained below.

SARI \cite{xu2016optimizing} is an automatic evaluation metric designed to measure the simpleness of the generated sentences. SARI requires access to source, predictions and references for evaluation. Computing SARI involves penalizing the n-gram additions to source which are inconsistent with the references. Similarly, deletions and keep operations are penalized. The overall score is a balanced sum of all the penalties. BLEU \cite{papineni2002bleu}, a popular metric to evaluate generations and translations is used to measure the correctness of the generations by measuring overlaps between the generated sentences and (multiple) references.

We also compute the average FE score difference between predictions and source in our evaluations. FE-difference measures whether the changes made by the model increase the readability ease of the generated sentence. \textit{Word Difference} is the average difference between number of words in the source sentence and generation. It is a simple and approximate metric proposed to detect if sentence shortening is occurring or not. Generations with lesser number of changes can still have high SARI and BLEU. Models with such generations can be ruled out by imposing a threshold on the word-diff metric.

Models with high word-diff, SARI and BLEU are picked during model-selection (with validation data). Model selection also involved manually examining the quality and relevance of generations.

We carry out a qualitative analysis of our system through human evaluation. For this the first 50 test samples were selected from the test data. Output of the seven systems reported in Table \ref{tab:results} along with the sources are presented to two native English speakers who would provide two ratings for each output: (a) \textit{Simpleness}, a binary score [0-1] indicating whether the output is a simplified version of the input or not, (b) \textit{Grammaticality} of the output in the range of [1-5], in the increasing order of fluency (c) \textit{Relatedness} score in the range of [1-5] showing if the overall semantics of the input is preserved in the output or not.

\subsection{Model Variants}
Using our design, we propose two different variants for evaluation: (i) Unsupervised Neural TS (\textsc{UNTS}) with SARI as the criteria for model selection, (ii) \textsc{UNTS} with minimal supervision using $10000$ labelled examples (UNTS+10K). Models selected using other selection criteria such as BLEU resulted in similar and/or reduced performance (details skipped for brevity).

We carried out the following basic post-processing steps on the generated outputs. The OOV(out of vocabulary) words in the generations are replaced by the source words with high attention weights. Words repeated consecutively  in the generated sentences are merged.

\subsection{Systems for Comparison}
\label{exp:exp}
In the absence of any other direct baseline for end-to-end TS, we consider the following unsupervised baselines. We consider the unsupervised NMT framework proposed by \cite{artetxe2018iclr} as a baseline. It uses techniques such as \textit{backtranslation} and \textit{denoising} techniques to synthesize more training examples. To use this framework, we treated the set of simple and complex sentences as two different languages. Same model configuration as reported by  \newcite{artetxe2018iclr} is used. We use the term \textsc{UNMT} for this system.

Similar to the \textsc{UNMT} system, we also consider unsupervised statistical machine translation (termed as \textsc{USMT}) proposed by \newcite{artetxe2018unsupervised}, with default parameter setting. Another system, based on the cross alignment technique proposed by \newcite{shen2017style} is also used for comparison. The system is originally proposed for the task of sentiment translation. We term this system as \textsc{ST}.

We also compare our approach with existing supervised and unsupervised lexical simplifications like \textsc{LightLS} \cite{glavavs2015simplifying}, Neural Text Simplification or \textsc{NTS} \cite{nisioi2017exploring}, Syntax based Machine Translation or \textsc{SbMT} \cite{xu2016optimizing}, and Phrase-based SMT simplification or \textsc{PbSMT} \cite{wubben2012sentence}. All the systems are trained using the Wikipedia-SimpleWikipedia dataset \cite{hwang2015aligning}. The test set is same for all of these and our models.

\section{Results}
\label{sec:result}
\begin{table}
\footnotesize
\begin{tabular}{lllll}
\toprule
\textbf{System} & \textbf{FE-diff} & \textbf{SARI} & \textbf{BLEU} & \textbf{Word-diff} \\
\midrule
\textsc{UNTS}+10K & 10.45 & 35.29 & 76.13 & 2.38 \\
\textsc{UNTS} & 11.15 & 33.8 & 74.24 & \textbf{3.55} \\
\midrule
\textsc{UNMT} & 6.60 & 33.72 & 70.84 & 0.74 \\
\textsc{USMT} & 13.84 & 32.11 & 87.36 & -0.01\\
\textsc{ST} & 54.38 & 14.97 & 0.73 & 5.61\\
\midrule
\textsc{NTS} & 5.37 & 36.1 & 79.38 & 2.73 \\
\textsc{SbMT} & \textbf{17.68} & \textbf{38.59} & 73.62 & -0.84 \\
\textsc{PbSMT} & 9.14 & 34.07 & 67.79 & 2.26 \\
\midrule
\textsc{LightLS} & 3.01 & 34.96 & \textbf{83.54} & -0.02 \\ \bottomrule
\end{tabular}
\caption{Comparison of evaluation metrics for proposed systems (\textsc{UNTS}), unsupervised baseline (\textsc{UNMT},\textsc{USMT}, and \textsc{ST}) and existing supervised and the unsupervised lexical simplification system \textsc{LightLS}.}
\label{tab:results}
\end{table}
\begin{table}[t]
\footnotesize
\begin{tabular}{llll}
\toprule
\textbf{System} & \textbf{Simpleness} & \textbf{Fluency} & \textbf{Relatedness}\\
\midrule
\textsc{UNTS}+10K & \textbf{57}\% & 4.13 & 3.93 \\
\textsc{UNTS} & 47\% & 3.86 & 3.73 \\
\midrule
\textsc{UNMT} & 40\% & 3.8 & \textbf{4.06} \\
\midrule
\textsc{NTS} & 49\% & 4.13 & 3.26 \\
\textsc{SbMT} & 53\% & \textbf{4.26} & \textbf{4.06} \\
\textsc{PbSMT} & 53\% & 3.8 & 3.93 \\
\midrule
\textsc{LightLS} & 6\% & 4.2 & 3.33 \\ \bottomrule
\end{tabular}
\caption{Average human evaluation scores for simpleness and grammatical correctness (fluency) and semantic relatedness between the output and input.}
\label{tab:Human}
\end{table}
\input{example_table.tex}
Table \ref{tab:results} shows evaluation results of our proposed approaches along with existing supervised and unsupervised alternatives. We observe that unsupervised baselines such as \textsc{UNMT} and \textsc{USMT} often, after attaining convergence, recreates sentences similar to the inputs. This explains why they achieve higher BLEU and reduced word-difference scores. The \textsc{ST} system did not converge for our dataset after significant number of epochs which affected the performance metrics. The system often produces short sentences which are simple but do not retain important phrases.

Other supervised systems such as \textsc{SbMT} and \textsc{NTS} achieve better content reduction as shown through SARI, BLEU and FE-diff scores; this is expected. However, it is still a good sign that the scores for the unsupervised system \textsc{UNTS} are not far from the supervised skylines. The higher word-diff scores for the unsupervised system also indicate that it is able to perform content reduction (a form of syntactic simplification), which is crucial to TS. This is unlike the existing unsupervised \textsc{LightLS} system which often replaces nouns with related non-synonymous nouns; sometimes increasing the complexity and affecting the meaning. Finally, it is worth noting that aiding the system with a very small amount of labeled data can also benefit our unsupervised pipeline, as suggested by the scores for the  \textsc{UNTS}+10K system.

In Table \ref{tab:Human}, the first column represents what percentage of output form is a simplified version of the input. The second and third columns present the average fluency (grammaticality) scores given by human evaluators and semantic relatedness with input scored through automatic means. Almost all systems are able to produce sentences that are somewhat grammatically correct and retain phrases from input. Supervised systems like \textsc{PbSMT}, as expected, simplify the sentences to the maximum extent. However, our unsupervised variants have scores competitive to the supervised skylines, which is a positive sign.

Table \ref{tab:gens} shows an anecdotal example, containing outputs from the seven systems. As can be seen, the quality of output from our unsupervised variants, is far from that of the reference output. However, the attempts towards performing lexical simplification (by replacing the word ``Neverthless'' with ``However'') and simplification of multi-word phrases (``Tagore emulated numerous styles''
getting translated to ``Tagore replaced many styles'') are quite visible and encouraging. Table \ref{tab:typesimplf} presents a few examples demonstrating the capabilities of our system in performing simplifications at lexical and syntactic level. We do observe that such operations are carried out only for a few instances in our test data. Also, our analysis in Appendix \ref{appendix:datavar}  indicate that the system can improve over time with addition of more data. Results for ablations on adversarial and diversification loss are also included in Appendix \ref{appendix:ablation}.
\section{Discussion}
\label{sec:discussion}
In Figure \ref{fig:analysis} we show that during training UnsupNTS-classf-disc baseline touches the 2-4 content reduction band several times during the training than other baselines. The generated simplifications during these times have different simplification schemes. This is because the model is oscillating between low-content reduction to high-content reduction states during training thus making it explore various simplifying alignments of words as well as different ways to conduct content-reduction. Table 2 shows a few examples that make our case. This oscillation cannot occur for long in supervised baselines due to limited data, but the unsupervised dataset we designed has enough examples for the model to explore various ways to simplify a sentence. This observation is consistent with the fact that a sentence can have multiple simplifications, thus inspiring a possibility of controlled simplification framework in which rate of simplification can be controlled by a control variate passed as input.

The algorithm \ref{algo:algo1} designed is such that sentences from simple domain are auto-encoded by simplifying pipeline and sentences from difficult domain are constrained to be simplified. Table \ref{tab:gens} shows predictions for  4 source sentences for different models of UnsupNTS-classf-adv baseline. The first 2 source sentences change drastically reflecting the effect of classifier and adversarial losses, whereas the last two source sentences don't change much when passed through simplifying pipeline because they are nearer to simple domain, .

\section{Details of semi-supervised version for TS (UNTS+10K)}
\input{9b_appendix.tex}
\section{Ablation Studies}
\label{appendix:ablation}
The following table shows results of the proposed system with ablations on adversarial loss (\textsc{UNTS-adv}) and diversification loss (\textsc{UNTS-div}).
\begin{table}[H]
\centering
\scriptsize
\begin{tabular}{lllll}
\toprule
\textbf{System} & \textbf{FE-diff} & \textbf{SARI} & \textbf{BLEU} & \textbf{Word-diff} \\
\midrule
\textsc{UNTS}+10K & 10.45 & 35.29 & 76.13 & 2.38 \\
\textsc{UNTS-div}+10K & 11.32 & 35.24 & 75.59 & {2.61} \\
\textsc{UNTS-adv}+10K & 10.32 & 35.08 & 76.19 & {2.64} \\
\midrule
\textsc{UNTS} & 11.15 & 33.8 & 74.24 & {3.55} \\
\textsc{UNTS-div} & 14.15 & 34.38 & 68.65 & {3.46}\\
\textsc{UNTS-adv} & 12.13 & 34.74 & 73.21 & {2.72}\\\hline
\end{tabular}
\caption{\textsc{UNTS-adv} does not use the adversarial loss, \textsc{UNTS-div} does not use the diversification loss.}
\label{tab:ablation-1}
\end{table}
\section{Effects of Variation in Labeled Data Size}
\label{appendix:datavar}
The following table shows the effect of labeled data size on the performance of the system. We supplied the system with 2K, 5K, and 10K pairs of complex and simple sentences. From the trained models, models with similar word-diff are chosen for fair comparison. Our observation is that, with increasing data, BLEU as well as SARI increases.
\begin{table}[H]
\centering
\scriptsize
\begin{tabular}{lllll}
\toprule
\textbf{System} & \textbf{FE-diff} & \textbf{SARI} & \textbf{BLEU} & \textbf{Word-diff} \\
\midrule
\textsc{UNTS}+10K & 11.65 & 35.14 & 75.71 & {3.05} \\
\textsc{UNTS}+5K & 11.69 & 34.39 & 70.96 & {3.01} \\
\textsc{UNTS}+2K & 11.64 & 34.17 & 72.63 & {3.26} \\
\textsc{UNTS} & 11.15 & 33.8 & 74.24 & {3.55}\\ \hline
\end{tabular}
\caption{Effect of variation in labeled data considered as additional help during training the unsupervised systems.}
\label{tab:ablation-2}
\end{table}

\section{Conclusion}
\label{sec:concl}
In this paper, we made a novel attempt towards unsupervised text simplification. We gathered unlabeled corpora containing simple and complex sentences and used them to train our system that is based on a shared encoder and two decoders. A novel training scheme is proposed which allows the model to perform content reduction and lexical simplification simultaneously through our proposed losses and denoising. Experiments were conducted for multiple variants of our system as well as known unsupervised baselines and supervised systems. Qualitative and quantitative analysis of the outputs for a publicly available test data demonstrate that our models, though unsupervised, can perform better than or competitive to these baselines. In future, we would like to improve the system further by incorporating better architectural designs and training schemes to tackle complex simplification operations.
\section{Central Idea}
\label{sec:central}
We propose a neural sequence to sequence architecture equipped with attention mechanism which is guided by an adversarial loss, as a result the decoder will attend to only relevant parts of the sentence and ignore the rest while decoding. We also use \textit{denoising} and on-fly \textit{backtranslation} to aid the model by taking advantage of dual nature of the simplification problem. Our training procedure is targeted such that lexical simplification, syntactic simplification and especially content reduction can be performed by the model during inference. We also solve the auto encoding problem faced by NTS models by introducing a mild classifier loss which constrains the model not to attend the source like an autoencoder does. This way our training process becomes sustainable for longer time without reaching an auto encoding state.

TODO: benefits introduced by our training procedure.