---
abstract: 'Pre-trained language models have recently been shown to be able to perform translation without finetuning via prompting. Inspired by these findings, we study improving the performance of pre-trained language models on translation tasks, where training neural machine translation models is the current *de facto* approach. We present ulti-tage rompting, a simple and lightweight approach for better adapting pre-trained language models to translation tasks. To make pre-trained language models better translators, we divide the translation process via pre-trained language models into three separate stages: the encoding stage, the re-encoding stage, and the decoding stage. During each stage, we independently apply different continuous prompts for allowing pre-trained language models better adapting to translation tasks. We conduct extensive experiments on low-, medium-, and high-resource translation tasks. Experiments show that our method can significantly improve the translation performance of pre-trained language models.'
author:
- 'Zhixing Tan$^{1,3,4}$, Xiangwen Zhang$^{6}$, Shuo Wang$^{1,3,4}$,'
- |
    Yang Liu$^{1,2,3,4,5}$\
    $^1$Department of Computer Science and Technology, Tsinghua University, Beijing, China\
    $^2$Institute for AI Industry Research, Tsinghua University, Beijing, China\
    $^3$Institute for Artificial Intelligence, Tsinghua University, Beijing, China\
    $^4$Beijing National Research Center for Information Science and Technology\
    $^5$Beijing Academy of Artificial Intelligence\
    $^6$Kuaishou Tech, Co.
bibliography:
- 'anthology.bib'
- 'custom.bib'
title: |
    MSP: Multi-Stage Prompting for Making\
    Pre-trained Language Models Better Translators
---

=1

Introduction
============

\[th\]

\[t\]<span>0.35</span> ![image](figs/pt.pdf){width="\textwidth"}

\[t\]<span>0.455</span> ![image](figs/msp.pdf){width="\textwidth"}

Recent years have witnessed the rapid development of pre-trained language models @devlin2019bert [@brown2020gpt3], with GPT-3 @brown2020gpt3 as the most representative model. By using prompts and a few examples, GPT-3 can perform various NLP tasks without using finetuning @brown2020gpt3, including translation, question-answering, and cloze tasks. This opens the possibility of using a single pre-trained language model to perform all NLP tasks @liu2021pre.

Neural machine translation (NMT) @sutskever2014sequence [@bahdanau2015nmt; @vaswani2017attention] is the current *de facto* paradigm for machine translation. With the breakthrough of pre-trained language models @devlin2019bert [@brown2020gpt3], efforts have been devoted to utilizing pre-trained language models for translation tasks @yang2020towards [@weng2020acquiring; @zhu2020incorporating; @guo2020incorporating; @stickland2021recipes; @sun2021multilingual]. Previous studies can be roughly divided into three categories: (1) finetuning pre-trained language models @yang2020towards [@weng2020acquiring]. (2) integrating pre-trained language models into neural machine translation models @zhu2020incorporating. (3) adapting pre-trained language models to translation tasks with adapters @guo2020incorporating [@stickland2021recipes; @sun2021multilingual]. Despite advances, these studies either treat pre-trained language models as a component of an NMT model or made non-subtle changes to pre-trained language models. Recent studies @brown2020gpt3 [@zhang2021cpm; @wei2021finetuned] have shed some light on using only pre-trained language models as translators. Via *prompting* @brown2020gpt3 [@li2021prefix; @lester2021power], pre-trained language models can perform translation tasks without modifying their network structures or parameters @brown2020gpt3 [@zhang2021cpm; @wei2021finetuned], which provides an efficient and elegant alternative approach for translation tasks. Compared with training separate neural models for translation tasks, we indicate two benefits of directly using pre-trained language models as translators:

1.  *Preserving the ability to perform multiple tasks simultaneously.* Using pre-trained language models as translators can preserve the ability to perform multiple tasks in a single batch by simply using different prompts @li2021prefix.

2.  *Effectiveness in exploiting large-scale raw data.* Pre-trained models have proved to be effective in utilizing abundant unlabeled data @devlin2019bert [@brown2020gpt3].

However, a naive prompting may not be sufficient to fully exploit the potential of pre-trained language models on translation tasks. Therefore we believe it is worthwhile to further investigate how to use pre-trained language models as translators.

In this paper, we present Multi-Stage Prompting (MSP), a simple and efficient approach for adapting GPT-style pre-trained language models to translation tasks. Inspired by neural machine translation models @sutskever2014sequence [@bahdanau2015nmt; @vaswani2017attention] which use separate networks for encoding and decoding, we explicitly divide the translation process via pre-trained language models into three different stages: the *encoding*, the *re-encoding*, and the *decoding* stages. By using different prompts during each stage, the pre-trained language models first learn to encode the source sentence in the encoding stage. Then the pre-trained language models encode more expressive source representations by re-encoding the source sentence using previously encoded activations. Finally, the pre-trained language models perform translations with the re-encoded activations during the decoding stage. Following prefix-tuning @li2021prefix and prompt tuning @lester2021power, we use trainable continuous prompts in different stages, which are learned through back-propagation. With MSP, we expect pre-trained language models can play different roles during different stages, and thus making the pre-trained models better translators. Figure \[fig:overview\] gives a comparison between previous approach and our proposed method. We conduct experiments using a multilingual GPT (mGPT) model on low-, medium-, and high-resource translation tasks. Experiments verify that our MSP can significantly improve the translation performance of pre-trained language models. Our method improves the translation performance of pre-trained language models via prefix-tuning by at least 1.2 BLEU points. Our method also outperforms a strong multi-lingual NMT model using the Transformer architecture by 1.8 BLEU points, showing the potential of using pre-trained language models as translators.

Background
==========

Prompting is a promising way of using pre-trained language models (PLMs) for downstream tasks @brown2020gpt3 [@li2021prefix; @gao2020making]. For example, we can use a template "English: $\mathbf{x}$ German: $\mathbf{y}$" and fill in a source sentence $\mathbf{x}$ to use a PLM to perform an English$\rightarrow$German translation task. Prompts can be either discrete sequences @brown2020gpt3 [@gao2020making] or continuous vectors @li2021prefix [@lester2021power], constructed by manually-designed @brown2020gpt3 or automatic search @gao2020making [@li2021prefix; @lester2021power].

Let $\mathbf{z}=[z_1, \ldots, z_n]$ be a sequence of tokens, we use $P(\mathbf{z})$ to denote the probability of the sequence $\mathbf{z}$. In this paper, we shall assume that $P(\mathbf{z})$ is modeled using an $N$-layered autoregressive Transformer network @vaswani2017attention $f_{\mathrm{LM}}(\bm{z}, \boldsymbol{\mathsf{H}};\bm{\theta})$, where $\bm{z}$ is a word embedding, $\boldsymbol{\mathsf{H}}$ is a sequence of past activations, and $\bm{\theta}$ denotes the parameters of the Transformer network. We use $d$ to denote the hidden size of the Transformer network and use $\bm{h}_t \in \mathbb{R}^{2Nd}$ to denote an activation at time step $t$, which is a concatenation of a set of key-value pairs $\{\langle \bm{k}^{(i)}, \bm{v}^{(i)} \rangle | i = 1\ldots N \}$ in the Transformer network. Given an input $\bm{z}_t \in \mathbb{R}^{d}$ and a sequence of past activations $\boldsymbol{\mathsf{H}}_{t-1} = [\bm{h}_1, ..., \bm{h}_{t-1}]$, the conditional probability $P(z_{t}|\mathbf{z}_{<t})$ is modeled as follows: $$\begin{aligned}
P(z_{t}|\mathbf{z}_{<t}) = \frac{\exp{(\bm{e}_{z_t}^{\mathsf{T}} \cdot \bm{g}_t})}{\sum_{i=1}^{V} \exp{(\bm{e}_{z_i}^{\mathsf{T}} \cdot \bm{g}_t})},\end{aligned}$$ where $V$ is the vocabulary size, $\bm{e}_{z_i}$ is the word embedding of $z_i$, and "$\cdot$" denotes matrix production. $\bm{g}_t$ is the output of the Transformer network: $$\begin{aligned}
\bm{g}_t, \bm{h}_t = f_{\mathrm{LM}}(\bm{e}_{z_t}, [\bm{h}_1, ..., \bm{h}_{t-1}]).\label{eq:plm}\end{aligned}$$

Instead of using discrete prompts, we use continuous prompts and define a prompt $\boldsymbol{\mathsf{P}}$ as a set of $L$ vectors $\{ \bm{p}_1,\ldots,\bm{p}_L \}$, where $\bm{p}_i$ is a trainable continuous vector that shares the same dimension as $\bm{h}_i$. @li2021prefix propose to prepend the prompt $\boldsymbol{\mathsf{P}}$ to the past activations $\boldsymbol{\mathsf{H}}$. Formally, the computation involved in Eq. (\[eq:plm\]) becomes $$\begin{aligned}
\bm{g}_t, \bm{h}_t = f_{\mathrm{LM}}(\bm{e}_{z_t}, [\bm{p}_1, \ldots, \bm{p}_L, \ldots, \bm{h}_{t-1}]).\end{aligned}$$ To make the notation simpler, we use the following equation to denote repeatedly application of $f_{\mathrm{LM}}$ over a sequence $\mathbf{z}_{i:j} = [z_i,\ldots,z_j]$ given past activations $\boldsymbol{\mathsf{H}}$: $$\begin{aligned}
\boldsymbol{\mathsf{G}}_{i:j}, \boldsymbol{\mathsf{H}}_{i:j} = f_{\mathrm{LM}}(\boldsymbol{\mathsf{Z}}_{i:j}, \boldsymbol{\mathsf{H}}),\end{aligned}$$ where $\boldsymbol{\mathsf{Z}}_{i:j}=[\bm{e}_{z_i},\ldots,\bm{e}_{z_j}]$. $\boldsymbol{\mathsf{G}}_{i:j}$ and $\boldsymbol{\mathsf{H}}_{i:j}$ have similar definitions.

By prepending the prompt $\boldsymbol{\mathsf{P}}$ and optimizing $\bm{p}_i$ using task-specific training data and gradient descent, the pre-trained LM can achieve strong performance on downstream tasks that comparable to finetuning while keeping $\bm{\theta}$ frozen @li2021prefix.

Proposed Method
===============

\[th\]

\[b\]<span>0.66</span> ![image](figs/enc.pdf){width="\textwidth"}

\[b\]<span>0.66</span> ![image](figs/reenc.pdf){width="\textwidth"}

\[b\]<span>0.66</span> ![image](figs/dec.pdf){width="\textwidth"}

We propose multi-stage prompting, a simple and lightweight method for adapting pre-trained LMs to translation tasks. We first describe MSP in section \[sec:msp\]. Then we describe the reparameterization of continuous prompts in section \[sec:reparam\]. Finally, we describe the training objective for learning prompts in section \[sec:obj\].

Multi-Stage Prompting {#sec:msp}
---------------------

@brown2020gpt3 treat the translation task using the GPT-3 model as a generation task given a few examples and a prompt. However, we believe there are two potential weaknesses of this approach:

-   Lack a separation of encoding and decoding. Unlike neural machine translation models which use two networks to model encoding and decoding, simply treating translation as a context generation task may not be optimal for making PLMs as translators.

-   Limited expressive power of source representations. The auto-regressive LM is unidirectional, and therefore is incapable of directly producing a bidirectional representation of the source sentence.

To overcome the above weaknesses, we propose to divide the procedure of using PLMs as translators into three separate stages: the encoding, the re-encoding, and the decoding stages. By providing different prompts at different stages, we believe the PLM can behave differently druing each stage, and is more capable for generating translations.

Given a source sentence $\mathbf{x}=[x_1,\ldots,x_S]$ and a target sentence $\mathbf{y}=[y_1,\ldots,y_T]$, the details of the three stages are described as follows:

#### The Encoding Stage.

In the encoding stage, the PLM encodes the source sentence $\mathbf{x}$ into a sequence of activations $\boldsymbol{\mathsf{H}}^e_{1:S}$ by using an encoding stage prompt $\boldsymbol{\mathsf{P}}^e$. This procedure is the same with naive prompting. Formally, it can be describe as follows: $$\begin{aligned}
\boldsymbol{\mathsf{H}}^e_{1:S} = f_{\mathrm{LM}}(\boldsymbol{\mathsf{X}}_{1:S}, \boldsymbol{\mathsf{P}}^{e}).\end{aligned}$$

#### The Re-encoding Stage.

In the re-encoding stage, the PLM produces fine-grained representations of the source sentence by re-encoding $\mathbf{x}$ given past activations $\boldsymbol{\mathsf{H}}^e_{1:S}$ and a re-encoding stage prompt $\boldsymbol{\mathsf{P}}^r$, which allows each representation to condition on all words in $\mathbf{x}$. This procedure can be described as $$\begin{aligned}
\boldsymbol{\mathsf{H}}^r_{1:S} = f_{\mathrm{LM}}(\boldsymbol{\mathsf{X}}_{1:S}, \llbracket \boldsymbol{\mathsf{P}}^r; \boldsymbol{\mathsf{H}}^e_{1:S} \rrbracket),\end{aligned}$$ where $\llbracket \boldsymbol{\mathsf{P}}^r; \boldsymbol{\mathsf{H}}^e_{1:S} \rrbracket$ denotes the concatenation of two sequences $\boldsymbol{\mathsf{P}}^r$ and $\boldsymbol{\mathsf{H}}^e_{1:S}$.

#### The Decoding Stage.

Finally, we obtain the hidden vectors $\boldsymbol{\mathsf{G}}_{1:T}$ for predicting the probability of the target sentence $\mathbf{y}$ in the decoding stage, given the refined source representation $\boldsymbol{\mathsf{H}}^r_{1:S}$ and a decoding stage prompt $\boldsymbol{\mathsf{P}}^d$: $$\begin{aligned}
\boldsymbol{\mathsf{G}}_{1:T} = f_{\mathrm{LM}}(\boldsymbol{\mathsf{Y}}_{1:S}, \llbracket \boldsymbol{\mathsf{P}}^d; \boldsymbol{\mathsf{H}}^r_{1:S} \rrbracket). \label{eq:dec}\end{aligned}$$

Figure \[fig:msp\] gives a detailed illustration of MSP. By dividing the translation process into multiple stages and applying different prompts, we expect the PLM model can generate better translations.

Reparameterization {#sec:reparam}
------------------

Learning better prompts for adapting pre-trained language models to translation tasks is challenging. Previous studies @li2021prefix [@liu2021gpt] suggest that using neural networks to reparameterize continuous prompts can bring significant improvements. We adopt the same architecture as @li2021prefix. Formally, we reparameterize $\boldsymbol{\mathsf{P}}^e$, $\boldsymbol{\mathsf{P}}^r$, and $\boldsymbol{\mathsf{P}}^d$ using the following network: $$\begin{aligned}
& \boldsymbol{\mathsf{H}} = \llbracket \boldsymbol{\mathsf{P}}_{\bm{\phi}}^e; \boldsymbol{\mathsf{P}}_{\bm{\phi}}^r; \boldsymbol{\mathsf{P}}_{\bm{\phi}}^d \rrbracket \cdot \bm{W}_1, \\
& \llbracket \boldsymbol{\mathsf{P}}^e; \boldsymbol{\mathsf{P}}^r; \boldsymbol{\mathsf{P}}^d \rrbracket = \tanh(\boldsymbol{\mathsf{H}}) \cdot \bm{W}_2,\end{aligned}$$ where $\bm{W}_1 \in \mathbb{R}^{d \times d}$, $\bm{W}_2 \in \mathbb{R}^{d \times 2Nd}$, $\boldsymbol{\mathsf{P}}_{\bm{\phi}}^e \in \mathbb{R}^{L \times d}$, $\boldsymbol{\mathsf{P}}_{\bm{\phi}}^r \in \mathbb{R}^{L \times d}$, and $\boldsymbol{\mathsf{P}}_{\bm{\phi}}^d \in \mathbb{R}^{L \times d}$ are trainable parameters. Once the training is done, we pre-compute $\boldsymbol{\mathsf{P}}^e$, $\boldsymbol{\mathsf{P}}^r$, and $\boldsymbol{\mathsf{P}}^d$ using the above network. And the network and all trainable parameters are then dropped.

Training Objective {#sec:obj}
------------------

We use the cross-entropy loss for learning prompts. Given $\boldsymbol{\mathsf{G}}_{1:T}=[\bm{g}_1,\ldots,\bm{g}_T]$ in Eq. (\[eq:dec\]), the training objective is formally described as follows: $$\begin{aligned}
\begin{split}
\mathcal{L} &= \sum_{t=1}^T P(y_t|\mathbf{y}_{<t},\mathbf{x}) \\
            &= \sum_{t=1}^T \frac{\exp{(\bm{e}_{z_t}^{\mathsf{T}} \cdot \bm{g}_t})}{\sum_{i=1}^{V} \exp{(\bm{e}_{z_i}^{\mathsf{T}} \cdot \bm{g}_t})}.
\end{split}\end{aligned}$$ Note that the parameters $\bm{\theta}$ of the PLM are fixed during training.

Experiments
===========

\[th\]

  **Model**           **Method**          **Bg**     **Es**     **It**     **Ru**     **Zh**    **Avg.**
  ------------- ----------------------- ---------- ---------- ---------- ---------- ---------- ----------
  Transformer              -               35.2       38.0       34.2       22.6       17.6       29.5
  mGPT               Prefix-Tuning         34.9       40.6       35.4       19.7       15.7       29.3
  mGPT           Multi-Stage Prompting   **37.0**   **42.1**   **37.8**   **24.4**   **18.3**   **31.9**

\[th\]

  **Model**           **Method**          **Bg**     **Es**     **It**     **Ru**    **Zh**   **Avg.**
  ------------- ----------------------- ---------- ---------- ---------- ---------- -------- ----------
  Transformer              -               29.2       34.0       29.2       16.7      21.2      26.1
  mGPT               Prefix-Tuning         32.7       38.2       32.1       16.3      14.1      26.7
  mGPT           Multi-Stage Prompting   **34.1**   **38.4**   **32.8**   **19.2**    14.9    **27.9**

Setup
-----

#### Pre-trained LM.

We used a multilingual GPT-2 (mGPT) @radford2019language model as the pre-trained language model in all our experiments. The mGPT model is trained using the Megatron-LM toolkit @shoeybi2019megatron [^1] on the mC4 dataset @xue2020mt5, which contains massive web crawled data covering 101 languages. The model consists of 24 transformer layers, and the hidden size $d$ of the model is set to 1,024. We used the same tokenization and vocabulary as the mT5 model @xue2020mt5. The number of parameters of the mGPT model is about 560M. [^2]

#### Datasets and Evaluation Metric.

We conduct experiments under three settings to verify our proposed method:

-   *Low-Resource Translation*: We conduct experiments on Bg$\leftrightarrow$En, Es$\leftrightarrow$En, It$\leftrightarrow$En, Ru$\leftrightarrow$En, and Zh$\leftrightarrow$En translation directions. We used the TedTalks dataset @qi2018and for both training and testing.

-   *Medium-Resource Translation*: We used the WMT14 English-German (En-De) dataset as the training corpus for the medium-resource translation task, which consists of 4.5M sentence pairs. The test set is `newstest2014`.

-   *High-Resource Translation*: We used the WMT20 English-Chinese (En-Zh) dataset as the training corpus for the high-resource translation task, which consists of 28M sentence pairs. The test set is `newstest2020`.

We used case-sensitive BLEU @papineni2002bleu as the evaluation metric. The BLEU score is calculated using the <span style="font-variant:small-caps;">SacreBLEU</span> toolkit @post2018call.[^3]

#### Baselines.

We compare our method with the following baselines:

-   Transformer @vaswani2017attention. State-of-the-art neural machine translation models.

-   Prefix-Tuning @li2021prefix. We use prefix-tuning for adapting the mGPT model to translation tasks.

#### Hyper-Parameters.

All our models are trained on a machine with 8 RTX 3090Ti GPUs. For transformer models, we used the *transformer-big* setting and used the same tokenization and vocabulary as of mGPT. All other settings are the same with @vaswani2017attention. For prefix tuning and MSP, we set the prompt length to 128. We use Adam @kingma2014adam ($\beta_1$ = 0.9, $\beta_2$ = 0.98 and $\epsilon$ = 1$\times$ $10^{-9}$) as the optimizer. Each mini-batch contains about 32k tokens. We train prompts for a total of 80k steps. We used the beam search algorithm to obtain translation from the mGPT model, and the beam size is set to 4. We implement our models with the open-source toolkits THUMT @tan2020thumt and Transformers @wolf2020transformers.

Results on the TedTalks Dataset
-------------------------------

Table \[tab:xen\] and \[tab:enx\] show the results on X$\rightarrow$En and En$\rightarrow$X translation tasks, respectively. Our method achieves an average of 31.9 BLEU points on X$\rightarrow$En translation tasks and an average of 27.9 BLEU En$\rightarrow$X translation tasks, outperforming the prefix-tuning baseline by 2.6 BLEU points and 1.2 BLEU points, respectively. Our method also outperforms a strong multilingual Transformer model by 2.4 BLEU and 1.8 BLEU points, respectively. The results indicate that pre-trained language models can effectively exploit large unlabeled raw data, and using pre-trained language models as translators can achieve superior performance than NMT models in low-resource translation scenarios.

Results on the WMT14 En-De Dataset
----------------------------------

  **Model**           **Method**         **BLEU**
  ------------- ----------------------- ----------
  Transformer              -               27.3
  mGPT              Prompt Encoder         25.9
  mGPT               Prefix-Tuning         17.5
  mGPT           Multi-Stage Prompting     21.2

  : Results on the WMT14 En-De dataset.<span data-label="tab:result_ende"></span>

Table \[tab:result\_ende\] shows the result for the WMT14 En-De translation task. With MSP, the mGPT model improves the translation performance by 3.7 BLEU points compared with the prefix-tuning baseline. However, the translation performance of mGPT with MSP is behind the NMT model by a large margin. We conjecture there are two reasons:

1.  Limited capacity of the mGPT model. Our mGPT model is relatively small and trained on massive multilingual data. As a result, the capacity of the mGPT model may limit the performance on translation tasks.

2.  Difficulty of adapting PLM to machine translation tasks. As translation is quite different with language modeling, it is generally difficult for adapting PLMs to translation tasks.

We conduct further experiments to validate our conjecture. We train a separate Transformer encoder to directly map a source sentence to a continuous prompt, leaving the mGPT model only serving as a decoder. Using this approach, the gap of translation performance between the mGPT model and the NMT model narrows to 1.1 BLEU points. The result verifies our assumption that the capacity of mGPT is limited and the difficulty of adapting PLM to translation tasks.

Results on the WMT20 En-Zh Dataset
----------------------------------

\[th\]

  **Model**                  **Architecture**   **\#Params.**        **Method**         **BLEU**
  ------------------------- ------------------ --------------- ----------------------- ----------
  mT5-XXL @zhang2021cpm      Encoder-Decoder         13B             Finetuning           24.0
  CPM-2 @zhang2021cpm        Encoder-Decoder         11B             Finetuning           26.2
  CPM-2 @zhang2021cpm        Encoder-Decoder         11B            Prompt Tuning         24.1
  Ernie 3.0 @sun2021ernie    Encoder-Decoder         10B             Finetuning           26.8
  mGPT                           Decoder            560M            Prompt Tuning         3.9
  mGPT                           Decoder            560M            Prefix-Tuning         21.9
  mGPT                           Decoder            560M        Multi-Stage Prompting   **28.1**

Table \[tab:enzh\] shows the results on the WMT20 En-Zh translation task. We also compare our method with previous works. Our method outperforms the results of mT5-XXL, CPM-2, and Ernie 3.0 models on this task, albeit using a much smaller pre-trained model. Using prompt tuning for adapting mGPT to the En-Zh translation task performs much worse than using prefix-tuning. Prompt tuning introduces fewer trainable parameters than prefix-tuning, which may be insufficient for adapting a relatively small pre-trained LM to translation tasks. Our approach outperforms the baseline using prefix-tuning by 6.2 BLEU points. The results indicate that on high-resource and complex translation directions, multi-stage prompting is more effective in adapting PLMs than prefix-tuning.

Ablation Study
--------------

\[th\]

   **\#**  **Method**                                                                                    **BLEU**
  -------- -------------------------------------------------------------------------------------------- ----------
     1     Multi-Stage Prompting                                                                           21.2
     2     Single prompt for all stages                                                                    19.8
     3     Prefix-Tuning (template: "$\mathbf{x}\ \verb|<S1>|\ \mathbf{x}\ \verb|<S2>|\ \mathbf{y}$")      18.8
     4     Prefix-Tuning (template: "$\mathbf{x}\ \verb|<S>|\ \mathbf{y}$")                                17.5

Table \[tab:ablation\] shows the ablation study on the WMT14 En-De translation task. Using a single prompt during the 3 stages drops the translation performance of mGPT model to 19.8 BLEU points (row 2 vs. row 1), which coincides with our intuition that using different prompts in different stages helps PLMs adapting to translation tasks. Using a double source template with prefix-tuning performs inferior to multi-stage prompting (row 3 vs. row 2), which indicates the necessity of differentiating stages. Repeating the source two times improves the translation performance (row 3 vs. row 4), which confirms that re-encoding is effective in improving the translation performance of PLMs.

Related Work
============

#### Prompting.

@brown2020gpt3 propose to use a few examples and prompts to adapt the GPT-3 model to downstream tasks, which is referred to as *in-context learning*. Their prompts are manually designed. @gao2020making present LM-BFF for automatic prompts generation. They use T5 model @raffel2019exploring to generate templates for prompting PLMs. @li2021prefix propose prefix-tuning, which uses continuous vectors as prompts. These prompts are trained using task-specific data and optimized through gradient descent. @lester2021power propose prompt tuning, which is similar to prefix-tuning but with fewer trainable parameters. @zhang2021cpm investigated using prompt tuning for adapting CPM-2 model to the WMT20 English-Chinese translation task. Our method is also based on prompting. We use continuous prompts for adapting PLMs to translation tasks. Unlike @li2021prefix and @lester2021power who present general frameworks, our method is focused on improving the translation performance of PLMs.

#### Utilizing Pre-trained Models for Machine Translation.

@yang2020towards present CT-NMT for making use of BERT and avoiding the catastrophic forgetting when finetuning the BERT model. Unlike their approach, we do not change the parameters of pre-trained language models during training. @weng2020acquiring introduce an APT framework for employing both the source- and target-side pre-trained models. @zhu2020incorporating propose using additional attention layers to incorporate source BERT models into NMT. Compared with their approaches, our method directly uses PLMs as translators while theirs treat PLMs as components of NMT models. @guo2020incorporating build a non-autoregressive NMT model by using a source BERT model as the encoder and a target BERT as the decoder with adapter layers. @sun2021multilingual propose grafting a source BERT model and a target GPT model for translation tasks. Compared with these approaches, our method only uses one multilingual GPT model. Moreover, we do not add trainable adapter networks into PLMs. @stickland2021recipes investigate using BART and mBART models for machine translation tasks, their approach relies on adapter networks and finetuning part of PLMs. Our approach is based on prompting, we only use prompts for adapting the PLMs to translation tasks. Furthermore, their approach applied for encoder-decoder architecture PLMs while ours applied for decoder-only PLMs. @wang2021language investigate using decoder-only architecture for translation tasks. Our method also uses a decoder-only architecture. However, our model is pre-trained on monolingual data and we only use bilingual data to learn prompts, while @wang2021language use parallel data to learn the whole model.

Conclusion
==========

We have presented multi-stage prompting, a method for making pre-trained models better translators. Experiments show that with multi-stage prompting, pre-trained language models can generate translation even better than neural machine translation models, showing the potential of using pre-trained language models for translation tasks. In future work, we plan to extend our methods to pre-trained language models with the encoder-decoder architecture.

[^1]: <https://github.com/NVIDIA/Megatron-LM>

[^2]: We release our checkpoint at <https://huggingface.co/THUMT/mGPT>.

[^3]: Signature: nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|\
    version:2.0.0

{'source': 'arXiv_out/2110'}