Question: Represent the scientific article to find a short TLDR

Question: Sequence generation models such as recurrent networks can be trained with a diverse set of learning algorithms. For example, maximum likelihood learning is simple and efficient, yet suffers from the exposure bias problem. Reinforcement learning like policy gradient addresses the problem but can have prohibitively poor exploration efficiency. A variety of other algorithms such as RAML, SPG, and data noising, have also been developed in different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation, and show that the apparently divergent algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of reward function and a couple of hyperparameters. The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency. Besides, based on the framework, we present a new algorithm that dynamically interpolates among the existing algorithms for improved learning. Experiments on machine translation and text summarization demonstrate the superiority of the proposed algorithm. Sequence generation is a ubiquitous problem in many applications, such as machine translation BID28, text summarization BID13 BID25, image captioning BID15, and so forth. Great advances in these tasks have been made by the development of sequence models such as recurrent neural networks (RNNs) with different cells BID12 BID6 and attention mechanisms BID1 BID19. These models can be trained with a variety of learning algorithms. The standard training algorithm is based on maximum-likelihood estimation (MLE) which seeks to maximize the log-likelihood of ground-truth sequences. Despite the computational simplicity and efficiency, MLE training suffers from the exposure bias BID24. That is, the model is trained to predict the next token given the previous ground-truth tokens; while at test time, since the ing model does not have access to the ground truth, tokens generated by the model itself are instead used to make the next prediction. This discrepancy between training and test leads to the issue that mistakes in prediction can quickly accumulate. Recent efforts have been made to alleviate the issue, many of which resort to the reinforcement learning (RL) techniques BID24 BID2 BID8. For example, BID24 adopt policy gradient BID29 that avoids the training/test discrepancy by using the same decoding strategy. However, RL-based approaches for sequence generation can face challenges of prohibitively poor sample efficiency and high variance. For more practical training, a diverse set of methods has been developed that are in a middle ground between the two paradigms of MLE and RL. For example, RAML adds reward-aware perturbation to the MLE data examples; SPG BID8 leverages reward distribution for effective sampling of policy gradient. Other approaches such as data noising BID34 ) also show improved . In this paper, we establish a unified perspective of the broad set of learning algorithms. Specifically, we present a generalized entropy regularized policy optimization framework, and show that the apparently diverse algorithms, such as MLE, RAML, SPG, and data noising, can all be re-formulated as special instances of the framework, with the only difference being the choice of reward and the values of a couple of hyperparameters (FIG0). In particular, we show MLE is equivalent to using a delta-function reward that assigns 1 to samples that exactly match data examples while −∞ to any other samples. Such extremely restricted reward has literally disabled any exploration of the model beyond training data, yielding the exposure bias. Other algorithms essentially use rewards that are more smooth, and also leverage model distribution for exploration, which generally in a larger effective exploration space, more difficult training, and better test-time performance. Besides the new understandings of the existing algorithms, the unified perspective also facilitates to develop new algorithms for improved learning. We present an example new algorithm that, as training proceeds, gradually expands the exploration space by annealing the reward and hyperparameter values. The annealing in effect dynamically interpolates among the existing algorithms. Experiments on machine translation and text summarization show the interpolation algorithm achieves significant improvement over the various existing methods. Sequence generation models are usually trained to maximize the log-likelihood of data by feeding the ground-truth tokens during decoding. Reinforcement learning (RL) addresses the discrepancy between training and test by also using models' own predictions at training time. Various RL approaches have been applied for sequence generation, such as policy gradient BID24 and actor-critic BID2. Softmax policy gradient (SPG) BID8 additionally incorporates the reward distribution to generate high-quality sequence samples. The algorithm is derived by applying a log-softmax trick to adapt the standard policy gradient objective. Reward augmented maximum likelihood (RAML) is an algorithm in between MLE and policy gradient. It is originally developed to go beyond the maximum likelihood criteria and incorporate task metric (such as BLEU for machine translation) to guide the model learning. Mathematically, RAML shows that MLE and maximum-entropy policy gradient are respectively minimizing KL divergences in opposite directions. We reformulate both SPG and RAML in a new perspective, and show they are precisely instances of a general entropy regularized policy optimization framework. The new framework provides a more principled formulation for both algorithms. Besides the algorithms discussed in the paper, there are other learning methods for sequence models. For example, Hal BID11 BID16; BID32 use a learningto-search paradigm for sequence generation or structured prediction. Scheduled Sampling adapts MLE by randomly replacing ground-truth tokens with model predictions as the input for decoding the next-step token. Our empirical comparison shows improved performance of the proposed algorithm. Policy optimization for reinforcement learning is studied extensively in robotics and game environment. For example, BID23 introduce a relative entropy regularization to reduce information loss during learning. BID26 develop a trust-region approach for monotonic improvement. BID7; BID17; BID0 study the policy optimization algorithms in a probabilistic inference perspective. The entropy-regularized policy optimization formulation presented here can be seen as a generalization of many of the previous policy optimization methods, as shown in the next section. Besides, we formulate the framework in the sequence generation context. We first present a generalized formulation of an entropy regularized policy optimization framework, to which a broad set of learning algorithms for sequence generation are connected. In particular, we show the conventional maximum likelihood learning is a special case of the policy optimization formulation. This provides new understandings of the exposure bias problem as well as the exploration efficiency of the algorithms. We further show that the framework subsumes as special cases other well-known learning methods that were originally developed in diverse perspectives. We thus establish a unified, principled view of the broad class of works. Let us first set up the notations for the sequence generation setting. Let x be the input and y = (y 1, . . ., y T) the sequence of T tokens in the target space. For example, in machine translation, x is the sentence in source language and y is in target language. Let (x, y *) be a training example drawn from the empirical data distribution, where y * is the ground truth sequence. We aim to learn a sequence generation model p θ (y|x) = t p θ (y t |y 1:t−1, x) parameterized with θ. The model can, for example, be a recurrent network. It is worth noting that though we present in the sequence generation context, the formulations can straightforwardly be extended to other settings such as robotics and game environment. Policy optimization is a family of reinforcement learning (RL) algorithms that seeks to learn the parameter θ of the model p θ (a.k.a policy). Given a reward function R(y|y *) ∈ R (e.g., BLEU score in machine translation) that evaluates the quality of generation y against the true y *, the general goal of policy optimization is to maximize the expected reward. A rich research line of entropy regularized policy optimization (ERPO) stabilizes the learning by augmenting the objective with information theoretic regularizers. Here we present a generalized formulation of ERPO. Assuming a general distribution q(y|x) (more details below), the objective we adopt is written as DISPLAYFORM0 where KL(· ·) is the Kullback-Leibler divergence forcing q to stay close to p θ; H(·) is the Shannon entropy imposing maximum entropy assumption on q; and α and β are balancing weights of the respective terms. In the RL literature, the distribution q has taken various forms, leading to different policy optimization algorithms. For example, setting q to a non-parametric policy and β = 0 in the prominent relative entropy policy search BID23 algorithm. Assuming q as a parametric distribution and α = 0 leads to the commonly-used maximum entropy policy gradient BID36 BID10. Letting q be a variational distribution and β = 0 corresponds to the probabilistic inference formulation of policy gradient BID0 BID17. Related objectives have also been used in other popular RL algorithms BID26 BID30.We assume a non-parametric q. The above objective can be maximized with an EM-style procedure that iterates two coordinate ascent steps optimizing q and θ, respectively. At iteration n: DISPLAYFORM1 The E-step is obtained with simple Lagrange multipliers. Note that q has a closed-form solution in the E-step. We can have an intuitive interpretation of its form. First, it is clear to see that if α → ∞, we have q n+1 = p n θ. This is also reflected in the objective Eq. where the weight α encourages q to be close to p θ. Second, the weight β serves as the temperature of the q softmax distribution. In particular, a large temperature β → ∞ makes q a uniform distribution, which is consistent to the outcome of an infinitely large maximum entropy regularization in Eq.. In terms of the M-step, the update rule can be interpreted as maximizing the log-likelihood of samples from the distribution q. In the context of sequence generation, it is sometimes more convenient to express the equations at token level, as shown shortly. To this end, we decompose R(y|y *) along the time steps: DISPLAYFORM2 where ∆R(y t |y *, y 1:t−1) measures the reward contributed by token y t. The solution of q in Eq. can then be re-written as: DISPLAYFORM3 The above ERPO framework has three key hyperparameters, namely (R, α, β). In the following, we show that different values of the three hyperparameters correspond to different learning algorithms (FIG0). We first connect MLE to the above general formulation, and compare and discuss the properties of MLE and regular ERPO from the new perspective. Maximum likelihood estimation is the most widely-used approach to learn a sequence generation model due to its simplicity and efficiency. It aims to find the optimal parameter value that maximizes the data log-likelihood: DISPLAYFORM0 As discussed in section 1, MLE suffers from the exposure bias problem as the model is only exposed to the training data, rather than its own predictions, by using the ground-truth subsequence y * 1:t−1 to evaluate the probability of y * t. We show that the MLE objective can be recovered from Eq. with specific reward and weight configurations. Consider a δ-reward defined as 1: DISPLAYFORM1 Let (R = R δ, α → 0, β = 1). From the E-step of Eq., we have q(y|x) = 1 if y = y * and 0 otherwise. The M-step is therefore equivalent to arg max θ log p θ (y * |x), which recovers precisely the MLE objective in Eq..That is, MLE can be seen as an instance of the policy optimization algorithm with the δ-reward and the above weight values. Any sample y that fails to match precisely the data y * will receive a negative infinite reward and never contribute to model learning. The ERPO reformulation of MLE provides a new statistical explanation of the exposure bias problem. Specifically, a very small α value makes the model distribution ignored during sampling from q, while the δ-reward permits only samples that match training examples. The two factors in effect make void any exploration beyond the small set of training data FIG1 ), leading to a brittle model that performs poorly at test time due to the extremely restricted exploration. On the other hand, however, a key advantage of the δ-reward specification is that its regular reward shape allows extreme pruning of the huge sample space, ing in a space that includes exactly the training examples. This makes the MLE implementation very simple and the computation very efficient in practice. On the contrary, common rewards (e.g., BLEU) used in policy optimization are more smooth than the δ-reward, and permit exploration in a broader space. However, such rewards usually do not have a regular shape as the δ-reward, and thus are not amenable to sample space pruning. Generally, a larger exploration space would lead to a harder training problem. Also, when it comes to the huge sample space, the rewards are still very sparse (e.g., most sequences have BLEU=0 against a reference sequence). Such reward sparsity can make exploration inefficient and even impractical.1 For token-level, define R δ (y1:t|y *) = t/T * if y1:t = y * 1:t and −∞ otherwise, where T * is the length of y *. Note that the R δ value of y = y * can also be set to any constant larger than −∞. Given the opposite algorithm behaviors in terms of exploration and computation efficiency, it is a natural idea to seek a middle ground between the two extremes to combine the advantages of both. A broad set of such approaches have been recently developed. We re-visit some of the popular ones, and show that these apparently divergent approaches can all be reformulated within our ERPO framework (Eqs.1-4) with varying reward and weight specifications. RAML was originally proposed to incorporate task metric reward into the MLE training, and has shown superior performance to the vanilla MLE. Specifically, it introduces an exponentiated reward distribution e(y|y *) ∝ exp{R(y|y *)} where R, as in vanilla policy optimization, is a task metric such as BLEU. RAML maximizes the following objective: DISPLAYFORM0 That is, unlike MLE that directly maximizes the data log-likelihood, RAML first perturbs the data proportionally to the reward distribution e, and maximizes the log-likelihood of the ing samples. The RAML objective reduces to the vanilla MLE objective if we replace the task reward R in e(y|y *) with the MLE δ-reward (Eq.6). The relation between MLE and RAML still holds within our new formulation (Eqs.1-2). In particular, similar to how we recovered MLE from Eq., let (α → 0, β = 1) 2, but set R to the task metric reward, then the M-step of Eq. is precisely equivalent to maximizing the above RAML objective. Formulating within the same framework allows us to have an immediate comparison between RAML and others. In particular, compared to MLE, the use of smooth task metric reward R instead of R δ permits a larger effective exploration space surrounding the training data FIG1 ), which helps to alleviate the exposure bias problem. On the other hand, α → 0 as in MLE still limits the exploration as it ignores the model distribution. Thus, RAML takes a step from MLE toward regular RL, and has effective exploration space size and exploration efficiency in between. SPG BID8 was developed in the perspective of adapting the vanilla policy gradient BID29 to use reward for sampling. SPG has the following objective: DISPLAYFORM0 where R is a common reward as above. As a variant of the standard policy gradient algorithm, SPG aims to address the exposure bias problem and shows promising BID8.We show SPG can readily fit into our ERPO framework. Specifically, taking gradient of Eq. w.r.t θ, we immediately get the same update rule as in Eq. FORMULA1 with (α = 1, β = 0, R = common reward).Note that the only difference between the SPG and RAML configuration is that now α = 1. SPG thus moves a step further than RAML by leveraging both the reward and the model distribution for full exploration FIG1 ). Sufficient exploration at training time would in theory boost the test-time performance. However, with the increased learning difficulty, additional sophisticated optimization and approximation techniques have to be used BID8 to make the training practical. Adding noise to training data is a widely adopted technique for regularizing models. Previous work BID34 has proposed several data noising strategies in the sequence generation context. For example, a unigram noising, with probability γ, replaces each token in data y * with a sample from the unigram frequency distribution. The ing noisy data is then used in MLE training. Though previous literature has commonly seen such techniques as a data pre-processing step that differs from the above learning algorithms, we show the ERPO framework can also subsume data noising as a special instance. Specifically, starting from the ERPO reformulation of MLE which takes (R = R δ, α → 0, β = 1) (section 3.2), data noising can be formulated as using a locally relaxed variant of R δ. For example, assume y has the same length with y * and let ∆ y,y * be the set of tokens in y that differ from the corresponding tokens in y *, then a simple data noising strategy that randomly replaces a single token y * t with another uniformly picked token is equivalent to using a reward R δ (y|y *) that takes 1 when |∆ y,y * | = 1 and −∞ otherwise. Likewise, the above unigram noising BID34 ) is equivalent to using a reward DISPLAYFORM0 where u(·) is the unigram frequency distribution. With a relaxed (i.e., smoothed) reward, data noising expands the exploration space of vanilla MLE locally FIG1 ). The effect is essentially the same as the RAML algorithm (section 3.3), except that RAML expands the exploration space based on the task metric reward. made an early attempt to address the exposure bias problem by exploiting the classic policy gradient algorithm BID29 and mixing it with MLE training. We show in the supplementary materials that the algorithm is closely related to the ERPO framework, and can be recovered with moderate approximations. Section 2 discusses more relevant algorithms for sequence generation learning. We have presented the generalized ERPO framework, and connected a series of well-used learning algorithms by showing that they are all instances of the framework with certain specifications of the three hyperparameters (R, α, β). Each of the algorithms can be seen as a point in the hyperparameter space FIG0 ). Generally, a point with a more restricted reward function R and a very small α tends to have a smaller effective exploration space and allow efficient learning (e.g., MLE), while in contrast, a point with smooth R and a larger α would lead to a more difficult learning problem, but permit more sufficient exploration and better test-time performance (e.g., (softmax) policy gradient).The unified perspective provides new understandings of the existing algorithms, and also facilitates to develop new algorithms for further improvement. Here we present an example algorithm that interpolates the existing ones. The interpolation algorithm exploits the natural idea of starting learning from the most restricted yet easiest problem configuration, and gradually expands the exploration space to reduce the discrepancy from the test time. The easy-to-hard learning paradigm resembles the curriculum learning BID4 ). As we have mapped the algorithms to points in the hyperparameter space, interpolation becomes very straightforward, which requires only annealing of the hyperparameter values. Specifically, in the general update rules Eq., we would like to anneal from using R δ to using smooth common reward, and anneal from exploring by only R to exploring by both R and p θ. Let R comm denote a common reward (e.g., BLEU). The interpolated reward can be written in the form R = λR comm + (1 − λ)R δ, for λ ∈. Plugging R into q in Eq. FORMULA1 and re-organizing the scalar weights, we obtain the numerator of q in the form: c · (λ 1 log p θ + λ 2 R comm + λ 3 R δ), whereModel BLEU MLE 26.44 ± 0.18 RAML 27.22 ± 0.14 SPG BID8 26.62 ± 0.05 MIXER BID24 26.53 ± 0.11 Scheduled Sampling 26.76 ± 0.17Ours 27.82 ± 0.11 Table 1: Results of machine translation.(λ 1, λ 2, λ 3) is defined as a distribution (i.e., λ 1 +λ 2 +λ 3 = 1), and, along with c ∈ R, are determined by (α, β, λ). For example, λ 1 = α/(α + 1). We gradually increase λ 1 and λ 2 and decrease λ 3 as the training proceeds. Further, noting that R δ is a Delta function (Eq.6) which would make the above direct function interpolation problematic, we borrow the idea from the Bayesian spike-and-slab factor selection method BID14. That is, we introduce a categorical random variable z ∈ {1, 2, 3} that follows the distribution (λ 1, λ 2, λ 3), and augment q as q(y|x, z) ∝ exp{c · (1(z = 1) log p θ + 1(z = 2)R comm + 1(z = 3)R δ )}. The M-step is then to maximize the objective with z marginalized out: DISPLAYFORM0 The spike-and-slab adaption essentially transforms the product of experts in q to a mixture, which resembles the bang-bang rewarded SPG method BID8 where the name bang-bang refers to a system that switches abruptly between extreme states (i.e., the z values). Finally, similar to BID8, we adopt the token-level formulation (Eq.4) and associate each token with a separate variable z. We provide the pseudo-code of the interpolation algorithm in the supplements. It is notable that Ranzato et al. FORMULA0 also develop an annealing strategy that mixes MLE and policy gradient training. As discussed in section 3 and the supplements, the algorithm can be seen as a special instance of the ERPO framework (with moderate approximation) we have presented. Next section shows improved performance of the proposed, more general algorithm compared to BID24. We evaluate the above interpolation algorithm in the tasks of machine translation and text summarization. The proposed algorithm consistently improves over a variety of previous methods. Code will be released upon acceptance. Setup In both tasks, we follow previous work BID24 and use an attentional sequence-to-sequence model BID19 where both the encoder and decoder are single-layer LSTM recurrent networks. The dimensions of word embedding, RNN hidden state, and attention are all set to 256. We apply dropout of rate 0.2 on the recurrent hidden state. We use Adam optimization for training, with an initial learning rate of 0.001 and batch size of 64. At test time, we use beam search decoding with a beam width of 5. Please see the supplementary materials for more configuration details. Dataset Our dataset is based on the common IWSLT 2014 BID5 German-English machine translation data, as also used in previous evaluation BID24. After proper pre-processing as described in the supplementary materials, we obtain the final dataset with train/dev/test size of around 146K/7K/7K, respectively. The vocabulary sizes of German and English are around 32K and 23K, respectively. Results The BLEU metric BID22 ) is used as the reward and for evaluation. Table 1 shows the test-set BLEU scores of various methods. Besides the approaches described above, we also compare with the Scheduled Sampling method which combats the exposure bias by feeding model predictions at randomly-picked decoding steps during training. From the table, we can see the various approaches such as RAML provide improved performance over the vanilla MLE, as more sufficient exploration is made at training time. Our proposed new algorithm performs best, as it interpolates among the existing algorithms to gradually increase the exploration space and solve the generation problem better. FIG2 shows the test-set BLEU scores against the training steps. We can see that, with annealing, our algorithm improves BLEU smoothly, and surpasses other algorithms to converge at a better point. Table 2: Results of text summarization. Dataset We use the popular English Gigaword corpus BID9 for text summarization, and pre-processed the data following BID25. The ing dataset consists of 200K/8K/2K source-target pairs in train/dev/test sets, respectively. More details are included in the supplements. Results The ROUGE metrics (including -1, -2, and -L) BID18 are the most commonly used metrics for text summarization. Following previous work BID8, we use the summation of the three ROUGE metrics as the reward in the learning algorithms. Table 2 show the on the test set. The proposed interpolation algorithm achieves the best performance on all the three metrics. For easier comparison, FIG3 shows the improvement of each algorithm compared to MLE in terms of ROUGE-L. The RAML algorithm, which performed well in machine translation, falls behind other algorithms in text summarization. In contrast, our method consistently provides the best . We have presented a unified perspective of a variety of well-used learning algorithms for sequence generation. The framework is based on a generalized entropy regularized policy optimization formulation, and we show these algorithms are mathematically equivalent to specifying certain hyperparameter configurations in the framework. The new principled treatment provides systematic understanding and comparison among the algorithms, and inspires further enhancement. The proposed interpolation algorithm shows consistent improvement in machine translation and text summarization. We would be excited to extend the framework to other settings such as robotics and game environments. A POLICY GRADIENT & MIXER BID24 made an early attempt to address the exposure bias problem by exploiting the policy gradient algorithm BID29. Policy gradient aims to maximizes the expected reward: DISPLAYFORM0 where R P G is usually a common reward function (e.g., BLEU). Taking gradient w.r.t θ gives: DISPLAYFORM1 We now reveal the relation between the ERPO framework we present and the policy gradient algorithm. Starting from the M-step of Eq. and setting (α = 1, β = 0) as in SPG (section 3.4), we use p θ n as the proposal distribution and obtain the importance sampling estimate of the gradient (we omit the superscript n for notation simplicity): DISPLAYFORM2 where Z θ = y exp{log p θ + R} is the normalization constant of q, which can be considered as adjusting the step size of gradient descent. We can see that Eq. recovers Eq. if we further set R = log R P G, and omit the scaling factor Z θ. In other words, policy gradient can be seen as a special instance of the general ERPO framework with (R = log R P G, α = 1, β = 0) and with Z θ omitted. The MIXER algorithm BID24 incorporates an annealing strategy that mixes between MLE and policy gradient training. Specifically, given a ground-truth example y *, the first m tokens y * 1:m are used for evaluating MLE loss, and starting from step m + 1, policy gradient objective is used. The m value decreases as training proceeds. With the relation between policy gradient and ERPO as established above, MIXER can be seen as a specific instance of the proposed interpolation algorithm (section 4) that follows a restricted annealing strategy for token-level hyperparameters (λ 1, λ 2, λ 3). That is, for t < m in Eq.4 (i.e.,the first m steps), (λ 1, λ 2, λ 3) is set to and c = 1, namely the MLE training; while for t > m, (λ 1, λ 2, λ 3) is set to (0.5, 0.5, 0) and c = 2. Algorithm 1 summarizes the interpolation algorithm described in section 4. Get training example (x, y *) for t = 0, 1,..., T do 5: DISPLAYFORM0 if z = 1 then Sample token y t ∼ exp{c · log p θ (y t |y 1:t−1, x)} 8: DISPLAYFORM1 Sample token y t ∼ exp{c · ∆R comm (y t |y 1:t−1, y *)} 10: DISPLAYFORM2 Sample token y t ∼ exp{c · ∆R δ}, i.e., set y t = y * t end if end for Update θ by maximizing the log-likelihood log p θ (y|x) Anneal λ by increasing λ 1 and λ 2 and decreasing λ 3 16: until convergence C EXPERIMENTAL SETTINGS C.1 DATA PRE-PROCESSING For the machine translation dataset, we follow BID20 for data pre-processing. In text summarization, we sampled 200K out of the 3.8M pre-processed training examples provided by BID25 for the sake of training efficiency. We used the refined validation and test sets provided by BID35. For RAML, we use the sampling approach (n-gram replacement) by BID20 to sample from the exponentiated reward distribution. For each training example we draw 10 samples. The softmax temperature is set to τ = 0.4.For Scheduled Sampling, the decay function we used is inverse-sigmoid decay. The probability of sampling from model i = k/(k + exp (i/k)), where k is a hyperparameter controlling the speed of convergence, which is set to 500 and 600 in the machine translation and text summarization tasks, respectively. For MIXER BID24, the advantage function we used for policy gradient is R(y 1:T |y *)− R(y 1:m |y *).For the proposed interpolation algorithm, we initialize the weights as (λ 1, λ 2, λ 3) = (0.04, 0, 0.96), and increase λ 1 and λ 2 while decreasing λ 3 every time when the validation-set reward decreases. Specifically, we increase λ 1 by 0.06 once and increase λ 2 by 0.06 for four times, periodically. For example, at the first time the validation-set reward decreases, we increase λ 1, and at the second to fifth time, we increase λ 2, and so forth. The weight λ 3 is decreased by 0.06 every time we increase either λ 1 or λ 2. Notice that we would not update θ when the validation-set reward decreases. Here we present additional of machine translation using a dropout rate of 0.3 (Table 3). The improvement of the proposed interpolation algorithm over the baselines is comparable to that of using dropout 0.2 (Table 1 in the paper). For example, our algorithm improves over MLE by 1.5 BLEU points, and improves over the second best performing method RAML by 0.49 BLEU points. (With dropout 0.2 in Table 1, the improvements are 1.42 BLEU and 0.64, respectively.) We tested with dropout 0.5 and obtained similar . The proposed interpolation algorithm outperforms existing approaches with a clear margin. FIG6 shows the convergence curves of the comparison algorithms. Model BLEU MLE 26.63 ± 0.11 RAML 27.64 ± 0.09 SPG BID8 26.89 ± 0.06 MIXER BID24 27.00 ± 0.13 Scheduled Sampling 27.03 ± 0.15 Ours 28.13 ± 0.12 Table 3: Results of machine translation when dropout is 0.3. Table 3 ) picked according to the validation set performance.

represent the short summary for finding the corresponding scientific article (long).

A unified perspective of various learning algorithms for sequence generation, such as MLE, RL, RAML, data noising, etc.

Represent the short summary for finding the corresponding scientific article (long)

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: Represent the article for finding a single-sentence TLDR

Question: We are reporting the SHINRA project, a project for structuring Wikipedia with collaborative construction scheme. The goal of the project is to create a huge and well-structured knowledge base to be used in NLP applications, such as QA, Dialogue systems and explainable NLP systems. It is created based on a scheme of ”Resource by Collaborative Contribution (RbCC)”. We conducted a shared task of structuring Wikipedia, and at the same, submitted are used to construct a knowledge base. There are machine readable knowledge bases such as CYC, DBpedia, YAGO, Freebase Wikidata and so on, but each of them has problems to be solved. CYC has a coverage problem, and others have a coherence problem due to the fact that these are based on Wikipedia and/or created by many but inherently incoherent crowd workers. In order to solve the later problem, we started a project for structuring Wikipedia using automatic knowledge base construction shared-task. The automatic knowledge base construction shared-tasks have been popular and well studied for decades. However, these tasks are designed only to compare the performances of different systems, and to find which system ranks the best on limited test data. The of the participated systems are not shared and the systems may be abandoned once the task is over. We believe this situation can be improved by the following changes: 1. designing the shared-task to construct knowledge base rather than evaluating only limited test data 2. making the outputs of all the systems open to public so that we can run ensemble learning to create the better than the best systems 3. repeating the task so that we can run the task with the larger and better training data from the output of the previous task (bootstrapping and active learning) We conducted “SHINRA2018” with the above mentioned scheme and in this paper we report the and the future directions of the project. The task is to extract the values of the pre-defined attributes from Wikipedia pages. We have categorized most of the entities in Japanese Wikipedia (namely 730 thousand entities) into the 200 ENE categories. Based on this data, the shared-task is to extract the values of the attributes from Wikipedia pages. We gave out the 600 training data and the participants are required to submit the attribute-values for all remaining entities of the same category type. Then 100 data out of them for each category are used to evaluate the system output in the shared-task. We conducted a preliminary ensemble learning on the outputs and found 15 F1 score improvement on a category and the average of 8 F1 score improvements on all 5 categories we tested over a strong baseline. Based on this promising , we decided to conduct three tasks in 2019; multi-lingual categorization task (ML), extraction for the same 5 categories in Japanese with a larger training data (JP-5) and extraction for 34 new categories in Japanese (JP-34). Based on "Resource by Collaborative Contribution (RbCC)" scheme, we conducted a shared task for structuring Wikipedia for the purpose of attracting participants, but at the same, submitted are used to construct a knowledge base. There have been a lot of shared-tasks, but the of the participated systems are not well used. These , which we believe are the great resources, are abandoned after the evaluation is over. We conducted a project which is a shared-task on AKBC, but at the same time the of the systems are gathered and used to produce even the better than the best participated system by ensemble learning. One of the trick is that we, the organizers, don't notify which is the test data to the participants. So they have to run their systems on all entities in Wikpedia and submit them, even though the evaluation are reported only on 100 test data among the entire data. By this methods, the organizers will receive the structured information for all entities from the participants. The accumulated are later become open to the public, and will be used to build the even better structured knowledge by the ensemble learning methods, for example. In other words, this is a project to use AKBC systems as a tool to construct a huge and well-structured Knowledge Base in collaborative manner. We will report "SHINRA2018" project which runs under the RbCC scheme. The task is the attribute extraction task, i.e. to extract the values of the attributes from Wikipedia pages. We have categorized most of Japanese Wikipedia entities (namely 730 thousand entities) into the 200 Extended Named Entity (ENE) categories prior to this project. Based on this data, the shared-task is to extract the values of the attributes which are defined for each category from texts and infobox in the Wikipedia pages. We gave out the 600 training data and the participants are required to submit the attribute-values for all remaining entities of the same category. Then 100 data, which is hidden to the participants for each category are used to evaluate the system as a shared-task. To this project, 8 groups submitted the based on 15 systems. We conducted a preliminary ensemble learning on the outputs in order to demonstrate how the RbCC scheme works. The of the ensemble learning shows a huge improvement over the best single system for each category. The improvements are 15 F-score on a category of "airport" in which the best system achieves 72 F-score, and 9 F-score on the average. This show that the RbCC scheme is very effective. Wikipedia is a great resource as a knowledge base of the entities in the world. However, Wikipedia is created for human to read rather than machines to process. Our goal is to transform the current Wikipedia to a machine readable format based on a clean structure. There are several machine readable knowledge bases (KB) such as CYC BID4, DBpedia BID3, YAGO BID7, Freebase BID0, Wikidata BID13 and so on, but each of them has problems to be solved. CYC has a coverage problem, and others have a coherence problem due to the fact that these are based on Wikipedia and/or created by many but inherently incoherent crowd workers. In order to solve these problems, we started a project for structuring Wikipedia using automatic knowledge base construction (AKBC) shared-task using a cleaner ontology definition. The automatic knowledge base construction shared-tasks have been popular for decades. In particular, there are popular shared-tasks in the field of Information Extraction, Knowledge Base population and attribute extraction, such as KBP[U.S. National Institute of Standards and Technology (NIST), 2018] and CoNLL. However, most of these tasks are designed only to compare the performances of participated systems, and to find which system ranks the best on limited test data. The outputs of the participated systems are not shared and the and the systems may be abandoned once the evaluation task is over. We believe this situation can be improved by the following changes:1. designing the shared-task to construct knowledge base rather than only evaluating on limited test data 2. making the outputs of all the systems open to public so that anyone can run ensemble learning to create the better than the best single system 3. repeating the task so that we can run the task with the larger and better training data from the output of the previous task (active learning and bootstrapping)We conducted "SHINRA2018" with the aforementioned ideas, we call it "Resource by Collaborative Contribution (RbCC)". In this paper we report the first and the future directions of the project. The task is to extract the values of the pre-defined attributes from Wikipedia entity pages. We used Extended Named Entity (ENE) as the definition of the category (in total 200 categories in the ontology) and the attributes (average of 20 attributes) for each category. We have categorized most of the entities in Japanese Wikipedia (namely 730 thousand entities) into the ENE categories prior to this project. Based on this data, the sharedtask is to extract values of the attributes defined for the category of each entity. At the SHINRA2018 project, we limited the target categories to 5, namely, person, company, city, airport and chemical compound. We gave out the 600 training data each for 5 categories at and the participants are supposed to submit the attribute-values for all remaining entities of the categories in Japanese Wikipedia. Then 100 data out of the entire pages of the category are used at the evaluation of the participated systems in the shared-task. For example, there are about 200K person entities in Japanese Wikipedia, and the participants have to extract the attribute-values, such as "birthday", "the organizations he/she have belonged", "mentor" and "awards" from all the remaining entities (i.e. 199.4K = 200K-600 entities). Before starting the project, the participants signed the contract that all the output will be shared among all participants, so that anyone can conduct the ensemble learning on those outputs, and hence create a better knowledge base than the best system in the task. Note that, for the sake of participant's interest, i.e. a company may want to keep the system as their property, the outputs are required to be shared, but their systems are not necessarily to be shared. A promising of the ensemble learning is achieved and we envision that it will lead to the cleaner machine readable knowledge base construction. Structured knowledge bases have considered as one of the most important knowledge resources in the fields of Natural Language Processing. There are several major projects targeted to construct structured knowledge bases in the past. One of the earliest project is CYC, and more recently there are Wikipedia based projects such as DBpedia, Yago, Freebase and Wikidata. Moreover, there are some shared-tasks aiming to build techniques for knowledge base structuring such as KBP and CoNLL. We will introduce these resources and projects and describe the points we consider as issues to be solved in those projects. CYC ontology is a large knowledge base constructed as common sense knowledge BID4. This is one of the large projects in the AI in 80-90's, which use the human labor to construct knowledge base. The cost of construction and maintenance of the handmade knowledge bases for the general domain is very high, and it is known that the knowledge bases have problems in the coverage and the consistency. DBpedia is a more recent project to construct a structured information from the semistructured data in Wikipedia such as infoboxes or categories BID3. DBpedia also has a problem of accuracy, coverage, and coherence. Like CYC, it is also created by human, but in this case, those who worked on creating the knowledge are non-experts of ontology. For example, "Shinjuku Station", which is a railway station, has a category "Odakyu Electric Railway", which is a railway company using the station. Of course, a station can't be an instance of a railway company, so this is not appropriate category. There are so many examples like this in DBpedia. Also there are many inconsistencies in the category hierarchy, and the attributes defined in the KB are not well organized in many categories. Yet Another Greater Ontology (YAGO) is a ontology constructed by mapping Wikipedia articles to the WordNet synsets BID7. YAGO has adopted attributes information extracted from infoboxes like as DBpedia because no attributes defined in WordNet synsets. Freebase is a project to construct a structured knowledge base by crowdsourcing, same as Wikipedia BID0. However, by the crowdsource approach, Freebase doesn't have a well-organized ontology. It has many noises and lack of coherence because these were created by unorganized crowds. Currently, the project of Freebase has paused and integrated into Wikidata. Wikidata is aiming to be a structured knowledge base based on corwdsourcing scheme. BID13. Wikidata also have noises and lack of coherence because it has constructed by bottom-up approach same as Wikipedia and Freebase. For example, just comparing the definition of "city", "town" and "human settlements", we can easily observe inconsistencies in the property (the number of properties are 30, 0 and 6, respectively), there are very biased properties such as "Danish urban area code" in "human settlements", there are many related ambiguous entities, such as "like a city", "city/town" and so on. Also, the category inconsistency can be easily found, for example, "city museum", "mayor" are subcategory of "city", although a mayor is not an instance of "city". Wikipedia allows topics to be included in a category, however, this policy prevents to make the category hierarchy as a well-designed ontology. KBP is a shared task organized by NIST for establishing a technology to construct a structured knowledge base from non-structured documents[U.S. National Institute of Standards and Technology (NIST), 2018]. KBP mainly consists of two tasks. One task is an Entity Discovery and Linking (EDL) which is to find and identify an entity defined in DB from documents. Another one is a Slot Filling which is to extract attribute information of the entity. KBP in general is limited entity types to Person, Location, and Organization in contrast to Wikipedia's wide coverage, and mostly it is a competition based project and no resource creation purpose. FIne Grained Entity Recognition (FIGER) is a project to identify 112 types of named entity classes that are finely defined, such as ENE, from documents BID6. The category of FIGER seems a bit biased, and it doesn't have attribute definitions for each category. In order to create structured knowledge base which is useful for NLP applications, we have learned that well structured ontology is needed and it has to be designed top-down manner. DBpedia, Freebase and Wikidata are created by crowds in bottom up manner,and these have inconsistent entries, imbalanced ontologies and adhoc attributes, as we described in the previous section. We believe the major cause is the fact that these are created bottomup manner, and a top-down design is essential to design the ontology and the attributes. As a top-down designed ontology for named entities, we employed the "Extended Named Entity (ENE) hierarchy" in our project. Extended Named Entity (ENE) is a named entity classification hierarchy along with the attribute definition for each category BID9 BID10. It includes fine-grained 200 categories of entities in hierarchy of up to 4 layers. It contains not only the finer categories of the typical NE categories, such as "city" and "lake for "location" or "company" for "organization", but also contains new named entity types such as "products", "event", "position" and so on. These categories are designed to cover a large amount of entities in the world using encyclopedia and many other resources. Figure 1 shows the ENE definition, version 7.1.0. Attributes are defined based on the investigation of the entities in each category. For example, the attributes for "airport" categories are as follows: "Reading", "IATA code", "ICAO code", "nickname", Figure 1: Definition of Extended Named Entity Hierarchy "name origin", "number of users per year", "the year of the statistic", "the number of airplane landing per year", "longitude", "latitude", "location", "old name", "elevation", "big city near by", "number of runaway" and so on. Please refer to the HP for the complete definition. In order to conduct the shared-task of the attribute-value extraction on Wikipedia entities, first, we have to assign one or more categories to each Wikipedia entity. For example, we have to know the Wikipedia page of "Chicago O'Hare Airport" belongs to an airport entity, and we supposed to extract attribute-value of airport from the page. We have annotated one or more of 200 ENE categories to 782,406 entities of Japanese Wikipedia (201711 version) prior to this project. At the annotation, we have excluded the less popular entities which have less than 5 incoming links (151K entities), and non-entity pages (about 53K pages) such as common nouns and simple numbers. This annotation was done by Machine learning method followed by hand checking on less reliable ones. The machine learning BID8 was conducted with 20K training data created by hand. Then a human check was conducted on the machine learning outputs with less reliable scores. In order to see the quality of the categorization, we evaluate a sample data by multiple annotators to see the accuracy of the data and we observed the accuracy of the categorization is 98.5%. The remaining 1.5% are those which are ambiguous in nature and are very difficult even for the human annotators. TAB0 show the most frequent categories with the frequencies in the data. In this section, we will describe the definition of the shared-task. The task is to extract the attribute-values of entities from the Wikipedia page. For this year's task (SHINRA2018), we picked 5 categories, namely "person", "city", "company", "airport" and "chemical compound" for the shared-task. This selection was done on the largest subcategories of "person", "location", "organization", which are the traditional three categories of named entity (person has no subcategories, and itself is the only category in ENE) "Airport" is selected as a category which has well-structured infobox in Wikipedia, on the other hand, "chemical compound" is selected because the information in infobox is not quite satisfactory for NLP purpose. The infobox for "Chemical compound" contains the factual information such as "boiling temperature", or "chemical formula", but it doesn't contain information such as "usage" or "production method", which we believe are important attributes, for example, for QA purpose. We gave out 600 training data for each category. In the training data, all attributevalues mentioned in the Wikipedia page are manually extracted and form the training data in JSON format. The participants also received the list of all entities, i.e. Wikipedia pages, for 5 categories, and they are required to extract attribute-values from Wikipedia pages of all entities. The evaluation is conducted on 100 entities for each category, but the participants are not notified which 100 entities are used for evaluation even after the evaluation is over. This is for the purpose of the data construction so that the participants have to do their best to produce the output for all data, and the purpose of the future comparison (if the test data is known, the participants could tune their system to the test data even unintentionally through a number of experiments). The are reported by precision, recall and F1 scores, as usual. The systems submitted before the deadline are reported as the formal and the submitted after the deadline are reported as a reference . In the ensemble learning experiment which will be reported later in this paper, we use all the regardless of the formal or reference so that we can achieve the best for the resource construction purpose. The participants are not required to submit the for all 5 categories, as some participants might be interested in a particular category or may have smaller machine resources to run their system for all the categories. The manual creation of the training and test data was not easy. In this section, we briefly described the data preparation, as this process is very interesting and could be a topic of one another paper by itself. We tried to use three types of annotators to create the data as a preliminary experiment.• Experts of the construction of linguistic data• Students who are supervised by experts• Workers on the crowdsourcing (Lancers)As we can imagine intuitively, we found out that the upper in the list, the more expensive, but at the same time the more accurate. Also, we found that the crowdsourcing has relatively high coverage based on our strategy of the crowdsourcing. The task of crowdsourcing is designed with three stages. The first stage is to identify the sections where the given attribute-value is written. In this stage, even the worker find the value in the page, they are not requested to extract the value. This identification of the sections will be repeated until two workers found no value is found, because some attributes have multiple values in one page. Then in the second stage, the values are extracted from the sections which are identified to contain the value(s). The final stage is to check if the extracted value is really the value for the attribute. Maybe this careful strategy might lead to produce the relatively high coverage. Based on the preliminary annotation experiments, we decided to use "expert" and "crowd" at the final data creation. The first round annotation is done by both "expert" and "crowd" independently for the same attributes, and then both are merged to create the final annotation by another "expert" (different from the one who annotate it initialy). The inter-annotator agreements between "crowd" and "experts" are 60-80% and that between "experts" are 80-90% depending on the attributes. The coverage by "crowds" is relatively high and it suggest missing information by the first "expert" at the final annotation by the "expert". In this section, we will report the of the shared-task. Five months are given to the participants to develop their systems and run their experiments from April to September 2018. 16 systems by 8 participants are submitted at SHINRA2018. The first two columns in TAB1 shows the participants (some in abbreviations) and their methods. Here "pattern" means that they created a hand-made patterns for the attribute-value extraction, and "DL" means some sort of "Deep Learning". "DrQA" is an open source QA system adapted Japanese QA system. In this system, the participant transformed the infobox into a sentence by pattern, e.g. "The birthday of Barack Obama is August 4, 1961" and attribute-values to be extracted is transformed to a question, e.g. "Who is the father of Barack Obama?" in order to extract "father" attribute-value of "Barack Obama" entity. Then they train and run DrQA for all categories together. For RbCC purpose, it is quite valuable to have technologies of wide variety used in this shared-task. The are shown in 3rd to 7th columns in the same table. The top is shown in bold for each category. The Unisys's DrQA system performs the best in three categories, most of which don't have so much information in infobox. As their method handles all the attributes in a single system (regardless of infobox or in the explanation sentence), the amount of training data for the system becomes relatively larger and it may receive the benefit in training data size at the situation where the training data is relatively small. TUT's pattern based system performed very well on airport category, in which the most of the required information are described in infobox, and practically only one infobox template is used in the category. Note that the category "person" has many different infobox templates depending on the vocation of the person, and the company's infobox templates vary depending on the type of the company. The goal of "Resource by Collaborative Contribution (RbCC)" is to produce more accurate KB than the KB created by the best single system. In order to see if RbCC scheme is practical and promising, we conducted a preliminary experiment of ensemble learning on the all system's outputs. Note that the ensemble learning is conducted in order to show an evidence if RbCC scheme is practical and promising. In the past, the ensemble learning methods have been studied with various ideas; such as Bagging BID5, Boosting BID2 or stacking BID15, BID1, BID11 ]. These methods are generally used to create a high accuracy system combining more than one ML systems. However, in our situation, the outputs of many systems are given and the objective is to produce the best output out of the system outputs by ensemble learning method. Because of this, the stacking method is best suitable for our purpose, but, first, we tried two simpler methods, i.e. the simple voting method and the weighted voting method base on the accuracy on held-out data as a preliminary ensemble learning experiment. First, we will explain the simple voting method. Assume there are n systems which outputs value v for an attribute of an entity. Then the value v receives score n. Separately, we compute the threshold t by maximizing the accuracy on the held-out data. If n > t, then we take the value v as the output of the ensemble system. In practice, we split the actual test data, which contains 100 samples, into two halves; one for the held-out data and the other for the test data for this experiment. We conducted the same experiment replacing the held-out data and test data; i.e. we conducted a 2 fold cross-validation experiment on 100 test data. The other method is the weighted ensemble method. We weighted the vote of the system by the accuracy of the system on the held-out data. Instead of a sum of the number of the systems which produce attribute-value v, we compute the sum of the accuracy as the score for the value v. The way to define the threshold and the cross-validation mechanism are the same to those of the simple voting method. We will show the precision, recall and F1 score of the baseline and the ensemble methods in Table 8. Also the relative improvement of those two methods compared to the baseline method is shown in FIG0. The baseline method is constructed by combining the best system outputs for each category, i.e. TUT system for "airport", AIP system for "city" and Unisys system for the rest, which is better than a single system, e.g. Unisys, though. We can observe from the table and the graph that the two voting methods performs better than the baseline methods in F1 score. Also the weighted voting method performs better than the simple voting method. The improvement exceed 15 F-score on "airport" category and 4 F-score for all categories. The average improvement is 10.4 F-score. The show the effectiveness of the ensemble learning methods and are the evidence that "Resource by Collaborative Contribution" scheme is promising and encouraging. Based on the success of the SHINRA2018 project, we decided to continue this project as SHINRA2019. We are planning to conduct three tasks as follows:• ML: Multi-lingual categorization task The multi-lingual task is to expand the benefit of RbCC to the knowledge base resources in languages other than Japanese. We are planning to run it on 9 languages with the largest numbers of "active users"; namely English, Spanish, French, German, Chinese, Russian, Portuguese, Italian and Arabic BID14. Actually, Japanese is the 10th ranked language on the measure, so Wikipedias of these 9 language have more users than that of Japanese. As we don't have the category information for those 9 language Wikipedia entities, the first task is to categorize the entities. For Japanese, we annotated 20K entities as the training data for the categorization, but now we have most of the Japanese entities categorized, we can utilize this information. There are links between equivalent Wikipedia entities in different languages. For example, we observed there are about 500K entities links from Japanese to English among 720K entities already categorized in Japanese. We can use them as the training data to categorize English entities. Likewise there are language links to other 8 language Wikipedias from Japanese Wikipedia, although the number of linked entities are much smaller and some noise may exist, the participants can use much bigger training data than that for the initial Japanese categorization experiment. As there are links between the Wikipedias of other languages and possibly different types of infoboxes exist in other language, too, the participants have a lot of information to be used in the categorization task. JP-5 is the task to extract the attribute-values for the same 5 categories in SHINRA2018; namely "person", "company", "city", "airport" and "chemical compound". At SHINRA2018, the values are prepared without contexts. In other words even there are more than one mention of a particular attribute, we didn't give out which one is the mention to that value. For example, assume the nationality for a person is "Japan", but the same string may be mentioned in the same person page but not necessarily be meant to indicate the nationality of the person, e.g. "He left Japan", we had no means to know that the context is not for the nationality. It is similar to the situation of the distant-supervision, so it is difficult to extract only the context of nationality. At SHINRA2019, we will annotate the attributevalue in the text, so that the exact context for the value can be extracted. We are also planning to expand the size of the training data from 600 to 1500, at least for the categories "person", "company" and "city" using the output of the ensemble system. This forms a bootstrapping scheme as the project year by year. JP-30 is the task to extract attribute-value for 30 new categories. As we mentioned in the previous section, creating the data is laborious, the size of the training data will be very small, namely 100. However the categories to be tested will be very close; 7 subcategories of Geographical Political Entities (GPE) such as country, prefecture/state and county, 8 subcategories of terrain such as mountain, island, river, lake and ocean, and organizational entities such as international organization, political organization, ethnic group and nationality. Although the number of training data is much smaller, we chose the very similar types and the similar attributes may exists. Some techniques of machine learning with adaptation might help creating a good . We expect to build a larger training data using bootstrapping scheme, just like JP-5 at SHINRA2018 and SHINRA2019.We hope to have many participants so that the better can be achieved by the ensemble learning methods to all three tasks. We proposed a scheme of knowledge base creation: "Resource by Collaborative Contribution". We conducted the Japanese Wikipedia structuring project, SHINRA2018, based on that scheme. Based on Extended Named Entity, the top-down definition of categories and attributed for named entities, the task is to extract the attribute-values from Japanese Wikipedia pages. 8 groups participated to the task, and the ensemble learning shows that the RbCC scheme is practical and promising. A quite big improvement over the the best single system was achieved on "airport" category (more than 15 F-score), and the average of 8 F-score improvement was achieved using the weighted voting methods. We are planning to conduct SHINRA2019 based on the RbCC scheme on 3 tasks. These are the multi-lingual categorization, the extraction of attribute-value on the same 5 categories, and the extraction of attribute-values on 30 new categories in Japanese. We'd like to express our deep appreciation to all the participants and collaborators who helped this project. Without the participation, we couldn't even try the ensemble learning and achieve the goal. We are hoping to expand and spread the idea of RbCC scheme, not only limited to this kind of task and resource.

We introduce a "Resource by Collaborative Construction" scheme to create KB, structured Wikipedia

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

The choice of the hub (target) language affects the quality of cross-lingual embeddings, which shouldn't be evaluated only  on English-centric dictionaries.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recent image super-resolution(SR) studies leverage very deep convolutional neural networks and the rich hierarchical features they offered, which leads to better reconstruction performance than conventional methods. However, the small receptive fields in the up-sampling and reconstruction process of those models stop them to take full advantage of global contextual information. This causes problems for further performance improvement. In this paper, inspired by image reconstruction principles of human visual system, we propose an image super-resolution global reasoning network (SRGRN) to effectively learn the correlations between different regions of an image, through global reasoning. Specifically, we propose global reasoning up-sampling module (GRUM) and global reasoning reconstruction block (GRRB). They construct a graph model to perform relation reasoning on regions of low resolution (LR) images. They aim to reason the interactions between different regions in the up-sampling and reconstruction process and thus leverage more contextual information to generate accurate details. Our proposed SRGRN are more robust and can handle low resolution images that are corrupted by multiple types of degradation. Extensive experiments on different benchmark data-sets show that our model outperforms other state-of-the-art methods. Also our model is lightweight and consumes less computing power, which makes it very suitable for real life deployment. Image Super-Resolution (SR) aims to reconstruct an accurate high-resolution (HR) image given its low-resolution (LR) counterpart. It is a typical ill-posed problem, since the LR to HR mapping is highly uncertain. In order to solve this problem, a large number of methods have been proposed, including interpolation-based (Zhang & Wu., 2006), reconstruction-based , and learning-based methods (; Peleg & Elad., 2014; ; ; ; ; a;). In recent years, deep learning based methods have achieved outstanding performance in superresolution reconstruction. Some effective residual or dense blocks b;;; Ahn et al.; ) have been proposed to make the network wider and deeper and achieved better . However, they only pay close attention to improving the feature extraction module, ignoring that the upsampling process with smaller receptive fields does not make full use of those extracted features. Small convolution receptive field means that the upsampling process can only perform super-resolution reconstruction based on local feature relationships in LR. As we all know, different features interact with each other, and features which are in different regions have corresponding effects on upsampling and reconstruction of a certain region. That is to say that a lot of information is lost in the process of upsampling and reconstruction due to the limitation of the receptive field, although the network extracts a large number of hierarchical features which are from low frequency to high frequency. Chariker et al. (2016; show that the brain generates the images we see based on a small amount of information observed by the human eye, ranther than acquiring the complete data from the point-by-point scan of the retina. This process of generating an image is similar to a SR process. According to their thought, we add global information in SR reconstruction and propose to use relational reasoning to implement the process that the human visual system reconstructs images with observed global information. In general, extracting global information requires a large receptive field. A large convolution receptive field usually requires stacking a large number of convolutional layers, but this method does not work in the upsampling and reconstruction process. Because this will produce a huge number of parameters. Based on the above analysis, we propose an image super-resolution global reasoning network (SR-GRN) which introduces the global reasoning mechanism to the upsampling module and the reconstruction layer. The model can capture the relationship between disjoint features of the image with a small respective field, thereby fully exploits global information as a reference for upsampling and reconstruction. We mainly propose global reasoning upsampling module (GRUM) and global reasoning reconstruction block (GRRB) as the core structure of the network. GRUM and GRRB first convert the LR feature map into N nodes, each of which not only represents a feature region in the LR image, but also contains the influence of pixels in other regions on this feature. Then they learn the relationship between the nodes and fuse the information of each node in a global scope. After that, GRUM learns the relationship between the channels in each node and amplifies the number of channels for the upsampling process. And then they convert N nodes into pixels with global reasoning information. Finally, GRUM and GRRB complete the upsampling and reconstruction process respectively. In general, our work mainly has the following three contributions: • We propose an image super-resolution global reasoning network (SRGRN) which draws on the idea of image reconstruction principles of human visual system. We mainly focus on the upsampling module and the reconstruction module. The model reconstructs SR images based on relational reasoning in a global scope. • We propose a global reasoning upsampling module (GRUM) and global reasoning reconstruction block (GRRB), which construct a graph model to implement the relational reasoning among the feature regions in an image via 1D and 2D convolution, and finally adds the information obtained by global reasoning to each pixel. It can provide more contextual information to help generate more accurate details. • Our proposed GRUM and GRRB are lightweight, which makes it suitable for real life deployment. More importantly, GRUM and GRRB balance the number of parameters and the reconstruction performance well. They can be easily inserted into other models. Deep CNN for SR and upsampling methods. Deep learning has achieved excellent performance in image super-resolution tasks. For the first time, Dong et al. applied convolutional neural networks to image SR. After this, Kim et al. proposed VDSR and DRCN which introduced residual learning to make the network depth reach 20 layers achieved significant improvement. And then more and more researchers are starting to pay attention to the improvement of the network feature extraction part. proposed EDSR and MDSR, which introduce residual scaling and remove unnecessary modules from the residual block. Concerned that the previous models only adopt the feather of the last CNN, Zhang et al. (2018b) proposed residual dense network to make full use of hierarchical features from each Conv layer. The above and most of the subsequent networks implement the upsampling based on either transposed convolution (; Zeiler & Fergus., 2014) or sub-pixel convolution . Although these models have achieved good , there exists a problem that these upsampling methods have only a small receptive field. This means that upsampling can only take advantage of contextual information within a small area. Recently, researchers propose some new super-resolution upsampling process. LapSRN allows low-resolution images to be directly input into the network for step-by-step amplification. exploit iterative up-and-down sampling layers and propose DBPN. further explore the application of feedback mechanism (weight sharing) in SR and propose the SRFBN. These models have achieved a better reconstruction performance. However, the Conv layers in these upsampling modules still have only a small receptive field. Global reasoning machansim. Recently, graph-based deep learning methods have begun to be widely used to solve relation reasoning. Santoro et al. propose Relation Networks (RN) to solve problems that depend on relational reasoning. propose SIN, which implement a object detection using a graph model for structure inference. Furthermore, model a Global Reasoning unit that consists of five convolutions for image classification, semantic segmentation and video action recognition task. Considering that the human visual system generates images based on the observed global information is also a reasoning process. Moreover, correlation between feature regions can be obtained through relational reasoning, which makes each pixel in the generated SR image jointly determined by the information in a global scope. Therefore, we propose a global reasoning network for SR. We will detail our SRGRN in next section. According to Chariker et al. (2016;, there is only little information transmitted from the retina to the visual cortex, and then the brain will reconstruct the real-world images based on the information received. We regard it as a reasoning process in a global scope. For image SR, the upsampling module constructs SR images base on features in LR images, which is substantially similar to detecting the category of each pixel of a SR image and generating these pixels based on contextual information of corresponding LR image. Due to the limitation of the convolution receptive field, only a small amount of contextual information can be utilized to generate HR images in most other models. This leads that many details in the HR image are not fine. Similarly, the above problem also exists in the reconstruction process. To solve these problems, we simulate the reasoning process that exists in human visual system, and then propose SRGRN to make full use of the contextual information to recover accurate details, which is achieved by constructing graph model and reasoning the relationship between these regions in an image. Figure 1, our SRGRN includes feature extraction part, global reasoning upsample module(GRUM) and global reasoning reconstruction block(GRRB). Let's denote I LR and I SR as the input and output of SRGRN. The feature extraction part can use the relevant architecture of most other models. Here we introduce the feature extraction part of the RDN (b) as an example. where H F EX (·) denotes a series of operations of feature extraction part. As with the previous work , the number of GRUM depends on the scaling factor, The GRUM receives F L as input. F S represents the output of the GRUM. GRUM can be expressed by the following mathematical formula: where H GRU M (·)denotes a series of operations of GRUM. More details about GRUM will be given in Section 3.2. We further conduct global reasoning reconstruction block(GRRB) to utilize the global contextual information to generate the output image. GRRB can be expressed by the following mathematical formula: where H GRRB (·) denotes a series of operations of GRRB. More details about GRRB will be given in Section 3.3. After the above operations, we get the corresponding SR image. In this section, we present details about our proposed global reasoning upsample module(GRUM) in Figure 2. In order to help achieve relation reasoning, we map each image to a graph model We first need a function to construct N nodes in the oriented graph, each of which represents a region in the image. In GRUM, we obtain relationship weights between these pixels through a 2D 1 × 1 convolution, and then convert the input F L into N nodes via element-wise product. The benefits of this approach are mainly reflected in the following aspects: It can not only aggregates a feature region of the input F L into a node, but also dig out the influence of other pixels in the image on this region. This is equivalent to adding global guidance of the image to each node. Using convolution means that these relationship weights are trainable. This process can be expressed by the following mathematical formula: refers to N nodes with C channels. After that, we use the 1D Conv -Leaky ReLu -1D Conv (CLC) structure to implement reasoning and interaction between N nodes in the graph. The parameters in CLC refer to the adjacency matrix of the weighted oriented complete graph, which store the correlations between the nodes. CLC can learn and reason the complex nonlinear relationship between nodes better than only one 1D Conv. We use the following formula to describe the reasoning process between nodes: where Conv(·) and LRelu(·) denote 1D convolution along node-wise and Leaky ReLU operation respectively. And then we use the bottleneck to achieve channel amplification. The bottleneck receives Y N ∈ R N ×C as input and redistributes these channels by modeling the relationship between the channels of each node, amplifying the number of channel to C × r 2, where r is the upscaling factor. The first convolution in bottleneck makes channel C drop toC = C/α, where α represents reduction ratio. Then the second convolution makes the channel dimensionC grow to C × r 2. The bottleneck not only fits the complex relationships between channels better and redistributes channels more accurately, but also greatly reduces the number of parameters compared to the method of utilizing a single convolution. We use the following formula to describe channel amplification: where Y N C ∈ R r 2 C×N refers to the output tensor. In order to expand the resolution by pixelshuffle like ESPCN , we need to retransform the N nodes (r 2 C × N) which have implemented the relational reasoning into a space whose shape is C × H × W. As above, we still learn a function to get a weight matrix whose shape is N ×HW through a 1 × 1 2D convolution, and then normalize the weight matrix along the column with softmax. Finally, Y rC and W P can be obtained through: where C×H×W is a feature map where each pixel is associated with N nodes. W P ∈ R N ×HW is the normalized weight matrix. The value of these weights ranges from 0 to 1. This means that the reconstruction of each pixel is affected by N nodes to varying degrees. Each pixel in the feature map contains information which is generated by global reasoning. After pixelshuffle, the output is multiplied by a parameter γ 1 and added to the upsampling without global reasoning. The initial value of γ 1 is set to 0. As the global reasoning module trains, the network will gradually learn to assign values to the γ 1, thereby fully exploiting global reasoning. This process can be expressed by the following formula: where H P S (·) denotes the operations of pixel shuffle and H U P (·) denotes the operations of subpixel convolution. Finally, F S can be obtained by: Figure 3: Global reasoning reconstruction block (GRRB) architecture As shown in Figure 3, the specific details are similar to GRUM. We also construct a graph model for reconstruction block. In GRRB, we first obtain the relationship weights W RN ∈ R N ×rHrW between pixels of F S by 2D 1x1 convolution, and then aggregate the regions in F S into N nodes by element-wise product. The output of this process Y RW ∈ R N ×C can be formulated as: After that, we use CLC to achieve the relationship reasoning between nodes. Then we exploit the weight matrix W RP ∈ R N ×rHrW = Sof tmax(V d (Conv(F S))) obtained by learning to redistribute the information of N nodes to the pixels. The output of this process Y RrC ∈ R C×rH×rW can be obtained by: where F CLC refers to the operations of CLC. In addition, we apply the idea of residual connection in GRRB, which multiplys the information generated via global reasoning by a parameter γ and then add it to the input feature map. The output is given by: The initial value of γ is set to 0. As the training progresses, the network assigns more weight to γ. Finally, we input the feature map with global reasoning into the two Convs for reconstruction. We can get the final output through: where H RL (·) denotes the operations of two Convs. In our proposed SRGRN, like the previous method , the number of GRUM depends on the scaling factor. For Conv layers with kernel size 3 × 3, we pad zeros to keep size fixed. We set the reduction ratio in bottleneck as α. The number of nodes in the graph model is set to N. We utilize Leaky ReLu with a negative slope of 0.2 as non-linear activation function. The feature extraction part of the network are the same as the RDN (b) settings. The final Conv layer has 1 or 3 output channels, as we output gray or color HR images. 4.1 SETTINGS Datasets and Metrics. We train all our models using 800 training images in the DIV2K (Agustsson & Timofte., 2017) dataset, which contains high-quality 2K images that can be used for image superresolution task. And We use five standard benchmark datasets to evaluate PSNR and SSIM metrics: Set5 , Set14 , B100 , Urban100 and Manga109 (Y.). The SR are evaluated on Y channel of transformed YCbCr space. Degradation Models. In order to make a fair comparison with existing models, bicubic dowmsampling(denoted as BI) is regarded as a standard degradation model. We use it to generate LR images with scaling factor ×2, ×3, and ×4 from ground truth HR images. To fully demonstrate the effectiveness of our model, we also use two other degradation models and conduct special experiments for them. Our second model, we defined it as BD, which blurs HR images with a Gaussian kernel of size 7 × 7 and a standard deviation of 1.6, and then downsamples the image with scaling factor ×3. In addition to BI and BD, we also built the DN model, which first performs bicubic downsampling with scaling factor ×3 and then adds Gaussian noise with a noise level of 30. Training Setting. In each training batch, 16 LR RGB patches of size 48 × 48 are extracted as inputs. We perform data enhancement on the training images, which are randomly rotated by 90 •, 180 •, 270 • and flipped horizontally. We use the Adam optimizer to update the parameters of the network with β 1 = 0.9, β 2 = 0.999, and ϵ = 10 −8. For all layers in the network, the initial learning rate is set to 0.0001, and then the learning rate is halved every 200 epochs. We use the Pytorch framework to implement our model with Tesla P100. Global reasoning upsampling module. In order to verify the importance of the GRUM, we remove the GRUM from the network, leaving only the GRRB in the network for relation reasoning. As shown in Table 1, after removing GRUM, the performance of the network drops from 32.45 dB to 32.40 dB. When the Case Index is equal to 1, the corresponding model is the baseline model. We can observe that after GRUM is added to the baseline model, the network performance is improved from 32.31 dB to 32.42 dB. It can be seen that although our baseline model has achieved quite good , GRUM can still improve the performance by relation reasoning in upsample module. This also indicates that relation reasoning can indeed in better prformance. These comparisons fairly demonstrate the effectiveness of the GRUM for SR tasks. Global reasoning reconstruction block. Then, we continue to study the effectiveness of GRRB for the network. After we add the GRRB to the baseline model, GRRB improves the performance of the model from 32.31 dB to 32.40 dB. Furthermore, the model with GRUM has achieved good performance. And it is difficult to obtain further improvements. But when we add the GRRB to it, the network performance shows a significant improvement, and the PSNR value on Set5 increases from 32.42 dB to 32.45 dB. These indicates that it is very essential for our network. Basic parameters. Moreover, we also study the effects of two basic parameters N and α on the performance of the model. As shown in Table 1, we observe that larger N and smaller α would lead to higher performance. Considering that larger N and smaller α will also bring more computation, we set 10 and 8 as the value of N and α respectively. Figure 4. Although our SRGRN has less parameter number than that of EDSR, MDSR and D-DBPN, our SRGRN and SRGRN+ achieve higher performance, having a better tradeoff between model size and performance. This demonstrates our method can well balance the number of parameters and the reconstruction performance. For BI degradation model, we compare our proposed SRGRN and SRGRN+ with other seven stateof-the-art image SR methods in quantitative terms. Following the previous works (; b;), we also introduced a self-ensemble strategy to further improve the performance. We denote the self-ensemble method as SRGRN+. A quantitative for ×2, ×3, and ×4 is shown in Table 2. We compare our models with other state-of-the-art methods on PSNR and SSIM. It can be seen that our proposed SRGRN outperforms other methods on all datasets without adding self-ensemble. After adopting self-ensemble, the performance further improves on the basis of SRGRN, and it achieved the best on all datasets. It is worth mentioning that SRFBN uses DIV2K+Flickr2K as their training set, which employs more training images than us. Previous research has come to a that more data in training set leads to a better . However, their are still not comparable to ours. Although RDN (b) is a state-of-the-art method, our SRGRN can achieve better performance in all datasets through relational reasoning in upsampling and reconstruction parts. The quantitative indicate that our GRUM and GRRB play a vital role in improving network performance. (Dong et al.) and VDSR for BD and DN degradation model because of mismatched degradation model. For BD and DN, there is no doubt that reconstruction has become more difficult. As shown in Table 3 and Table 4, in the case of images with a lot of artifacts and noise, our SRGRN can get a excellent performance. This shows that SRGRN can effectively denoise and alleviate blurring artifacts. And when added to self-ensemble, SRGRN+ can achieve a better improvement. To prove that our SRGRN can be widely used in the real world and performs robustly, we also conduct SR experiments on representative real-world images. We reconstruct some low resolution images in the real world that lack a lot of high frequency information. Moreover, in this case, the original HR images are not available and the degradation model is unknown either. Experiments show our SRGRN can recover finer and more faithful real-world images than other state-of-the-art methods under this bad condition. This further reflects the superiority of relation reasoning. In this paper, inspired by the process of reconstructing images from the human visual system, we propose an super-resolution global reasoning network (SRGRN) for image SR, which aims at completing the reconstruction of SR images through global reasoning. We mainly propose global reasoning upsampling module (GRUM) and global reasoning reconstruction block (GRRB) as the core of the network. The GRUM can give the upsampling module the ability to perform relational reasoning in a global scope, which allows this process to overcome the limitations of the receptive field and recover more faithful details by analyzing more contextual information. The GRRB also enables the reconstruction block to make full use of the interaction between the regions and pixels to reconstruct SR images. We exploit SRGRN not only to handle low resolution images that are corrupted by three degradation model, but also to handle real-world images. Extensive benchmark evaluations demonstrate the importance of GRUM and GRRB. It also indicates that our SRGRN achieves superiority over state-of-the-art methods through global reasoning. Visual comparison with BI degradtion model. As shown in Figure 5, we show a visual comparison on 4× SR. For image "img_078" from Urban100, we observe that most methods, even RDN and SRFBN, cannot recover these lattices and suffer from extremely severe blurring artifacts. Only our SRGRN can alleviate these blurring artifacts, recovers sharper and clearer edges and finer texture. For image "MukoukizuNoChonbo" from Manga109, There are heavy blurrings artifacts in all comparison methods, and the outline of some letters are broken. However, our proposed SRGRN can accurately recover these outlines, more faithful to the ground truth. The above comparison are mainly due to the fact that SRGRN can enable upsampling and reconstruction modules to utilize more contextual information through relation reasoning. Visual comparison with BD and DN degradtion model. In Figure 6, we show the comparison of SRGRN with other models in visual . For image "img_014", we use bicubic upsampling to recover these images whose HR images are blurred with a Gaussian kernel before bicubic downsampling, then we obtain SR images with a lot of noticeable blurring artifacts. We have also observed that most methods, including RDN and SRFBN, do not clearly recover the lines around the window. Only our SRGRN can suppress blurring artifacts and recover these clear enough lines close to the ground truth by relation reasoning. For image "img_002", a large amount of noise corrupt the LR image and make it loss some detail. It can be seen that when using bicubic for upsampling, the obtained image not only has a large number of blurring artifacts but also a large amount of noise. However, we find that our SRGRN has great potential for removing noise efficiently and recover more detail. This fully demonstrates the effectiveness and robustness of our SRGRN for BD and DN degradation models. Visual comparison on Real-World Images. In figure 7, the resolution of these images is so small that there is a lot of high frequency information missing from them. Moreover, in this case, the original HR images are not available and the degradation model is unknown either. For image "window"(with 200 × 160 pixels), only our SRGRN is able to recover sharper window edges and produce clearer SR image. For image "flower"(with 256 × 200 pixels), most other methods recover images whose upper left corner produces the edge of the pistil that looks unreal. And their edges of the petals in the whole image are very blurry. Our SRGRN can recovers sharper edges and finer details than other state-of-the-art methods. The above analysis indicate our model perform robustly unknown degradation models. This further reflects the superiority of relation reasoning.

Represent the TLDR for finding the corresponding scientific article

A state-of-the-art model based on global reasoning for image super-resolution

represent this text to find a much much longer article about it with say hundreds-thousands of words.

In this paper, we explore an internal dense yet external sparse network structure of deep neural networks and analyze its key properties.