Question: Represent the scientific article to find a short TLDR

Question: Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art in domains such as language modeling and machine translation. Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially-observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting. In this work we demonstrate that the standard transformer architecture is difficult to optimize, which was previously observed in the supervised learning setting but becomes especially pronounced with RL objectives. We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XL variant. The proposed architecture, the Gated Transformer-XL (GTrXL), surpasses LSTMs on challenging memory environments and achieves state-of-the-art on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture. We show that the GTrXL, trained using the same losses, has stability and performance that consistently matches or exceeds a competitive LSTM baseline, including on more reactive tasks where memory is less critical. GTrXL offers an easy-to-train, simple-to-implement but substantially more expressive architectural alternative to the standard multi-layer LSTM ubiquitously used for RL agents in partially-observable environments. It has been argued that self-attention architectures deal better with longer temporal horizons than recurrent neural networks (RNNs): by construction, they avoid compressing the whole past into a fixed-size hidden state and they do not suffer from vanishing or exploding gradients in the same way as RNNs. Recent work has empirically validated these claims, demonstrating that self-attention architectures can provide significant gains in performance over the more traditional recurrent architectures such as the;;. In particular, the Transformer architecture has had breakthrough success in a wide variety of domains: language modeling;, machine translation , summarization (Liu & Lapata), question answering (;, multi-task representation learning for NLP (; ;, and algorithmic tasks . The repeated success of the transformer architecture in domains where sequential information processing is critical to performance makes it an ideal candidate for partially observable RL problems, where episodes can extend to thousands of steps and the critical observations for any decision often span the entire episode. Yet, the RL literature is dominated by the use of LSTMs as the main mechanism for providing memory to the agent; ). Despite progress at designing more expressive memory architectures; that perform better than LSTMs in memory-based tasks and partially-observable environments, they have not seen widespread adoption in RL agents perhaps due to their complex implementation, with the LSTM being seen as the go-to solution for environments where memory is required. In contrast to these other memory architectures, the transformer is well-tested in many challenging domains and has seen several open-source implementations in a variety of deep learning frameworks 1. Motivated by the transformer's superior performance over LSTMs and the widespread availability of implementations, in this work we investigate the transformer architecture in the RL setting. In particular, we find that the canonical transformer is significantly difficult to optimize, often ing in performance comparable to a random policy. This difficulty in training transformers exists in the supervised case as well. Typically a complex learning rate schedule is required (e.g., linear warmup or cosine decay) in order to train (;, or specialized weight initialization schemes are used to improve performance . These measures do not seem to be sufficient for RL. , for example, transformers could not solve even simple bandit tasks and tabular Markov Decision Processes (MDPs), leading the authors to hypothesize that the transformer architecture was not suitable for processing sequential information. However in this work we succeed in stabilizing training with a reordering of the layer normalization coupled with the addition of a new gating mechanism to key points in the submodules of the transformer. Our novel gated architecture, the Gated Transformer-XL (GTrXL) (shown in Figure 1, Right), is able to learn much faster and more reliably and exhibit significantly better final performance than the canonical transformer. We further demonstrate that the GTrXL achieves state-ofthe-art when compared to the external memory architecture MERLIN on the multitask DMLab-30 suite . Additionally, we surpass LSTMs significantly on memory-based DMLab-30 levels while matching performance on the reactive set, as well as significantly outperforming LSTMs on memory-based continuous control and navigation environments. We perform extensive ablations on the GTrXL in challenging environments with both continuous actions and high-dimensional observations, testing the final performance of the various components as well as the GTrXL's robustness to seed and hyperparameter sensitivity compared to LSTMs and the canonical transformer. We demonstrate a consistent superior performance while matching the stability of LSTMs, providing evidence that the GTrXL architecture can function as a drop-in replacement to the LSTM networks ubiquitously used in RL. The transformer network consists of several stacked blocks that repeatedly apply self-attention to the input sequence. The transformer layer block itself has remained relatively constant since its original introduction (; ;). Each layer consists of two submodules: an attention operation followed by a position-wise multi-layer network (see Figure 1 (left)). The input to the transformer block is an embedding from the previous layer E (l−1) ∈ R T ×D, where T is the number of time steps, D is the hidden dimension, and l ∈ [0, L] is the layer index with L being the total number of layers. We assume E is an arbitrarily-obtained input embedding of dimension [T, D], e.g. a word embedding in the case of language modeling or an embedding of the per-timestep observations in an RL environment. Multi-Head Attention: The Multi-Head Attention (MHA) submodule computes in parallel H softattention operations for every time step. A residual connection (a) and layer normalization are then applied to the output (see Appendix C for more details): Multi-Layer Perceptron: The Multi-Layer Perceptron (MLP) submodule applies a 1 × 1 temporal convolutional network f (l) (i.e., kernel size 1, stride 1) over every step in the sequence, producing a new embedding tensor E (l) ∈ R T ×D. As in, the network output does not include an activation function. After the MLP, there is a residual update followed by layer normalization: Relative Position Encodings: The basic MHA operation does not take sequence order into account explicitly because it is permutation invariant. Positional encodings are a widely used solution in Canonical Transformer(-XL) block with multi-head attention and position-wise MLP submodules and the standard layer normalization placement with respect to the residual connection (a). Center: TrXL-I moves the layer normalization to the input stream of the submodules. Coupled with the residual connections, there is a gradient path that flows from output to input without any transformations. Right: The GTrXL block, which additionally adds a gating layer in place of the residual connection of the TrXL-I. domains like language where order is an important semantic cue, appearing in the original transformer architecture . To enable a much larger contextual horizon than would otherwise be possible, we use the relative position encodings and memory scheme used in. In this setting, there is an additional T -step memory tensor M (l) ∈ R T ×D, which is treated as constant during weight updates. The MHA submodule then becomes: where StopGrad is a stop-gradient function that prevents gradients flowing backwards during backpropagation. We refer to Appendix C for a more detailed description. While the transformer architecture has achieved breakthrough in modeling sequences for supervised learning tasks (; ;, a demonstration of the transformer as a useful RL memory has been notably absent. Previous work has highlighted training difficulties and poor performance . When transformers have not been used for temporal memory but instead as a mechanism for attention over the input space, they have had success-notably in the challenging multi-agent Starcraft 2 environment. Here, the transformer was applied solely across Starcraft units and not over time. Multiplicative interactions have been successful at stabilizing learning across a wide variety of architectures (; ; . Motivated by this, we propose the introduction of powerful gating mechanisms in place of the residual connections within the transformer block, coupled with changes to the order of layer normalization in the submodules. As will be empirically demonstrated, the "Identity Map Reordering" and gating mechanisms are critical for stabilizing learning and improving performance. Our first change is to place the layer normalization on only the input stream of the submodules, a modification described in several previous works (b; ;). The model using this Identity Map Reordering is termed TrXL-I in the following, and is depicted visually in Figure 1 (center). A key benefit to this reordering is that it now enables an identity map from the input of the transformer at the first layer to the output of the transformer after the last layer. This is in contrast to the canonical transformer, where there are a series of layer normalization operations that non-linearly transform the state encoding. Because the layer norm reordering causes a path where two linear layers are applied in sequence, we apply a ReLU activation to each sub-module output before the residual connection (see Appendix C for equations). The TrXL-I already exhibits a large improvement in stability and performance over TrXL (see Section 4.3.1). One hypothesis as to why the Identity Map Reordering improves is as follows: assuming that the submodules at initialization produce values that are in expectation near zero, the state encoding is passed un-transformed to the policy and value heads, enabling the agent to learn a Markovian policy at the start of training (i.e., the network is initialized such that π(·|s t, . . ., s 1) ≈ π(·|s t) and V π (s t |s t−1, . . ., s 1) ≈ V π (s t |s t−1)). In many environments, reactive behaviours need to be learned before memory-based ones can be effectively utilized, i.e., an agent needs to learn how to walk before it can learn how to remember where it has walked. We further improve performance and optimization stability by replacing the residual connections in Equations 4 and 2 with gating layers. We call the gated architecture with the identity map reordering the Gated Transformer(-XL) (GTrXL). The final GTrXL layer block is written below: where g is a gating layer function. A visualization of our final architecture is shown in Figure 1 (right), with the modifications from the canonical transformer highlighted in red. In our experiments we ablate a variety of gating layers with increasing expressivity: Input: The gated input connection has a sigmoid modulation on the input stream, similar to the short-cut-only gating from He et al. (2016b): The gated output connection has a sigmoid modulation on the output stream: g ) y Highway: The highway connection modulates both streams with a sigmoid: Sigmoid-Tanh: The sigmoid-tanh (SigTanh) gate (Van den) is similar to the Output gate but with an additional tanh activation on the output stream: g y) Gated-Recurrent-Unit-type gating: The Gated Recurrent Unit (GRU) is a recurrent network that performs similarly to an LSTM but has fewer parameters. We adapt its powerful gating mechanism as an untied activation function in depth: We have claimed that the Identity Map Reordering aids policy optimization because it initializes the agent close to a Markovian policy / value function. If this is indeed the cause of improved stability, we can explicitly initialize the various gating mechanisms to be close to the identity map. This is the purpose of the bias b g in the applicable gating layers. We later demonstrate in an ablation that initially setting b In this section, we provide experiments on a variety of challenging single and multi-task RL domains: DMLab-30 , Numpad and Memory Maze (see Fig. 8). Crucially we demonstrate that the proposed Gated Transformer-XL (GTrXL) not only shows substantial improvements over LSTMs on memory-based environments, but suffers no degradation of performance on reactive environments. The GTrXL also exceeds MERLIN , an external memory architecture which used a Differentiable Neural Computer coupled with auxiliary losses, surpassing its performance on both memory and reactive tasks. For all transformer architectures except when otherwise stated, we train relatively deep 12-layer networks with embedding size 256 and memory size 512. These networks are comparable to the state-of-the-art networks in use for small language modeling datasets (see enwik8 in). We chose to train deep networks in order to demonstrate that our do not necessarily sacrifice complexity for stability, i.e. we are not making transformers stable for RL simply by making them shallow. Our networks have receptive fields that can potentially span any episode in the environments tested, with an upper bound on the receptive field of 6144 (12 layers × 512 memory). Future work will look at scaling transformers in RL even further, e.g. towards the 52-layer network in. See App. B for experimental details. For all experiments, we used V-MPO , an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) (a; b) that performs approximate policy iteration based on a learned state-value function V (s) instead of the state-action value function used in MPO. Rather than directly updating the parameters in the direction of the policy gradient, V-MPO uses the estimated advantages to first construct a target distribution for the policy update subject to a sample-based KL constraint, then calculates the gradient that partially moves the parameters toward that target, again subject to a KL constraint. V-MPO was shown to achieve state-of-the-art for LSTM-based agents on the multi-task DMLab-30 benchmark suite. We first present of the best performing GTrXL variant, the GRU-type gating, against a competitive LSTM baseline, demonstrating a substantial improvement on the multi-task DMLab-30 do- 75.2 ± 10.4 GTrXL (SigTanh) 101.0 ± 1.3 83.9 ± 0.7 Table 1: Final human-normalized return averaged across all 30 DMLab levels for baselines and GTrXL variants. We also include the 100-capped score where the per-level mean score is clipped at 100, providing a metric that is proportional to the percentage of levels that the agent is superhuman. We see that the GTrXL (GRU) surpasses LSTM by a significant gap and exceeds the performance of MERLIN trained for 100 billion environment steps. The GTrXL (Output) and the proposed reordered TrXL-I also surpass LSTM but perform slightly worse than MERLIN and are not as robust as GTrXL (GRU) (see Sec. 4.3.2). We sample 6-8 hyperparameters per model. We include standard error over runs. Figure 3: Numpad demonstrating that the GTrXL has much better memory scaling properties than LSTM. Left: As the Numpad environment's memory requirement increases (because of larger pad size), the GTrXL suffers much less than LSTM. However, because of the combinatorial nature of Numpad, the GTrXL eventually also starts dropping in performance at 4x4. We plot mean and standard error of the last 200 episodes after training each model for 0.15B, 1.0B and 2.0B environment steps for Numpad size 2, 3 and 4, respectively. Center, Right: Learning curves for the GTrXL on 2 × 2 and 4 × 4 Numpad. Even when the LSTM is trained for twice as long, the GTrXL still has a substantial improvement over it. We plot 5 hyperparameter settings per model for learning curves. main . DMLab-30 is a large-scale, multitask benchmark comprising 30 firstperson 3D environments with image observations and has been widely used as a benchmark for architectural and algorithmic improvements (; ;). The levels test a wide agent competencies such as language comprehension, navigation, handling of partial observability, memory, planning, and other forms of long horizon reasoning, with episodes lasting over 4000 environment steps. Figure 2 shows mean return over all levels as training progresses, where the return is human normalized as done in previous work (meaning a human has a per-level mean score of 100 and a random policy has a score of 0), while Table 1 has the final performance at 10 billion environment steps. The GTrXL has a significant gap over a 3-layer LSTM baseline trained using the same V-MPO algorithm. Furthermore, we included the final of a previously-published external memory architecture, MERLIN . Because MERLIN was trained for 100 billion environment steps with a different algorithm, IMPALA, and also involved an auxiliary loss critical for the memory component to function, the learning curves are not directly comparable and we only report the final performance of the architecture as a dotted line. Despite the differences, our demonstrate that the GTrXL can match the previous state-of-the-art on DMLab-30. An informative split between a set of memory-based levels and more reactive ones (listed in Appendix D) reveals that our model specifically has large improvements in environments where memory plays a critical role. Meanwhile, GTrXL also shows improvement over LSTMs on the set of reactive levels, as memory can still be effectively utilized in some of these levels. Figure 4: Learning curves for the gating mechanisms, along with MERLIN score at 100 billion frames as a reference point. We can see that the GRU performs as well as any other gating mechanism on the reactive set of tasks. On the memory environments, the GRU gating has a significant gain in learning speed and attains the highest final performance at the fastest rate. We plot both mean (bold) and the individual 6-8 hyperparameter samples per model (light). Figure 5: Sensitivity analysis of GTrXL variants versus TrXL and LSTM baselines. We sample 25 different hyperparameter sets and seeds and plot the ranked average return at 3 points during training (0.5B, 1.0B and 2.0B environment steps). Higher and flatter lines indicate more robust architectures. The same hyperparameter sampling distributions were used across models (see Appendix B). We plot human performance as a dotted line. We next demonstrate that the GTrXL scales better compared to an LSTM when an environment's temporal horizon is increased, using the "Numpad" continuous control task of which allows an easy combinatorial increase in the temporal horizon. In Numpad, a robotic agent is situated on a platform resembling the 3x3 number pad of a telephone (generalizable to N × N pads). The agent can interact with the pads by colliding with them, causing them to be activated (visualized in the environment state as the number pad glowing). The goal of the agent is to activate a specific sequence of up to N 2 numbers, but without knowing this sequence a priori. The only feedback the agent gets is by activating numbers: if the pad is the next one in the sequence, the agent gains a reward of +1, otherwise all activated pads are cleared and the agent must restart the sequence. Each correct number in the sequence only provides reward once, i.e. each subsequent activation of that number will no longer provide rewards. Therefore the agent must explicitly develop a search strategy to determine the correct pad sequence. Once the agent completes the full sequence, all pads are reset and the agent gets a chance to repeat the sequence again for more reward. This means higher reward directly translates into how well the pad sequence has been memorized. An image of the scenario is provided in Figure 3. There is the restriction that contiguous pads in the sequence must be contiguous in space, i.e. the next pad in the sequence can only be in the Moore neighborhood of the previous pad. Furthermore, no pad can be pressed twice in the sequence. We present two in this environment in Figure 3. The first measures the final performance of the trained models as a function of the pad size. We can see that LSTM performs badly on all 3 pad sizes, and performs worse as the pad size increases from 2 to 4. The GTrXL performs much better, and almost instantly solves the environment with its much more expressive memory. On the center Figure 6: Learning curves comparing a thinner GTrXL (GRU) with half the embedding dimension of the other presented gated variants and TrXL baselines. The Thin GTrXL (GRU) has fewer parameters than any other model presented but still matches the performance of the best performing counterpart, the GTrXL (Output), which has over 10 million more parameters. We plot both mean (bold) and 6-8 hyperparameter settings (light) per model. and right images, we provide learning curves for the 2 × 2 and 4 × 4 Numpad environments, and show that even when the LSTM is trained twice as long it does not reach GTrXL's performance. We demonstrated that the GRU-type-gated GTrXL can achieve state-of-the-art on DMLab-30, surpassing both a deep LSTM and an external memory architecture, and also that the GTrXL has a memory which scales better with the memory horizon of the environment. However, the question remains whether the expressive gating mechanisms of the GRU could be replaced by simpler alternatives. In this section, we perform extensive ablations on the gating variants described in Section 3.2, and show that the GTrXL (GRU) has improvements in learning speed, final performance and optimization stability over all other models, even when controlling for the number of parameters. We first report the performance of the gating variants in DMLab-30. Table 1 and Figure 4 show the final performance and training curves of the various gating types in both the memory / reactive split, respectively. The canonical TrXL completely fails to learn, while the TrXL-I improves over the LSTM. Of the gating varieties, the GTrXL (Output) can recover a large amount of the performance of the GTrXL (GRU), especially in the reactive set, but as shown in Sec. 4.3.2 is generally far less stable. The GTrXL (Input) performs worse than even the TrXL-I, reinforcing the identity map path hypothesis. Finally, the GTrXL (Highway) and GTrXL (SigTanh) are more sensitive to the hyperparameter settings compared to the alternatives, with some settings doing worse than TrXL-I. Beyond improved performance, we next demonstrate a significant reduction in hyperparameter and seed sensitivity for the GTrXL (GRU) compared to baselines and other GTrXL variants. We use the "Memory Maze" environment, a memory-based navigation task in which the agent must discover the location of an apple randomly placed in a maze of blocks. The agent receives a positive reward for collecting the apple and is then teleported to a random location in the maze, with the apple's position held fixed. The agent can make use of landmarks situated around the room to return as quickly as possible to the apple for subsequent rewards. Therefore, an effective mapping of the environment in more frequent returns to the apple and higher reward. We chose to perform the sensitivity ablation on Memory Maze because it requires the use of longrange memory to be effective and it includes both continuous and discrete action sets (details in Appendix A) which makes optimization more difficult. In Figure 5, we sample 25 independent V-MPO hyperparameter settings from a wide range of values and train the networks to 2 billion environment steps (see Appendix B). Then, at various points in training (0.5B, 1.0B and 2.0B), we rank all runs by their mean return and plot this ranking. Models with curves which are both higher and flatter are thus more robust to hyperparameters and random seeds. Our demonstrate that the GTrXL (GRU) can learn this challenging memory environment in much fewer environment steps than LSTM, and that GTrXL (GRU) beats the other gating variants in stability by a large margin, thereby offering a substantial reduction in necessary hyperparameter tuning. The values in Table 3 list what percentage of the 25 runs per model had losses that diverged to infinity. We can see that the only model reaching human performance in 2 billion environment steps is the GTrXL (GRU), with 10 runs having a mean score 8 and above. For the final gating ablation, we compare transformer variants while tracking their total parameter count to control for the increase in capacity caused by the introduction of additional parameters in the gating mechanisms. To demonstrate that the advantages of the GTrXL (GRU) are not solely due to an increase in parameter count, we halve the number of attention heads (which also effectively halves the embedding dimension due to the convention that the embedding size is the number of heads multiplied by the attention head dimension). The effect is a substantial reduction in parameter count, ing in less parameters than even the canonical TrXL. Fig. 6 and Tab. 2 compare the different models to the "Thin" GTrXL (GRU), with Tab. 2 listing the parameter counts. We include a parameter-matched LSTM model with 12 layers and 512 hidden size. The Thin GTrXL (GRU) surpasses every other model (within variance) except the GTrXL (GRU), even surpassing the next best-performing model, the GTrXL (Output), with over 10 million less parameters. All applicable gating variants in the previous sections were trained with the gated identity initialization. We observed in initial Memory Maze that the gated identity initialization significantly improved optimization stability and learning speed. Figure 7 compares an otherwise identical 4-layer GTrXL (GRU) trained with and without the gated identity initialization. Similarly to the previous sensitivity plots, we plot the ranked mean return of 10 runs at various times during training. As can be seen from Fig. 7, there is a significant gap caused by the bias initialization, suggesting that preconditioning the transformer to be close to Markovian in large learning speed gains. Gating has been shown to be effective to address the vanishing gradient problem and thus improve the learnability of recurrent models. LSTM networks rely on an input, forget and output gate that protect the update of the cell. GRU is another popular gated recurrent architecture that simplifies the LSTM cell, reducing the number of gates to two. Finding an optimal gating mechanism remains an active area of research, with other existing proposals (; ;), as well as works trying to discover optimal gating by neural architecture search More generally, gating and multiplicative interactions have a long history . Gating has been investigated previously for improving the representational power of feedforward and recurrent models (Van den ;), as well as learnability . Initialization has also played a crucial role in making deep models trainable (; ;). There has been a wide variety of work looking at improving memory in reinforcement learning agents. External memory approaches typically have a regular feedforward or recurrent policy interact with a memory database through read and write operations. Priors are induced through the design of the specific read/write operations, such as those resembling a digital computer (; or an environment map . An alternative non-parametric perspective to memory stores an entire replay buffer of the agent's past observations, which is made available for the agent to itself reason over either through fixed rules or an attention operation . Others have looked at improving performance of LSTM agents by extending the architecture with stacked hierarchical connections / multiple temporal scales and auxiliary losses ) or allowing an inner-loop update to the RNN weights . Other work has examined self-attention in the context of exploiting relational structure within the inputspace or within recurrent memories. In this paper we provided evidence that confirms previous observations in the literature that standard transformer models are unstable to train in the RL setting and often fail to learn completely . We presented a new architectural variant of the transformer model, the GTrXL, which has increased performance, more stable optimization, and greater robustness to initial seed and hyperparameters than the canonical architecture. The key contributions of the GTrXL are reordered layer normalization modules and a gating layer instead of the standard residual connection. We performed extensive ablation experiments testing the robustness, ease of optimization and final performance of the gating layer variations, as well as the effect of the reordered layer normalization. These empirically demonstrate that the GRU-type gating performs best across all metrics, exhibiting comparable robustness to hyperparameters and random seeds as an LSTM while still maintaining a performance improvement. Furthermore, the GTrXL (GRU) learns faster, more stably and achieves a higher final performance (even when controlled for parameters) than the other gating variants on the challenging multitask DMLab-30 benchmark suite. Having demonstrated substantial and consistent improvement in DMLab-30, Numpad and Memory Maze over the ubiquitous LSTM architectures currently in use, the GTrXL makes the case for wider adoption of transformers in RL. A core benefit of the transformer architecture is its ability to scale to very large and deep models, and to effectively utilize this additional capacity in larger datasets. In future work, we hope to test the limits of the GTrXL's ability to scale in the RL setting by providing it with a large and varied set of training environments. Numpad: Numpad has three actions, two of which move the sphere towards some direction in the x,y plane and the third allows the agent to jump in order to get over a pad faster. The observation consists of a variety of proprioceptive information (e.g. position, velocity, acceleration) as well as which pads in the sequence have been correctly activated (these will shut off if an incorrect pad is later hit), and the previous action and reward. Episodes last a fixed 500 steps and the agent can repeat the correct sequence any number of times to receive reward. Observations were processed using a simple 2-layer MLP with tanh activations to produce the transformer's input embedding. Ignoring the "jump" and "crouch" actions which we do not use, an action in the native DMLab action space consists of 5 integers whose meaning and allowed values are given in Table 4. Following previous work on DMLab , we used the reduced action set given in Table 5 with an action repeat of 4. Observations are 72 × 96 RGB images. Some levels require a language input, and for that all models use an additional 64-dimension LSTM to process the sentence. , the DMLab Arbitrary Visuomotor Mapping task was specifically used to highlight the MERLIN architecture's ability to utilize memory. In Figure 9 we show that, given a similarly reduced action set as used in , see Table 6, the GTrXL architecture can also reliably attain human-level performance on this task. Memory Maze: An action in the native Memory Maze action space consists of 8 continuous actions and a single discrete action whose meaning and allowed values are given in Table 7. Unlike for DMLab, we used a hybrid continuous-discrete distribution to directly output policies in the game's native action space. Observations are 72 × 96 RGB images. Table 6: Simplified action set for DMLab Arbitrary Visuomotor Mapping (AVM). This action set is the same as the one used for AVM in but with an additional no-op, which may also be replaced with the Fire action. Image Encoder: For DMLab-30 and Memory Maze, we used the same image encoder as in for multitask DMLab-30. The ResNet was adapted from and each of its layer blocks consists of a (3 × 3, stride 1) convolution, followed by (3 × 3, stride 2) max-pooling, followed by 2 3 × 3 residual blocks with ReLU non-linearities. For all experiments, beyond sampling independent random seeds, each run also has V-MPO hyperparameters sampled from a distribution (see Table 8). The sampled hyperparameters are kept fixed across all models for a specific experiment, meaning that if one of the α sampled is 0.002, then all models will have 1 run with α = 0.002 and so on for the rest of the samples. The exception is for the DMLab-30 LSTM, where a more constrained range was found to perform better in preliminary experiments. Each model had 8 seeds started, but not all runs ran to completion due to compute issues. These hyperparameter settings were dropped randomly and not due to poor environment performance. We report how many seeds ran to completion for all models. At least 6 seeds finished for every model tested. We list architecture details by section below. All LSTM models have residual skip connections in depth. All experiments in this work were carried out in an actor-learner framework that utilizes TF-Replicator for distributed training on TPUs in the 16-core Figure 10: The 25 hyperparameter settings sampled for the sensitivity ablation (Sec. 4.3.2). X-axis is in log scale and values are sampled from the corresponding ranges given in Table 8. MultiHeadAttention(E (l−1) ): where we used Einstein summation notation to denote the tensor multiplications, MaskedSoftmax is a causally-masked softmax to prevent addressing future information, Linear is a linear layer applied per time-step and we omit reshaping operations for simplicity. The basic MHA operation does not take sequence order into account explicitly because it is permutation invariant, so positional encodings are a widely used solution in domains like language where order is an important semantic cue, appearing in the original transformer architecture . To enable a much larger contextual horizon than would otherwise be possible, we use the relative position encodings and memory scheme described in. In this setting, there is an additional T -step memory tensor M (l) ∈ R T ×D, which is treated as constant during weight updates. Table 13: Partition of DMLab-30 levels into a memory-based and reactive split of levels. Model Median Human Normalized Score LSTM 136.6 ± 3.4 GTrXL 137.1 ± 5.0 Table 14: Final human-normalized median return across all 57 Atari levels for LSTM and GTrXL at 11.4 billion environment steps (equivalent to 200 million per individual game). Both models are 256 dimensions in width. We include standard error over runs. Figure 11: Median human-normalized returns as training progresses for both GTrXL and LSTM models. We run 8 hyperparameter settings per model. In this section, we run the GTrXL on the multitask Atari-57 benchmark. Although Atari-57 was not designed specifically to test an agent's memory capabilities, we include these here to demonstrate that we suffer no performance regression on a popular environment suite, providing further evidence that GTrXL can be used as an architectural replacement to the LSTM. The LSTM and GTrXL are matched in width at 256 dimensions. The GTrXL is 12 layers deep to show our model's learning stability even at large capacity. The LSTM architecture matches the one reported in. We train for 11.4 billion environment steps, equivalent to 200 million per environment. We run 8 hyperparameter settings per model.

represent the tldr for finding the corresponding scientific article

We succeed in stabilizing transformers for training in the RL setting and demonstrate a large improvement over LSTMs on DMLab-30, matching an external memory architecture.

Represent the TLDR for finding the corresponding scientific article


Question: represent the article for finding a single-sentence tldr

Question: Knowledge distillation is an effective model compression technique in which a smaller model is trained to mimic a larger pretrained model. However in order to make these compact models suitable for real world deployment, not only do we need to reduce the performance gap but also we need to make them more robust to commonly occurring and adversarial perturbations. Noise permeates every level of the nervous system, from the perception of sensory signals to the generation of motor responses. We therefore believe that noise could be a crucial element in improving neural networks training and addressing the apparently contradictory goals of improving both the generalization and robustness of the model. Inspired by trial-to-trial variability in the brain that can from multiple noise sources, we introduce variability through noise at either the input level or the supervision signals. Our show that noise can improve both the generalization and robustness of the model. ”Fickle Teacher” which uses dropout in teacher model as a source of response variation leads to significant generalization improvement. ”Soft Randomization”, which matches the output distribution of the student model on the image with Gaussian noise to the output of the teacher on original image, improves the adversarial robustness manifolds compared to the student model trained with Gaussian noise. We further show the surprising effect of random label corruption on a model’s adversarial robustness. The study highlights the benefits of adding constructive noise in the knowledge distillation framework and hopes to inspire further work in the area. The design of Deep Neural Networks (DNNs) for efficient real world deployment involves careful consideration of following key elements: memory and computational requirements, performance, reliability and security. DNNs are often deployed in resource constrained devices or in applications with strict latency requirements such as self driving cars which leads to a necessity for developing compact models that generalizes well. Furthermore, since the environment in which the models are deployed are often constantly changing, it is important to consider their performance on both indistribution data as well as out-of-distribution data. Thereby ensuring the reliability of the models under distribution shift. Finally, the model needs to be robust to malicious attacks by adversaries . Many techniques have been proposed for achieving high performance in compressed model such as model quantization, model pruning, and knowledge distillation. In our study, we focus on knowledge distillation as an interactive learning method which is more similar to human learning. Knowledge Distillation involves training a smaller network (student) under the supervision of a larger pre-trained network (teacher). In the original formulation, proposed mimicking the softened softmax output of the teacher model which consistently improves the performance of the student model compared to the model trained without teacher assistance. However, despite the promising performance gain, there is still a significant performance gap between the student and the teacher model. Consequently an optimal method of capturing knowledge from the larger network and transferring it to a smaller model remains an open question. While reducing this generalization gap is important, in order to truly make these models suitable for real world deployment, it is also pertinent to incorporate methods into the knowledge distillation framework that improve the robustness of the student model to both commonly occurring and malicious perturbations. For our proposed methods, we derive inspiration from studies in neuroscience on how humans learn. A human infant is born with billions of neurons and throughout the course of its life, the connections between these neurons are constantly changing. This neuroplasticity is at the very core of learning . Much of the learning for a child happens not in isolation but rather through collaboration. A child learns by interacting with the environment and understanding it through their own experience as well as observations of others. Two learning theories are central to our approach: cognitive bias and trial-to-trial response variation. Human decision-making shows systematic simplifications and deviations from the tenets of rationality ('heuristics') that may lead to sub-optimal decisional outcomes ('cognitive biases') . These biases are strengthened through repeatedly rewarding a particular response to the same stimuli. Trial-to-trial response variation in the brain, i.e. variation in neural responses to the same stimuli, encodes valuable information about the stimuli . We hypothesize that introducing constructive noise in the student-teacher collaborative learning framework to mimic the trial-to-trial response variation in humans can act as a deterrent to cognitive bias which is manifested in the form of memorization and over-generalization in neural networks. When viewed from this perspective, noise can be a crucial element in improving learning and addressing the apparent contradictory goals of achieving accurate and robust models. In this work, we present a compelling case for the beneficial effects of introduction of noise in knowledge distillation. We provide a comprehensive study on the effects of noise on model generalization and robustness. Our contributions are as follows: • A comprehensive analysis on the effects of adding a diverse range of noise types in different aspects of the teacher-student collaborative learning framework. Our study aims to motivate further work in exploring how noise can improve both generalization and robustness of the student model. • A novel approach for transferring teacher model's uncertainty to a student using Dropout in teacher model as a source of trial-to-trial response variability which leads to significant generalization improvement. We call this method "Fickle Teacher". • A novel approach for using Gaussian noise in the knowledge distillation which improves the adversarial robustness of the student model by an order of magnitude while significantly limiting the drop in generalization. we refer to this method as "Soft Randomization". • Random label corruption as a strong deterrent to cognitive bias and demonstrating its surprising ability to significantly improve adversarial robustness with minimal reduction in generalization. Many experimental and computational methods have reported the presence of noise in the nervous system and how it affects the the function of system . Noise as a common regularization technique has been used for ages to improve generalization performance of overparameterized deep neural networks by adding it to the input data, the weights or the hidden units Steijvers & Grünwald, 1996;;; ). Many noise techniques have been shown to improve generalization such as Dropout and injection of noise to the gradient (; Neelakantan et al.). Many works show that noise is crucial for non-convex optimization;; ). A family of randomization techniques that inject noise in the model both during training and inference time are proven to be effective to the adversarial attacks (? ; ;). Randomized smoothing transforms any classifier into a new smooth classifier that has certifiable l 2 -norm robustness guarantees . Label smoothing improves the performance of deep neural networks across a range of tasks . However, Müller et al. reports that label smoothing impairs knowledge distillation. We believe the knowledge distillation framework with the addition of constructive noise might offer a promising direction towards the design goal mentioned earlier, i.e. achieving lightweight well generalizing models with improved robustness to both adversarial and naturally occurring perturbations. For our empirical analysis, we adopted CIFAR-10 because of its pervasiveness in both knowledge distillation and robustness literature. Furthermore, the size of the dataset allows for extensive experimentation. To study the effect of noise addition in the knowledge distillation framework, we use Hinton method which trains the student model by minimizing the Kullback-Leibler divergence between the smoother output probabilities of the student and teacher model. In all of our experiments we use α = 0.9 and τ = 4. We conducted our experiments on Wide Residual Networks (WRN) (b). Unless otherwise stated, we normalize the images between 0 and 1 and use standard training scheme as used in (a;) To evaluate the out of distribution generalization of our models, we used the ImageNet images from the CINIC dataset . For adversarial robustness evaluation, we use the Projected Gradient Descent (PGD) attack from and run for multiple step sizes. We report the worst robustness accuracy for 5 random initialization runs. Finally, we test the robustness of our models to commonly occurring corruptions and perturbations proposed by in CIFAR-C as a proxy for natural robustness. For details of the methods, please see appendex. In this section, we propose injecting different types of noise in the student-teacher learning framework of knowledge distillation and analyze their effect on the generalization and robustness of the model. Here, we add a signal-dependent noise to the output logits of the teacher model. For each sample, we add zero-mean Gaussian noise with variance that is proportional to the output logits in the given sample (z i).ẑ We study the effect for the noise range [0 − 0.5] at steps of 0.1. Figure 1 shows for noise levels up to 0.1, the random signal-dependent noise improves the generalization to CIFAR-10 test set compared to the Hinton method without noise while marginally reducing the out-of-distribution generalization to CINIC-ImageNet. Figure 1 and Figure 11 show a slight increase in the adversarial robustness and natural robustness of the models. Müller et al. reported that when the teacher model is trained with label smoothing, the knowledge distillation to the student model is impaired and the student model performs worse. On the contrary, for lower level of noise, our method improves the effectiveness of distillation process. Our method differs from their approach in that we train the teacher model without any noise and only when distilling knowledge to the student, we add noise to its softened logits. Inspired by trial-to-trial variability in the brain and its constructive role in learning, we propose using dropout in the teacher model as a source of variability in the supervision signal from the teacher. We train the teacher model with dropout and while training the student model, we keep the dropout active in the teacher model. As a , repeated representation of the same input image leads to different output prediction of teacher. Gal & Ghahramani used dropout to obtain principled uncertainty estimates from deep learning networks. Gurau et al. utilize knowledge distillation to better calibrate a student model with the same architecture as the teacher model by using the soft target distribution obtained by averaging the Monte Carlo samples. Our proposed method differs from their method in a number of ways. We use dropout as a source of uncertainty encoding noise for distilling knowledge to a compact student model. Also, instead of averaging Monte Carlo simulations, we used the logits returned by the teacher model with activate dropout and train the student for more epochs so that it can capture the uncertainty of the teacher directly. Figure 2: Encoding the uncertainty of teacher helps the student to (a)generalize better on both unseen data and out-of-distribution data, and (b) to ave higher generalization to PGD attack. Note that for higher dropout rate the performance of teacher drops. We compare the generalization and robustness of the proposed method for dropout in the range [0 − 0.5] at steps of 0.1. For training parameters, please see the appendex. Figure 12a show that training the student model with dropout using our scheme significantly improves both in-distribution and outof-distribution generalization over the Hinton method. Interestingly, even when the performance of the teacher model used to train the model is decreasing after drop rate 0.2, the student model performance still improves up to drop rate 0.4. For dropout rate upto 0.2, both PGD Robustness (Figure 12b) and natural robustness increases (Figure 6). This suggest that as per our hypothesis, adding trial-to-trial variability helps in distilling knowledge to the student model. Pinot et al. provided theoretical evidence for the relation between adversarial robustness and the intensity of random noise injection in the input image. They show that injection of noise drawn from the exponential family such as Gaussian or Laplace noise leads to guaranteed robustness to adversarial attack. However this improved robustness comes at the cost of generalization. We propose a novel method for adding Gaussian noise in the input image while distilling knowledge to the student model. Since the knowledge distillation framework provides an opportunity to combine multiple sources of information, we hypothesize that using the teacher model trained on clean images, to train the student model with random Gaussian noise can retain the adversarial robustness gain observed with randomized training and mitigate the loss in generalization. Our method involves minimizing the following loss function in the knowledge distillation framework. where S denotes the output of student, S τ and T τ denote the soften logits of student and teacher models by temperature τ, respectively. α and τ are the balancing factor and temperature parameters from the Hinton method. We trained the models with six Gaussian noise levels and observe a significant increase in adversarial robustness and a decrease in generalization. However, our proposed method outperforms the compact model trained with Gaussian noise without teacher assistance for both generalization and robustness (Figures: 3 and 4). Our method is able to increase the adversarial robustness even at lower noise intensity For σ = 0.05, our method achieves 33.85% compared to 3.53% for the student model trained alone. In addition, our method also improves the robustness to common corruptions. Figure 5 shows that the robustness to noise and blurring corruptions improves significantly as the Gaussian noise intensity increases. For weather corruptions, it improves robustness except for fog and frost. Finally for digital corruption except for contrast and saturation, the robustness improves. We also observe changes in the effect at different intensities, for example for frost, the robustness increases at lower noise level and then decreases for higher intensities. Our method allows the use of lower noise intensity for increasing adversarial robustness while keeping the loss in generalization very low compared to other adversarial training methods. Following the analogy with cognitive bias in humans, and relating it to the memorization and over generalization in deep neural networks, we propose a counter intuitive regularization technique based on label noise. For each sample in the training process, with probability p, we randomly change the one hot encoded target labels to an incorrect class. The intuition behind this method is g a u ss ia n _n o is e im p u ls e _n o is e sh o t_ n o is e sp e ck le _n o is e d e fo cu s_ b lu r g a u ss ia n _b lu r g la ss _b lu r m o ti o n _b lu r zo o m _b lu r b ri g h tn e ss fo g fr o st sn o w sp a tt e r co n tr a st e la st ic _t ra n sf o rm jp e g _c o m p re ss io n p ix e la te sa tu ra te that by randomly relabeling a fraction of the samples in each epoch, we encourage the model to not be overconfident in its predictions and discourage memorization. There has been a number of studies on improving the tolerance of the DNNs to noisy labels (; ;). However, to the best of our knowledge, random label noise has not been explored as a source of constructive noise to improve the generalization of the model. g a u ss ia n _n o is e im p u ls e _n o is e sh o t_ n o is e sp e ck le _n o is e d e fo cu s_ b lu r g a u ss ia n _b lu r g la ss _b lu r m o ti o n _b lu r zo o m _b lu r b ri g h tn e ss fo g fr o st sn o w sp a tt e r co n tr a st e la st ic _t ra n sf o rm jp e g _c o m p re ss io n p ix e la te sa tu ra te We extensively study the effect of random label corruption on a range of p values and at multiple levels: teacher model alone, student model alone, both student and teacher model. When the label corruption is only used during knowledge distillation to student (Corrupted-S), both in-distribution and out-of-distribution generalization increases even for very high corruption levels. When the label corruption is used for training the teacher model and then used to train the student model with (Corrupted-TS) and without (Corrupted-T) label corruption, the generalization drops (Figure 7). In general. knowledge, for high level of label corruption, knowledge distillation outperforms the teacher model. Interestingly, random label corruption leads to a huge increase in adversarial robustness. Just by training with 5% random labels, the PGD-20 robustness of the teacher model increases from 0% to 10.89%. We see this increase in robustness for Corrupted-T and Corrupted-TS. Up to 40% random label corruption, the adversarial robustness increases and slightly decreases for 50%. We believe that this observed phenomenon warrants further study. Inspired by trial-to-trial variability in the brain, we introduce variability in the knowledge distillation framework through noise at either the input level or the supervision signals. For this purpose, we proposed novel ways of introducing noise at multiple levels and studied their effect on both generalization and robustness. Fickle teacher improves the both in-distribution and out of distribution generalization significantly while also slightly improving robustness to common and adversarial perturbations. Soft randomization improves the adversarial robustness of the student model trained alone with Gaussian noise by a huge margin for lower noise intensities while also reducing the drop in generalization. We also showed the surprising effect of random label corruption alone in increasing the adversarial robustness by an order of magnitude in addition to improving the generalization. Our strong empirical suggest that injecting noises which increase the trial-to-trial variability in the knowledge distillation framework is a promising direction towards training compact models with good generalization and robustness. A APPENDIX In this section we provide details for the methods relevant our study. Hinton et al. proposed to use the final softmax function with a raised temperature and use the smooth logits of the teacher model as soft targets for the student model. The method involves minimizing the Kullback-Leibler divergence between the smoother output probabilities: where L CE denotes cross-entropy loss, σ denotes softmax function, z S student output logit, z T teacher output logit, τ and α are the hyperparameters which denote temperature and balancing ratio, respectively. Neural networks tend to generalize well when the test data comes from the same distribution as the training data . However, models in the real world often have to deal with some form of domain shift which adversely affects the generalization performance of the models ((; ; ;). Therefore, test set performance alone is not the optimal metric for evaluation the generalization of the models in test environment. To measure the out-of-distribution performance, we used the ImageNet images from the CINIC dataset . CINIC contains 2100 images randomly selected for each of the CIFAR-10 categories from the ImageNet dataset. Hence the performance of models trained on CIFAR-10 on these 21000 images can be considered as a approximation for a model's out-of-distribution performance. A.3.1 ADVERSARIAL ROBUSTNESS Deep Neural Networks have been shown to be highly vulnerable to carefully crafted imperceptible perturbations designed to fool a neural networks by an adversary . This vulnerability poses a real threat to deep learning model's deployment in the real world . Robustness to these adversarial attacks has therefore gained a lot of traction in the research community and progress has been to better evaluate robustness to adversarial attacks (; ;) and defend our models against these attacks . To evaluate the adversarial robustness of models in this study, we use the Projected Gradient Descent (PGD) attack from. The PGD-N attack initializes the adversarial image with the original image with the addition of a random noise within some epsilon bound,. For each step it takes the loss with respect to the input image and moves in the direction of loss with the step size and then clips it within the epsilon bound and the range of valid image. where denote epsilon-bound, α step size and X original image. The projection operator,d (A) denotes element-wise clipping, with A i,j clipped to the range [X i,j −, X i,j +] and within valid data range. In all of our experiments, we use 5 random initializations and report the worst adversarial robustness. While robustness to adversarial attack is important from security perspective, it is an instance of worst case distribution shift. The model also needs to be robust to naturally occurring perturbations which it will encounter frequently in the test environment. Recent works have shown that Deep Neural Networks are also vulnerable to commonly occurring perturbations in the real world which are far from the adversarial examples manifold. curated a set of real-world, unmodified and naturally occurring examples that causes classifier accuracy to significantly degrade. measured model's robustness to the minute transformations found across video frames which they refer to as natural robustness and found state-of-the-art classifier to be brittle to these transformations. In their study, they found robustness to synthetic color distortions as a good proxy for natural robustness. In our study we use robustness to the common corruptions and perturbations proposed by in CIFAR-C as a proxy for natural robustness. A.3.3 TRADE OFF BETWEEN GENERALIZATION AND ADVERSARIAL ROBUSTNESS While making our model's robust to adversarial attacks, we need to be careful not to overemphasize robustness to norm bounded perturbation and rigorously test their effect on model's in-distribution and out-of-distribution generalization as well as robustness to naturally occurring perturbation and distribution shift. Recent study have highlighted the adverse affect of adversarially trained model on natural robustness. showed that even a semantics-preserving transformations on the input data distribution significantly degrades the performance of adversarial trained models but only slightly affects the performance of standard trained model. showed that adversarially trained models improve robustness to mid and high frequency perturbations but at the expense of low frequency perturbations which are more common in the real world. Furthermore, in the adversarial literature, a number of studies has shown an inherent trade-off between adversarial robustness and generalization;;. We would like to point out that these studies were conducted under adversarial setting and do not necessarily hold true for general robustness of the model. To exploit the uncertainty of the teacher model for a sample, we propose random swapping noise methods that select a sample with some probability p and then swap the softened softmax logits if the difference is below a threshold. We propose two variants of random swapping: 1. Swap Top 2: Swap the top two logits if the difference between them is below the threshold. 2. Swap All: Consider all consecutive pairs iteratively and swap them if the difference is below the threshold value. These methods improve the in-distribution generalization but adversely affects the out-ofdistribution generalization (Figure 9 . It does not have a pronounced affect on the robustness (Figures: 9b, 10). A.5 TRAINING SCHEME FOR DISTILLATION WITH DROPOUT Because of the variability in the teacher model, the student model needs to be trained to more epochs in order for it to converge and effectively capture the uncertainty of the teacher model. We used the same initial learning rate of 0.1 and decay factor of 0.2 as per the standard training scheme. For dropout rate of 0.1 and 0.2, we train for 250 epochs and reduce learning rate at 75, 150 and 200 epochs. For dropout rate 0.3, we train for 300 epochs and reduce learning rate at 90, 180 and 240 epochs. Finally for drop rate of 0.4 and 0.5, due to the increased variability, we train for 350 epochs and reduce learning rate at 105, 210 and 280 epochs. Adversarial Robustness Figure 9: Noise on the supervision from teacher by swapping all logits or the top 2 (a) improves the accuracy of student on unseen data, but not the generalization to out-of-distribution data. g a u ss ia n _n o is e im p u ls e _n o is e sh o t_ n o is e sp e ck le _n o is e d e fo cu s_ b lu r g a u ss ia n _b lu r g la ss _b lu r m o ti o n _b lu r zo o m _b lu r b ri g h tn e ss fo g fr o st sn o w sp a tt e r co n tr a st e la st ic _t ra n sf o rm jp e g _c o m p re ss io n p ix e la te sa tu ra te g a u ss ia n _n o is e im p u ls e _n o is e sh o t_ n o is e sp e ck le _n o is e d e fo cu s_ b lu r g a u ss ia n _b lu r g la ss _b lu r m o ti o n _b lu r zo o m _b lu r b ri g h tn e ss fo g fr o st sn o w sp a tt e r co n tr a st e la st ic _t ra n sf o rm jp e g _c o m p re ss io n p ix e la te sa tu ra te

represent the tldr for finding the corresponding scientific article!

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.

represent the short summary for finding the corresponding scientific article (long)

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We introduce a novel end-to-end approach for learning to cluster in the absence of labeled examples. Our clustering objective is based on optimizing normalized cuts, a criterion which measures both intra-cluster similarity as well as inter-cluster dissimilarity. We define a differentiable loss function equivalent to the expected normalized cuts. Unlike much of the work in unsupervised deep learning, our trained model directly outputs final cluster assignments, rather than embeddings that need further processing to be usable. Our approach generalizes to unseen datasets across a wide variety of domains, including text, and image. Specifically, we achieve state-of-the-art on popular unsupervised clustering benchmarks (e.g., MNIST, Reuters, CIFAR-10, and CIFAR-100), outperforming the strongest baselines by up to 10.9%. Our generalization are superior (by up to 21.9%) to the recent top-performing clustering approach with the ability to generalize. Clustering unlabeled data is an important problem from both a scientific and practical perspective. As technology plays a larger role in daily life, the volume of available data has exploded. However, labeling this data remains very costly and often requires domain expertise. Therefore, unsupervised clustering methods are one of the few viable approaches to gain insight into the structure of these massive unlabeled datasets. One of the most popular clustering methods is spectral clustering (; ;), which first embeds the similarity of each pair of data points in the Laplacian's eigenspace and then uses k-means to generate clusters from it. Spectral clustering not only outperforms commonly used clustering methods, such as k-means , but also allows us to directly minimize the pairwise distance between data points and solve for the optimal node embeddings analytically. Moreover, it is shown that the eigenvector of the normalized Laplacian matrix can be used to find the approximate solution to the well known normalized cuts problem . In this work, we introduce CNC, a framework for Clustering by learning to optimize expected Normalized Cuts. We show that by directly minimizing a continuous relaxation of the normalized cuts problem, CNC enables end-to-end learning approach that outperforms top-performing clustering approaches. We demonstrate that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet, which consequently in better clustering accuracy. Let us motivate CNC through a simple example. In Figure 1, we want to cluster 6 images from CIFAR-10 dataset into two clusters. The affinity graph for these data points is shown in Figure 1 (a) (details of constructing such graph is discussed in Section 4.2). In this example, it is obvious that the optimal clustering is the of cutting the edge connecting the two triangles. Cutting this edge will in the optimal value for the normalized cuts objective. In CNC, we define a new differentiable loss function equivalent to the expected normalized cuts objective. We train a deep learning model to minimize the proposed loss in an unsupervised manner without the need for any labeled datasets. Our trained model directly returns the probabilities of belonging to each cluster (Figure 1(b) ). In this example, the optimal normalized cuts is 0.286 (Equation 1), and as we can see, the CNC loss also converges to this value (Figure 1(c) Optimal Normalized cuts #edge cuts = 1 per cluster volume = 2+2+3 = 7 1/7 + 1/7 = 0.286 Cluster 2 Cluster 2 Cluster 1 Figure 1: Motivational example: (a) affinity graph of 6 images from CIFAR-10, the objective is to cluster these images into two clusters. (b) CNC model is trained to minimize expected normalized cuts in an unsupervised manner without the need for any labeled data. For each data point, our model directly outputs the probabilities of it belonging to each of the clusters. (c) The CNC loss converges to the optimal normalized cuts value. In Algorithm 1 we show how we can scale this approach through a batch processing technique to large datasets. We compare the performance of CNC to several learning-based clustering approaches (SpectralNet, DEC , DCN , VaDE , DEPICT , IMSAT , and IIC ) on four datasets: MNIST, Reuters, CIFAR10, and CIFAR100. Our show up to 10.9% improvement over the baselines. Moreover, generalizing spectral embeddings to unseen data points, a task commonly referred to as out-of-sample-extension (OOSE), is a non-trivial task (; ;). Our confirm that CNC generalizes to unseen data. Our generalization are superior (by up to 21.9%) to SpectralNet, the recent top-performing clustering approach with the ability to generalize. Recent deep learning approaches to clustering attempt to embed the input data into a form that is amenable to clustering by k-means or Gaussian Mixture Models. focused on learning representations for clustering. To find the clustering-friendly latent representations and to better cluster the data, DCN proposed a joint dimensionality reduction (DR) and K-means clustering approach in which DR is accomplished via learning a deep neural network. DEC simultaneously learns cluster assignment and the underlying feature representation by iteratively updating a target distribution to sharpen cluster associations. Several other approaches rely on a variational autoencoder that utilizes a Gaussian mixture prior (; ; ; ;). These approaches are mainly based on data augmentation, where the network is trained to maximize the mutual information between inputs and predicted clusters, while regularizing the network so that the cluster assignment of the data points is consistent with the assignment of the augmented points. Different clustering objectives, such as self-balanced k-means and balanced min-cut, have also been exhaustively studied (; ;). One of the most effective techniques is spectral clustering, which first generates node embeddings in the eigenspace of the graph Laplacian, and then applies k-means clustering to these vectors (; ;). To address the fact that clusters with the lowest graph conductance tend to have few nodes , proposed regularized spectral clustering to encourage more balanced clusters. Generalizing clustering to unseen nodes and graphs is nontrivial (; ;). A recent work, SpectralNet, takes a deep learning approach to spectral clustering that generalizes to unseen data points. This approach first learns embeddings of the similarity of each pair of data points in Laplacian's eigenspace and then applies k-means to those embeddings to generate clusters. Unlike SpectralNet, we propose an end-to-end learning approach with a differentiable loss that directly minimizes the normalized cuts. We show that our approach indeed can produce lower normalized cut values than the baseline methods such as SpectralNet, which consequently in better clustering accuracy. Our evaluation show that CNC improves generalization accuracy on unseen data points by up to 21.9%. Since CNC objective is based on optimizing normalized cuts, in this section, we briefly overview the formal definition of this metric. Let G = (V, E, W) be a graph where V = {v i} and E = {e(v i, v j)|v i ∈ V, v j ∈ V } are the set of nodes and edges in the graph and w ij ∈ W is the edge weight of the e(v i, v j). Let n be the number of nodes. A graph G can be clustered into g disjoint sets S 1, S 2,... S g, where the union of the nodes in those sets are V (g k=1 S k = V), and each node belongs to only one set (g k=1 S k = ∅), by simply removing edges connecting those sets. For example, in Figure 1(a), by removing one edge two disjoint clusters are formed. Normalized cuts (Ncuts) which is defined based on the graph conductance, has been studied by , and the cost of a cut that forms disjoint sets S 1, S 2,... S g is computed as: WhereS k represents the complement of S k, i.e.,S k = i =k S i. cut(S k,S k) is called cut and is the total weight of the edges that are removed from G in order to form disjoint sets is the total edge weights (w ij), whose end points (v i, or v j) belong to S k. The cut and vol are: In running example (Figure 1), since the edge weights are one, cut(S 1,S 1) = cut(S 2,S 2) = 1, and Thus the Ncuts(S 1, S 2) = Finding the cluster assignments that minimizes the normalized cuts is NP-complete and an approximation to the this problem is based on the eigenvectors of the normalized graph Laplacian which has been studied in . CNC, on the other hand, is a neural network framework for learning to cluster in the absence of labeled examples by directly minimizing the continuous relaxation of the normalized cuts. As shown in Algorithm 1, end-to-end training of the CNC contains two steps, i.e, (i) data points embedding (line 3), and (ii) clustering (lines 4-9). In data points embedding, the goal is to learn embeddings that capture the affinity of the data points, while the clustering step uses those embeddings to learn the CNC model and outputs the cluster assignments. Next, we first focus on the clustering step and we introduce our new differentiable loss function to train CNC model. Later in Section 4.2, we discuss the details of the embedding step. In this section, we describe the clustering step in Algorithm 1 (lines 4-9). For each data point x i, the input to clustering step is embedding v i ∈ R d (detail in Section 4.2). The goal is to learn CNC model, which represents the assignment probabilities over g clusters. Clearly for n data points, it returns Y ∈ R n×g where Y ik represents the probability that v i belongs to cluster S k. The CNC model F θ is implemented using a neural network, where the parameter vector θ denotes the network weights. We propose a loss function based on output Y to calculate the expected normalized cuts. Thus CNC learns the F θ by minimizing this loss (Equation 7). Recall that cut(S k,S k) is the total weight of the edges that are removed from G in order to form disjoint sets S k andS k. In our setup, embeddings are the nodes in graph G, and neighbors of an embedding v i are based on the k-nearest neighbors. Let Y ik be the probability that node v i belongs to cluster S k. The probability that node v j does not belong to S k would be 1 − Y jk. Therefore, Compute affinity graph W ∈ R b×b over the M based on the k-nearest neighbors 7: Use M and W to train CNC model F θ: R d → R g that minimizes the expected normalized cuts (Equation 6) via backpropagation. For a data point with embedding v i the output y i = F θ (v i) represents the assignment probabilities over g clusters. 8: end while Inference, cluster assignments 9: For every data points x i whose embedding is v i return arg max of y i = F θ (v i) as its cluster assignment. Since the weight matrix W represents the edge weights adjacent nodes, we can rewrite Equation 3: The element-wise product with the weight matrix (W) ensures that only the adjacent nodes are considered. Moreover, the of W is an n × n matrix and reduce-sum is the sum over all of its elements. From Equation 2, vol(S k, V) is the total edge weights (w ij), whose end points (v i, or v j) belong to S k. Let D be a column vector of size n where D i is the total edge weights from node v i. We can update Equation 3 as follows to find the expected normalized cuts. The matrix representation is given in Equation 6, where Γ = Y D is a vector in R g, and g is the number of sets/clusters. is element-wise division and the of (Y Γ)(1 − Y) W is a n × n matrix where reduce-sum is the sum over all of its elements. CNC model F θ is implemented using a neural network, where the parameter θ denotes the network weights (y i = F θ (v i)). CNC is trained to optimize Equation 7 via backpropagation (Algorithm 1). arg min As you can see the affinity graph W is part of the CNC loss (Equation 7). Clearly, when the number of data points (n) is large, such calculation can be expensive. However, in our experimental , we show that for large dataset (e.g., Reuters contains 685,071 documents), it is possible to optimize the loss on randomly sampled minibatches of data. We also build the affinity graph over a given minibach using the embeddings and based on their k nearest-neighbor (Algorithm 1 (lines 5-6)). Specifically, in our implementation, CNC model F θ is a fully connected layer followed by gumble softmax, trained on randomly sampled minibatches of data to minimize Equation 6. In Section 5.7 through a sensitivity analysis we show that the minibatch size affects the accuracy of our model. When training is over, the final assignment of a data point with embedding v i to a cluster is the arg max of y i = F θ (v i) (Algorithm 1 (line 9)). In this section, we discuss the embedding step (line 3 in Algorithm 1). Different affinity measures, such as simple euclidean distance or nearest neighbor pairs combined with a Gaussian kernel, have been used in spectral clustering. Recently it is shown that unsupervised application of a Siamese network to determine the distances improves the quality of the clustering. In this work, we also use Siamese networks to learn embeddings that capture the affinities of the data points. Siamese network is trained to learn an adaptive nearest neighbor metric. It learns the affinities directly from euclidean proximity by "labeling" points x i, x j positive if x i − x j is small and negative otherwise. In other words, it generates embeddings such that adjacent nodes are closer in the embedding space and non-adjacent nodes are further. Such network is typically trained to minimize contrastive loss: We evaluate the performance of CNC in comparison to several deep learning-based clustering approaches on four real world datasets: MNIST, Reuters, CIFAR-10, and CIFAR-100. The details of the datasets are as follows: • MNIST is a collection of 70,000 28×28 gray-scale images of handwritten digits, divided into 60,000 training images and 10,000 test images. • The Reuters dataset is a collection of English news labeled by category. Like SpectralNet, DEC, and VaDE, we used the following categories: corporate/industrial, government/social, markets, and economics as labels and discarded all documents with multiple labels. Each article is represented by a tfidf vector using the 2000 most frequent words. The dataset contains 685,071 documents. We divided the data randomly to a 90%-10% split to evaluate the generalization ability of CNC. We also investigate the imapact of training data size on the generalization by considering following splits: 90%-10%, 70%-30%, 50%-50%, 20%-80%, and 10%-90%. • CIFAR-10 consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. • CIFAR-100 has 100 classes containing 600 images each with a 500/100 train/test split per class. In all runs we assume the number of clusters is given. In MNIST and CIFAR-10 number of clusters (g) is 10, g = 4 in Reuters, g = 100 in CIFAR-100. We compare CNC to SpectralNet, DEC , DCN , VaDE , DEPICT , IMSAT , and IIC . While focused on learning representations for clustering, other approaches (; ; ; ;) rely on a variational autoencoder that utilizes a Gaussian mixture prior. SpectralNet, takes a deep learning approach to spectral clustering that generalizes to unseen data points. Table 1 shows the reported for these six methods. Similar to, for MNIST and Reuters we use publicly available and pre-trained autoencoders 1. The autoencoder used to map the Reuters data to code space was trained based on a random subset of 10,000 samples from the full dataset. Similar to , for CIFAR-10 and CIFAR-100 we applied 50-layer pre-trained deep residual networks trained on ImageNet to extract features and used them for clustering. We use two commonly used measures, the unsupervised clustering accuracy (ACC), and the normalized mutual information (NMI) in to evaluate the accuracy of the clustering. Both ACC and NMI are in, with higher values indicating better correspondence the clusters and the true labels. Note that true labels never used neither in training, nor in test. Clustering Accuracy (ACC): For data points X = {x 1, . . . x n}, let l = (l 1, . . . l n) and c = (c 1, . . . c n) be the true labels and predicted clusters respectively. The ACC is defined as: where is the collection of all permutations of 1,... g. The optimal permutation π can be computed using the Kuhn-Munkres algorithm . Normalized Mutual Information (NMI): Let I(l; c) be the mutual information between l and c, and H be their entropy. The NMI is: For each dataset we trained a Siamese network to learn embeddings which represents the affinity of data points by only considering the k-nearest neighbors of each data. In Table 1, we compare clustering performance across four benchmark datasets. Since most of the clustering approaches do not generalize to unseen data points, all data has been used for the training (Later in Section 5.4, to evaluate the generalizability we use 90%-10% split for training and testing). While the improvement of CNC is marginal over MNIST, it performs better across other three datasets. Specifically, over CIFAR-10, CNC outperforms SpectralNet and IIC on ACC by 20.1% and 10.9% respectively. Moreover, the NMI is improved by 12.3%. The over Reuters, and CIFAR-100, show 0.021% and 11% improvement on ACC. The NMI is also 27% better over CIFAR-100. The fact that our CNC outperforms existing approaches in most datasets suggests the effectiveness of using our deep learning approach to optimize normalized cuts for clustering. We performed an ablation study to evaluate the impact of embeddings by omitting this step in Algorithm 1. We find that on both MNIST and Reuters datasets, adding the embedding step improves the performance, but CNC without embeddings still outperforms SpectralNet without embeddings. On MNIST, the ACC and NMI are 0.945 and 0.873, whereas with the embeddings, ACC and NMI increase to 0.972 and 0.924 (Table 1). Without embeddings, CNC outperforms SpectralNet (with ACC of 0.8 and NMI of 0.814). On Reuters, the ACC and NMI are 0.684 and 0.428, whereas with the embeddings, ACC and NMI increase to 0.824 and 0.583. Again, even without embeddings, CNC outperforms SpectralNet (with ACC of 0.605 and NMI of 0.401). ]. CNC is able to find better cuts than the SpectralNet We further evaluate the generalization ability of CNC by dividing the data randomly to a 90%-10% split and training on the training set and report the ACC and NMI on the test set (Table 2). Among seven methods in Table 1, only SpectralNet is able to generalize to unseen data points. CNC outperforms SpectralNet in most datasets by up to 21.9% on ACC and up to 10.7% on NMI. Note that simple arg max over the output of CNC retrieves the clustering assignments while SpectralNet relies on k-means to predict the final clusters. To evaluate the impact of normalized cuts for the clustering task, we calculate the numerical value of the Normalized cuts (Equation 1) over the clustering of the CNC and SpectralNet. Since such calculation over whole dataset is very expensive we only show this over the test set. Table 3 shows the numerical value of the Normalized cuts over the clustering of the CNC and SpectralNet. As one can see CNC is able to find better cuts than the SpectralNet. Moreover, we observe that for those datasets that the improvement of the CNC is marginal (MNIST and Reuters), the normalized cuts of CNC are also only slightly better than the SpectralNet, while for the CIFAR-10 and CIFAR-100 that the accuracy improved significantly the normalized cuts of CNC are also much smaller than SpectralNet. The higher accuracy (ACC in Table 2) and smaller normalized cuts (Table 3), verify that indeed CNC loss function is a good notion for clustering task. As you may see in generalization (Table 2), when we reduce the size of the training data to 90% the accuracy of CNC slightly changed in compare to training over the whole data (Table 1). Based on this observation, we next investigate how varying the size of the training dataset affects the generalization. In other words, how ACC and NMI of test data change when we vary the size of the training dataset. We ran experiment over Routers dataset by dividing the data randomly based on the following data splits: 90%-10%, 70%-30%, 50%-50%, 20%-80%, and 10%-90%. For example, in 10%-90%, we train CNC over 10% of the data and we report the ACC and NMI of CNC over the 90% test set. Figure 3 shows how the ACC and NMI of CNC over the test data change as the size of the training data is varied. For example, when the size of the training data is 90%, the ACC of CNC over the test data is 0.824. As we expected and shown in Figure 3 the ACC and NMI of CNC increased as the size of the training data is increased. Interestingly, we observed that with only 10% training data the ACC of CNC is 0.68 which is only 14% lower than the ACC with 90% training data. Similarly the NMI of CNC with 10% training data is only 18% lower than the NMI with 90% training data. Here are the details of the CNC model for each dataset. • MNIST: The Siamese network has 4 layers sized with ReLU (Embedding size d is 10). The clustering module has 2 layers sized with a final gumbel softmax layer. Batch sized is 256 and we only consider 3 nearest neighbors to find the embeddings and constructing the affinity graph for each batch. We use Adam with lr = 0.005 with decay 0.5. Temperature starts at 1.5 and the minimum is set to 0.5. • Reuters: The Siamese network has 3 layers sized with ReLU (Embedding size d is 128). The clustering module has 3 layers sized with tanh activation and a final gumbel softmax layer. Batch sized is 128 and we only consider 3 nearest neighbors to find the embeddings and constructing the affinity graph for each batch. We use Adam with lr = 1e-4 with decay 0.5. Temperature starts at 1.5 and the minimum is set to 1.0. • CIFAR-10: The Siamese network has 2 layers sized with ReLU (Embedding size d is 256). The clustering module has 2 layers sized with tanh activation and a final gumbel softmax layer. Batch sized is 256 and we only consider 2 nearest neighbors to find the embeddings and constructing the affinity graph for each batch. We use Adam with lr = 1e-4 with decay 0.1. Temperature starts at 2.5 and the minimum is set to 0.5. • CIFAR-100: The Siamese network has 2 layers sized with ReLU (Embedding size d is 256). The clustering module has 3 layers sized with tanh activation and a final gumbel softmax layer. Batch sized is 1024 and we only consider 3 nearest neighbors to find the embeddings and constructing the affinity graph for each batch. We use Adam with lr = 1e-3 with decay 0.5. Temperature starts at 1.5 and the minimum is set to 1.0. Hyper-parameter Sensitivity: We train the CNC on the Reuters dataset by fixing some hyperparameters and varying others. We noticed that CNC benefits from tuning the number of hidden layers (hl), learning rate (lr), batch size (b), and the number of nearest neighbors (k), but is not particularly sensitive to any of the other hyper-parameters, including decay rate, patience parameter (cadence of epochs where decay is applied), Gumbel-Softmax temperature or minimum temperature (Figure 4). More precisely, we varied decay rate over the range [0.1-1.0], patience from epochs, Gumbel-Softmax temperature from [1.0-2.0], and minimum temperature from [0.5-1.0]. When we fix hl=3, lr=5e-5, b=64, and k=3, the average accuracy is 0.803 ± 2e − 3. With hl=3, lr=5e-4, b=512, and k=10, the average accuracy is 0.811 ± 2e − 3. With hl=3, lr=5e-4, b=128, and k=3, the average accuracy is 0.821 ± 4e − 3. With hl=2, lr=1e-4, b=256, and k=3, the average accuracy is 0.766 ± 9e − 4. And finally with hl=4, lr=1e-5, b=512, and k=3, the average accuracy is 0.766 ± 7e − 3. As one can see, the accuracy varied from 0.766 to 0.821. We propose CNC (Clustering by learning to optimize Normalized Cuts), a framework for learning to cluster unlabeled examples. We define a differentiable loss function equivalent to the expected normalized cuts and use it to train CNC model that directly outputs final cluster assignments. CNC achieves state-of-the-art on popular unsupervised clustering benchmarks (MNIST, Reuters, CIFAR-10, and CIFAR-100 and outperforms the strongest baselines by up to 10.9%. CNC also enables generation, yielding up to 21.9% improvement over SpectralNet, the previous best-performing generalizable clustering approach. Table 4 : Generalization : CNC is trained on VGG and validated on MNIST-conv. During inference, the model is applied to unseen TensorFlow graphs: ResNet. Inception-v3, and AlexNet. The ground truth for AlexNet is Bal = 99%, Cut = 4.6%, for Inception-v3, is Bal = 99%, Cut = 3.7%, and for ResNet is Bal = 99% and Cut = 3.3%. GraphSAGE-on generalizes better than the other models. To show that CNC generalizes effectively on unseen graphs, we train CNC on a single TensorFlow graph, VGG, and validate on MNIST-conv. During inference, we test the trained model on unseen TensorFlow graphs: AlexNet, ResNet, and Inception-v3. We consider the best quality among hMETIS, KaFFPa, and KaFFPaE as the ground truth. The ground truth for AlexNet is Bal = 99%, Cut = 4.6%, for Inception-v3, is Bal = 99%, Cut = 3.7%, and for ResNet is Bal = 99% and Cut = 3.3%. Table 4 shows the of our experiments, and illustrates the importance of graph embeddings in generalization. The operation type (such as Add, Conv2d, and L2loss in TensorFlow) is used as the node feature as a one-hot. We leverage GCN and GraphSAGE to capture similarities across graphs. In GraphSAGE-on both node embedding and clustering modules are trained jointly, while in GCN and GraphSAGE-off, only the clustering module is trained. Table 4 shows that the GraphSAGE-on (last row) achieves the best performance and generalizes better than the other models. Note that this model is trained on a single graph, VGG with only 1325 nodes, and is tested on AlexNet, ResNet, and Inception-v3 with 798, 20586, and 27114 nodes respectively. On the other hand, the ground truth is the of running different partitioning algorithms on each graph individually. In this work, our goal is not to beat the existing graph partitioning algorithms which involve a lot of heuristics on a given graph. Our generalization show promise that rather than using heuristics, CNC is able to learn graph structure for generalizable graph partitioning. Model Architecture and Hyper-parameters: The details of the model with the best performance (GraphSAGE-on) are as follows: the input feature dimension is 1518. GraphSAGE has 5 layers sized 512 with shared pooling, and the graph clustering module has 3 layers sized 64 with a final softmax layer. We use ReLU, Xavier initialization , and Adam with lr = 7.5e-5.

We introduce a novel end-to-end approach for learning to cluster in the absence of labeled examples. We define a differentiable loss function equivalent to the expected normalized cuts.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the article for finding a single-sentence TLDR

Question: We introduce the largest (among publicly available) dataset for Cyrillic Handwritten Text Recognition and the first dataset for Cyrillic Text in the Wild Recognition, as well as suggest a method for recognizing Cyrillic Handwritten Text and Text in the Wild. Based on this approach, we develop a system that can reduce the document processing time for one of the largest mathematical competitions in Ukraine by 12 days and the amount of used paper by 0.5 ton. Text is one of the main ways to transfer the information and it can take many forms. It can be handwritten or printed, in the form of business documents, notes, bills, historical documents, advertisements, logos etc. Therefore, the method for its recognition should be flexible enough to work with different text styles and under the different external conditions. Although for the English language the task of text recognition is well studied,, for Cyrillic languages such studies are almost missing, the main reason being the lack of extensive publicly available datasets. To the best of our knowledge, the only public Cyrillic dataset consists only of individual letters, while others,,, are unavailable. In our research, we will focus on developing a single model for Handwritten Text Recognition and Text Recognition in the Wild, as the extreme case of printed text. Currently the amount of data provided by the existing Cyrillic datasets for text recognition is insufficient for training deep networks. That is why we have created a synthetic dataset and annotated datasets for Handwritten Text and Text in the Wild recognition. We approach the problem of generating Synthetic Cyrillic text recognition dataset by adapting the idea proposed by Jaderberg et al., but in our Cyrillic setup. For the text sampling stage, a word is randomly sampled from the UberText Corpus with the requirement to include only Cyrillic letters. For the font rendering, a font is randomly selected from the subset of fonts from UKR-Fonts, Google Fonts, Font Squirrel including all glyphs needed for generation of the sampled word. In total, the dataset consists of 881309 samples of 180994 different words (Figure 1). As mentioned above, at the moment there are no existing datasets for Text in the Wild recognition of Cyrillic words. We addressed this problem by collecting the dataset by annotating photos from the Internet depicting different cities all around Ukraine. The dataset consist of 505 samples in different orientations from 151 different photos and in total has 454 different words (Figure 1). The extended version of this dataset consists not only of rectangles of words, but also their locations in the images (although this information is not used in the research). The handwritten dataset was collected by processing data from one of the largest mathematical competitions in Ukraine. The dataset is extracted from the forms that were filled by children aged 7 to 18 from different parts of Ukraine and contains mainly their surnames. It consists of 82061 samples and 37007 words (Figure 1) and is divided into 3 parts (train 60%, validate 20% and test 20%), with the same distribution of each word in each part of the dataset. We developed a single model that was pretrained on Synthetic Cyrillic text and then finetuned on a mixture of Synthetic Cyrillic text and Handwritten Cyrillic text. The architecture consists of two Conv blocks with CRelu and Relu followed by six Conv blocks with Instance Normalisation and Leaky Relu. The obtained word error rate and character error rate are reported in Table 1. The method was used for the development of the new system for processing responses of participants of one of the largest math competitions in Ukraine (International Mathematics Contest "Kangaroo"). The new system allows to remove part of the form that was used by participants for manual annotation of their surnames and to reduce the size of the form from B5 to A5. This competition is organized twice a year and attracts around 500000 participants each time. The new system will allow to reduce the processing (scanning) time by 12 days, and the amount of used paper by 0.5 tons on each round.

represent the tldr for finding the corresponding scientific article:

We introduce several datasets for Cyrillic OCR and a method for its recognition

We introduce a lightweight architecture for named entity recognition and carry out incremental active learning, which is able to match state-of-the-art performance with just 25% of the original training data.