Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this paper, a new intrinsic reward generation method for sparse-reward reinforcement learning is proposed based on an ensemble of dynamics models. In the proposed method, the mixture of multiple dynamics models is used to approximate the true unknown transition probability, and the intrinsic reward is designed as the minimum of the surprise seen from each dynamics model to the mixture of the dynamics models. In order to show the effectiveness of the proposed intrinsic reward generation method, a working algorithm is constructed by combining the proposed intrinsic reward generation method with the proximal policy optimization (PPO) algorithm. Numerical show that for representative locomotion tasks, the proposed model-ensemble-based intrinsic reward generation method outperforms the previous methods based on a single dynamics model. Reinforcement learning (RL) with sparse reward is an active research area (; de ; ;). In typical model-free RL, an agent learns a policy to maximize the expected cumulative reward under the circumstance that the agent receives a non-zero reward from the environment for each action of the agent. On the contrary, in sparse reward RL, the environment does not return a non-zero reward for every action of the agent but returns a non-zero reward only when certain conditions are met. Such situations are encountered in many action control problems (; . As in conventional RL, exploration is important at the early stage of learning in sparse reward RL, whereas the balance between exploration and exploitation is required on the later stage. Methods such as the -greedy strategy and the control of policy gradient with Gaussian random noise a) have been applied to various tasks for exploration. However, these methods have been revealed to be insufficient for successful learning when reward is sparse . In order to overcome such difficulty, intrinsically motivated RL has been studied to stimulate better exploration by generating intrinsic reward for each action by the agent itself, even when reward is sparse. Recently, many intrinsically-motivated RL algorithms have been devised to deal with the sparsity of reward, e.g., based on the notion of curiosity ) and surprise . It is shown that these algorithms are successful and outperform the previous approaches. In essence, these algorithms use a single estimation model for the next state or the environment dynamics to generate intrinsic reward. In this paper, in order to further improve the performance of sparse reward model-free RL, we propose a new method to generate intrinsic reward based on an ensemble of estimation models for the environment dynamics. The rationale behind our approach is that by using a mixture of several distributions, we can increase degrees of freedom for modeling the unknown underlying model dynamics and designing a better reward from the ensemble of estimation models. Numerical show that the proposed model-ensemble-based intrinsic reward generation method yields improved performance as compared to existing reward generation methods for continuous control with sparse reward setting. In this paper, we consider a discrete-time continuous-state Markov Decision Process (MDP), denoted as (S, A, P, r, ρ 0, γ), where S and A are the sets of states and actions, respectively, P: S × A × S → is the transition probability function (called model dynamics), r: S ×A×S → R is the extrinsic reward function, ρ 0: S → is the distribution of the initial state, and γ is the discounting factor. A (stochastic) policy is represented by π: S × A →, where π(a|s) represents the probability of choosing action a ∈ A for given state s ∈ S. In sparse reward RL, the environment does not return a non-zero reward for every action but returns a non-zero reward only when certain conditions are met by the current state, the action and the next state (; . The goal of this paper is to optimize the policy π to maximize the expected cumulative return η(π) by properly generating intrinsic reward in such sparse reward environments. We assume that the true transition model P is unknown to the agent. Intrinsically-motivated RL adds a properly designed intrinsic reward to the actual extrinsic reward to yield a non-zero total reward for training even when the extrinsic reward returned by the environment is zero (de ; ;). One way to design such an intrinsic reward for action control is based on surprise, which is a measure of the unexpectedness of observing the next state for a given current state and action pair and is especially useful to yield better exploration . In this context, a recent work proposed a promising direction of intrinsic reward design in which the agent tries to optimize its policy π according to for some constant c > 0, where P φ is the learning model parameterized by φ that the agent has regarding the true unknown transition probability P of the environment. is the Kullback-Leibler divergence (KLD) between two distributions P and P φ of the next state for given current state-action pair (s, a), and E (s,a)∼π is the expectation over (s, a) following the policy π. Thus, the surprise is quantified as & ). Furthermore, the KLD D KL (P ||P φ)|(s t, a t) at timestep t can be lower-bounded as with an arbitrary choice of the parameter φ. Therefore, the intrinsic reward at timestep t is determined as r t,int (s t, a t, s t+1) = log P φ (st+1|st,at) P φ (st+1|st,at) , where P φ needs to be designed properly. With φ = φ(t) and φ = φ(t −), where P φ(t) and P φ(t −) are respectively the agent's model for P at timestep t and the model before the update at timestep t, the intrinsic reward is given by the computable quantity named as the 1-step surprise: The proposed 1-step intrinsic reward performs well compared to the previously designed intrinsic reward, and it is based on a single model P φ for P, where P φ for given (s, a) is modeled as Gaussian distribution . In this paper, we take the principle that D KL (P ||P φ)|(s, a) is a reasonable measure for surprise to promote exploration, and generalize the intrinsic reward design under this measure. However, instead of using a single learning model for P as in the previous approach, we propose using an ensemble of K dynamics models P φ 1, · · ·, P φ K for P, constructing the mixture distribution with the mixing coefficients q i ≥ 0 and K i=1 q i = 1, and using P K in as an estimate for the true unknown P. The rationale behind this is that by using a mixture of several distributions we increase degrees of freedom for modeling the underlying model dynamics and designing a better intrinsic reward. For the j-th model P φ j, j = 1, · · ·, K, we have as in. Thus, for P φ j, the intrinsic reward at timestep t is determined as Furthermore, can be modified to yield a 1-step surprise intrinsic reward as where P φ j l (t) and P φ j l(t)−1 are the j-th model at the update period l corresponding to timestep t and the previous update period l − 1, respectively (l(t) will become clear in the subsection 2.3). Since the mixture model has the increased model order for modeling the underlying dynamics distribution beyond single-mode distributions, we have more freedom to design intrinsic reward. That is, we now have K values, r j t,int (s t, a t, s t+1), j = 1, · · ·, K, for candidates for intrinsic reward. In order to devise a proper use of this extra freedom, we consider the following two objective functions: where τ is a sample trajectory, c is a positive constant, and P (·|s, a) and P (·|s, a) are the true transition probability of an environment and its estimation model, respectively. The first objective function η(π) is the actual desired expected cumulative return for policy π and the second objective functionη(π) is the expected cumulative sum of the actual reward and intentionally-added surprise for policy π. We define π * andπ * as optimal solutions which maximize the objective functions and, respectively. Note that with additional intrinsic reward, the agent learnsπ *. Regarding η(π *) and η(π *), we have the following proposition: Proposition 1. Let η(π) be the actual expected discounted sum of extrinsic rewards defined in. Then, the following inequality holds: where c is a positive constant. Proposition 1 implies that better estimation of the true transition probability P by model P makes η(π *) closer to η(π *), whereπ * is learned based onη(π). Thus, for given P we want to minimize 10) over our estimation model P so that we have a tighter gap between η(π *) and η(π *), and the policyπ * learned with the aid of surprise intrinsic reward well approximates the true optimal policy π *. Regarding this minimization for tight gap, we have the following proposition: Proposition 2. Let P φ i (·|s, a), i = 1,..., K be the ensemble of model distributions, and P (·|s, a) be an arbitrary true transition probability distribution. Then, the minimum of average KLD between P (·|s, a) and the mixture model P = i q i P φ i (·|s, a) over the mixture weights {q 1, · · ·, q K |q i ≥ 0, i q i = 1} is upper bounded by the minimum of average KLD between P and P φ i over {i}: i.e., As seen in the proof of Proposition 2 in Appendix A, ] within the class of linear combinations of the individual surprise values {E (s,a)∼ π * D KL P ||P φ i |(s, a), i = 1, 2, · · ·, K}. Propositions 1 and 2 motivate us to use the minimum among the K available individual surprises for our intrinsic reward to reduce the gap between the actual target reward sum η(π *) of the intrinsic reward-aided learned policyπ * and η(π *) of the true optimal policy π *. Note that with the aid of intrinsic reward, we optimizeη(π) in fact and this makes our policy (try to) approachπ * and the sample trajectory approach (s, a) ∼π *. So, with E (s,a)∼ π * in the right-hand side of replaced simply by the computable instantaneous sample-based value and D KL P ||P φ i |(s, a) replaced by the approximation, we propose using the minimum of r j t,int (s t, a t, s t+1), j = 1, · · ·, K as the single value of intrinsic reward from the K candidates. That is, the agent selects the index j * as where r j t,int is given by, and the intrinsic reward is determined as For the dynamics models P φ 1, · · ·, P φ K, we adopted the fully-factorized Gaussian distributions (; . Then, P K in becomes the class of K-modal Gaussian mixture distributions. We first update the model ensemble P φ 1, · · ·, P φ K and the corresponding mixing coefficients q 1,..., q K. At the beginning, the parameters φ 1, · · ·, φ K are independently initialized, and q i's are set to 1 K for all i = 1, · · ·, K. At every batch period l, in order to jointly learn φ i and q i, we apply maximum-likelihood estimation with an L 2 -norm regularizer with KL constraints : where φ i old is the parameter of the i-th model before the update, α is the regularization coefficient, and κ is a positive constant. To solve this optimization problem with respect to {φ i}, we apply the method based on second-order approximation (a). For the update of {q i}, we apply the method proposed by and set q i as follows: where q old i is the mixing coefficient of the i-th model before the update. For numerical stability, we use the "log-sum-exp" trick for computing as well as L likelihood and ∇ φ i L likelihood. In addition, we apply simultaneous update of all φ i's and q i's, which was found to perform better than one-by-one alternating update of the K models for our problem. Although the proposed intrinsic reward generation method can be combined with general RL algorithms, we here consider the PPO algorithm, which is a popular on-policy algorithm and generates a batch of experiences of length L with every current policy. Let D be the batch of experiences for training the policy, i.e., D = (s t, a t, r total t, s t+1, · · ·, r total t+L−2, s t+L−1, a t+L−1, r total t+L−1), where a t ∼ π θ l (·|s t), s t+1 ∼ P (·|s t, a t), and r total t is the total reward. Here, π θ l is the parameterized policy at the batch period l corresponding to timestep t, · · ·, t + L − 1 (the batch period index l is now included in π θ l for clarity). The total reward at timestep t for training the policy is given by M AX: the maximum index of batch period l, K: the number of dynamics models. 5: Initialize the policy π θ0, the K transition probability models, and the corresponding mixing coefficients q 1, · · ·, q K. 6: Generate trajectories with π θ0 and add them to the initially empty replay buffer M. 7: for Batch period l = 0, · · ·, M AX − 1 do 8: by performing gradient updates for, and update q 1, · · ·, q K by performing iterations with. For this, we draw a batch D of size L randomly and uniformly from M, and perform the updates with minibatches of size L mini drawn from D for N epochs. 9: Collect s t from the environment and a t with the policy π θ l. Collect s t+1 and the extrinsic reward r t from the environment and add (s t, a t, s t+1) to M. Calculate the preliminary intrinsic reward r t,int in. 14: Acquire the normalized intrinsic rewards of the current batch D of size L by using. Train π θ l by using PPO with the total rewards and minibatch size L mini for N epochs. 16: end for where r t (s t, a t, s t+1) is the actual sparse extrinsic reward at timestep t from the environment, r t,int (s t, a t, s t+1) is the normalized intrinsic reward at timestep t, and β > 0 is the weighting factor. Note that the actually-used intrinsic rewardr t,int (s t, a t, s t+1) is obtained by applying normalization to improve numerical stability as where the unnormalized intrinsic reward r t,int (s t, a t, s t+1) is given by. Then, the policy π θ l can be updated at every batch period l with D by following the standard PPO procedure based on the total reward. Summarizing the above, we provide the pseudocode of our algorithm in Algorithm 1, which assumes PPO as the algorithm. Note that the proposed intrinsic reward generation method can also be applied to other RL algorithms. In order to evaluate the performance, we considered sparse reward environments for continuous control. The considered tasks were five environments of Mujoco , OpenAI Gym : Ant, Hopper, HalfCheetah, Humanoid, and Walker2d. To implement sparse reward setting, we adopted the delay method. We first accumulate extrinsic rewards generated from the considered environments for every ∆ timesteps or until the episode ends. Then, we provide the accumulated sum of rewards to the agent at the end of the ∆ timesteps or at the end of the episode, and repeat this process. We set ∆ = 40 for our experiments. All simulations were conducted over 10 fixed random seeds. The y-axis in each figure with the title "Average Return" represents the mean value of the extrinsic returns of the most recent 100 episodes averaged over the 10 random seeds. Each colored band in figure represents the interval of ±σ around the mean curve, where σ is the standard deviation of the 10 instances of data from the 10 random seeds. (Please see Appendix B for detailed description of the overall hyperparameters for simulations.) First, in order to validate the proposed approach in which the intrinsic reward is given by the minimum surprise over the ensemble, we investigated several methods of obtaining a single intrinsic reward value from the multiple preliminary reward values r 1 t,int, · · ·, r K t,int in from the K models: the proposed minimum selection- and other possible methods such as the maximum selection, the average value taking method, and a pure 1-step surprise method with the mixture with the intrinsic reward defined as from the idea that we simply replace the unimodal model P φ in with the mixture model P K in. Figure 1: Impact of single intrinsic reward value extraction for K = 2: minimum selection (min), maximum selection (max), average (avg), and 1-step surprise with ensemble (1-step). Fig. 1 shows the mean performance of the four single intrinsic reward value extraction methods for K = 2: the proposed minimum selection-, the maximum selection, the average method, and the pure 1-step surprise with mixture. As inferred from Propositions 1 and 2, it is seen that the minimum selection yields the best performance in all the environments. (The average method yields similar performance in HalfCheetah and Humanoid.) Interestingly, the proposed approach motivated by Propositions 1 and 2 outperforms the simple mixture replacement in. With this validation, we use the minimum selection method- for all remaining studies. Next, we investigated the impact of the model order K. Since we adopt Gaussian distributions for the dynamics models P φ 1, · · ·, P φ K, the mixture P K in is a Gaussian mixture for given state-action pair (s, a). According to a recent , the model order of Gaussian mixture need not be too large to capture the underlying dynamics effectively in practice. Thus, we evaluated the performance for K = 1, 2, 3, 4. Fig. 2 shows the mean performance as a function of K for the considered sparse reward environments. It is observed that in general the performance improves as K increases, and once the proper model order is reached, the performance does not improve further or degrades a bit due to more difficult model estimation for higher model orders, as expected from our intuition. From this , we found that K = 2 seems reasonable for our model order, so we used K = 2 for all the five environments in the following performance comparison 3.3. yielded similar performance to that of K = 3, so we omitted the curve of K = 4 for simplicity) With the above verification, we compared the proposed method with existing intrinsic reward generation methods by using PPO as the algorithm. We considered the existing intrinsic reward generation methods: curiosity , hashing , information gain approximation (de), and single-model surprise . We also considered the method using intrinsic reward module among the most recent works introduced in Appendix C, which uses delayed sparse reward setup and provides an implementation code. For fair comparison, we used PPO with the same neural network architecture and common hyperparameters, and applied the same normalization technique in for all the considered intrinsic reward generation methods so that the performance difference only from the intrinsic reward generation method. The weighting factor β in between the extrinsic reward and the intrinsic reward should be determined for all intrinsic reward generation methods. Since each of the considered methods yields different scale of the intrinsic reward, we used an optimized β for each algorithm for each environment. In the case of the single-model surprise method and the proposed method, the hyperparameters of the single-model surprise method are tuned to yield best performance and then the proposed method employed the same hyperparameters as the single-model surprise method. We also confirmed that the hyperparameters associated with the other four methods were well-tuned in the original papers (de ; ; ;, and we used the hyperparameters provided by these methods. (Please see Appendix B for detailed description of the hyperparameters for simulations.) Fig. 3 shows the comparison . It is seen that the proposed model-ensemble-based intrinsic reward generation method yields top-level performance. Note that the performance gain by the proposed method is significant in sparse Hopper and sparse Walker2d. Various types of intrinsic motivation such as curiosity, information gain, and surprise have been investigated in cognitive science , and intrinsically-motivated RL has been inspired from these studies. used the information gain on the dynamics model as additional reward based on the notion of curiosity. defined an intrinsic reward's work with the idea of homeostasis in biology. The concept of surprise was exploited to yield intrinsic rewards . In parallel with intrinsically motivated RL, researchers developed model-based approaches for learning itself, in which the agent uses the trained dynamics model and fictitious samples generated from the model for training. suggested using the trained dynamics model to initialize the policy network at the beginning of model-free learning. proposed the policy optimization using trust-region method with a model ensemble, in which multiple prediction models for the next state for given pair of current state and action are constructed and trained by using actual samples, and the policy is trained by multiple fictitious sample trajectories from the multiple models. Our work differs from these works in that we use a model ensemble for the environment transition probability distribution and generates intrinsic reward based on this ensemble of dynamics models to enhance the performance of model-free RL with sparse reward. (Please see Appendix C for more related works.) In this paper, we have proposed a new intrinsic reward generation method based on an ensemble of dynamics models for sparse-reward reinforcement learning. In the proposed method, the mixture of multiple dynamics models is used to better approximate the true unknown transition probability, and the intrinsic reward is designed as the minimum of the intrinsic reward computed from each dynamics model to the mixture to capture the most relevant surprise. The proposed intrinsic reward generation method was combined with PPO to construct a working algorithm. Ablation study has been performed to investigate the impact of the hyperparameters associated with the proposed ensemblebased intrinsic reward generation. Numerical show that the proposed model-ensemble-based intrinsic reward generation method outperforms major existing intrinsic reward generation methods in the considered sparse environments. A PROOFS Proposition 1. Let η(π) be the actual expected discounted sum of extrinsic rewards defined in. Then, the following inequality holds: where c is a positive constant. Proof. The inequality (a) is trivial from the definition of π *, that is, π * is an optimal policy maximizing η(π). The inequality (b) holds since Proposition 2. Let P φ i (·|s, a), i = 1,..., K be the ensemble of model distributions, and P (·|s, a) be an arbitrary true transition probability distribution. Then, the minimum of average KLD between P (·|s, a) and the mixture model P = i q i P φ i (·|s, a) over the mixture weights {q 1, · · ·, q K |q i ≥ 0, i q i = 1} is upper bounded by the minimum of average KLD between P and P φ i over {i}: i.e., Proof. Here, is valid due to the convexity of the KL divergence in terms of the second argument for a fixed first argument. is valid due to the linearity of expectation. is valid since the minimum in the right-hand side of is achieved when we assign all the mass to q i that has the minimum value of E (s,a)∼ π * D KL P ||P φ i |(s, a). (Note that the optimal {q i} in is not the same as the optimal {q i} achieving the minimum in.) Note that each step in the proof is tight except in which the convexity of the KL divergence in terms of the second argument is used. This part involves the function f (x) = − log x for 0 < x ≤ 1 since D KL (p 1 ||p 2) = p 1 (y) log p1(y) p2(y) dy, but the convexity of f (x) = − log x for 0 < x ≤ 1 is not so severe if x is not so close to zero. For the actual implementation, the code implemented by is used. The policy and dynamics models were designed by fully-connected neural networks all of which had two hidden layer of size ). The tanh activation function was used for all of the networks (; . The means of the fully factorized Gaussian dynamics models were the outputs of our networks, and the variances were trainable variables which were initialized to 1 . Other than the variances, all initialization is randomized so that each of dynamics models was set differently . For the implementation of the policy model, our method and all the considered intrinsic reward generation method used the same code for the module method. Although a recent work used TRPO (a) as the baseline learning engine, we used PPO, one of the currently most popular algorithms for continuous action control, as our baseline algorithm. While the same basic hyperparameters as those in the previous work were used, some hyperparameters were tuned for PPO. λ for the GAE method (b) was fixed to 0.95, while the discounting factor was set to γ = 0.99. The batch size L for the training of the policy was fixed to 2048. For the policy update using PPO, the minibatch size L mini was set to 64, the epoch number N 10, the clipping constant 0.2, and the entropy coefficient 0.0. The maximum number of timesteps was 1M for all five environments, and the maximum index of batch period, M AX, is Each of the single-model surprise method, the hashing method and our proposed method requires a replay buffer. The size of the used replay buffer for all these three methods is 1.1M. Before the beginning of the iterations, 2048 × B samples from real trajectories generated by the initial policy were added to the replay buffer. We set B = 40 for our experiments. For the methods not requiring a replay buffer, i.e., Curiosity, Information Gain, Module, and PPO Only, we ran 2048 × B = 81920 timesteps before measuring performance for fair comparison. (Therefore, every x-axis in Fig. 1, 2, and 3 shows the total timesteps of 1.08192M.) For the dynamics model learning, we set the batch size L = 2048, L mini = 64, and N = 4. The optimization was solved based on second-order approximation (a). When K = 1, the optimization reduces to the model learning problem in. , the constraint constant κ in the second-order optimization was well-tuned as 0.001. So, we used this value of κ not only to the case of K = 1 but also to the case of K ≥ 2. We further tuned the value of α in for each environment, and we set α = 0.01. For the information gain method, we need another hyperparameter h which is the weight to balance the original intrinsic reward and the homeostatic regulation term (de). We tuned this hyperparameter for each environment and the used value of h is shown in Table 1. Table 1 summarizes the weighting factor β as well as the hyperparameter h in information gain method. Here, we used the optimized weighting factor β in for each algorithm for each environment. As aforementioned, the major hyperparameters for the proposed model-ensemble-based method are the same as those used for the single-model surprise method. For the intrinsic reward module method, we checked that the provided source code in github reproduced in, as shown in Fig. 4.' Module 0.01' represents the module method with training using the sum of intrinsic reward and scaled extrinsic reward with scaling factor 0.01.' Module 0' represents training using intrinsic reward only (no addition of extrinsic reward). Both methods are introduced in, and we checked reproducibility when B = 0, i.e., we ran 2048 × B = 0 timesteps before measuring performance. We observed that our used code yielded the same as those in. Recent advanced exploration methods can be classified mainly into two categories. One is to generate intrinsic reward explicitly and to train the agent with the total reward which is the sum of the extrinsic reward and the adequately scaled intrinsic reward. The other is indirect methods which do not explicitly generate intrinsic reward. Our work belongs to the first category. There exist many exploration techniques on image spaces (; a; b;) but these works are not directly related to our work here. suggested a new intrinsic reward for sparse and binary extrinsic reward environments, based on sampling additional states from the replay buffer and setting those data as new goals. In their work the policy was based on the input of both state and goal. In our work, on the other hand, the concept of goal is not necessary. A recent work by used a delayed reward environment to propose training the module to generate intrinsic reward apart from training the usual policy. This delayed reward environment for sparse reward setting is different from the previous sparse reward environment based on thresholding, i.e., the agent get non-zero reward when the agent achieves a certain physical quantity (such as the distance from the origin) larger than the predefined threshold. proposed generating intrinsic reward by applying a generative model with the Wasserstein-1 distance. With the concept of state-action embedding, adopted the Jensen-Shannon divergence (JSD) to construct a new variational lower bound of the corresponding mutual information, guaranteeing numerical stability. Our work differs from these two recent works in that we used a model ensemble to generate intrinsic reward. Recent indirect methods can further be classified mainly into two groups: (i) revising the original objective function to stimulate exploration, which exploits intrinsic motivation implicitly, and (ii) perturbing the parameter space of policy. In the first group, proposed that exploration can be stimulated by exploiting novel state-action pairs from the past, and used sparse reward environments by delaying extrinsic rewards. revised the original objective function for training by considering maximization of the divergence between the current policy and recent policies, with an adaptive scaling technique. The concept of dropout was applied to the PPO algorithm to encourage the stochastic behavior of the agent episode-wisely . Convex combination of the target policy and any given policy is considered as a new exploratory policy , which corresponds to solving a surrogate Markov Decision Process, generalizing usual exploration methods such as -greedy or Gaussian noise. In the second group, proposed a goal-based exploration method for continuous control, which alternates generating parameter-outcome pair and perturbing certain parameters based on randomly drawn goal from the outcome space. Recently, inspired by , considered pure exploration MDP without any extrinsic reward with the notion of utility, where utility is based on JSD and the Jensen-Rényi divergence (Rényi et al., 1961). In this work, they consider a number of models for transition function but they used this to compute utility based on average entropy of the multiple models. Our work uses the minimum of surprise from the multiple dynamics models under the existence of explicit reward whether it is extrinsic or intrinsic.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

For sparse-reward reinforcement learning, the ensemble of multiple dynamics models is used to generate intrinsic reward designed as the minimum of the surprise.

The paper designs two algorithms for the stochastic AUC maximization problem with state-of-the-art complexities when using deep neural network as predictive model, which are also verified by empirical studies.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Given the fast development of analysis techniques for NLP and speech processing systems, few systematic studies have been conducted to compare the strengths and weaknesses of each method. As a step in this direction we study the case of representations of phonology in neural network models of spoken language. We use two commonly applied analytical techniques, diagnostic classifiers and representational similarity analysis, to quantify to what extent neural activation patterns encode phonemes and phoneme sequences. We manipulate two factors that can affect the outcome of analysis. First, we investigate the role of learning by comparing neural activations extracted from trained versus randomly-initialized models. Second, we examine the temporal scope of the activations by probing both local activations corresponding to a few milliseconds of the speech signal, and global activations pooled over the whole utterance. We conclude that reporting analysis with randomly initialized models is crucial, and that global-scope methods tend to yield more consistent and interpretable and we recommend their use as a complement to local-scope diagnostic methods. As end-to-end architectures based on neural networks became the tool of choice for processing speech and language, there has been increased interest in techniques for analyzing and interpreting the representations emerging in these models. A large array of analytical techniques have been proposed and applied to diverse tasks and architectures. Given the fast development of analysis techniques for NLP and speech processing systems, relatively few systematic studies have been conducted to compare the strengths and weaknesses of each methodology and to assess the reliability and explanatory power of their outcomes in controlled settings. This paper reports a step in this direction: as a case study, we examine the representation of phonology in neural network models of spoken language. We choose three different models that process speech signal as input, and analyze their learned neural representations. We use two commonly applied analytical techniques: (i) diagnostic models and (ii) representational similarity analysis to quantify to what extent neural activation patterns encode phonemes and phoneme sequences. In our experiments, we manipulate two important factors that can affect the outcome of analysis. One pitfall not always successfully avoided in work on neural representation analysis is the role of learning. Previous work has shown that sometimes non-trivial representations can be found in the activation patterns of randomly initialized, untrained neural networks . Here we investigate the representations of phonology in neural models of spoken language in light of this fact, as extant studies have not properly controlled for role of learning in these representations. The second manipulated factor in our experiments is the scope of the extracted neural activations. We control for the temporal scope, probing both local activations corresponding to a few milliseconds of the speech signal, as well as global activations pooled over the whole utterance. When applied to global-scope representations, both the methods detect a robust difference between the trained and randomly initialized target models. However we find that in our setting, RSA applied to local representations shows low correlations between phonemes and neural activation patterns for both trained and randomly initialized target models, and for one of the target models the local diagnostic classifier only shows a minor difference in the decodability of phonemes from randomly initialized versus trained network. This highlights the importance of reporting analy-sis with randomly initialized models as a baseline. Many current neural models of language learn representations that capture useful information about the form and meaning of the linguistic input. Such neural representations are typically extracted from activations of various layers of a deep neural architecture trained for a target task such as automatic speech recognition or language modeling. A variety of analysis techniques have been proposed in the academic literature to analyze and interpret representations learned by deep learning models of language as well as explain their decisions; see and for a review. Some of the proposed techniques aim to explain the behavior of a network by tracking the response of individual or groups of neurons to an incoming trigger (e.g., ;). In contrast, a larger body of work is dedicated to determining what type of linguistic information is encoded in the learned representations. This type of analysis is the focus of our paper. Two commonly used approaches to analyzing representations are Probing techniques, or diagnostic classifiers, i.e. methods which use the activations from different layers of a deep learning architecture as input to a prediction model (e.g., ; ;); Representational Similarity Analysis (RSA) borrowed from neuroscience and used to correlate similarity structures of two different representation spaces (; ;) We use both techniques in our experiments to systematically compare their output. Research on the analysis of neural encodings of language has shown that in some cases, substantial information can be decoded from activation patterns of randomly initialized, untrained recurrent networks. It has been suggested that the dynamics of the network together with the characteristics of the input signal can in non-random activation patterns . Using activations generated by randomly initialized recurrent networks has a history in speech recognition and computer vision. Two betterknown families of such techniques are called Echo State Networks (ESN) and Liquid State Machines (LSM) . The general approach (also known as reservoir computing) is as follows: the input signal is passed through a randomly initialized network to generate a nonlinear response signal. This signal is then used as input to train a model to generate the desired output at a reduced cost. We also focus on representations from randomly initialized neural models but do so in order to show how training a model changes the information encoded in the representations according to our chosen analysis methods. Since the majority of neural models of language work with text rather than speech, the bulk of work on representation analysis has been focused on (written) word and sentence representations. However, a number of studies analyze neural representations of phonology learned by models that receive a speech signal as their input. As examples of studies that track responses of neurons to controled input, analyze local representations acquired from a deep model of phoneme recognition and show that both individual and groups of nodes in the trained network are selective to various phonetic features, including manner of articulation, place of articulation, and voicing. use a similar approach and suggest that phonemes are learned as an intermediate representation for predicting graphemes, especially in very deep layers. Others predominantly use diagnostic classifiers for phoneme and grapheme classification from neural representations of speech. In one of the their experiments use a linear classifier to predict phonemes from local activation patterns of a grounded language learning model, where images and their spoken descriptions are processed and mapped into a shared semantic space. Their show that the network encodes substantial knowledge of phonology on all its layers, but most strongly on the lower recurrent layers. use diagnostic classifiers to study the encoding of phonemes in an end-to-end ASR system with convolutional and recurrent layers, by feeding local (frame-based) representations to an MLP to predict a phoneme label. They show that phonological information is best represented in lowest input and convolutional layers and to some extent in low-to-middle recurrent layers. extend their previous work to multiple languages (Arabic and English) and different datasets, and show a consistent pattern across languages and datasets where both phonemes and graphemes seem to be encoded best in the middle recurrent layers. None of these studies report on phoneme classification from randomly initialized versions of their target models, and none use global (i.e., utterancelevel) representations in their analyses. In this section we first describe the speech models which are the targets of our analyses, followed by a discussion of the methods used here to carry out these analyses. We will release source code to run all our analyses on the publication of this paper. We tested the analysis methods on three target models trained on speech data. The first model is a transformer model trained on the automatic speech recognition (ASR) task. More precisely, we used a pretrained joint CTC-Attention transformer model from the ESPNet toolkit , trained on the Librispeech dataset . 1 The architecture is based on the hybrid CTC-Attention decoding scheme presented by but adapted to the transformer model. The encoder is composed of two 2D convolutional layers (with stride 2 in both time and frequency) and a linear layer, followed by 12 transformer layers, while the decoder has 6 such layers. The convolutional layers use 512 channels, which is also the output dimension of the 1 We used ESPnet code from commit 8fdd8e9 with the pretrained model available from https://drive.google. com/open?id=1BtQvAnsFvVi-dp_qsaFP7n4A_ 5cwnlR6 linear and transformer layers. The dimension of the flattened output of the two convolutional layers (along frequencies and channel) is then 20922 and 10240 respectively: we omit these two layers in our analyses due to their excessive size. The input to the model is made of a spectrogram with 80 coefficients and 3 pitch features, augmented with the SpecAugment method . The output is composed of 5000 SentencePiece subword tokens . The model is trained for 120 epochs using the optimization strategy from , also known as Noam optimization. Decoding is performed with a beam of size 60 for reported word error rates of 2.6% and 5.7% on the test set (for the clean and other subsets respectively). The Visually Grounded Speech (VGS) model is trained on the task of matching images with their corresponding spoken captions, first introduced by and. We use the architecture of which implemented several improvements over the RNN model of, and train it on the Flickr8K Audio Caption Corpus . The speech encoder consists of one 1D convolutional layer (with 64 output channels) which subsamples the input by a factor of two, and four bidirectional GRU layers (each of size 2048) followed by a self-attention-based pooling layer. The image encoder uses features from a pre-trained ResNet-152 model followed by a linear projection. The loss function is a margin-based ranking objective. we trained the model using the Adam optimizer with a cyclical learning rate schedule . The input are MFCC features with total energy and delta and double-delta coefficients with combined size 39. This model is a middle ground between the two previous ones. It is trained as a speech recognizer similarly to the transformer model but the architecture of the encoder follows the RNN-VGS model (except that the recurrent layers are one-directional in order to fit the model in GPU memory). The last GRU layer of the encoder is fed to the attention-based decoder from , here composed of a single layer of 1024 GRU units. The model is trained with the Adadelta optimizer . The input features are identical to the ones used for the VGS model; it is also trained on the Flickr8k dataset spoken caption data, using the original written captions as transcriptions. The architecture of this model is not optimized for the speech recognition task: rather it is designed to be as similar as possible to the RNN-VGS model while still performing reasonably on speech recognition. We consider two analytical approaches: • Diagnostic model is a simple, often linear, classifier or regressor trained to predict some information of interest given neural activation patterns. To the extent that the model successfuly decodes the information, we conclude that this information is present in the neural representations. • Representational similarity analysis (RSA) is a second-order approach where similarities between pairs of some stimuli are measured in two representation spaces: e.g. neural activation pattern space and a space of symbolic linguistic representations such as sequences of phonemes or syntax trees (see . Then the correlation between these pairwise similarity measurements quantifies how much the two representations are aligned. The diagnostic models have trainable parameters while the RSA-based models do not, except when using a trainable pooling operation. We also consider two ways of viewing activation patterns in hidden layers as representations: • Local representations at the level of a single frame or time-step; • Global representations at the level of the whole utterance. Combinations of these two facets give rise to the following concrete analysis models. Local diagnostic classifier. We use single frames of input (MFCC or spectrogram) features, or activations at a single timestep as input to a logistic diagnostic classifier which is trained to predict the phoneme aligned to this frame or timestep. Local RSA. We compute two sets of similarity scores. For neural representations, these are cosine similarities between neural activations from pairs of frames. For phonemic representations our similarities are binary, indicating whether a pair of frames are labeled with the same phoneme. Pearson's r coefficient computed against a binary variable, as in our setting, is also known as point biserial correlation. Global diagnostic classifier. We train a linear diagnostic classifier to predict the presence of phonemes in an utterence based on global (pooled) neural activations. For each phoneme j the predicted probability that it is present in the utterance with representation h is denoted as P(j|h) and computed as: where Pool is one of the pooling function in Section 3.2.1. Global RSA. We compute pairwise similarity scores between global (pooled; see Section 3.2.1) representations and measure Pearson's r with the pairwise string similarities between phonemic transcriptions of utterances. We define string similarity as: where | · | denotes string length and Levenshtein is the string edit distance. The representations we evaluate are sequential: sequences of input frames, or of neural activation states. In order to pool them into a single global representation of the whole utterance we test two approaches. Mean pooling. We simply take the mean for each feature along the time dimension. Attention-based pooling. Here we use a simple self-attention operation with parameters trained to optimize the score of interest, i.e. the RSA score or the error of the diagnostic classifier. The attentionbased pooling operator performs a weighted average over the positions in the sequence, using scalar weights. The pooled utterance representation Pool(h) is defined as: with the weights α computed as: where w are learnable parameters, and h t is an input or activation vector at position t. 2 For RSA we use Pearson's r to measure how closely the activation similarity space corresponds to the phoneme or phoneme string similarity space. For the diagnostic classifiers we use the relative error reduction (RER) over the majority class baseline to measure how well phoneme information can be decoded from the activations. Effect of learning In order to be able to assess and compare how sensitive the different methods are to the effect of learning on the activation patterns, it is important to compare the score on the trained model to that on the randomly initialized model; we thus always display the two jointly. We posit that a desirable property of an analytical method is that it is sensitive to the learning effect, and that the scores on trained versus randomly initialized models are clearly separated. Coefficient of partial determination Correlation between similarity structures of two representational spaces can, in principle, be partly due to the fact that both these spaces are correlated to a third space. For example, were we to get a high value for global RSA for one of the top layers of the RNN-VGS model, we might suspect that this is due to the fact that string similarities between phonemic transcriptions of captions are correlated to visual similarities between their corresponding images, rather than due to the layer encoding phoneme strings. In order to control for this issue, we can carry out RSA between two spaces while controling for the third, confounding, similarity space. We do this by computing the coefficient of partial determination defined as the relative reduction in error caused by including variable X in a linear regression model 2 Note that the visually grounded speech models of; Chrupała; use similar mechanisms to aggregate the activations of the final RNN layer; here we use it as part of the analytical method to pool any sequential representation of interest. A further point worth noting is that we use scalar weights αt and apply a linear model for learning them in order to keep the analytic model simple and easy to train consistently. for Y: where e Y ∼X+Z is the sum squared error of the model with all variables, and e Y ∼Z is the sum squared error of the model with X removed. Given the scenario above with the confounding space being visual similarity, we identify Y as the pairwise similarities in phoneme string space, X as the similarities in neural activation space, and Z as similarities in the visual space. The visual similarities are computed via cosine similarity on the image feature vectors corresponding to the stimulus utterances. All analytical methods are implemented in Pytorch . The diagnostic classifiers are trained using Adam with learning rate schedule which is scaled by 0.1 after 10 epochs with no improvement in accuracy. We terminate training after 50 epochs with no improvement. Global RSA with attention-based pooling is trained using Adam for 60 epochs with a fixed learning rate (0.001). For all trainable models we snapshot model parameters after every epoch and report the for the epoch with best validation score. In all cases we sample half of the available data for training (if applicable), holding out the other half for validation. Sampling data for local RSA. When computing RSA scores it is common practice in neuroscience research to use the whole upper triangular part of the matrices containing pairwise similarity scores between stimuli, presumably because the number of stimuli is typically small in that setting. In our case the number of stimuli is very large, which makes using all the pairwise similarities computationally taxing. More importantly, when each stimulus is used for computing multiple similarity scores, these scores are not independent, and score distribution changes with the number of stimuli. We therefore use an alternative procedure where each stimulus is sampled without replacement and used only in a single similarity calculation. Figures 1-3 display the outcome of analyzing our target models. All three figures are organized in a 2 × 3 matrix of panels, with the top row showing the diagnostic methods and the bottom row the RSA methods; the first column corresponds to local scope; column two and three show global scope with mean and attention pooling respectively. The data points are displayed in the order of the hierarchy of layers for each architecture, starting with the input (layer id = 0). In all the reported experiments, the score of the diagnostic classifiers corresponds to relative error reduction (RER), whereas for RSA we show Pearson's correlation coefficient. Figure 4 shows the of global RSA with mean pooling on the RNN-VGS target model, while controling for visual similarity as a confound. We will discuss the patterns of observed for each model separately in the following sections. As can be seen in Figure 1, most reported experiments (with the exception of the local RSA) suggest that phonemes are best encoded in pre-final layers of the deep network. The also show a strong impact of learning on the predictions of the analytical methods, as is evident by the difference between the performance using representations of the trained versus randomly initialized models. Local RSA shows low correlation values overall, and does not separate the trained versus random conditions well. Most experimental findings displayed in Figure 2 suggest that phonemes are best encoded in RNN layers 3 and 4 of the VGS model. They also show that the representations extracted from the trained model encode phonemes more strongly than the ones from the random version of the model. However, the impact of learning is more salient with global than local scope: the scores of both local classifier and local RSA on random vs. trained representations are close to each other for all layers. For the global representations the performance on trained representations quickly diverges from the random representations from the first RNN layer onward. Furthermore, as demonstrated in Figure 4, for top RNN layers of this architecture, the correlation between similarities in the neural activation space and the similarities in the phoneme string space is not solely due to both being correlated to visual similarities: indeed similarities in activation space contribute substantially to predicting string similarities, over and above the visual similarities. The overall qualitative patterns for this target model are the same as for RNN-VGS. The absolute scores for the global diagnostic variants are higher, and the curves steeper, which may reflect that the objective for this target model is more closely aligned with encoding phonemes than in the case of RNN-VGS. Here we discuss the impact of each factor in the outcome of our analyses. Choice of method. The choice of RSA versus diagnostic classifier interacts with scope, and thus these are better considered as a combination. Specifically, local RSA as implemented in this study shows only weak correlations between neural activations and phoneme labels. It is possibly related to the range restriction of point biserial correlation with unbalanced binary variables. Impact of learning. Applied to the global representations, both analytical methods are equally sensitive to learning. The on random vs. trained representations for both methods start to diverge noticeably from early recurrent layers. The separation for the local diagnostic classifiers is weaker for the RNN models. Representation scope. Although the temporal scale of the extracted representations (especially those of spoken language) has not received much attention and scrutiny, our experimental findings suggest that this is an important choice. Specifically, global representations seem to be more sensitive to learning, and more consistent across different analysis methods. Results with attention-based learned pooling are in general more erratic than with mean pooling. This reflects the fact that analytical models which incorporate learned pooling are more difficult to optimize and require more careful tuning compared to mean pooling. Given the above findings, we now give tentative recommendations regarding how to carry out representational analyses of neural models. • Analyses should be run on randomly initialized target models. We saw that in most cases scores on such models are substantially above zero, and in some cases relatively close to scores on trained models. • It is unwise to rely on a single analytical approach, even a widely used one such as the local diagnostic classifier. With solely this method we would have concluded that, in RNN models, learning has only a weak effect on the encoding of phonology. • Global methods applied to pooled representations should be considered as a complement to standard local diagnostic methods. In our experiments they show more consistent . We carried out a systematic study of analysis methods for neural models of spoken language and offered some suggestions on best practices in this endeavor. Nevertheless our work is only a first step, and several limitations remain. The main challenge is that it is often difficult to completely control for the many factors of variation present in the target models, due to the fact that a particular objective function, or even a dataset, may require relatively important architectural modifications. In future we plan to sample target models with a larger number of plausible combinations of factors. Likewise, a choice of an analytical method may often entail changes in other aspects of the analysis: for example unlike a global diagnostic classifier, global RSA captures the sequential order of phonemes. In future we hope to further disentangle these differences.

Represent the TLDR for finding the corresponding scientific article

We study representations of phonology in neural network models of spoken language with several variants of analytical techniques.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.


Question: Represent the article for finding a single-sentence TLDR

Question: Super Resolution (SR) is a fundamental and important low-level computer vision (CV) task. Different from traditional SR models, this study concentrates on a specific but realistic SR issue: How can we obtain satisfied SR from compressed JPG (C-JPG) image, which widely exists on the Internet. In general, C-JPG can release storage space while keeping considerable quality in visual. However, further image processing operations, e.g., SR, will suffer from enlarging inner artificial details and in unacceptable outputs. To address this problem, we propose a novel SR structure with two specifically designed components, as well as a cycle loss. In short, there are mainly three contributions to this paper. First, our research can generate high-qualified SR images for prevalent C-JPG images. Second, we propose a functional sub-model to recover information for C-JPG images, instead of the perspective of noise elimination in traditional SR approaches. Third, we further integrate cycle loss into SR solver to build a hybrid loss function for better SR generation. Experiments show that our approach achieves outstanding performance among state-of-the-art methods. With the marvelous achievement of deep learning (DL) in computer vision (CV), Super Resolution (SR) attracts much attention for its crucial value as the basis of many high-level CV tasks ). Deep learning Super Resolution (DL-SR) algorithms (; ; ; c; b) strive for finding the complex nonlinear mapping between low resolution (LR) images and their high resolution (HR) counterparts. However, the learned model only reflects the inverse of down-scaled mapping, which is used to obtain LR images from their HR fathers. In other words, if there are some spots/stains in LR inputs, the SR model will treat them as inherent elements, and the corresponding SR outputs will enlarge these undesirable details. In reality, on the Internet, JPG compression is probably the most commonly used pattern for storage space reduction. That is to say, the LR image will be further processed into a compressed JPG (C-JPG) image. The quality of C-JPG will greatly drop, and the compression may yield unpleasant artifacts, for example, the presence of obvious partition lines, which vastly deteriorates the overall visual feeling. Hence, directly solving high-level CV tasks with these C-JPG images will lead to poor performance. In this paper, we propose a lossless SR model to obtain images with satisfying quality from the low-quality inputs (C-JPG). The deterioration in C-JPG makes the SR processing a huge challenge. In this paper, we focus on the more realistic C-JPG SR problem. Many SR methods regarding to the real-world condition images have been already developed, such as Zhang et al. (2018a);. Among them, some models regard the noise as a kernel estimating problem which can be solved by addictive Gaussian noises. However, the distribution of most real images are inconsistent with the hypothetical Gaussian distribution. Taking C-JPG images as a example, the image compression operation is related to decreasing information from original image instead of adding specific noises. Other models learn the related information from irrelevant LR-HR images to obtain similar representations by unsupervised strategy. All of them cannot solve the problem well. In general, most LR images are produced through performing traditional interpolation method (mostly bicubic) on their HR fathers. The SR training process should recover this down-scaled mapping in a reverse manner. Referring to our C-JPG SR issue, when searching images from Google, a lot of unpleasant details are displayed, especially in the edges of objects. However, the low quality of image makes former SR methods fail to generate applicable images. As shown in Fig. 1, it is shown that the SR generations of traditional bicubic interpolation, leading SR algorithm RCAN, and RCAN with pre-denoising input all demonstrate poor quality with the low quality C-JPG inputs. Damaged grids are apparently enlarged by the approaches designed for traditional non-JPG datasets. More specialized analysis can be found in the research of Köhler et al.. Note that the image pairs with fixed down-scaled kernel have been successfully learnt by SR models, such as SRGAN , EDSR , and RDN (c). In this study, we deliberately build a more complicated dataset by adding JPG format LR images to the training data. To be specific, we have three kinds of training inputs: C-JPG LR, LR, and HR images. The whole training process includes two separate functional components: missing detail recuperative part (JPG recovering stage) and SR mapping learning part (SR generating stage). In order to remove ring, checkerboard effects, as well as other noise, the former half sub-model is trained with pre-processed C-JPG LR images as inputs, and original LR ones as the supervised information. The function of this stage is to recover LR image from its compression counterpart. Hence, the outputs (LR(C − JP G)) of the first part are greatly improved and free of partition lines phenomenon. Based on these improved LR images, the latter sub-model continues to learn the mapping between (LR(C −JP G)) and HR. Therefore, an integrated pipeline for SR representation between C-JPG and HR images is achieved through the jointly two sub-models. In short, there are mainly three contributions in this study: • Our research can be regarded as an universal SR method that generates SR images from C-JPG inputs, which is empirically proved to be more difficult than SR with non-JPG inputs. • We regard this specific SR task as a recovering information process for the inputs, compared with the former denoising assumption including down-sampling and degradation parts. In this viewpoint, a recovering model is firstly introduced to generate satisfied intermediates from C-JPG inputs. • We further propose an integrated SR model training pipeline with two-level data, i.e., C-JPG LR and LR images, as well as a new integrated loss function. The experimental demonstrate our method can surpass traditional SR models. Single image super-resolution (SISR) is a widely concerned task. A bundle of researches is developed to resolve this challenge, and great achievements have been achieved. At first, researchers obtain SR generations with an estimated pixel-value interpolation function. For instance, Bicubic regards pixel spreading in a limited patch as a specified linear parametric function. However, these methods are prone to generate fuzzy SR . Later, depending on external and internal dataset, two different classes of SR methods (external and internal based models) are developed. In detail, external models learn the mapping upon many image pairs with various learning algorithms, such as nearest neighbor , kernel ridge regression , sparse learning (;, cluster learning (;, manifold learning , and neural networks . Internal models leverage similar patches within the image or across scales, and the approaches mainly focus on obtaining more self-similar patches, such as;;;;. Recently, deep learning method, which has achieved excellent performance in many high-level CV tasks, is introduced to SR. DL-SR focuses on learning the relationship between LR image and its corresponding HR one. To our knowledge, the most cutting-edge SR algorithms DL-SR, thanks to the powerful learning ability of deep learning. In detail, firstly propose end-to-end DL-SR network, SRCNN. Though there are only three convolutional layers in SRCNN, it greatly surpasses the former traditional methods in peak-signal-to-nosie-ratio (PSNR). Ever since SRCNN, a number of novel models arise for better generation ability, such as FSRCNN , VDSR , SRGAN , EDSR , RDN (c), DBPN , ESRGAN , and RCAN (b). In DL-SR, the goal is developed into two different aspects, i.e., higher accuracy factor (high PSNR and SSIM values) and more photo-realistic (better feature similarity) in visual sense. Based on this consideration, various structures are designed to extract more crucial features from the LR image. For example, VDSR introduces the global residual learning and trains a multi-scale model through sharing parameters across different scales. Inspired by the stunning performance of ResNet , the SRGAN replaces most basic convolutional blocks with residual ones. Moreover, it combines MSE loss, GANs loss loss, perceptual loss , and global loss for photo-realistic SR generations. Experiments show the great power in generating photo-realistic images. Based on the generator of SRGAN, EDSR removes all batch normalization to reduce the computation complexity, and replaces MSE loss with L1 norm loss. Moreover, the model increases the channel number to 256 in each layer and achieves state-of-the-art in NTIRE2017. However, there are over 40 million parameters in EDSR. Benefit from the joint of densely architecture, such as Densely Network , RDN (c) surpasses EDSR in accuracy. RCAN (b) upgrades the densely block to residual in residual (RIR) block and combines with channel attention mechanism. To our knowledge, RCAN is the leading method in accuracy pursuing SISR. Meanwhile, some researchers aim to build a lightweight and accurate model, such as CARN , s-LWSR, and FALSR . These methods try to reduce parameters and operations while keeping the decent performance. CARN leverages the cascading mechanism upon a residual block for obtaining better feature representation from multi-level layers. At the same time, to decrease the model size, some residual blocks are replaced by group convolution parts, which is similar to depth-wise convolution in MobileNets . The s-LWSR tries to introduce a more flexible SR model with an additional information pool. Inspired by the NAS , FALSR automatically searches a desirably lightweight and accurate SR model without human-designing. All methods mentioned above try to solve the SR problem on algorithmic LR-HR image pairs. However, the scale-kernel and degradation function are usually undefined in real-world, which in the SR image accompanied by amplified noise. To overcome this weakness, ZSSR proposes an unsupervised Zero-shot SR method. Firstly, the method generates many derived HR and LR images from the input and builds a simple CNN network to learn the mapping between pre-proposed LR image and its HR counterpart. So far as we know, ZSSR greatly surpasses other supervised SR models on the non-ideal condition mentioned above. Learning from propose a cycle-in-cycle structure called CincGAN to address the blind SR issue. Inputs with noise are firstly processed to generate intermediate LR images with less noise. Then, these LR images are jointly restored and scaled up with the help of an extra pre-trained SR model. Besides, proposes the multi-gram losses in UMGSR to generate perceptual satisfying images. For supervised method, propose a new pipeline to learn to generate SR images from the generated realistic data of raw image. Challenge Formulation In general, SISR issue can be formulated as: y = F(x) + z, where y and x represent HR and LR images respectively. Here, F is the SR mapping in the hypothesis set space (e.g., neural networks), and z denotes the additional irrelevant information, such as noises and blurred details. Normally, most SISR models are trained on the standard dataset, where the input LR is directly downsampled from HR by chosen method (e.g., Bicubic). As a , we assume that z equals to zero in the dataset. In this paper, we investigate a specific situation where the LR is low-quality C-JPG image. This situation commonly exists on the Internet, due to reducing storage or protecting copyright. Since LR inputs are firstly deteriorated to low-quality images, we redefine the above SISR formulation as: y = F(x + w). Here, w refers to the missing information due to the compression. Based on the specialization of low-quality JPG-SR issue, our model can be separated into two functional stages: JPG recovering and SR generating. Hence, our SISR formulation consists in two stages: As shown in Figure 2, training data are: C-JPG LR images (LR(C − JP G)); normal LR images (LR(normal)), which provide supervised information in the first stage; and HR images, which are the supervised information in the second stage. Stage I: JPG Recovering. For low quality JPG images, the compression operation discards most of useful details. As a , to rebuild details according to the original LR image plays a key role in this stage. To this end, we design a specialized network to learn the mapping between LR(C−JP G) and LR(normal). The comparison of LR(C − JP G) and LR(normal) is shown in Figure 3. Given the original LR image, we firstly re-save it to its low quality JPG version as our training material. In detail, the JPG compression abandon information in Cb/Cr channel on every 8 × 8 patch. As shown in Figure 3, the size of the original LR image is 82K bytes, while its C-JPG version (80 percent lower in quality) (b) C-JPG (20% quality) 12K Figure 3: The comparison of compressed JPG image and its LR counterpart. We choose a typical building image from Urban100 dataset . Given the LR image, we leverage Pillow to compress it with 80 percent lower in quality. Benefiting from the compression process, C-JPG takes only 12K storage space, compared to the 82K LR image. However, the visual feeling of C-JPG displays more unpleasant details than LR. is only 12K bytes. However, the quality of image are greatly decreased. This phenomenon can be clearly inferred from the visual contrast in Figure 3 that mussy details are generated in C-JPG image, such as the irregular shape of windows and the blurry object edge. Normally, to achieve better balance between storage cost and customer satisfaction, many websites provide images with partialquality JPG ones. When performing SR operation on these images, the effect of these deuterogenic noises is enlarged, leading to aggressively unpleasant feeling in visual. Since this recovering task shares similar goal with SR task, that is, trying to restore accurate information in pixel level, we employ an effective SR model, s-LWSR, to address the details recovering issue. Learning from stage I in Figure 2, LR(C-JPG) images firstly go through the recovering model. In detail, a simple convolutional layer transfers the basic RGB layers into fixed numbers. The main body part of recovering stage shares the same structure with our SR generating: stacking 26 residual blocks with additional information pool to intensively extract information. Experiments in s-LWSR have proven the powerful learning ability of this architecture. Finally, a convolution layer inversely realize the transformation between middle layers and final 3 RGB layers. Eventually, the recovered LR image for C-JPG achieves visual satisfactory in certain degree, and the artificial trace appears smooth, which greatly reduces the pixel inconsistency. Furthermore, SR generation will benefit from these better quality LR input deriving from C-JPG in stage I. Stage II: SR Generating. Because SR generating acts as a complementary task and is not the main contribution in our paper, we just bring in a state-of-the-art SR method called s-LWSR in our model. As is mentioned, s-LWSR leverages the combination of multi-level information from the front half of model. More details are shown in Figure 4. Specifically, chosen layers are stacked to form the information pool, which transfers the combination of abundant low-level information to high-level layers through a series of activation operations. However, in the last part, a specified upsampling block is applied to scale up the image to the ideal one. Normally, 2×, 3×, 4×, and 8× are the most frequent SR tasks. Some models, such as MDSR , RCAN (b), can deal with all these scale-up tasks upon only one block with sharing parameters. In this paper, following most SR algorithms, we deal with single scale-up factor 4× SR problem. As a , the upsampling stage in our model contains two subpixel interpolation layers as in ESPCN . In detail, each upsample layer performs one 2× scale-up. Finally, we obtain SR generations with satisfactory visual quality. Figure 4: The s-LWSR architecture. The framework of our approach includes two typical tasks: JPG recovering and SR generating, involving three similarity loss functions as follows. In stage I, each pixel in C-JPG image corresponds to one counterpart in non-JPG LR image. The aim is to recover accurate value in pixels. As a , we choose the following L1 loss: where W and H refer to the width and height of image respectively. G Rec is the transformation of stage I. On the basis of pre-processed C-JPG, we further scale the intermediate (LR rec) up to the default size of HR image through SR operation. Since our SR model pursues a better accurate SR generation, loss function in stage II inherits from that of former outstanding SR models: where s is the scale factor. Different from normal SR task, C-JPG SR involves more difference between the input C-JPG and its final supervised HR image. Inspired by marvelous development of unsupervised style-to-style learning, like CycleGAN, CinCGAN , and WESPE , we apply cycle loss as the third component in our final loss function. Taking CycleGAN as an example, images in different domains are transferred with inconsistent contents. Cycle loss can keep same content of the original image, while changing the style features, such as colors and texture. In this paper, SR generations are downscaled to input size to further compare with the corresponding non-JPG LR images. Here, we also use L1 loss, and Bic refers to bicubic interpolation downsampling. The final loss function is the combination of three L1 losses above with equal weight: In this section, we first describe the implementation details of our model. Then, we further analyze effects of all the proposed strategies in our model by ablation study. Finally, we compare our method with other leading SR models to validate the advantage. Dataset and Pre-processing Models. As mentioned in the modeling part, the training process is based on compressed LR (C-JPG), LR, and HR images. Following other leading SR methods, we choose DIV2K as our training dataset. There are 800 HR-LR training image pairs and 100 validation images. To get the C-JPG LR images, all LR images are firstly processed by pillow package in python. In order to exhibit more contrast with different compression levels, we respectively save JPG images with 20% and 50%. Moreover, we pre-process C-JPG with MatLab2018, which contains the default deblurring model. All images (C-JPG LR and LR) are processed by data augmentation strategy which randomly rotates images by 90 •, 180 •, 270 • and flips horizontally., BSD100 , and Urban100. Moreover, in order to prove the practical effectiveness, we perform our approach on the downloaded JPG images from the Internet. For both of the two stages, we apply Adam optimizer , where β 1 = 0.9 and β 2 = 0.999. The learning rate is initialized as 1 × 10 −4 and half decreased every 2 × 10 2 iterations of back-propagation. Our model is trained over 1 × 10 3 times until reaching its convergence. We implement our model by Pytorch with a Titan 2080Ti GPU. Evaluation Standards. For accurate related SR models, the most common evaluation standards are. Aligning with these algorithms, we evaluate our SR on Y channel (i.e., luminance) of transformed YCbCr space instead of direct RGB channels. To clarify the effectiveness of the recovering stage in our model, we apply ablation study to our model. Two strategies are employed. First, the recovering stage is removed from the model. Second, we use denoising with Matlab2018 to replace the recovering stage. The recovering stage plays the crucial role in our model. In this part, we remove it from the model. As a , C-JPG inputs directly go through the stage II: SR generating. Correspondingly, we just train the model with L 2 L1 loss. The remaining architecture is the same. As shown in Fig.5, there are a lot of undesired artifacts compared with C-JPG model. Both of the PSNR and the SSIM decrease a large number. In fact, single model is hard to solve SR and recovering the C-JPG image simultaneously. The huge difference in supervised information leads to large variance among middle layers, which represent ideal details serving the final SR model. More comparisons are shown in the Appendix. To further analyze the learning ability of our recovering stage, we replace it by the denoising code in Matlab2018. The C-JPG images are firstly processed to a set of clean intermediate inputs: inter. Then, we train the SR model with I inter and I HR pairs. The corresponding are shown in Fig.5. It clearly illustrates that pre-denoisng operation can remove many artifacts, but it is still obviously worse than our model. In our opinion, more details should be recovered instead of just removing noise. Denoising only makes these C-JPG inputs clearer, while recovering brings accurate information to C-JPG images. , BSD100 , and Urban100 . Because our research is SR generation from compressed JPG image, we firstly process images of the mentioned datasets to 20% quality JPG ones. Then, we compare our model with the sate-of-the-art SR methods: EDSR , DBPN , and RCAN (b). We present all comparison in Figure 5 and In this paper, we propose a lossless SISR model for low-quality C-JPG images which is extensive used on the Internet. Based on our redefined C-JPG SR pipeline, two functional stages are integrated to fulfill the SR task on C-JPG images. In addition, we employ cycle loss to guarantee the consistency after above two stages. The intensive experiments demonstrate that our model can learn capable representations of LR inputs for C-JPG SR task and outperform other cutting edges in SISR. More exploration should be executed on other CV tasks with C-JPG images inputs as the future work.

We solve the specific SR issue of low-quality JPG images by functional sub-models.

Represent the short summary for finding the corresponding scientific article (long)

We introduced a shallow featrue extraction network with a large receptive field for stereo matching tasks, which uses a simple structure to get better performance.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Keyword spotting—or wakeword detection—is an essential feature for hands-free operation of modern voice-controlled devices. With such devices becoming ubiquitous, users might want to choose a personalized custom wakeword. In this work, we present DONUT, a CTC-based algorithm for online query-by-example keyword spotting that enables custom wakeword detection. The algorithm works by recording a small number of training examples from the user, generating a set of label sequence hypotheses from these training examples, and detecting the wakeword by aggregating the scores of all the hypotheses given a new audio recording. Our method combines the generalization and interpretability of CTC-based keyword spotting with the user-adaptation and convenience of a conventional query-by-example system. DONUT has low computational requirements and is well-suited for both learning and inference on embedded systems without requiring private user data to be uploaded to the cloud. A more natural method for custom wakeword detection is "query-by-example" keyword spotting. In a query-by-example system, the user teaches the system the desired wakeword by recording a few training examples, and the keyword spotter uses some form of template matching to compare incoming audios with these training examples to detect the wakeword. In dynamic time warping (DTW)-based keyword spotting, for example, a variable-length sequence of feature vectors, such as Mel-filterbank cepstral coefficients (MFCCs) BID6 or phoneme posteriors [8; 9; 10], is extracted from the query audio and test audio, and the DTW alignment score between query and test is used as the detection score. Other template-matching approaches compare fixed-length feature vectors, such as the final hidden states of a pre-trained recurrent neural network (RNN) BID10 or the output of a Siamese network [12; 13], using the cosine distance. Systems that use template matching are difficult to interpret, and therefore difficult to debug and optimize. For instance, it is hard to say why a keyword is incorrectly detected or not detected in a system based on dynamic time warping (DTW) simply by inspecting the DTW matrix. Likewise, the hidden states of RNNs can sometimes be interpreted (c.f. BID13, BID14), but this is currently only possible with some luck and ingenuity. In contrast, a CTC-based model is easy to interpret. The wakeword model itself is interpretable: it consists simply of a human-readable string, like "ALEXA" or "AH L EH K S AH", rather than a vector of real numbers. Inference is interpretable because the neural network outputs are peaky and sparse (the "blank" symbol has probability ≈1 at almost all timesteps), so it is easy to determine what the network "hears" for any given audio and whether it hears the labels of the wakeword BID15. This is a useful property because it enables the system designer to take corrective action. For instance, one might identify that a particular label is not well-recognized and augment the training data with examples of this label. In this paper, we propose a new method for custom wakeword detection that combines the convenience and speaker-adaptive quality of query-by-example methods with the generalization power and interpretability of CTC-based keyword spotting. We call our method "DONUT", since detection requires O(N U T) operations given the neural network output, where N, U, and T are small numbers defined later in the paper. The method works as follows: the user records a few training examples of the keyword, and a beam search is used to estimate the labels of the keyword. The algorithm maintains an N -best list of label sequence hypotheses to minimize the error that may be incurred by incorrectly estimating the labels. At inference time, each hypothesis is scored using the forward algorithm, and the hypothesis scores are aggregated to obtain a single detection score. In the rest of the paper, we describe the proposed method and show that it achieves good performance compared with other query-by-example methods, yet generates easily interpretable models and matches the user's pronunciation better than when the label sequence is supplied through text. This section describes the model, learning, and inference for DONUT FIG0, as well as the memory, storage, and computational requirements. The proposed method uses a model composed of a wakeword model and a label model. Here we give more detail on these two components. We can model the user's chosen wakeword as a sequence of labels y = {y u ∈ A | u = 1, . . ., U}, where A is the set of possible labels, and U is the length of the sequence. The labels could be phonemes, graphemes, or other linguistic subunits; in this work, we use phonemes. It is generally not possible to perfectly estimate y from only a few training examples. Therefore, we maintain multiple hypotheses as to what the true sequence might be, along with a confidence for each hypothesis, and make use of all of these hypotheses during inference. A trained wakeword model thus consists of a set of label sequences and confidences. Beam Search DISPLAYFORM0 score:-3.1 The label model φ is a neural network trained using CTC on a speech corpus where each audio has a transcript of labels from the label set A. The network accepts an audio in the form of a sequence of acoustic feature vectors x = {x t ∈ R d | t = 1, . . ., T}, where d is the number of features per frame, and T is the number of frames. The network outputs a posteriorgram π = f φ (x) = {π t ∈ R 1+|A| | t = 1, . . ., T} representing the posterior probabilities of each of the labels and the CTC "blank" symbol at each timestep. Algorithm 1 describes the learning phase. The user records three examples of the wakephrase, here denoted by x train,1, x train,2, and x train,3. Once the user has recorded the audios, the label posteriors π train,i for each audio are computed using the label model φ. The CTCBeamSearch function then runs a beam search of width B over the label posteriors and returns a list of B probable label sequences and their corresponding log probabilities. More details on the beam search algorithm for CTC models can be found in BID16. The top N hypothesesŷ train,i 1,...,N are kept, and their log probabilities are converted to "confidences", which are also stored. Since not every hypothesis is equally good, the confidences can be used to weight the hypotheses during inference. We use an "acoustic-only" approach, in the sense that we do not use any sort of language model or pronunciation dictionary to prune the N -best list. Algorithm 2 describes how the wakeword is detected after the wakeword model has been learned. A voice activity detector (VAD) is used to determine which frames contain speech audio; only these frames are sent to the label model. The VAD thus reduces power consumption by reducing the amount of computation performed by the label model. After the label posteriors are computed by the network, the log probability of each hypothesis in the wakeword model is computed. The CTCForward function returns the log probability of a hypothetical label sequence given the audio by efficiently summing over all possible alignments of the label sequence to the audio BID2. The log probabilities are weighted by their respective confidences before they are summed to obtain a score. If the score is above a certain pre-determined threshold, the wakeword is detected. For clarity, we have written Algorithm 2 as though the posteriors are only computed after a complete audio x test has been acquired; it is preferable to reduce latency by computing the posteriors and Require: DISPLAYFORM0 beam, beam_scores:= CTCBeamSearch(π train,i, B)5:for j = 1 to N do end for 11: end for 12: return wake_model updating the hidden states as each speech frame becomes available from the VAD. Likewise, the forward algorithm can ingest a slice of π test at each timestep to compute that timestep's forward probabilities. Require: DISPLAYFORM0 score:= score + log p φ (ŷ|x test) · w 6: end for 7: return score DONUT is fast and suitable for running online on an embedded device. The memory, storage, and computational requirements of running DONUT online can be broken down into two parts: running the label model and running the wakeword model. The runtime requirements are dominated by the label model (the neural network). The complexity of running the neural network is O(nT), where n is the number of parameters and T is the duration of the audio in frames. We use an RNN with frame stacking BID17: that is, pairs of contiguous acoustic frames are stacked together so that the RNN operates at 50 Hz instead of 100 Hz, cutting the number of operations in half at the expense of slightly more input-hidden parameters in the first layer. The wakeword model requires little storage, as it consists of just 3N short strings and one real-valued confidence for each string. The CTC forward algorithm requires O(U T) operations to process a single label sequence. If the algorithm is run separately for N hypotheses, and the hypotheses have length U on average, then O(N U T) operations are required. The number of operations could be reduced by identifying and avoiding recomputing shared terms for the forward probabilities (e.g. using a lattice BID18), at the cost of a more complicated implementation. However, since N and U are small values, this kind of optimization is not crucial. (In the experiments described below, n is 168k, N is 10, and U is on average 10, so it is apparent that in general O(nT) » O(N U T).) The system requires O(N U) memory to store the forward probabilities for a single timestep; the memory for the previous timestep can be overwritten with the current timestep after the current forward probabilities have been computed. All audio data in our experiments is sampled at 16,000 Hz and converted to sequences of 41-dimensional Mel filterbank (FBANK) feature vectors using a 25 ms window with a stride of 10 ms. Here, we describe the two types of datasets used in our experiments: the dataset used to train the label models, and the datasets used to train and test the wakeword detectors. Label dataset We used LibriSpeech BID19, an English large vocabulary continuous speech recognition (LVCSR) dataset, to train label models. We used the Montreal Forced Aligner BID20 to obtain phoneme-level transcripts written in ARPAbet of the 100-and 360-hour subsets of the dataset. We trained a unidirectional GRU network with 3 layers and 96 hidden units per layer (168k parameters) on LibriSpeech with CTC using the phoneme-level transcripts. Wakeword datasets We created two wakeword datasets: one based on the 500-hour subset of LibriSpeech (LibriSpeech-Fewshot) and one based on crowdsourced English recordings (EnglishFewshot). Both datasets are composed of a number of few-shot learning "episodes". Each episode contains support examples and test examples. The support set contains three examples of the target phrase spoken by a single speaker. The test set contains a number of positive and negative examples. An example of an episode is shown in FIG2. The episodes are split into one subset for hyperparameter tuning and another subset for reporting performance. To create the LibriSpeech-Fewshot dataset, we split the LibriSpeech recordings into short phrases between 500 ms and 1,500 ms long, containing between one and four words. These short phrases were selected and grouped together to form 6,047 episodes. The test set contains eight positive examples by the same speaker and 24 negative examples by random speakers. Of the negative examples, twenty are phonetically similar ("confusing"), and four are phonetically dissimilar ("non-confusing"). To produce the confusing examples, we generated a phoneme-level transcript for each example, calculated the phoneme edit distance between the target phrase and all other available phrases, and chose the 20 phrases with the lowest phoneme edit distance. The non-confusing examples were chosen at random from the remaining phrases. To create the English-Fewshot dataset, we used crowdsourcing to record speakers saying phrases consisting of "Hello" followed by another word: for example, "Hello Computer". Like the LibriSpeech- Fewshot dataset, this dataset has positive examples from the same speaker and negative examples from different speakers; however, here there are also negative examples from the same speaker, so as to show that the models are not simply performing speaker verification. Due to data-gathering constraints, we were unable to obtain "imposter" examples in which a different speaker says the target phrase, but we plan to explore this in the future. All wakeword models used beam width B = 100 and kept N = 10 hypotheses per training example. We use the receiver operating characteristic (ROC) curve to measure the performance of a wakeword detector. A single detection threshold is used across all episodes. Two performance metrics are reported: the equal error rate (EER; lower is better) and the area-under-ROC-curve (AUC; higher is better) metric. An EER of 0% or an AUC of 1 indicates a perfect classifier. In the first experiment, we compare the performance of DONUT with two other query-by-example keyword spotting methods: dynamic time warping (DTW) based on the raw FBANK input and DTW based on the posteriorgram (the output of the label model). We used the 2 norm to compare FBANK features, and we used the distance-like metric suggested in BID7 to compare posteriorgram features: DISPLAYFORM0 where λ is a small positive number (we used 1e−5) and u is a uniform distribution (a vector with entries equal to 1 1+|A| ; used to prevent log by smoothing the peaky output distribution). We also tried removing the softmax, using the 2 norm as the distance metric, and using a label model trained using the framewise cross-entropy loss instead of the CTC loss. None of these modifications improved performance; we report the best with the CTC model here. TAB0 shows the performance of the query-by-example methods on English-Fewshot. We report the performance for three separate cases, in decreasing order of difficulty: the cases when the negative examples are 1) confusing and taken from the same speaker, 2) non-confusing and taken from the same speaker, and 3) non-confusing and taken from different speakers. DONUT outperforms both DTW methods in all three cases. In this experiment, we compare the performance of our method with the performance of conventional CTC keyword spotting when the "true" label sequence is provided (e.g., by the user through a text interface). The phoneme sequence for each phrase in the LibriSpeech-Fewshot dataset was obtained using forced alignment and used as the wakeword model for each episode. TAB1 shows that for phonetically confusing examples, DONUT outperforms the text-based approach, and for non-confusing examples, the two approaches perform roughly the same, with the text-based approach performing very slightly better. This indicates that not only does DONUT provide a more convenient interface than query-by-string keyword spotting, it also has the same or even better performance. Like conventional CTC keyword spotting, DONUT is interpretable, which makes it easy for a system designer to identify problems with the model and improve it. For example, FIG2 shows an example of a wakeword model learned for the phrase "of dress". In the first two training examples, the network hears an "N" sound where one would expect the "V" phoneme in "of". This information can be used to improve the model: one could retrain the label model with more examples short words such as "of" and "on", to help the model distinguish short sounds more easily. Alternately, it could become apparent after listening to the training examples that for the speaker's personal accent the phrase does indeed contain an "N" sound. Debugging the inference phase is also made easier by the use of CTC. It is possible to decode phoneme sequences from the test audio using a beam search, although this is not necessary to do during inference. One could inspect the decoded sequences from an audio that causes a false accept to identify hypotheses that should be removed from the model to make the false accept less likely to occur. If a false reject occurs, one could check whether the wakeword model hypotheses are found in the decoded sequences or if the network hears something completely different. DONUT has a few hyperparameters: the beam width B, the number of hypotheses kept from the beam search N, the label model φ, and the way in which the hypothesis scores are aggregated. Here we explore the impact of these hyperparameters on performance using the English-Fewshot dataset. Increasing the number of hypotheses generally improves performance (Table 3), though we have found that this may yield diminishing returns. Even a simple greedy search (B = 1, N = 1), which can be implemented by picking the top output at each timestep, works fairly well for our system. With respect to the impact of the choice of label model, we find that label models with lower phoneme error rate (edit distance between the true label sequence and the model's prediction) for the original corpus they were trained on have a lower error rate for wakeword detection TAB2 ). This suggests that making an improvement to the label model can be expected to translate directly to a decrease in EER/increase in AUC.In the inference algorithm described above (Algorithm 2), the hypotheses' scores are aggregated by taking a weighted sum, where each weight is the inverse of the log probability of that hypothesis given its corresponding training example. Without the weighting, performance was hurt because some hypotheses are a worse fit to the data than others. A more principled approach to aggregating the scores might be to treat the hypotheses' log probabilities from training as log priors and add them to the scores, since multiplying by a prior is equivalent to adding a log prior, and to take the logsumexp of the scores plus their log priors, since adding two probabilities is equivalent to taking the logsumexp of two log probabilities. However, we have found that this does not work as well as the weighted sum approach, perhaps because the logsumexp function acts like max and tends to pick out a single hypothesis instead of smoothly blending the hypotheses. In this paper, we proposed DONUT, an efficient algorithm for online query-by-example keyword spotting using CTC. The algorithm learns a list of hypothetical label sequences from the user's speech during enrollment and uses these hypotheses to score audios at test time. We showed that the model is interpretable, and thus easy to inspect, debug, and tweak, yet at the same time has high accuracy. Because training a wakeword model amounts to a simple beam search, it is possible to train a model on the user's device without uploading a user's private voice data to the cloud. Our technique is in principle applicable to any domain in which a user would like to teach a system to recognize a sequence of events, such as a melody (a sequence of musical notes) or a gesture (a sequence of hand movements). It would be interesting to see how well the proposed technique transfers to these other domains.

represent the tldr for finding the corresponding scientific article

We propose an interpretable model for detecting user-chosen wakewords that learns from the user's examples.

Represent the short summary for finding the corresponding scientific article (long).

Language generation using seq2seq models which produce word embeddings instead of a softmax based distribution over the vocabulary at each step enabling much faster training while maintaining generation quality