Question: Represent the text to find a very short summary (1-2 sentences max)

Question: To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics: (a) it should build an abstract state representing the condition of the world; (b) it should form a belief which represents uncertainty on the world; (c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction. Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning. Generative models of sequential data have received a lot of attention, due to their wide applicability in domains such as speech synthesis BID18, neural translation BID3, image captioning BID22, and many others. Different application domains will often have different requirements (e.g. long term coherence, sample quality, abstraction learning, etc.), which in turn will drive the choice of the architecture and training algorithm. Of particular interest to this paper is the problem of reinforcement learning in partially observed environments, where, in order to act and explore optimally, agents need to build a representation of the uncertainty about the world, computed from the information they have gathered so far. While an agent endowed with memory could in principle learn such a representation implicitly through model-free reinforcement learning, in many situations the reinforcement signal may be too weak to quickly learn such a representation in a way which would generalize to a collection of tasks. Furthermore, in order to plan in a model-based fashion, an agent needs to be able to imagine distant futures which are consistent with the agent's past. In many situations however, planning step-by-step is not a cognitively or computationally realistic approach. To successfully address an application such as the above, we argue that a model of the agent's experience should exhibit the following properties:• The model should learn an abstract state representation of the data and be capable of making predictions at the state level, not just the observation level.• The model should learn a belief state, i.e. a deterministic, coded representation of the filtering posterior of the state given all the observations up to a given time. A belief state contains all the information an agent has about the state of the world and thus about how to act optimally.• The model should exhibit temporal abstraction, both by making'jumpy' predictions (predictions several time steps into the future), and by being able to learn from temporally separated time points without backpropagating through the entire time interval. To our knowledge, no model in the literature meets these requirements. In this paper, we develop a new model and associated training algorithm, called Temporal Difference Variational Auto-Encoder (TD-VAE), which meets all of the above requirements. We first develop TD-VAE in the sequential, non-jumpy case, by using a modified evidence lower bound (ELBO) for stochastic state space models (; BID12 BID8 which relies on jointly training a filtering posterior and a local smoothing posterior. We demonstrate that on a simple task, this new inference network and associated lower bound lead to improved likelihood compared to methods classically used to train deep state-space models. Following the intuition given by the sequential TD-VAE, we develop the full TD-VAE model, which learns from temporally extended data by making jumpy predictions into the future. We show it can be used to train consistent jumpy simulators of complex 3D environments. Finally, we illustrate how training a filtering a posterior leads to the computation of a neural belief state with good representation of the uncertainty on the state of the environment.2 MODEL DESIDERATA Autoregressive models. One of the simplest way to model sequential data (x 1, . . ., x T) is to use the chain rule to decompose the joint sequence likelihood as a product of conditional probabilities, i.e. log p(x 1, . . ., x T) = t log p(x t | x 1, . . ., x t−1). This formula can be used to train an autoregressive model of data, by combining an RNN which aggregates information from the past (recursively computing an internal state h t = f (h t−1, x t)) with a conditional generative model which can score the data x t given the context h t. This idea is used in handwriting synthesis BID15, density estimation , image synthesis (van den BID19, audio synthesis (van den BID20, video synthesis , generative recall tasks BID13, and environment modeling (; BID9 .While these models are conceptually simple and easy to train, one potential weakness is that they only make predictions in the original observation space, and don't learn a compressed representation of data. As a , these models tend to be computationally heavy (for video prediction, they constantly decode and re-encode single video frames). Furthermore, the model can be computationally unstable at test time since it is trained as a next step model (the RNN encoding real data), but at test time it feeds back its prediction into the RNN. Various methods have been used to alleviate this issue; BID14 BID0.State-space models. An alternative to autoregressive models are models which operate on a higher level of abstraction, and use latent variables to model stochastic transitions between states (grounded by observation-level predictions). This enables to sample state-to-state transitions only, without needing to render the observations, which can be faster and more conceptually appealing. They generally consist of decoder or prior networks, which detail the generative process of states and observations, and encoder or posterior networks, which estimate the distribution of latents given the observed data. There is a large amount of recent work on these type of models, which differ in the precise wiring of model components BID4 BID10; BID1 BID12;; BID8; BID17.Let z = (z 1, . . ., z T) be a state sequence and x = (x 1, . . ., x T) an observation sequence. We assume a general form of state-space model, where the joint state and observation likelihood can be written as p(x, z) = t p(z t | z t−1)p(x t | z t).1 These models are commonly trained with a VAEinspired bound, by computing a posterior q(z | x) over the states given the observations. Often, the posterior is decomposed autoregressively: q(z | x) = t q(z t | z t−1, φ t (x)), where φ t is a function of (x 1, . . ., x t) for filtering posteriors or the entire sequence x for smoothing posteriors. This leads to the following lower bound: DISPLAYFORM0 A key feature of sequential models of data is that they allow to reason about the conditional distribution of the future given the past: p(x t+1, . . ., x T | x 1, . . ., x t). For reinforcement learning in partially observed environments, this distribution governs the distribution of returns given past observations, and as such, it is sufficient to derive the optimal policy. For generative sequence modeling, it enables conditional generation of data given a context sequence. For this reason, it is desirable to compute sufficient statistics b t = b t (x 1, . . ., x t) of the future given the past, which allow to rewrite the conditional distribution as p(x t+1, . . ., x T | x 1, . . ., x t) ≈ p(x t+1, . . ., x T | b t). For an autoregressive model as described in section 2.1, the internal RNN state h t can immediately be identified as the desired sufficient statistics b t. However, for the reasons mentioned in the previous section, we would like to identify an equivalent quantity for a state-space model. For a state-space model, the filtering distribution p(z t | x 1, . . ., x t), also known as the belief state in reinforcement learning, is sufficient to compute the conditional future distribution, due to the Markov assumption underlying the state-space model and the following derivation: DISPLAYFORM1 Thus, if we train a network that extracts a code DISPLAYFORM2, b t would contain all the information about the state of the world the agent has, and would effectively form a neural belief state, i.e. a code fully characterizing the filtering distribution. Classical training of state-space model does not compute a belief state: by computing a joint, autoregressive posterior q(z | x) = t q(z t | z t−1, x), some of the uncertainty about the marginal posterior of z t may be'leaked' in the sample z t−1. Since that sample is stochastic, to obtain all information from (x 1, . . ., x t) about z t, we would need to re-sample z t−1, which would in turn require re-sampling z t−2 all the way to z 1.While the notion of a belief state itself and its connection to optimal policies in POMDPs is well known BID2; ), it has often been restricted to the tabular case (Markov chain), and little work investigates computing belief states for learned deep models. A notable exception is , which uses a neural form of particle filtering, and represents the belief state more explicitly as a weighted collection of particles. Related to our definition of belief states as sufficient statistics is the notion of predictive state representations (PSRs) ; see also BID21 for a model that learns PSRs which, combined with a decoder, can predict future observations. Our last requirement for the model is that of temporal abstraction. We postpone the discussion of this aspect until section 4. In this section, we develop a sequential model that satisfies the requirements given in the previous section, namely (a) it constructs a latent state-space, and (b) it creates a online belief state. We consider an arbitrary state space model with joint latent and observable likelihood given by DISPLAYFORM0, and we aim to optimize the data likelihood log p(x). We begin by autoregressively decomposing the data likelihood as: log p(x) = t log p(x t | x <t). For a given t, we evaluate the conditional likelihood p(x t | x <t) by inferring over two latent states only: z t−1 and z t, as they will naturally make belief states appear for times t − 1 and t: DISPLAYFORM1 Because of the Markov assumptions underlying the state-space model, we can simplify DISPLAYFORM2. Next, we choose to decompose q(z t−1, z t | x ≤t) as a belief over z t and a one-step smoothing distribution DISPLAYFORM3. We obtain the following belief-based ELBO for state-space models: DISPLAYFORM4 Both quantities p(z t−1 | x ≤t−1) and q(z t | x ≤t) represent the belief state of the model at different times, so at this stage we approximate them with the same distribution DISPLAYFORM5 representing the belief state code for z t. Similarly, we represent the smoothing posterior over z t−1 as q(z t−1 | z t, b t−1, b t). We obtain the following loss: DISPLAYFORM6 We provide an intuition on the different terms of the ELBO in the next section. The model derived in the previous section expresses a state model p(z t | z t−1) that describes how the state of the world evolves from one time step to the next. However, in many applications, the relevant timescale for planning may not be the one at which we receive observations and execute simple actions. Imagine for example planning for a trip abroad; the different steps involved (discussing travel options, choosing a destination, buying a ticket, packing a suitcase, going to the airport, and so on), all occur at vastly different time scales (potentially months in the future at the beginning of the trip, and days during the trip). Certainly, making a plan for this situation does not involve making second-by-second decisions. This suggests that we should look for models that can imagine future states directly, without going through all intermediate states. Beyond planning, there are several other reasons that motivate modeling the future directly. First, training signal coming from the future can be stronger than small changes happening between time steps. Second, the behavior of the model should ideally be independent from the underlying temporal sub-sampling of the data, if the latter is an arbitrary choice. Third, jumpy predictions can be computationally efficient; when predicting several steps into the future, there may be some intervals where the prediction is either easy (e.g. a ball moving straight), or the prediction is complex but does not affect later time steps -which Neitz et al. FORMULA1 call inconsequential chaos. There is a number of research directions that consider temporal jumps. Koutnik et al. FORMULA1 and BID11 consider recurrent neural network with skip connections, making it easier to bridge distant timesteps. BID8 temporally sub-sample the data and build a jumpy model (for fixed jump size) of this data; but by doing so they also drop the information contained in the skipped observations. Neitz et al. FORMULA1 and Jayaraman et al. FORMULA1 predict sequences with variable time-skips, by choosing as target the most predictable future frames. They predict the observations directly without learning appropriate states, and only focus on nearly fully observed problems (and therefore do not need to learn a notion of belief state). For more general problems, this is a fundamental limitation, as even if one could in principle learn a jumpy observation model p(x t+δ |x ≤t), it cannot be used recursively (feeding x t+δ back to the RNN and predicting x t+δ+δ). This is because x t+δ does not capture the full state of the system and so we would be missing information from t to t + δ to fully characterize what happens after time t + δ. In addition, x t+δ might not be appropriate even as target, because some important information can only be extracted from a number of frames (potentially arbitrarily separated), such as a behavior of an agent. Motivated by the model derived in section 3, we extend sequential TD-VAE to exhibit time abstraction. We start from the same assumptions and architectural form: there exists a sequence of states z 1,..., z T from which we can predict the observations x 1,..., x T. A forward RNN encodes a belief state b t from past observations x ≤t. The main difference is that, instead of relating information known at times t and t + 1 through the states z t and z t+1, we relate two distant time steps t 1 and t 2 through their respective states z t1 and z t2, and we learn a jumpy, state-to-state model p(z t2 | z t1) between z t1 and z t2. Following equation 5, the negative loss for the TD-VAE model is: DISPLAYFORM0 To train this model, one should choose the distribution of times t 1, t 2; for instance, t 1 can be chosen uniformly from the sequence, and t 2 − t 1 uniformly over some finite range [1, D]; other approaches could be investigated. FIG2 describes in detail the computation flow of the model. Finally, it would be desirable to model the world with different hierarchies of state, the higher-level states predicting the same-level or lower-level states, and ideally representing more invariant or abstract information. For this reason, we also develop stacked (hierarchical) version of TD-VAE, which uses several layers of latent states. Hierarchical TD-VAE is detailed in the appendix. In this section, we provide a more intuitive explanation behind the computation and loss of the model. Assume we want to predict a future time step t 2 from all the information we have up until time t 1. All relevant information up until time t 1 (respectively t 2) has been compressed into a code b t1 (respectively b t2). We make an observation x t of the world 2 at every time step t, but posit the existence of a state z t which fully captures the full condition of the world at time t. Consider an agent at the current time t 2. At that time, the agent can make a guess of what the state of the world is by sampling from its belief model p B (z t2 | b t2). Because the state z t2 should entail the corresponding observation x t2, the agent aims to maximize p(x t2 | z t2) (first term of the loss), with a variational bottleneck penalty − log p(z t2 | b t2) (second term of the loss) to prevent too much information from the current observation x t2 from being encoded into z t2. Then follows the question'could the state of the world at time t 2 have been predicted from the state of the world at time t 1?'. In order to ascertain this, the agent must estimate the state of the world at time t 1. By time t 2, the agent has aggregated observations between t 1 and t 2 that are informative about the state of the world at time t 1, which, together with the current guess of the state of the world z t2, can be used to form an ex post guess of the state of the world. This is done by computing a smoothing distribution q(z t1 |z t2, b t1, b t2) and drawing a corresponding sample z t1. Having guessed states of the world z t1 and z t2, the agent optimizes its predictive jumpy model of the world state p(z t2 | z t1) (third term of the loss). Finally, it should attempt to see how predictable the revealed information was, or in other words, to assess whether the smoothing distribution q(z t1 | z t2, b t2) could have been predicted from information only available at time t 1 (this is indirectly predicting z t2 from the state of knowledge b t1 at time t 1 -the problem we started with). The agent can do so by minimizing the KL between the smoothing distribution and the belief distribution at time DISPLAYFORM0 (fourth term of the loss). Summing all the losses described so far, we obtain the TD-VAE loss. In reinforcement learning, the state of an agent represents a belief about the sum of discounted rewards R t = τ r t+τ γ τ. In the classic setting, the agent only models the mean of this distribution represented by the value function V t or action dependent Q-function Q a t . Recently in , a full distribution over R t has been considered. To estimate V t1 or Q a t1 at time t 1, one does not usually wait to get all the rewards to compute R t1. Instead, one uses an estimate at some future time t 2 as a bootstrap to estimate V t1 or Q a t1 (temporal difference). In our case, the model expresses a belief p B (z t | b t) about possible future states instead of the sum of discounted rewards. The model trains the belief p B (z t1 | b t1) at time t 1 using belief p B (z t2 | b t2) at some time t 2 in the future. It accomplishes this by (variationally) auto-encoding a sample z t2 of the future state into a sample z t1, using the approximate posterior distribution q(z t1 | z t2, b t1, b t2) and the decoding distribution p(z t2 | z t1). This auto-encoding mapping translates between states at t 1 and t 2, forcing beliefs at the two time steps to be consistent. Sample z t1 forms the target for training the belief p B (z t1 | b t1), which appears as a prior distribution over z t1. The first experiment using sequential TD-VAE, which enables a direct comparison to related algorithms for training state-space models. Subsequent experiments use the full TD-VAE model. We use a partially observed version of the MiniPacman environment (Racanière et al., 2017), shown in FIG0. The agent (Pacman) navigates a maze, and tries to eat all the food while avoiding being eaten by a ghost. Pacman sees only a 5 × 5 window around itself. To achieve a high score, the agent needs to form a belief state that captures memory of past experience (e.g. which parts of the maze have been visited) and uncertainty on the environment (e.g. where the ghost might be).We evaluate the performance of sequential (non-jumpy) TD-VAE on the task of modeling a sequence of the agent's observations. We compare it with two state-space models trained using the standard ELBO of equation 1:• A filtering model with encoder q(z | x) = t q(z t | z t−1, b t), where b t = RNN(b t−1, x t).• A mean-field model with encoder q(z | x) = t q(z t | b t), where b t = RNN(b t−1, x t). FIG0 shows the ELBO and estimated negative log probability on a test set of MiniPacman sequences for each model. TD-VAE outperforms both baselines, whereas the mean-field model is the least well-performing. We note that b t is a belief state for the mean-field model, but not for the filtering model; the encoder of the latter explicitly depends on the previous latent state z t−1, hence b t ELBO and estimated negative log probability on a test set of MiniPacman sequences. Lower is better. Log probability is estimated using importance sampling with the encoder as proposal. is not its sufficient statistics. This comparison shows that naively restricting the encoder in order to obtain a belief state hurts the performance significantly; TD-VAE overcomes this difficulty. In this experiment, we show that the model is able to learn the state and roll forward in jumps. We consider sequences of length 20 of images of MNIST digits. For each sequence, a random digit from the dataset is chosen, as well as the direction of movement (left or right). At each time step, the digit moves by one pixel in the chosen direction, as shown in FIG1. We train the model with t 1 and t 2 separated by a random amount t 2 − t 1 from the interval. We would like to see whether the model at a given time can roll out a simulated experience in time steps t 1 = t + δ 1, t 2 = t 1 + δ 2,... with δ 1, δ 2,... > 1, without considering the inputs in between these time points. Note that it is not sufficient to predict the future inputs x t1,... as they do not contain information about whether the digit moves left or right. We need to sample a state that contains this information. We roll out a sequence from the model as follows: (a) b t is computed by the aggregation recurrent network from observations up to time t; (b) a state z t is sampled from p B (z t | b t); (c) a sequence of states is rolled out by repeatedly sampling z ← z ∼ p(z | z) starting with z = z t; (d) each z is decoded by p(x | z), producing a sequence of frames. The ing sequences are shown in FIG1. We see that indeed the model can roll forward the samples in steps of more than one elementary time step (the sampled digits move by more than one pixel) and that it preserves the direction of motion, demonstrating that it rolls forward a state. We would like to demonstrate that the model can build a state even when little information is present in each observation, and that it can sample states far into the future. For this we consider a 1D sequence obtained from a noisy harmonic oscillator, as shown in Figure 4 (first and fourth rows). The frequencies, initial positions and initial velocities are chosen at random from some range. At every update, noise is added to the position and the velocity of the oscillator, but the energy is approximately preserved. The model observes a noisy version of the current position. Attempting to predict the input, which consists of one value, 100 time steps in the future would be uninformative; such a Figure 4: Skip-state prediction for 1D signal. The input is generated by a noisy harmonic oscillator. Rollouts consist of (a) a jumpy state transition with either dt = 20 or dt = 100, followed by 20 state transitions with dt = 1. The model is able to create a state and predict it into the future, correctly predicting frequency and magnitude of the signal.prediction wouldn't reveal what the frequency or the magnitude of the signal is, and because the oscillator updates are noisy, the phase information would be nearly lost. Instead, we should try to predict as much as possible about the state, which consists of frequency, magnitude and position, and it is only the position that cannot be accurately predicted. The aggregation RNN is an LSTM; we use a hierarchical TD-VAE with two layers, where the latent variables in the higher layer are sampled first, and their are passed to the lower layer. The belief, smoothing and state-transition distributions are feed-forward networks, and the decoder simply extracts the first component from the z of the first layer. We also feed the time interval t 2 − t 1 into the smoothing and state-transition distributions. We train on sequences of length 200, with t 2 − t 1 taking values chosen at random from with probability 0.8 and from We analyze what the model has learned as follows. We pick time t 1 = 60 and sample z t1 ∼ p B (z t1 | b t1). Then, we choose a time interval δ t ∈ {20, 100} to skip, sample from the forward model p(z 2 | z 1, δ t) to obtain z t2 at t 2 = t 1 + δ t. To see the content of this state, we roll forward 20 times with time step δ = 1 and plot the , shown in Figure 4. We see that indeed the state z t2 is predicted correctly, containing the correct frequency and magnitude of the signal. We also see that the position (phase) is predicted well for dt = 20 and less accurately for dt = 100 (at which point the noisiness of the system makes it unpredictable).Finally, we show that TD-VAE training can improve the quality of the belief state. For this experiment, the harmonic oscillator has a different frequency in each interval,,,. The first three frequencies f 1, f 2, f 3 are chosen at random. The final frequency f 4 is chosen to be one fixed value f a if f 1 > f 2 and another fixed value f b otherwise (f a and f b are constants). In order to correctly model the signal in the final time interval, the model needs to learn the relation between f 1 and f 2, store it over length of 100 steps, and apply it over a number of time steps (due to the noise) in the final interval. To test whether the belief state contains the information about this relationship, we train a binary classifier from the belief state to the final frequency f 4 at points just before the final interval. We compare two models with the same recurrent architecture (an LSTM), but trained with different objective: next-step prediction vs TD-VAE loss. The figure on the right shows the classification accuracy for the two methods, averaged over 20 runs. We found that the longer the separating time interval (containing frequency f 3) and the smaller the size of the LSTM, the better TD-VAE is compared to next-step predictor. In the final experiment, we analyze the model on a more visually complex domain. We use sequences of frames seen by an agent solving tasks in the DeepMind Lab environment BID5. We aim to demonstrate that the model holds explicit beliefs about various possible futures, and that it can roll out in jumps. We suggest functional forms inspired by convolutional DRAW: we use convolutional LSTMs for all the circles in FIG5 and make the model 16 layers deep (except for the forward updating LSTMs which are fully connected with depth 4).We use time skips t 2 − t 1 sampled uniformly from and analyze the content of the belief state b. We take three samples z 1, z 2, z 3 from p B (z | b), which should represent three instances of possible futures. FIG3 (left) shows that they decode to roughly the same frame. To see what they represent about the future, we draw 5 samples z k i ∼ p(ẑ | z), k = 1,..., 5 and decode them, as shown in FIG3 (right). We see that for a given i, the predicted samples decode to similar frames (images in the same row). However z's for different i's decode to different frames. This means b represented a belief about several different possible futures, while different z i each represent a single possible future. Finally, we show what rollouts look like. We train on time separations t 2 − t 1 chosen uniformly from on a task where the agent tends to move forward and rotate. FIG4 shows 4 rollouts from the model. We see that the motion appears to go forward and into corridors and that it skips several time steps (real single step motion is slower). In this paper, we argued that an agent needs a model that is different from an accurate step-by-step environment simulator. We discussed the requirements for such a model, and presented TD-VAE, a sequence model that satisfies all requirements. TD-VAE builds states from observations by bridging time points separated by random intervals. This allows the states to relate to each other directly over longer time stretches and explicitly encode the future. Further, it allows rolling out in state-space and in time steps larger than, and potentially independent of, the underlying temporal environment/data step size. In the future, we aim to apply TD-VAE to more complex settings, and investigate a number of possible uses in reinforcement learning such are representation learning and planning. In section 3, we derive an approximate ELBO which forms the basis of the training loss of the one-step TD-VAE. One may wonder whether a similar idea may underpin the training loss of the jumpy TD-VAE. Here we show how to modify the derivation to provide an approximate ELBO for a slightly different training regime. Assume a sequence (x 1, . . ., x T), and an arbitrary distribution S over subsequences x s = (x t1, . . ., x tn) of x. For each time index t i, we suppose a state z ti, and model the subsequence x s with a jumpy state-space model p(x s) = i p(z ti |z ti−1)p(x ti |z ti); denote z s = (z t1, . . ., z tn) the state subsequence. We use the exact same machinery as the next-step ELBO, except that we enrich the posterior distribution over z s by making it depend not only on observation subsequence x s, but on the entire sequence x. This is possible because posterior distributions can have arbitrary contexts; the observations which are part of x but not x s effectively serve as auxiliary variable for a stronger posterior. We use the full sequence x to form a sequence of belief states b t at all time steps. We use in particular the ones computed at the subsampled times t i. By following the same derivation as the one-step TD-VAE, we obtain: DISPLAYFORM0 which, using the same belief approximations as the next step TD-VAE, becomes: DISPLAYFORM1 which is the same loss as the TD-VAE for a particular choice of the sampling scheme S (only sampling pairs). In this section we start with a general recurrent variational auto-encoder and consider how the desired properties detailed in sections 1 and 2 constrain the architecture. We will find that these constraints in fact naturally lead to the TD-VAE model. Let us first consider a relatively general form of temporal variational auto-encoder. We consider recurrent models where the same module is applied at every step, and where outputs are sampled one at a time (so that arbitrarily long sequences can be generated). A very general form of such an architecture consist of forward-backward encoder RNNs and a forward decoder RNN (Figure 7) but otherwise allowing for all the connections. Several works BID10; BID1 BID12; BID14 BID8 ) fall into this framework. Now let us consider our desired properties. In order to sample forward in latent space, the encoder must not feed into the decoder or the prior of the latent variables, since observations are required to compute the encoded state, and we would therefore require the sampled observations to compute the distribution over future states and observations. We next consider the constraint of computing a belief state b t. The belief state b t represents the state of knowledge up to time t, and therefore cannot receive an input from the backwards decoder. Figure 7: Recurrent variational auto-encoder. General recurrent variational auto-encoder, obtained by imposing recurrent structure, forward sampling and allowing all potential connections. Note that the encoder can have several alternating layers of forward and backward RNNs. Also note that the connection 1 has to be absent if the backwards encoder is used. Possible skip connections are not shown as they can directly be implemented in the RNN weights. If connections 2 are absent, the model is capable of forward sampling in latent space without going back to observations. Furthermore, b t should have an unrestricted access to information; it should ideally not be disturbed by sampling (two identical agents with the same information should compute the same information; this will not be the case if the computation involves sampling), nor go through information bottlenecks. This suggests using the forward encoder for computing the belief state. This prevents running the backwards inference from the end of the sequence. However if we assume that p B represents our best belief about the future, we can take a sample from it as an instance of the future: z t2 ∼ p B (z t2 |b t2). It forms a type of bootstrap information. Then we can go backwards and infer what would the world have looked like given this future (e.g. the object B was still in the box even if we don't see it). Using VAE training, we sample z 1 from its posterior q(z t1 |z t2, b t2, b t1) (the conditioning variables are the ones we have available locally), using p B (z t1 |b t1) as prior. Conversely, for t 2, we sample from p B (z t2 |b t2) as posterior, but with p(z t2 |z t1) as prior. We therefore obtain the VAE losses log q(z 1 |z 2, s 1, s 2) − log p B (z 1 |s 1) at t 1 and log p B (z 2 |s 2) − log p P (z 2 |z 1) at t 2. In addition we have the reconstruction term p D (x 2 |z 2) that grounds the latent in the input. The whole algorithm is presented in the FIG2 In the main paper we detailed a framework for learning models by bridging two temporally separated time points. It would be desirable to model the world with different hierarchies of state, the higherlevel states predicting the same-level or lower-level states, and ideally representing more invariant or abstract information. In this section we describe a stacked (hierarchical) version of the model. The first part to extend to L layers is the RNN that aggregates observations to produce the belief state b. Here we simply use a deep LSTM, but with layer l receiving inputs also from layer l + 1 from the previous time step. This is so that the higher layers can influence the lower ones (and vice versa). and setting b 0 = b L and b L+1 = ∅.We create a deep version of the belief part of the model by stacking the shallow one, as shown in FIG5. In the usual spirit of deep directed models, the model samples downwards, generating higher level representations before the lower level ones (closer to pixels). The model implements deep inference, that is, the posterior distribution of one layer depends on the samples from the posterior distribution in previously sampled layers. The order of inference is a design choice, and we use the same direction as that of generation, from higher to lower layers, as done for example by BID16;;. We implement the dependence of various distributions on latent variables sampled so far using a recurrent neural network that summarizes all such variables (in a given group of distributions). We don't share the weights between different layers. Given these choices, we can allow all connections consistent with the model. Next we describe the functional forms used in our model.= KL(q DISPLAYFORM0 The hidden layer of the D maps is 50; the size of each z l t is 8. Belief states have size 50. We use the Adam optimizer with learning rate 0.0005.The same network works for the MNIST experiment with the following modifications. Observations are pre-processed by a two hidden layer MLP with ReLU nonlinearity. The decoder p D also have a two layer MLP, which outputs the logits of a Bernoulli distribution. δ t was not passed as input to any network.

represent the short summary for finding the corresponding scientific article (long):

Generative model of temporal data, that builds online belief state, operates in latent space, does jumpy predictions and rollouts of states.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: represent the article for finding a single-sentence tldr

Question: This paper introduces an information theoretic co-training objective for unsupervised learning. We consider the problem of predicting the future. Rather than predict future sensations (image pixels or sound waves) we predict ``hypotheses'' to be confirmed by future sensations. More formally, we assume a population distribution on pairs $(x,y)$ where we can think of $x$ as a past sensation and $y$ as a future sensation. We train both a predictor model $P_\Phi(z|x)$ and a confirmation model $P_\Psi(z|y)$ where we view $z$ as hypotheses (when predicted) or facts (when confirmed). For a population distribution on pairs $(x,y)$ we focus on the problem of measuring the mutual information between $x$ and $y$. By the data processing inequality this mutual information is at least as large as the mutual information between $x$ and $z$ under the distribution on triples $(x,z,y)$ defined by the confirmation model $P_\Psi(z|y)$. The information theoretic training objective for $P_\Phi(z|x)$ and $P_\Psi(z|y)$ can be viewed as a form of co-training where we want the prediction from $x$ to match the confirmation from $y$. We give experiments on applications to learning phonetics on the TIMIT dataset.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Presents an information theoretic training objective for co-training and demonstrates its power in unsupervised learning of phonetics.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: represent the article for finding a single-sentence tldr

Question: Generative models for singing voice have been mostly concerned with the task of "singing voice synthesis," i.e., to produce singing voice waveforms given musical scores and text lyrics. In this work, we explore a novel yet challenging alternative: singing voice generation without pre-assigned scores and lyrics, in both training and inference time. In particular, we experiment with three different schemes: 1) free singer, where the model generates singing voices without taking any conditions; 2) accompanied singer, where the model generates singing voices over a waveform of instrumental music; and 3) solo singer, where the model improvises a chord sequence first and then uses that to generate voices. We outline the associated challenges and propose a pipeline to tackle these new tasks. This involves the development of source separation and transcription models for data preparation, adversarial networks for audio generation, and customized metrics for evaluation. The task of computationally producing singing voices is usually referred to as singing voice synthesis (SVS) in the literature . Most researchers assume that the note sequence and the lyrics of the waveform to be generated are given as the model input, and aim to build synthesis engines that sound as natural and expressive as a real singer (; ; ; a;). As such, the content of the produced singing voice is largely determined by the given model input, which is usually assigned by human. And, accordingly, progress in SVS has followed closely with that in text-to-speech (TTS) synthesis (; ;). However, we argue that singing according to a pre-assigned musical score and lyrics is only a part of the human singing activities. For human beings, singing can also be a spontaneous activity. We learn to spontaneously sing when we were children . We do not need a score to sing when we are humming on the road or in the bathroom. The voices sung do not have to be intelligible. Jazz vocalists can improvise according to a chord progression, an accompaniment, or even nothing. We aim to explore such a new task in this paper: teaching a machine to sing with a training collection of singing voices, but without the corresponding musical scores and lyrics of the training data. Moreover, the machine has to sing without pre-assigned score and lyrics as well even in the inference (generation) time. This task is challenging in that, as the machine sees no lyrics at all, it hardly has any knowledge of the human language to pronounce or articulate either voiced or unvoiced sounds. And, as the machine sees no musical scores at all, it has to find its own way learning the language of music in creating plausible vocal melodies. It also makes the task different from TTS. Specifically, we consider three types of such score-and lyrics-free singing voice generation tasks, as shown in Figures 1(b) - (d). A free singer sings with only random noises as the input. An accompanied singer learns to sing over a piece of instrumental music, which is given as an audio waveform (again without score information). Finally, a solo singer also sings with only noises as the input, but it uses the noises to firstly generate some kind of'inner ideas' of what to sing. From a technical point of view, we can consider SVS as a strongly conditioned task for generating singing voices, as the target output is well specified by the input. In contrast, the proposed tasks are either unconditioned or weakly conditioned. This work therefore contributes to expanding the "spectrum" (in terms of the strength of conditional signals) of singing voice generation. Doing so has at least two implications. First, while our models are more difficult to train than SVS models, they enjoy more freedom in the generation output. Such freedom may be desirable considering the artistic nature of singing. Second, we can more easily use a larger training set to train our model- due to the difficulty in preparing time-aligned scores and lyrics, the training set employed in existing work on SVS usually consists of tens of songs only (a); in contrast, in our case we do not need labeled and aligned data and can therefore use more than hundreds of songs for training. This may help establish a universal model based on which extensions can be made. The proposed accompanied singer also represents one of the first attempts to produce singing voice given an accompaniment. One intuitive approach to achieve this is to first generate a score according to an accompaniment in the symbolic domain and then synthesize the singing voices according to the score. The second step of synthesis is relatively well-established, but the first step of generating a score given an accompaniment is not explored yet. Extensive researches have been done in generating scores of one or several instruments (; ;). However, to the best of our knowledge, very few, if any, researches have been done on generating scores of singing voices given an accompaniment. Our approach bypasses the step of generating scores by directly generating the mel-spectrogram representation. We outline below the challenges associated with the proposed tasks and the solutions we investigate. First, the tasks are unsupervised as we do not provide any labels (e.g., annotations of phonemes, pitches, or onset times) for the training singing files. The machine has to learn the complex structure of music directly from audio signals. We explore the use of generative adversarial network (GAN) to address this issue, for its demonstrated effectiveness for SVS and pitch-conditioned instrument note synthesis . Specifically, we design a novel GAN-based architecture to learn to generate the mel-spectrogram of singing voice, and then use WaveRNN , a single-layer recurrent neural network, as the vocoder to generate the audio waveform. Rather than considering the mel-spectrograms as a fixedsize image as done in recent work on audio generation , we use gated recurrent units (GRUs) and dilated convolutions (van den) in both the generator and discriminator, to model both the local and sequential patterns in music and to facilitate the generation of variable-length waveforms. Second, for training the free singer, unaccompanied vocal tracks are needed. As for the accompanied singer, we need additionally an accompaniment track for each vocal track. However, public-domain multi-track music data is hard to find. We choose to implement a vocal source separation model with state-of-the-art separation quality for data preparation. The proposed pipeline for training and evaluating an accompanied singer is illustrated in Figure 2. The advantage of having a vocal separation model is that we can use as many audio files as we have to compile the training data. The downside is that the singing voice generation models may suffer from the artifacts of the source separation model, which is moderate but not negligible. Third, for the accompanied singer, there is no single "ground truth" and the relationship between the model input and output may be one-to-many. This is because there are plenty of valid ways to Figure 2: A pipeline for building the accompanied singer. We use source separation to get separated singing voice and accompaniment from professionally recorded audio files. Then, we use the separated tracks to train the generators and discriminators in the GAN. In inference time, we feed an unseen accompaniment to the trained singer model and let it "sing." sing over an accompaniment track. For diversity and artistic freedom, we cannot ask the machine to generate any specific singing voice in response to an accompaniment track, even if we have paired data of vocal and accompaniment tracks. We investigate using conditional GAN to retain the possibility of generating singing voices with multiple modes. Fourth, as the proposed tasks are new, there are no established ways for performance evaluation. According to our setting, we desire our machine to generate audio waveforms with high quality and diversity, vocal-like timbre, plausible pitch contour, emotion expression, and, for the accompanied singer, that are in harmony with the given accompaniment track. But, the singing does not have to be intelligible. We propose customized objective and subjective metrics to evaluate our models in these aspects. For example, we adapt the melody harmonization model proposed by to measure the matchness between the generated vocal track and the given accompaniment track. Finally, reproducibility is a major issue, especially for a subjective task. We intend to use publiclyavailable copyright-free instrumental music as the conditional signals for testing the accompanied singer, so that other researchers can use the same testing conditions for model comparison in the future. We will also release the testing conditions for the solo singer, the generated singing voices for all our models, as well as open source our code through a public git repository [URL removed]. We focus on Jazz music in this work. Samples of the generated singing voices can be found at https://bit.ly/2mIvoIc. Our models have many possible use cases. For example, we can use the accompanied singer as a backing vocalist. In addition, we can use the free singer as a sound source-to demonstrate this, we make a song by hand in the style of Jazz Hiphop by sampling the output of our free singer. This song can be listened to at https://bit.ly/2QkUJoJ. A free singer takes no conditions at all as the input. We want it to sing freely. The singing voices from a free singer may not even sound good, but they should sound like singing voice. A free singer is like we are freely humming or singing on the road walking or in the bathroom taking a shower. We may not even know what we are singing and likely there is no underlying musical score. From the viewpoint of a generative model, training a free singer amounts to modeling a distribution P (Y), where Y ∈ R K×T is a matrix representing a sequence of K-dimensional features and T is the number of time frames. A free singing is sampled from this distribution without conditions. An accompanied singer takes as the input a sequence of accompaniment-derived features. An accompanied singer tries to generate singing voices that match the accompaniment track in some way. It is similar to the case of Karaoke, where a backing accompaniment track is played from a speaker, the lyrics and a video are displayed on a screen, and a user tries to sing according to the lyrics and the backing track. The difference is that, this time the user is a trained model and we do not ask it to follow the lyrics or the exact pitch contour of the accompaniment. The note sequence found in the singing has to be in harmony with, but not a duplicate of, that in the backing track. Training an accompanied singer amounts to modeling a distribution P (Y|A), where Y ∈ R K×T represents a feature sequence of the vocal track, and A ∈ R H×T is a feature sequence of the given accompaniment track. In our implementation, we use the mel-spectrograms for Y in compliance with the need of the vocoder. For A, different features can be tried, and we investigate using a transcription model to extract pitch features. See Section 4.1 for details. A solo singer is similar to a free singer in that both takes no conditions as the input. However, a solo singer would generate an'inner idea' first, and then sing according to that. In other words, it learns a joint distribution P (Y|I)Q(I), where I ∈ R J×T is a matrix representing the idea sequence. The singer first samples I from a distribution Q, and then uses that to conditionally sample Y from P. The inner idea I can take several forms. In this work, we instantiate this scheme with I being a chord progression (namely a sequence of chord labels). The distribution Q is modeled by an autoregressive recurrent network we build for chord progression generation (with a network architecture adapted from that in ), as described in Section 3.3. Alternatively, we can think of a solo singer as a combination of an idea generator and an an accompanied singer. For an accompanied singer, the information extracted from the given accompaniment track can take several forms such as transcribed pitches and chord progressions. A solo singer learns to generate such information on its own, without reference to an actual accompaniment track. To account for the absence of supervised data and the highly complicated spatio-temporal patterns in audio spectrograms, we propose a new adversarial net that features heavy use of ), dilated convolutions (van den ;), and feature grouping to build our singer models. We provide the algorithmic details below. Network architectures with stacked blocks of GRUs and dilated convolutions have been used to attain state-of-the-art performance in blind musical source separation . In a source separation task, a model learns to decompose, or unmix, different sources (e.g., vocal, piano, bass, drum) from a mixture signal . This requires the abilities to model the relationships between different sources as well as the relationships between neighboring time frames. The output spectrograms are also expected to be distortion-less and of high audio quality. For it has demonstrated its capability in source separation, we adopt it as a building block of the singer models. Especially, we want the singer models to also consider accompaniment information. Specifically, one such block we adopted in our models is a stack of GRU, dilated convolution with feature grouping, and group normalization . The input to the GRU, the output of the GRU, and the output of the group normalization are summed to form the output of the block. We note that the original'D2 block' used in uses dilated GRU and uses weight normalization for the dilated convolution layers. However, empirically we find that it is easier for the singer models to converge by replacing weight normalization with group normalization, and using plain GRUs is as good as using dilated GRUs. We refer to our blocks as GRU-grouped dilated convolution-group normalization block ('G3 block'). The accompanied singers and solo singers have to take conditions as part of their input. One desirable property of the models is the ability to generate voices with arbitrary length, as the conditional signal can be of variable length. Besides, the model has to deal with the one-to-many issue mentioned in Section 1, and the absence of supervisory signals. With these issues in mind, we design a GAN architecture for score and lyrics-free voice generation. In particular, we pay special attention to the following three components: 1) the network architecture, 2) the input noises for GAN, and 3) the loss function of the discriminator. Let us first take a look at two existing GAN models for audio generation: and. Their generators and discriminators are both based on 2D convolutions, transposed 2D convolutions and dense (linear) layers. The generators take a vector z ∈ R U as the input noise and use transposed convolutions to expand z so that a temporal dimension emerges in the expanded intermediate matrices. The number of temporal frames in the final output depends on the total strides used in all the transposed convolutions. The discriminators take the output of the generators or the real signal as the input, and compress the input matrix with convolution layers until the output becomes a single value represents the prediction of true (real) or false (generated) data. A main reason why existing models cannot generate variable-length output is the need to expand z by transposed convolution layers. We remedy this by using an architecture consisting of the proposed G3 blocks, and convolutions without strides, for both the generators G(·) and discriminators D(·). Moreover, instead of using a single noise vector, our models take as input a sequence of noise vectors, denoted as Z ∈ R U ×T, that has the same temporal length as the desired output Y. Each column of Z is sampled independently from a Gaussian distribution N ormal. At the first glance, it might feel unnatural to have one noise vector per frame as that may in fast oscillations in the noises. However, we note that the output of G(·) for the t-th frame depends not only on the t-th column of Z (and C or I), but the entire Z (and the condition matrices), due to the recurrent GRUs in the model. We expect that the GRUs in the discriminator D(·) would force G(·) to generate consistent consecutive frames. Therefore, the effect of the frame-wise noises might be introducing variations to the generation (e.g., by adjusting the modes of the generated frame-wise features). As for the loss function of D(·), we experiment with the following three options: the vanilla GAN, the LSGAN that adopts the least squares loss function for the discriminator, and the boundary equilibrium GAN (BEGAN) that adopts an "auto-encoder style" discriminator loss. The D(·) in either GAN or LSGAN is implemented as a classifier aiming to distinguish between real and generated samples, whereas the D(·) in BEGAN is an autoencoder aiming to reconstruct its input. Specifically, in BEGAN, the loss functions l D and l G for the discriminator and generator, as in the case of the accompanied singer, are respectively: where X ∈ R K×T is the feature sequence of a real vocal track sampled from the training data, G(Z, C) ∈ R K×T is the feature sequence for the generated vocal track, and L(·) is a function that measures how well the discriminator D(·), implemented as an auto-encoder, reconstructs its input: where we use M w,t to denote the (w, t)-th element of a matrix M (and similarly for D(M, C) w,t ). Moreover, the variable τ s in Eq. is introduced by BEGAN to balance the power of D(·) and G(·) during the learning process. It is dynamically set to be for each training step s, with τ s ∈. λ and γ are manually-set hyperparameters. Empirical comparison of the performance of GAN, LSGAN and BEGAN can be found in Appendix D. It turns out that the BEGAN-based one, referred to as G3BEGAN hereafter, works the best. We also use auto-regressive RNN to build a chord progression generator for implementing the solo singer. Our chord generator is trained on the Wikifonia dataset (http://www.wikifonia. org/), a set of 6,670 songs in the leadsheet format (i.e., with separated symbolic melody and chord tracks). Its chord vocabulary covers 612 different chords. We set the harmonic rhythm of the chord generator such that a chord change may occur every beat. We desire the chord generator to freely generate chord progressions across different tempo values, time signatures, and keys. Moreover, the generated chord progression has to be rhythmically correct. In achieving so, we encode the tempo, time signatures, and key information (which are available in the Wikifonia dataset) as the initial hidden state of the RNN, and concatenate the chord vector from last time step with the beat position (e.g., 1, 2, 3, 4) of that time step as the input to the RNN. For data augmentation, we transpose the chord progressions found in Wikifonia to 12 possible keys. Once trained, we use the chord generator to create chord progressions for testing the solo singer. With a one-hot encoding, each column of the condition matrix I would have dimension J = 660. More details of G3BEGAN and the chord generator can be found in Appendices A.1 and A.2. In our implementation, we use 80-dimensional mel-spectrograms as the acoustic features modeled and generated by the singer models (i.e., K = 80). We use the python package librosa , with default settings, to compute the mel-spectrograms from audio. A mel-spectrogram is passed to a WaveRNN vocoder to generate an audio signal from melspectrograms. Our implementation of the WaveRNN vocoder is based on the code from Fatchord. Instead of using off-the-shelf pre-trained vocoders, which are typically trained for TTS, we train our vocoder from scratch with a set of 3,500 vocal tracks we separate (by a separation model) from an in-house collection of music that covers diverse musical genres. One main difficulty in conducting this research is to get vocal tracks for training our singer models. Existing multitrack datasets that contain clean vocal tracks are usually diverse in musical genre and the singing timbre, making it hard for our models to learn the relationship between singing and accompaniment. As we set out to focus on Jazz music in this work, we opt for collecting our own vocal tracks from Jazz music only. We therefore implement our source separation model following the architecture proposed by , which represents the state-of-the-art as evaluated on the MUSDB benchmark . We use the whole MUSDB dataset to train our separation model. This dataset contains clean vocal and accompaniment tracks. However, for constructing the condition matrix A of our accompanied singer, we desire to have separated piano tracks as well (we will explain why shortly). We therefore collect additionally 4.5 hours of Jazz piano solo audio and use them to augment MUSDB for training our source separation model, which can as a isolate out not only the vocal track but also the piano track from an arbitrary song. We collect 17.4 hours of Jazz songs containing female voices and 7.6 hours of Jazz songs with male voices. We use the aforementioned separation model to get the vocal tracks. For batched training, we divide the tracks into 10-second sub-clips. Sub-clips that contain less than 40% vocals, as measured from energy, are removed. This leads to 9.9-hour and 5.0-hour training data for female and male Jazz vocals, respectively. 200 and 100 sub-clips are reserved from the training set as the validation set for female singing and male singing, respectively. Each model is trained for 500 epochs. For GAN and LSGAN, we use the models at the 500th epoch for evaluation. For BEGAN, the parameters of the epoch with the best convergence rate are used for evaluation. For the accompanied singer, we experiment with extracting pitch-related information from the accompaniment track to form the matrix A that conditions the generation of the vocal track. The assumption here is that whether the generated vocal track is in harmony with the accompaniment track can be largely determined by pitch-related information. For this purpose, we implement a piano transcription model to transcribe the separated piano track, leading to 88-dimensional transcribed frame-wise pitch as the accompaniment condition (i.e., H = 88, as there are 88 piano notes). We implement a piano transcription model with the G3 blocks introduced in Section 3.1, following the training procedure of . We also implement the model of. Under the same training setting, we find that ours is slightly worse than theirs in note F1 score (0.779 vs 0.794) but slightly better in the note precision score (0.834 vs 0.823). We decide to use our model for the better precision, but the difference of using either of them should be small. The clips in the training set of our singer models may not contain piano playing (see Table 5 for a summary of the datasets). Even if a clip contains piano playing, the piano may not play across the entire clip. Hence, the models have to learn to sing either with or without the piano accompaniment. For performance evaluation, we collect 5.3 hours of Jazz music from Jamendo (https://www. jamendo.com), an online platform for sharing copyright-free music. As said in Section 1, this test set is meant to be public. We apply source separation to the audios, divide each track into 20-second sub-clips, 4 and remove those that do not contain piano. Piano transcription is also applied to the separated piano track, yielding 402 20-second sub-clips for evaluation. 402 chord progressions are generated by the chord generator to evaluate the solo singer and to compute the matchness. As this is a new task, there is no previous work that we can compare with. Therefore, we establish the baselines by 1) computing the baseline objective metrics (see Section 4.3) from the training data of the singing models, and 2) using existing SVS systems for synthesizing singing voices. For the SVS baselines, we employ Sinsy and Synthesizer V , the two well-known SVS systems that are publicly accessible. For Sinsy, we use the publicly available repository 5 to query the Sinsy API (http://sinsy.jp/); we use the HMM version instead of the deep learning version as the latter cannot generate male voices. For Synthesizer V, we use their software (https://synthesizerv.com/). We use Sinsy for both objective and subjective tests, but Synthesizer V for subjective test only, for the latter does not provide a functionality to batch process a collection of MIDI files and lyrics. SVS systems have to take lyrics and a melody to synthesize singing voices. For the lyrics, we choose to use multiple'la,' the default lyrics for Synthesizer V. 6 For the melodies, we consider two methods: 1. Vocal transcription from singer training data. We use CREPE to transcribe the separated vocals from the singer training data, and convert it to MIDI format. 2. Piano transcription from the Jamendo testing data. As described in Section 4.1, we have separated and transcribed the piano part of the Jamendo data. Yet, the piano transcription often contains multiple notes at the same time. We use the skyline algorithm to the transcription to get a melody line comprising the highest notes. The best way to evaluate the performance of the singer models is perhaps by listening to the generated . Therefore, we encourage our readers to listen to the audio files provided in the supplementary material. However, objective evaluation remains desirable, either for model development or for gaining insights into the generation . We propose the following metrics for our tasks. • Vocalness measures whether an audio clip contains singing voices. There are different publicly available tools for detecting singing voices in an audio mixture (e.g., ). We choose the JDC model for it represents the state-of-theart. In this model, the pitch contour is also predicted in addition to the vocal activation. If the pitch at a frame is outside a reasonable human pitch range (73-988 Hz defined by JDC), the pitch is set to 0 at that frame. We consider a frame as being vocal if it has a vocal activation ≥ 0.5 AND has a pitch > 0. Moreover, we define the vocalness of an audio clip as the proportion of its frames that are vocal. The tool is applied to the non-silence part of an audio 7 of the generated singing voices only, excluding the accompaniment. • Average pitch: We estimate the pitch (in Hz) for each frame with two pitch detection models: the state-of-the-art monophonic pitch tracker CREPE (a), and JDC. The average pitch is computed by averaging the pitches across the frames with confidence higher than 0.5 for CREPE, and across the frames that are estimated to be vocal for JDC. 4 Please note that this is longer than the 10-second sub-clips we used to train the singer models. This is okay as our model can generate variable-length output. 5 https://github.com/mathigatti/midi2voice 6 As our models do not contain meaningful lyrics, to be fair the baselines should not contain meaningful lyrics either. We choose'la' because people do sometimes sing with'la' and it has no semantic meaning. An alternative way to get the lyrics is by randomly sampling a number of characters. However, randomly sampling a reasonable sequence of characters is not a trivial task as well. 7 The non-silence frames are derived by using the librosa function'effects. signal to frame nonsilent.' 312 ± 70 310 ± 56 0.60 ± 0.14 -9.24 ± 3.09 Singer train data (vocals, male) 263 ± 93 258 ± 75 0.64 ± 0.16 -9.09 ± 3.22 Singer train data (accomp., female) --0.05 ± 0.09 -Singer train data (accomp., male) --0.12 ± 0.15 -MUSDB clean vocals 271 ± 81 283 ± 75 0.59 ± 0.14 - Table 1: Result of objective evaluation for our singer models and a few baseline methods. • Singing-accompaniment matchness: As detailed in the appendix, to objectively measure matchness, we build a melody harmonization RNN by adapting the chord generator described in Section 3.3. Given a pair of melody and chord sequences, the model computes the likelihood of observing that chord sequence as the output when taking the melody sequence as the model input. We use the average of the log likelihood across time frames as the matchness score. As the harmonization model considers symbolic sequences, we use CREPE to transcribe the generated voices, and Madmom (Böck et al., 2016) to recognize the chord sequence from the accompaniment track. Several observations can be made from the shown in Table 1. In terms of the average pitch, we can see that the of our model is fairly close to that of the singing voices in the training data. Moreover, the average pitch of the generated female voices is higher than that of the generated male voices as expected. We can also see that the Sinsy singing voices tend to have overly high pitches, when the melody line is derived from a piano playing (denoted as 'testing piano skyline.'). In terms of vocalness, our models score in general lower than Sinsy, and the singing voices in the training data. However, the difference is not that far. As a reference, we also compute the vocalness of the accompaniments in the training set (denoted as 'accomp.') and it is indeed quite low. As for matchness, we show in Table 1 the score computed from the real melody-chords pairs of Wikifonia (-7.04) and that from random pairs of Wikifonia (-13.16). We can see that the accompanied singers score higher than the random baseline and the free singer as expected. 9 Moreover, the matchenss scores of the accompanied singers are close to that of the singer training data. Examples of the generated spectrograms of our models can be found in the appendix. From visually inspecting the spectrograms and listening to the , the models seem to learn the characteristics of the singing melody contour (e.g., the F0 is not stable over time). Moreover, the female singer models learn better than the male counterparts, possibly because of the larger training set. Vocalness Expression Matchness G3BEGAN (20 epochs) 1.59 ± 0.82 1.93 ± 0.99 1.98 ± 0.88 2.18 ± 1.08 G3BEGAN (240 epochs) 2.24 ± 0.93 2.66 ± 1.01 2.60 ± 1.01 2.58 ± 1.05 G3BEGAN (final) 2.38 ± 0.96 2.98 ± 1.02 2.85 ± 1.00 2.74 ± 1.04 Table 2: Mean opinion scores (MOS) and standard deviations with respect to four evaluation criteria collected from the user study, for three different versions of accompanied singer (female). The scores are in 5-point Likert scale and are from 1 to 5; the higher the better. Sound quality Vocalness Expression Matchness G3BEGAN (final) 1.71 ± 0.70 2.39 ± 1.11 2.27 ± 1.06 2.34 ± 1.16 Sinsy 3.19 ± 1.07 2.90 ± 1.01 2.40 ± 0.98 2.10 ± 0.90 Synthesizer V 3.57 ± 1.07 3.30 ± 1.24 3.25 ± 1.10 3.35 ± 1.15 Table 3: MOS from the second user study, comparing our model and two existing SVS systems. We conduct two online, non-paid user studies to evaluate the accompanied singer, the female one. In the first user study, we compare the'final' model (with the number of epochs selected according to a validation set) against two early versions of the model trained with less epochs. In the second one, we compare the proposed accompanied singer with Sinsy and Synthesizer V. In the first study, we recruit 39 participants to each rate the generated singing for three different accompaniment tracks (each 20 seconds), one accompaniment track per page. The subjects are informed the purpose of our research (i.e., score and lyrics-free singing voice generation) and the user study (to compare three computer models), and are asked to listen in a quiet environment with proper headphone volume. No post-processing (e.g., noise removal, EQ adjustment) is applied to the audio. The ordering of the of the three models is randomized. The process of the second study is similar to the first one, but it includes five different accompaniments (randomly chosen from those used in the first user study) and the respective generated/synthesized singing voices. The melodies used for synthesis are those from the piano skyline of the test data, so that our model can be compared with the synthesis methods with the same accompaniment. A separate set of 21 subjects participate in this study. The audio files used in this user study can be downloaded from https://bit.ly/2qNrekv. Tables 2 and 3 show the of the two studies. We can see that the model indeed learns better with more epochs. Among the four evaluation criteria, the Sound Quality is rated lower than the other three in both studies, suggesting room for improvement. By comparing the proposed model with the two SVS systems, we see that Synthesizer V performs the best for all the evaluation criteria. Our model achieves better Matchness than Sinsy, and achieves a rating close to Sinsy in Expression. In general, we consider the as promising considering that our models are trained from scratch with little knowledge of human language. While early work on SVS is mainly based on digital signal processing (DSP) techniques such as sampling concatenation , machine learning approaches offer greater flexibility and have been more widely studied in recent years. Hidden Markov models 10 We note that Sinsy and Synthesizer V have an unfair advantage on matchness because their singing voices are basically synthesized according to the melody lines of the accompaniment. From Table 3, we see that Synthesizer V does exhibit this advantage, while Sinsy does not. We observe that the Sinsy singing voices do not always align with the provided scores. The fact that Synthesizer V has higher audio quality seem to promote its score in the other criteria. The presence of the of Synthesizer V seems to also make the subjects in the second study rate the proposed model lower than the subjects do in the first study. (HMMs), in particular, have been shown to work well for the task . The Sinsy system, a baseline model in Section 4, is also based on HMMs . report improved naturalness by using deep neural nets instead of HMMs. Since then, many neural network models have been proposed. The model presented by uses simple fully-connected layers to map symbolic features extracted from the user-provided scores and lyrics, to a vector of acoustic features for synthesis. The input and output features are time-aligned frame-by-frame beforehand by well-trained HMMs. The input features consist of score-related features (e.g., the key of the current bar and the pitch of the current musical note), and lyrics-related ones (the current phoneme identify, the number of phonemes in the current syllable, and the duration of the current phoneme). The output features consist of spectral and excitation parameters and their dynamic features , which altogether can then be turned into audio with a DSP technique called the MLSA filter . The aforementioned model has been extended in many aspects. For instance, using convolutional layers and recurrent layers in replacement of the fully-connected layers for learning the mapping between input and output features has been respectively investigated by and Kim et al. (2018b). Using neural vocoders such as the WaveNet (van den) instead of the MLSA filter has been shown to improve naturalness by. Rather than using hand-crafted features for the input and output, Lee et al. (2019a) train a model to predict the mel-spectrogram directly from time-aligned lyrics and pitch labels, and then use the Griffin-Lim algorithm to synthesize the audio. Modern techniques such as adversarial loss and attention module have also been employed (a). A follow-up work adds a speaker encoder to the network to achieve multi-singer SVS (b). Synthesizer V , the other baseline model we employ in Section 4, is based on a hybrid structure that uses both deep learning and sample-based concatenation. While exciting progress has been made to SVS, the case of score and lyrics-free singing voice generation, to our best knowledge, has not been tackled thus far. Similar to (a), we do not use hand-crafted features and we train our model to predict the mel-spectrograms. Using neural nets for score-conditioned instrumental audio generation have also been investigated in recent years. However, existing work is mostly concerned with the generation of single notes of, for example, 4-second long (Défossez et al., 2018;). A deep neural network that is capable of generating variable-length audio (e.g., a "recurrent generator") as the proposed singer models do, to our knowledge, has not been much studied. In this paper, we have introduced a novel task of singing voice generation that does not use musical scores and lyrics. Specifically, we proposed three singing schemes with different input conditions: free singer, accompanied singer, and solo singer. We have also proposed a BEGAN based architecture that uses GRUs and grouped dilated convolutions to learn to generate singing voices in an adversarial way. For evaluating such models, we proposed several objective metrics and implemented a model to measure the compatibility between a given accompaniment track and the generated vocal track. The evaluation shows that the audio quality of the generated voices still leave much room for improvement, but in terms of humanness and emotion expression our models work fine. Score and lyrics-free singing voice generation is a new task, and this work represents only a first step tackling it. There are many interesting ideas to pursue. For example, we have chosen to extract pitch-related information only from the accompaniment track for the accompanied singer, but a more interesting way is to let the model learns to extract relevant information itself. In the near future, we plan to investigate advanced settings that allow for timbre and expression control, and experiment with other network architectures, such as coupling a fine-grained auto-regressive model with a multiscale generation procedure as done in MelNet , using a discriminator that examines different chunks of the generated audio as done in PatchGAN for the vision domain , or using multiple discriminators that evaluate the generated audio based on multi-frequency random windows as done in GAN-TTS (Bińkowski et al., 2019) The generator in G3BEGAN is implemented with a stack of two G3 blocks. Please see Table 4 for details of the network architecture. The chord generator is aimed to generate chord progressions freely under some given conditions. It supports 12 major and 12 minor keys, 10 tempo options from 60 to 240 BPM, 6 time signature options, and 51 chord qualities (612 chords in total). The conditions, key, tempo, and time signatures, are encoded into one-hot representation and concatenated together as a 40-dimension vector. The model mainly consists with 3 stacked GRU layers, each with 512 hidden variables. The input of each time step is a 524-dimensional vector consisting of a chord embedding and a beat-related one-hot positional encoding (to encourage the model to follow certain rhythmical pattern. This input array passes through a fully-connected layer to 512-dimension and is used as the input of the GRUs. The training data are the leadsheets from the Wikifonia dataset. We augmented the data by rotating the keys, leading to in total 80,040 leadsheets for training. Table 5 : A summary of the datasets employed in this work. The first three datasets and the 4.5-hour Piano solo part of the last dataset are Jazz music, whereas the others may not. The 'Processing' column indicates the music processing models that have been applied to process the respective dataset. (Notation-Tr.: training. Ev.: evaluating. SS: source separation. PT: piano transcription) The melody harmonization model is modified from the chord generator described in Appendix A.2, using additionally the melody tracks found in the Wikifonia dataset. Specifically, the model intends to generate a chord sequence given a melody sequence. Such a model can be learned by using the pairs of melody and chord tracks in Wikifonia. We add the chroma representation of the melody with window size of a quarter-note to the input vector. The matchness of the real pairs of melody and chord progression in Wikifonia is -7.04±2.91. If we pair a melody with a randomly selected chord progression and calculate the matchness, the score becomes -13.16±3.72. B APPENDIX: DATASETS C APPENDIX: EXAMPLES OF THE SPECTROGRAM OF THE GENERATED SINGING VOICES Examples of the spectrograms of the generated singing voices can be found in Figures 3 and 4. (a) The given instrumental, accompaniment track (b) The voices generated by the accompanied singer (female) (c) The voices generated by the accompanied singer (male) Figure 3: Samples of spectrograms generated by our accompanied singers: (left) the given accompaniment tracks, the voices generated by (middle) the female singer and (right) the male singer. Average pitch (Hz) Vocalness Model CREPE JDC JDC BEGAN (female) 288 ± 28 292 ± 28 0.48 ± 0.09 GAN (female) 307 ± 7 371 ± 25 0.22 ± 0.17 LSGAN (female) 1130 ± 9 453 ± 14 0.28 ± 0.03 Table 6: Result of using different GAN losses, for free singer (female). We experiment different GANs, including BEGAN , vanilla GAN (, and LSGAN , for the case of building the free singer. The output of the discriminators in the vanilla GAN and LSGAN is a single real/fake value. To compare the three GANs as fairly as possible, the discriminators used in vanilla GAN and LSGAN are almost the same as the one used in the BEGAN. The only difference is that the discriminators used in vanilla GAN and LSGAN have an extra average-pooling layer in the output. The validation losses at different training epochs are shown in Figure 5, and the metrics are shown in Table 6. In Figure 5, we can see that, according to our implementation, only BEGAN converges. In Table 6, we can see that the BEGAN model has much higher Vocalness than other models. By listening to the generated singing voices of the GAN and LSGAN models, we find that they are basically noises. Figure 4: Samples of spectrograms generated by our (left) free singers and (right) solo singers. We can see salient pitch contour in the spectrograms. Moreover, the pitches sung by the male singers seem on average lower than those sung by the female singers. which can be used to examine how well the model converges.

Our models generate singing voices without lyrics and scores. They take accompaniment as input and output singing voices.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We automatically extract fingering information from videos of piano performances, to be used in automatic fingering prediction models.