Question: represent the scientific article to find a short tldr

Question: Meta-learning algorithms learn to acquire new tasks more quickly from past experience. In the context of reinforcement learning, meta-learning algorithms can acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can in principle be used to train optimal meta-learners. Our experimental indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and significantly exceeds the performance of learning from scratch. Reusing past experience for faster learning of new tasks is a key challenge for machine learning. Meta-learning methods achieve this by using past experience to explicitly optimize for rapid adaptation (; ; ; a; ;). In the context of reinforcement learning (RL), meta-reinforcement learning (meta-RL) algorithms can learn to solve new RL tasks more quickly through experience on past tasks (b; a). Typical meta-RL algorithms assume the ability to sample from a pre-specified task distribution, and these algorithms learn to solve new tasks drawn from this distribution very quickly. However, specifying a task distribution is tedious and requires a significant amount of supervision (b; b) that may be difficult to provide for large, real-world problem settings. The performance of meta-learning algorithms critically depends on the meta-training task distribution, and meta-learning algorithms generalize best to new tasks which are drawn from the same distribution as the meta-training tasks. In effect, meta-RL offloads much of the design burden from algorithm design to designing a sufficiently broad and relevant distribution of meta-training tasks. While this offloading helps in acquiring representations for fast adaptation to the specified task distribution, specifying this is often tedious and challenging. A natural question is whether we can do away with manual task design and develop meta-RL algorithms that learn only from unsupervised environment interaction. In this paper, we take an initial step toward the formalization and design of such methods. Our goal is to automate the meta-training process by removing the need for hand-designed metatraining tasks. To that end, we introduce unsupervised meta-RL: meta-learning from a task distribution that is acquired automatically, rather than requiring manual design of the meta-training tasks. Unsupervised meta-RL methods must solve two difficult problems together: meta-RL with broad task distributions, and unsupervised exploration for proposing a wide variety of tasks for meta-learning. Since the assumptions of our method differ fundamentally from prior meta-RL methods (we do not assume access to hand-specified meta-training tasks that use human-specified reward functions), the best points of comparison for our approach are learning meta-test tasks entirely from scratch with conventional RL algorithms. Our method can also be thought of as automatically acquiring an environment-specific learning procedure for deep neural network policies, somewhat related to data-driven initialization procedures explored in supervised learning (Krähenbühl et al., 2015;). The primary contributions of our work are to propose a framework for unsupervised meta-RL; to sketch out a family of unsupervised meta-RL algorithms; to provide a theoretical derivation that allows us to reason about the optimality of unsupervised meta-RL methods in terms of mutual information objectives; and to describe an instantiation of an algorithm from this family that builds on a recently proposed procedure for unsupervised exploration and modelagnostic meta-learning (MAML) (a). In addition to our theoretical derivations, we provide an empirical evaluation that studies the performance of two variants of our approach on simulated control tasks. Our experimental evaluation shows that, for a variety of tasks, unsupervised meta-RL can effectively acquire RL procedures that perform significantly better than standard RL methods that learn from scratch, without requiring additional task knowledge. Our work lies at the intersection of meta-RL, goal generation, and unsupervised exploration. Metalearning algorithms use data from multiple tasks to learn how to learn, acquiring rapid adaptation procedures from experience (; ; ; ; ; ; ; ; ; a; ;). These approaches have been extended into the setting of RL (b; ; a; ; ; ; ; a; . In practice, the performance of meta-learning algorithms depends on the user-specified meta-training task distribution. We aim to lift this limitation and provide a general recipe for avoiding manual task engineering for meta-RL. To that end, we make use of unsupervised task proposals. These proposals can be obtained in a variety of ways, including adversarial goal generation , information-theoretic methods (; ;), and even random functions. We argue that, theoretically, methods based on mutual information have the potential to provide optimal task proposals for unsupervised meta-RL. Exploration methods that seek out novel states are also closely related to goal generation methods (; ; ; ;), but do not by themselves aim to generate new tasks or learn to adapt more quickly to new tasks, only to achieve wide coverage of the state space. These methods are complementary to our approach, but address a distinct problem. While model-based RL methods (; ; ; b;) might likewise use unsupervised experience to learn a dynamics model, they are not equipped with a strategy to employ that model to learn effectively at test time. Related to our work are prior methods that study the training of goal-conditioned policies (; ;). Indeed, our theoretical derivation first studies the goal reaching case, before generalizing it to the more general case of arbitrary tasks. This generalization allows unsupervised meta-learning methods to solve arbitrary tasks at meta-test time without being restricted to a task parameterization or goal specification. Additionally, in Section 3.4 we discuss why the framework of meta-learning gives us theoretical benefits over the goal reaching paradigm. Figure 1: Unsupervised meta-reinforcement learning: Given an environment, unsupervised meta-RL produces an environment-specific learning algorithm that quickly acquire new policies that maximize any task reward function. In this paper, we consider the problem of automatically tailoring (i.e., learning) a reinforcement learning algorithm to solve tasks on a specific environment. This learning algorithm should be meta-learned without requiring any human supervision or specification of task distribution. The implicit assumption in this formulation will be that all test-time tasks share the same dynamics but use different reward functions. To talk about environments on which many tasks might be performed, we consider a controlled Markov process (CMP) -a Markov decision process without a reward function, C = (S, A, P, γ, ρ), with state space S, action space A, transition dynamics P, discount factor γ and initial state distribution ρ. The CMP along with this reward function r z produces a Markov decision processes M i = (S, A, P, γ, ρ, r z). The goal of the learning algorithm f is to learn an optimal policy π * i (a | s) for any reward function r z that is provided with the CMP. With this notation in place, we formally define the unsupervised meta-RL as follows: Definition 1. Unsupervised Meta-Reinforcement Learning: Given a CMP C, output a learning algorithm f that can learn new tasks efficiently in MDPs defined by CMP C together with an (unknown) reward function r z. The data available to unsupervised meta-RL are unsupervised interactions with the CMP, and does not include observing any of the reward functions r z. We illustrate this problem setting in Figure 1. In contrast, in standard meta-reinforcement learning, a task distribution P (T) is known and we simply try to optimize for a fast learning algorithm on that known task distribution. A detailed definition can be found in Section 3.2. In the rest of this section, we will first sketch out a general recipe for an unsupervised meta-RL algorithm, present a theoretical derivation for an optimal unsupervised meta-learning method, and then instantiate a practical approximation to this theoretically-motivated approach using components from recently proposed exploration and meta-learning algorithms. Our framework, unsupervised meta-RL, consists of a task proposal mechanism and a meta-learning method. Formally, we will define the task distribution as a mapping from a latent variable z ∼ p(z) to a reward function r z (s, a): S × A → R 1. That is, for each value of the random variable z, we have a different reward function r z (s, a). Under this formulation, learning a task distribution amounts to optimizing a parametric form for the reward function r z (s, a) that maps each z ∼ p(z) to a different reward function. The choice of this parametric form represents an important design decision for an unsupervised meta-learning method, and the ing set of tasks is often referred to as a task or goal proposal procedure. The second component is the meta-learning 2 algorithm, which takes the family of reward functions induced by p(z) and r z (s, a) along with the associated CMP, and meta-learns a RL algorithm f that can quickly adapt to any task from the task distribution defined by 1 In most cases p(z) is chosen to be a uniform categorical so it is not challenging to specify 2 A meta-reinforcement learning algorithm learns how to learn: it uses a set of meta-training tasks to learn a learning function f, which can then learn a new task. We refer to this learned learning function f as an "acquired reinforcement learning procedure," following prior work, such as MAML (a) and RL2 (b) p(z) and r z (s, a) in the given CMP. The meta-learned algorithm f can then learn new tasks quickly at meta-test time, when a user-specified reward function is actually provided. This generic design for an unsupervised meta-RL algorithm is summarized in Figure 1. The "no free lunch theorem" might lead us to expect that a truly generic approach to proposing a task distribution would not yield a learning procedure f that is effective on any real tasks, or even on the meta-training tasks. However it is important to note that even without a reward function, an unsupervised meta-learning algorithm can collect and organize meaningful information about the environment. For example, this information could include policies for reaching certain states, estimates for the distances between states, a map of the environment, or a model of the dynamics. Importantly, the meta-learner is not restricted to learning information that is human-interpretable, especially since human-interpretable representations may not be optimal for fast adaptation. In the following sections, we will discuss how to formulate an optimal unsupervised meta-learner that minimizes regret on new meta-test tasks in the absence of any prior knowledge. We first discuss the notion of an optimal meta-learner and then show how we can train one without requiring task distributions to be hand-specified. A more general version of this algorithm might also use f to inform the acquisition of tasks, allowing for an alternating optimization procedure the iterates between learning r z (s, a) and updating f, for example by designing tasks that are difficult for the current algorithm f to handle. However, in this paper we will consider the stagewise approach, which acquires a task distribution once and meta-trains on it, leaving the iterative variant for future work. We first define an abstract notion of an optimal meta-learner, when given a task distribution. This definition will be useful to showing that for certain test-time task distributions, meta-training on self-proposed tasks (i.e., unsupervised meta-training) will yield the optimal meta-learner at test-time. We assume that an optimal meta-learner takes in a distribution over tasks for a given CMP and outputs a learning procedure f that minimizes expected regret when learning tasks drawn from the same distribution as seen in meta-training. As before, the task distribution is defined by a variable z ∼ p(z) and a reward function r z (s, a): S × A → R. An optimal meta-learner optimizes where p(r z) is a distribution over reward functions, parameterized by z, R(f, r z) is the total return obtained by using meta learner f on reward function r z, and f is an optimal meta-learner. This is equivalent to the expected reward objective used by most meta-RL methods (a; b). Note that the behavior of the optimal meta-learner depends on the specific task distribution. For example, the optimal behavior for manipulation tasks involves moving a robot's arms, while the optimal behavior for locomotion tasks involves moving a robot's legs. Optimizing Equation 1 requires inferring new tasks and acquiring the policy for each task. While learning an internal model of the environment might assist in these steps, an optimal meta-learner would also need to know how to leverage this model. In the remainder, we will abstract away in the internal representation of the meta-learner and instead focus on the effect of the training task distribution on the behavior induced in an optimal meta-learner, and how this suggests a method to construct unsupervised meta-learning algorithms. We will now derive an optimal unsupervised meta-learner for the special case of goal reaching tasks, and then generalize this approach to solve arbitrary tasks in Section 3.4. In both, we will restrict our attention to CMPs with deterministic dynamics. In this goal-reaching setting, we consider episodes with finite horizon T and a discount factor of γ = 1. Tasks correspond to reaching an unknown goal state s g. We will only consider the agent's state at the last time step in each episode, so the (unknown) reward function is always of the form We will first assume that goal states are drawn from some known distribution p(s g), and later will show how we can remove this assumption. We define ρ T π (s) as the probability that policy π visits state s at time step t = T. If s g is the true goal, then the event that the policy π reaches s g at the final step of an episode is a Bernoulli random variable with parameter p = ρ T π (s g). Thus, the expected hitting time of this goal state is HITTINGTIME π (s g) = 1 ρ T π (s g) We now define a meta-learner f π in terms of an exploration policy π. Before the goal is found, f π uses policy π to explore. Once the goal s g is found, the meta-learner takes actions to always return to state s g. Meta-learner f π incurs one unit of regret for each step before it has found the goal, and zero regret afterwards. The expected cumulative regret is therefore the expectation of the hitting time, taken with respect to p(s g) = p(r g): In this special case, an optimal meta-learner as defined in Section 3.2 will explore for a number of episodes until it finds the goal state. After the meta-learner finds the goal state, it would always choose the trajectory that reaches that goal state under deterministic dynamics. Thus, the cumulative regret of the meta-learner is the number of episodes required to find the goal state. By our assumption that the meta-learner only receives information about the task if it has reached the goal state at the end of the episode, the meta-learner cannot use information about multiple goals within a single episode. We can minimize the regret in Equation 2 w.r.t. the marginal distribution ρ T π. Using the calculus of variations (for more details refer to Appendix C in), the (exploration) policy for the optimal meta-learner, π *, satisfies: The analysis so far tells us how to obtain the optimal meta-learner if were were given the goal sampling distribution, p(s g). If we do not know this distribution, then we cannot compute the optimal policy using Equation 3. In this case, we resort to bounding the worst-case regret of our policy: Lemma 1. Let π be a policy for which ρ T π (s) is uniform. Then f π has lowest worst-case regret. The proof is in Appendix A. Given this , we know that the exploration policy of the optimal meta-learner should have a uniform state marginal distribution. While Lemma 1 is only directly applicable in settings where there exists a policy that achieves a uniform state marginal distribution, we might expect the intuition to carry over to other settings. The minimax optimal meta-learner corresponds to exploring over a uniform distribution over goal states, so we can acquire this metalearner by training on a goal-reaching task distribution where the goals are uniformly distributed. Constructing this distribution is hard, especially in high-dimensional state spaces. How might we propose uniformly-distributed goals as tasks during unsupervised meta-training? The key idea is that this uniform goal proposal distribution can be obtained by maximizing the mutual information between z and the final state s T: where µ denotes a probability distribution over terminal states s T and latent variables z, and P µ is a subset of such distributions. Observe that this objective contains two competing terms. is a Dirac distribution, centered at a state s z specified by the latent z. Then any distribution µ that maximizes mutual information I µ (s T ; z) The key idea is that a joint distribution µ(s T, z) can be defined implicitly via a latent-conditioned policy. This policy is not a meta-learned model, but rather will become part of the task proposal mechanism. For a given prior µ(z) and latent-conditioned policy µ(a | s, z), the joint likelihood is µ(τ, z) = µ(z)p(s 1) t p(s t+1 | s t, a t)µ(a t | s t, z), and the marginal likelihood is simply given by µ(s T, z) = µ(τ, z)ds 1 a 1 · · · a T −1. We define P µ as all distributions µ(s T, z) that can be constructed in such a way using a Markovian policy. Note that the assumption in Lemma 2 does not always hold, as stochasticity in the environment may mean that µ(s T | z) is never a Dirac distribution, or the existence of unreachable states may mean that µ(s T) can never be uniform. Finally, we construct a distribution over tasks from the joint distribution µ. We then define our task proposal distribution by sampling z ∼ p(z) and using the corresponding reward function r z (s T, a T) log p(s T | z). This reward function does not depend on the action a T. In the case when both the prior µ(z) and the marginal µ(s T) are uniform, this reward function is equivalent to the DIAYN reward function log µ(z | s T), up to additive constants. Since each latent z corresponds to a single state s T when the mutual information is maximized, our task distribution corresponds exactly to sampling goals uniformly when z is sampled uniformly. As argued above, successfully meta-learning on the uniform goal distribution produces the minimax-optimal meta-learner. To extend the analysis in the previous section to the general case, and thereby derive a framework for optimal unsupervised meta-learning, we will consider "trajectory-matching" tasks. These tasks are a trajectory-based generalization of goal reaching: while goal reaching tasks only provide a positive reward when the policy reaches the goal state, trajectory-matching tasks only provide a positive reward when the policy executes the optimal trajectory. The trajectory matching case is more general because, while trajectory matching can represent different goal-reaching tasks, it can also represent tasks that are not simply goal reaching, such as reaching a goal while avoiding a dangerous region or reaching a goal in a particular way. As before, we will restrict our attention to CMPs with deterministic dynamics. These non-Markovian tasks essentially amount to a problem where an RL algorithm must "guess" the optimal policy, and only receives a reward if its behavior is perfectly consistent with that optimal policy. We will show that mutual information between z and trajectories yields the minimum regret solution in this case, and then show that unsupervised meta-learning for the trajectory-matching task is at least as hard as unsupervised meta-learning for general tasks (though, in practice, general tasks may be easier). Formally, we define a distribution of trajectory-matching tasks by a distribution over goal trajectories, p(τ *). For each goal trajectory τ *, the corresponding trajectory-level reward function is Analogous to before, we define the hitting time as the expected number of episodes to match the target trajectory: We then define regret as the expected hitting time: Using the same derivation as before, the exploration policy for the optimal meta-learner is However, obtaining such a policy requires knowing the trajectory distribution p(τ). In the setting where p(τ) is unknown, the minimax policy is simply uniform: Lemma 3. Let π be a policy for which π(τ) is uniform. Then f π has lowest worst-case regret. How can we acquire a policy with a uniform trajectory distribution? Repeating the steps above, we learn a collection of skills using a trajectory-level mutual information objective: Using the same reasoning as Section 3.3, the optimal policy for this objective has a uniform distribution over trajectories that, conditioned on a particular latent z, deterministically produces a single trajectory in a deterministic CMP. Analogous to Section 3.3, we define a distribution over reward functions as r z (τ) log p(τ | z). At optimality, each z corresponds to exactly one trajectory τ z, so the reward function r z (τ) simply indicates whether τ is equal to τ z. Since the distribution over trajectories p(τ | z)p(z)dz is uniform at the optimum, the distribution of reward functions r z corresponds to a uniform distribution over trajectories. Thus, meta-learning on the rewards from trajectory-level mutual information in the minimax-optimal meta-learner. Now that we have derived the optimal meta-learner for trajectory-matching tasks, observe that trajectory-matching is a super-set of the problem of optimizing any possible Markovian reward function at test-time. For a given initial state distribution, each reward function is optimized by a particular trajectory. However, trajectories produced by a non-Markovian policy (i.e., a policy with memory) are not necessarily the unique optimum for any Markovian reward function. Let R τ denote the set of trajectory-level reward functions, and R s,a denote the set of all state-action level reward functions. Bounding the worst-case regret on R τ minimizes an upper bound on the worst-case regret on R s,a: This inequality holds for all policies π, including the policy that maximizes the LHS. While we aim to maximize the RHS, we only know how to maximize the LHS, which gives us a lower bound on the RHS. This inequality holds for all policies π, so it also holds for the policy that maximizes the LHS. In general, this bound is loose because the set of all Markovian reward functions is smaller than the set of all trajectory-level reward functions (i.e., trajectory-matching tasks). However, this bound becomes tight when considering meta-learning on the set of all possible (non-Markovian) reward functions. In the discussion of meta-learning thus far, we have considered tasks where the reward is provided at the last time step T of each episode. In this particular case, the best that an optimal meta-learner can do is go directly to a goal or execute a particular trajectory at every episode according to the optimal exploration policy: for trajectory matching. All intermediate states in the trajectory are uninformative, thus making instances of meta-learning algorithms which explore via schemes like posterior sampling optimal for this class of problems. In the more general case with arbitrary reward functions, intermediate rewards along a trajectory may be informative, and the optimal exploration strategy may be different from posterior sampling (; b;). Nonetheless, the analysis presented in this section provides us insight into the behavior of optimal meta-learning algorithms and allows us to understand the qualities desirable for unsupervised task proposals. The general proposed scheme for unsupervised meta-learning has a number of benefits over standard universal value function and goal reaching style algorithms: it can be applied to arbitrary reward functions going beyond simple goal reaching and it can acquire a significantly more effective exploration strategy than posterior sampling by using intermediate states in a trajectory for intelligent information gathering. Through our analysis, we introduced the notion of optimal meta-learners and analyze their exploration behavior and regret on a class of goal reaching problems. We showed that on these problems, when the test-time task distribution is unknown, the optimal meta-training task distribution for minimizing worst-case test-time regret is uniform over the space of goals. We also showed that this optimal task distribution can be acquired by a simple mutual information maximization scheme. We subsequently extend the analysis to the more general case of matching arbitrary trajectories, as a proxy for the more general class of arbitrary reward functions. In the following section, we will discuss how we can derive a practical algorithm for unsupervised meta-learning from this analysis. Following the derivation in the previous section, we can instantiate a practical unsupervised meta-RL algorithm by constructing a task proposal mechanism based on a mutual information objective. A variety of different mutual information objectives can be formulated, including mutual information between single states and z, pairs of start and end states and z , and entire trajectories and z . We will use DIAYN and leave a full examination of possible mutual information objectives for future work. Data: M \ R, an MDP without a reward function Result: a learning algorithm f: Extract corresponding task reward functions rz(s) using D φ (z|s) Update f using MAML with reward rz(s) DIAYN optimizes mutual information by training a discriminator network D φ (z|·) that predicts which z was used to generate the states in a given rollout according to a latent-conditioned policy π(a|s, z). Our task proposal distribution is thus defined by r z (s, a) = log(D φ (z|s)). The complete unsupervised meta-learning algorithm follows the recipe in Figure 1: first, we acquire r z (s, a) by running DIAYN, which learns D φ (z|s) and a latent-conditioned policy π(a|s, z) (which is discarded). Then, we use z ∼ p(z) to propose tasks r z (s, a) to a standard meta-RL algorithm. This meta-RL algorithm uses the proposed tasks to learn how to learn, acquiring a fast learn algorithm f which can then learn new tasks quickly. While, in principle, any meta-RL algorithm could be used, the discussion at the end of Section 3.4 suggests that meta-RL methods based on posterior sampling might be suboptimal for some tasks. We use MAML (a) as our meta-learning algorithm. Note that the learning algorithm f returned by MAML is defined simply as running gradient descent using the parameters found by MAML as initialization. While it is not immediately apparent how a parameter initialization can define an entire learning algorithm, insightful prior work provides an in-depth discussion of this property. The complete method is summarized in Algorithm 1. In addition to mutual information maximizing task proposals, we will also consider random task proposals, where we also use a discriminator as the reward, according to r(s, z) = log D φrand (z|s), but where the parameters φ rand are chosen randomly (i.e., a random weight initialization for a neural network). While such random reward functions are not optimal, we find that they can be used to acquire useful task distributions for simple tasks, though they are not as effective as the tasks become more complicated. This empirically reinforces the claim that unsupervised meta-RL does not in fact violate any "no free lunch" principle -even simple task proposals that cause the meta-learner to explore the CMP can already accelerate learning of new tasks. In our experiments, we aim to understand whether unsupervised meta-learning as described in Section 3.1 can provide us with an accelerated RL procedure on new tasks. Whereas standard meta-learning requires a hand-specified task distribution at meta-training time, unsupervised meta-learning learns the task distribution through unsupervised interaction with the environment. A fair baseline that likewise uses requires no reward supervision at training time, and only uses rewards at test time, is learning via RL from scratch without any meta-learning. As an upper bound, we include the unfair comparison to a standard meta-learning approach, where the meta-training distribution is manually designed. This method has access to a hand-specified task distribution that is not available to our method. We evaluate two variants of our approach: (a) task acquisition based on DIAYN followed by meta-learning using 2D navigation Half-Cheetah Ant Figure 3: Unsupervised meta-learning accelerates learning: After unsupervised meta-learning, our approach (UML-DIAYN and UML-RANDOM) quickly learns a new task significantly faster than learning from scratch, especially on complex tasks. Learning the task distribution with DIAYN helps more for complex tasks. Results are averaged across 20 evaluation tasks, and 3 random seeds for testing. UML-DIAYN and random also significantly outperform learning with DIAYN initialization or an initialization with a policy pretrained with VIME. MAML, and (b) task acquisition using a randomly initialized discriminator followed by meta-learning using MAML. Our experiments study three simulated environments of varying difficulty: 2D point navigation, 2D locomotion using the "HalfCheetah," and 3D locomotion using the "Ant," with the latter two environments are modifications of popular RL benchmarks (a). While the 2D navigation environment allows for direct control of position, HalfCheetah and Ant can only control their center of mass via feedback control with high dimensional actions (6D for HalfCheetah, 8D for Ant) and observations (17D for HalfCheetah, 111D for Ant). The evaluation tasks, shown in Figure 6, are similar to prior work (a;): 2D navigation and ant require navigating to goal positions, while the half cheetah must run at different goal velocities. These tasks are not accessible to our algorithm during meta-training. We used the default hyperparameters for MAML across all tasks, varying the meta-batch size according to the number of skills: 50 for pointmass, 20 for cheetah, and 20 ant. We found that the default architecture, a 2 layer MLP with 300 units each and ReLU non-linearities, worked quite well for meta-training. We also used the default hyperparameters for DIAYN to acquire skills. We swept over learning rates for learning from scratch via vanilla policy gradient, and found that using ADAM with adaptive step size is the most stable and quick at learning. The comparison between the two variants of unsupervised meta-learning and learning from scratch is shown in Figure 3. We also add a comparison to VIME , a standard noveltybased exploration method, where we pretrain a policy with the VIME reward and then finetune it on the meta-test tasks. In all cases, the UML-DIAYN variant of unsupervised meta-learning produces an RL procedure that outperforms RL from scratch and VIME-init, suggesting that unsupervised interaction with the environment and meta-learning is effective in producing environment-specific but task-agnostic priors that accelerate learning on new, previously unseen tasks. The comparison with VIME shows that the speed of learning is not just about exploration but is indeed about fast adaptation. In our experiments thus far, UML-DIAYN always performs better than learning from scratch, although the benefit varies across tasks depending on the actual performance of DIAYN. We added additional comparisons to using TRPO as a stronger optimizer from scratch with the same trends being observed. We also compared with a baseline of simply initializing from a DIAYN trained contextual policy, and then finetuning the best skill with the actual task reward. From the comparisons in Fig 3, it is apparent that this works poorly as compared with UMRL variants. Interestingly, in many cases (in Figure 4) the performance of unsupervised meta-learning with DIAYN matches that of the hand-designed task distribution. We see that on the 2D navigation task, while handcrafted meta-learning is able to learn very quickly initially, it performs similarly after 100 steps. For the cheetah environment as well, handcrafted meta-learning is able to learn very quickly to start off, but is quickly matched by unsupervised meta-RL with DIAYN. On the ant task, we see that hand-crafted meta-learning does do better than UML-DIAYN, likely because the task distribution is more challenging, and a better unsupervised task proposal algorithm would improve the performance of a meta-learner. The comparison between the two unsupervised meta-learning variants is also illuminating: while the DIAYN-based variant of our method generally achieves the best performance, even the random discriminator is often able to provide a sufficient diversity of tasks to produce meaningful acceleration over learning from scratch in the case of 2D navigation and ant. This has two interesting implications. First, it suggests that unsupervised meta-learning is an effective tool for learning an environment prior. Although the performance of unsupervised meta-learning can be improved with better coverage using DIAYN (as seen in Figure 3), even the random discriminator version provides competitive advantages over learning from scratch. Second, the comparison provides a clue for identifying the source of the structure learned through unsupervised meta-learning: though the particular task distribution has an effect on performance, simply interacting with the environment (without structured objectives, using a random discriminator) already allows meta-RL to learn effective adaptation strategies in a given environment. That is, the performance cannot be explained only by the unsupervised procedure (DIAYN) capturing the right task distribution. We also provide an analysis of the task distributions acquired by the DIAYN procedure in Appendix B.1. We presented an unsupervised approach to meta-RL, where meta-learning is used to acquire an efficient RL procedure without requiring hand-specified task distributions for meta-training. This approach accelerates RL without relying on the manual supervision required for conventional metalearning algorithms. We provide a theoretical derivation that argues that task proposals based on mutual information maximization can provide for a minimum worst-case regret meta-learner, under certain assumptions. We then instantiate an approximation to the theoretically-motivated method by building on recently developed unsupervised task proposal and meta-learning algorithms. Our experiments indicate that unsupervised meta-RL can accelerate learning on a range of tasks, outperforming learning from scratch and often matching the performance of meta-learning from hand-specified task distributions. As our work is the first foray into unsupervised meta-RL, our approach opens a number of questions about unsupervised meta-learning algorithms. One limitation of our analysis is that it only considers deterministic dynamics, and only considers task distributions where posterior sampling is optimal. Extending our analysis to stochastic dynamics and more realistic task distributions may allow unsupervised meta-RL to acquire learning algorithms that can explore and adapt more intelligently, and more effectively solve real-world tasks. Lemma 1 Let π be a policy for which ρ T π (s) is uniform. Then π has lowest worst-case regret. Proof of Lemma 1. To begin, we note that all goal distributions p(s g) have equal regret for policies where ρ T π (s) = 1/|S| is uniform: Now, consider a policy π for which ρ T π (s) is not uniform. For simplicity, we will assume that the argmin is unique, though the proof holds for non-unique argmins as well. The worst-case goal distribution will choose the state s − where that the policy is least likely to visit: Thus, the worst-case regret for policy π is strictly greater than the regret for a uniform π: Thus, a policy π for which ρ T π is non-uniform cannot be minimax, so the optimal policy has a uniform marginal ρ T π. Lemma 2: Mutual information I(s T ; z) is maximized by a task distribution p(s g) which is uniform over goal states. Proof of Lemma 2. We define a latent variable model, where we sample a latent variable z from a uniform prior p(z) and sample goals from a conditional distribution p(s T | z). To begin, note that the mutual information can be written as a difference of entropies: The conditional entropy H p [s T | z] attains the smallest possible value (zero) when each latent variable z corresponds to exactly one final state, s z. In contrast, the marginal entropy H p [s T] attains the largest possible value (log |S|) when the marginal distribution p(s T) = p(s T | z)p(z)dz is uniform. Thus, a task uniform distribution p(s g) maximizes I(s T ; z). Note that for any non-uniform task distribution q(s T), we have is zero, no distribution can achieve a smaller conditional entropy. This, for all non-uniform task distributions q, we have I q (s T ; z) < I p (s T ; z). Thus, the optimal task distribution must be uniform. To understand the method performance more clearly, we also add an ablation study where we compare the meta-test performance of policies at different iterations along meta-training. This shows the effect that additional meta-training has on the fast learning performance for new tasks. This comparison is shown in Figure 5. As can be seen here, at iteration 0 of meta-training the policy is not a very good initialization for learning new tasks. As we move further along the meta-training process, we see that the meta-learned initialization becomes more and more effective at learning new tasks. This shows a clear correlation between additional meta-training and improved meta test-time performance. Ant Half-Cheetah Figure 6: Learned meta-training task distribution and evaluation tasks: We plot the center of mass for various skills discovered by point mass and ant using DIAYN, and a blue histogram of goal velocities for cheetah. Evaluation tasks, which are not provided to the algorithm during meta-training, are plotted as red'x' for ant and pointmass, and as a green histogram for cheetah. While the meta-training distribution is broad, it does not fully cover the evaluation tasks. Nonetheless, meta-learning on this learned task distribution enables efficient learning on a test task distribution. We can analyze the tasks discovered through unsupervised exploration and compare them to tasks we evaluate on at meta-test time. Figure 6 illustrates these distributions using scatter plots for 2D navigation and the Ant, and a histogram for the HalfCheetah. Note that we visualize dimensions of the state that are relevant for the evaluation tasks -positions and velocities -but these dimensions are not specified in any way during unsupervised task acquisition, which operates on the entire state space. Although the tasks proposed via unsupervised exploration provide fairly broad coverage, they are clearly quite distinct from the meta-test tasks, suggesting the approach can tolerate considerable distributional shift. Qualitatively, many of the tasks proposed via unsupervised exploration such as jumping and falling that are not relevant for the evaluation tasks. Our choice of the evaluation tasks was largely based on prior work, and therefore not tailored to this exploration procedure. The for unsupervised meta-RL therefore suggest quite strongly that unsupervised task acquisition can provide an effective meta-training set, at least for MAML, even when evaluating on tasks that do not closely match the discovered task distribution. For all our experiments, we used DIAYN to acquire the task proposals using 20 skills for half-cheetah and for ant and 50 skills for the 2D navigation. We ran the domains using the standard DIAYN hyperparameters described in https://github.com/ben-eysenbach/sac to acquire task proposals. These proposals were then fed into the MAML algorithm https://github.com/ cbfinn/maml_rl, with inner learning rate 0.1, meta learning rate 0.01, inner batch size 40, outer batch size 20, path length 100, using 2 layer networks with 300 units each with ReLu nonlinearities. The test time learning is done with the same parameters for the UMRL variants, and done using REINFORCE with the Adam optimizer for the comparison with learning from scratch.

represent the short summary for finding the corresponding scientific article (long)
The provided query could be "We propose a new auto-encoder incorporated with multiway delay-embedding transform toward interpreting deep image prior." and the positive "Deep image prior (DIP), which utilizes a deep convolutional network (ConvNet) structure itself as an image prior, has attracted huge attentions in computer vision community. It empirically shows the effectiveness of ConvNet structure for various image restoration applications. However, why the DIP works so well is still unknown, and why convolution operation is essential for image reconstruction or enhancement is not very clear. In this study, we tackle these questions. The proposed approach is dividing the convolution into ``delay-embedding'' and ``transformation (\ie encoder-decoder)'', and proposing a simple, but essential, image/tensor modeling method which is closely related to dynamical systems and self-similarity. The proposed method named as manifold modeling in embedded space (MMES) is implemented by using a novel denoising-auto-encoder in combination with multi-way delay-embedding transform. In spite of its simplicity, the image/tensor completion and super-resolution of MMES are quite similar even competitive to DIP in our extensive experiments, and these would help us for reinterpreting/characterizing the DIP from a perspective of ``low-dimensional patch-manifold prior''. The most important piece of information for image/tensor restoration would be the "prior" which usually converts the optimization problems from ill-posed to well-posed, and/or gives some robustness for specific noises and outliers. Many priors were studied in computer science problems such as low-rank representation (; ; ;), smoothness (; ;), sparseness , non-negativity , statistical independence , and so on. Particularly in today's computer vision problems, total variation (TV) , low-rank representation (; ; ;), and non-local similarity priors are often used for image modeling. These priors can be obtained by analyzing basic properties of natural images, and categorized as "unsupervised image modeling". By contrast, the deep image prior (DIP) has been come from a part of "supervised" or "data-driven" image modeling framework (i.e., deep learning) although the DIP itself is one of the state-of-the-art unsupervised image restoration methods. The method of DIP can be simply explained to only optimize an untrained (i.e., randomly initialized) fully convolutional generator network (ConvNet) for minimizing squares loss between its generated image and an observed image (e.g., noisy image), and stop the optimization before the overfitting. explained the reason why a high-capacity ConvNet can be used as a prior by the following statement: Network resists "bad" solutions and descends much more quickly towards naturally-looking images, and its phenomenon of "impedance of ConvNet" was confirmed by toy experiments. However, most researchers could not be fully convinced from only above explanation because it is just a part of whole. One of the essential questions is why is it ConvNet? or in more practical perspective, to explain what is "priors in DIP" with simple and clear words (like smoothness, sparseness, low-rank etc) is very important. In this study, we tackle the question why ConvNet is essential as an image prior, and try to translate the "deep image prior" with words. For this purpose, we divide the convolution operation into "embedding" and "transformation" (see Fig. 9 in Appendix). Here, the "embedding" stands for delay/shift-embedding (i.e., Hankelization) which is a copy/duplication operation of image-patches by sliding window of patch size (τ, τ). The embedding/Hankelization is a preprocessing to capture the delay/shift-invariant feature (e.g., non-local similarity) of signals/images. This "transformation" is basically linear transformation in a simple convolution operation, and it also indicates some nonlinear transformation from the ConvNet perspective. To simplify the complicated "encoder-decoder" structure of ConvNet used in DIP, we consider the following network structure: Embedding H (linear), encoding φ r (non-linear), decoding ψ r (non-linear), and backward embedding H † (linear) (see Fig. 1). Note that its encoder-decoder part (φ r, ψ r) is just a simple multi-layer perceptron along the filter domain (i.e., manifold learning), and it is sandwitched between forward and backward embedding (H, H †). Hence, the proposed network can be characterized by Manifold Modeling in Embedded Space (MMES). The proposed MMES is designed as simple as possible while keeping a essential ConvNet structure. Some parameters τ and r in MMES are corresponded with a kernel size and a filter size in ConvNet. When we set the horizontal dimension of hidden tensor L with r, each τ 2 -dimensional fiber in H, which is a vectorization of each (τ, τ)-patch of an input image, is encoded into r-dimensional space. Note that the volume of hidden tensor L looks to be larger than that of input/output image, but representation ability of L is much lower than input/output image space since the first/last tensor (H,H) must have Hankel structure (i.e., its representation ability is equivalent to image) and the hidden tensor L is reduced to lower dimensions from H. Here, we assume r < τ 2, and its lowdimensionality indicates the existence of similar (τ, τ)-patches (i.e., self-similarity) in the image, and it would provide some "impedance" which passes self-similar patches and resist/ignore others. Each fiber of Hidden tensor L represents a coordinate on the patch-manifold of image. It should be noted that the MMES network is a special case of deep neural networks. In fact, the proposed MMES can be considered as a new kind of auto-encoder (AE) in which convolution operations have been replaced by Hankelization in pre-processing and post-processing. Compared with ConvNet, the forward and backward embedding operations can be implemented by convolution and transposed convolution with one-hot-filters (see Fig. 12 in Appendix for details). Note that the encoder-decoder part can be implemented by multiple convolution layers with kernel size and non-linear activations. In our model, we do not use convolution explicitly but just do linear transform and non-linear activation for "filter-domain" (i.e., horizontal axis of tensors in Fig. 1). The contributions in this study can be summarized as follow: A new and simple approach of image/tensor modeling is proposed which translates the ConvNet, effectiveness of the proposed method and similarity to the DIP are demonstrated in experiments, and most importantly, there is a prospect for interpreting/characterizing the DIP as "low-dimensional patch-manifold prior". Note that the idea of low-dimensional patch manifold itself has been proposed by and. Peyre had firstly formulated the patch manifold model of natural images and solve it by dictionary learning and manifold pursuit. Osher et al. formulated the regularization function to minimize dimension of patch manifold, and solved Laplace-Beltrami equation by point integral method. In comparison with these studies, we decrease the dimension of patch-manifold by utilizing AE shown in Fig. 1. A related technique, low-rank tensor modeling in embedded space, has been studied recently by. However, the modeling approaches here are different: multi-linear vs nonlinear manifold. Thus, our study would be interpreted as manifold version of in a perspective of tensor completion methods. Note that applied their model for only tensor completion task. By contrast, we investigate here tensor completion, super-resolution, and deconvolution tasks. Another related work is devoted to group sparse representation (GSR) (a). The GSR is roughly characterized as a combination of similar patch-grouping and sparse modeling which is similar to the combination of embedding and manifold-modeling. However, the computational cost of similar patch-grouping is obviously higher than embedding, and this task is naturally included in manifold learning. The main difference between above studies and our is the motivation: Essential and simple image modeling which can translate the ConvNet/DIP. The proposed MMES has many connections with ConvNet/DIP such as embedding, non-linear mapping, and the training with noise. From a perspective of DIP, there are several related works. First, the deep geometric prior utilises a good properties of a multi-layer perceptron for shape reconstruction problem which efficiently learn a smooth function from 2D space to 3D space. It helps us to understand DIP from a perspective of manifold learning. For example, it can be used for gray scale image reconstruction if an image is regarded as point could in 3D space (i, j, X ij). However, this may not provide the good image reconstruction like DIP, because it just smoothly interpolates a point cloud by surface like a Volonoi interpolation. Especially it can not provide a property of self-similarity in natural image. Second, deep decoder reconstructs natural images from noises by nonconvolutional networks which consists of linear channel/color transform, ReLU, channel/color normalization, and upsampling layers. In contrast that DIP uses over-parameterized network, deep decoder uses under-parameterized network and shows its ability of image reconstruction. Although deep decoder is a non-convolutional network, Authors emphasize the closed relationship between convolutional layers in DIP and upsampling layers in deep decoder. In this literature, Authors described "If there is no upsampling layer, then there is no notion of locality in the ant image" in deep decoder. It implies the "locality" is the essence of image model, and the convolution/upsampling layer provides it. Furthermore, the deep decoder has a close relationship with our MMES. Note that the MMES is originally/essentially has only decoder and inverse MDT (see Eq.), and the encoder is just used for satisfying Hankel structure. The decoder and inverse MDT in our MMES are respectively corresponding linear operation and upsampling layer in deep decoder. Moreover, concept of under-parameterization is also similar to our MMES. From this, we can say the essence of image model is the "locality", and its locality can be provided by "convolution", "upsampling", or "delay-embedding". This is why the image restoration from single image with deep convolutional networks has highly attentions which are called by zero-shot learning, internal learning, or self-supervised learning (; ; ; ; ; ;). Recently, two generative models: SinGAN and InGAN learned from only a single image, have been proposed. Key concept of both papers is to impose the constraint for local patches of image to be natural. From a perspective of the constraint for local patches of image, our MMES has closed relationship with these works. However, we explicitly impose a low-dimensional manifold constraint for local patches rather than adversarial training with patch discriminators. Here, on the contrary to Section 1, we start to explain the proposed method from the concept of MMES, and we systematically derive the MMES structure from it. Conceptually, the proposed tensor reconstruction method can be formulated by minimize where Y ∈ R J1×J2×···×J N is an observed corrupted tensor, X ∈ R I1×I2×···×I N is an estimated tensor, F: R I1×I2×···×I N → R J1×J2×···×J N is a linear operator which represents the observation system, H: R I1×I2×···×I N → R D×T is padding and Hankelization operator with sliding window of size (τ 1, τ 2, ..., τ N), and we impose each column of matrix H can be sampled from an r-dimensional manifold M r in D-dimensional Euclid space (see Appendix B for details). We have r ≤ D. For simplicity, we putted D:= n τ n and T:= n (I n +τ n −1). For tensor completion task, F:= P Ω is a projection operator onto support set Ω so that the missing elements are set to be zero. For superresolution task, F is a down-sampling operator of images/tensors. For deconvolution task, F is a convolution operator with some blur kernels. Fig. 2 shows the concept of proposed manifold modeling in case of image inpainting (i.e., N = 2). We minimize the distance between observation Y and reconstruction X with its support Ω, and all patches in X should be included in some restricted manifold M r. In other words, X is represented by the patch-manifold, and the property of the patch-manifold can be image priors. For example, low dimensionality of patch-manifold restricts the non-local similarity of images/tensors, and it would be related with "impedance" in DIP. We model X indirectly by designing the properties of patch-manifold M r. We consider an AE to define the r-dimensional manifold M r in (n τ n)-dimensional Euclidean space as follows: where. Note that, in general, the use of AE models is a widely accepted approach for manifold learning . The properties of the manifold M r are determined by the properties of φ r and ψ r. By employing multi-layer perceptrons (neural networks) for φ r and ψ r, encoder-decoder may provide a smooth manifold. In this section, we combine the conceptual formulation and the AE guided manifold constraint to derive a equivalent and more practical optimization problem. First, we redefine a tensor X as an output of generator: Algorithm 1 Optimization algorithm for tensor reconstruction input: where l t ∈ R r, and H † is a pseudo inverse of H. At this moment, X is a function of {l t} T t=1, however Hankel structure of matrix H can not be always guaranteed under the unconstrained condition of l t. For guaranteeing the Hankel structure of matrix H, we further transform it as follow: where we put A r: R D×T → R D×T as an operator which auto-encodes each column of a input matrix with (ψ r,φ r), and [g 1, g 2, ..., g T] as a matrix, which has Hankel structure and is transformed by Hankelization of some input tensor Z ∈ R I1×I2×···×I N. Note that Z is the most compact representation for Hankel matrix [g 1, g 2, ..., g T]. Eq. describes the MMES network shown in Fig. 1: H,φ r,ψ r and H † are respectively corresponding to forward embedding, encoding, decoding, and backward embedding, where encoder and decoder can be defined e.g. by multi-layer perceptrons (i.e., repetition of linear transformation and non-linear activation). F, where A r is an AE which defines the manifold M r. In this study, the AE/manifold is learned from an observed tensor Y itself, thus the optimization problem is finally formulated as where we refer respectively the first and second terms by a reconstruction loss and an auto-encoding loss, and λ > 0 is a trade-off parameter for balancing both losses. Optimization problem consists of two terms: a reconstruction loss, and an auto-encoding loss. Hyperparameter λ is set to balance both losses. Basically, λ should be large because auto-encoding loss should be zero. However, very large λ prohibits minimizing the reconstruction loss, and may lead to local optima. Therefore, we adjust gradually the value of λ in the optimization process. Algorithm 1 shows an optimization algorithm for tensor reconstruction and/or enhancement. For AE learning, we employs a strategy of denoising-auto-encoder (see Appendix in detail). Adaptation of λ is just an example, and it can be modified appropriately with data. Here, the trade-off parameter λ is adjusted for keeping L rec > L AE, but for no large gap between both losses. By exploiting the convolutional structure of H and H † (see Appendix B.1), the calculation flow of L rec and L AE can be easily implemented by using neural network libraries such as TensorFlow. We employed Adam optimizer for updating (Z, A r). Here, we show the selective experimental to demonstrate the close similarity and some slight differences between DIP and MMES. First, toy examples with a time-series signal and a gray-scale image were recovered by the proposed method to show its basic behaviors. Thereafter, we show the main by comparison with DIP and other selective methods on color-image inpainting, superresolution, and deconvolution tasks. Optional of optimization behavior, hyper-parameter sensitivity, and volumetric/3D image completion are shown in Appendix. In this section, we apply the proposed method into a toy example of signal recovery. Fig. 3 shows a of this experiment. A one-dimensional time-series signal is generated from Lorentz system, and corrupted by additive Gaussian noise, random missing, and three block occlusions. The corrupted signal was recovered by the subspace modeling , and the proposed manifold modeling in embedded space. Window size of delay-embedding was τ = 64, the lowest dimension of auto-encoder was r = 3, and additive noise standard deviation was set to σ = 0.05. Manifold modeling catched the structure of Lorentz attractor much better than subspace modeling. Similar patches are located near each other, and the smooth change of patterns can be observed. It implies the relationship between non-local similarity based methods (; ; ; a), and the manifold modeling (i.e., DAE) plays a key role of "patch-grouping" in the proposed method. The difference from the non-local similarity based approach is that the manifold modeling is "global" rather than "non-local" which finds similar patches of the target patch from its neighborhood area. In this section, we compare performance of the proposed method with several selected unsupervised image inpainting methods: low-rank tensor completion (HaLRTC) , parallel lowrank matrix factorization (TMac) , tubal nuclear norm regularization (tSVD) (b), Tucker decomposition with rank increment (Tucker inc.) , lowrank and total-variation (LRTV) regularization (;, smooth PARAFAC tensor completion (SPC) , GSR (a), multi-way delay embedding based Tucker modeling (MDT-Tucker) , and DIP . Implementation and detailed hyper-parameter settings are explained in Appendix. Basically, we carefully tuned the hyper-parameters for all methods to perform the best scores of peak-signal-tonoise ratio (PSNR) and structural similarity (SSIM). Fig. 5(a) shows the eight test images and averages of PSNR and SSIM for various missing ratio {50%, 70%, 90%, 95%, 99%} and for selective competitive methods. The proposed method is quite competitive with DIP. Fig. 6 shows the illustration of . The 99% of randomly selected voxels are removed from 3D-tensors, and the tensors were recovered by various methods. Basically low-rank priors (HaLRTC, TMac, tSVD, Tucker) could not recover such highly incomplete image. In piecewise smoothness prior (LRTV), over-smoothed images were reconstructed since the essential image properties could not be captured. There was a somewhat jump from them by SPC (i.e., smooth prior of basis functions in low-rank tensor decomposition). MDT-Tucker further improves it by exploiting the shift-invariant multi-linear basis. GSR nicely recovered the global pattern of images but details were insufficient. Finally, the reconstructed images by DIP and MMES recovered both global and local patterns of images. In this section, we compare the proposed method with selected unsupervised image super-resolution methods: Bicubic interpolation, GSR (a), ZSSR and DIP . Implementation and detailed hyper-parameter settings are explained in Appendix. Basically, we carefully tuned the hyper-parameters for all methods to perform the best scores of PSNR and SSIM. Fig. 5(b) shows values of PSNR and SSIM of the computer simulation . We used three color images, and six color images. Super resolution methods scaling up them from four or eight times down-scaled images of them with Lanczos2 kernels. According to this quantitative evaluation, bicubic interpolation was clearly worse than others. ZSSR worked well for up-scaling from, however the performances were substantially decreased for upscaling from. Basically, GSR, DIP, and MMES were very competitive. In detail, DIP was slightly better than GSR, and the proposed MMES was slightly better than DIP. More detailed PSNR/SSIM values are given by Table 3 in Appendix. Fig. 7 shows selected high resolution images reconstructed by four super-resolution methods. In general, bicubic method reconstructed blurred images and these were visually worse than others. GSR had smooth outlines in all images, but these were slightly blurred. ZSSR was weak for very low-resolution images. DIP reconstructed visually sharp images but these images had jagged artifacts along the diagonal lines. The proposed MMES reconstructed sharp and smooth outlines. In this section, we compare the proposed method with DIP for image deconvolution/deblurring task. Three color images are prepared and blurred by using three different Gaussian filters. For DIP we choose the best early stopping timing from {1000, 2000, ..., 10000} iterations. For MMES, we employed the fixed AE structure as [32τ 2, r, 32τ 2], and parameters as τ = 4, r = 16, and σ = 0.01 for all nine cases. Fig. 8 shows the reconstructed deblurring images by DIP and MMES. Tab. 1 shows the PSNR and SSIM values of these . We can see that the similarity of the methods qualitatively and quantitatively. It is well known that there is no mathematical definition of interpretability in machine learning and there is no one unique definition of interpretation. We understand the interpretability as a degree to which a human can consistently predict the model's or performance. The higher the interpretability of a deep learning model, the easier it is for someone to comprehend why certain performance or predictions or expected output can be achieved. We think that a model is better interpretable than another model if its performance or behaviors are easier for a human to comprehend than performance of the other models. The manifold learning and associated auto-encoder (AE) can be viewed as the generalized non-linear version of principal component analysis (PCA). In fact, manifold learning solves the key problem of dimensionality reduction very efficiently. In other words, manifold learning (modeling) is an approach to non-linear dimensionality reduction. Manifold modeling for this task are based on the idea that the dimensionality of many data sets is only artificially high. Although the patches of images (data points) consist of hundreds/thousands pixels, they may be represented as a function of only a few or quite limited number underlying parameters. That is, the patches are actually samples from a low-dimensional manifold that is embedded in a high-dimensional space. Manifold learning algorithms attempt to uncover these parameters in order to find a low dimensional representation of the images. In our MMES approach to solve the problem we applied original embedding via multi-way delay embedding transform (MDT or Hankelization). Our algorithm is based on the optimization of cost function and it works towards extracting the low-dimensional manifold that is used to describe the high-dimensional data. The manifold is described mathematically by Eq. and cost function is formulated by Eq.. As mentioned at introduction, reported an important phenomenon of noise impedance of ConvNet structures. Here, we provide a prospect for explaining the noise impedance in DIP through the MMES. Let us consider the sparse-land model, i.e. noise-free images are distributed along low-dimensional manifolds in the high-dimensional Euclidean space and images perturbed by noises thicken the manifolds (make the dimension of the manifolds higher). Under this model, the distribution of images can be assumed to be higher along the low-dimensional noise-free image manifolds. When we assume that the image patches are sampled from low-dimensional manifold like sparse-land model, it is difficult to put noisy patches on the low-dimensional manifold. Let us consider to fit the network for noisy images. In such case the fastest way for decreasing squared error (loss function) is to learn "similar patches" which often appear in a large set of image-patches. Note that finding similar image-patches for denoising is well-known problem solved, e.g., by BM3D algorithm, which find similar image patches by template matching. In contrast, our auto-encoder automatically maps similar-patches into close points on the low-dimensional manifold. When similar-patches have some noise, the low-dimensional representation tries to keep the common components of similar patches, while reducing the noise components. This has been proved by so that a (denoising) auto-encoder maps input image patches toward higher density portions in the image space. In other words, a (denoising) auto-encoder has kind of a force to reconstruct the low-dimensional patch manifold, and this is our rough explanation of noise impedance phenomenon. Although the proposed MMES and DIP are not completely equivalent, we see many analogies and similarities and we believe that our MMES model and associated learning algorithm give some new insight for DIP. A beautiful manifold representation of complicated signals in embedded space has been originally discovered in a study of dynamical system analysis (i.e., chaos analysis) for time-series signals . After this, many signal processing and computer vision applications have been studied but most methods have considered only linear approximation because of the difficulty of non-linear modeling (; ; ; ;). However nowadays, the study of non-linear/manifold modeling has been well progressed with deep learning, and it was successfully applied in this study. Interestingly, we could apply this non-linear system analysis not only for time-series signals but also natural color images and tensors (this is an extension from delay-embedding to multi-way shiftembedding). The best of our knowledge, this is the first study to apply Hankelization with AE into general tensor data reconstruction. MMES is a novel and simple image reconstruction model based on the low-dimensional patchmanifold prior which has many connections to ConvNet. We believe it helps us to understand how work ConvNet/DIP through MMES, and support to use DIP for various applications like tensor/image reconstruction or enhancement (; ; ;). Finally, we established bridges between quite different research areas such as the dynamical system analysis, the deep learning, and the tensor modeling. The proposed method is just a prototype and can be further improved by incorporating other methods such as regularizations, multi-scale extensions, and adversarial training. We can see the anti-diagonal elements of above matrix are equivalent. Such matrix is called as "Hankel matrix". For a two-dimensional array we consider unfold of it and inverse folding by unfold, and The point here is that we scan matrix elements column-wise manner. Hankelization of this twodimensional array (matrix) with τ = is given by scanning a matrix with local-window column-wise manner, and unfold and stack each local patch left-to-right. Thus, it is given as We can see that it is not a Hankel matrix. However, it is a "block Hankel matrix" in perspective of block matrix, a matrix that its elements are also matrices. We can see the block matrix itself is a Hankel matrix and all elements are Hankel matrices, too. Thus, Hankel matrix is a special case of block Hankel matrix in case of that all elements are scalar. In this paper, we say simply "Hankel structure" for block Hankel structure. Figure 9 shows an illustrative explanation of valid convolution which is decomposed into delayembedding/Hankelization and linear transformation. 1D valid convolution of f with kernel h = [h 1, h 2, h 3] can be provided by matrix-vector product of the Hankel matrix and h. In similar way, 2D valid convolution can be provided by matrix-vector product of the block Hankel matrix and unfolded kernel. Multiway-delay embedding transform (MDT) is a multi-way generalization of Hankelization proposed by. In , MDT is defined by using the multi-linear tensor product with multiple duplication matrices and tensor reshaping. Basically, we use the same operation, but a padding operation is added. Thus, the multiway-delay embedding used in this study is defined by where D×T is an unfolding operator which outputs a matrix from an input N -th order tensor, and S n ∈ R is a duplication matrix. Fig. 10 shows the duplication matrix with τ. For example, our Hankelization with reflection padding of f = [f 1, f 2, ..., f 7] with τ = 3 is given by Fig. 11 shows an example of our multiway-delay embedding in case of second order tensors. The overlapped patch grid is constructed by multi-linear tensor product with S n. Finally, all patches are splitted, lined up, and vectorized. The Moore-Penrose pseudo inverse of H is given by where, and trim τ = pad † τ is a trimming operator for removing (τ n −1) elements at start and end of each mode. Note that H † •H is an identity map, but H • H † is not, that is kind of a projection. Delay embedding and its pseudo inverse can be implemented by using convolution with all onehot-tensor windows of size (τ 1, τ 2, ..., τ N). The one-hot-tensor windows can be given by folding a D-dimensional identity matrix I D ∈ R D×D into I D ∈ R τ1×···×τ N ×D. Fig. 12 shows a calculation flow of multi-way delay embedding using convolution in a case of N = 2. Multi-linear tensor product is replaced with convolution with one-hot-tensor windows. Pseudo inverse of the convolution with padding is given by its adjoint operation, which is called as the "transposed convolution" in some neural network library, with trimming and simple scaling with Figure 10: Duplication matrix. In case that we have I columns, it consists of (I − τ + 1) identity matrices of size (τ, τ). In this section, we discuss how to design the neural network architecture of auto-encoder for restricting the manifold M r. The simplest way is controlling the value of r, and it directly restricts the dimensionality of latent space. There are many other possibilities: Tikhonov regularization , drop-out , denoising auto-encoder , variational auto-encoder (Diederik P), adversarial auto-encoder , alpha-GAN (, and so on. All methods have some perspective and promise, however the cost is not low. In this study, we select an attractive and fundamental one: "denoising auto-encoder"(DAE) . The DAE is attractive because it has a strong relationship with Tikhonov regularization , and decreases the entropy of data . Furthermore, learning with noise is also employed in the deep image prior. Finally, we designed an auto-encoder with controlling the dimension r and the standard deviation σ of additive zero-mean Gaussian noise. In case of multi-channel or color image recovery case, we use a special setting of generator network because spacial pattern of individual channels are similar and the patch-manifold can be shared. Fig. 14 shows an illustration of the auto-encoder shared version of MMES in a case of color image recovery. In this case, we put three channels of input and each channel input is embedded, independently. Then, three block Hankel matrices are concatenated, and auto-encoded simultaneously. Inverted three images are stacked as a color-image (third-order tensor), and finally color-transformed. The last color-transform can be implemented by convolution layer with kernel size, and it is also optimized as parameters. It should be noted that the input three channels are not necessary to correspond to RGB, but it would be optimized as some compact color-representation. Here, we explain detailed experimental settings in Section 4.2. In this section, we compared performance of the proposed method with several selected unsupervised image inpainting methods: low-rank tensor completion (HaLRTC) , parallel low-rank matrix factorization (TMac) , tubal nuclear norm regularization (tSVD) (b), Tucker decomposition with rank increment (Tucker inc.) , low-rank and total-variation (LRTV) regularization 2 (;, smooth PARAFAC tensor completion (SPC) 3 , GSR 4 (a), multi-way . For this experiments, hyper-parameters of all methods were tuned manually to perform the best peaksignal-to-noise ratio (PSNR) and for structural similarity (SSIM), although it would not be perfect. For DIP, we did not try the all network structures with various kernel sizes, filter sizes, and depth. We just employed "default architecture", which the details are available in supplemental material 7 of , and employed the best at the appropriate intermediate iterations in optimizations based on the value of PSNR. For the proposed MMES method, we adaptively selected the patch-size τ, and dimension r. Table 2 shows parameter settings of τ = [τ, τ] and r for MMES. Noise level of denoising auto-encoder was set as σ = 0.05 for all images. For auto-encoder, same architecture shown in Fig. 13 was employed. Initial learning rate of Adam optimizer was 0.01 and we decayed the learning rate with 0.98 every 100 iterations. The optimization was stopped after 20,000 iterations for each image. Here, we explain detailed experimental settings in Section 4.3. In this section, we compare performance of the proposed method with several selected unsupervised image super-resolution methods: bicubic interpolation, GSR 8 (a), ZSSR 9 and DIP . In this experiments, DIP was conducted with the best number of iterations from {1000, 2000, 3000, ..., 9000}. For four times (x4) up-scaling in MMES, we set τ = 6, r = 32, and σ = 0.1. For eight times (x8) up-scaling in MMES, we set τ = 6, r = 16, and σ = 0.1. For all images in MMES, the architecture of auto-encoder consists of three hidden layers with sizes of [8τ 2, r, 8τ 2]. We assumed the same Lanczos2 kernel for down-sampling system for all super-resolution methods. Tab. 3 shows values of PSNR and SSIM of the . We used three color images, and six color images. Super resolution methods scaling up them from four or eight times down-scaled images of them. According to this quantitative evaluation, bicubic interpolation was clearly worse than others. ZSSR was good for color images, however the performance were substantially decreased for color image. Basically, GSR, DIP, and MMES were very competitive. In detail, DIP was slightly better than GSR, and the proposed MMES was slightly better than DIP. For this experiment, we recovered 50% missing gray-scale image of'Lena'. We stopped the optimization algorithm after 20,000 iterations. Learning rate was set as 0.01, and we decayed the learning rate with 0.98 every 100 iterations. λ was adapted by Algorithm 1 every 10 iterations. Fig. 15 shows optimization behaviors of reconstructed image, reconstruction loss L rec, auto-encoding loss L DAE, and trade-off coefficient λ. By using trade-off adjustment, the reconstruction loss and the auto-encoding loss were intersected around 1,500 iterations, and both losses were jointly decreased after the intersection point. We evaluate the sensitivity of MMES with three hyper-parameters: r, σ, and τ. First, we fixed the patch-size as, and dimension r and noise standard deviation σ were varied. the reconstruction of a 99% missing image of'Lena' by the proposed method with different settings of (r, σ). The proposed method with very low dimension (r = 1) provided blurred , and the proposed method with very high dimension (r = 64) provided which have many peaks. Furthermore, some appropriate noise level (σ = 0.05) provides sharp and clean . For reference, Fig. 16 shows the difference of DIP optimized with and without noise. From both , the effects of learning with noise can be confirmed. Next, we fixed the noise level as σ = 0.05, and the patch-size were varied with some values of r. Fig. 18 shows the with various patch-size settings for recovering a 99% missing image. The patch sizes τ of or were appropriate for this case. Patch size is very important because it depends on the variety of patch patterns. If patch size is too large, then patch variations might expand and the structure of patch-manifold is complicated. By contrast, if patch size is too small, then the information obtained from the embedded matrix H is limited and the reconstruction becomes difficult in highly missing cases. The same problem might be occurred in all patch-based image reconstruction methods (; ; ; a). However, good patch sizes would be different for different images and types/levels of corruption, and the estimation of good patch size is an open problem. Multi-scale approach may reduce a part of this issue but the patch-size is still fixed or tuned as a hyper-parameter. In this section, we show the of MR-image/3D-tensor completion problem. The size of MR image is. We randomly remove 50%, 70%, and 90% voxels of the original MR-image and recover the missing MR-images by the proposed method and DIP. For DIP, we implemented the 3D version of default architecture in TensorFlow, but the number of filters of shallow layers were slightly reduced because of the GPU memory constraint. For the proposed method, 3D patch-size was set as τ =, the lowest dimension was r = 6, and noise level was σ = 0.05. Same architecture shown in Fig. 13 was employed. Fig. 19 shows reconstruction behavior of PSNR with final value of PSNR/SSIM in this experiment. From the values of PSNR and SSIM, the proposed MMES outperformed DIP in low-rate missing cases, and it is quite competitive in highly missing cases. The some degradation of DIP might be occurred by the insufficiency of filter sizes since much more filter sizes would be required for 3D ConvNet than 2D ConvNet. Moreover, computational times required for our MMES were significantly shorter than that of DIP in this tensor completion problem., r=4, r=8, r=16, r=32, r=48"

Meta-learning on self-proposed task distributions to speed up reinforcement learning without human specified task distributions

Represent the short summary for finding the corresponding scientific article (long)

DCEM learns latent domains for optimization problems and helps bridge the gap between model-based and model-free RL --- we create a differentiable controller and fine-tune parts of it with PPO