Question: Represent the article for finding a single-sentence TLDR

Question: We propose a fully-convolutional conditional generative model, the latent transformation neural network (LTNN), capable of view synthesis using a light-weight neural network suited for real-time applications. In contrast to existing conditional generative models which incorporate conditioning information via concatenation, we introduce a dedicated network component, the conditional transformation unit (CTU), designed to learn the latent space transformations corresponding to specified target views. In addition, a consistency loss term is defined to guide the network toward learning the desired latent space mappings, a task-divided decoder is constructed to refine the quality of generated views, and an adaptive discriminator is introduced to improve the adversarial training process. The generality of the proposed methodology is demonstrated on a collection of three diverse tasks: multi-view reconstruction on real hand depth images, view synthesis of real and synthetic faces, and the rotation of rigid objects. The proposed model is shown to exceed state-of-the-art in each category while simultaneously achieving a reduction in the computational demand required for inference by 30% on average. Generative models have been shown to provide effective frameworks for representing complex, structured datasets and generating realistic samples from underlying data distributions BID8. This concept has also been extended to form conditional models capable of sampling from conditional distributions in order to allow certain properties of the generated data to be controlled or selected BID20. These generative models are designed to sample from broad classes of the data distribution, however, and are not suitable for inference tasks which require identity preservation of the input data. Models have also been proposed which incorporate encoding components to overcome this by learning to map input data to an associated latent space representation within a generative framework BID18. The ing inference models allow for the defining structure/features of inputs to be preserved while specified target properties are adjusted through conditioning BID34. Conventional conditional models have largely relied on rather simple methods, such as concatenation, for implementing this conditioning process; however, BID21 have shown that utilizing the conditioning information in a less trivial, more methodical manner has the potential to significantly improve the performance of conditional generative models. In this work, we provide a general framework for effectively performing inference with conditional generative models by strategically controlling the interaction between conditioning information and latent representations within a generative inference model. In this framework, a conditional transformation unit (CTU), Φ, is introduced to provide a means for navigating the underlying manifold structure of the latent space. The CTU is realized in the form of a collection of convolutional layers which are designed to approximate the latent space operators defined by mapping encoded inputs to the encoded representations of specified targets (see FIG7). This is enforced by introducing a consistency loss term to guide the CTU mappings during training. In addition, a conditional discriminator unit (CDU), Ψ, also realized as a collection of convolutional layers, is included in the network's discriminator. This CDU is designed to improve the network's ability to identify and eliminate transformation specific artifacts in the network's predictions. The network has also been equipped with RGB balance parameters consisting of three values {θ R, θ G, θ B} designed to give the network the ability to quickly adjust the global color balance of FIG7: The conditional transformation unit Φ constructs a collection of mappings {Φ k} in the latent space which produce high-level attribute changes to the decoded outputs. Conditioning information is used to select the appropriate convolutional weights ω k for the specified transformation; the encoding l x of the original input image x is transformed to l y k = Φ k (l x) = conv(l x, ω k) and provides an approximation to the encoding l y k of the attribute-modified target image y k.the images it produces to better align with that of the true data distribution. In this way, the network is easily able to remove unnatural hues and focus on estimating local pixel values by adjusting the three RGB parameters rather than correcting each pixel individually. In addition, we introduce a novel estimation strategy for efficiently learning shape and color properties simultaneously; a task-divided decoder is designed to produce a coarse pixel-value map along with a refinement map in order to split the network's overall task into distinct, dedicated network components. 1. We introduce the conditional transformation unit, with a family of modular filter weights, to learn high-level mappings within a low-dimensional latent space. In addition, we present a consistency loss term which is used to guide the transformations learned during training.2. We propose a novel framework for color inference which separates the generative process into three distinct network components dedicated to learning i) coarse pixel value estimates, ii) pixel refinement scaling factors, and iii) the global RGB color balance of the dataset.3. We introduce the conditional discriminator unit designed to improve adversarial training by identifying and eliminating transformation-specific artifacts present in generated images. Each contribution proposed above has been shown to provide a significant improvement to the network's overall performance through a series of ablation studies. The ing latent transformation neural network (LTNN) is placed through a series of comparative studies on a diverse range of experiments where it is seen to outperform existing state-of-the-art models for (i) simultaneous multi-view reconstruction of real hand depth images in real-time, (ii) view synthesis and attribute modification of real and synthetic faces, and (iii) the synthesis of rotated views of rigid objects. Moreover, the CTU conditioning framework allows for additional conditioning information, or target views, to be added to the training procedure ad infinitum without any increase to the network's inference speed. BID4 has proposed a supervised, conditional generative model trained to generate images of chairs, tables, and cars with specified attributes which are controlled by transformation and view parameters passed to the network. The range of objects which can be synthesized using the framework is strictly limited to the pre-defined models used for training; the network can generate different views of these models, but cannot generalize to unseen objects to perform inference tasks. Conditional generative models have been widely used for geometric prediction BID23 BID32. These models are reliant on additional data, such as depth information or mesh models, to perform their target tasks, however, and cannot be trained using images alone. Other works have introduced a clamping strategy to enforce a specific organizational structure in the latent space BID27 BID17; these networks require extremely detailed labels for supervision, such as the graphics code parameters used to create each example, and are therefore very difficult to implement for more general tasks (e.g. training with real images). have proposed the appearance flow network (AFN) designed specifically for the prediction of rotated viewpoints of objects from images. This framework also relies on geometric concepts unique to rotation and is not generalizable to other inference tasks. The conditional variational autoencoder (CVAE) incorporates conditioning information into the standard variational autoencoder (VAE) framework BID16 and is capable of synthesizing specified attribute changes in an identity preserving manner BID29 BID34. CVAE-GAN BID0 further adds adversarial training to the CVAE framework in order to improve the quality of generated predictions. have introduced the conditional adversarial autoencoder (CAAE) designed to model age progression/regression in human faces. This is achieved by concatenating conditioning information (i.e. age) with the input's latent representation before proceeding to the decoding process. The framework also includes an adaptive discriminator with conditional information passed using a resize/concatenate procedure. BID13 have proposed Pix2Pix as a general-purpose image-to-image translation network capable of synthesizing views from a single image. The IterGAN model introduced by BID6 is also designed to synthesize novel views from a single image, with a specific emphasis on the synthesis of rotated views of objects in small, iterative steps. To the best of our knowledge, all existing conditional generative models designed for inference use fixed hidden layers and concatenate conditioning information directly with latent representations; in contrast to these existing methods, the proposed model incorporates conditioning information by defining dedicated, transformation-specific convolutional layers at the latent level. This conditioning framework allows the network to synthesize multiple transformed views from a single input, while retaining a fully-convolutional structure which avoids the dense connections used in existing inference-based conditional models. Most significantly, the proposed LTNN framework is shown to outperform state-of-the-art models in a diverse range of view synthesis tasks, while requiring substantially less FLOPs for inference than other conditional generative models (see Tables 1 & 2). In this section, we introduce the methods used to define the proposed LTNN model. We first give a brief overview of the LTNN network structure. We then detail how conditional transformation unit mappings are defined and trained to operate on the latent space, followed by a description of the conditional discriminator unit implementation and the network loss function used to guide the training process. Lastly, we describe the task-division framework used for the decoding process. The basic workflow of the proposed model is as follows:1. Encode the input image x to a latent representation l x = Encode(x).2. Use conditioning information k to select conditional, convolutional filter weights ω k.3. Map the latent representation l x to l y k = Φ k (l x) = conv(l x, ω k), an approximation of the encoded latent representation l y k of the specified target image y k.4. Decode l y k to obtain a coarse pixel value map and a refinement map.5. Scale the channels of the pixel value map by the RGB balance parameters and take the Hadamard product with the refinement map to obtain the final prediction y k.6. Pass real images y k as well as generated images y k to the discriminator, and use the conditioning information to select the discriminator's conditional filter weights ω k.7. Compute loss and update weights using ADAM optimization and backpropagation. A detailed overview of the proposed network structure is provided in Section A.1 of the appendix. Provide: Labeled dataset x, {y k} k∈T with target transformations indexed by a fixed set T, encoder weights θ E, decoder weights θ D, RGB balance parameters {θ R, θ G, θ B}, conditional transformation unit weights {ω k} k∈T, discriminator D with standard weights θ D and conditionally selected weights {ω k} k∈T, and loss function hyperparameters γ, ρ, λ, κ corresponding to the smoothness, reconstruction, adversarial, and consistency loss terms, respectively. The specific loss function components are defined in detail in Equations 1 -5 in Section 3.2.1: procedure TRAIN 2:x, {y k} k∈T = get train batch # Sample input and targets from training set 3: DISPLAYFORM0 for k in T do 5: DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 # Assemble final network prediction for target 9: 10:# Update encoder, decoder, RGB, and CTU weights 11: DISPLAYFORM4 # Update discriminator and CDU weights 19: DISPLAYFORM5 Generative models have frequently been designed to explicitly disentangle the latent space in order to enable high-level attribute modification through linear, latent space interpolation. This linear latent structure is imposed by design decisions, however, and may not be the most natural way for a network to internalize features of the data distribution. Several approaches have been proposed which include nonlinear layers for processing conditioning information at the latent space level. In these conventional conditional generative frameworks, conditioning information is introduced by combining features extracted from the input with features extracted from the conditioning information (often using dense connection layers); these features are typically combined using standard vector concatenation, although some have opted to use channel concatenation (; BID0 . Six of these conventional conditional network designs are illustrated in FIG0 along with the proposed LTNN network design for incorporating conditioning information. Rather than directly concatenating conditioning information, we propose using a conditional transformation unit (CTU), consisting of a collection of distinct convolutional mappings in the network's latent space; conditioning information is then used to select which collection of weights, i.e. which CTU mapping, should be used in the convolutional layer to perform a specified transformation. For view point estimation, there is an independent CTU per viewpoint. Each CTU mapping maintains its own collection of convolutional filter weights and uses Swish activations BID26. The filter weights and Swish parameters of each CTU mapping are selectively updated by controlling the gradient flow based on the conditioning information provided. The CTU mappings are trained to transform the encoded, latent space representation of the network's input in a manner which produces high-level view or attribute changes upon decoding. This is accomplished by introducing a consistency term into the loss function which is minimized precisely when the CTU mappings behave as depicted in FIG7. In this way, different angles of view, light directions, and deformations, for example, can be synthesized from a single input image. The discriminator used in the adversarial training process is also passed conditioning information which specifies the transformation which the model has attempted to make. The conditional discriminator unit (CDU), consisting of convolutional layers with modular weights similar to the CTU, is trained to specifically identify unrealistic artifacts which are being produced by the corresponding conditional transformation unit mappings. For view point estimation, there is an independent CDU per viewpoint. The incorporation of this context-aware discriminator structure has significantly boosted the performance of the network (see Table 4 in the appendix).The proposed model uses the adversarial loss as the primary loss component. The discriminator, D, is trained using the adversarial loss term L D adv defined below in Equation 1. Additional loss terms corresponding to structural reconstruction, smoothness BID14, and a notion of consistency, are also used for training the encoder/decoder: DISPLAYFORM0 DISPLAYFORM1 where y k is the modified target image corresponding to an input x, ω k are the weights of the CDU mapping corresponding to the k th transformation, Φ k is the CTU mapping for the k th transformation, y k = Decode Φ k Encode[x] is the network prediction, and τ i,j is the two-dimensional, discrete shift operator. The final loss function for the encoder and decoder components is given by: DISPLAYFORM2 with hyperparameters typically selected so that λ, ρ γ, κ. The consistency loss is designed to guide the CTU mappings toward approximations of the latent space mappings which connect the latent representations of input images and target images as depicted in FIG7. In particular, the consistency term enforces the condition that the transformed encoding, DISPLAYFORM3, approximates the encoding of the k th target image, l y k = Encode[y k], during the training process. The decoding process has been divided into three tasks: estimating the refinement map, pixel-values, and RGB color balance of the dataset. We have found this decoupled framework for estimation helps the network converge to better minima to produce sharp, realistic outputs. The decoding process begins with a series of convolutional layers followed by bilinear interpolation to upsample the low resolution latent information. The last component of the decoder's upsampling process consists of two distinct transpose convolutional layers used for task separation; one layer is allocated for predicting the refinement map while the other is trained to predict pixel-values. The refinement map layer incorporates a sigmoidal activation function which outputs scaling factors intended to refine the coarse pixel value estimations. RGB balance parameters, consisting of three trainable variables, are used as weights for balancing the color channels of the pixel value map. The Hadamard product of the refinement map and the RGB-rescaled value map serves as the network's final output: DISPLAYFORM0 In this way, the network has the capacity to mask values which lie outside of the target object (i.e. by setting refinement map values to zero) which allows the value map to focus on the object itself during the training process. Experimental show that the refinement maps learn to produce masks which closely resemble the target objects' shapes and have sharp drop-offs along the boundaries. BID33. The input depth-map hand pose image is shown to the far left, followed by the network predictions for 9 synthesized view points. The views synthesized using LTNN are seen to be sharper and also yield higher accuracy for pose estimation (see FIG4). To show the generality of our method, we have conducted a series of diverse experiments: (i) hand pose estimation using a synthetic training set and real NYU hand depth image data BID33 for testing, (ii) synthesis of rotated views of rigid objects using the real ALOI dataset BID7 and synthetic 3D chair dataset BID1, (iii) synthesis of rotated views using a real face dataset BID5, and (iv) the modification of a diverse range of attributes on a synthetic face dataset . For each experiment, we have trained the models using 80% of the datasets. Since ground truth target depth images were not available for the real hand dataset, an indirect metric has been used to quantitatively evaluate the model as described in Section 4.1. Ground truth data was available for all other experiments, and models were evaluated directly using the L 1 mean pixel-wise error and the structural similarity index measure (SSIM) BID23 BID19 (the masked pixel-wise error L M 1 BID6) was used in place of the L 1 error for the ALOI experiment). More details regarding the precise training configurations and the creation of the synthetic datasets can be found in the appendix. To evaluate the proposed framework with existing works, two comparison groups have been formed: conditional inference models (CVAE-GAN, CVAE, and CAAE) with comparable encoder/decoder structures for comparison on experiments with non-rigid objects, and view synthesis models (MV3D BID32, IterGAN, Pix2Pix, AFN , and TVSN BID23 ) for comparison on experiments with rigid objects. Additional experiments have been performed to compare the proposed CTU conditioning method with other conventional concatenation methods (see FIG0 ; are shown in FIG3 . Qualitative and comparisons for each experiment are provided in the appendix. Hand pose experiment: Since ground truth predictions for the real NYU hand dataset were not available, the LTNN model has been trained using a synthetic dataset generated using 3D mesh hand models. The NYU dataset does, however, provide ground truth coordinates for the input hand pose; using this we were able to indirectly evaluate the performance of the model by assessing the accuracy of a hand pose estimation method using the network's multi-view predictions as input. More specifically, the LTNN model was trained to generate 9 different views which were then fed into the pose estimation network from BID3 (also trained using the synthetic dataset).A comparison of the quantitative hand pose estimation is provided in FIG3 where the proposed LTNN framework is seen to provide a substantial improvement over existing methods; qualitative are also available in FIG1. With regard to real-time applications, the proposed model runs at 114 fps without batching and at 1975 fps when applied to a mini-batch of size 128 (using a single TITAN Xp GPU and an Intel i7-6850K CPU). The stereo face database BID5, consisting of images of 100 individuals from 10 different viewpoints, was used for experiments with real faces; these faces were segmented using the method of BID22 and then cropped and centered to form the final dataset. The LTNN model was trained to synthesize images of input faces corresponding to three consecutive horizontal rotations. As shown in FIG4, our method significantly outperforms the CVAE-GAN, CAAE, and IterGAN models in both the L 1 and SSIM metrics; qualitative are also available in FIG5 and Section A.6 of the appendix. Real object experiment: The ALOI dataset BID7, consisting of images of 1000 real objects viewed from 72 rotated angles (covering one full 360• rotation), has been used for experiments on real objects. As shown in Table 1 and in Figure 8, our method outperforms other state-of-the-art methods with respect to the L 1 metric and achieves comparable SSIM metric scores. Of note is the fact that the LTNN framework is capable of effectively performing the specified rigid-object transformations using only a single image as input, whereas most state-of-the-art view synthesis methods require additional information which is not practical to obtain for real datasets. For example, MV3D requires depth information and TVSN requires 3D models to render visibility maps for training which is not available in the ALOI dataset. We have tested our model's ability to perform 360• view estimation on the chairs and compared the with the other state-of-the-art methods. The proposed model outperforms existing models specifically designed for the task of multi-view prediction and require the least FLOPs for inference compared with all other methods (see Table 2). Table 2: Results for 3D chair 360• view synthesis. The proposed method uses significantly less parameters during inference, requires the least FLOPs, and yields the fastest inference times. FLOP calculations correspond to inference for a single image with resolution 256×256×3. To evaluate the proposed framework's performance on a more diverse range of attribute modification tasks, a synthetic face dataset and five conditional generative models with comparable encoder/decoder structures to the LTNN model have been selected for comparison. These models have been trained to synthesize discrete changes in elevation, azimuth, light direction, and age from a single grayscale image; are shown in Table 3 and ablation are available in Table 4. Near continuous attribute modification is also possible within the proposed framework, and distinct CTU mappings can be composed with one another to synthesize multiple modifications simultaneously; more details and related figures are provided in sections A.7.4 and A.7.5 of the appendix. Table 3: Results for simultaneous colorization and attribute modification on synthetic face dataset. DISPLAYFORM0 In this work, we have introduced an effective, general framework for incorporating conditioning information into inference-based generative models. We have proposed a modular approach to incorporating conditioning information using CTUs and a consistency loss term, defined an efficient task-divided decoder setup for deconstructing the data generation process into managable subtasks, and shown that a context-aware discriminator can be used to improve the performance of the adversarial training process. The performance of this framework has been assessed on a diverse range of tasks and shown to outperform state-of-the-art methods. At the bottle-neck between the encoder and decoder, a conditional transformation unit (CTU) is applied to map the 2×2 latent features directly to the transformed 2×2 latent features on the right. This CTU is implemented as a convolutional layer with filter weights selected based on the conditioning information provided to the network. The noise vector z ∈ R 4 from normal distribution N is concatenated to the transformed 2×2 features and passed to the decoder for the face attributes task only. The 32×32 features near the end of the decoder component are processed by two independent convolution transpose layers: one corresponding to the value estimation map and the other corresponding to the refinement map. The channels of the value estimation map are rescaled by the RGB balance parameters, and the Hadamard product is taken with the refinement map to produce the final network output. For the ALOI data experiment, we have followed the encoder and decoder structure, and for the stereo face dataset BID5 experiment, we have added an additional Block v1 layer in the encoder and decoder to utilize the full 128×128×3 resolution images. The encoder incorporates two main block layers, as defined in Figure A. 2, which are designed to provide efficient feature extraction; these blocks follow a similar design to that proposed by, but include dense connections between blocks, as introduced by BID10. We normalize the output of each network layer using the batch normalization method as described in BID12. For the decoder, we have opted for a minimalist design, inspired by the work of BID24. Standard convolutional layers with 3 × 3 filters and same padding are used through the penultimate decoding layer, and transpose convolutional layers with 5 × 5 filters and same padding are used to produce the value-estimation and refinement maps. All parameters have been initialized using the variance scaling initialization method described in BID9.Our method has been implemented and developed using the TensorFlow framework. The models have been trained using stochastic gradient descent (SGD) and the ADAM optimizer BID15 with initial parameters: learning rate = 0.005, β 1 = 0.9, and β 2 = 0.999 (as defined in the TensorFlow API r1.6 documentation for tf.train.AdamOptimizer)., along with loss function hyperparameters: λ = 0.8, ρ = 0.2, γ = 0.0002, and κ = 0.00005 (as introduced in FORMULA8). The discriminator is updated once every two encoder/decoder updates, and one-sided label smoothing BID28 has been used to improve stability of the discriminator training procedure. All datasets have also been normalized to the interval for training. Once the total number of output channels, N out, is specified, the remaining N out − N in output channels are allocated to the non-identity filters (where N in denotes the number of input channels). For the Block v1 layer at the start of the proposed LTNN model, for example, the input is a single grayscale image with N in = 1 channel and the specified number of output channels is N out = 32. One of the 32 channels is accounted for by the identity component, and the remaining 31 channels are the three non-identity filters. When the remaining channel count is not divisible by 3 we allocate the remainder of the output channels to the single 3 × 3 convolutional layer. Swish activation functions are used for each filter, however the filters with multiple convolutional layers (i.e. the right two filters in the Block v1 diagram) do not use activation functions for the intermediate 3 × 3 convolutional layers (i.e. those after the 1 × 1 layers and before the final 3 × 3 layers). A.2.1 DATASET A kinematic hand model with 33 degrees of freedom has been used to generate 200,000 distinct hand poses with nine depth images from different viewpoints for each pose. We sampled hand pose uniformly from each of the 18 joint angle parameters, covering a full range of hand articulations. The nine viewpoints are centered around a designated input view and correspond to 30• changes in the spherical coordinates of the viewer (i.e. 30• up, 30• right, 30• up and 30• right, etc.). Testing was performed on the MSRA and NYU BID33 hand datasets. We follow the training procedure proposed by BID3 for the hand pose estimation network; after training, the LTNN multi-view predictions are fed into the network and the accuracy of the predicted angles is used to assess how well these network predictions approximate the unknown, true views. As seen in Figure A.3, the optimal are obtained when all 9 synthesized views points are fed into the pose estimation network., we have used images of resolution 256 × 256 × 3 from the ALOI database for training and testing. While the LTNN model is capable of making simultaneous predictions of multiple viewpoints, as illustrated in the hand and chair experiments, the Pix2Pix BID13 and IterGAN BID6 networks are designed to produce a single synthesized view. To make fair comparisons between these existing networks and the proposed LTNN model, each model has been trained only to produce a single 30• rotated view of the ALOI objects. In particular, only two CTU mappings were trained: one corresponding to the identity, and one corresponding to the rotated view. Figure A.8: Experiment on unseen objects from the ALOI real object dataset. First row of is ground truth, second row is ours, third row is ours without task-division, fourth row is IterGAN, and bottom row is Pix2Pix. As shown from the figure, our methods are sharper and realistic than other methods in the majority of generated views. Chairs from the ShapeNet BID2 ) 3D model repository have been rotated horizontally 20• 17 times and vertically 10 • 3 times; 6742 chairs have been selected following the data creation methodology from BID23. A.6 REAL FACE EXPERIMENT A.6.1 DATASET The original dataset has and 100 identification and 5 different views from 2 distinct cameras, which 1000 face images in total. Since the dataset is not huge, we would like to reduce noise and perform face segmentation with BID22 and manually filtered badly segmented faces from the with the segmentation method and create 300 × 300 × 3 face images. For training, we resize the original images into 128 × 128 × 3 resolution. Each face has been rendered at four distinct age ranges, and four different lighting directions have been used. The orientation of faces is allowed to vary in elevation from −20• to 29• by increments of 7• and in azimuth from 10 • to 150• by increments of 20•. To demonstrate the model's colorization capabilities, the input images have been converted to gray-scale using the luminosity method. Table 4: Ablation/comparison using identical encoder, decoder, and training procedure. DISPLAYFORM0 Task-division: An overview of the task-division decoding procedure applied to the synthetic face dataset is provided in Figure A.13. As noted in Section 3.3, the refinement maps tend to learn to produce masks which closely resemble the target objects' shapes and have sharp drop-offs along the objects' boundaries. In addition to masking extraneous pixels, these refinement maps have been shown to apply local color balancing by, for example, filtering out the green and blue channels near lips when applied to human faces (i.e. the refinement maps for the green and blue channels show darker regions near the lips, thus allowing for the red channel to be expressed more distinctly).The use of a task-divided decoder can also be seen to remove artifacts in the generated images in Figure A. 13; e.g. removal of the blurred eyebrow (light), elimination of excess hair near the side of ear (azimuth), and reduction of the reddish vertical stripe on the forehead (age). As noted in Section 4.3, near-continuous attribute modification can be performed by piecewise-linear interpolation in the latent space. For example, we can train 9 CTU mappings {Φ k} 8 k=0 corresponding to discrete, incremental 7• changes in elevation {θ k}. In this setting, the network predictions for an elevation change of θ 0 = 0• and θ 1 = 7• are given by Decode[Φ 0 (l x)] and Decode[Φ 1 (l x)], respectively (where l x denotes the encoding of the input image). To predict an elevation change of 3.5•, we can perform linear interpolation in the latent space between the representations Φ 0 (l x) and Φ 1 (l x); that is, we may take our network prediction for the intermediate change of 3.5• to be: y = Decode[l y] where l y = 0.5 · Φ 0 (l x) + 0.5 · Φ 1 (l x)Likewise, to approximate a change of 10.5• in elevation we may take Decode[l y], where l y = 0.5 · Φ 1 (l x) + 0.5 · Φ 2 (l x), as the network prediction. More generally, we can interpolate between the latent CTU map representations to predict a change θ via: DISPLAYFORM0 with k ∈ {0, . . ., 7} and λ ∈ chosen so that θ = λ · θ k + (1 − λ) · θ k+1. Accordingly, the proposed framework naturally allows for continuous attribute changes to be approximated by using this piecewise-linear latent space interpolation procedure. Figure A.19: Near continuous attribute modification is attainable using piecewise-linear interpolation in the latent space. Provided a gray-scale image (corresponding to the faces on the far left), modified images corresponding to changes in light direction (first), age (second), azimuth (third), and elevation (fourth) are produced with 17 degrees of variation. These attribute modified images have been produced using 9 CTU mappings, corresponding to varying degrees of modification, and linearly interpolating between the discrete transformation encodings in the latent space. Additional qualitative for near-continuous attribute modification can be found in Section A.7.6.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce an effective, general framework for incorporating conditioning information into inference-based generative models.

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.


Question: Represent the article for finding a single-sentence TLDR

Question: We describe three approaches to enabling an extremely computationally limited embedded scheduler to consider a small number of alternative activities based on resource availability. We consider the case where the scheduler is so computationally limited that it cannot backtrack search. The first two approaches precompile resource checks (called guards) that only enable selection of a preferred alternative activity if sufficient resources are estimated to be available to schedule the remaining activities. The final approach mimics backtracking by invoking the scheduler multiple times with the alternative activities. We present an evaluation of these techniques on mission scenarios (called sol types) from NASA's next planetary rover where these techniques are being evaluated for inclusion in an onboard scheduler. Embedded schedulers must often operate with very limited computational resources. Due to such limitations, it is not always feasible to develop a scheduler with a backtracking search algorithm. This makes it challenging to perform even simple schedule optimization when doing so may use resources needed for yet unscheduled activities. In this paper, we present three algorithms to enable such a scheduler to consider a very limited type of preferred activity while still scheduling all required (hereafter called mandatory) activities. Preferred activities are grouped into switch groups, sets of activities, where each activity in the set is called a switch case, and exactly one of the activities in the set must be scheduled. They differ only by how much time, energy, and data volume they consume and the goal is for the scheduler to schedule the most desirable activity (coincidentally the most resource consuming activity) without sacrificing any other mandatory activity. The target scheduler is a non-backtracking scheduler to be onboard the NASA Mars 2020 planetary rover BID9 that schedules in priority first order and never removes or moves an activity after it is placed during a single run of the scheduler. Because the scheduler does not backtrack, it is challenging to ensure that scheduling a consumptive switch case will not use too many resources Copyright c 2019, California Institute of Technology. Government Sponsorship Acknowledged. and therefore prevent a later (in terms of scheduling order, not necessarily time order) mandatory activity from being scheduled. The onboard scheduler is designed to make the rover more robust to run-time variations by rescheduling multiple times during execution BID4. If an activity ends earlier or later than expected, then rescheduling will allow the scheduler to consider changes in resource consumption and reschedule accordingly. Our algorithms to schedule switch groups must also be robust to varying execution durations and rescheduling. We have developed several approaches to handle scheduling switch groups. The first two, called guards, involve reserving enough sensitive resources (time, energy, data volume) to ensure all later required activities can be scheduled. The third approach emulates backtracking under certain conditions by reinvoking the scheduler multiple times. These three techniques are currently being considered for implementation in the Mars 2020 onboard scheduler. For the scheduling problem we adopt the definitions in (Rabideau and Benowitz 2017). The scheduler is given• a list of activities A 1 p 1, d 1, R 1, e 1, dv 1, Γ 1, T 1, D 1... A n p n, d n, R n, e n, dv n, Γ n, T n, D n• where p i is the scheduling priority of activity A i;• d i is the nominal, or predicted, duration of activity A i;• R i is the set of unit resources R i1... R im that activity A i will use;• e i and dv i are the rates at which the consumable resources energy and data volume respectively are consumed by activity A i;• Γ i1... Γ ir are non-depletable resources used such as sequence engines available or peak power for activity A i;• T i is a set of start time windows [T ij start, T i j pref erred, DISPLAYFORM0 • D i is a set of activity dependency constraints for activity A i where A p → A q means A q must execute successfully before A p starts. The goal of the scheduler is to schedule all mandatory activities and the best switch cases possible while respecting individual and plan-wide constraints. Each activity is assigned a scheduling priority. This priority determines the order in which the activity will be considered for addition to the schedule. The scheduler attempts to schedule the activities in priority order, therefore: higher priority activities can block lower priority activities from being scheduled and higher priority activities are more likely to appear in the schedule. Mandatory Activities are activities, m 1... m j ⊆ A, that must be scheduled. The presumption is that the problem as specified is valid, that is to say that a schedule exists that includes all of the mandatory activities, respects all of the provided constraints, and does not exceed available resources. In addition, activities can be grouped into Switch Groups. The activities within a switch group are called switch cases and vary by how many resources (time, energy, and data volume) they consume. It is mandatory to schedule exactly one switch case and preferable to schedule a more resource intensive one, but not at the expense of another mandatory activity. For example, one of the Mars 2020 instruments takes images to fill mosaics which can vary in size; for instance we might consider 1x4, 2x4, or 4x4 mosaics. Taking larger mosaics might be preferable, but taking a larger mosaic takes more time, takes more energy, and produces more data volume. These alternatives would be modeled by a switch group that might be as follows: DISPLAYFORM1 The desire is for the scheduler to schedule the activity M osaic 4x4 but if it does not fit then try scheduling M osaic 2x4, and eventually try M osaic 1x4 if the other two fail to schedule. It is not worth scheduling a more consumptive switch case if doing so will prevent a future, lower priority mandatory activity from being scheduled due to lack of resources. Because our computationally limited scheduler cannot search or backtrack, it is a challenge to predict if a higher level switch case will be able to fit in the schedule without consuming resources that will cause another lower priority mandatory activity to be forced out of the schedule. Consider the following example in FIG1 where the switch group consists of activities B1, B2, and B3 and DISPLAYFORM2 Each activity in this example also has one start time window from T istart to T i end.B3 is the most resource intensive and has the highest priority so the scheduler will first try scheduling B3. As shown in FIG1, scheduling B3 will prevent the scheduler from placing activity C at a time satisfying its execution constraints. So, B3 should not be scheduled. The question might arise as to why switch groups cannot simply be scheduled last in terms of scheduling order. This is difficult for several reasons: 1) We would like to avoid gaps in the schedule which is most effectively done by scheduling primarily left to right temporally, and 2) if another activity is dependent on an activity in a switch group, then scheduling the switch group last would introduce complications to ensure that the dependencies are satisfied. The remainder of the paper is organized as follows. First, we describe several plan wide energy constraints that must be satisfied. Then, we discuss two guard approaches to schedule preferred activities, which place conditions on the scheduler that restrict the placement of switch cases under certain conditions. We then discuss various versions of an approach which emulates backtracking by reinvoking the scheduler multiple times with the switch cases. We present empirical to evaluate and compare these approaches. There are several energy constraints which must be satisfied throughout scheduling and execution. The scheduling process for each sol, or Mars day, begins with the assumption that the rover is asleep for the entire time spanning the sol. Each time the scheduler places an activity, the rover must be awake so the energy level declines. When the rover is asleep the energy level increases. Two crucial energy values which must be taken into account are the Minimum State of Charge (SOC) and the Minimum Handover State of Charge. The state of charge, or energy value, cannot dip below the Minimum SOC at any point. If scheduling an activity would cause the energy value to dip below the Minimum SOC, then that activity will not be scheduled. In addition, the state of charge cannot be below the Minimum Handover SOC at the Handover Time, in effect when the next schedule starts (e.g., the handover SOC of the previous plan is the expected beginning SOC for the subsequent schedule).In order to preserve battery life, the scheduler must also consider the Maximum State of Charge constraint. Exceeding the Maximum SOC hurts long term battery performance and the rover will perform shunting. To prevent it from exceeding this value, the rover may be kept awake. First we will discuss two guard methods to schedule switch cases, the Fixed Point guard and the Sol Wide guard. Both of these methods attempt to schedule switch cases by reserving enough time and energy to schedule the remaining mandatory activities. For switch groups, this means that resources will be reserved for the least resource consuming activity since it is mandatory to schedule exactly one activity in the switch group. The method through which both of these guard approaches reserve enough time to schedule future mandatory activities is the same. They differ in how they ensure there is enough energy. While the Fixed Point guard reserves enough energy at a single fixed time pointthe time at which the least resource consuming switch case is scheduled to end in the nominal schedule, the Sol Wide guard attempts to reserve sufficient energy by keeping track of the energy balance in the entire plan, or sol. In this discussion, we do not attempt to reserve data volume while computing the guards as it is not expected to be as constraining of a resource as time or energy. We aim to take data volume into account as we continue to do work on this topic. Both the time and energy guards are calculated offline before execution occurs using a nominal schedule. Then, while rescheduling during execution, the constraints given by the guards are applied to ensure that scheduling a higher level switch case will not prevent a future mandatory activity from being scheduled. If activities have ended sufficiently early and freed up resources, then it may be possible to reschedule with a more consumptive switch case. First, we will discuss how the Fixed Point and Sol Wide guards ensure enough time will be reserved to schedule remaining mandatory activities while attempting to schedule a more resource consuming switch case. If a preferred time, T i j pref erred, is specified for an activity, the scheduler will try to place an activity closest to its preferred time while obeying all other constraints. Otherwise, the scheduler will try to place the activity as early as possible. Each switch group in the set of activities used to create a nominal schedule includes only the nominal, or least resource consuming switch case, and all activities take their predicted duration. First, we generate a nominal schedule and find the time at which the nominal switch case is scheduled to complete, as shown in FIG2. We then manipulate the execution time constraints of the more resource intensive switch cases, B2 and B3 in FIG2, so that they are constrained to complete by T N ominal as shown in Equation 2. Thus, a more (time) resource consuming switch case will not use up time from any remaining lower priority mandatory activities. If an activity has more than one start time window, then we only alter the one which contains T N ominal and remove the others. If a prior activity ends earlier than expected during execution and frees up some time, then it may be possible to schedule a more consumptive switch case while obeying the time guard given by the altered execution time constraints. DISPLAYFORM0 Since we found that the above method was quite conservative and heavily constrained the placement of a more resource consuming switch case, we attempted a preferred time method to loosen the time guard. In this approach, we set the preferred time of the nominal switch case to its latest start time before generating the nominal schedule. Then, while the nominal schedule is being generated, the scheduler will try to place the nominal switch case as late as possible since the scheduler will try to place an activity as close to its preferred time as possible. As a , T N ominal will likely be later than what it would be if the preferred time were not set in this way. As per Equation 2, the latest start times, T Bi j end, of the more resource consuming switch cases may be later than what they would be using the previous method where the preferred time was not altered, thus allowing for wider start time windows for higher level switch cases. This method has some risks. If the nominal switch case was placed as late as possible, it could use up time from another mandatory activity with a tight execution window that it would not otherwise have used up if it was placed earlier, as shown in Fixed Point Minimum State of Charge Guard The Fixed Point method attempts to ensure that scheduling a more resource consuming switch case will not cause the energy to violate the Minimum SOC while scheduling any future mandatory activities by reserving sufficient energy at a single, fixed point in time, T N ominal as shown in FIG5. The guard value for the Minimum SOC is the state of charge value at T N ominal while constructing the nominal schedule. When attempting to schedule a more resource intensive switch case, a constraint is placed on the scheduler so that the energy cannot fall below the Minimum SOC guard value at time T N ominal. If an activity ends early (and uses fewer resources than expected) during execution, it may be possible to satisfy this guard while scheduling a more consumptive switch case. Then, while attempting to place a more consumptive switch case, a constraint is placed on the scheduler so that the extra energy required by the switch case does not exceed Energy Leftover from the nominal schedule as in FIG6. For example, if we have a switch group consisting of three activities, B1, B2, and B3 and d B3 > d B2 > d B1 and each switch case consumes e Watts of power, we must ensure that the following inequality holds at the time the scheduler is attempting to schedule a higher level switch case: DISPLAYFORM0 There may be more than one switch group in the schedule. Each time a higher level switch case is scheduled, the Energy Leftover value is decreased by the extra energy required to schedule it. When the scheduler tries to place a switch case in another switch group, it will check against the updated Energy Leftover. Sol Wide Handover State of Charge Guard The Sol Wide handover SOC guard only schedules a more resource consumptive switch case if doing so will not cause the energy to dip below the Handover SOC at handover time. First, we use the nominal schedule to calculate how much energy is needed to schedule remaining mandatory activities. Having a Maximum SOC constraint while calculating this value may produce an inaccurate since any energy that would exceed the Maximum SOC would not be taken into account. So, in order to have an accurate prediction of the energy balance as activities are being scheduled, this value is calculated assuming there is no Maximum SOC constraint. 8. The Maximum SOC constraint is only removed while computing the guard offline to gain a clear understanding of the energy balance but during execution it is enforcedAs shown in FIG8, the energy needed to schedule the remaining mandatory activities is the difference between the energy level just after the nominal switch case has been scheduled, call this E1, and after all activities have been scheduled, call this energy level E2. Energy N eeded = E1 − E2 Then, a constraint is placed on the scheduler so that the energy value after a higher level switch case is scheduled must be at least: DISPLAYFORM1 By placing this energy constraint, we hope to prevent the energy level from falling under the Minimum Handover SOC by the time all activities have been scheduled. Sol Wide Minimum State of Charge Guard While we ensure that the energy will not violate the minimum Handover SOC by keeping track of the energy balance, it is possible that scheduling a longer switch case will cause the energy to fall below the Minimum SOC. To limit the chance of this happening, we run a Monte Carlo of execution offline while computing the sol wide energy guard. We use this Monte Carlo to determine if a mandatory activity was not scheduled due to a longer switch case being scheduled earlier. If this occurs in any of the Monte Carlos of execution, then we increase the guard constraint in Equation 5. We first find the times at which each mandatory activity was scheduled to finish in the nominal schedule. Then, we run a Monte Carlo of execution with the input plan containing the guard and all switch cases. Each Monte Carlo differs in how long each activity takes to execute compared to its original predicted duration in the schedule. If a mandatory activity was not executed in any of the Monte Carlo runs and a more resource consuming switch case was executed before the time at which that mandatory activity was scheduled to complete in the nominal schedule, then we increase the Sol Wide energy guard value in Equation 5 by a fixed amount. We aim to compose a better heuristic to increase the guard value as we continue work on this subject. The Multiple Scheduler Invocation (MSI) approach emulates backtracking by reinvoking the scheduler multiple times with the switch cases. MSI does not require any precomputation offline before execution as with the guards and instead reinvokes the scheduler multiple times during execution. During execution, the scheduler reschedules (e.g., when activities end early) with only the nominal switch case as shown in FIG10 until an MSI trigger is satisfied. At this point, the scheduler is reinvoked multiple times, at most once per switch case in each switch group. In the first MSI invocation, the scheduler attempts to schedule the highest level switch case as shown in FIG10. If the ing schedule does not contain all mandatory activities, then the scheduler will attempt to schedule the next highest level switch case, as in 7c, and so on. If none of the higher level switch cases can be successfully scheduled then the schedule is regenerated with the nominal switch case. If activities have ended early by the time MSI is triggered and ed in more resources than expected, then the goal is for this approach to generate a schedule with a more consumptive switch case if it will fit (assuming nominal activity durations for any activities that have not yet executed).There are multiple factors that must be taken into consideration when implementing MSI:When to Trigger MSI There are two options to trigger the MSI process (first invocation while trying to schedule the switch case):1. Time Offset. Start MSI when the current time during execution is some fixed amount of time, X, from the time at which the nominal switch case is scheduled to start in the current schedule (shown in FIG11).2. Switch Ready. Start MSI when an activity has finished executing and the nominal switch case activity is the next activity scheduled to start (shown in FIG12). If the highest level switch case activity is not able to be scheduled in the first invocation of MSI, then the scheduler must be invoked again. We choose to reschedule as soon as possible after the most recent MSI invocation. This method risks over-consumption of the CPU if the scheduler is invoked too frequently. To handle this, we may need to rely on a process within the scheduler called throttling. Throttling places a constraint which imposes a minimum time delay between invocations, preventing the scheduler from being invoked at too high of a rate. An alternative is to reschedule at an evenly split, fixed cadence to avoid over-consumption of the CPU; we plan to explore this approach in the future. In some situations, the nominal switch case activity in the original plan may become committed before or during the MSI invocations as shown in FIG1. An activity is committed if its scheduled start time is between the start and end of the commit window BID2. A committed activity cannot be rescheduled and is committed to execute. If the nominal switch case remains committed, the scheduler will not be able to elevate to a higher level switch case. There are two ways to handle this situation:1. Commit the activity. Keep the nominal switch case activity committed and do not try to elevate to a higher level switch case.2. Veto the switch case. Veto the nominal switch case so that it is no longer considered in the current schedule. When an activity is vetoed, it is removed from the current schedule and will be considered in a future invocation of the scheduler. Therefore, by vetoing the nominal switch case, (a) B1 is the nominal switch case. Since an activity has not finished executing and B1 is not the next activity, MSI cannot begin yet.(b) Since A finished executing early, and B1 is the next activity, the MSI process can begin. it will no longer be committed and the scheduler will continue the MSI invocations in an effort to elevate the switch case. Handling Rescheduling After MSI Completes but before the Switch Case is Committed After MSI completes, there may be events that warrant rescheduling (e.g., an activity ending early) before the switch case is committed. When the scheduler is reinvoked to account for the event, it must know which level switch case to consider. If we successfully elevated a switch case, we choose to reschedule with that higher level switch case. Since the original schedule generated by MSI with the elevated switch case was in the past and did not undergo changes from this rescheduling, it is possible the schedule will be inconsistent and may lead to complications while scheduling later mandatory activities. An alternative we plan to explore in the future is to disable rescheduling until the switch case is committed. However, this approach would not allow the scheduler to regain time if an activity ended early and caused rescheduling. In order to evaluate the performance of the above methods, we apply them to various sets of inputs comprised of activities with their constraints and compare them against each other. The inputs are derived from sol types. Sol types are currently the best available data on expected Mars 2020 rover operations (Jet Propulsion Laboratory 2017a). In order to construct a schedule and simulate plan execution, we use the Mars 2020 surrogate scheduler -an implementation of the same algorithm as the Mars 2020 onboard scheduler (Rabideau and Benowitz 2017), but intended for a Linux workstation environment. As such, it is expected to produce the same schedules as the operational scheduler but runs much faster in a workstation environment. The surrogate scheduler is expected to assist in validating the flight scheduler implementation and also in ground operations for the mission BID1. Each sol type contains between 20 and 40 activities. Data from the Mars Science Laboratory Mission (Jet Propulsion Laboratory 2017b; BID4 BID5 indicates that activity durations were quite conservative and completed early by around 30%. However, there is a desire by the mission to operate with a less conservative margin to increase productivity. In our model to determine activity execution durations, we choose from a normal distribution where the mean is 90% of the predicted, nominal activity duration. The standard deviation is set so that 10 % of activity execution durations will be greater than the nominal duration. For our analysis, if an activity's execution duration chosen from the distribution is longer than its nominal duration, then the execution duration is set to be the nominal duration to avoid many complications which from activities running long (e.g., an activity may not be scheduled solely because another activity ran late). Detailed discussion of this is the subject of another paper. We do not explicitly change other activity resources such as energy and data volume since they are generally modeled as rates and changing activity durations implicitly changes energy and data volume as well. We create 10 variants derived from each of 8 sol types by adding one switch group to each set of inputs for a total of 80 variants. The switch group contains three switch cases, A nominal, A 2x, and A 4x where DISPLAYFORM0 In order to evaluate the effectiveness of each method, we have developed a scoring method based on how many and what type of activities are able to be scheduled successfully. The score is such that the value of any single mandatory activity being scheduled is much greater than that of any combination of switch cases (at most one activity from each switch group can be scheduled).Each mandatory activity that is successfully scheduled, including whichever switch case activity is scheduled, contributes one point to the mandatory score. A successfully scheduled switch case that is 2 times as long as the original activity contributes 1/2 to the switch group score. A successfully scheduled switch case that is 4 times as long as the original, nominal switch case contributes 1 to the switch group score. If only the nominal switch case is able to be scheduled, it does not contribute to the switch group score at all. There is only one switch group in each variant, so the maximum switch group score for a variant is 1. Since scheduling a mandatory activity is of much higher importance than scheduling any number of higher level switch case, the mandatory activity score is weighted at a much larger value then the switch group score. In the following empirical , we average the mandatory and switch groups scores over 20 Monte Carlo runs of execution for each variant. We compare the different methods to schedule switch cases over varying incoming state of charge values (how much energy exists at the start) and determine which methods in 1) scheduling all mandatory activities and 2) the highest switch group scores. The upper bound for the theoretical maximum switch group score is given by an omniscient scheduler-a scheduler which has prior knowledge of the execution duration for each activity. Thus, this scheduler is aware of the amount of resources that will be available to schedule higher level switch cases given how long activities take to execute compared to their predicted, nominal duration. The input activity durations fed to this omniscient scheduler are the actual execution durations. We run the omniscient scheduler at most once per switch case. First, we try to schedule with only the highest level switch case and if that fails to schedule all mandatory activities, then we try with the next level switch case, and so on. First, we determine which methods are able to successfully schedule all mandatory activities, indicated by the Maximum Mandatory Score in FIG1. Since scheduling a mandatory activity is worth much more than scheduling any number of higher level switch cases, we only compare switch group scores between methods that successfully schedule all mandatory activities. In order to evaluate the ability of each method to schedule all mandatory activities, we also compare against two other methods, one which always elevates to the highest level switch case while the other always elevates to the medium level switch case. We see in FIG1 that always elevating to the highest (3rd) level performs the worst and drops approximately 0.25 mandatory activities per sol, or 1 activity per 4 sols on average while always elevating to the second highest level drops close to 0.07 mandatory activities per sol, or 1 activity per 14 sols on average. For comparison, the study described in BID4 showed that approximately 1 mandatory activity was dropped every 90 sols, indicating that both of these heuristics perform poorly. We found that using preferred time to guard against time FIG1: Switch Group Score vs Incoming SOC for Methods which Schedule all Mandatory Activities caused mandatory activities to drop for both the fixed point and sol wide guard (for the reason described in the Guarding for Time section) while using the original method to guard against time did not. We see in FIG1 that the preferred time method with the fixed point guard drops on average about 0.04 mandatory activities per sol, or 1 activity every 25 sols while the sol wide guard drops on average about 0.1 mandatory activities per sol, or 1 activity every 10 sols. We also see that occasionally fewer mandatory activities are scheduled with a higher incoming SOC. Since using preferred time does not properly ensure that all remaining activities will be able to be scheduled, a higher incoming SOC can allow a higher level switch case to be scheduled, preventing future mandatory activities from being scheduled. The MSI approaches which veto to handle the situation where the nominal switch case becomes committed before or during MSI drop mandatory activities. Whenever an activity is vetoed, there is always the risk that it will not be able to be scheduled in a future invocation, more so if the sol type is very tightly time constrained, which is especially true for one of our sol types. Thus, vetoing the nominal switch case can in dropping the activity, accounting for this method's inability to schedule all mandatory activities. The MSI methods that keep the nominal switch case committed and do not try to elevate to a higher level switch case successfully schedule all mandatory activities, as do the guard methods. We see that the Fixed Point guard, Sol Wide guard, and two of the MSI approaches are able to successfully schedule all mandatory activities. As shown in FIG1, the Sol Wide guard and MSI approach using the options Time Offset and Commit in the highest switch group scores closest to the upper bound for the theoretical maximum. Both MSI approaches have increasing switch group scores with increasing incoming SOC since a higher incoming energy will in more energy to schedule a consumptive switch case during MSI. The less time there is to complete all MSI invocations, the more likely it is for the nominal switch case to become committed. Since we give up trying to elevate switch cases and keep the switch case committed if this occurs, fewer switch cases will be elevated. Because our time offset value, X, in FIG11 is quite large (15 minutes), this situation is more likely to occur using the Switch Ready approach to choose when to start MSI, explaining why using Switch Ready in a lower switch score than Time Offset. The Fixed Point guard in a significantly lower switch case score because it checks against a state of charge constraint at a particular time regardless of what occurs during execution. Even if a switch case is being attempted to be scheduled at a completely different time than T N ominal in FIG2, (e.g., because prior activities ended early), the guard constraint will still be enforced at that particular time. Since we simulate activities ending early, more activities will likely complete by T N ominal, causing the energy level to fall under the Minimum SOC Guard value. Unlike the Fixed Point guard, since the the Sol Wide guard checks if there is sufficient energy to schedule a higher level switch case at the time the scheduler is attempting to schedule it, not at a set time, it is better able to consider resources regained from an activity ending early. We also see that using the Fixed Point guard begins to in a lower switch group score with higher incoming SOC levels after the incoming SOC is 80% of the Maximum SOC. Energy is more likely to reach the Maximum SOC constraint with a higher incoming SOC. The energy gained by an activity taking less time than predicted will not be able to be used if the ing energy level would exceed the Maximum SOC. If this occurs, then since the extra energy cannot be used, the energy level may dip below the guard value in FIG5 at time T N ominal while trying to schedule a higher level switch case even if an activity ended sufficiently early, as shown in FIG1. Just-In-Case Scheduling BID3 ) uses a nominal schedule to determine areas where breaks in the schedule are most likely to occur and produces a branching (tree) schedule to cover execution contingencies. Our approaches all (re) schedule on the fly although the guard methods can be vewied as forcing schedule branches based on time and resource availability. BID8 solve RCPSP (resource-constrained project scheduling problem) where all activities that must be scheduled are not known in advance and the scheduler must decide whether or not to perform certain activities of varying resource consumption. Similarly, our scheduler does not know which of the switch cases to schedule in advance, using runtime resource information to drive (re) scheduling. Integrated planning and scheduling can also be considered scheduling disjuncts (chosen based on prevailing conditions (e.g., BID0)) but these methods typically search whereas we are too computationally limited to search. There are many areas for future work. Currently the time guard heavily limits the placement of activities. As we saw, using preferred time to address this issue ed in dropping mandatory activities. Ideally analysis of start time windows and dependencies could determine where an activity could be placed without blocking other mandatory activities. Additionally, in computing the guard for Minimum SOC using the Sol Wide Guard, instead of increasing the guard value by a predetermined fixed amount which could in over-conservatism, binary search via Monte Carlo analysis could more precisely determine the guard amount. Currently we consider only a single switch group per plan, the Mars 2020 rover mission desires support for multiple switch groups in the input instead. Additional work is needed to extend to multiple switch groups. Further exploration of all of the MSI variants is needed. Study of starting MSI invocations if an activity ends early by at least some amount and the switch case is the next activity is planned. We would like to analyze the effects of evenly spacing the MSI invocations in order to avoid relying on throttling and we would like to try disabling rescheduling after MSI is complete until the switch case has been committed and understand if this in major drawbacks. We have studied the effects of time and energy on switch cases, and we would like to extend these approaches and analysis to data volume. We have presented several algorithms to allow a very computationally limited, non-backtracking scheduler to consider a schedule containing required, or mandatory, activities and sets of activities called switch groups where each activity in such sets differs only by its resource consumption. These algorithms strive to schedule the most preferred, which happens to be the most consumptive, activity possible in the set without dropping any other mandatory activity. First, we discuss two guard methods which use different approaches to reserve enough resources to schedule remaining mandatory activities. We then discuss a third algorithm, MSI, which emulates backtracking by reinvoking the scheduler at most once per level of switch case. We present empirical analysis using input sets of activities derived from data on expected planetary rover operations to show the effects of using each of these methods. These implementations and empirical evaluation are currently being evaluated in the context of the Mars 2020 onboard scheduler.

Represent the TLDR for finding the corresponding scientific article

This paper describes three techniques to allow a non-backtracking,  computationally limited scheduler to consider a small number of alternative activities based on resource availability.

Represent the short summary for finding the corresponding scientific article (long)

Techniques for combining generalized policies with search algorithms to exploit the strengths and overcome the weaknesses of each when solving probabilistic planning problems


Question: Represent the article for finding a single-sentence TLDR

Question: Inspired by the modularity and the life-cycle of biological neurons,we introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on the pruning of neurons of low activity. In this method, an L1 regulator is used to promote the presence of neurons of zero or low activity whose connections to previously active neurons is permanently severed at the end of training. Subsequent tasks are trained using these pruned neurons after reinitialization and cause zero deterioration to the performance of previous tasks. We show empirically that this biologically inspired method leads to state of the art beating or matching current methods of higher computational complexity. Continual learning, the ability of models to learn to solve new tasks beyond what has previously been trained, has garnered much attention from the machine learning community in recent years. The main obstacle for effective continual learning is the problem of catastrophic forgetting: machines trained on new problems forget about the tasks that they were previously trained on. There are multiple approaches to this problem, from employing networks with many submodules to methods which penalize changing the weights of the network that are deemed important for previous tasks. These approaches either require specialized training schemes or still suffer catastrophic forgetting, albeit at a smaller rate. Furthermore, from a biological perspective, the current fixed capacity approaches generally require the computation of a posterior in weight space which is non-local and hence biologically implausible. Motivated by the life-cycle of biological neurons, we introduce a simple continual learning algorithm for fixed capacity networks which can be trained using standard gradient descent methods and suffers zero deterioration on previously learned problems during the training of new tasks. In this method, the only modifications to standard machine learning algorithms are simple and biologically plausible: i.e. a sparsifying L 1 regulator and activation threshold based neural pruning. We demonstrate empirically that these modifications to standard practice lead to state of the art performance on standard catastrophic forgetting benchmarks. Lifelong learning. Prior work addressing catastrophic forgetting generally fall under two categories. In the first category, the model is comprised of many individual modules at each layer and forgetting is prevented either by routing the data through different modules or by successively adding new modules for each new task. This approach often has the advantage of suffering zero forgetting, * Work completed while at New York Unviersity. however, the structure of these networks is specialized. In the case of, the model is not fixed capacity and in the case of training is done using a tournament selection genetic algorithm. In the second category of approaches to lifelong learning the network structure and training scheme are standard, and forgetting is addressed by penalizing changes of weights which are deemed important. These approaches, generally referred to as weight elasticity methods, have the advantage of simpler training schemes but still suffer catastrophic forgetting, albeit at a smaller rate than unconstrained training. Sparsification. While sparsification is a crucial tool that we use, it is not in itself a focus of this work. For accessibility, we use a simple neuron/filter based sparsification scheme which can be thought of as a single iteration variation of. The core idea of our method is to take advantage of the fact that neural networks are vastly over-parametrized. A manifestation of this over-parametrization is through the practice of sparsification, i.e. the compression of neural network with relatively little loss of performance. As an example, it was shown in show that VGG-16 can be compressed by more than 16 times. In this section we first show that given an activation based sparse network, we can leverage the unused capacity of the model to develop a continual learning scheme which suffers no catastrophic forgetting. We then discuss the idea of graceful forgetting to address the tension between sparsification and model performance in the context of lifelong learning. In what follows we will discuss sparsity for fully connected layers by looking at the individual neurons. The same argument goes through identically for individual channels of convolutional layers. Output Figure 1: The partition of an network with neuronal sparsity into active, inactive and interference parts. Let us assume that we have a trained network which is sparse in the sense that only a subset of the neurons of the network are active. Networks with this form of sparsity can be thought of as narrow subnetworks embedded inside the original structure. There are many approaches that aim to train such sparse networks with little loss of performance (e.g. ). We will discuss our sparsification method in detail in §3.2. Fig. 1 shows a cartoon of our approach, where we have a network with activation based neuronal sparsity, where the active and inactive neurons are respectively denoted by blue and grey nodes. Based on the connectivity structure, the weights of the network can also be split into three classes. First, denoted in blue in Fig. 1, we have the active weights W act which connect active nodes to active nodes. Next we have the weights which connect any node to inactive nodes, we call these the free weights W free, denoted in grey. Finally we have the weights which connect the inactive nodes to the active nodes, we call these the interference weights W int, denoted in red dashed lines. A more precise definition of the active and inactive neurons and weights is given in §3.2. The crux of our approach is the simple observation that if all the interference weightsW int are set to zero, the free weightsW free can be changed arbitrarily without causing any change whatsoever to the output of the network. We can therefore utilize these weights to train new tasks without any catastrophic forgetting of the previous tasks. We can further split the free weights into two groups. First, the weights which connect active nodes to inactive nodes. These are the weights that take advantage of previously learned features and are therefore responsible for transfer learning throughout the network. We also have the weights that connect inactive nodes to inactive nodes. These weights can form completely new pathways to the input and train new features. A simple measure of the amount of transfer learning taking place is the number of new active neurons at each layer after the training of subsequent tasks. Given that an efficient sparse training scheme would not need to relearn the features that are already present in the network, the number of new neurons grown at each stage of training is an indicator of the sufficiency of the already learned features for the purposes of the new task. For example, if the features learned at some layer for previous tasks provide sufficient statistics for purposes of a subsequent task, no new neurons need to be trained at this layer during the training of the subsequent task. We will see more of this point in §4. Output architecture. To fully flesh out a continual learning scheme, we need to specify the connectivity structure of the output nodes. There are two intuitive routes that we can take. In order to train a new task, one option is to use a new output layer (i.e. a new head) while saving the previous output layer. This option, demonstrated in Fig. 2 on the left, is known as the multi-head approach and is standard in continual learning. Because each new output layer comes with its own sets of weights which connect to the final hidden layer neurons, this method is not a fully fixed capacity method. Note that in our approach to continual learning, training a multi-head network with a fully depleted core structure, i.e. a network where are no more free neurons left, is equivalent to final layer transfer learning. In scenarios where the output layer of the different tasks are structurally compatible, for example when all tasks are classification on the same number of classes, we can use a single-head approach. Demonstrated in Fig. 2 on the right, in this approach we use the same output layer for all tasks, but for each task, we mask out the neurons of the final hidden layer that were trained on other tasks. In the case of Fig. 2, only green nodes in the final hidden layer are connected to the output for the second task and only blue nodes for the first task. This is equivalent to a dynamic partitioning of the final hidden layer into multiple unequal sized parts, one part for each task. In practice this is done using a multiplicative masking operation with a task dependent mask, denoted in Fig. 2 by dashed lines after the final hidden layer. This structure, being truly fixed, is more restrictive to train than its multi-head counterpart. Because of this, single head continual algorithms were not possible previously, and as far as we are aware, CLNP is the first viable such algorithm. In what follows we will assume that we are using Rectifier Linear Units (ReLU). While we have only tested our methodology with ReLU networks, we expect it to work similarly with other activations. Sparsification. So far in this section we have shown that given a sparse network trained on a number of tasks, we can train the network on new tasks without suffering any catastrophic forgetting. We now discuss the specific scheme that we use to achieve this sparisty, which is similar in spirit to the network trimming approach put forward in Ref.. Our sparsification method is comprised of two parts. First, during the training of each task, we add an L 1 weight regulator to promote sparsity and regulate the magnitude of the weights of the network. This is akin to biological energy requirements for synaptic communication. The coefficient of α of this regulator is a hyperparameter of our approach. We can also gain more control over the amount of sparsity in each layer by choosing a differentα for different layers. The second part of our sparsification scheme is post-training neuron pruning based on the average activity of each neuron. This step is the analogue of long term depression of synaptic connections between neurons without correlated activities. Subsequently at the beginning of training a new task, the connections of these pruned neurons are reinitialized in a manner reminiscent of the lifecycle of biological neurons. Note that most efficient sparsification algorithms include a third part which involves adjusting the surviving weights of the network after pruning. This step is referred to as fine-tuning and is done by retraining the network for a few epochs while only updating the weights which survive sparsification. This causes the model to regain some of its lost performance because of pruning. To achieve a yet higher level of sparsity, one can iterate the pruning and fine-tuning steps multiple times. For simplicity, unless otherwise specified, we only perform one iteration of pruning without the fine tuning step. Partitioning the network. In §3.1, we split the network into active and inactive parts which we define as follows. Given network N, comprised of L layers, we denote the neurons of each layer as N l with l = 1···L. Let us also assume that the network N has been trained on dataset S. In order to find the active and inactive neurons of the network, we compute the average activity over the entire dataset S for each individual neuron. In a network with ReLU activations, we identify the active neurons N. We can therefore view N act l as a compression of the network into a sub-network of smaller width. Based on their connectivity structure, the weights of each layer are again divided into active, free and interference parts, respectively corresponding to the blue, grey and red lines in Fig. 2. Graceful forgetting. While sparsity is crucial in our approach for the training of later tasks, care needs to be taken so as not to overly sparsify and thereby reduce the model's performance. In practice, model sparsity has a similar relationship with generalization as other regularization schemes. As sparsity increases, initially the generalization performance of the model improves. However, as we push our sparsity knobs (i.e. the L 1 regulator α and activity threshold θ) higher, eventually both training and validation accuracy will suffer and the network fails to fit the data properly. This means that in choosing these hyperparameters, we have to make a compromise between model performance and remaining network capacity for future tasks. This brings us to a subject which is often overlooked in lifelong learning literature generally referred to as graceful forgetting. This is the general notion that it would be preferable to sacrifice some accuracy in a controlled manner, if it reduces future catastrophic forgetting of this task and also helps in the training of subsequent tasks. We believe any successful fixed capacity continual learning algorithm needs to implement some form of graceful forgetting scheme. In our approach, graceful forgetting is implemented through the sparsity vs. performance compromise. In other words, after the training of each task, we sparsify the model up to some acceptable level of performance loss (given by a margin parameter m) in a controlled manner. We then move on to subsequent tasks knowing that the model no longer suffers any further deterioration from training future tasks. This has to be contrasted with other weight elasticity approaches which use soft constraints on the weights of the network and cannot guarantee future performance of previously trained tasks. The choice of sparsity hyperparameters is made based on this incarnation of graceful forgetting as follows. We scan over a range of hyperparameters (α, the L 1 weight regulator and ξ, the learning rate) using grid search and note the value of the best validation accuracy across all hyperparameters. We then pick the models which achieve validation accuracy within a margin of m% of this best validation accuracy. The margin parameterm controls how much we are willing to compromise on accuracy to regain capacity and in experiments we take it to be generally in the range of 0.05% to 2% depending on the task. We sparsify the picked models using the highest activation thresholdθ such that the model remains within this margin of the best validation accuracy. We finally pick the hyperparameters which give the highest sparsity among these models. In this way, we efficiently find the optimal hyperparameters α * (m), θ * (m) and ξ * (m) which afford the highest sparsity model with validation accuracy within m% of the highest validation accuracy among all hyperparameters. After pruning away the unused weights and neurons of the model with the hyperparameters chosen as above, we report the test accuracy of the sparsified network. This algorithm for training and hyperparameter grid search does not incur any significant additional computational burden over standard practice. The hyperparameter search is performed in standard fashion, and the additional steps of selecting networks within the acceptable margin, scanning the threshold, and selecting the highest sparsity network only require evaluation and do not include any additional network training. Permuted MNIST. In this experiment, we look at the performance of our approach on ten tasks derived from the MNIST dataset via ten random permutations of the pixels. To compare with previous work, we start with the same structure and hyperparameters as in Ref.: a multi-head MLP architecture with two hidden layers, each with 2000 neurons and ReLU activation and a softmax multi-class crossentropy loss trained with Adam optimizer and batch size 256. In order to make the task more challenging we look at two variations of this structure: For the first variation, we employ only a single-head to demonstrate the viability of our single-head approach. For the second variation we use layers of width 100 instead of 2000. For the first network variation (wide single-head structure), we do a search over the hyperparameters on the first task using a heldout validation set, just as in Ref.. For the remaining tasks, we settle on learning rate of 0.002 and L 1 weight regularization α = 10 −7,10 −5,10 −6 respectively for the first, second and final layers. Finally, when sparsifying after training each task, we allow for graceful forgetting with a small margin ofm = 0.05%. With test error within 0.05% of single task SGD training, CLNP virtually eliminates catastrophic forgetting and achieves an average accuracy of 98.42±0.04 which is just shy of the single task performance of 98.48±0.05 (mean + STD over 5 iterations of the experiment). In the second variation of the network (narrow multi-head structure), perform sparsification with 2 iterations of fine-tuning and specifically choose a graceful forgetting margin (m = 3%) such that the network is saturated (runs out of free neurons) after all 10 tasks have been trained. In this case, our method attains an average of 95.8% over 10 tasks. In both variations of the network, our achieve state of the art performance on networks with comparable size, matching or exceeding prior methods of much higher conceptual and computational complexity (e.g. ). For an exhaustive comparison of these with previous methods see Tab. 2 in. Split CIFAR-10/CIFAR-100. In this experiment, we train an image classifier sequentially, first on CIFAR-10 (task 1) and then on CIFAR-100 split into 10 different tasks, each with 10 classes (tasks 2-11). We employ the same multi-head network used in Ref., and we use two different training schemes comprised of maximum graceful forgetting of m = 1% and m = 2%. The validation accuracy of the 6 tasks after training them sequentially is shown in Fig. 3a. We see that we again achieve state of the art performance. The more ambitious m = 1% scheme (which only allowed for a graceful forgetting of less than 1%) runs out of capacity after the fourth task is trained. We notice that after the model capacity is depleted (tasks 5 and 6 denoted with red dashed lines), the performance of the m = 1% scheme plummets, showing the necessity for unused neurons for the performance of the network. The more moderate forgetting scheme m = 2% (denoted in orange), however, maintains high performance throughout all tasks and does not run out of capacity until final task is trained. We repeated the experiment with a graceful forgetting of m = 4% but this time followed by fine-tuning, i.e. retraining of the remaining weights after pruning. The of this method are given in Fig. 3a in green. We see that here there is virtually no catastrophic forgetting on the first task (the model performs even better after pruning and retraining as has been reported in previous sparsity literature ). The remaining tasks also get a significant boost from this improved sparsification method. This is a simple demonstration of the potential of sparsification based continual learning methods given more advanced sparsification schemes. We also use a wider single-head network for comparison. In Fig. 3b, we can see the number of new channels learned at each layer for each consecutive task. Of note, the first convolutional layer trains new channels only for tasks 1 and 2. The second and third convolutional layers, grow new channels up to task 3 and task 5 respectively. The fourth layer keeps training new channels up to the last task. The fact that the first layer grows no new channels after the second task implies that the features learned during the training of the first two tasks are eemed sufficient for the training of the subsequent tasks. The fact that this sufficiency happens after training more tasks for layers 2 and 3 is a verification of the fact that features learned in lower layers are more general and thus more transferable in comparison with the features of the higher layers which are known to specialize. This observation implies that models which hope to be effective at continual learning need to be wider in the higher layers to accommodate for this lack of transferability of the features at these scales. In this work we have introduced an intuitive lifelong learning method which leverages the over-parametrization of neural networks to train new tasks in the inactive neurons/filters of the network without suffering any catastrophic forgetting in the previously trained tasks. We implemented a controlled way of graceful forgetting by sacrificing some accuracy at the end of the training of each task in order to regain network capacity for training new tasks. We showed empirically that this method leads to which exceed or match the current state-of-the-art while being less computationally intensive. Because of this, we can employ larger models than otherwise possible, given fixed computational resources. Our methodology comes with simple diagnostics based on the number of free neurons left for the training of new tasks. Model capacity usage graphs are informative regarding the transferability and sufficiency of the features of different layers. Using such graphs, we have verified the notion that the features learned in earlier layers are more transferable. We can leverage these diagnostic tools to pinpoint any layers that run out of capacity prematurely, and resolve these bottlenecks in the network by increasing the number of neurons in these layers when moving on to the next task. In this way, our method can expand to accommodate more tasks and compensate for sub-optimal network width choices.

Represent the TLDR for finding the corresponding scientific article.

We use simple and biologically motivated modifications of standard learning techniques to achieve state of the art performance on catastrophic forgetting benchmarks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Filter level sparsity emerges implicitly in CNNs trained with adaptive gradient descent approaches due to various phenomena, and the extent of sparsity can be inadvertently affected by different seemingly unrelated hyperparameters.