Question: Represent the article for finding a single-sentence TLDR

Question: Disentangling underlying generative factors of a data distribution is important for interpretability and generalizable representations. In this paper, we introduce two novel disentangling methods. Our first method, Unlabeled Disentangling GAN (UD-GAN, unsupervised), decomposes the latent noise by generating similar/dissimilar image pairs and it learns a distance metric on these pairs with siamese networks and a contrastive loss. This pairwise approach provides consistent representations for similar data points. Our second method (UD-GAN-G, weakly supervised) modifies the UD-GAN with user-defined guidance functions, which restrict the information that goes into the siamese networks. This constraint helps UD-GAN-G to focus on the desired semantic variations in the data. We show that both our methods outperform existing unsupervised approaches in quantitative metrics that measure semantic accuracy of the learned representations. In addition, we illustrate that simple guidance functions we use in UD-GAN-G allow us to directly capture the desired variations in the data. Generative Adversarial Networks (GANs) are generative model estimators, where two neural networks (generator and discriminator) are trained in an adversarial setting, so that likelihood-based probabilistic modeling is not necessary. This works particularly well for sampling from a complex probability distribution, such as images. Although GANs yield realistic looking images BID18, the original formulation in only allows for randomly sampling from the data distribution without disentangled structural or semantic control over the generated data points. One way to disentangle the generation process is to use conditional GANs BID16 BID17. These models modify the generator by conditioning it with supervised labels. Then, they either take the same labels as input in the discriminator BID16 and measure the image-label compatibility, or classify the correct label at the output, given the generated image BID17. Conditional GANs rely on a dataset with labels, which might not always be available or might be time-consuming to collect. In this paper, we propose two GAN-based methods that learns disentangled representations without using labeled data. Our first method, Unlabeled Disentangling GAN (UD-GAN), generates image pairs, then embeds them with Siamese Networks BID2, and finally learns a distance metric on a disentangled representation space. Whereas our second method, UD-GAN-G, uses guidance functions to restrict the input to our siamese networks, so that they capture desired semantic variations. There have been many studies on learning disentangled representations in generative models, which can be grouped into the level of supervision/labeled data they require. In BID28 BID26, the identity and the viewpoint of an object are disentangled via reconstructing the same object from a different viewpoint and minimizing a reconstruction loss. Whereas in BID15, the style and category of an object is separated via autoencoders, where an encoder embeds the style of an input image to a latent representation, and a decoder takes the category and style input to reconstruct the input image. In BID24, autoencoders and GANs are combined to decompose identity and attribute of an object, where the disentangled representation is obtained at the encoder outputs, and image labels are used at the output of the discriminator. Disentangled representations (semi-supervised). In BID20, they clamp the hidden units for a pair of images with the same identity but with different pose or expression to have the same identity representation. Whereas in BID11, synthesized images are used to disentangle pose, light, and shape of an object by passing a batch of images where only one attribute varies and the rest of the representation is clamped to be the same. These techniques only require a batch of samples with one attribute different at a time. Disentangled representations (unsupervised). InfoGAN BID1 is an unsupervised technique that discovers categorical and continuous factors by maximizing the mutual information between a GAN's noise variables and the generated image. β-VAE BID7 and DIP-VAE BID12 are unsupervised autoencoder-based techniques that disentangle different factors in the latent representation of an encoded image. In β-VAE, the KL-divergence between the latent and a prior distribution is weighted with a factor β > 1 to encourage disentanglement in the posterior latent distributions. Wheres in DIP-VAE, the covariance matrix of the latent distribution is encouraged to be an identity matrix, thus leading to uncorrelated latent representations. For all of the unsupervised methods, after a model is trained, a human needs to investigate which factors map to which semantic property. In addition, as the methods are unsupervised, not all desirable factors might be represented. In contrast, our method builds on existing approaches with two important modifications: (i) We operate on pairs of similar/dissimilar image pairs. (ii) We compute the image embeddings using separate networks, which allows us to guide the disentangling process with information restriction. In GANs, the generator, G, maps a latent variable z, which has an easy-to-sample distribution, into a more complex and unknown distribution, such as images. On the other hand, the discriminator D tries to distinguish real images from the ones that are generated by G. In, the training is performed as a minimax game as follows: DISPLAYFORM0 where P R and P Z are the probability distributions of real images and the latent variable z, respectively. We train our GAN by using the loss in equation 1. In order to increase stability, we modify the generator loss by maximzing log(D(G(z))), instead of minimizing the second term in equation 1. In a standard GAN setting, all of the variation in the distribution of real images is captured by the latent variable z. However, a single dimension or a slice of z does not necessarily have a semantic meaning. In this paper, our target is to slice the latent variable into multiple vectors, where each vector controls a different semantic variation. Our network architecture is visualized in Figure 1. In our method, the latent vector DISPLAYFORM0, which represent different attributes we aim to disentangle. One can add a final variable that captures the variation (and the noise) that is not picked up by the knobs. In our experiments, this additional variable did not have a notable effect. In our notation, q i refers to all of the knobs, except q i. In order to train our model, first, for each q i, we sample two different vectors, q The image pairs that are generated with the same q i vectors, {x 11, x 12} or {x 21, x 22}, should have the same i th attribute, regardless of the values of q i. We can ensure this via embedding the generated image pairs into a representation space with Siamese Networks BID2, which are denoted as φ i , and then learning a distance metric on the embedding vectors by employing Contrastive Loss BID6 ). An optional guidance function is used to restrict the information that goes into a siamese network, thus letting us approximate a desired representation space. The guidance is disabled for our unsupervised UD-GAN approach. Whereas for UD-GAN-G, the guidance is a simple, user-defined function, which is discussed in Section 3.3.We use a Contrastive Loss function to pull similar image pairs together, and push dissimilar pairs apart as follows: DISPLAYFORM1 where, L φ i is the Contrastive Loss for the i th Siamese Network φ i , the function DISPLAYFORM2 ) 2 is a shorthand for embedding distance between x ni1 and x ni2, and γ DISPLAYFORM3 is an adaptive margin of the form γ DISPLAYFORM4. Using an adaptive margin makes the distance between two latent samples semantically meaningful and we empirically found that it improves the training stability. The discriminator network D is not modified and is trained to separate real and generated image distributions. BID3 use a similar latent variable slicing for capturing illumination and pose variations of a face with a fixed identity. Their discriminator needs image pairs, which must be labeled for real images, to judge the quality and identity of the faces. Our method does not require any labels for the real images. Instead, we create similar and dissimilar image pairs via concatenating latent variables and generating image batches. Our final loss function is: DISPLAYFORM5 where, L W GAN is the GAN loss described in equation 1, λ φ i is the weight of the embedding loss, and the sampling of the latent variables depends on i and is performed as described above. A guidance function reduces the information content that flows into a siamese network and causes the corresponding embedding space to capture only the variations in the restricted input. For example, consider we want to capture the hair-related attributes in the CelebA dataset BID14, which contains aligned images of human faces. By cropping every region but the top part of a generated image, we are able to guide φ top to learn only the variations in the "Hair Color" as shown in the first row of FIG1. Note that, the knob q top (that corresponds to φ top ) changes the hair color not only at the cropped part of the image but as a whole. This is due to the interplay between the adversarial part of our loss (see equation 3), which enforces global realism in images, and the contrastive loss, which administers disentangled representations. As shown in FIG1, different guidance functions leads to capturing different variations in the CelebA dataset. Crop Guidance We can gain a probabilistic interpretation of our method on a toy example. Let us assume a problem, where we want to generate images of colored polygons (see Figure 1), where there are two independent factors of variation: shape and color, which we want to capture using two knobs q i and q j, respectively. When we set q j to a certain value and vary q i, we want to generate polygons with the same color, but different shapes, and vice versa. Let P be the probability distribution of colored polygons. For each attribute, P can be decomposed into a mixture distribution as follows: DISPLAYFORM0 is a mixture component and π DISPLAYFORM1 is its corresponding probability of choosing it, and N i is the number of different values an attribute (in our example, i corresponds to shape) can take. A similar explanation can be made for attribute j, i.e. color. For the sake of this analysis, we accept that for each attribute, P can be decomposed into different discrete mixture distributions as shown in Figure 3. For this specific case, Q i and Q i are the distributions of colored squares and colored diamonds, respectively. For the color attribute, which is indexed by j, each Q (k) j corresponds to a distribution of polygons with a single color (i.e., green polygons).Our contrastive loss in equation 2 has two terms. The first term is minimizing the spread of each mixture component Q (k) i. This spread is inversely related to disentanglement. If all samples from DISPLAYFORM2 Figure 3: Illustration of the embedding spaces and separated probability distributions after training our model. DISPLAYFORM3 are mapped to the same embedding vector, the effect of j (and any other attribute) on the representation φ i disappears and disentangling is achieved. During training, we stochastically go through all embedding spaces and minimize their spread, thus ing in a disentangled representation in TAB6 in Appendix G.The second term in equation 2 separates all Q (k) i from each other using an adaptive margin γ i. This margin depends on the difference between input latent pairs, so that the ing embedding space is smooth. In other words, we separate rectangles, circles, and ovals from each other, but circles should be closer to ovals than squares, due to their relative similarity. In the following, we focus on the shape attribute that is represented by i, however, derivations carry over to the color attribute j. In order to separate the probability distributions over image embeddings, one can maximize a divergence between all pairs from Q (k) i. One way to measure the distance between these distributions is to use the unbiased estimator of the energy distance BID23: DISPLAYFORM4 The energy distance in equation 5 can be interpreted as an instance of Maximum Mean Discrepancy BID0 and resembles the Contrastive Loss BID6. We can rewrite equation 5 using the Contrastive Loss in equation 2 as follows: DISPLAYFORM5 Each element in the second sum is quadratic function and has its minimum at DISPLAYFORM6 /2 and the value of the minimum is γ i 2 /2. So, we can rewrite equation 6 as follows: DISPLAYFORM7 Therefore, as the margin γ i depends only on the input latent variables and is not trainable, minimizing our embedding loss L φ i maximizes the lower bound for the energy distance D E. This corresponds to learning a Siamese Network φ i that separates two probability distributions Q We perform our experiments on a server with Intel Xeon Gold 6134 CPU, 256GB system memory, and an NVIDIA V100 GPU with 16GB of graphics memory. Our generator and discriminator architectures are outlined in our Appendix A. Each knob is a 1-dimensional slice of the latent variable and is sampled from Unif (−1, 1). We use ADAM BID9 as an optimizer for our training with the following parameters: learning rate=0.0002 and β 1 = 0.5. We will release our code after the review process. Datasets. We evaluate our method on two image datasets: (i) the CelebA dataset BID14, which consists of over 200,000 images of aligned faces. We cropped the images to 64 × 64 pixels in size. (ii) the 2D Shapes BID7, which is a dataset that is synthetically created with different properties, such as shape, scale, orientation, and x-y locations. Both datasets are divided into training and test sets with a 90%-10% ratio. The weight values for the contrastive loss is λ φ = 1 for the CelebA dataset and λ φ = 5 for the 2D shapes dataset. We use a 32 and 10-dimensional latent variables for the CelebA and the 2D Shapes datasets, respectively. Baselines. We have two versions of our algorithm. UD-GAN refers to the that are obtained without any guidance at the input of our siamese networks, whereas UD-GAN-G represents a guided training. We compare our method against β-VAE , DIP-VAE BID12, and InfoGAN BID1 to compare against both autoencoder and GAN-based approaches. We get the quantitative and visual for β-VAE and DIP-VAE from BID7 and BID12, and use our own implementation of InfoGAN for training and testing. The same generator/discriminator architecture is used for InfoGAN and our method. Guidance. For the CelebA dataset, the first 28 of 32 latent knobs are unguided and therefore are processed by the same siamese network that outputs a 28-dimensional embedding vector 1. Whereas the remaining four knobs correspond to four siamese networks (φ top, φ miu, φ mil, φ bot) that are guided with cropped images in FIG1. For the 2D shapes dataset, we have 10 knobs, where the first 7 dimensions are unguided. In order to guide the remaining three networks, we estimate the center of mass (M x,M y) and the sizeŜ of the generated object and feed them to our siamese networks, φ X (M x), φ Y (M y), and φ S (Ŝ). More information for this computation can be found in Appendix D. Disentanglement Metric. This metric was proposed by BID7 and measures whether learned disentangled representations can capture separate semantic variations in a dataset. In β-VAE and DIP-VAE, this representation is the output of the encoder, i.e., the inferred latent variable. For InfoGAN, we use the representation learned by the discriminator. In our method, we use the concatenated outputs of our siamese networks, which we denote as φ.The disentanglement metric scores for different methods are illustrated in TAB0. Here, we can see that both of our methods outperforms the baseline on the CelebA dataset. All of the baseline approaches relate the latent variables to generated images on per-image basis. Whereas our approach attempts to relate similarities/differences of latent variable pairs to image pairs, which provides a discriminative image embedding, where each dimension is invariant to unwanted factors BID6. For both datasets, our guided network (UD-GAN-G) performs better than our unguided approach, especially on the CelebA dataset. This might be due to the correlations between irrelevant attributes. For example the correlation coefficient between "Wearing Lipstick" and "Wavy Hair" attributes is 0.36, although they are not necessarily dependent. One of our guided networks receive the cropped image around the mouth of a person, which prevents cluttering it with hairstyle. Therefore, this guidance provides better disentanglement and in an improved score as shown in TAB0. Due to containing simple synthetic images, our disentanglement scores for the 2D shapes dataset are very high. The reason we get 100.0 score on our guided method is because of the guidances we choose, which are highly correlated with the ground truth labels, as shown in TAB4 in Appendix D. TAB1, we compare our method against baseline approaches on CelebA attribute classification accuracy using the aforementioned projection vector. Similar to the in TAB0, our guided approach slightly outperforms our unguided method and the other completely unsupervised techniques. This is because some attributes in the CelebA dataset can be spatially isolated via cropping, which leads to a better classification performance. For example, the attributes that are related to hair (Black Hair, Blond Hair, Wavy Hair) and mouth (Mouth Slightly Open, Wearing Lipstick) are captured better by the guided approach, because our top and bottom crops (see FIG1) are detaching the effects of other variations and are making attributes less correlated. The accuracy on the attribute "Bangs" is worse on the guided approach. This might be due to heuristic cropping we perform that divides the relevant image region into two slits. Table 3, we illustrate images generated by different methods on the CelebA dataset. Each of the three rows capture the change in a semantic property: smile, azimuth, and hair color, respectively. Within each image group, a latent dimension is varied (from top to bottom) to visualize the semantic change in that property. Compared to adversarial methods, such as InfoGAN and UD-GAN-G, the DIP-VAE method generates blurrier images, due to the data likelihood term in VAE-based approaches, which is usually implemented as a pixel-wise image reconstruction loss. In GAN-based approaches, this is handled via a learnable discriminator in an adversarial setting. In TAB0 and 2, we quantitatively show the advantage of using our guided approach. Another advantage is to have better control over the captured attributes. For example, in all unsupervised approaches (including UD-GAN), we need to check which latent dimension represents corresponds to which visual attribute. In some cases, a semantic attribute might not be captured due to the correlated nature of a dataset. Whereas, in UD-GAN-G, we directly obtain the variations in smile, azimuth, and hair color through cropping the bottom, middle, and top part of our images, respectively. Thanks to our guidance in FIG1, we can directly manipulate these three attributes using the knobs q bot, q mil, and q top as shown in Table 3.The same trend is true for the 2D Shapes dataset in Table 4. Although the X and Y positions and the scale of the synthetic object is captured by both our unsupervised and guided approaches, the guidance we choose directly captures the desired feature on in advance chosen knobs q X, q Y, and q S, respectively. Table 3: Images generated by varying a latent dimension, which corresponds to a semantic property. Table 4: Generated images for the 2D Shapes dataset by varying a latent dimension, which corresponds to a semantic property (first row: UD-GAN, second row: UD-GAN-G). DISPLAYFORM0 DISPLAYFORM1 In completely unsupervised approaches, there is no guarantee to capture all of the desired semantic variations. The main premise behind UD-GAN-G is to find very simple, yet effective ways to capture some of the variation in the data. This weak supervision helps us to obtain proxies to certain semantic properties, so that we get the desired features without training the model multiple times with different hyperparameters or initializations. In the aligned the CelebA dataset, each face is roughly centered around the nose. This reduces the variation and simplifies the problem of guidance design, as we show in FIG1. In more complex scenarios, where the objects can appear in a large variety of scales, translations, and viewpoints, one can use a pre-trained object detection and localization method, such as YOLO BID19, as a guidance network. This enables us to use the knowledge obtained from a labeled dataset, such as ImageNet BID21 to disentangle a new unlabeled dataset. Note that backpropagating the gradients of a deep network into an image might cause adversarial samples BID22. However, the discriminator can alleviate this by rejecting problematic images. In order to backpropagate the gradients from the siamese networks to the generator, the guidance function we use needs to be differentiable. This might pose a limitation to our method; however, differentiable relaxations can instead be used to guide our network. For example, one can employ differentiable relaxation of the superpixel segmentation in BID8 ) to disentangle a low-level image segmentation. Our latent variables are sampled from a uniform distribution. In addition, image similarity is measured by using L2-distance between a pair of image embeddings. We experimented with modeling some latent dimensions as categorical variables. However, we encountered training stability issues, due to computing the softmax loss between two learnable categorical image embeddings, instead of one embedding and one fixed label vector as it is usually done. We plan to tackle that problem in our future work. In this paper we introduced UD-GAN and UD-GAN-G, novel GAN formulations which employ Siamese networks with contrastive losses in order to make slices of the latent noise space disentangled and more semantically meaningful. Our experiments encompassed guided and unguided approaches for the embedding networks, and illustrated how our methods can be used for semantically meaningful image manipulation. Our qualitative and quantiative confirm that our method can adjust well to the intrinsic factors of variation of the data and outperform the current state-of-the-art methods on the CelebA and 2D Shapes datasets. In future work, we plan to investigate more powerful forms of embedders, e.g. extracting information from pre-trained networks for semantic segmentation and landmark detection. This allows for even more powerful novel image manipulation techniques. In TAB2, we show the neural network layers we use in our generator for different datasets. Our discriminator and siamese network architectures are the inverted version of our generator. Each fully connected and Conv2D layer is followed by a Leaky ReLU non-linearity, except the last layer. DISPLAYFORM0 The Siamese Networks φ i are desired to map images into embedding spaces, where they can be grouped within a distinct semantic context. For the example shown in FIG3, where we disentangle the shape and the color, this might not be directly achievable in a completely unsupervised setting, because the separation in equation 4 is not unique. However, we can still benefit from the disentangling capability of our method via small assumptions and domain knowledge, without collecting labeled data. Consider the toy example, where we extend the MNIST dataset BID13 to have a random color, sampled from a uniform RGB color distribution. We define our problem to independently capture the shape of a digit with q 1 and its color with q 2.In FIG3 (a), we show images created by a generator, which is trained along with two networks, φ 1 and φ 2, without any guidance in an unsupervised setting. We can see that the knobs, q 1 and q 2, capture the variations in the data, however, these variations are coupled with multiple semantic properties. Each knob modifies a complicated combination of shape and color. However, if we design a network architecture in a slightly smarter way, we should be able to separate the shape and the color attributes. This is exemplified in FIG3 (b), where instead of feeding the whole image to φ 2, we feed the average color of some randomly sampled pixels from a generated image. This choice prevents φ 2 to capture the spatial structure of the generated digit and to focus only on color. After the training our method with a modified φ 2, the first network captures shape of a digit, and the second one captures the color variations. This can also be observed in FIG3 (c) and 4(d), where we use t-SNE (van der BID25 to visualize embedding spaces for shape and color, respectively. In order to show the effect of the guided siamese networks, we perform three experiments on the MS-Celeb dataset BID5 by using different guiding proxies. In the first experiment, only one of the two networks is guided with an edge detector at the input. Results of this experiment are shown in TAB3 . We can see that the first knob, which is connected to edges, captures the overall outline and roughly controls the identity of the generated face. On the other hand, the unguided second knob modifies the image with minimal changes to image edges. This change, in this case, corresponds to the lighting of the face. DISPLAYFORM0 We perform a second experiment with the edge detector, where in this case, the second knob is guided with the average color of the generated image. In TAB3, we can observe the of our disentangled image manipulation. The first knob with the edge detector again captures the outline of the face, and the second average color knob modifies a combination of the light and the skin color, similar to the in TAB3 .In our third experiment, we employ the cropped guidance networks. The two knobs receive the cropped top and bottom part of the image for training. Although these image crops are not independent, we still get acceptable that are shown in TAB3 . Adjusting the first knob only modifies the upper part of the face; the hair and the eyes. Similarly, the second knob is responsible for determining the chin and mouth shape. In order to guide our siamese networks for the 2D shapes dataset, we estimate the center of mass of the generated image, and the size of the generated object as follows: DISPLAYFORM1 where, x is a generated image, x[c x, c y] is the pixel intensity at image coordinates [c x, c y], (M x,M y) are the coordinates of the center of mass of x, andŜ is the size estimate for the generated object. As the 2D shapes dataset is relatively simple and contain only one object, these guidances are highly correlated with the ground truth attributes as shown in TAB4. FIG5, we illustrate additional semantic properties that are captured by UD-GAN-G. In TAB5, we compare the classification perfromance of our method to InfoGAN on all attributes in the CelebA dataset. G ATTRIBUTE CORRELATIONS.In TAB6, we compare the correlation between different embedding (or latent) dimensions and the correlation between embedding dimensions and the CelebA attributes. Although DIP-VAE encodes a more un-correlated representation, due to the correlated nature of CelebA attributes, it does not necessarily transfer to a disentangled semantic representation, as illustrated by the quantitative in TAB0 and 2.

represent this text to find a much much longer article about it with say hundreds-thousands of words

------

For example, 'This paper propose a mask method which solves the previous blurred results of unsupervised monocular depth estimation caused by occlusion' should be close to 'Unsupervised monocular depth estimation has made great progress after deep learning is involved. Training with binocular stereo images is considered as a good option as the data can be easily obtained. However, the depth or disparity prediction show poor performance for the object boundaries. The main reason is related to the handling of occlusion areas during the training. In this paper, we propose a novel method to overcome this issue. Exploiting disparity maps property, we generate an occlusion mask to block the back-propagation of the occlusion areas during image warping. We also design new networks with flipped stereo images to induce the networks to learn occluded boundaries. It shows that our method achieves clearer boundaries and better evaluation on KITTI driving dataset and Virtual KITTI dataset. Monocular depth estimation becomes an active research topic as deep learning is applied in various computer vision tasks. It has many applications, from navigation through to scene understanding. A single traditional camera can be a cheaper alternative to the expensive LIDAR sensor for automotive cars if accurate estimation can be achieved. Meanwhile, single camera simplifies the design of depth estimation solution which can be adopted quite widely at a low cost. One straight-forward way to train deep depth estimation models is to use ground truth depth images as the supervision signals BID1. However, supervised deep learning method is eager for massive data with ground truth. Collecting large datasets with ground truth depth in varied real scenarios is challenge and expensive. Instead, training using stereo images without depth label is an alternative option. BID7 proposed a method to exploit the left-right consistency of stereo images to tackle the monocular depth estimation, which achieved quite promising . However, the depth predicted by their method has blurred boundaries. The issue is mainly due to the occlusions during the image warping. Though it can be alleviated in some extent with proper post processing, the fundamental problem is not well addressed. In this paper, we propose a new method to overcome the blurred boundaries when using stereo pairs to train the monocular depth model. An example is illustrated in FIG0. During the image warping, we generate an occlusion mask using the disparity map to block the inappropriate back-propagation gradients for occlusion areas. However, the mask only cannot guarantee clear boundaries as there is no constrain for the masked areas. Then we design new networks to fully exploit the information of stereo images. With flipped stereo pairs, the network is induced to learn clear boundaries for occlusion areas. Our method provides a solution to the fundamental learning difficulty of occluded areas introduced by image warping in depth estimation. Empirical evaluation on KITTI driving dataset BID6 ) and Virtual KITTI dataset BID4 ) demonstrates the effectiveness of our approach. Moreover, we find the depth label of KITTI 2015 is usually very sparse near the object boundaries, which is not very sensitive to evaluate the clearness of boundaries. Large amounts of multi-view based approaches have been proposed such as stereo matching BID22 ), difference view point BID3 ) or temporal sequence BID18 ). Here we briefly review work based on single view depth estimation which is usually harder since reasoning depth from monocular colored image is an ill-posed problem. The most intuition way is treating monocular depth estimation problem as a supervised problem by taking RGB images as inputs and Lidar depth points as ground truth. BID21 proposed a method known as Make3d which breaks the image into homogeneous patches. For each small homogeneous patch, Saxena et al used a Markov Random Field (MRF) to infer a set of plane parameters that capture both the 3D location and 3D orientation. However, this approach has a hard time capturing thin structures since the predictions are made locally. BID1 first exploited CNN in a coarse to fine manner. BID13 BID12 proposed a network jointly explore the capacity of deep CNN and continuous conditional random field (CRF). BID10 incorporated semantic segmentations in to single-view depth estimation task since they are closely tied to the property of perspective geometry. BID11 proposed a residual network with up-sampling module using fully convolutional architecture. Also, reverse Huber loss was introduced. However, large amount of high-quality labelled data is needed, which is hard to require in practice. To overcome the lack of high quality labelled data, several semi-supervised and fully unsupervised methods have been proposed. BID2 first proposed a view synthesis based method called DeepStereo which generates new view image from nearby image. BID24 FIG2 proposed a method which generates the right image through probability distribution over all the possible disparities for each pixel. BID5 first proposed a warp based method by aligning reconstructed image with the ground truth left image as described in 3.1. However, their loss is not fully differentiable. BID7 improve this methods by introducing a novel loss. BID19 extended the network into two separate channel with 6 or 12 losses which improves the . Based on Garg et al, BID9 proposed a semi-supervised methods that exploited both the sparse Lidar points as supervision and stereo pairs as unsupervion signals. The semisupervised method was further improved by BID15, they decoupled the monocular depth prediction problem into two procedure, a view synthesis procedure followed by stereo matching. Recently, several work using only monocular temporal sequence comes out which enables more training data such as video sequence on YouTube. BID26 proposed a network that predicts depth and camera pose separately. Using the predicted depth and camera pose, relative temporal image can be reconstructed by image warping with which final loss can be constructed. BID16 performed a novel 3D loss that enforces consistency of the estimated 3D point clouds and ego-motion across consecutive frames and combined it with 2D photometric loss. proposed a differentiable implementation of Direct Visual Odometry (DVO) and a novel depth normalization strategy. However, all the temporal sequence based training meet the same problem of object motion. This problem can be alleviated by including stereo pairs during training known as trinocular training BID25 BID17 BID8 ). However, all these warp based methods have the difficulty of learning occluded area which would infect the . Godard et al achieves state of art of unsupervised monocular depth estimation with only stereo images. Follow Godard et al, we proposed monocular depth estimation network with novel mask methods which can be trained end-to-end and without ground-truth label. Our method is superior to Godard et al in quality with clearer boundaries, especially on dense evaluation dataset such as virtual-KITTI BID4 ). In general, our goal is to learn a network that can predict a pixel-wise dense depth map from single colored image(DISPLAYFORM0 However, all supervised methods have a hard time with acquiring large dense labelled data in real scenarios. Thus, several unsupervised methods have been proposed to overcome the obstacle. Among these methods, training by image reconstruction using rectified stereo pairs became more and more popular currently due to its high accuracy and easy accessibility of training data. BID5, Left-right consistency network proposed by Godard et al, Our refined network without shared parameters and our refined network with shared parameters First proposed by BID5, the monodepth estimation network takes left image (I l) as input and outputs the disparity aligned with the right image (d r) which can be used to sample from the left image (I l) to reconstruct the right image (Ĩ r) during training. Thus, image reconstruction loss can be constructed between the reconstructed right image(Ĩ r) and original right image(I r). When testing, only one colored image are required, the predicted disparity can be converted to depth simply using depth = b * f /disparity, where b and f are given baseline and camera focal length respectively. It is worth to mention that disparities (d l and d r) are a scalar per pixel as the images are rectified. The network was further improved by BID7 by introducing left-right consistency loss and refined encoder-decoder network. Given the input left image, the network predicts both the left and right disparities simultaneously which enables constructing the left-right consistency. This consistency restriction leads to more accurate and less artifacts. Also, fully differentiable backward bilinear sampling was used to reconstruct the left image which makes the model easier to optimize. With better network architecture and better loss, Godard et al achieved the state of art of unsupervised monodepth estimation only with rectified stereo pairs, and even outperforming supervised methods. Their network architectures are shown in FIG1.However, there still are unsatisfactory artifacts at the occlusion boundaries showing blurred ramps on the left side of image and of the occluders. Even with the post-processing step which weighted sums the flipped disparity of the flipped input image and the disparity of the input image, the blurred ramps are still visible near the objects, especially for those near the cameras with higher disparities as illustrated in FIG0. Though common backward warping using bilinear sampler is fully differentiable which enables us to train the model end to end, some undesirable duplicates and artifacts are introduced during the warping process because of occlusions according to BID14. In FIG2, we use SYTHIA dataset BID20 ), a synthesized virtual driving dataset to illustrate. Though ground truth disparities are used for warping, there still exists obvious duplicates and even a huge black region on the left of the reconstructed image(Ĩ l) because of the occlusion. If those inevitable artifacts and duplicates are back propagated during the training process, unwanted high losses will be introduced forcing the network learn to blur in those regions, as the blurriness(disparity ramps) in the occluded regions will make the reconstructed image show stretched patterns (Fig 7) which are more similar to original ground truth image compared to duplicates and large black regions FIG2 ).. We used a warping mask which is generated automatically from disparities to block the back-propagation of those artifacts. The final output is shown in (e), where the white regions are masked. In order to block the back propagation process of those warping induced artifacts, we designed an algorithm which can mask the occlusion region automatically. In backward bilinear sampling process, a disparity map was used to sample from the source image. The intuition here is that, if any pixel in the source image has never been sampled during the bilinear sampling process, this pixel should only be visible in the source image and should be masked when reconstructing this source image later on. Thus, the mask methods takes, say, left disparity as input while generating the mask of reconstructed right image and vice versa. The pseudo code is shown in Algorithm 1 and the mask schematic diagram is shown in Fig 4. However, the mask method alone cannot guarantee the clearness since no constrain is added in masked region. Specifically, though the masks block the back propagation of duplicates and artifacts induced by warping, they also block the further learning process of those regions. Once the disparities start to blur, hardly can we correct the network back to clearness. To solve this problem, we refined the network architecture and introduced a flip-over training scheme. Though the mask blocks the process of further learning of the blurred regions, we find that the blurred side are definite which can be exploited to reactive the learning process. For example, when the network is only trained on left images and takes corresponding right images as ground truth, the disparity ramps (blurred regions) will only appear on the left side of the occluders. So, if we randomly flipped the input images horizontally, the disparity ramps will still appear on the left side. When flipped the output disparities back for warping, the blurred regions will appear on the right side where no mask is added. Examples are shown in the last column of Fig 7. Thus, those blurred regions will make distortions when reconstructing images, and back propagate despite the 3 for x,y in grid(h,w) do DISPLAYFORM0 13 end 14 end 15 return mask l, mask r mask. Also, because the flip-over scheme is performed randomly, any blurriness on the definite side will receive punishment on average which restrict the prediction to be clear. It is worth to mention that flip-over scheme will not affect masking process and we still use masks to block the back propagation of duplicates. The flip-over schematic diagram is shown in Fig. 4.However, the predicted disparities of the right branch won't make any sense if we take the flipped left image as input and are totally mismatched with the ground truths as is shown in Fig 4. As a , we delete the predicting branch of the right disparity and add another encoder-decoder network which takes right images as input and predicts right disparities as is shown in FIG1 This doubled encoder-decoder network enables us to preform left-right consistency loss at the cost of doubling the training parameters. However, it won't slow down the test speed since only one branch of encoder-decoder network is used when testing. We also tried another network architecture with shared the encoder-decoder network which achieves comparable as non-shared network while halves the training time. More details can be found in 6.1 We use similar training loss as BID7. The losses are performed on four different scale and are finally summed as the total loss. C = 4 s=1 C s. For each scale, three different losses including appearance match loss(C ap), disparity smoothness loss(C ds) and LR consistency loss(C lr) are performs as follows. DISPLAYFORM0 Intuitively, appearance matching loss measures photometric error between the reconstructed image(Ĩ l ij)) and the ground truth image(I l ij), which is defined as the weighted sum of L1 and SSIM shown as follows, DISPLAYFORM1 Disparity smoothness loss aims to force the smoothness of predicted disparities through Figure 4: Schematic diagram. The orange squares refer to an object such as pedestrians or vehicles. The blue and black squares refer to blurred disparity region and masked region respectively. Left: Naive masking process. Warping mask(black) is overlap with blurred region(blue). As a , masks will block the learning of those region. Right: Flip-over training scheme. We flipped the left images(f (I l)) as input and flipped the output disparities(f (d(f (I l)))) back to reconstruct the image(Ĩ l). Different from the diagram shown in left, the blurred region will switch to the opposite side to avoid the mask. As a , losses will be introduced leading to clear boundaries. We can also observed that the right branch (flipped predicted right disparity of flipped left image(f (d(I r)))) is totally mismatch with the ground truth right image, so we delete this branch and add another encoder-decoder branch as shown in FIG1. DISPLAYFORM2 Finally, L1 penalty is added to constrict the left-right consistency, which further reduce the artifacts, DISPLAYFORM3 Our network is based on BID7 and is implemented in Tensorflow (Abadi et al.). The network contains encoder network based on VGG16 or Resnet50 and decoder network with 7 upconvolutional layers, 7 merging layers and 4 disparity prediction layers. With 7 skip connections, the network can better handles features at different scales. Different from Godard et al, we modified the channel number of disparity prediction layers to predict only one disparity instead of two. Also, we tuned the default hyper parameters α and α ds. The rest are the same with Godard et al with α ap = 1, α lr = 1. The learning rate λ remains 10 −4 for the first 30 epoch and halves every 10 epoch when trained for 50 epoch. We also tried batch norm but might lead to unstable . Data augmentation is performed on the fly similar as Godard et al including flipping and color shifting. We trained our model on rectified stereo image pairs in KITTI and Cityscapes dataset and evaluated mainly on KITTI split BID6 ), Eigen split BID1 ) and virtual-KITTI datasets BID4 ). Also, we find the problem of KITTI 2015 evaluation such as sparsity and man-made defects which infects our evaluation . When testing on dense datasets like virtual-KITTI, the superiority becomes more obvious. For comparison, we use the same split as BID7 high quality disparity image issued by official KITTI dataset. It is worth mentioned that though these disparity images are of better quality than projected velodyne laser point and with CAD models inserted in cars, most part of ground truth are extremely sparse, especially in terms of occluded regions. As the red boxes in the FIG3 shown, there is some blank space on the left side of image and occluded regions which is not involved in the evaluation. Thus, those blank space just covers the shortcomings the disparity ramps on the occluded regions. As a , our shows less superiority over Godard et al on KITTI stereo 2015 dataset. As shown in TAB2, we use the metrics from BID1 and D1-all metrics from KITTI. Our model achieves comparable with Godard et al when both trained on KITTI dataset only for 50 epoch, and superior when both trained longer for 100 epoch. BID6 ). K means the model is only trained on KITTI dataset, while CS + K means the model is trained on Cityscapes dataset and then finetune on KITTI dataset. Also, pp means post processing step, and more details about post processing can be found in 6.3. For a fair comparison, we train the network proposed by Godard et al and our non-shared network both to 50 and 100 epoch, and find that our network has larger improvement and better performance than BID7 when trained longer to 100 epoch. We guess that our network requires longer time to converge. We choose different hyperparameters from Godard et al for our network. Though it is unfair to evaluation on sparse KITTI dataset, our still outperforms that of Godard et al. Similar to Godard et al, we use the test split of 697 images proposed by BID1. Each image contains 3D points captured by velodyne laser which was used to generate ground truth depth. We keep the same 22600 stereo pairs for training. According to Godard et al, all evaluation are done under the Garg crop BID5 ) except for Eigen for a fair comparison. We also present the uncropped which our models' superiority become more obvious, because it will crop the disparity ramps on the left side for evaluation which boosting Godard et al's . Also, ground truths are captured by velodyne Lidar thus rather sparse which would reduce our superiority over Godard et al. The evaluation is in TAB4 4.3 VIRTUAL-KITTITo prevent sparsity induced inaccurate , we evaluate models on the Virtual-KITTI dataset BID6 ). Virtual-KITTI dataset contains 50 monocular videos generated from five different virtual worlds in urban settings under different weather conditions with corresponding pixel-level ground truth depth. With the same resolution, scenes and camera parameters as KITTI 2015, Virtual-KITTI dataset can be implement naively when testing. Since KITTI 2015 dataset we used for training does not cover those weathers, only 2126 labbeled images without weather condi-Method Supervised Dataset Abs Rel Sq Rel RMSE RMSE log δ < 1.25 δ < 1.25 2 δ < 1.25 3 BID1 BID1. K is KITTI and CS is cityscapes dataset for training. We use the evaluation provided in BID7. For a fair comparison, we use the crop the same as BID5 except for BID1 and apply the same hyper-parameters as Godard et al on our model. Besides, we set the maximun evaluation depth to 50 meters(cap 50m) in the second row which is the same as BID5, while others remain 80 meters. Also, we compared uncropped with Godard et al, on which the superiority become more obvious.tion are used for evaluation. The evaluation is shown in TAB5: Evaluation on virtual-KITTI. Once our ground truth become dense, our model out performance other models for its sharp and clear boundaries on predicted depth map. Even we crop the black edges of Godard et al (the left 10% of the whole disparity map) in the second row, our model is still superior to BID7 (the state of art unsupervised monocular depth estimation using left and right information only). We evaluate on 3 different methods: naive, crop the black edge, both crop the edge and set the maximun evaluation depth to 50 meters In this work, we present an occlusion mask and filp-over training scheme to enable effective learning of object boundaries when using image warping. With our new network, our model achieves state of art using only stereo images. Moreover, as warping based image reconstruction is commonly used in depth estimation problem, our method provides a solution to the fundamental difficulty of occluded areas introduced by image warping. In the future, our method can be incorporated with more accurate network trained on trinocular data (temporal Stereo sequence) such as BID25, BID17 and BID8, which would further boost the accuracy.6 SUPPLEMENTARY MATERIALS The shared weight network is similar to the non-shared weight network except for the shared weight. The architecture is shown in 2. Fortunately, the shared weight model naturally handled the problem. Because the blurriness always occur in the occluded region, say, disparity ramps (burriness) in left disparity maps will only appear on the left side of the occluders, the network cannot distinguish which side the input image belongs to and which side to blur when trained on the left and right image simultaneously with shared weights. Thus, any wrong blurriness (blurriness on the wrong side, say, right blurriness in the left disparity maps) will lead to punishment in raising reconstructed loss since no mask was adding on the other side (wrong side, say, right side). As a , the network achieves comparable as non-shared network while halves the training time. However, shared method is slightly inferior to non-shared network but still out performance Godard FIG0, Godard et al used non-linear weight. Because our model do not predict black edge, so we simply average them as our post-processing. Here we use the network by BID7 to illustrate how flip-over scheme could solve the problem of back-propagate, so the predicted disparities are rather blurred. In Fig 7, from left to right: left disparity(d l), reconstructed left image(Ĩ l), corresponding mask(mask l), ground truth(I l), flipped disparity of flipped input imagef (d(f (I l))). To minimize the reconstruction error, the network predicts blurred forcing the network to sample from the , which leads to more similar stretched patterns instead of high loss duplicates as shown in FIG2. However, the blurred regions are aligned with the mask, which freezes the fine-tuning process of blurred regions. To solve the problem, we randomly flip the colored image as input(Input = f (I l)), then flipped the output back as final disparities for warping(f (d(f (I l)))). As shown in the last column, the blurred regions are no longer aligned with the masks which enables further learning process. As the training process goes on, the boundaries will become clearer as shown in FIG0. We also use schematic diagram to illustrate in Fig. 4 Figure 7: Example of blurred and flip-over scheme.'

We use Siamese Networks to guide and disentangle the generation process in GANs without labeled data.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Conditional VAE on top of latent spaces of pre-trained generative models that enables  transfer between drastically different domains while preserving locality and semantic alignment.


Question: We present Predicted Variables, an approach to making machine learning (ML) a first class citizen in programming languages. There is a growing divide in approaches to building systems: using human experts (e.g. programming) on the one hand, and using behavior learned from data (e.g. ML) on the other hand. PVars aim to make using ML in programming easier by hybridizing the two. We leverage the existing concept of variables and create a new type, a predicted variable. PVars are akin to native variables with one important distinction: PVars determine their value using ML when evaluated. We describe PVars and their interface, how they can be used in programming, and demonstrate the feasibility of our approach on three algorithmic problems: binary search, QuickSort, and caches. We show experimentally that PVars are able to improve over the commonly used heuristics and lead to a better performance than the original algorithms. As opposed to previous work applying ML to algorithmic problems, PVars have the advantage that they can be used within the existing frameworks and do not require the existing domain knowledge to be replaced. PVars allow for a seamless integration of ML into existing systems and algorithms. Our PVars implementation currently relies on standard Reinforcement Learning (RL) methods. To learn faster, PVars use the heuristic function, which they are replacing, as an initial function. We show that PVars quickly pick up the behavior of the initial function and then improve performance beyond that without ever performing substantially worse -- allowing for a safe deployment in critical applications. Machine Learning (ML) has had many successes in the past decade in terms of techniques and systems as well as in the number of areas in which it is successfully applied. However, using ML has some cost that comes from the additional complexity added to software systems BID24. There is a fundamental impedance mismatch between the approaches to system building. Software systems have evolved from the idea that experts have full control over the behavior of the system and specify the exact steps to be followed. ML on the other hand has evolved from learning behavior by observing data. It allows for learning more complex but implicit programs leading to a loss of control for programmers since the behavior is now controlled by data. We believe it is very difficult to move from one to another of these approaches but that a hybrid between them needs to exist which allows to leverage both the developer's domain-specific knowledge and the adaptability of ML systems. We present Predicted Variables (PVars) as an approach to hybridize ML with programming. We leverage the existing concept of a variable which is universal across all programming modalities and add a new type, a predicted variable. PVars are akin to native variables with one important distinction: a PVar determines its value using ML when evaluated. A developer will be able to use a PVar just like any other variable, combine it with heuristics, domain specific knowledge, problem constraints, etc. in ways that are fully under the programmer's control. This represents an inversion of control compared to how ML systems are usually built. PVars allow to integrate ML tightly into algorithms whereas traditional ML systems are build around the model. PVars aim to make using ML in software development easier by avoiding the overhead of going through the traditional steps of building an ML system: collecting and preparing training data, defining a training loss, training an initial model, tweaking and optimizing the model, integrating the model into their system, and continuously updating and improving the model to adjust for drift in the distribution of the data processed. We show how these properties of PVars allow for applying ML in domains that have traditionally not been using ML. We demonstrate that ML can help improve the performance of "classical" algorithms that typically rely on a heuristic. The concrete implementation of PVars in this paper is based on standard deep reinforcement learning (RL). We emphasize that this is just one possible implementation. Other types of machine learning are in scope for PVars: supervised learning can be used when ground truth is available, or active learning is applicable when humans are in the loop to provide feedback. While in this paper we show PVars in the context of the Python programming language and use concepts from object oriented programming, everything described here applies directly to functional or procedural programming languages as well. We describe the framework around the central concept of a predicted variable but depending on the language the notion of a predicted function can be used interchangeably. We also introduce the notion of an initial function which can be the current heuristic that a PVar is replacing. It allows the PVar to minimize regret in critical applications and allow for safe deployment. This is a key strengths of our hybrid approach: it allows for better solutions while also providing better guarantees to the programmer. We demonstrate the feasibility of our approach on three algorithmic problems: binary search, QuickSort, and caches where we replace and enrich commonly used heuristics. We show improvements over common heuristics by injecting a predicted variable into an existing program, leaving much of the algorithm (including the domain-specific knowledge) untouched. We consider these problems the first applications of our newly defined interface and see the main contribution of this paper in the general applicability of the framework. The problem selection in this paper was driven by the desire for a self-contained setup and ease of reproducibility. PVars are applicable to more general problems across a large variety of domains from system optimization to user modelling. In our experiments we do not focus on the actual run time but rather on the effectiveness of the ML models. While for the algorithmic examples in this paper, in a practical scenario speed is the key metric, we see PVars as a more general interface that can be applied across a more diverse set of problems including user modelling, predicting user preference, or content recommendations. In many applications, speed is not a meaningful metric. Further, we believe that advances in specialized hardware will enable running machine learning models at insignificant cost .Our main contributions are:• we introduce the PVar API to smoothly integrate ML into software development;• we show how standard RL methods can be leveraged through the PVars interface;• we propose an approach to learn using the initial function, leveraging off-policy learning;• we demonstrate the feasibility of our approach on 3 standard algorithmic problems. The remainder of this paper is structured as follows: We describe how PVars can be used in software development in sec. 2 and how we make use of the heuristics that we are replacing to guide the training and avoid unstable behavior in sec. 3. Sec. 4 describes our implementation and the application of PVars to three algorithmic applications. We also describe the experiments that we performed to demonstrate that PVars are an intuitive approach to bring ML closer to software development and are applicable to different problems. We describe related work in sec. 5. A PVar has a simple API that allows the developer to provide enough information about its context, predict its value, and provide feedback about the quality of its predictions. PVars invert the control compared to common ML approaches that are model centric. Here, the developer has full control over how data and feedback are provided to the model, how inference is called, and how its are used. To create a PVar, the developer chooses its output type (float, int, category, ...), shape, and range; defines which data the PVar is able to observe (type, shape, range); and optionally provides an initial function. In the following example we instantiate a scalar float PVar taking on values between 0 and 1, which can observe three scalar floats (each in the range between 0 and 10), and which uses a simple initial function: The PVar can then be used like a normal variable. It determines its value at read time by using inference in the underlying ML model, e.g. value = pvar. Predict Specifically, developers should be able to use a PVar instead of a heuristic or an arbitrarily chosen constant. PVars can also take the form of a stochastic variable, shielding the developer from the underlying complexity of inference, sampling, and explore/exploit strategies. The PVar determines its value on the basis of observations about the context that the developer passes in: A developer might provide additional side-information into the PVar that an engineered heuristic would not be using but which a powerful model is able to use in order to improve performance. The developer provides feedback about the quality of previous predictions once it becomes available: DISPLAYFORM0 In this example we provide numerical feedback. Following common RL practice a PVar aims to maximize the sum of reward values received over time (possibly discounted). In other setups, we might become aware of the correct value in hindsight and provide the "ground truth" answer as feedback, turning the learning task into a supervised learning problem. Some problems might have multiple metrics to optimize for (run time, memory, network bandwidth) and the developer might want to give feedback for each dimension. This API allows for integrating PVars easily and transparently into existing applications with little overhead. See listing 1 for how to use the PVar created above in binary search. In addition to the API calls described above, model hyperparameters can be specified through additional configuration, which can be tuned independently. The definition of the PVar only determines its interface (i.e. the types and shapes of inputs and outputs). We allow for the developer to pass an initial function to the PVar. We anticipate that in many cases the initial function will be the heuristic that the PVar is replacing. Ideally it is a reasonable guess at what values would be good for the PVar to return. The PVar will use this initial function to avoid very bad performance in the initial predictions and observe the behavior of the initial function to guide its own learning process, similar to imitation learning BID13. The existence of the initial function should strictly improve the performance of a PVar. In the worst case, the PVar could choose to ignore it completely, but ideally it will allow the PVar to explore solutions which are not easily reachable from a random starting point. Further, the initial function plays the role of a heuristic policy which explores the state and action space generating initial trajectories which are then used for learning. Even though such exploration is biased, off-policy RL can train on this data. In contrast to imitation learning where an agent tries to become as good as the expert, we explicitly aim to outperform the initial function as quickly as possible, similar to BID23.For a PVar to make use of the initial heuristic, and to balance between learning a good policy and the safety of the initial function, it relies on a policy selection strategy. This strategy switches between exploiting the learned policy, exploring alternative values, and using the initial function. It can be applied at the action or episode level depending on the requirements. Finally, the initial function provides a safety net: in case the learned policy starts to misbehave, the PVar can always fallback to the initial function with little cost. In the following we describe how PVars can be used in three different algorithmic problems and how a developer can leverage the power of machine learning easily with just a few lines of code. We show experimentally how using PVars helps improving the algorithm performance. The interface described above naturally translates into an RL setting: the inputs to Observe calls are combined into the state, the output of the Predict call is the action, and Feedback is the reward. To evaluate the impact of PVars we measure cumulative regret over training episodes. Regret measures how much worse (or better when it is negative) a method performs compared to another method. Cumulative regret captures whether a method is better than another method over all previous decisions. For practical use cases we are interested in two properties: Regret should never be very high to guarantee acceptable performance of the PVar under all circumstances. Cumulative regret should become permanently negative as early as possible. This corresponds to the desire to have better performance than the baseline model as soon as possible. Unlike the usual setting which distinguishes a training and evaluation mode, we perform evaluation from the point of view of the developer without this distinction. The developer just plugs in the PVar and starts running the program as usual. Due to the online learning setup in which PVars are operating, overfitting does not pose a concern BID3. The (cumulative) regret numbers thus do contain potential performance regressions due to exploration noise. This effect could be mitigated by performing only a fraction of the runs with exploration. For our feasibility study we do not account for the computational costs of inference in the model. PVars would be applicable to a wide variety of problems even if these costs were high, particularly for problems relying on expensive approximation heuristics or working with inherently slow hardware, such as filesystems. Our implementation currently is a small library exposing the PVar interface to client applications (FIG1 . A PVar assembles observations, actions, and feedback into episode logs that are passed to a replay buffer. The models are trained asynchronously. When a new checkpoint becomes available the PVar loads it for use in consecutive steps. To enable PVars we leverage recent progress in RL for modelling and training. It allows to apply PVars to the most general use cases. While we are only looking at RL methods here, PVars can be used with other learning methods embedded such as supervised learning or multi-armed bandit methods. We are building our models on DDQN BID9 for categorical outputs and on TD3 BID4 for continuous outputs. DDQN is a de facto standard in RL since its success in AlphaGo . TD3 is a recent modification to DDPG BID19 using a second critic network to avoid overestimating the expected reward. We summarize the hyperparameters used in our experiments in the appendix TAB5 . While these hyperparameters are now new parameters that the developer can tweak, we hypothesize that on the one hand, tuning hyperparameters is often simpler than manually defining new problem-specific heuristics and on the other hand that improvements on automatic model tuning from the general machine learning community will be easily applicable here too. Our policy selection strategy starts by only evaluating the initial function and then gradually starts to increase the use of the learned policy. It keeps track of the received rewards of these policies adjusts the use of the learned policy depending on its performance. We show the usage rate of the initial function when we use it ( FIG3 demonstrating the effectiveness of this strategy. Similar to many works that build on RL technology we are faced with the reproducibility issues described by BID10 . Among multiple runs of any experiment, only some runs exhibit the desired behavior, which we report. However, in the "failing" runs we observe baseline performance because the initial function acts as a safety net. Thus, our experiments show that we can outperfom the baseline heuristics without a high risk to fail badly. We do not claim to have a solution to these reproducibility issues but any solution developed by the community will be applicable here. To quantify the reproducibility of our for the different problems, we provide the performance of the learned policies in the appendix when re-running the same experiments multiple times. Binary search BID26) is a standard algorithm for finding the location l x of a target value x in a sorted array A = {a 0, a 1, . . ., a N −1} of size N. Binary search has a worst case runtime complexity of log 2 (N) steps when no further knowledge about the distribution of data is available. Knowing more about the distribution of the data can help to reduce expected runtime. For example, if the array values follow a uniform distribution, the location of x can be approximated using linear interpolation l x ≈ (N − 1)(x − a 0)/(a N −1 − a 0). We show how PVars can be used to speed up binary search by learning to estimate the position l x for a more general case. The simplest way of using a PVar is to directly estimate the location l x and incentivize the search to do so in as few steps as possible by penalizing each step by the same negative reward (listing 1). At each step, the PVar observes the values a L, a R at both ends of the search interval and the target x. The PVar output q is used as the relative position of the next read index m, such that m = qL + (1 − q)R.In order to give a stronger learning signal to the model, the developer can incorporate problemspecific knowledge into the reward function or into how the PVar is used. One way to shape the reward is to account for problem reduction. For binary search, reducing the size of the remaining search space will speed up the search proportionally and should be rewarded accordingly. By replacing the step-counting reward in listing 1 (line 9) with the search range reduction DISPLAYFORM0 we directly reward reducing the size of the search space. By shaping the reward like this, we are able to attribute the feedback signal to the current prediction and to reduce the problem from RL to contextual bandit (which we implement by using a discount factor of 0).Alternatively we can change the way the prediction is used to cast the problem in a way that the PVar learns faster and is unable to predict very bad values. For many algorithms (including binary search) it is possible to predict a combination of (or choice among) several existing heuristics rather than predicting the value directly. We use two heuristics: (a) vanilla binary search which splits the search range {a L, . . ., a R} into two equally large parts using the split location l v = (L + R)/2, and (b) interpolation search which interpolates the split location as DISPLAYFORM1 We then use the value q of the PVar to mix between these heuristics to get the predicted split position l q = ql v + (1 − q)l i. Since in practice both of these heuristics work well on many distributions, any point in between will also work well. This reduces the risk for the PVar to pick a value that is really bad which in turn helps learning. A disadvantage is that it's impossible to find the optimal strategy with values outside of the interval between l v and l i. To evaluate our approaches we are using a test environment where in each episode, we sample an array of 5000 elements from a randomly chosen distribution (uniform, triangular, normal, pareto, power, gamma and chisquare), sort it, scale to [−10 4, 10 4] and search for a random element. FIG3 shows the for the different variants of binary search using a PVar and compares them to the vanilla binary search baseline. The show that the simplest case (pink line) where we directly predict the relative position with the simple reward and without using an initial function performs poorly initially but then becomes nearly as good as the baseline (cumulative regret becomes nearly constant after an initial bad period). The next case (yellow line) has an identical setup but we are using the initial function and we see that the initial regret is substantially smaller. By using the shaped reward (blue line), the PVar is able to learn the behavior of the baseline quickly. Both approaches that are mixing the heuristics (green and red lines) significantly outperform the baselines. In the appendix TAB1 we give details about when each of the different variants of using a PVar in binary search reaches break-even. QuickSort BID12 sorts an array in-place by partitioning it into two sets (smaller/larger than the pivot) recursively until the array is fully sorted. QuickSort is one of the most commonly used sorting algorithms where many heuristics have been proposed to choose the pivot element. While the average time complexity of QuickSort is θ(N log(N)), a worst case time complexity of O(N 2) can happen when the pivot elements are badly chosen. The optimal choice for a pivot is the median of the range, which splits it into two parts of equal size. To improve QuickSort using a PVar we aim at tuning the pivot selection heuristic. To allow for sorting arbitrary types, we decided to use the PVar to determine the number of elements that are sampled from the array to be sorted and then pick the median from these samples as the pivot (listing 2). As feedback signal for a recursion step we use an estimate of its impact on the computational cost ∆c. DISPLAYFORM0 where n is the size of the array, a and b are the sizes of the partitions with n = a + b and c pivot = c median +c partition is the cost to compute the median of the samples and to partition the array. ∆c recursive takes into account how close the current partition is to the ideal case (median). The cost is a weighted sum of number of reads, writes, and comparisons. Similar to the shaped reward in binary search, this reward allows us to reduce the RL problem to a contextual bandit problem and we use a discount of 0.For evaluation we are using a test environment where we sort randomly shuffled arrays. Results of the experiments are presented in fig. 3. It shows that the learned method outperforms all baseline heuristics within less than 100 episodes.' Vanilla' corresponds to a standard QuickSort implementation that picks one pivot at random in each step.'Random3' and'Random9' sample 3 and 9 random elements respectively and use the median of these as pivots.' Adaptive' uses the median of max(1, log 2 (n) − 1 ) randomly sampled elements as pivot when partitioning a range of size n. It uses more samples at for larger arrays, leading to a better approximation of the median, and thus to faster problem size reduction. Fig. 4 shows that the PVar learns a non-trivial policy. The PVar learns to select more samples at larger array sizes which is similar to the behavior that we hand-coded in the adaptive baseline but in this case no manual heuristic engineering was necessary and a better policy was learned. Also, note that a PVar-based method is able to adapt to changing environments which is not the case for engineered heuristics. One surprising is that the PVar prefers 13 over 15 samples at large array sizes. We hypothesize this happens because relatively few examples of large arrays are seen during training (one per episode, while arrays of smaller sizes are seen multiple times per episode). Caches are a commonly used component to speed up computing systems. They use a cache replacement policy (CRP) to determine which element to evict when the cache is full and a new element needs to be stored. Probably the most popular CRP is the least recently used (LRU) heuristic which evicts the element with the oldest access timestamp. A number of approaches have been proposed to improve cache performance using machine learning (see sec. 5). We propose two different approaches how PVars can be used in a CRP to improve cache performance. Discrete (listing 3): A PVar directly predicts which element to evict or chooses not to evict at all (by predicting an invalid index). That is, the PVar learns to become a CRP itself. While this is the simplest way to use a PVar, it makes it more difficult to learn a CRP better than LRU (in fact, even learning to be on par with LRU is non-trivial in this setting).Continuous (listing 4): A PVar is used to enhance LRU by predicting an offset to the last access timestamp. Here, the PVar learns which items to keep in the cache longer and which items to evict sooner. In this case it becomes trivial to be as good as LRU by predicting a zero offset. The PVar value in (−1, 1) is scaled to get a reasonable value range for the offsets. It is also possible to choose not to store the element by predicting a sufficiently negative score. In both approaches the feedback given to the PVar is whether an item was found in the cache (+1) or not (−1). In the discrete approach we also give a reward of −1 if the eviction actually takes place. In our implementation the observations are the history of accesses, memory contents, and evicted elements. The PVar can observe keys as a categorical input or features of the keys. Observing keys as categorical input allows to avoid feature engineering and enables directly learning the properties of particular keys (e.g. which keys are accessed the most) but makes it difficult to deal with rare and unseen keys. To handle keys as input we train an embedding layer shared between the actor and critic networks (FIG6).As features of the keys we observe historical frequencies computed over a window of fixed size. This approach requires more effort from the developer to implement such features, but pays off with better performance and the fact that the model does not rely on particular key values. We experiment with three combinations of these options: discrete caches observing keys, continuous caches observing keys, continuous caches observing frequencies. For evaluation we use a cache with size 10 and integer keys from 1 to 100. We use two synthetic access patterns of length 1000, sampled i.i.d. from a power law distribution with α = 0.1 and α = 0.5. FIG7 shows for the three variants of predicted caches, a standard LRU cache, and an oracle cache to give a theoretical, non-achievable, upper bound on the performance. We look at the hit ratio without exploration to understand the potential performance of the model once learning has converged. However, cumulative regret is still reported under exploration noise. Both implementations that work directly on key embeddings learn to behave similar to the LRU baseline without exploration (comparable hit ratio). However, the continuous variant pays a higher penalty for exploration (higher cumulative regret). Note that this means that the continuous variant learned to predict constant offsets (which is trivial), however the discrete implementation actually learned to become an LRU CRP which is non-trivial. The continuous implementation with frequencies quickly outperforms the LRU baseline, making the cost/benefit worthwhile long-term (negative cumulative regret after a few hundred episodes). Similar to our proposed interface, probabilistic programming BID5 introduces interfaces which simplify the developer complexity when working with statistical models and conditioning variable values on run-time observations. In contrast to PVars, the introduced interfaces are specialized on working with distributions and graphical models. In the space of approximate computing, BID22 propose a programming interface for approximate computation. While similar in spirit, this work does not explicitly target machine learning models. Similar in spirit to our approach is which proposes to incorporate neural models into database systems by replacing existing index structures with neural models that can be both faster and smaller. PVars in contrast aim not to replace existing data structures or algorithms but transparently integrate with standard algorithms and systems. PVars are general enough to be used to improve the heuristics in algorithms (as done here), to optimize database systems (similar to), or to simply replace an arbitrarily chosen constant. Another approach that is similar to PVars is Spiral BID2 but it is far more limited in scope than PVars in that it aims to predict boolean values only and relies on ground truth data for model building. Similarly, a number of papers apply machine learning to algorithmic problems, e.g. Neural Turing Machines BID7 aims to build a full neural model for program execution. BID14; BID15 BID1 propose end-to-end ML approaches to combinatorial optimization problems. In contrast to PVars these approaches replace the existing methods with an ML-system. These are a good demonstration of the inversion of control mentioned above: using ML requires to give full control to the ML system. There are a few approaches that are related to our use of the initial function, however most common problems where RL is applied do not have a good initial function. Generally related is the idea of imitation learning BID13 where the agent aims to replicate the behavior of an expert. Typically the amount of training data created by an expert is very limited. Based on imitation learning is the idea to use previously trained agents to kickstart the learning of a new model BID23 where the authors concurrently use a teacher and a student model and encourage the student model to learn from the teacher through an auxiliary loss that is decreased over time as the student becomes better. In some applications it may be possible to obtain additional training data from experts from other sources, e.g. BID11 BID0 leverage YouTube videos of gameplay to increase training speed of their agents. These approaches work well in cases where it is possible to leverage external data sources. Caches are an interesting application area where multiple teams have shown in the past that ML can improve cache performance BID27 BID20 BID8 BID21 BID6. In contrast to our approach, all ML models work on caches specifically and build task-dependent models that do not generalize to other tasks. Algorithm selection has been an approach to apply RL for improving sorting algorithms BID17. Search algorithms have also been improved using genetic algorithms to tweak code optimization BID18. We have introduced a new programming concept called a predicted variable (PVar) aiming to make it easier for developers to use machine learning from their existing code. Contrary to other approaches, PVars can easily be integrated and hand full control to the developer over how ML models are used and trained. PVars bridge the chasm between the traditional approaches of software systems building and machine learning modeling and thus allow for the developer to focus on refining their algorithm and metrics rather than working on building pipelines to incorporate machine learning. PVars achieve this by reusing the existing concept of variables in programming in a novel way where the value of the variable is determined using machine learning. PVar observes information about its context and receives feedback about the quality of predictions instead of being assigned a value directly. We have studied the feasibility of PVars in three algorithmic problems. For each we show how easy PVars can be incorporated, how performance improves in comparison to not using a PVar at all. Specifically, through our experiments we highlight both advantages and disadvantages that reinforcement learning brings when used as a solution for a generic interface as PVars. Note that we do not claim to have the best possible machine learning model for each of these problems but our contribution lies in building a framework that allows for using ML easily, spreading its use, and improving the performance in places where machine learning would not have been used otherwise. PVars are applicable to more general problems across a large variety of domains from system optimization to user modelling. Our current implementation of PVars is built on standard RL methods but other ML methods such as supervised learning are in scope as well if the problem is appropriate. In this paper we barely scratch the surface of the new opportunities created with PVars. The current rate of progress in ML will enable better and wider applicability of PVars to new applications. We hope that PVars will inspire the use of ML in places where it has not been considered before. We plan to release the code to reproduce the in this paper. Further, we hope to make PVars a standard feature in C++29, Python 4, and Java 12.;) TAB1 gives details about when each of the different variants of using a PVar in binary search reaches break-even. The numbers indicate how many episodes it takes for the cumulative regret to become permanently negative, which means that for any additional evaluations after that point the user has a net benefit from using a PVar compared to not using ML at all. The table shows that reward shaping and using the predictions smartly improve performance but it also shows that even simple methods are able to give improvements. Note, that no model outperforms interpolation search on a uniform distribution as it is the best approximation for this distribution. We do not claim to have solved reinforcement learning reproducibility and throughout our experiments we are facing the same issues as the larger community. The core aspect of the PVars framework is the ability to rely on the initial function or algorithmic formulation to limit the costs of a poorly performing learned policy. We illustrate this by looking at metrics over 100 restarts of the different experiments and highlight that, while some experiments for some problems are more reproducible than others, we do not perform worse than the initial function provided by the developer. The design construct specific to PVars and what distinguishes it from standard Reinforcement Learning is that it is applied in software control where often developers are able to provide safe initial functions or write the algorithm in a way that limits the cost of a poorly performing policy. To quantify the reproducibility, we ran the experiment from Sec. 4.2 120× and report the cumulative regret per episode (average number of extra steps per search episode) compared to vanilla binary search. On average, the cumulative regret is: -1.59 @5K (-2.20 @50K). The break-even point is reached in 85% of the cases, and within an average of 1965 episodes. The performance breakdown by percentile, and the number of steps at which the break-even point is reached are referenced in table 2. To quantify the reproducibility, we ran the experiment described in Sec. 4.3 115× and report the cumulative regret per episode (average number of extra operations, as read=write=1, compare=0.5 per sort) compared to vanilla QuickSort. On average, the cumulative regret per episode is -913 @1K (-1064 @10K) on a total operation cost of 25.1K per sort. The break-even point is reached in 94% of the cases, and in an average after 368 episodes. The performance breakdown by percentile, and the number of steps at which the break-even point is reached are referenced in table 3. In order to quantify the reproducibility of our experiments we ran 100 times the same experiment illustrated in sec. 4.4 and we report the performance of the learned cache policy using predicted variables when compared with LRU heuristic, by looking at the cumulative regret metric after 20000 episodes. We break down the cumulative regret by percentiles in table 4. When counting the number of runs for which there exists an episode where the cumulative regret is strictly negative until the end, we note that this happens for 26% of the runs. For 60% of the runs the cumulative regret does not become positive, meaning that using the learned cache policy is at least as good as using the LRU heuristic. This leaves us with 14% of the runs ing in strictly worse performance than relying on the LRU heuristic. In table 5 we provide the hyperparameters used for different experiments in order to ease reproducibility of our work. Together with Sec. B this details our entire experimental and setup.

Represent the short summary for finding the corresponding scientific article (long)

We present Predicted Variables, an approach to making machine learning a first class citizen in programming languages.

Represent the TLDR for finding the corresponding scientific article

We check DNN models for catastrophic forgetting using a new evaluation scheme that reflects typical application conditions, with surprising results.