Question: Represent the article for finding a single-sentence TLDR

Question: The recent direction of unpaired image-to-image translation is on one hand very exciting as it alleviates the big burden in obtaining label-intensive pixel-to-pixel supervision, but it is on the other hand not fully satisfactory due to the presence of artifacts and degenerated transformations. In this paper, we take a manifold view of the problem by introducing a smoothness term over the sample graph to attain harmonic functions to enforce consistent mappings during the translation. We develop HarmonicGAN to learn bi-directional translations between the source and the target domains. With the help of similarity-consistency, the inherent self-consistency property of samples can be maintained. Distance metrics defined on two types of features including histogram and CNN are exploited. Under an identical problem setting as CycleGAN, without additional manual inputs and only at a small training-time cost, HarmonicGAN demonstrates a significant qualitative and quantitative improvement over the state of the art, as well as improved interpretability. We show experimental in a number of applications including medical imaging, object transfiguration, and semantic labeling. We outperform the competing methods in all tasks, and for a medical imaging task in particular our method turns CycleGAN from a failure to a success, halving the mean-squared error, and generating images that radiologists prefer over competing methods in 95% of cases. Image-to-image translation BID15 aims to learn a mapping from a source domain to a target domain. As a significant and challenging task in computer vision, image-to-image translation benefits many vision and graphics tasks, such as realistic image synthesis BID15 BID41, medical image generation BID39 BID9, and domain adaptation BID13. Given a pair of training images with detailed pixel-to-pixel correspondences between the source and the target, image-to-image translation can be cast as a regression problem using e.g. Fully Convolutional Neural Networks (FCNs) BID23 by minimizing e.g. the per-pixel prediction loss. Recently, approaches using rich generative models based on Generative Adaptive Networks (GANs) BID11 BID27 BID0 have achieved astonishing success. The main benefit of introducing GANs BID11 to image-to-image translation BID15 is to attain additional image-level (often through patches) feedback about the overall quality of the translation, and information which is not directly accessible through the per-pixel regression objective. The method by BID15 is able to generate high-quality images, but it requires paired training data which is difficult to collect and often does not exist. To perform translation without paired data, circularity-based approaches BID41 BID17 BID37 have been proposed to learn translations from a set to another set, using a circularity constraint to establish relationships between the source and target domains and forcing the generated from a sample in the source domain to map back and generate the original sample. The original image-to-image translation problem BID15 ) is supervised in pixel-level, whereas the unpaired image-to-image translation task BID41 ) is considered unsupervised, with pixel-level supervision absent but with adversarial supervision at the image-level (in the target domain) present. By using a cycled regression for the pixel-level prediction (source→target→source) plus a term for the adversarial difference between the transferred images and the target images, CycleGAN is able to successfully, in many cases, train a translation model without paired source→target supervision. However, lacking a mechanism to enforce regularity in the translation creates problems like in Fig To combat the above issue, in this paper we look at the problem of unpaired image-to-image translation from a manifold learning perspective BID33 BID28. Intuitively, the problem can be alleviated by introducing a regularization term in the translation, encouraging similar contents (based on textures or semantics) in the same image to undergo similar translations/transformations. A common principle in manifold learning is to preserve local distances after the unfolding: forcing neighboring (similar) samples in the original space to be neighbors in the new space. The same principle has been applied to graph-based semisupervised learning BID44 where harmonic functions with graph Laplacians BID45 BID2 are used to obtain regularized labels of unlabeled data points. During the translation/transformation, some domain-specific attributes are changed, such as the colors, texture, and semantics of certain image regions. Although there is no supervised information for these changes, certain consistency during the transformation is desirable, meaning that for image contents similar in the source space should also be similar in the target space. Inspired by graphbased semi-supervised learning BID45 BID44, we introduce smoothness terms to unpaired image-to-image translation BID41 by providing a stronger regularization for the translation/transformation between the source and target domains, aiming to exploit the "manifold structure" of the source and target domains. For a pair of similar samples (two different locations in an image; one can think of them as two patches although the receptive fields of CNN are quite large), we add the smoothness term to minimize a weighted distance of the corresponding locations in the target image. Note that two spatially distant samples might be neighbors in the feature space. We name our algorithm HarmonicGAN as it behaves harmonically along with the circularity and adversarial constraints to learn a pair of dual translations between the source and target domains, as shown in FIG0. Distance metrics defined on two alternative features are adopted: a low-level soft RGB histograms; and CNN (VGG) features with pre-trained semantics. We conduct experiments in a number of applications, showing that in each of them our method outperforms existing methods quantitatively, qualitatively, and with user studies. For a medical imaging task BID6 that was recently calling attention to a major CycleGAN failure case (learning to accidentally add/remove tumors in an MRI image translation task), our proposed method provides a large improvement over CycleGAN, halving the mean-squared error, and generating images that radiologists prefer over competing methods in 95% of cases. CONTRIBUTIONS 1. We introduce smooth regularization over the graph for unpaired image-to-image translation to attain harmonic translations.2. When building an end-to-end learning pipeline, we adopt two alternative types of feature measures to compute the weight matrix for the graph Laplacian, one based on a soft histogram BID35 and another based on semantic CNN (VGG) features BID31.3. We show that this method in significantly improved consistency for transformations. With experiments on multiple translation tasks, we demonstrate that HarmonicGAN outperforms the state-of-the-art. As discussed in the introduction, the general image-to-image translation task in the deep learning era was pioneered by BID15, but there are prior works such as image analogies BID12 ) that aim at a similar goal, along with other exemplar-based methods BID10 BID8 BID1. After BID15, a series of other works have also exploited pixel-level reconstruction constraints to build connections between the source and target domains BID34. The image-to-image translation framework BID15 ) is very powerful but it requires a sufficient amount of training data with paired source to target images, which are often laborious to obtain in the general tasks such as labeling BID23, synthesis BID5, and style transfer BID14.Unpaired image-to-image translation frameworks BID41 BID22 BID29 BID17 such as CycleGAN remove the requirement of having detailed pixellevel supervision. In CycleGAN this is achieved by enforcing a bi-directional prediction from source to target and target back to source, with an adversarial penalty in the translated images in the target domain. Similar unsupervised circularity-based approaches BID17 BID37 have also been developed. The CycleGAN family models BID41 point to an exciting direction of unsupervised approaches but they also create artifacts in many applications. As shown in FIG1, one reason for this is that the circularity constraint in CycleGAN lacks the straightforward description of the target domain, so it may change the inherent properties of the original samples and generate unexpected which are inconsistent at different image locations. These failures have been prominently explored in recent works, showing that CycleGAN BID41 ) may add or remove tumors accidentally in cross-modal medical image synthesis BID6, and that in the task of natural image transfiguration, e.g. from a horse to zebra, regions in the may also be translated into a zebra-like texture ) (see FIG0).Here we propose HarmonicGAN that introduces a smoothness term into the CycleGAN framework to enforce a regularized translation, enforcing similar image content in the source space to also be similar in the target space. We follow the general design principle in manifold learning BID33 BID28 and the development of harmonic functions in the graph-based semi-supervised learning literature BID45 BID2 BID44 BID36. There has been previous work, DistanceGAN BID3, in which distance preservation was also implemented. However, DistanceGAN differs from HarmonicGAN in motivation, formulation, implementation, and performance. The primary motivation of DistanceGAN demonstrates an alternative loss term for the per-pixel difference in CycleGAN.In HarmonicGAN, we observe that the cycled per-pixel loss is effective and we aim to make the translation harmonic by introducing additional regularization. The smoothness term acts as a graph Laplacian imposed on all pairs of samples (using random samples in implementation). In the experimental , we show that the artifacts in CycleGAN are still present in DistanceGAN, whereas HarmonicGAN provides a significant boost to the performance of CycleGAN.In addition, it is worth mentioning that the smoothness term proposed here is quite different from the binary term used in the Conditional Random Fields literature BID20 BID19, either fully supervised BID4 BID40 or weakly-supervised BID32 BID21. The two differ in output space (multi-class label vs. highdimensional features), mathematical formulation (a joint conditional probably for the neighboring labels vs. a Laplacian function over the graph), application domain (image labeling vs. image translation), effectiveness (boundary smoothing vs. manifold structure preserving), and FORMULA7 the role in the overall algorithm (post-processing effect with relatively small improvement vs. large-area error correction). Following the basic formulation in CycleGAN BID41, for the source domain X and the target domain Y, we consider unpaired training samples {x k} N k=1 where x k ∈ X, and DISPLAYFORM0 where y k ∈ Y. The goal of image-to-image translation is to learn a pair of dual mappings, including forward mapping G: X → Y and backward mapping F: Y → X. Two discriminators D X and D Y are adopted in BID41 to distinguish between real images and generated images. In particular, the discriminator D X aims to distinguish real image {x} from the generated image DISPLAYFORM1 Therefore, the objective of adversarial constraint is applied in both source and target domains, expressed in BID41 as: DISPLAYFORM2 and DISPLAYFORM3 For notational simplicity, we denote the GAN loss as DISPLAYFORM4 Since the data in the two domains are unpaired, a circularity constraint is introduced in BID41 to establish relationships between X and Y. The circularity constraint enforces that G and F are a pair of inverse mappings, and that the translated sample can be mapped back to the original sample. The circularity constraint contains consistencies in two aspects: the forward cycle DISPLAYFORM5 Thus, the circularity constraint is formulated as BID41: DISPLAYFORM6 Here we rewrite the overall objective in BID41 to minimize as: DISPLAYFORM7 where the weights λ GAN and λ cyc control the importance of the corresponding objectives. The full objective of circularity-based approach contains adversarial constraints and a circularity constraint. The adversarial constraints ensure the generated samples are in the distribution of the source or target domain, but ignore the relationship between the input and output of the forward or backward translations. The circularity constraint establishes connections between the source and target domain by forcing the forward and backward translations to be the inverse of each other. However, CycleGAN has limitations: as shown in FIG1, the circular projection might perfectly match the input, and the translated image might look very well like a real one, but the translated image may not maintain the inherent property of the input and contain a large artifact that is not connected to the input. Here we propose a smoothness term to enforce a stronger correlation between the source and target domains that focuses on providing similarity-consistency between image patches during the translation. The smoothness term defines a graph Laplacian with the minimal value achieved as a harmonic function. We define the set consisting of individual image patches as the nodes of the graph G. x i is referred to as the feature vector of the i-th image patch in x ∈ X. For the image set X, we define the set that consists of individual samples (image patches) of source image set X as S = {x(i), i = 1..M } where M is the total number of the samples/patches. An affinity measure (similarity) computed on image patch x(i) and image patch x(j), w ij (X) (a scalar), defines the edge on the graph G of S. The smoothness term acts as a graph Laplacian imposed on all pairs of image patches. Therefore, we define a smoothness term over the graph as DISPLAYFORM0 where BID45 defines the affinity between two patches x(i) and x(j) based on their distances (e.g. measured on histogram or CNN features). Dist[G( y)(i), G(y)(j)] defines the distance between two image patches after translation at the same locations. In implementation, we first normalize the features to the scale of and then use the L1 distance of normalized features as the Dist function (for both histogram and CNN features). Similarly, we define a smoothness term for the backward part as DISPLAYFORM1 DISPLAYFORM2 The combined loss for the smoothness thus becomes smoothness term provides a stronger similarity-consistency between patches to maintain inherent properties of the images. DISPLAYFORM3 Combining Eqn. FORMULA7 and Eqn., the overall objective for our proposed HarmonicGAN under the smoothness constraint becomes DISPLAYFORM4 Similar to the graph-based semi-supervised learning definition BID45 BID44, the solution to Eqn. leads to a harmonic function. The optimization process during training obtains: DISPLAYFORM5 The effectiveness of the smoothness term of Eqn. FORMULA11 is evident. In Fig. 4, we show (using t-SNE BID24) that the local neighborhood structure is being preserved by HarmonicGAN, whereas CycleGAN in two similar patches being far apart after translation. In the smoothness constraint, the similarity of a pair of patches is measured on the features for each patch (sample point). All the patches in an image form a graph. Here we adopt two types of features: a low-level soft histogram, and pre-trained CNN (VGG) features that carry semantic information. Soft histogram features are lightweight and easy to implement but without much semantic information; VGG requires an additional CNN network but carries more semantics. We first design a weight matrix based on simple low-level RGB histograms. To make the end-to-end learning system work, it is crucial to make the computation of gradient in the histograms derivable. We adopt a soft histogram representation proposed in BID35 but fix the means and the bin size. This histogram representation is differentiable and its gradient is back-propagateable. This soft histogram function contains a family of linear basis functions ψ b, b = 1,..., B, where B is the number of bins in the histogram. As x i represents the i-th patch in image domain X, for each pixel j in x i, ψ b (x i (j)) represents pixel j voting for the b-th bin, expressed as: DISPLAYFORM0 where µ b and w b are the center and width of the b-th bin. The representation of x i in the RGB space is the linear combination of linear basis functions on all the pixels in x i, expressed as: DISPLAYFORM1 where φ h is the RGB histogram feature, b is the index of dimension of the RGB histogram representation, and j represents any pixel in the patch x i. The RGB histogram representation φ h (X, i) of x i is a B-dimensional vector. For some domains we instead use semantic features to acquire higher-level representations of patches. The semantic representations are extracted from a pre-trained Convolutional Neural Network (CNN). The CNN encodes semantically relevant features from training on a large-scale dataset. It extracts semantic information of local patches in the image through multiple pooling or stride operators. Each point in the feature maps of the CNN is a semantic descriptor of the corresponding image patch. Additionally, the semantic features learned from the CNN are differentiable and the CNN can be integrated into HarmonicGAN and be trained end-to-end. We instantiate the semantic feature φ s as a pre-trained CNN model e.g. VGGNet BID30. In implementation, we select the layer 4 3 after ReLU from VGG-16 network for computing the semantic features. We evaluate the proposed method on three different applications: medical imaging, semantic labeling, and object transfiguration. We compare against several unpaired image-to-image translation methods: CycleGAN BID41, DiscoGAN BID17, DistanceGAN BID3, and UNIT BID22. We also provide two user studies as well as qualitative . The appendix provides additional and analysis. Medical imaging. This task evaluates cross-modal medical image synthesis, Flair ↔ T1. The models are trained on the BRATS dataset BID26 which contains paired MRI data to allow quantitative evaluation. Similar to the previous work BID6, we use a training set of 1400 image slices (50% healthy and 50% tumors) and a test set of 300, and use their unpaired training scenario. We adopt the Mean Absolute Error (MAE) and the Mean Squared Error (MSE) between the generated images and the real images to evaluate the reconstruction errors, and further use the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) to evaluate the reconstruction quality of generated images. Semantic labeling. We also test our method on the labels ↔ photos task using the Cityscapes dataset BID7 under the unpaired setting as in the original CycleGAN paper. For quantitative evaluation, in line with previous work, for labels → photos we adopt the "FCN score" BID15, which evaluates how interpretable the generated photos are according to a semantic segmentation algorithm. For photos → labels, we use the standard segmentation metrics, including per-pixel accuracy, per-class accuracy, and mean class Intersection-Over-Union (Class IOU).Object transfiguration. Finally, we test our method on the horse ↔ zebra task using the standard CycleGAN dataset (2401 training images, 260 test images). This task does not have a quantitative evaluation measure, so we instead provide a user study together with qualitative . We apply the proposed smoothness term on the framework of CycleGAN BID41. Similar with CycleGAN, we adopt the architecture of BID16 as the generator and the PatchGAN BID15 as the discriminator. The log likelihood objective in the original GAN is replaced with a least-squared loss BID25 for more stable training. We resize the input images to the size of 256 × 256. For the histogram feature, we equally split the RGB range of to 16 bins, each with a range of 16. Images are divided into non-overlapping patches of 8 × 8 and the histogram feature is computed on each patch. For the semantic feature, we adopt a VGG network pre-trained on ImageNet to obtain semantic features. We select the feature map of layer relu4 3 in VGG. The loss weights are set as λ GAN = λ Smooth = 1, λ cyc = 10. Following CycleGAN, we adopt the Adam optimizer BID18 with a learning rate of 0.0002. The learning rate is fixed for the first 100 epochs and linearly decayed to zero over the next 100 epochs. Medical imaging. Table 1 shows the reconstruction performance on medical image synthesis, Flair ↔ T1. The proposed method yields a large improvement over CycleGAN, showing lower MAE and MSE reconstruction losses, and higher PSNR and SSIM reconstruction scores, highlighting the significance of the proposed smoothness regularization. HarmonicGAN based on histogram and VGG features shows similar performance; the reconstruction losses of histogram-based HarmonicGAN are slightly lower than the VGG-based one in Flair → T1, while they are slightly higher in T1 → Flair, indicating that both low-level RGB values and high-level CNN features can represent the inherent property of medical images well and help to maintain the smoothness-consistency of samples. Table 1: Reconstruction evaluation of cross-modal medical image synthesis on the BRATS dataset. Semantic labeling. We report semantic labeling in TAB1. The proposed method using VGG features yields a 3% improvement in Pixel Accuracy in translation scores for photo ↔ label and also shows stable improvements in other metrics, clearly outperforming all competing methods. The performance using a histogram is slightly lower than CycleGAN; we hypothesize that the reason is that the objects in photos have a large intra-class variance and inter-class similarity in appearance, e.g. cars have different colors, while vegetation and terrain have similar colors, thus the regularization of the RGB histogram is not appropriate to extract the inherent property of photos. DISPLAYFORM0 Medical imaging. We randomly selected 100 images from BRATS test set. For each image, we showed one radiologist the real ground truth image, followed by images generated by CycleGAN, DistanceGAN and HarmonicGAN (different order for each image set to avoid bias). The radiologist was told to evaluate similarity by how likely they would lead to the same clinical diagnosis, and was asked to rate similarity of the generation methods on a Likert scale from 1 to 5 (1 is not similar at all, 5 is exactly same). Results are in shown in TAB2. In 95% of cases, the radiologist preferred images generated by our method over the competing methods, and the average Likert score was 4.00 compared to 1.68 for CycleGAN, confirming that our generated images are significantly better. This is significant as it confirms that we solve the issue presented in a recent paper BID6 showing that CycleGAN can learn to accidentally add/remove tumors in images. Object transfiguration. We evaluate our algorithm on horse ↔ zebra with a human perceptual study. We randomly selected 50 images from the horse2zebra test set and showed the input images and three generated images from CycleGAN, DistanceGAN and HarmonicGAN (with generated images in random order). 10 participants were asked to score the generated images on a Likert scale from 1 to 5 (as above). As shown in TAB3, the participants give the highest score to the proposed method (in 72% of cases), significantly more often than CycleGAN (in 28% of cases). Additionally, the average Likert score of our method was 3.60, outperforming 3.16 of CycleGAN and 1.08 of DistanceGAN, indicating that our method generates better . Medical imaging. Object transfiguration. FIG4 shows a qualitative comparison of our method on the horse ↔ zebra task. We observe that we correct several problems in CycleGAN, including not changing the and performing more complete transformations. More and analysis are shown in FIG6 and FIG0. We introduce a smoothness term over the sample graph to enforce smoothness-consistency between the source and target domains. We have shown that by introducing additional regularization to enforce consistent mappings during the image-to-image translation, the inherent self-consistency property of samples can be maintained. Through a set of quantitative, qualitative and user studies, we have demonstrated that this in a significant improvement over the current state-of-the-art methods in a number of applications including medical imaging, object transfiguration, and semantic labeling. In a medical imaging task in particular our method provides a very significant improvement over CycleGAN. They show different motivations and formulations. The distance constraint aims to preserve the distance between samples in the mapping in a direct way, so it minimizes the expectation of differences between distances in two domains. The distance constraint in DistanceGAN is not doing a graph-based Laplacian to explicitly enforce smoothness. In contrast, the smoothness constraint is designed from a graph Laplacian to build the similarity-consistency between image patches. Thus, the smoothness constraint uses the affinity between two patches as weight to measure the similarityconsistency between two domains. The whole idea is based on manifold learning. The smoothness term defines a Laplacian ∆ = D − W, where W is our weight matrix and D is a diagonal matrix with D i = j w ij, thus, the smoothness term defines a graph Laplacian with the minimal value achieved as a harmonic function. They are different in implementation. The smoothness constraint in HarmonicGAN is computed on image patches while the distance constraint in DistanceGAN is computed on whole image samples. Therefore, the smoothness constraint is fine-grained compared to the distance constraint. Moreover, the distances in DistanceGAN is directly computed from the samples in each domain. They scale the distances with the precomputed means and stds of two domains to reduce the effect of the gap between two domains. Differently, the smoothness constraint in HarmonicGAN is measured on the features (Histogram or CNN features) of each patch, which maps samples in two domains into the same feature space and removes the gap between two domains. They show different . FIG5 shows the qualitative of CycleGAN, DistanceGAN and the proposed HarmonicGAN on the BRATS dataset. As shown in FIG5, the problem of randomly adding/removing tumors in the translation of CycleGAN is still present in the of Distance-GAN, while HarmonicGAN can correct the location of tumors. Table 1 shows the quantitative on the whole test set, which also yields the same . The of DistanceGAN on four metrics are even worse than CycleGAN, while HarmonicGAN yields a large improvement over CycleGAN. There are some fundamental differences between the CRF literature and our work. They differ in output space, mathematical formulation, application domain, effectiveness, and the role in the over-all algorithm. The similarity between CRF and HarmonicGAN lies the adoption of a regularization term: a binary term in the CRF case and a Laplacian term in HarmonicGAN.The smoothness term in HarmonicGAN is not about obtaining'smoother' images/labels in the translated domain, as seen in the experiments; instead, HarmonicGAN is about preserving the overall integrity of the translation itself for the image manifold. This is the main reason for the large improvement of HarmonicGAN over CycleGAN.To further demonstrate the difference of HarmonicGAN and CRF, we perform an experiment applying the pairwise regularization of CRFs to the CycleGAN framework. For each pixel of the generated image, we compute the unary term and binary term with its 8 neighbors, and then minimize the objective function of CRF. The are shown in TAB6. The pairwise regularization of CRF is unable to handle the problem of CycleGAN illustrated in FIG0. What's worse, using the pairwise regularization may over-smooth the boundary of generated images, which in extra artifacts. In contrast, HarmonicGAN aims at preserving similarity from the overall view of the image manifold, and can thus exploit similarity-consistency of the generated images, rather than over-smooth the boundary. human to a zebra-like texture. In contrast, HarmonicGAN does better in region and achieves an improvement in some regions of of the human (Putin's face), but it still fails on the human body. We hypothesize this is because the semantic features used by HarmonicGAN have not been trained on humans without a shirt. to apple, facade to label, label to facade, aerial to map, map to aerial, summer to winter, winter to summer.

Smooth regularization over sample graph for unpaired image-to-image translation results in significantly improved consistency

Represent the short summary for finding the corresponding scientific article (long)!

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Multiview stereo aims to reconstruct scene depth from images acquired by a camera under arbitrary motion. Recent methods address this problem through deep learning, which can utilize semantic cues to deal with challenges such as textureless and reflective regions. In this paper, we present a convolutional neural network called DPSNet (Deep Plane Sweep Network) whose design is inspired by best practices of traditional geometry-based approaches. Rather than directly estimating depth and/or optical flow correspondence from image pairs as done in many previous deep learning methods, DPSNet takes a plane sweep approach that involves building a cost volume from deep features using the plane sweep algorithm, regularizing the cost volume via a context-aware cost aggregation, and regressing the depth map from the cost volume. The cost volume is constructed using a differentiable warping process that allows for end-to-end training of the network. Through the effective incorporation of conventional multiview stereo concepts within a deep learning framework, DPSNet achieves state-of-the-art reconstruction on a variety of challenging datasets. Various image understanding tasks, such as semantic segmentation BID3 and human pose/action recognition BID29 BID33, have been shown to benefit from 3D scene information. A common approach to reconstructing 3D geometry is by multiview stereo, which infers depth based on point correspondences among a set of unstructured images BID10;. To solve for these correspondences, conventional techniques employ photometric consistency constraints on local image patches. Such photo-consistency constraints, though effective in many instances, can be unreliable in scenes containing textureless and reflective regions. Recently, convolutional neural networks (CNNs) have demonstrated some capacity to address this issue by leveraging semantic information inferred from the scene. The most promising of these methods employ a traditional stereo matching pipeline, which involves computation of matching cost volumes, cost aggregation, and disparity estimation BID5; BID19; BID14; BID0. Some are designed for binocular stereo BID31; BID19; BID0 and cannot readily be extended to multiple views. The CNN-based techniques for multiview processing BID5; BID14 both follow the plane-sweep approach, but require plane-sweep volumes as input to their networks. As a , they are not end-to-end systems that can be trained from input images to disparity maps. In this paper, we present Deep Plane Sweep Network (DPSNet), an end-to-end CNN framework for robust multiview stereo. In contrast to previous methods that employ the plane-sweep approach BID14; BID5, DPSNet fully models the plane-sweep process, including construction of plane-sweep cost volumes, within the network. This is made possible through the use of a differentiable warping module inspired by spatial transformer networks BID17 to build the cost volumes. With the proposed network, plane-sweep stereo can be learned in an end-to-end fashion. Additionally, we introduce a cost aggregation module based on local cost-volume filtering BID26 for context-aware refinement of each cost slice. Through this cost-volume regularization, the effects of unreliable matches scattered within the cost volume are reduced considerably. With this end-to-end network for plane-sweep stereo and the proposed cost aggregation, we obtain state-of-the-art over several standard datasets. Ablation studies indicate that each of these technical contributions leads to appreciable improvements in reconstruction accuracy. CNN-based depth estimation has been studied for stereo matching, depth from single images, and multiview stereo. Recent work in these areas are briefly reviewed in the following. Stereo matching Methods for stereo matching address the particular case of depth estimation where the input is a pair of rectified images captured by a stereo rig. Various network structures have been introduced for this problem. BID38 present a Siamese network structure to compute matching costs based on the similarity of two image patches. The estimated initial depth is then refined by traditional cost aggregation and refinement as post-processing. BID24 directly stack several convolution and deconvolution layers upon the matching costs and train the network to minimize the distance between the estimates and ground truth. BID1 propose a CNN that estimates initial disparity and then refines it using both prior and posterior feature consistency in an end-to-end manner. BID19 leverage geometric knowledge in building a cost volume from deep feature representations. It also enables learning of contextual information in a 3D volume and regresses disparity in an end-to-end manner. BID0 introduce a pyramid pooling module for incorporating global contextual information into image features and a stacked hourglass 3D CNN to extend the regional support of contextual information. Depth from single images Similar to these stereo matching approaches, single-image methods extract CNN features to infer scene depths and perform refinements to increase depth accuracy. The first of these methods was introduced by BID4, which demonstrated that CNN features could be utilized for depth inference. Later, BID22 combined a superpixel-based conditional random field (CRF) to a CNN to improve the quality of depth estimates from single images. To facilitate training, recent studies; BID23; BID36 present an end-to-end learning pipeline that utilizes the task of view synthesis as supervision for single-view depth and camera pose estimation. These systems consist of a depth network and a pose estimation network which simultaneously train on sequential images with a loss computed from images warped to nearby views using the estimated depth. View synthesis has similarly been used as supervision by warping between stereo image pairs BID7; BID8. In contrast to these single-image works which employ warping as a component of view synthesis for self-supervised learning, our network computes warps with respect to multiple depth planes to produce plane-sweep cost volumes both for training and at test time. The cost volumes undergo further processing in the form of cost aggregation and regularization to improve the robustness of depth estimates. Multi-view stereo In multi-view stereo, depth is inferred from multiple input images acquired from arbitrary viewpoints. To solve this problem, some methods recover camera motion between the unstructured images but are designed to handle only two views BID31;. The DeMoN system BID31 consists of encoder-decoder networks for optical flow, depth/motion estimation, and depth refinement. By alternating between estimating optical flow and depth/motion, the network is forced to use both images in estimating depth, rather than resorting to single-image inference. perform monocular visual odometry in an unsupervised manner. In the training step, the use of stereo images with extrinsic parameters allows 3D depth estimation to be estimated with metric scale. Among networks that can handle an arbitrary number of views, camera parameters are assumed to be known or estimated by conventional geometric methods. BID18 introduce an endto-end learning framework based on a viewpoint-dependent voxel representation which implicitly encodes images and camera parameters. The voxel representation restricts the scene resolution that can be processed in practice due to limitations in GPU memory. BID16 formulate a geometric relationship between optical flow and depth to refine the estimated scene geometry, but is designed for image sequences with a very small baseline, i.e., an image burst from a handheld camera. BID14 compute a set of plane-sweep volumes using calibrated pose data as input for the network, which then predicts an initial depth feature using an encoder-decoder network. In the depth prediction step, they concatenate a reference image feature to the decoder input as an intra-feature aggregation, and cost volumes from each of the input images are aggregated by max-pooling to gather information for the multiview matching. Its estimated depth map is refined using a conventional CRF. By contrast, our proposed DPSNet is developed to be trained end-to-end from input images to the depth map. Moreover, it leverages conventional multiview stereo concepts by incorporating context-aware cost aggregation. Finally, we would like to refer the reader to the concurrent work by BID35 that also adopts differential warping to construct a multi-scale cost volume, then refined an initial depth map guided by a reference image feature. Our work is independent of this concurrent effort. Moreover, we make distinct contributions: We focus on dense depth estimation for a reference image in an end-to-end learning manner, different from BID35 which reconstructs the full 3D of objects. Our cost volume is constructed by concatenating input feature maps, which enables inference of accurate depth maps even with only two-view matching. Our work refines every cost slice by applying context features of a reference image, which is beneficial for alleviating coarsely scattered unreliable matches such as for large textureless regions. Our Deep Plane Sweep Network (DPSNet) is inspired by traditional multiview stereo practices for dense depth estimation and consists of four parts: feature extraction, cost volume generation, cost aggregation and depth map regression. The overall framework is shown in FIG1. We first pass a reference image and target images through seven convolutional layers (3 × 3 filters except for the first layer, which has a 7 × 7 filter) to encode them, and extract hierarchical contextual information from these images using a spatial pyramid pooling (SPP) module BID12 with four fixed-size average pooling blocks (16 × 16, 8 × 8, 4 × 4, 2 × 2). The multi-scale features extracted by SPP have been shown to be effective in many visual perception tasks such as visual recognition BID12, scene parsing BID39 and stereo matching BID14. After upsampling the hierarchical contextual information to the same size as the original feature map, we concatenate all the feature maps and pass them through 2D convolutional layers. This process yields 32-channel feature representations for all the input images, which are next used in building cost volumes. We propose to generate cost volumes for the multiview images by adopting traditional plane sweep stereo Collins (images for each pixel. In a similar manner to traditional plane sweep stereo, we construct a cost volume from an input image pair. To reduce the effects of image noise, multiple images can be utilized by averaging cost volumes for other pairs. For this cost volume generation network, we first set the number of virtual planes perpendicular to the z-axis of the reference viewpoint and uniformly sample them in the inverse-depth space as follows: DISPLAYFORM0 where L is the total number of depth labels and d min is the minimum scene depth as specified by the user. Then, we warp all the paired features F i, (i = 1, .., N), where i is an index of viewpoints and N is the total number of input views, into the coordinates of the reference feature (of size W idth × Height × CHannel) using pre-computed intrinsics K and extrinsic parameters consisting of a rotation matrix R i and a translation matrix t i of the i th camera: DISPLAYFORM1 where u,ũ l are the homogeneous coordinates of a pixel in the reference view and the projected coordinates onto the paired view, respectively. F il (u) denotes the warped features of the paired image through the l th virtual plane. Unlike the traditional plane sweeping method which utilizes a distance metric, we use a concatenation of features in learning a representation and carry this through to the cost volume as proposed in BID19. We obtain a 4D volume (W ×H ×2CH ×L) by concatenating the reference image features and the warped image features for all of the depth labels. In Eq., we assume that all images are captured by the same camera, but it can be directly extended to images with different intrinsics. For the warping process, we use a spatial transformer network BID17 for all hypothesis planes, which does not require any learnable parameters. In TAB4, we find that concatenating features improves performance over the absolute difference of the features. Given the 4D volume 1, our DPSNet learns a cost volume generation of size W × H × L by using a series of 3D convolutions on the concatenated features. All of the convolutional layers consist of 3 × 3 × 3 filters and residual blocks. In the training step, we only use one paired image (while the other is the reference image) to obtain the cost volume. In the testing step, we can use any number of paired images (N ≥ 1) by averaging all of the cost volumes. The key idea of cost aggregation BID26 is to regularize the noisy cost volume through edge-preserving filtering BID11 volume filtering, we introduce a context-aware cost aggregation method in our end-to-end learning process. The context network takes each slice of the cost volume and the reference image features extracted from the previous step, and then outputs the refined cost slice. We run the same process for all the cost slices. The final cost volume is then obtained by adding the initial and residual volumes as shown in FIG2.Here, we use dilated convolutions in the context network for cost aggregation to better exploit contextual information; BID37. The context network consists of seven convolutional layers with 3 × 3 filters, where each layer has a different receptive field (1, 2, 4, 8, 16, 1, and 1). We jointly learn all the parameters, including those of the context network. All cost slices are processed with shared weights of the context network. Then, we upsample the cost volume, whose size is equal to the feature size, to the original size of the images via bilinear interpolation. We find that this leads to moderate performance improvement as shown in TAB4. We regress continuous depth values using the method proposed in BID19. The probability of each label l is calculated from the predicted cost c l via the softmax operation σ(·). The predicted labell is computed as the sum of each label l weighted by its probability. With the predicted label, the depth is calculated from the number of labels L and minimum scene depth d min as follows:d DISPLAYFORM0 We set L and d min to 64 and 0.5, respectively. Let θ be the set of all the learnable parameters in our network, which includes feature extraction, cost volume generation and cost aggregation (plane sweep and depth regression have no learnable parameters). Letd,d denote the predicted depth from the initial and refined cost volumes, respectively, and let d gt be the corresponding supervision signal. The training loss is then formulated as DISPLAYFORM0 where | · | H denotes the Huber norm, referred to as SmoothL1Loss in PyTorch. The weight value λ for depth from the initial cost volume is set to 0.7. In the training procedure, we use image sequences, ground-truth depth maps for reference images, and the provided camera poses from public datasets, namely SUN3D, RGBD, and Scenes11 2. We train our model from scratch for 1200K iterations in total. All models were trained end-to-end with the ADAM optimizer (β 1 = 0.9, β 2 = 0.999). We use a batch size of 16 and set the learning rate to 2e−4 for all iterations. The training is performed with a customized version of PyTorch on four NVIDIA 1080Ti GPUs, which usually takes four days. A forward pass of the proposed network takes about 0.5 seconds for 2-view matching and an additional 0.25 seconds for every new frame matched (640 × 480 image resolution). In our evaluations, we use common quantitative measures of depth quality: absolute relative error (Abs Rel), absolute relative inverse error (Abs R-Inv), absolute difference error (Abs diff), square relative error (Sq Rel), root mean square error and its log scale (RMSE and RMSE log) and inlier ratios (δ < 1.25 i where i ∈ {1, 2, 3}). All are standard metrics used in a public benchmark suite 3.For our comparisons, we choose state-of-the-art methods for traditional geometry-based multiview stereo (COLMAP), depth from unstructured two-view stereo (DeMoN) BID31 and CNN-based multiview stereo (DeepMVS) BID14. We estimate the depth maps from two unstructured views using the test sets in MVS, SUN3D, RGBD and Scenes11, as done for DeMoN 4. The are reported in Table 1. Our DPSNet provides the best performance on nearly all of the measures. Of particular note, DPSNet accurately recovers scene depth in homogeneous regions as well as along object boundaries as exhibited in FIG3. DeMoN generally produces good depth estimates but often fails to reconstruct scene details such as the keyboard (third row) and fine structures (first, second and fourth rows). By contrast, DPSNet estimates accurate depth maps at those regions because the differential feature warping penalizes inaccurate reconstructions, playing a role similar to the left-right consistency check that has been used in stereo matching BID7. The first and third rows of FIG3 exhibit problems of COLMAP and DeepMVS in handling textureless regions. DPSNet instead produces accurate , courtesy of the cost aggregation network. For a more balanced comparison, we adopt measures used in BID14 as additional evaluation criteria: completeness, which is the percentage of pixels whose errors are below a certain threshold. geometry error, taking the L1 distance between the estimated disparity and the ground truth. photometry error, which is the L1 distance between the reference image and warped image using the estimated disparity map. The for COLMAP, DeMoN and DeepMVS are directly reported from BID14 in TAB1. In this experiment, we use the ETH3D dataset on which all methods are not trained. Following BID35, we take 5 images with 1152 × 864 resolution and set 192 depth labels based on ground-truth depth to obtain optimal for MVSNet. For the DPSNet , we use 4 views with 810 × 540 resolution and set 64 labels whose range is determined by the minimum depth values of the ground truth. In TAB1, our DPSNet shows the best performance overall among the all the comparison methods, except for filtered COLMAP. Although filtered COLMAP achieves the best performance, its completeness is only 71% and its unfiltered version shows a significant performance drop in all error metrics. On the other hand, our DPSNet with 100% completeness shows promising on all measures. We note that our DPSNet has a different purpose compared to COLMAP and MVSNet. COLMAP and MVSNet are designed for full 3D reconstruction with an effective outlier rejection process, while DPSNet aims to estimate a dense depth map for a reference view. An extensive ablation study was conducted to examine the effects of different components on DPSNet performance. We summarize the in TAB4.Cost Volume Generation In TAB4 (a) and (e), we compare the use of cost volumes generated using the traditional absolute difference BID2 and using the concatenation of features from the reference image and warped image. The absolute difference is widely used for depth label selection via a winner-take-all strategy. However, we observe that feature concatenation provides better performance in our network than the absolute difference. A possible reason is that the CNN may learn to extract 3D scene information from the tensor of stacked features. The tensor is fed into the CNN to produce an effective feature for depth estimation, which is then passed through our cost aggregation network for the initial depth refinement. Cost Aggregation For our cost aggregation sub-network, we compare DPSNet with and without it in TAB4 (e) and (b), respectively. It is shown that including the proposed cost aggregation leads to significant performance improvements. Examples of depth map refinement with the cost aggregation are displayed in FIG5.Our cost aggregation is also compared to using a stacked hourglass to aggregate feature information along the depth dimension as well as the spatial dimensions as done recently for stereo matching BID0. Although the stacked hourglass is shown in For further analysis of cost aggregation, we display slices of 3D cost volumes after the softmax operation (in Eq. FORMULA2) that span depth labels and the rows of the images. The cost slices in Figure 6 (c), (d) show that our feature-guided cost aggregation regularizes noisy cost slices while preserving edges well. The cleaner cost profiles that ensue from the cost aggregation lead to clearer and edge-preserving depth regression . As mentioned in a recent study BID13, a cost profile that gives confident estimates should have a single, distinct minimum (or maximum), while an ambiguous profile has multiple local minima or multiple adjacent labels with similar costs, making it hard to exactly localize the global minimum. Based on two quantitative confidence measures BID13 on cost volumes in TAB5, the proposed aggregation improves the reliability of the correct match corresponding to the minimum cost. Depth Label Sampling In the plane sweep procedure, depth labels can be sampled in either the depth domain or the inverse depth domain, which provides denser sampling in areas closer to a camera. TAB4 (d) and (e) show that uniform depth label sampling in the inverse depth domain produces more accurate depth maps in general. We examine the performance of DPSNet with respect to the number of input images. As displayed in FIG7, a greater number of images yields better , since cost volume noise is reduced through averaging over more images, and more viewpoints help to provide features from areas unseen in other views. FIG7 shows that adding input views aids in distinguishing object boundaries. Note that the performance improvement plateaus when seven or more images are used. Rectified Stereo Pair CNNs-based stereo matching methods have similarity to DPSNet, but differ from it in that correspondences are obtained by shifting learned features in BID24; BID19 BID30. The purpose of this study is to show readers that not only descriptor shift but also plane sweeping can be applied to rectified stereo matching. We apply DPSNet on the KITTI dataset, which provides rectified stereo pairs with a specific baseline. As shown in FIG8, although DPSNet is not designed to work on rectified stereo images, it produces reasonable . In particular, DPSNet fine-tuned on the KITTI dataset in TAB7 achieves performance similar to BID24 in terms of D1-all score, with 4.34% for all pixels and 4.05% for non-occluded pixels in the KITTI benchmark. We expect that the depth accuracy would improve if we were to adopt rectified stereo pair-specific strategies, such as the feature consistency check in BID1. We developed a multiview stereo network whose design is inspired by best practices of traditional non-learning-based techniques. The plane sweep algorithm is formulated as an end-to-end network via a differentiable construction of plane sweep cost volumes and by solving for depth as a multilabel classification problem. Moreover, we propose a context-aware cost aggregation method that leads to improved depth regression without any post-processing. With this incorporation of traditional multiview stereo schemes into a deep learning framework, state-of-the-art reconstruction are achieved on a variety of datasets. Directions exist for improving DPSNet. One is to integrate semantic instance segmentation into the cost aggregation, similar to the segment-based cost aggregation method of BID25. Another direction is to improve depth prediction by employing viewpoint selection in constructing cost volumes BID6, rather than by simply averaging the estimated cost volumes as currently done in DPSNet. Lastly, the proposed network requires pre-calibrated intrinsic and extrinsic parameters for reconstruction. Lifting this restriction by additionally estimating camera poses in an end-to-end learning framework is an important future challenge.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

A convolution neural network for multi-view stereo matching whose design is inspired by best practices of traditional geometry-based approaches

We propose a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The ability to design biological structures such as DNA or proteins would have considerable medical and industrial impact. Doing so presents a challenging black-box optimization problem characterized by the large-batch, low round setting due to the need for labor-intensive wet lab evaluations. In response, we propose using reinforcement learning (RL) based on proximal-policy optimization (PPO) for biological sequence design. RL provides a flexible framework for optimization generative sequence models to achieve specific criteria, such as diversity among the high-quality sequences discovered. We propose a model-based variant of PPO, DyNA-PPO, to improve sample efficiency, where the policy for a new round is trained offline using a simulator fit on functional measurements from prior rounds. To accommodate the growing number of observations across rounds, the simulator model is automatically selected at each round from a pool of diverse models of varying capacity. On the tasks of designing DNA transcription factor binding sites, designing antimicrobial proteins, and optimizing the energy of Ising models based on protein structure, we find that DyNA-PPO performs significantly better than existing methods in settings in which modeling is feasible, while still not performing worse in situations in which a reliable model cannot be learned. Driven by real-world obstacles in health and disease requiring new drugs, treatments, and assays, the goal of biological sequence design is to identify new discrete sequences x which optimize some oracle, typically an experimentally-measured functional property f (x). This is a difficult black-box optimization problem over a combinatorially large search space in which function evaluation relies on slow and expensive wet-lab experiments. The setting induces unusual constraints in black-box optimization and reinforcement learning: large synchronous batches with few rounds total. The current gold standard for biomolecular design is directed evolution, which was recently recognized with a Nobel prize and is a form of randomized local search. Despite its impact, directed evolution is sample inefficient and relies on greedy hillclimbing to the optimal sequences. Recent work has demonstrated that machine-learning-guided optimization (Section 3) can find better sequences faster. Reinforcement learning (RL) provides a flexible framework for black-box optimization that can harness modern deep generative sequence models. This paper proposes a simple method for improving the sample efficiency of policy gradient methods such as PPO for black-box optimization by using surrogate models that are trained online to approximate f (x). Our method updates the policy's parameters using sequences x generated by the current policy π θ (x), but evaluated using a learned surrogate f w (x), instead of the true, but unknown, oracle reward function f (x). We learn the parameters of the reward model, w, simultaneously with the parameters of the policy. This is similar to other model-based RL methods, but simpler, since in the context of sequence optimization, the state-transition model is deterministic and known. Initially the learned reward model, f w (x), is unreliable, so we rely entirely on f (x) to assess sequences and update the policy. This allows a graceful fallback to PPO when the model is not effective. Over time, the reward model becomes more reliable and can be used as a cheap surrogate, similar to Bayesian optimization methods . We show empirically that cross-validation is an effective heuristic for assessing the model quality, which is simpler than the inference required by Bayesian optimization. We rigorously evaluate our method on three in-silico sequence design tasks that draw on experimental data to construct functions f (x) characteristic of real-world design problems: optimizing binding affinity of DNA sequences of length 8 (search space size 4 8); optimizing anti-microbial peptide sequences (search space size 20 50), and optimizing binary sequences where f (x) is defined by the energy of an Ising model for protein structure (search space size 20 50). These do not rely on wet lab experiments, and thus allow for large-scale benchmarking across a range of methods. We show that our DyNA-PPO method achieves higher cumulative reward for a given budget (measured in terms of number of calls to f (x)) than existing methods, such as standard PPO, various forms of the cross-entropy method, Bayesian optimization, and evolutionary search. In summary, our contributions are as follows: • We provide a model-based RL algorthm, DyNA-PPO, and demonstrate its effectiveness in performing sample efficient batched black-box function optimization. • We address model bias by quantifying the reliability and automatically selecting models of appropriate complexity via cross validation. • We propose a visitation-based exploration bonus and show that it is more effective than entropy-regularization in identifying multiple local optima. • We present a new optimization task for benchmarking methods for biological sequence design based on protein energy Ising models. Let f (x) be the rounds of experiments and that B sequences can be measured per round. Let D n = {(x, f (x))} be the data acquired in round n with |D n | = B. For simplicity, we assume that the sequence length T is constant, but our approach based on generating sequences autoregressively easily generalizes to variable-length sequences. We formulate the design of a single sequence x as a Markov decision process M = (S, A, p, r) with state space S, action space A, transition function p, and reward function r. The state space S = ∪ t=1... T V t is the set of all possible sequence prefixes and A corresponds to the vocabulary V. A sequence is generated left to right. At time step t, the state s t = a 0,..., a t−1 corresponds to the t last tokens and the action a t ∈ A to the next token. The transition function p(s t + 1|s t) = s t a t is deterministic and corresponds to appending a t to s t. The reward r(s t, a t) is zero except at the last step T, where it corresponds to the functional measurement f (s T −1). For generating variable length sequences, we extend the vocabulary by a special end-of-sequence token and terminate sequence generation when this token is selected. We train a policy π θ (a t |s t) to optimize the expected sum of rewards: 1: Input: Number of experiment rounds N 2: Input: Number of model-based training rounds M 3: Input: Set of candidate models S = {f} 4: Input: Minimum model score τ for model-based training 5: Input: Policy π θ with initial parameters θ 6: for n = 1, 2,... N do 7: Collect samples Dn = {x, f (x)} using policy π θ 8: Train policy π θ on Dn 9: Fit candidate models f ∈ S on n i=1 Di and compute their score by cross-validation 10: Select the subset of models S ⊆ S with a R 2 score ≥ τ 11: if S is not empty then 12: for m = 1, 2,...M do 13: Sample a batch of sequences x from π θ and observe the reward Update π θ on {x, f (x)} 15: end for 16: end if 17: end for We use proximal policy optimization (PPO) with KL trust-region constraint , which we have found to be more stable and sample efficient than REINFORCE . We have also considered off-policy deep Q-learning and categorical distributional deep Q-learning , which are in principle more sample-efficient than on-policy learning using PPO since they can reuse samples multiple times. However, they performed worse than PPO in our experiments (Appendix C). We implement algorithms using the TF-Agents RL library . We employ autoregressive models with one fully-connected layer as policy and value networks since they are faster to train and outperformed recurrent networks in our experiments. At time step t, the network takes as input the W last characters a t−W,..., a t−1 that are one-hot encoded, where the context window size W is a hyper-parameter. To provide the network with information about the current position of the context window, it also receives the time step t, which is embedded using a sinusoidal positional encoding , and concatenated with the one-hot characters. The policy network outputs a distribution π θ (a t |s t) over next the token a t. The value network V (s t), which approximates the expected future reward for being in state s t, is used as a baseline to reduce the variance of stochastic estimates of equation 1 . Model-based RL learns a model of the environment that is used as a simulator to provide additional pseudo-observations. While model-free RL has been successful in domains where interaction with the environment is cheap, such as those where the environment is defined by a software program, its high sample complexity may be unrealistic for biological sequence design. In model-based RL, the MDP M = (S, A, p, r) is approximated by a model M = (S, A, p, r) with the same state space S and action space A as M (, Ch. 8). Since the transition function p is deterministic in our case, only the reward function r(s t, a t) needs to be approximated by r (s t, a t). Since r(s T, a T) is non-zero at the last step T and then corresponds to f (x) with x == s T −1, the problem reduces to approximating f (x). This can be done by supervised regression by fitting a regressor f (x) on the data ∪ r <=r D r from all previous rounds. We then use the ing model to collect additional observations (x, f (x)) and update the policy in a simulation phase, instead of only using observations (x, f (x)) from the the true environment, which are expensive to collect. We call our method DyNA-PPO since it is similar to the DYNA architecture and since can be used for DNA sequence design. Model-based RL provides the promise of improved sample efficiency when the model is accurate, but it can reduce performance if insufficient data are available for training a trustworthy model. In this case, the policy is prone to exploit regions where the model is inaccurate . To reap the benefit of model-based RL when the model is accurate and avoid reduced performance when it is not, we (i) automatically select the model from a set of candidate models of varying complexity, (ii) only use the selected model if it is accurate, and iii) stop model-based training as soon the the model uncertainty increases by a certain threshold. After each round of experiment, we fit each candidate model on all available data to estimate f (x) via supervised regression. We quantify model accuracy by the R 2 score, which we estimate by five-fold cross validation. If the R 2 score of all candidate model is below a pre-specified threshold τ, we do not perform modelbased training in that round. Otherwise, we build an ensemble model that includes all models with a score greater or equal than τ, and use the average prediction as reward for training the policy. We considered τ as a tunable hyper-parameter, were we found τ = 0.5 to be optimal for all problems (see figure 14 . By ignoring the model if it is inaccurate, we aim to prevent the policy from exploiting deficiencies of the model . We perform up to M model-based optimization rounds (see algorithm 1) and stop as soon as the model uncertainty increased by a certain factor relative to the model uncertainty at the first round (m = 1). This is motivated by the observation that the model uncertainty is strongly correlated with the unknown model error, and prevents from training the policy with inaccurate model predictions (see figure 12, 13) as soon as the model starts to explore regions on which the model was not trained on. For models, we consider nearest neighbor regression, Bayesian ridge regression, random forests, gradient boosting trees, Gaussian processes, and ensemble of deep neural networks. Within each model family, we additionally use cross-validation for tuning hyper-parameters, such as the number of trees, tree depth, kernels and kernel parameters, or number of hidden layers and units (see Appendix A.7 for details). By testing and optimizing the hyper-parameters of different models automatically, the model capacity can dynamically increase as data becomes available. In Bayesian optimization, non-parametric models such as Gaussian processes are popular regressors, and they also automatically grow model capacity as more data arrives . However, with Bayesian optimization there is no opportunity to ignore the regressor entirely if it is unreliable. Furthermore, Bayesian optimization relies on the ability to do (approximate) Bayesian inference, which in practice is sensitive to the choice of approximation and to hyper-parameter choice . Here, we have find it simpler to do cross-validation-based model selection. Overall, our method combines the positive attributes of both generative and discriminative approaches to sequence design. Our experiments do not compare to prior work on model-based RL, since these methods primarily focus on estimating a dynamics model for state transitions. Learning policies to generate diverse sequences is important because of several reasons. In many applications, f (x) is an in-vitro (an experiment taking place outside a living organism) surrogate for an in-vivo (reaction occurring inside a living organism) functional measurement that is even more expensive to evaluate than f (x). The in-vivo measurement may depend on properties that are correlated with f (x) and others that are not captured at all in-vitro, such as off-target effects or toxicity. Therefore, it is desirable for the optimization procedure to discover a diverse set of candidate optima, to improve the chance that a sequence satisfying the ultimate in-vivo criteria is found within this set. Here, diversity is a downstream metric, for which training the policy π θ (x) to maximize equation 1 will not necessarily yield good performance. For example, a high-quality policy can learn to always generate the same sequence x with a high value of f (x), and will therefore in zero diversity. An additional reason that diversity matters is that it yields a good exploration strategy, even for scenarios where optimizing equation 1 is sufficient. Finally, use of strategies that reward high-diversity policies can reduce the policies' tendency to generate exact duplicates. To increase sequence diversity, we employ a simple exploration reward bonus based on the density of proposed sequences, similar to existing exploration techniques based on state visitation frequency . Specifically, we define the final reward r T = f (x) − λ · dens (x), where dens (x) ∈ N + is the weighted number of sequences that have been proposed in previous rounds with a distance of less than away from x, where the weight decays linearly with the distance. This reward penalizes proposing similar sequences multiple times, where the strength of the penalty is controlled by λ. As a , the policy learns not to generate related sequences and hence explores the search space more effectively. We used the edit-distance as distance metric and tuned the distance radius, where setting > 0 improved exploration on high-dimensional problems (see figure 11). We also considered an alternative penalty based on the nearest neighbor distance of the proposed sequence to past sequences, which we found to be less effective (see figure 9). Recently, machine learning approaches have been shown to be effective in optimizing real-world DNA and protein sequences; de;;; ). Existing methods for biological sequence design fall into three broad categories: evolutionary search, optimization using discriminative models (e.g. Bayesian optimization), and optimization using generative models (e.g. the cross entropy method). Evolutionary approaches perform direct local search in the space of sequences. They include the aforementioned directed evolution and derivatives with application-specific mutation and recombination steps. Evolutionary approaches are appealing since they are simple and can easily incorporate human intuition into the design process, but generally suffer from low sample efficiency. Optimization methods based on discriminative models alternate between two steps: (i) using the data that have been collected so far to fit a regressor f w (x) to approximate f (x), and (ii) using f w (x) to define an acquisition function that is optimized to select the next batch of sequences. Recently, such an approach was used to optimize the binding affinity of IgG antibodies , where a neural network ensemble was used for f w (x). In general, optimizing the acquisition function is a non-trivial combinatorial optimization problem. employ activation maximization, where gradient-based optimization is performed on a continuous relaxation of the discrete search space. However, this requires f w (x) to be differentiable and optimization of a continuous relaxation is vulnerable to leaving the data manifold (cf. deep dream ). Bayesian optimization defines an acquisition function such as expected improvement based on the uncertainty of f w (x), which enables balancing exploration and exploitation. Gaussian processes (GPs) are commonly used for Bayesian black box optimization since they provide calibrated uncertainty estimates. for an overview. Unfortunately, GPs are hard to scale to large, high-dimensional datasets and are sensitive to the choice of hyperparameters. In response, recent work has performed continuous black box optimization in the latent space of a deep generative model (Gómez-). However, this approach requires a pre-trained model such as a variational autoencoder to obtain the latent embeddings. Our modelbased reinforcement learning approach is similar to these approaches in that we train a reinforcement learning policy to optimize a model f w (x). However, our policy is also trained directly on observations of f (x) and is able to resort to model-free training by automatically identifying if the model f w (x) is too inaccurate to be used as a simulation of f (x). investigate conditions in which an estimate of model generalization (their analysis uses validation accuracy) could justify model usage in such model-based policy optimization settings. propose using a cascade of classifiers, one per round, to guide sampling progressively better candidates. Optimization methods based on generative models seek to learn a distribution p θ (x) parameterized by θ that maximizes the expected value of f (x):. We note that this is the same form as variational optimization objectives, which allow the use of parameter-space evolutionary strategies (; ;). Variants of the cross entropy method (; a) optimize θ, by alternating two steps: (i) sampling x ∼ p θ (x) and evaluating f(x), and (ii) updating θ to maximize this expectation. Methods differ in how step (ii) is performed. For example, hillclimb-MLE performs maximum-likelihood training on the top k sequences from step (i). Similarly, Feedback GAN (FBGAN) uses samples whose target function value f (x) exceeds a fixed threshold for training a generative adversarial network . Design by Adaptive Sampling (DbAs) performs weighted MLE of variational autoencoders , where a sample's weight corresponds to the probability that f (x) is greater than a quantile cutoff under an noise model . In Brookes et al. (2019b), p θ (x) is further restricted to stay close to a prior distribution over sequences. An alternative approach for optimizing the above expectation is RL. While RL has been used for generating natural text , small molecules , and RNA se-quences that fold into a particular structure , we are not aware of applications of RL to optimizing DNA and protein sequences. DyNA PPO is related to existing work on model-based RL for sample efficient control (; ; ; ;), with the key difference that the state transition function is known and the reward function is unknown in our work, whereas most existing model-based RL approaches seek to model the state-transition function and consider the reward function as known. Prior work on sequence generation incorporates non-differentiable rewards, like BLEU in machine translation, via weighted maximum likelihood (MLE). introduce reward augmented MLE, while fine tune an MLE-pretrained model using actor-critic methods. Reinforcement learning has also been applied to solving combinatorial optimization problems (; ; ;). In this setting sample complexity is less important because evaluating f (x) only involves a fast software program. Recent work has proposed generative models of protein structures or generative models of amino acids conditional on protein structure . Such methods are outside of the scope of this paper's experiments, since they could only be used in experimental settings where protein structures, which are expensive to measure, are available. Finally, DNA and protein design differs from small molecule design (Griffiths & Hernández-; ; Gómez-; ; ;) in the following points: (i) the number of sequences measured in parallel in the lab is typically higher (hundred or thousands vs. dozens) due to the maturity of DNA synthesis and sequencing technology, (ii) the search space is a set of sequences instead of molecular graphs, which require specialized network architectures for both discriminative and generative models, and (iii) molecules must be optimized subject to the constraint that there is a set of reactions to synthesize them, whereas practically all DNA or protein sequences are synthesizable. In the next three sections, we compare DyNA-PPO to existing methods on three in-silico optimization problems that we designed in collaboration with life scientists to faithfully simulate the behavior of real wet-lab experiments, which would be cost prohibitive for a comprehensive methodological evaluation. Along the way, we present ablation experiments to help better understand the behavior of DyNA-PPO. We compare the performance of model-free policy optimization (PPO) and model-based optimization (DyNA-PPO) with the following methods that we discussed in Section 3. Further details for each method can be found in Appendix A: • RegEvolution: Local search based on regularized evolution , which has performed well on other black-box optimization tasks and can be seen as an instance of directed evolution. • DbAs: Cross-entropy optimization using variational autoencoders . • FBGAN: Cross entropy optimization using generative adversarial networks . • Bayesopt GP: Bayesian optimization using a Gau ssian process regressor and activation maximization as acquisition function solver. • Bayesopt ENN Bayesian optimization using an ensemble of neural network regressors and activation maximization as acquisition function solver. • Random: Guessing sequences uniformly at random. We quantify optimization performance by the cumulative maximum of f (x) for sequences proposed up to a given round. We quantify sequence diversity (Section 2.4) in terms of the mean pairwise hamming distance between the sequences proposed at each round. For problems with known optima, we also report the fraction of global optima found. We replicate experiments with 50 random seeds. depending on the number of inner policy optimization rounds using the surrogate model. Using 0 rounds corresponds to PPO training. Since the surrogate model is sufficiently accurate, it is useful to perform many rounds of updating the policy using it before querying f (x) again. Right: the R 2 of the surrogate model. Since it is always above the threshold for model-based training (0.5; dashed line), it is always used for training. We first consider synthetic black-box optimization problems based on the 3D structure of naturallyoccurring proteins. Ising models fit on sets of evolutionary-related protein sequences have been shown to be accurate predictors for proteins' 3D structure (; ; ; Sułkowska et al., 2012). We consider the inverse problem: given a protein, we seek to find the amino acid sequence that minimizes the energy of the Ising model parameterized by its structure. Optimizers are given a budget of 10 rounds with batch size 1000 and we consider sequences of length 50 (search space size 20 50). The functional form of the energy function is given in Appendix B.1. On the left of Figure 1 we consider the optimization trajectory for a representative protein and on the right we compare the best f (x) found for each method across a range of proteins. We find that DyNA-PPO considerably outperforms the other methods. We expect that this is because this synthetic reward landscape can be well-described by a model fit using few examples, which also explains the good performance of Bayesian optimization. On the left of Figure 2 we vary the number of inner-loop optimization rounds of the policy using interaction with the model-based simulated environment, where using 0 rounds corresponds to performing standard PPO. We find that with DyNA-PPO it is worthwhile to spend considerable resources updating the policy using the simulator: the best system performs the most steps. Doing so is possible because the surrogate model is high quality. On the right we plot the score of the regressor fit by automated model selection. Its high accuracy helps DyNA-PPO learn to generate high-quality sequences using very few evaluations of f (x). Table 1: Comparison of methods across transcription factor binding sites. Mean rank of methods across all 41 hold-out tasks of the transcription factor binding dataset. Ranks were computed within each task using the average of metrics across optimization rounds, and then averaged across tasks. The higher the rank the better. 7 is the maximum rank. DyNA-PPO outperforms the other methods on both optimization of f (x) and its ability to identify multiple well-separated local optima. Transcription factors are protein sequences that bind to DNA sequences and regulate their activity. , the binding affinity of numerous transcription factors was measured against all possible length-8 DNA sequences (V = 4). The ing dataset defines 158 different Right: performance of entropy regularization as a function of the regularization strength. The top row shows that PPO finds about 80% of local optima with a relatively mild density penalty of λ = 0.1, whereas only about 45% local optima are found when using entropy regularization. The bottom row shows that varying the density penalty enables to control the sequence diversity quantified by the mean pairwise hamming distance between sequences. discrete optimization tasks, where the goal of each task is to find a DNA sequence of length eight that maximizes the affinity towards one of the transcription factors. It is well suited for in-silico benchmarking since (i) it is exhaustive and thereby does not require estimating missing f (x) and (ii) the distinct local optima of all tasks are known and can be used to quantify exploration (see Appendix B.2 for details). The optimization methods are given a budget of 10 rounds with a batch size of B = 100 sequences and the search space size is 4 8. We use one task (CRX REF R1) for optimizing the hyper-parameters of all methods, and test performance on 41 heterogeneous hold-out tasks. as a function of the total number of sequences measured so far. We find that DyNA-PPO and PPO outperform all other methods in terms of both the cumulative maximum f (x) found as well as the fraction of local optima discovered. We also find that the diversity of proposed sequences quantified by the fraction of global optima found is high compared to other generative approaches. This shows that our method continues to explore the search space by proposing novel sequences instead of converging to a single sequence or a handful of sequences-a desired property as discussed in section 2.4. Across all tasks DyNA-PPO and PPO rank highest compared with other methods (Table 1). In Figures 4 and 5 we analyze the effects two key design decisions of DyNA-PPO: model-based training and an exploration bonus. We find that automated model selection automatically increases the complexity of the model, but that the models are not always accurate enough to be used for model-based training. This explains the relatively small improvement of DyNA-PPO over PPO. We also find that the exploration bonus outlined in Section 2.4 is more effective than entropy regularization in finding multiple local optima and promoting sequence diversity. Next, we seek to design antimicrobial peptides (AMPs). AMPs are relatively short (8 -75 amino acids) protein sequences (|V | = 20 amino acids) that are promising candidates against multi-resistant pathogens due to their wide range of antimicrobial activities. We use the dataset proposed by , which contains 6,760 unique AMP sequences and their antimicrobial activity towards multiple pathogens. We follow for preprocessing the dataset and generating non-AMP sequences as negative training samples. Unlike the transcription factor binding site dataset, we do not have wet-lab experiments for every sequence in the search space. Therefore, we fit random forest classifiers to predict if a sequence is antimicrobial towards a certain pathogen in the dataset (see Section B.3), and use the predicted probability as the functional measurement f (x) to optimize. Given the high accuracy of the classifiers (cross validated AUC 0.94 and 0.99), we expect that the reward landscape of f (x) is of realistic difficulty. We perform 8 rounds with a batch size 250 and restrict the sequence length to at most 50 characters (search space size 20 50). Figure 6 compares methods on C. alibicani. We find that model-based optimization using DyNA PPO enables finding high reward sequences in early rounds, though model-free PPO slightly surpasses the performance of DyNA PPO later on. Both DyNA PPO and PPO considerably outperform the other methods in terms of the maximum f (x) found. The density based exploration bonus prevents PPO and DyNA PPO from generating non-unique sequences (figure 11). Stopping modelbased training as soon as the model uncertainty increased by a certain factor prevents DyNA PPO from converging to a sub-optimal solution when performing many model-based optimization rounds. (figure 12,13). We have shown that RL is an attractive alternative to existing methods for designing DNA and protein sequences. We have proposed DyNA-PPO, a model-based extension of PPO with automatic model selection that improves sample efficiency, and incorporates a reward function that promotes exploration by penalizing identical sequences. By approximating an expensive wet-lab experiment with a surrogate model, we can perform many rounds of optimization in simulation. While this work has been focused on showing the benefit of DyNA-PPO for biological sequence design, we believe that the large-batch, low-round optimization setting described here may well be of general interest, and that model-based RL may be applicable in other domains such as agriculture, education, and economics. A IMPLEMENTATION DETAILS Regularized evolution is a variant of directed evolution that regularizes the search by keeping a fixed number of individuals alive as candidates for selection (analogous to death by aging). At each round, it generates a batch of child sequences by sampling two parent sequences per child from the population via tournament selection, i.e. selecting the fittest out of K randomly sampled individuals. It then performs crossover of the two parent sequences by copying the characters of one parent from left to right and randomly transitioning to transcribing from the other parent sequence with some crossover probability at each step. Child sequences are mutated by substituting characters independently by other characters with some substitution probability. For variable length sequences, we also allowed insertion and deletion mutations. As hyper-parameters, we tune the tournament size, substitution-, insertion-, and deletion-probabilities. A.2 MCMC AND SIMULATED ANNEALING MCMC and simulated annealing resemble evolution with no crossover, and selection only occurring between an individual and its parent. Beginning with a random population, each individual evolves as a single chain, with neighborhood structure defined by the mutation operator described in section A.1. We denote x and x as a parent and child sequence, respectively. A transition x → x is always accepted if the reward increases (f (x) > f (x)). Otherwise, the transition is accepted with some acceptance probability. For MCMC, the acceptance probability is f (x)/f (x), while for simulated annealing it is exp((f (x) − f (x))/T ) for some temperature T. A high temperature increases the likelihood of accepting a move that decreases the reward. The next mutation on the chain begins from x if the transition is rejected, and from x otherwise. We treated the temperature T as a tunable hyper-parameter in addition to the evolution hyper-parameters described in section A.1. We follow the methodology suggested by. Instead of using a constant threshold for selecting positive sequences as described in the original publication, we used a quantile cutoff, which does not depend on the absolute scale of f (x) and performed better in our experiments. As hyper-parameters, we tuned the quantile cutoff, learning rate, batch size, discriminator and generator training epochs, the gradient penalty weight, the Gumble softmax temperature, and the number of latent variables of the generator. We follow the methodology suggested by. As hyper-parameters, we optimized the quantile for selecting training samples, learning rate, batch size, training epochs, number of hidden units of the MLP generator and discriminator, and number of latent variables. The generative model is an variational autoencoder with a multi-layer perceptron decoder. We also considered DbAs with a LSTM as generative model, which performed slightly better than a VAE on the TfBind8 problem but worse on the PdbIsing and AMP problem (see figure 8). As regressors, we considered a Gaussian process (GP) with RBF kernel on one-hot features, and an ensemble of ten fully-connected neural networks with one fully connected layer and 128 hidden units. We used the regressor output to compute the expected improvement or posterior mean acquisition function, which we maximized by gradient ascent for a certain number of acquisition following. We took the ing B unique sequences with highest acquisition function value as sequences to measure in the next round. We tuned the length scale and variance of the RBF kernel, and the learning rate, batch size, and number of training epochs of the neural network ensemble. We further tuned the number of gradient ascent steps for activation maximization. A.6 PPO AND DYNA PPO We used the PPO implementation of the TF-Agents RL library . After each round, we trained trained the agent on the collected batch of sequences for a relatively high number of steps (about 72) since it ed in a performance increase compared with performing only a single training step. We used the adaptive KL trust region penalty, which performed slightly better than importance ratio clipping in our experiments. We used a policy and value network with one fully connected layer and 128 hidden units. Both networks take the W last generated characters and positions as input, which we padded at the beginning of the sequence. We set the context window W to the minimum of the total sequence length and 50. As hyperparameters, we tuned the learning rate, number of training steps, adaptive KL target, and entropy regularization. For DyNA PPO, we also tuned the maximum number of model-based optimization rounds M (see section 2.3). Automatic model selection optimizes the hyper-parameters of a set of candidate models by randomized search, and evaluates each hyper-parameter configuration by five-fold cross-validation using the R 2 score. To account for randomness in the R 2 score between models due to different crossvalidation splits, we used the same split for evaluating each of the models per round. We considered the following candidate models (implemented in scikit learn) and corresponding hyper-parameters: • KNeighborsRegressor: n neighbors • BayesianRidge: alpha 1, alpha 2, lambda 1, lamdba 2 • RandomForestRegressor: max depth, max features, n estimators • ExtraTreesRegressor: max depth, max features, n estimators • GradientBoostingRegressor: learning rate, max depth, n estimators • GaussianProcessRegressor: with RBF, RationalQuadratic, and Matern kernel We also considered an ensemble of 10 neural networks with two convolutional layers and one fully connected layer, and optimized the learning rate and number of training epochs. Given a protein in the Protein Data Bank , we compute the energy E(x) for sequence x as E(x) = i φ i (x i) + ij C ij φ(x i, x j), where x i refers to the character in the i-th position of sequence x. C ij is an indicator for whether the Cα atoms of the residues at positions i and j are separated by less than 6 Angstroms when the protein folds. φ(x i, x j) is a widely-used'pair potential' based on co-occurence probabilities derived from the structures of real-world proteins . The same 20 x 20 table of pair potentials is used at all positions in the sequence, and thus the difference in energy functions across proteins is dictated only by their differing contact map structure. We set the local term φ i (x i) to zero. In future work, it would be interesting to consider non-zero local terms. Our experiments consider a set of qualitatively-different proteins listed at the bottom-right of Figure 1. We identify the local optima using the same procedure as in Section B.2, except without accounting for reverse complements. We used the dataset described by , and min-max normalized binding affinities between zero and one. To reduce computational costs, we only considered the first replicate (REF R1) of each wild type transcription factor in the dataset, which ed in 41 optimization targets that we used for comparing optimizers as described in Section 4.2. We extracted local optima for each binding target as follows. First, we separated sequences into forward and reverse sequences by ordering sequences lexicographically and including each sequence in the set of forward sequences unless the set already contained its reverse complement. We then chose the 100 forward sequences with the highest binding affinity and clustered them using the hamming distance metric, where we determined the number of clusters by finding the number of PCA components required to explain 95% of variance. We then used the sequences with the highest reward per cluster and their reverse complement as local optima. We downloaded the dataset 1 provided by , and followed the paper for preprocessing sequences and generating non-AMP sequences as negative training samples. We additionally excluded sequences containing cysteine and sequences shorter than 15 or longer than 50 amino acids. We fit one classifier to predict if a sequence is antimicrobial towards either E.coli, S.aureus, P.aeruginosa, or B.subtilis, which we used for hyper-parameter tuning, and a second classifier for C. alibicani, which we used for hold-out evaluation. We used C. alibicani as hold-out target since its antimicrobial activity was least correlated with the activity of other pathogenes in the dataset with more than 1000 AMP sequences. We used random forest classifiers since they were more accurate (cross validated AUC 0.99 and 0.94) than alternative models such as k-nearest neighbors, Gaussian processes, or neural networks. Since sequences are variable-length, we padded them to the maximum sequence length of 50 and extended the vocabulary by an additional end of sequence token. Tokens after the fist end of sequence token were ignored when evaluating f (x). Protein Ising AMP DyNA-PPO is built on PPO, which we have found to outperform other policy-based and value-based RL methods in practice on our problems. In Figure 7 we contrast the performance of PPO , REINFORCE , deep Q-learning , and categorical distributional deep Q-learning on all problems considered in section 4. We find that PPO has better exploration properties than REINFORCE, which tends to converge too soon to a local optimum. The poor performance of DQN and CatDQN can be explained by the sparse reward (the reward is only non-zero at the terminal state), such that the Bellman error and training loss for updating the Q network are zero in most states. We also found the performance of DQN and CatDQN to be sensitive to the choice of the epsilon greedy rate and Boltzmann temperature for trading-off exploration and exploitation and increasing diversity. TF Bind Protein Ising AMP Figure 8: Comparison of additional baselines. We consider the performance of optimizers based on MCMC (section A.2). Such methods are known to be effective optimizers when evaluating the black box function is inexpensive, and thus many iterations of sampling can be performed. The focus of our experiments is on resource-constrained black box optimization. We find that their low sample efficiency makes them undesirable for biological sequence design. We also consider DbAS with a LSTM generative model instead of a VAE with multi-layer perceptron decoder to disentangle the choice of generative model in DbAS from the overall optimization strategy. DbAs VAE outperforms DbAs RNN on all problems except for TF bind. Figure 9: Comparison of alternative approaches for promoting diversity. Left column: The proposed density based exploration bonus as described in section 2.4, which adds a penalty to the reward of a sequence x that is proportional to the distance-weighted number of past sequence that are less than a specified distance away from x (here edit distance one). Middle column: An alternative approach where the exploration bonus of a sequence is proportional to the distance to the nearest neighboring past sequence. Right column: standard entropy regularization. Shown are the cumulative maximum reward and alternative metrics for quantifying diversity depending on the penalty strength (λ in section 2.4) of each exploration approach. Without exploration bonus (penalty = 0.0; red line), PPO does not find the optimal solution (cumulative maximum is below 1.0) and the hamming distance and uniqueness of sequences within a batch converge to zero. PPO finds the optimal solutions and continues to generate diverse sequences by increasing the strength of any of the three exploration approaches. The density based exploration bonus is most effective in recovering all optima (second row, left plot) and enables a more fine-grained control of diversity compared to the distance based approach. Results are shown for target CRX REF R1 of the transcription factor binding problem. Figure 11: Analysis of density-based exploration bonus on the AMP problem. The top row shows the sensitivity to the distance radius and the bottom row to the regularization strength λ (section 2.4). Diversity correlates positively with the distance radius and regularization strength. = 2 and λ = 0.1 provides the best trade-off between optimization performances (cumulative maximum reward) and diversity (mean pairwise hamming distance and uniqueness). Penalizing only exact duplicates (= 0) is less effective in maintaining a high hamming distance than taking neighboring sequences into account (> 0). Figure 13: Optimization performance on the AMP problem depending on the uncertainty threshold for stopping model-based optimization and the maximum number model optimization rounds M. Without threshold (Inf; red line), DyNA PPO converges to a sub-optimal solution, in particular when the maximum number of model-based optimization rounds M is high. A threshold of 0.5 prevents a performance decrease due to inaccuracy of the model (see figure 12). Protein Ising AMP Figure 14: Sensitivity of DyNA PPO depending on the choice of the minimum cross-validation score τ for model-based optimization. Shown are the for the transcription factor binding-, protein contact Ising-, and AMP problem. DyNA PPO reduces to PPO if τ is above the maximum cross-validation score of models that are considered during model selection, e.g. if τ = 1.0. If τ is too low, also inaccurate models are selected, which reduces the overall accuracy of the ensemble model and optimization performance. A cross-validation score between 0.4 and 0.5 is best for all problems.

Represent the TLDR for finding the corresponding scientific article

We augment model-free policy learning with a sequence-level surrogate reward functions and count-based visitation bonus and demonstrate effectiveness in the large batch, low-round regime seen in designing DNA and protein sequences.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.