Question: Represent the article for finding a single-sentence TLDR

Question: There is mounting evidence that pretraining can be valuable for neural network language understanding models, but we do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn. With this in mind, we compare four objectives---language modeling, translation, skip-thought, and autoencoding---on their ability to induce syntactic and part-of-speech information, holding constant the genre and quantity of training data. We find that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data, which suggests that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information. We also find that a randomly-initialized, frozen model can perform strikingly well on our auxiliary tasks, but that this effect disappears when the amount of training data for the auxiliary tasks is reduced. Representation learning with deep recurrent neural networks has revolutionized natural language processing and replaced many of the expert-designed, linguistic features previously used. Recently, researchers have begun to investigate the properties of representations learned by networks by training auxiliary classifiers that use the hidden states of frozen pretrained models to perform other tasks. These investigations have shown that when deep LSTM RNNs are trained on tasks like machine translation, they latently identify substantial syntactic and semantic information about their input sentences, including part-of-speech (; a,b;).These intriguing findings lead us to ask the following questions:1. How does the training task affect how well models latently learn syntactic properties? Which tasks are better at inducing these properties?2. How does the amount of data the model is trained on affect these ? When does training on more data help?We investigate these questions by holding the data source and model architecture constant, while varying both the training task and the amount of training data. Specifically, we examine models trained on English-German (En-De) translation, language modeling, skip-thought , and autoencoding, in addition to an untrained baseline model. We control for the data domain by exclusively training on datasets from the 2016 Conference on Machine Translation (WMT;). We train models on all tasks using the parallel En-De corpus and a small subset of that corpus, which allows us to make a fair comparison across all five models. Additionally, we augment the parallel dataset with a large monolingual corpus from WMT to examine how the performance of the unsupervised tasks (all but translation) scale with more data. Throughout our work, we focus on the syntactic evaluation tasks of part-of-speech (POS) tagging and Combinatorial Categorical Grammar (CCG) supertagging. Supertagging is a building block for parsing as these tags constrain the ways in which words can compose, largely determining the parse of the sentence. CCG supertagging thus allows us to measure the degree to which models learn syntactic structure above the word. We focus our analysis on representations learned by language models and by the encoders of sequence-to-sequence models, as translation encoders have been found to learn richer representations of POS and morphological information than translation decoders (a).We find that for POS and CCG tagging, bidirectional language models (BiLMs)-created by separately training forward and backward language models, and concatenating their hidden statesoutperform models trained on all other tasks. Even BiLMs trained on relatively small amounts of data (1 million sentences) outperform translation and skip-thought models trained on larger datasets (5 million and 63 million sentences respectively).Our inclusion of an untrained LSTM baseline allows us to study the effect of training on state representations. We find, surprisingly, that randomly initialized LSTMs underperform our best trained models by only a few percentage points when we use all of the available labeled data to train classifiers for our auxiliary tasks. When we reduce the amount of classifier training data though, the performance of the randomly initialized LSTM model drops far below those of trained models. We hypothesize that this occurs because training the classifiers on large amounts of auxiliary task data allows them to memorize configurations of words seen in the training set and their associated tags. We test this hypothesis by training classifiers to predict the identity of neighboring words from a given hidden state, and find that randomly initialized models outperform all trained models on this task. Our findings demonstrate that our best trained models do well on the tagging tasks because they are truly learning representations that conform to our notions of POS and CCG tagging, and not because the classifiers we train are able to recover neighboring word identity information well. introduce the idea of examining sentence vector representations by training auxiliary classifiers to take sentence encodings and predict attributes like word order. Belinkov et al. (2017a) build on this work by using classifiers to examine the hidden states of machine translation models in terms of what they learn about POS and morphology. They find that translating into morphologically poorer languages leads to a slight improvement in encoder representations. However, this effect is small, and we expect that our study of English-German translation will nonetheless provide a reasonable overall picture of the representations that can be learned in data-rich translation. Beyond translation, find that deep LSTMs latently learn hierarchical syntax when trained on a variety of tasks-including semantic role labeling, language modeling, and dependency parsing. We build on this work by controlling for model size, and the quantity and genre of the training data, which allows us to make direct comparisons between different tasks on their ability to induce syntactic information. Transfer Learning of Representations Much of the work on sentence-level pretraining has focused on sentence-to-vector models and evaluating learned representations on how well they can be used to perform sentence-level classification tasks. Skip-thought -the technique of training a sequence-to-sequence model to predict the sentence preceding and following each sentence in a running text-represents a prominent early success in this area with unlabeled data, and InferSent -the technique of pretraining encoders on natural language inference data-yields strikingly better performance when such labeled data is available. Work in this area has recently moved beyond strict sentence-to-vector mapping. Newer models that incorporate LSTMs pretrained on datarich tasks, like translation and language modeling, have achieved state-of-the-art on many tasks-including semantic role labeling and coreference resolution (; ;). Although comparisons have previously been made between translation and language modeling as pretraining tasks , we investigate this issue more thoroughly by controlling for the quantity and content of the training data. Training Dataset Size The performance of neural models depends immensely on the amount of training data used. find that when training machine translation models on corpora with fewer than 15 million words (English side), statistical machine translation approaches outperform neural ones. study data volume dependence on several tasks-including translation and image classification-and find that for small amounts of data, neural models perform about as well as, 2017). Our method of training auxiliary classifiers on randomly initialized RNNs builds on the tradition of reservoir computing, in which randomly initialized networks or "reservoirs" are fixed and only "read-out" classifier networks are trained (Lukoševičius and). Echo state networks-reservoir computing with recurrent models-have been used for tasks like speech recognition, language modeling, and time series prediction (; ;). We use the parallel English-German (En-De) dataset from the 2016 ACL Conference on Machine Translation (WMT) shared task on news translation . This dataset contains 5 million ordered sentence translation pairs. We also use the 2015 English monolingual news dataset from the same WMT shared task, which contains approximately 58 million ordered sentences. To examine how the volume of training data affects learned representations, we use four corpus sizes: 1, 5, 15, and 63 million sentences (translation is only trained on the smaller two sizes). We create the 1 million sentence corpora from the 5 million sentence dataset by sampling (i) sentence pairs for translation, (ii) English sentences for autoencoders, and (iii) ordered English sentence pairs for skip-thought and language models. Note, we initialize language model LSTM states with the final state after reading the previous sentence. Similarly, we create 15 million sentence corpora for the unsupervised tasks by sampling sentences from the entire corpus of 63 million sentences. We use word-level representations throughout and use the Moses package to tokenize and truecase our data. Finally, we limit both the English and German vocabularies to the 50k most frequent tokens in the training set. We train all our models using OpenNMT-py and use the default options for model sizes, hyperparameters, and training procedure-except that we increase the size of the LSTMs, make the encoders bidirectional, and use validation-based learning rate decay instead of a fixed schedule. Specifically, all our models (except language models) are 1000D, twolayer encoder-decoder LSTMs with bidirectional encoders (500D LSTMs in each direction) and 500D embeddings; we train models both with and without attention . For language models, we train a single 1000D forward language model and a bidirectional language model-two 500D language models (one forward, one backward) trained separately, whose hidden states are concatenated. All models are randomly initialized with a uniform distribution between −0.1 and 0.1, the default in OpenNMT-py. We use the same training procedure for all our models. We evaluate on the validation set every epoch when training on the 1 and 5 million sentence datasets, and evaluate approximately every 5 million sentences when training on the larger datasets. We use SGD with an initial learning rate of 1. Whenever a model's validation loss increases relative to the previous evaluation, we halve the learning rate and stop training when the learning rate reaches 0.5 15. For each training task and dataset size, we select the model with the lowest validation perplexity to perform auxiliary task evaluations on. We report model performance in terms of perplexity and BLEU in Table 1. For translation we use beam search (B = 5) when decoding. For CCG supertagging, we use CCG Bank , which is based on PTB WSJ. CCG supertagging provides fine-grained information about the role of each word in its larger syntactic context and is considered almost parsing, since sequences of tags map sentences to small subsets of possible parses. The entire dataset contains approximately 50k sentences and 1327 tag types. We display POS and CCG tags for an example sentence in FIG2 To study the impact of auxiliary task data volume, for both datasets we create smaller classifier training sets by sampling 10% and 1% of the sentences. We truecase both datasets using the same truecase model trained on WMT and restrict the vocabularies to the 50k tokens used in our LSTM models. We use the word-conditional most frequent class as a baseline, which is the most frequently assigned tag class for each word in the training set; for this baseline we restrict the vocabulary to that of our encoder models (we map all out-of-vocabulary words to a single UNK token). Note that while PTB and WMT are both drawn from news text, there is slight genre mismatch. Word Identity For this task, the classifier takes a single LSTM hidden state as input and predicts the identity of the word at a different time step, for example, three steps previous (shift of -3). We use the WSJ data for this task. , we take all words that occur between 100 and 1000 times (about 1000 words total) as the possible targets for neighboring word prediction. Classifier Training Procedure We train multilayer perceptron (MLP) classifiers that take an LSTM hidden state (from one time step and one layer) and output a distribution over the possible labels (tags or word identities). The MLPs we train have a single 1000D hidden layer with a ReLU activation. For classifier training, we use the same training and learning rate decay procedure used for pretraining the encoders. In this section, we discuss the main POS and CCG tagging displayed in FIG3. Overall, POS and CCG tagging accuracies tend to increase with the amount of data the LSTM encoders are trained on. However, the amount of this improvement is generally small, especially when encoders are already trained on large amounts of data. Language Modeling and Translation For all pretraining dataset sizes and tasks, bidirectional language model (BiLM) and translation encoder representations perform best in terms of both POS and CCG tagging. Translation encoders, however, slightly underperform BiLMs, even when both models are trained on the same amount of data. Interestingly, even BiLMs trained on the smallest amount of data (1 million sentences) outperform models trained on all other tasks and dataset sizes (up to 5 million sentences for translation, and 63 million sentences for skip-thought and autoencoding). The consistent superior performance of BiLMs-along with the fact that language models do not require aligned data-suggests that for transfer learning of syntactic information, BiLMs are superior to translation encoders. For all amounts of training data, the BiLMs significantly outperform the 1000D forward-only language models. The gap in performance between bidirectional and forward language models is greater for CCG supertagging than for POS tagging. When using all available auxiliary training data, there is a 2 and 8 percentage point performance gap on POS and CCG tagging respectively. This difference in relative performance suggests that bidirectional context information is more important when identifying syntactic structure than when identifying part of speech. Figure 2 also illustrates how the best performing BiLMs and translation models tend to be more robust to decreases in classifier data than models trained on other tasks. When training on less auxiliary task data, POS tagging performance tends to drop less than CCG supertagging performance. For the best model (BiLM trained on 63 million sentences), when using 1% rather than all of the auxiliary task training data, CCG accuracy drops 9 percentage points, while POS accuracy only drops 2 points. Further examinations of the effect of classifier data volume are displayed in FIG5.Skip-Thought Although skip-thought encoders consistently underperform both BiLMs and translation encoders in all data regimes we examine, skip-thought models improve the most when increasing the amount of pretraining data, and are the only models whose performance does not seem to have plateaued by 63 million training sentences. Skip-thought models without attention are very similar to language models-the main difference is that while skip-thought models have separate encoder and decoder weights (and a bidirectional encoder), language models share weights between the encoder and decoder. Thus language models can be interpreted as regularized versions of skip-thought. The increased model capacity of skip-thought, compared to language models, could explain the difference in learned representation quality-especially when these models are trained on smaller amounts of data. Random Initialization For our randomly initialized, untrained LSTM encoders we use the default weight initialization technique in OpenNMTpy, a uniform distribution between -0.1 and 0.1; the only change we make is to set all biases to zero. We find that this baseline performs quite well when using all auxiliary data, and is only 3 and 8 percentage points behind the BiLM on POS and CCG tagging, respectively. We find that decreasing the amount of classifier data leads to a significantly greater drop in the randomly initialized encoder performance compared to trained models. In the 1% classifier data regime, the performance of untrained encoders on both tasks drops below that of all trained models and below even the wordconditional most-frequent class baseline. We hypothesize that the randomly initialized baseline is able to perform well on tagging tasks with large amounts of auxiliary task training data, because the classifier can learn the identity of neighboring words from a given time step's hidden state, and simply memorize word configurations and their associated tags from the training data. We test this hypothesis directly in Section 6 and find that untrained LSTM representations are in fact better at capturing neighboring word identity information than any trained model. Autoencoder Models trained on autoencoding are the only ones that do not consistently improve with the amount of training data, which is unsurprising as unregularized autoencoders are prone to learning identity mappings . When training on 10% and 1% of the auxiliary task data, autoencoders outperform randomly initialized encoders and match the word-conditional most frequent class baseline. When training on all the auxiliary data though, untrained encoders outperform autoencoders. These suggest that autoencoders learn some useful structure that Figure 4: POS and CCG tagging accuracies in terms of percentage points over the word-conditional most frequent class baseline. We display for the best performing models for each task.is useful in the low auxiliary data regime. However, the representations autoencoders learn do not capture syntactically rich features, since random encoders outperform them in the high auxiliary data regime. This is further supported by the extremely poor performance of the second layer of an autoencoder without attention on POS tagging (almost 10 percentage points below the most frequent class baseline), as seen in Figure 4a. Embeddings (Layer 0) We find that randomly initialized embeddings consistently perform as well as the word-conditional most frequent class baseline on POS and CCG tagging, which serves as an upper bound on performance for the embedding layer. As these are untrained, the auxiliary classifiers are learning to memorize and classify the random vectors. When using all the auxiliary classifier data, there is no significant difference in the performance of trained and untrained embeddings on the tagging tasks. Only for smaller amounts of classifier data do trained embeddings consistently outperform randomly initialized ones. Belinkov et al. (2017a) find that, for translation models, the first layer consistently outperforms the second on POS tagging. We find that this pattern holds for all our models, except in BiLMs, where the first and second layers perform equivalently. The pattern holds even for untrained models, suggesting that POS information is stored on the lower layer, not because the training task encourages this, but because of fundamental properties of the deep LSTM architecture. We also find that for CCG supertagging, the first layer also outperforms the second layer on untrained models. For the trained models though, behavior is mixed, with the second layer performing better in some cases. Which layer performs best appears to be independent of absolute performance on the supertagging task. Our layer analysis are displayed in Figure 4. Our on word identity prediction are summarized in FIG7 and given in more detail in Appendix A. We find that randomly initialized LSTMs outperform all trained models. We hypothesize this occurs because a kind of useful forgetting occurs during training, as the LSTMs learn that information about certain word patterns are more important to remember than others. In this regard, randomly initialized models are less biased and process inputs more uniformly. The fact that untrained encoders outperform trained ones on word identity prediction, but underperform trained models on POS and CCG tagging, confirms that trained models genuinely capture substantial syntactic features, beyond mere word identity, that the auxiliary classifiers can use. Effect of Depth We find that for both trained and untrained models, the first layer outperforms the second layer when predicting the identity of the immediate neighbors of a word. However, the second layer tends to outperform the first at predicting the identity of more distant neighboring words. This effect is especially apparent for the randomly initialized encoders. Our findings suggest that, as is the case for convolutional neural networks, depth in recur- By controlling for the genre and quantity of the training data, we make fair comparisons between several data-rich training tasks in their ability to induce syntactic information. We find that bidirectional language models (BiLMs) do better than translation and skip-thought encoders at extracting useful features for POS tagging and CCG supertagging. Moreover, this improvement holds even when the BiLMs are trained on substantially less data than competing models. Although, due to limited parallel data, we could not compare BiLMs and translation encoders on more than 5 million sentences, our suggest that for syntactic information, there is no need to compare these two models trained on more data, as BiLMs consistently outperform translation encoders in all data regimes. We also find that randomly initialized encoders extract usable features for POS and CCG tagging, at least when the auxiliary POS and CCG classifiers are themselves trained on reasonably large amounts of data. However, the performance of untrained models drops sharply relative to trained ones when using smaller amounts of the classifier data. We investigate further and find that untrained models outperform trained ones on the task of neighboring word identity prediction, which confirms that trained encoders do not perform well on tagging tasks because the classifiers are simply memorizing word identity information. We also find that both trained and untrained LSTMs store more local neighboring word identity information in lower layers and more distant word identity information in upper layers, which suggests that depth in LSTMs allow them to capture larger context information. Our suggest that for transfer learning, bidirectional language models like ELMo capture more useful features than translation encoders-and that this holds even on genres or languages for which data is not abundant. However, the scope of our experiments is limited, and we still know little about the representations of models trained on other supervised tasks, or precisely how the choice of training task affects the type of syntactic information that is learned. Our work also highlights the interesting behavior of randomly initialized LSTMs, which show an ability to preserve the contents of their inputs significantly better than trained models. Figure 6: Here we display for the word identity prediction task with randomly initialized LSTM encoders with up to 4 layers. Lower layers have a more peaked shape and upper layers a more flat shape, meaning that the lower layers encode relatively more nearby neighboring word information, while upper layers encode relatively more distant neighboring word information. Table 4: Here we display for training on 1% of auxiliary task data. Word-conditional most frequent class baselines for this amount of training data are 81.8% for POS tagging and 62.3% for CCG supertagging. For each task, we underline the best performance for each training dataset size and bold the best overall performance.

Representations from language models consistently perform better than translation encoders on syntactic auxiliary prediction tasks.

The choice of the hub (target) language affects the quality of cross-lingual embeddings, which shouldn't be evaluated only  on English-centric dictionaries.


Question: represent the scientific article to find a short tldr

Question: We consider the problem of generating configurations that satisfy physical constraints for optimal material nano-pattern design, where multiple (and often conflicting) properties need to be simultaneously satisfied. Consider, for example, the trade-off between thermal resistance, electrical conductivity, and mechanical stability needed to design a nano-porous template with optimal thermoelectric efficiency. To that end, we leverage the posterior regularization framework andshow that this constraint satisfaction problem can be formulated as sampling froma Gibbs distribution. The main challenges come from the black-box nature ofthose physical constraints, since they are obtained via solving highly non-linearPDEs. To overcome those difficulties, we introduce Surrogate-based Constrained Langevin dynamics for black-box sampling. We explore two surrogate approaches. The first approach exploits zero-order approximation of gradients in the Langevin Sampling and we refer to it as Zero-Order Langevin. In practice, this approach can be prohibitive since we still need to often query the expensive PDE solvers. The second approach approximates the gradients in the Langevin dynamics with deep neural networks, allowing us an efficient sampling strategy using the surrogate model. We prove the convergence of those two approaches when the target distribution is log-concave and smooth. We show the effectiveness of both approaches in designing optimal nano-porous material configurations, where the goal is to produce nano-pattern templates with low thermal conductivity and reasonable mechanical stability. In many real-world design problems, the optimal design needs to simultaneously satisfy multiple constraints, which can be expensive to estimate. For example, in computational material design, the goal is to come up with material configurations, or samples, satisfying a list of physical constraints that are given by black-box numerical Partial Differential Equations (PDE) solvers. Such solvers (for example, the Boltzmann Transport Equation solver) are often complex, expensive to evaluate, and offer no access to their inner variables or their gradients. We pose this design-under-constraints problem as sampling from a Gibbs distribution defined on some compact support. The problem of sampling from a distribution with unknown likelihood that can only be point-wise evaluated is called black-box sampling . We show in this paper that constrained black-box sampling can be cast as a constrained Langevin dynamics with gradient-free methods. Zero-order optimization via Gaussian smoothing was introduced in and extended to black-box sampling with Langevin dynamics in. We extend this approach to the constrained setting from a black-box density with compact support. However, one shortcoming of this approach is that it is computationally very expensive since it requires repeatedly querying PDE solvers in order to get an estimate of the gradient. To alleviate computational issues, we propose Surrogate Model Based Langevin dynamics, that consists of two steps: (i) Learning (using training data) an approximation of the gradient of the potential of the Gibbs distribution. We show that learning the gradient, rather than the potential itself, is important for the mixing of the Langevin dynamics towards the target Gibbs distribution. We devise several objective functions, as well as deep neural-network architectures for parameterizing the approximating function class, for learning the gradient of the potential function. (ii) We then use the surrogate gradient model in the constrained Langevin dynamics in lieu of the black-box potential. Using the surrogate enables more efficient sampling, since it avoids querying the expensive PDE solvers, and obtaining gradients is as efficient as evaluating the functions themselves using automatic differentiation frameworks such as PyTorch or TensorFlow. To summarize, our main contributions are as follows: 1. We cast the problem of generating samples under constraints in the black-box setting as sampling from a Gibbs distribution. 2. We introduce Constrained Zero-Order Langevin Monte Carlo, using projection or proximal methods, and provide the proof of its convergence to the target Gibbs distribution. 3. We introduce Surrogate Model Based Projected Langevin Monte Carlo via learning the gradient of the potential of the Gibbs distribution using deep neural networks or reproducing kernel spaces, and prove its convergence to the target distribution when used in conjunction with projection or proximal based methods. We shed the light on the importance of the approximation of the gradient of the potential, and we show how to achieve this using Hermite and Taylor learning. 4. We showcase the usability and effectiveness of the proposed methods for the design of nanoporous configurations with improved thermoelectric efficiency. The design consists of finding new configurations with optimized pore locations, such that the ing configurations have favorable thermal conductivity (i.e., minimal κ) and desired mechanical stability (von Mises Stress σ ≤ τ, where τ is some preset threshold). In black-box optimization problems (such as the material design under consideration), the goal is to find a posterior distribution q of samples satisfying a list of equality and inequality constraints: ψ j (x) = y k, j = 1... C e, and φ k (x) ≤ b k, k = 1... C i where x ∈ Ω and Ω ⊂ R d is a bounded domain. We assume a prior distribution p 0 (whose analytical form is known). The main challenge in black-box optimization is that the functions ψ j and φ k can be only evaluated point-wise, and neither do we have functional forms nor access to their gradients. For example, ψ and φ might be obtained via aggregating some statistics on the solution of a nonlinear PDE given by a complex solver. To make the problem of learning under constraints tractable, we choose Lagrangian parameters λ j > 0 and obtain the following relaxed objective: The formulation in Eq. 1 is similar in spirit to the posterior regularization framework of;. However, we highlight two differences: (i) our focus is on constrained settings (where Ω is bounded), and (ii) we assume a black-box setting. We first obtain: Lemma 1 (Constraint Satisfaction as Sampling from a Gibbs Distribution). The solution to the distribution learning problem given in Eq. 1 is given by: where Lemma 1 shows that the constraint satisfaction problem formulated in Eq. 1 amounts to sampling from a Gibbs distribution defined on a compact support given in Eq. 2. Sampling from a Gibbs 1 Note that both properties κ and σ for a given configuration are obtained by numerically solving highly non-linear PDEs. The material configuration is defined by the pore locations, the material used, and the response of the material to heat (thermal) or stress (mechanical) flows. distribution (also known as Boltzmann distribution) has a long history using Langevin dynamics. In the white-box setting when the functions defining the constraints have explicit analytical forms as well as their gradients, Langevin dynamics for Gibbs distribution sampling defined on a compact domain Ω and their mixing properties were actively studied in;. In the next Section, we provide a more detailed review. Remark 1 (Relation to Bayesian Optimization). While in Bayesian optimization we are interested in finding a point that satisfies the constraints, in our setting we are interested in finding a distribution of candidate samples that satisfy (black-box) constraints. See for more details. Remark 2. For the rest of the paper, we will assume p 0 to be the uniform distribution on Ω, which means that its gradients are zero on the support of the domain Ω. Otherwise, if p 0 is known and belongs to, for instance, an exponential family or a generative model prior (such as normalizing flows), we can sample from π using a mixture of black-box sampling on the constraints (ψ j, φ k) and white-box sampling on log(p 0). We review in this section Langevin dynamics in the unconstrained case (Ω = R d) and the constrained setting (Ω ⊂ R d). Below, · denotes the Euclidean norm unless otherwise specified. We are interested in sampling from Preliminaries. We give here assumptions, definitions and few preliminary known facts that will be useful later. Those assumptions are commonly used in Langevin sampling analysis (; ; ;). We assume Ω is a convex such that 0 ∈ Ω, Ω contains a Euclidean ball of radius r, and Ω is contained in a Euclidean ball of radius R. (For example, Ω might encode box constraints.) The projection onto Ω, P Ω (x) is defined as follows: for all x ∈ Ω, P Ω (x) = arg min z∈Ω x − z 2. Let R = sup x,x ∈Ω ||x − x || < ∞. We assume that U is convex, β-smooth, and with bounded gradients: The Total Variation (TV) distance between two measures µ, ν is defined as follows: Unconstrained Langevin Dynamics. In the unconstrained case, the goal is to sample from a Gibbs distribution π(x) = exp(−U (x))/Z that has unbounded support. This sampling can be done via the Langevin Monte Carlo (LMC) algorithm, which is given by the following iteration: where ξ k ∼ N (0, I d), η is the learning rate, and λ > 0 is a variance term. Constrained Langevin Dynamics. In the constrained case, the goal is to sample from π(x) = exp(−U (x))/Z1 x∈Ω,. We discuss two variants: Projected Langevin Dynamics. Similar to projected gradient descent, introduced Projected Langevin Monte Carlo (PLMC) and proved its mixing propreties towards the stationary distribution π. PLMC is given by the following iteration: In essence, PLMC consists of a single iteration of LMC, followed by a projection on the set Ω using the operator P Ω. Proximal Langevin Dynamics. Similar to proximal methods in constrained optimization, introduced Proximal LMC (ProxLMC) that uses the iteration: where η is the step size and γ is a regularization parameter. In essence, ProxLMC performs an ordinary LMC on where i Ω (x) = 0 for x ∈ Ω and i Ω (x) = ∞ for x / ∈ Ω. Therefore, the update in Eq. 6 is a regular Langevin update (as in Eq. 4) with potential gradient We denote by µ PLMC K and µ ProxLMC K the distributions of X K obtained by iterating Eq. 5 and Eq. 6 respectively. Under Assumptions A and B, both these distributions converge to the target Gibbs distribution π in the total variation distance. In particular, showed that for η =Θ(R 2 /K), we obtain: showed that for 0 < η ≤ γ(1 + β 2 γ 2) −1, we obtain: where the notation α n =Ω(β n) means that there exists c ∈ R, C > 0 such that α n ≥ Cβ n log c (β n). We now introduce our variants of constrained LMC for the black-box setting where explicit potential gradients are unavailable. We explore in this paper two strategies for approximating the gradient of U in the black-box setting. In the first strategy, we borrow ideas from derivative-free optimization (in particular, evolutionary search). In the second strategy we learn a surrogate deep model that approximates the gradient of the potential. Below, let G: Ω → R d be a vector valued function that approximates the gradient of the potential, ∇ x U. We make: Surrogate Projected Langevin Dynamics. Given Y 0, the Surrogate Projected LMC (S-PLMC) replaces the potential gradient ∇ x U in Eq. 5 with the surrogate gradient G: Surrogate Proximal Langevin Dynamics. Similarly, the Surrogate Proximal LMC (S-ProxLMC) replaces the unknown potential gradient ∇ x U in Eq. 6 with the gradient surrogate G: We now present our main theorems on the approximation properties of surrogate LMC (S-PLMC, and S-ProxLMC). We do so by bounding the total variation distance between the trajectories of the surrogate Langevin dynamics (S-PLMC, and S-ProxLMC) and the true LMC dynamics (PLMC and ProxLMC). Theorem 1 is an application of techniques in Stochastic Differential Equations (SDE) introduced in and is mainly based on a variant of Grisanov's Theorem for change of measures and Pinsker's Inequality that bounds total variation in terms of Kullback-Leibler divergence. Theorem 1 (S-PLMC and S-ProxLMC Mixing Properties). Under Assumption C, we have: be the distribution of the random variable X K obtained by iterating PLMC Eq. 5, and µ S-PLMC K be the distribution of the random variable Y K obtained by iteration S-PLMC given in Eq. 9. We have: 2. S-ProxLMC Convergence. Let µ ProxLMC K be the distribution of the random variable X K obtained by iterating ProxLMC Eq. 6, and µ S-ProxLMC K be the distribution of the random variable Y K obtained by iterating S-ProxLMC given in Eq. 10. We have: From Theorem 1, we see that it suffices to approximate the potential gradient ∇ x U (X) (and not the potential U (X)) in order to guarantee convergence of surrogate-based Langevin sampling. Using the triangle inequality, and combining Theorem 1 and bounds in Eqs 7 and 8 we obtain: Theorem 2. (Convergence of Surrogate Constrained LMC to the Gibbs distribution.) Under assumptions A,B and C we have: we have: In zero-order optimization (; ; ;), one considers the Gaussian smoothed potential, and its gradient is given by where g 1,... g n are i.i.d. standard normal vectors. Zero-Order sampling from log-concave densities was recently studied in. We extend it here to the constrained sampling case of log-concave densities with compact support. We define Constrained Zero-Order Projected LMC (Z-PLMC) and Zero-Order Proximal LMC (Z-ProxLMC) by setting G(x) =Ĝ n U(x) in Eq. 9 and Eq. 10 respectively. Lemma 2 (Zero-Order Gradient Approximation ). Under Assumption B, we have for all x ∈ Ω: Thanks to Lemma 2 that ensures uniform approximation of gradients in expectation, we can apply Theorem 2 and get the following corollary for Z-PLMC and Z-ProxLMC: Corollary 1 (Zero-order Constrained Langevin approximates the Gibbs distribution). Under As- we have the following bounds in expectation: 2. Set λ = 1, and we have: Remark 3. For simplicity, we state the above bound in terms of expectations over the randomness in estimating the gradients. It is possible to get finite-sample bounds using the Vector Bernstein concentration inequality, coupled with covering number estimates of Ω but omit them due to space. Despite its theoretical guarantees, zero-order constrained Langevin (Z-PLMC and Z-ProxLMC) has a prohibitive computation cost as it needs O(nK) black-box queries (in our case, invocations of a nonlinear PDE solver). To alleviate this issue, we introduce in this Section a neural surrogate model as an alternative to the gradient of the true potential. From Theorem 2, we saw that in order to guarantee the convergence of constrained Langevin dynamics, we need a good estimate of the gradient of the potential of the Gibbs distribution. Recall that the potential given in Lemma 1 depends on ψ j and φ k, which are scalar outputs of computationally heavy PDE solvers in our material design problem. To avoid this, we propose to train surrogate neural network models approximating each PDE output and their gradients. Concretely, suppose we are given a training set S for a PDE solver for the property ψ (dropping the index j for simplicity): where ρ Ω is the training distribution andĜ n ψ is the zero-order estimate of the gradient of ψ given in Eq. 13. We propose to learn a surrogate model belonging to a function class H θ,f θ ∈ H θ, that regresses the value of ψ and matches the zero-order gradient estimates as follows: The problem in Eq. 17 was introduced and analyzed in where H θ is a ball in a Reproducing Kernel Hilbert Space (RKHS). , we refer to this type of learning as Hermite Learning. In the deep learning community, this type of learning is called Jacobian matching and was introduced in; where H θ is a deep neural network parameterized with weights θ. When f θ is a deep network, we can optimize this objective efficiently using common deep learning frameworks (PyTorch, TensorFlow). have shown that when H θ is an RKHS ball and whenỹ i = ∇ x ψ(x i) are exact gradients, for a sufficiently large training set with N = O(1/ 1/(2rζ) ) (where r, ζ are exponents in that depend on the regularity of the function ψ). Under the assumption that ψ ∈ H θ we have: Since we are using inexact zero-order gradients, we will incur an additional numerical error that is also bounded as shown in Lemma 2. While Jacobian matching of zero-order gradients is a sound approach, it remains expensive to construct the dataset, as we need for each point to have 2n + 1 queries of the PDE solver. We exploit in this section the Taylor learning framework of gradients that was introduced in; , and. In a nutshell, suggests to learn a surrogate potential f θ and gradient G Λ that are consistent with the first-order taylor expansion. Given a training set suggest the following objective: where, H θ is an RKHS ball of scalar valued functions, and H d Λ is an RKHS ball of vector valued functions. Under mild assumptions, shows that we have for We simplify the problem in Eq. 18 and propose the following two objective functions and leverage the deep learning toolkit to parameterize the surrogate f θ: The objective in Eq. 19 uses a single surrogate to parameterize the potential and its gradient. The objective in Eq. 20 is similar in spirit to the Jacobian matching formulation in the sense that it adds a regularizer on the gradient of the surrogate to be consistent with the first-order Taylor expansion in local neighborhoods. The advantage of the Taylor learning approach is that we do not need to perform zero-order estimation of gradients to construct the training set and we rely instead on first-order approximation in local neighborhood. Consider the surrogate model f θ obtained via Hermite Learning (Eq. 17) or via Taylor learning (Eqs 18, 19, 20). We are now ready to define the surrogate model LMC by replacing in the constrained Langevin dynamics in Eqs 9 and 10. Both Hermite and Taylor learning come with theoretical guarantees when the approximation function space is an RKHS under some mild assumptions on the training distribution and the regularity of the target function ψ. In Hermite learning (Theorem 2 in) we have: ) (where exponents ζ, r ∈ depend on regularity of ψ). In Taylor Learning with the objective function given in Eq. 18 (Proposition 7 in we have: . In order to apply Theorem 2 we need this gradient approximation error to hold in expectation on all intermediate distributions in the Langevin sampling. Hence, we need the following extra-assumption on the training distribution p Ω : Assumption D: Assume we have a learned surrogate G on training distribution ρ Ω such that 2 ≤ . Assume ρ Ω (x) > 0, ∀x ∈ Ω and that it is a dominating measure of Langevin (PLMC, S-PLMC, Prox-LMC, S-ProxLMC) intermediate distributions µ k, i.e. there exists C > 0 such that: and hence we can apply Theorem 2 for δ = C, and we obtain ε-approximation of the target Gibbs distribution in terms of total variation distance. Remark 4. Assumption D on the -approximation of the gradient can be achieved for a large enough training set N, when we use Hermite learning in RKHS under mild assumptions and in Taylor learning. The assumption on the dominance of the training distribution is natural and means that we need a large training set that accounts to what we may encounter in Surrogate LMC iterations. In what follows we refer to surrogate constrained LMC, as x-PLMC or x-ProxLMC where x is one of four suffixes ({Z-Hermite, Taylor-2, Taylor-1, Taylor-Reg}). Zero-Order Methods. Zero-order optimization with Gaussian smoothing was studied in and in the convex setting. Non-convex zero order optimization was also addressed in. The closest to our work is the zero-order introduced recently for black-box sampling from log concave density. The main difference in our setting, is that the density has a compact support and hence the need to appeal to projected LMC and Proximal LMC . It is worth nothing that introduced recently mirror Langevin sampling that can also be leveraged in our framework. Gradients and Score functions Estimators. We used the approach of gradient distillation and learning gradients of , since they are convenient for training on different constraints and they come with theoretical guarantees. However, other approaches can be also leveraged such as the score matching approach for learning the gradient of the log likelihood (Hyvärinen, 2005) and other variants appealing to dual embeddings . Estimating gradients can be also performed using Stein's method as in , or via maintaining a surrogate of the gradient as in Stein descent without gradient . Optimization approaches. Due to space limitation, we restrict the discussion to the optimization methods that are most commonly and recently used for optimal material (or molecule) design. A popular approach to deal with optimization of expensive black-box functions is Bayesian Optimization (BO) (; ;). The standard BO protocol is comprised of estimating the black-box function from data through a probabilistic surrogate model, usually a Gaussian process, and maximizing an acquisition function to decide where to sample next. BO is often performed over a latent space, as in (Gómez-). Hernández- proposed an information-theoretic framework for extending BO to address optimization under black-box constraints, which is close to current problem scenario. Genetic Algorithms (GA), a class of meta-heuristic based evolutionary optimization techniques, is another widely used approach for generating (material) samples with desired property and has been also used for handling optimization under constraints . However, GA typically requires a large number of function evaluations, can get stuck in local optima, and does not scale well with complexity. has used deep reinforcement learning technique of Deep Q-networks to optimize molecules under a specific constraint using desired properties as rewards. The advantage of our framework is that we obtain a distribution of optimal configurations (as opposed to a single optimized sample) that does not rely on training on a specific pre-existing dataset and can be further screened and tested for their optimality for the task at hand. In this section, we demonstrate the usability of our black-blox Langevin sampling approach for the design of nano-porous configurations. We first show the performance of the surrogate models in learning the potential function, showcasing the using four different variants: standard regression, Taylor regularization, Taylor-1 and Taylor-2. We then show how well the surrogate-based Langevin MC generates new samples under the thermal and mechanical constraints. We compare the sample quality on multiple criteria between the surrogate and zero-order approaches with either projection or proximal update step. Data. We want to learn surrogate models to approximate the gradient of the potential from data. To this end, we generate a dataset of 50K nano-porous structures, each of size 100nm × 100nm. One such example is displayed in Fig. 1. Number of pores is fixed to 10 in this study and each pore is a square with a side length of 17.32nm. We sample the pore centers uniformly over the unit square and construct the corresponding structure after re-scaling them appropriately. Then, using the solvers OpenBTE and Summit , we obtain for each structure x a pair of values: thermal conductivity κ and von Mises stress σ. Finally, we collect two datasets: with the same inputs x i's and N = 50K samples. More details are given in Appendices B and C on the PDEs and their corresponding solvers. Features. The pore locations are the natural input features to the surrogate models. Apart from the coordinates, we also derive some other features based on physical intuitions. For example, the distances between pores and the alignment along axes are informative of thermal conductivity . As such, we compute pore-pore distances along each coordinate axis and add them as additional features. Surrogate gradient methods. We use feed-forward neural networks to model the surrogates since obtaining gradients for such networks is efficient thanks to automatic differentiation frameworks. We use networks comprised of 4 hidden layers with sizes 128, 72, 64, 32 and apply the same architecture to approximate the gradients for κ and σ separately. The hidden layers use ReLU activations whereas sigmoid was used at the output layer (after the target output is properly normalized). For the Taylor-2 variant (in Eq. 18), we have an additional output vector of the same size as the input for the gradient prediction. The networks are trained on the corresponding objective functions set up earlier by an Adam optimizer with learning rate 10 −4 and decay 1.0. We fine-tune the networks with simple grid-search and select the best models for comparison. Due to the space constraint, we present the in Appendix A and emphasize that Z-Hermite is not included in the entire comparison but in a small experiment performed with a more lightweight OpenBTE version. Incorporating constraints and comparison metrics. We demonstrate the usability of our proposed black-box Langevin sampling for the design of nano-configurations under thermal conductivity and mechanical stability constraints that are provided by the corresponding PDE solvers. To compare sampling outcomes, we use the following metrics. We report the minimum value of κ and Monte Carlo estimates for both κ and σ to compare the samples generated by different sampling methods and surrogate models. The Monte Carlo estimates are computed on 20 samples. Single constraint. Our first task is to design nano-configurations under the thermal conductivity constraint where we want κ as low as possible in order to achieve high thermo-electric efficiency. From the posterior regularization formulation Section 2, we pose the constraint satisfaction as sampling from the following Gibbs distribution: where p 0 (x) is the uniform distribution over the unit square, which is equivalent to the Poisson process of 10 pores on the square, and κ(x) is the thermal conductivity we want to minimize. Starting from 20 samples initialized from p 0 (x), we run our proposed black-box Langevin MCs and obtain 20 new realizations from the target distribution π(x). We use four different surrogates (including simple regression, Taylor-Reg, Taylor-1 and zero-order) and each surrogate with either projection or proximal update. We show the summary statistics of these samples in Table 1. The regression-PMLC in the first row and regression-ProxLMC in the fifth represent the sampling where the surrogate model are fitted on solely the mean square error objective. In all methods, we set λ = 100, the step size η = 1e−3 and the exponential decay rate 0.8. Since keeping track of the true κ value is expensive, we stop after K = 10 iterations. We first observe that the regression-based method (PLMC, ProxLMC) is less effective than the others simply because they do not have an implicit objective for approximating the gradients. Taylor-Reg and Taylor-1 demonstrate its effectiveness in approximating the gradient and are able to achieve lower thermal conductivity. In particular, Taylor-1-ProxLMC and Zero-order-PLMC perform in the similar range in terms of the minimum achieved, but the learned surrogate offers 17x speed up (per sample) over zero order methods. Due to the space limit, we do not report Taylor-2 in Table 1, and note that Taylor-2 works in the similar vein as Taylor-1. Multiple constraints. Achieving the minimal thermal conductivity can be fulfilled without much difficulty (e.g. structures with all pores aligned along the vertical axis), but such structures are often mechanically unstable. In the next step, we study whether adding more (conflicting) constraints helps us design better nano-configurations. Hence, we consider both thermal conductivity κ and mechanical stability provided via von Mises stress σ. We want a sample x that minimizes κ(x) to achieve high thermo-electric efficiency while maintaining σ(x) less than some threshold (which we explain below). Like the single constraint case, we pose this as sampling from the following Gibbs distribution: where p 0 (x) is the same as above, σ(x) is the von Mises stress and τ is a threshold on the maximum value of σ. With this framework, we relax the inequality constraint to the Hinge loss term on von Mises stress. The are summarized in Table 2. Note that all the surrogate Langevin MCs are initialized from the same set of 20 samples as above. In this experiment, we set τ = 0.5, λ 1 = 100, λ 2 = 10 the step size η = 1e−3 and the exponential decay rate 0.8. Comparing with Table 1, one can see that not only better κ be achieved but also the σ can be reduced simultaneously. These suggest that our approach can effectively sample new configurations under multiple competing constraints. Examples of new nano-configurations are show in Fig. 1 Table 2: Summary statistics of 20 new samples obtained by our sampling method on π(x) with κ and σ constraints Eq. 22. The starting samples are reused from the single constraint case (min κ = 0.0759, mean κ = 0.1268, and mean σ = 0.8181; note that σ can be as high as 16.) In this paper we introduced Surrogate-Based Constrained Langevin Sampling for black-box sampling from a Gibbs distribution defined on a compact support. We studied two approaches for defining the surrogate: the first through zero-order methods and the second via learning gradient approximations using deep neural networks. We showed the proofs of convergence of the two approaches in the log-concave and smooth case. While zero-order Langevin had prohibitive computational cost, learned surrogate model Langevin enjoy a good tradeoff of lightweight computation and approximation power. We applied our black-box sampling scheme to the problem of nano-material configuration design, where the black box constraints are given by expensive PDE solvers, and showed the efficiency and the promise of our method in finding optimal configurations. Among different approaches for approximating the gradient, the zero-order ones (PLMC, ProxLMC) show overall superior performance, at a prohibitive computational cost. We established that the deep the surrogate (Taylor-1 ProxLMC) is a viable alternative to zero-order methods, achieving reasonable performance, and offering 15x speedup over zero-order methods. Surrogate gradient methods We use feed-forward neural networks to model the surrogates since obtaining gradients for such networks is efficient thanks to automatic differentiation frameworks. We use networks comprised of 4 hidden layers with sizes 128, 72, 64, 32 and apply the same architecture to approximate the gradients for κ and σ separately. The hidden layers compute ReLU activation whereas sigmoid was used at the output layer (after the target output is properly normalized). For the Taylor-2 variant (in Eq. 18), we have an output vector for the gradient prediction. The networks are trained on the corresponding objective functions set up earlier by Adam optimizer with learning rate 10 −4 and decay 1.0. We fine-tune the networks with simple grid-search and select the best models for comparison. As emphasized throughout, our focus is more on approximating the gradient rather than learning the true function. However, we need to somehow evaluate the surrogate models on how well they generalize on a hold-out test set. Like canonical regression problems, we compare the surrogate variants against each other using root mean square error (RMSE) on the test set. Figures 2 and 3 shows the . The left figure shows RMSE for predicting κ and the right one shows RMSE for the von Mises stress σ. We can see that the Taylor-Reg generalizes better and also converges faster than Taylor-1 and Taylor-2 to target RMSE for κ, while all methods similarly for σ prediction. This is reasonable because the objectives of Taylor-1 and Taylor-2 are not to optimize the mean square error, which we evaluate on here. Figure 3 shows the learning in terms of sample complexity. Again, Taylor-Reg outperforms Taylor-1 and Taylor-2 for κ prediction. In contrast, most models work similarly for σ regression, particularly when the training size is reduced to 50% (25K). Effectiveness of Z-Hermite learning Notice that Z-Hermite learning is not included in this comparison and as a surrogate model in the black-blox Langevin sammpling in Section 8. The reason is that apart from the usual sample pair (x i, y i), we need the gradientỹ i (See Eq. 17). Since we can query the solvers, this gradient can only be estimated using finite difference. For both κ and σ in our experiment, obtaining such data is extremely expensive. As a consequence, we do not have the full of the Z-Hermite model. Instead, we ran a separate study to show the effectiveness of Z-Hermite surrogate LMC on a smaller data with a lightweight OpenBTE version (0.9.55). The in Table 3 shows the working of Z-Hermite learning in learning the gradient of κ(x). Here, the entropy is based nearest neighbor estimate to demonstrate the diversity of the pore centers in the unit square. With the (x p, y p)-coordinates of each pore p, the entropy estimate is given by: Mean A hybrid algorithm between zero-order and Taylor-1 surrogate We can see in Tables 1, 2 and 3 the trade-off between computation and accuracy of our approach. While zero-order PLMC and ProxLMC can achieve the lowest thermal conductivity, their computational costs are prohibitive. In contrast, deep surrogate models (including Taylor-Reg, Taylor-1) are far more time-efficient but slightly worse in terms of achieving the optimal κ. To mitigate the trade-off, we propose a simple hybrid method that combines the best of the zero-order and Taylor-1 surrogate models. The algorithm is shown in Figure A that alternates between using the gradient from the zero-order estimate and the gradient of the deep surrogate depending on whether taking this step would decrease the potential function (i.e. κ). We show and compare the achieved κ and running time in Table 3. Examples of the samples generated by Zero-order PLMC, Taylor-1 PLMC and the hybrid method are also depicted in Figure 4. The hybrid achieves the thermal conductivity that is lower than Taylor-1 PMLC while running almost 2x faster than zero-order PLMC. This suggests that the hybrid strategy offers a better trade-off in accuracy and computation. One way to further improve the hybrid is to collect the zero-order gradients while mixing and re-update the surrogate with Z-Hermite learning. Algorithm 1 A hybrid PLMC algorithm alternating between zero-order and Taylor-1 surrogate gradients. Train a network f θ (x) with Taylor-1 Randomly sample x 0 from the uniform p(x) Perform a Langevin dynamic step Additional generated samples We show additional configurations generated by our sampling approach (Taylor-Reg ProxLMC, Taylor At the nanoscale, heat transport may exhibit strong ballistic behaviour and a non-diffusive model must be used . In this work we use the Boltzmann transport equation under the relaxation time approximation and in the mean-free-path (MFP) formulation Λŝ where T (Λ) is the effective temperature associated to phonons with MFP Λ and directionŝ; the notation. stands for an angular average. The coefficients α(Λ) are given by where K(Λ) is the bulk MFP distribution. In general, such a quantity can span several orders of magnitude; however, for simplicity we assume the gray model, i.e. all phonons travel with the same MFP, Λ 0. Within this approximation, we have K(Λ) = κ bulk δ(Λ − Λ 0). In this work we choose Λ 0 = 10 nm, namely as large as the unit cell, so that significant phonons size effects occur. With no loss of generality, we set κ bulk = 1 Wm −1 K −1. Eq. 23 is an integro-differential PDE, which is solved iteratively for each phonon direction over an unstructured mesh . We apply periodic boundary conditions along the unit cell while imposing a difference of temperature of ∆T = 1 K along the x-axis. At the pores' walls we apply diffusive boundary conditions. Upon convergence, the effective thermal conductivity is computed using Fourier's law, i.e. where J = (κ bulk /Λ 0) T (Λ 0)ŝ n is the heat flux, L is the size of the unit cell, A is the area of the cold contact (with normaln). Throughout the text we use the quantity κ = κ eff /κ bulk as a measure of phonon size effects. We model mechanical stress by using the continuum linear elasticity equations where f i is the body force (which is zero in this case), and σ ij is the stress tensor. Note that we used the Einstein notation, i.e. repeated indexes are summed over. The strain kl is related to the stress via the fourth-rank tensor elastic constant C ijkl σ ij = C ijkl kl. The strain is then related to the displacement u via kl = 1 2 We apply periodic boundary conditions along the unit-cell and applied solicitation is a small in-plane expansion. Once the stress tensor is calculated, we compute the von Mises stress as where σ i are the principal stress axis. As a mechanical stability estimator we use σ = max x∈D (σ V M) where D is the simulation domain. To avoid material's plasticity, σ needs to be smaller than the yield stress of a given material. For mechanical simulation we used the SUMIT code (Setting first order optimality conditions on q, we have for x ∈ Ω: Hence we have:, x ∈ Ω and q(x) = 0, x / ∈ Ω, First order optimality on η give us: Ω q(x) = 1, we conclude by setting e exp(−η) = Z. Proof of Theorem 1 1) Projected Langevin. Let us define the following continuous processes by interpolation of X k and Y K (Piecewise constant): whereŨ t (X) = − ∞ k=0 ∇ x U (X kη)1 t∈[kη,(k+1)η] (t). Similarly let us define: where G t (Ỹ) = − Note that: Hence we have: Assume that X 0 = Y 0 there exists Q such that, X T = Q({W t} t∈ [0,T] ) and Y T = Q((W t) t∈[0,T] ). Let µX T be the law ofX t∈ [0,T]. Same for µỸ T. The proof here is similar to the proof of Lemma 8 in . By the data processing inequality we have:

We propose surrogate based Constrained Langevin sampling with application in nano-porous material configuration design.

Represent the short summary for finding the corresponding scientific article (long)!

We propose a new class of inference models that iteratively encode gradients to estimate approximate posterior distributions.


Question: Represent the following document

Question: There is growing interest in geometrically-inspired embeddings for learning hierarchies, partial orders, and lattice structures, with natural applications to transitive relational data such as entailment graphs. Recent work has extended these ideas beyond deterministic hierarchies to probabilistically calibrated models, which enable learning from uncertain supervision and inferring soft-inclusions among concepts, while maintaining the geometric inductive bias of hierarchical embedding models. We build on the Box Lattice model of , which showed promising in modeling soft-inclusions through an overlapping hierarchy of sets, parameterized as high-dimensional hyperrectangles (boxes). However, the hard edges of the boxes present difficulties for standard gradient based optimization; that work employed a special surrogate function for the disjoint case, but we find this method to be fragile. In this work, we present a novel hierarchical embedding model, inspired by a relaxation of box embeddings into parameterized density functions using Gaussian convolutions over the boxes. Our approach provides an alternative surrogate to the original lattice measure that improves the robustness of optimization in the disjoint case, while also preserving the desirable properties with respect to the original lattice. We demonstrate increased or matching performance on WordNet hypernymy prediction, Flickr caption entailment, and a MovieLens-based market basket dataset. We show especially marked improvements in the case of sparse data, where many conditional probabilities should be low, and thus boxes should be nearly disjoint. Embedding methods have long been a key technique in machine learning, providing a natural way to convert semantic problems into geometric problems. Early examples include the vector space BID17 and latent semantic indexing BID4 ) models for information retrieval. Embeddings experienced a renaissance after the publication of Word2Vec BID12, a neural word embedding method BID2 BID13 ) that could run at massive scale. Recent years have seen an interest in structured or geometric representations. Instead of representing e.g. images, words, sentences, or knowledge base concepts with points, these methods instead associate them with more complex geometric structures. These objects can be density functions, as in Gaussian embeddings BID21 BID0, convex cones, as in order embeddings BID20 BID9, or axis-aligned hyperrectangles, as in box embeddings BID22 BID18. These geometric objects more naturally express ideas of asymmetry, entailment, ordering, and transitive relations than simple points in a vector space, and provide a strong inductive bias for these tasks. In this work, we focus on the probabilistic Box Lattice model of BID22, because of its strong empirical performance in modeling transitive relations, probabilistic interpretation (edges in a relational DAG are replaced with conditional probabilities), and ability to model complex joint probability distributions including negative correlations. Box embeddings (BE) are a generalization of order embeddings (OE) BID20 and probabilistic order embeddings (POE) BID9 that replace the vector lattice ordering (notions of overlapping and enclosing convex cones) in OE and POE with a more general notion of overlapping boxes (products of intervals).While intuitively appealing, the "hard edges" of boxes and their ability to become easily disjoint, present difficulties for gradient-based optimization: when two boxes are disjoint in the model, but have overlap in the ground truth, no gradient can flow to the model to correct the problem. This is of special concern for (pseudo-)sparse data, where many boxes should have nearly zero overlap, while others should have very high overlap. This is especially pronounced in the case of e.g. market basket models for recommendation, where most items should not be recommended, and entailment tasks, most of which are currently artificially resampled into a 1:1 ratio of positive to negative examples. To address the disjoint case, BID22 introduce an ad-hoc surrogate function. In contrast, we look at this problem as inspiration for a new model, based on the intuition of relaxing the hard edges of the boxes into smoothed density functions, using a Gaussian convolution with the original boxes. We demonstrate the superiority of our approach to modeling transitive relations on WordNet, Flickr caption entailment, and a MovieLens-based market basket dataset. We match or beat existing state of the art , while showing substantial improvements in the pseudosparse regime. As mentioned in the introduction, there is much related work on structured or geometric embeddings. Most relevant to this work are the order embeddings of BID20, which embed a nonprobabilistic DAG or lattice in a vector space with order given by inclusion of embeddings' forward cones, the probabilistic extension of that model due to BID9, and the box lattice or box embedding model of BID22, which we extend. Concurrently to BID22, another hyperrectangle-based generalization of order embeddings was proposed by BID18, also called box embeddings. The difference between the two models lies in the interpretation: the former is a probabilistic model that assigns edges conditional probabilities according to degrees of overlap, while the latter is a deterministic model in the style of order embeddings -an edge is considered present only if one box entirely encloses another. Methods based on embedding points in hyperbolic space BID16 BID5 have also recently been proposed for learning hierarchical embeddings. These models, similar to order embeddings and the box embeddings of BID18, are nonprobabilistic and optimize an energy function. Additionally, while the negative curvature of hyperbolic space is attractively biased towards learning tree structures (since distances between points increase the farther they are from the origin), this constant curvature makes the models not as suitable for learning non-treelike DAGs. Our approach to smoothing the energy landscape of the model using Gaussian convolution is common in mollified optimization and continuation methods, and is increasingly making its way into machine learning models such as Mollifying Networks BID7, diffusion-trained networks BID14, and noisy activation functions BID6.Our focus on embedding orderings and transitive relations is a subset of knowledge graph embedding. While this field is very large, the main difference of our probabilistic approach is that we seek to learn an embedding model which maps concepts to subsets of event space, giving our model an inductive bias especially suited for transitive relations as well as fuzzy concepts of inclusion and entailment. We begin with a brief overview of two methods for representing ontologies as geometric objects. First, we review some definitions from order theory, a useful formalism for describing ontologies, then we introduce the vector and box lattices. FIG0 shows a simple two-dimensional example of these representations. A non-strict partially ordered set (poset) is a pair P,, where P is a set, and is a binary relation. For all a, b, c ∈ P, Reflexivity: a a Antisymmetry: a b a implies a = b Transitivity: a b c implies a cThis generalizes the standard concept of a totally ordered set to allow some elements to be incomparable. Posets provide a good formalism for the kind of acyclic directed graph data found in many knowledge bases with transitive relations. A lattice is a poset where any subset of elements has a single unique least upper bound, and greatest lower bound. In a bounded lattice, the set P contains two additional elements, (top), and ⊥ (bottom), which denote the least upper bound and greatest lower bound of the entire set. A lattice is equipped with two binary operations, ∨ (join), and ∧ (meet). a∨b denotes the least upper bound of a, b ∈ P, and a ∧ b denotes their greatest lower bound. A bounded lattice must satisfy these properties: DISPLAYFORM0 Note that the extended real numbers, R ∪ {−∞, ∞}, form a bounded lattice (and in fact, a totally ordered set) under the min and max operations as the meet (∧) and join (∨) operations. So do sets partially ordered by inclusion, with ∩ and ∪ as ∧ and ∨. Thinking of these special cases gives the intuition for the fourth property, absorption. The ∧ and ∨ operations can be swapped, along with reversing the poset relation, to give a valid lattice, called the dual lattice. In the real numbers this just corresponds to a sign change. A semilattice has only a meet or join, but not both. Note. In the rest of the paper, when the context is clear, we will also use ∧ and ∨ to denote min and max of real numbers, in order to clarify the intuition behind our model. A vector lattice, also known as a Riesz space , or Hilbert lattice when the accompanying vector space has an inner product, is a vector space endowed with a lattice structure. A standard choice of partial order for the vector lattice R n is to use the product order from the underlying real numbers, which specifies for all x, y ∈ R n x y ⇐⇒ ∀i ∈ {1..n}, x i ≤ y i Under this order, meet and join operations are pointwise min and max, which gives a lattice structure. In this formalism, the Order Embeddings of BID20 embed partial orders as vectors using the reverse product order, corresponding to the dual lattice, and restrict the vectors to be positive. The vector of all zeroes represents, and embedded objects become "more specific" as they get farther away from the origin. FIG0 demonstrates a toy, two-dimensional example of the Order Embedding vector lattice representation of a simple ontology. Shading represents the probability measure assigned to this lattice in the probabilistic extension of BID9. Vilnis et al. FORMULA2 introduced a box lattice, wherein each concept in a knowledge graph is associated with two vectors, the minimum and maximum coordinates of an axis-aligned hyperrectangle, or box (product of intervals).Using the notion of set inclusion between boxes, there is a natural partial order and lattice structure. To represent a box x, let the pairs (x m,i, x M,i) be the maximum and minimum of the interval at each coordinate i. Then the box lattice structure (least upper bounds and greatest lower bounds), with ∨ and ∧ denoting max and min when applied to the scalar coordinates, is DISPLAYFORM0 Here, denotes a set (cartesian) product -the lattice meet is the largest box contained entirely within both x and y, or bottom (the empty set) where no intersection exists, and the lattice join is the smallest box containing both x and y. To associate a measure, marginal probabilities of (collections of) events are given by the volume of boxes, their complements, and intersections under a suitable probability measure. Under the uniform measure, if event x has an associated box with interval boundaries (x m, x M), the probability p(x) is given by n i (x M,i − x m,i). Use of the uniform measure requires the boxes to be constrained to the unit hypercube, so that p(x) ≤ 1. p(⊥) is taken to be zero, since ⊥ is an empty set. As boxes are simply special cases of sets, it is intuitive that this is a valid probability measure, but it can also be shown to be compatible with the meet semilattice structure in a precise sense BID10. When using gradient-based optimization to learn box embeddings, an immediate problem identified in the original work is that when two concepts are incorrectly given as disjoint by the model, no gradient signal can flow since the meet (intersection) is exactly zero, with zero derivative. To see this, note that for a pair of 1-dimensional boxes (intervals), the volume of the meet under the uniform measure p as given in Section 3.3 is DISPLAYFORM1 where m h is the standard hinge function, m h (x) = 0 ∨ x = max(0, x).The hinge function has a large flat plateau at 0 when intervals are disjoint. This issue is especially problematic when the lattice to be embedded is (pseudo-)sparse, that is, most boxes should have very little or no intersection, since if training accidentally makes two boxes disjoint there is no way to recover with the naive measure. The authors propose a surrogate function to optimize in this case, but we will use a more principled framework to develop alternate measures that avoid this pathology, improving both optimization and final model quality. DISPLAYFORM2 Figure 2: One-dimensional example demonstrating two disjoint indicators of intervals before and after the application of a smoothing kernel. The area under the purple product curve is proportional to the degree of overlap. The intuition behind our approach is that the "hard edges" of the standard box embeddings lead to unwanted gradient sparsity, and we seek a relaxation of this assumption that maintains the desirable properties of the base lattice model while enabling better optimization and preserving a geometric intuition. For ease of exposition, we will refer to 1-dimensional intervals in this section, but the carry through from the representation of boxes as products of intervals and their volumes under the associated product measures. The first observation is that, considering boxes as indicator functions of intervals, we can rewrite the measure of the joint probability p(x ∧ y) between intervals x = [a, b] and y = [c, d] as an integral of the product of those indicators: DISPLAYFORM3 since the product has support (and is equal to 1) only in the areas where the two intervals overlap. A solution suggests itself in replacing these indicator functions with functions of infinite support. We elect for kernel smoothing, specifically convolution with a normalized Gaussian kernel, equivalent to an application of the diffusion equation to the original functional form of the embeddings (indicator functions) and a common approach to mollified optimization and energy smoothing BID15 BID7 BID14. This approach is demonstrated in one dimension in Figure 2.Specifically, given x = [a, b], we associate the smoothed indicator function DISPLAYFORM4 We then wish to evaluate, for two lattice elements x and y with associated smoothed indicators f and g, DISPLAYFORM5 This integral admits a closed form solution. Proposition 1. Let m Φ (x) = Φ(x)dx be an antiderivative of the standard normal CDF. Then the solution to equation 2 is given by, DISPLAYFORM6 where σ = σ 2 1 + σ 2 2, soft(x) = log(1 + exp(x)) is the softplus function, the antiderivative of the logistic sigmoid, and ρ = σ 1.702.Proof. The first line is proved in Appendix A, the second approximation follows from the approximation of Φ by a logistic sigmoid given in BID3.Note that, in the zero-temperature limit, as ρ goes to zero, we recover the formula DISPLAYFORM7 with equality in the last line because (a, b) and (c, d) are intervals. This last line is exactly our original equation equation 1, which is expected from convolution with a zero-bandwidth kernel (a Dirac delta function, the identity element under convolution). This is true for both the exact formula using Φ(x)dx, and the softplus approximation. Unfortunately, for any ρ > 0, multiplication of Gaussian-smoothed indicators does not give a valid meet operation on a function lattice, for the simple reason that f 2 = f, except in the case of indicator functions, violating the idempotency requirement of Section 3.1.More importantly, for practical considerations, if we are to treat the outputs of p φ as probabilities, the consequence is DISPLAYFORM8 which complicates our applications that train on conditional probabilities. However, by a modification of equation 3, we can obtain a function p such that p(x ∧ x) = p(x), while retaining the smooth optimization properties of the Gaussian model. Recall that for the hinge function m h and two intervals (a, b) and (c, d), we have DISPLAYFORM9 where the left hand side is the zero-temperature limit of the Gaussian model from equation 3. This identity is true of the hinge function m h, but not the softplus function. However, an equation with a similar functional form as equation 6 (on both the left-and right-hand sides) is true not only of the hinge function from the unsmoothed model, but also true of the softplus. For two intervals x = (a, b) an y = (c, d), by the commutativity of min and max with monotonic functions, we have DISPLAYFORM10 In the zero-temperature limit, all terms in equations 3 and 7 are equivalent. However, outside of this, equation 7 is idempotent for x = y = (a, b) = (c, d) (when considered as a measure of overlap, made precise in the next paragraph), while equation 3 is not. This inspires us to define the probabilities p(x) and p(x, y) using a normalized version of equation 7 in place of equation 3. For the interval (one-dimensional box) case, we define DISPLAYFORM11 which satisfies the idempotency requirement, p(x) = p(x, x).Because softplus upper-bounds the hinge function, it is capable of outputting values that are greater than 1, and therefore must be normalized. In our experiments, we use two different approaches to normalization. For experiments with a relatively small number of entities (all besides Flickr), we allow the boxes to learn unconstrained, and divide each dimension by the measured size of the global minimum and maximum (G DISPLAYFORM12 For data where computing these values repeatedly is infeasible, we project onto the unit hypercube and normalize by m soft. The final probability p(x) is given by the product over dimensions DISPLAYFORM13 Note that, while equivalent in the zero temperature limit to the standard uniform probability measure of the box model, this function, like the Gaussian model, is not a valid probability measure on the entire joint space of events (the lattice). However, neither is factorization of a conditional probability table using a logistic sigmoid link function, which is commonly used for the similar tasks. Our approach retains the inductive bias of the original box model, is equivalent in the limit, and satisfies the necessary condition that p(x, x) = p(x). A comparison of the 3 different functions is given in FIG2, with the softplus overlap showing much better behavior for highly disjoint boxes than the Gaussian model, while also preserving the meet property. Note that in order to achieve high overlap, the Gaussian model must drastically lower its temperature, causing vanishing gradients in the tails. We perform experiments on the WordNet hypernym prediction task in order to evaluate the performance of these improvements in practice. The WordNet hypernym hierarchy contains 837,888-edges after performing the transitive closure on the direct edges in WordNet. We used the same train/dev/test split as in BID20. Positive examples are randomly chosen from the 837k edges, while negative examples are generated by swapping one of the terms to a random word in the dictionary. Experimental details are given in Appendix D.1. The smoothed box model performs nearly as well as the original box lattice in terms of test accuracy 1. While our model requires less hyper-parameter tuning than the original, we suspect that our performance would be increased on a task with a higher degree of sparsity than the 50/50 positive/negative split of the standard WordNet data, which we explore in the next section. In order to confirm our intuition that the smoothed box model performs better in the sparse regime, we perform further experiments using different numbers of positive and negative examples from the WordNet mammal subset, comparing the box lattice, our smoothed approach, and order embeddings (OE) as a baseline. The training data is the transitive reduction of this subset of the mammal WordNet, while the dev/test is the transitive closure of the training data. The training data contains 1,176 positive examples, and the dev and test sets contain 209 positive examples. Negative examples are generated randomly using the ratio stated in the table. As we can see from the table, with balanced data, all models include OE baseline, Box, Smoothed Box models nearly match the full transitive closure. As the number of negative examples increases, the performance drops for the original box model, but Smoothed Box still outperforms OE and Box in all setting. This superior performance on imbalanced data is important for e.g. real-world entailment graph learning, where the number of negatives greatly outweigh the positives. Table 5: F1 scores of the box lattice, order embeddings, and our smoothed model, for different levels of label imbalance on the WordNet mammal subset. We conduct experiments on the Flickr entailment dataset. Flickr is a large-scale caption entailment dataset containing of 45 million image caption pairs. In order to perform an apples-to-apples comparison with existing we use the exact same dataset from BID22. In this case, we do constrain the boxes to the unit cube, using the same experimental setup as BID22, except we apply the softplus function before calculating the volume of the boxes. Experimental details are given in Appendix D.3.We report KL divergence and Pearson correlation on the full test data, unseen pairs (caption pairs which are never occur in training data) and unseen captions (captions which are never occur in training data). As shown in TAB2, we see a slight performance gain compared to the original model, with improvements most concentrated on unseen captions. We apply our method to a market-basket task constructed using the MovieLens dataset. Here, the task is to predict users' preference for movie A given that they liked movie B. We first collect all pairs of user-movie ratings higher than 4 points (strong preference) from the MovieLens-20M dataset. From this we further prune to just a subset of movies which have more than 100 user ratings to make sure that counting statistics are significant enough. This leads to 8545 movies in our dataset. We calculate the conditional probability P (A|B) = P (A,B) We compare with several baselines: low-rank matrix factorization, complex bilinear factorization BID19, and two hierarchical embedding methods, POE BID9 and the Box Lattice BID22. Since the training matrix is asymmetric, we used separate embeddings for target and conditioned movies. For the complex bilinear model, we added one additional vector of parameters to capture the "imply" relation. We evaluate on the test set using KL divergence, Pearson correlation, and Spearman correlation with the ground truth probabilities. Experimental details are given in Appendix D.4. DISPLAYFORM0 From the in TAB4, we can see that our smoothed box embedding method outperforms the original box lattice as well as all other baselines' performances, especially in Spearman correlation, the most relevant metric for recommendation, a ranking task. We perform an additional study on the robustness of the smoothed model to initialization conditions in Appendix C. We presented an approach to smoothing the energy and optimization landscape of probabilistic box embeddings and provided a theoretical justification for the smoothing. Due to a decreased number of hyper-parameters this model is easier to train, and, furthermore, met or surpassed current state-ofthe-art on several interesting datasets. We further demonstrated that this model is particularly effective in the case of sparse data and more robust to poor initialization. Tackling the learning problems presented by rich, geometrically-inspired embedding models is an open and challenging area of research, which this work is far from the last word on. This task will become even more pressing as the embedding structures become more complex, such as unions of boxes or other non-convex objects. To this end, we will continue to explore both function lattices, and constraint-based approaches to learning. A PROOF OF GAUSSIAN OVERLAP FORMULA We wish to evaluate, for two lattice elements x and y, with associated smoothed indicators f and g, DISPLAYFORM0 Since the Gaussian kernel is normalized to have total integral equal to 1, so as not to change the overall areas of the boxes, the concrete formula is DISPLAYFORM1 Since the antiderivative of φ is the normal CDF, this may be recognized as the difference Φ(x; a, σ 2) − Φ(x; b, σ 2), but this does not allow us to easily evaluate the integral of interest, which is the integral of the product of two such functions. To evaluate equation 8, recall the identity BID8 BID21 DISPLAYFORM2 For convenience, let τ: DISPLAYFORM3. Applying Fubini's theorem and using equation 9, we have DISPLAYFORM4 and therefore, with σ = τ −1, DISPLAYFORM5 The MovieLens dataset, while not truly sparse, has a large proportion of small probabilities which make it especially suitable for optimization by the smoothed model. The rough distribution of probabilities, in buckets of width 0.1, is shown in FIG0. We perform an additional set of experiments to determine the robustness of the smoothed box model to initialization. While the model is normally initialized randomly so that each box is a product of intervals that almost always overlaps with the other boxes, we would like to determine the models robustness to disjoint boxes in a principled way. While we can control initialization, we cannot always control the intermediate of optimization, which may drive boxes to be disjoint, a condition from which the original, hard-edged box model may have difficulty recovering. So, parametrizing the initial distribution of boxes with a minimum coordinate and a positive width, we adjust the width parameter so that approximately 0%, 20%, 50%, and 100% of boxes are disjoint at initialization before learning on the MovieLens dataset as usual. These are presented in table 8. The smoothed model does not seem to suffer at all from disjoint initialization, while the performance of the original box model degrades significantly. From this we can speculate that part of the strength of the smoothed box model is its ability to smoothly optimize in the disjoint regime. We give a brief overview of our methodology and hyperparameter selection methods for each experiment. Detailed hyperparameter settings and code to reproduce experiments can be found at https://github.com/Lorraine333/smoothed_box_embedding. For the WordNet experiments, the model is evaluated every epoch on the development set for a large fixed number of epochs, and the best development model is used to score the test set. Baseline models are trained using the parameters of BID22, with the smoothed model using hyperparameters determined on the development set. We follow the same routine as the WordNet experiments section to select best parameters. For the 12 experiments we conducted in this section, negative examples are generated randomly based on the ratio for each batch of positive examples. We do a parameter sweep for all models then choose the best for each model as our final . The experimental setup uses the same architecture as BID22 and BID9, a single-layer LSTM that reads captions and produces a box embedding parameterized by min and delta. Embeddings are produced by feedforward networks on the output of the LSTM. The model is trained for a large fixed number of epochs, and tested on the development data at each epoch. The best development model is used to report test set score. Hyperparameters were determined on the development set. For all MovieLens experiments, the model is evaluated every 50 steps on the development set, and optimization is stopped if the best development set score fails to improve after 200 steps. The best development model is used to score the test set.

Represent the short summary for finding the corresponding scientific article (long):

Improve hierarchical embedding models using kernel smoothing

we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.