Question: Represent the scientific article to find a short TLDR

Question: During the last years, a remarkable breakthrough has been made in AI domain thanks to artificial deep neural networks that achieved a great success in many machine learning tasks in computer vision, natural language processing, speech recognition, malware detection and so on. However, they are highly vulnerable to easily crafted adversarial examples. Many investigations have pointed out this fact and different approaches have been proposed to generate attacks while adding a limited perturbation to the original data. The most robust known method so far is the so called C&W attack. Nonetheless, a countermeasure known as fea- ture squeezing coupled with ensemble defense showed that most of these attacks can be destroyed. In this paper, we present a new method we call Centered Initial Attack (CIA) whose advantage is twofold: first, it insures by construc- tion the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process that degrades the quality of attacks. Second, it is robust against recently introduced defenses such as feature squeezing, JPEG en- coding and even against a voting ensemble of defenses. While its application is not limited to images, we illustrate this using five of the current best classifiers on ImageNet dataset among which two are adversarialy retrained on purpose to be robust against attacks. With a fixed maximum perturbation of only 1.5% on any pixel, around 80% of attacks (targeted) fool the voting ensemble defense and nearly 100% when the perturbation is only 6%. While this shows how it is difficult to defend against CIA attacks, the last section of the paper gives some guidelines to limit their impact. Since the skyrocketing of data volumes and parallel computation capacities with GPUs during the last years, deep neural networks (DNN) have become the most effective approaches in solving many machine learning problems in several domains like computer vision, speech recognition, games playing etc. They are even intended to be used in critical systems like autonomous vehicle BID17, BID18. However, DNN as they are currently built and trained using gradient based methods, are very vulnerable to attacks a.k.a. adversarial examples BID1. These examples aim to fool a classifier to make it predict the class of an input as another one, different from the real class, after bringing only a very limited perturbation to this input. This can obviously be very dangerous when it comes to systems where human life is in stake like in self driven vehicles. Companies IT networks and plants are also vulnerable if DNN based intrusion detection systems were to be deployed BID20.Many approaches have been proposed to craft adversarial examples since the publication by Szegedy et al. of the first paper pointing out DNN vulnerability issue BID4. In their work, they generated adversarial examples using box-constrained L-BFGS. Later in BID1, a fast gradient sign method (FGSM) that uses gradients of a loss function to determine in which direction the pixels intensity should be changed is presented. It is designed to be fast not optimize the loss function. Kurakin et al. introduced in BID13 a straightforward simple improvement of this method where instead of taking a single step of size in the direction of the gradient-sign, multiple smaller steps are taken, and the is clipped in the end. Papernot et al. introduced in BID3 an attack, optimized under L0 distance, known as the Jacobian-based Saliency Map Attack (JSMA). Another simple attack known as Deepfool is provided in. It is an untargeted attack technique optimized for the L2 distance metric. It is efficient and produces closer adversarial examples than the L-BFGS approach discussed earlier. Evolutionary algorithms are also used by authors in BID14 to find adversarial example while maintaining the attack close to the initial data. More recently, Carlini and Wagner introduced in BID0 the most robust attack known to date as pointed out in BID5. They consider different optimization functions and several metrics for the maximum perturbation. Their L2-attack defeated the most powerful defense known as distillation BID7. However, authors in BID6 showed that feature squeezing managed to destroy most of the C&W attacks. Many other defenses have been published, like adversarial training BID3, gradient masking BID9, defenses based on uncertainty using dropout BID10 as done with Bayesian networks, based on statistics BID11, BID12, or principal components BID22, BID23. Later, while we were carrying out our investigation, paper BID16 showed that not less than ten defense approaches, among which are the previously enumerated defenses, can be defeated by C&W attacks. It also pointed out that feature squeezing also can be defeated but no thorough investigation actually was presented. Another possible defense but not investigated is based on JPEG encoding when dealing with images. It has never been explicitly attacked even after it is shown in BID13 that most attacks are countered by this defense. Also, to our knowledge, no investigation has been conducted when dealing with ensemble defenses. Actually, attacks transferability between models that is well investigated and demonstrated in BID19 in the presence of an oracle (requesting defense to get labels to train a substitute model) is not guaranteed at all when it is absent. Finally, when the maximum perturbation added to the original data is strictly limited, clipping is needed at the end of training (adversarial crafting) even if C&W attacks are used. The quality of crafted attacks is therefore degraded as the brought perturbation during the training is brutally clipped. We tackle all these points in our work while introducing a new attack we call Centered Initial Attack (CIA). This approach considers the perturbation limits by construction and consequently no alteration is done on the CIA ing examples. To make it clearer for the reader, an example is given below to illustrate the clipping issue. FIG0 shows a comparison between CIA and C&W L2 attack before and after clipping on an example, a guitar targeted as a potpie with max perturbation equal to 4.0 (around 1.5%). The same number of iterations FORMULA4 is considered for both methods. As can be seen on FIG0, CIA generates the best attack with 96% confidence. C&W is almost as good with a score of 95% but it is degraded to 88% after applying the clipping to respect the imposed max perturbation. Avoiding this degradation due to clipping is the core motivation of our investigation. The remaining of this paper is organized as follows. Section I presents some mathematical formulations and the principle of CIA strategy. Then Section II investigates the application of CIA against ensemble defense, feature squeezing and JPEG encoding defenses. Then Section III provides some guidelines to limit the impact of CIA attacks. Finally, we give some possible future investigations in the . Before entering into details of CIA, let us give some useful formulations. A neural network can be seen as a function F (x) = y that accepts an input x and produces an output y. The function F depends actually on some model parameters often called weights an biases. These are the variables that are adjusted during the learning process to fit the training data on one hand and generalize well to unseen data on the other hand. Since they do not change in our models, we omit them in our notations. The input x can be a vector or an array of any dimension. So, without loss of generality, we consider x ∈ n as it can be flattened in any case. So, the i th component of x is noted x i with integer i ∈ [1, n].Since we consider m-class classifiers, the output is calculated using the sof tmax function. The output y = F (x) can be seen as a vector of m probabilities p j with j ∈ [1, m]. The component with the biggest value gives the predicted class C(x). This can be written as: DISPLAYFORM0 We note the output corresponding to the correct class as C c (x). An adversarial examplex is crafted as a non targeted attack so as to get C(x) = C c (x) or a targeted one to get C(x) = t where t is the target class. Crafting an example can then be formulated using a loss function L(F, x) to maximize the probability of getting a class different from the correct one. Cross entropy is used in our work. The adversarial examplex can be written asx = x + δ where δ is the added perturbation. In our work we constrain δ to be within domain [−∆, ∆] as considered for instance in Google Brain-Kaggle competition, ∆ being the maximum perturbation. With the existing approaches, adversarial examples are generated through some iterations then clipped in the end to respect this constraint. With C&W attacks for instance, the loss function includes a norm term δ to minimize perturbation δ but the clipping is still needed as we saw above. The main idea behind Centered Initial Attack is to find for each component i the center x * i of the domain in whichx i is allowed to be (green segment on Figure. 2). This is not trivial since we have to insure at the same time the componentx i to be within another domain [α i, β i] to be valid, α i and β i are respectively the minimum and maximum values that can be taken by the i To find the center of domain definition ofx, three cases are to be considered actually, not four since ∆ i is much smaller than (β i − α i), as can be seen on FIG2. For recall, ∆ i is the i th component of ∆.The three cases are: DISPLAYFORM1 DISPLAYFORM2 Now if we consider a continuous differentiable function g such that: g: → [−1, +1], then we can write every componentx i as: DISPLAYFORM3 This equation can be rewritten using arrays as: DISPLAYFORM4 where operator is the elementwise product and r is a new variable on which we optimize the loss function. Finally, the loss function can be written as: DISPLAYFORM5 No constraint is to be considered on the variable r since it is well defined in domain (−∞, +∞). Any initialization of r is possible but we consider it as zero for simplicity. So, the initial attackx is therefore different from x whereas it is centered in its domain of definition (green segment). This explains the CIA attack name. Regarding g, many continuous functions can be used. For instance we tried three functions: DISPLAYFORM6 Obviously other functions can be considered. In our experiments, as they all lead to similar , we always considered tanh. It is interesting to note that with CIA, we can define a different maximum perturbation from a component x i to another x j. Likewise, it is easy for instance to limit the crafting to only a portion of an image by considering a zero max perturbation on the other regions, without changing anything in the training algorithm. This is an advantage with regard to existing approaches as it is difficult with the current machine learning frameworks to select from the same array only some variables to optimize on. Gradients masking would be a solution but not desired as it is a clipping operation. An example of such partial crafting is displayed on Figure. 3 where only a 50px band on the top and right sides is modified. We generated it using ∆ = 32 to make the difference visible on the paper but the image (spider) is also classified as the target (dog) even with smaller values. Also, any gradient descent optimizer can be used to craft attacks as the case with BID0. Adam BID15 turns out to be the fastest in our experiments. We used it for training and considered 20 iterations, a good compromise between computation time and attacks crafting convergence, for all the adversarial examples crafting. To reproduce the , the Adam hyperparameters to be considered are {learning rate = 0.2, DISPLAYFORM0 Finally, it is worth noting that CIA is not limited to images. It can be used for any type of data with bounded continuous features. In order to check the effectiveness of CIA attacks, we consider mainly targeted attacks as they are more difficult to craft against three different strategies of defense; ensemble defense with many classifiers, feature squeezing, and JPEG encoding. A combination of these defenses is also considered as we will see later. In this paper, we consider only white box attacks where we have full access to defense models parameters. Other works BID19 pointed out the transferability property between models when it is possible to quest the defense classifier and get the labels back to train a substitute model to be used for crafting attacks. When it is the case, an attack generated using the substitute is likely to remain an attack on the defense. When it is not the case however, this transferability is inexistent as we will see below. So, attacking as many models as possible at once is required. In order to check the robustness of CIA in attacking many classifiers at once, we consider the five best classifiers on ImageNet dataset : Inception V3 (IncV3 a), Inception V4 (IncV4), InceptionResnet V2 (IncRes a), adversarialy trained Inception V3(IncV3 b) and adversarialy InceptionResnet V2 (IncRes b). The accuracy of these classifiers is around 80% on the whole ImageNet dataset. The accuracies on 1000 images dataset we consider are showed in TAB0. In this experiment, we attack IncV3 b classifier and present the success rate of targeted attacks and the miss-classification rate of each classifier. The are showed in TAB1. As can be seen on TAB1, the transferability is inexistent whatever the maximum perturbation used when looking at the targeted attacks success rate. However, a small increase in misclassification rate is noticed especially with IncV3 a, raising from 3.9% to 7%. This was verified when attacking any other classifier alone and checking the impact on the others. This demonstrates clearly the need to attack many classifiers at once. To do so, we considered an optimization using a sum of losses, each loss being related to one classifier: DISPLAYFORM0 where F i is the function relative to the i th classifier. A weighed version can be considered to target a classifier more aggressively than another. In our experiments, they are all attacked equally. The are displayed in TAB2. As we can see on TAB2, the success rate of the targeted attacks against the voting ensemble is high, around 80% for ∆ = 4.0 and approaching 100% for ∆ = 16.0. It is also interesting to notice that the success rate of attacking IncV3 b has decreased compared to the case when it was attacked alone. With ∆ = 4.0 for instance, it went from 97.6% to 92,5%. This can be explained by the fact that the gradients are balanced in a way to change the input in a direction that minimizes all the losses at the same time. As a of this section, attacking an ensemble defense using CIA is effective when we have complete access to all defense models (white box attacks). Section 3.3 will show that the transferability to an anknown model, while attacking four among the five, is limited but not negligible (more than 30%). This defense approach has been developed to counter attacks using smoothing filters BID6. The intuition behind this idea was that smoothing removes the sharp changes brought while crafting adversarial examples. There are different feature squeezing possibilities but we consider spatial smoothing in the current study. The other ones will be addressed in future experiments. While adding a filter, one should care about the possible loss of accuracy of the defense classifier. Authors in BID6 showed that a 3x3 filter is a good compromise that gives an effective defense while limiting the loss of accuracy. We noticed it too in the current investigation and therefore considered this kernel size. Also, different smoothing strategies are possible like Gaussian, diagonal, mean, etc. As the are quite similar in our experiments we carried out, we consider the mean filter in the sequel for its calculation simplicity. The filter used for defense can actually be replaced by a convolution layer before the neural network as shown in Figure. 4. Figure 4: spatial smoothing modeling using a convolution layer. As can be seen on Figure 4, adding a convolution layer in a new network that could be represented using a new function F. We can therefore simply craft an adversarial example using this new function. As a start, we conducted an attack against only one network (IncV3 a). The are shown in TAB3. As we can notice in TAB3, the success rate of targeted attacks and the miss-classification rate are nearly 100%. This means that the spatial smoothing based defense is not effective. Once again, the transferability between models is inexistent. An interesting point is to check the success of the same attacks when the defense is not actually using spatial smoothing for sure. Said in other words, we suspect the defense to use spatial smoothing but we are not sure about it. So, we craft attacks as before for a guarantee. Or not! The are showed in TAB4.As we can clearly see, the attacks success rate depleted to only 3.2% and the misclassification rate to 18.2%. The attack is therefore not effective in case of filter based defense uncertainty. This disagrees with the intuition behind spatial smoothing as an efficient defense against attacks presented in BID6. As demonstrated indeed, the filtered adversarial example can be an effective attack but not the unfiltered one! Then, how to overcome this issue for a more robust attack? To answer this question, we consider a hybrid network where both filtered and non filtered inputs are used for optimization as represented on Figure. 5. The loss function to be used is given as a sum of two terms (weighing would be useful for more robustness) as follows: DISPLAYFORM0 where a, b are real positive numbers. We conducted a new experiment using this hybrid loss function and the are given in TAB5. 6.3% TAB5 shows that the success rate is this time very high in both cases: 98.4% in filter based defense and nearly 100% in no filter defense. We conclude that this attack is robust whether filtering is used or not in defense. Another question arises from previous given the lack of transferability of attacks between models. What if an ensemble defense is used and filter use is uncertain? Once again, we consider the sum of all losses, but use the hybrid losses this time as follows: DISPLAYFORM1 where L Hi is the hybrid loss relative to the i th classifier. The are presented in TAB6 Even with ∆ = 4.0 and considering only targeted attacks, the success and miss-classification rates are high, around 50% when all classifiers use filters and much higher (around 80%) when no filters are used. Other experiments we conducted showed that attacks rate for filter based defense can be improved by assigning a greater weight b (twice the weight a) in the hybrid loss equation FORMULA9. An investigation conducted by authors in BID13 showed that most adversarial examples are countered if they are JPEG encoded before classifying them. Lets check if it is the case with CIA attacks. We conducted an attack against IncV3 a and classified the adversarial examples after being JPEG encoded. The encoding uses different compression quality values Q. A higher Q means a better quality of image with a bigger size however since it undergoes less loss and compression. The are displayed on TAB7. TAB7 shows that CIA is robust when performing non targeted attacks as almost 100% of them are successful with Q = 80 and around 50% with Q = 20. The targeted attacks are less successful with the highest score of 20% when Q = 80 and 0% when Q = 20.The is somehow mitigated with regard to targeted attaks . Indeed, one has to keep in mind that a Q = 20 would not be a reasonable defense as this will degrade highly the accuracy of the classifier. Nonetheless, we tried to improve the attacks success score by finding a suitable approximation JPEG transformations. Obviously, JPEG encoding cannot be modeled accurately using a differentiable function that can be included in crafting examples process as we did before with spatial smoothing. As a brief recall, this encoding implies the passage from RGB space to another color space called YCbCr where Y is brightness, Cb and Cr components represent the chrominance. Actually, humans can see considerably more fine detail in the brightness of an image (the Y component) than in the hue and color saturation of an image (the Cb and Cr components). Considering this fact, Cb and Cr can be downsampled by a factor of 2 or 3 without sensitive change of receptivity to human eye. Another fact is the eye not being sensitive to sharp changes in images. So, removing the high frequencies from the spectral space after a DCT (Discrete Cosine Transform) of an image would not affect its quality remarkably too. These are the important facts used when making a JPEG compression. Other steps like dividing the image into blocks, the quantization of frequency components, the encoding of these components and so on are thoroughly well documented for interested readers BID21. We do not take them into account. Our idea for approximating JPEG is represented in FIG5.As shown on FIG5, we first transform RGB images to YCrCb space using a function T (product an sum operations) then we filter each component using a convolution layer. Given the facts enumerated before about filtering high frequencies and down sampling the chrominance, we consider a mean 3x3 kernel for brightness component and a 6x6 kernel for Cr and Cb. Once filtered, the is brought back to RGB space using T −1 before feeding the neural network. The of crafting attacks against IncV3 a using this approximation are given in TAB8. As can be noticed, the are almost the same as those of TAB7. This is a bit untriguing as if all the filters used in the approximation are inexistent. However, a similar remark as in the filterbased defense case can be made. The attacks crafted using JPEG encoding are no longer attacks if JPEG is not used by the defense. This means that the considered approximation of JPEG encoding is not that bad but not enough accurate to give strong attacks. It has obviously to be improved. We are working on it. As we saw previously, defending against attacks is hard as along as the defense can be modeled using a function that could be added to form a new network that can be used for crafting examples. As a , one has to find a transformation that is hardly represented or approximated using simple functions. This is obviously not trivial as we saw with JPEG defense. Another, and more short term realizable defense, is to consider a big number of well performing classifiers including limited accuracy variance transformations like spatial smoothing. As we saw in TAB6, the success rate decreased to around 50% when using five classifiers including filters for defense. This is not guaranteed but it worth being investigated. For instance, when we attack all the classifiers except IncV4, the transferability is somehow limited. Indeed, the success rate of attacks is almost 100% against the attacked classifiers whereas it is only around 34% for IncV4 as can be noticed on TAB0. In this paper we presented a new strategy called CIA for crafting adversarial examples while insuring the maximum perturbation added to the original data to be smaller than a fixed threshold. We demonstrated also its robustness against some defenses, feature squeezing, ensemble defenses and even JPEG encoding. For future work, it would be interesting to investigate the transferability of CIA attacks to the physical world as it is shown in BID13 that only a very limited amount of FGDM attacks, around 20%, survive this transfer. Another interesting perspective is to consider partial crafting attacks while selecting regions taking into account the content of the data. With regard to images for instance, this would be interesting to hide attacks with big but imperceptible perturbations.

Represent the TLDR for finding the corresponding scientific article

In this paper, a new method we call Centered Initial Attack (CIA) is provided. It insures by construction the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process.

Represent the short summary for finding the corresponding scientific article (long)

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.


Question: Represent the scientific article to find a short TLDR

Question: Answering complex logical queries on large-scale incomplete knowledge graphs (KGs) is a fundamental yet challenging task. Recently, a promising approach to this problem has been to embed KG entities as well as the query into a vector space such that entities that answer the query are embedded close to the query. However, prior work models queries as single points in the vector space, which is problematic because a complex query represents a potentially large set of its answer entities, but it is unclear how such a set can be represented as a single point. Furthermore, prior work can only handle queries that use conjunctions ($\wedge$) and existential quantifiers ($\exists$). Handling queries with logical disjunctions ($\vee$) remains an open problem. Here we propose query2box, an embedding-based framework for reasoning over arbitrary queries with $\wedge$, $\vee$, and $\exists$ operators in massive and incomplete KGs. Our main insight is that queries can be embedded as boxes (i.e., hyper-rectangles), where a set of points inside the box corresponds to a set of answer entities of the query. We show that conjunctions can be naturally represented as intersections of boxes and also prove a negative that handling disjunctions would require embedding with dimension proportional to the number of KG entities. However, we show that by transforming queries into a Disjunctive Normal Form, query2box is capable of handling arbitrary logical queries with $\wedge$, $\vee$, $\exists$ in a scalable manner. We demonstrate the effectiveness of query2box on two large KGs and show that query2box achieves up to 25% relative improvement over the state of the art. Knowledge graphs (KGs) capture different types of relationships between entities, e.g., Canada citizen −−−−→ Hinton. Answering arbitrary logical queries, such as "where did Canadian citizens with Turing Award graduate?", over such KGs is a fundamental task in question answering, knowledge base reasoning, as well as AI more broadly. First-order logical queries can be represented as Directed Acyclic Graphs (DAGs) (Fig. 1(A) ) and be reasoned according to the DAGs to obtain a set of answers (Fig. 1(C) ). While simple and intuitive, such approach has many drawbacks: Computational complexity of subgraph matching is exponential in the query size, and thus cannot scale to modern KGs; Subgraph matching is very sensitive as it cannot correctly answer queries with missing relations. To remedy one could impute missing relations (; Džeroski, 2009; ;) but that would only make the KG denser, which would further exacerbate issue (; Krompaß et al., 2014). Recently, a promising alternative approach has emerged, where logical queries as well as KG entities are embedded into a low-dimensional vector space such that entities that answer the query are embedded close to the query (; ;). Such approach robustly handles missing relations and is also orders of magnitude faster, as answering an arbitrary logical query is reduced to simply identifying entities nearest to the embedding of the query in the vector space. However, prior work embeds a query into a single point in the vector space. This is problematic because answering a logical query requires modeling a set of active entities while traversing the KG (Fig. 1(C) ), and how to effectively model a set with a single point is unclear. Furthermore, it We then obtain query embedding according to the computation graph (B) as a sequence of box operations: start with two nodes TuringAward and Canada and apply Win and Citizen projection operators, followed by an intersection operator (denoted as a shaded intersection of yellow and orange boxes) and another projection operator. The final embedding of the query is a green box and query's answers are the entities inside the box. is also unnatural to define logical operators (e.g., set intersection) of two points in the vector space. Another fundamental limitation of prior work is that it can only handle conjunctive queries, a subset of first-order logic that only involves conjunction (∧) and existential quantifier (∃), but not disjunction (∨). It remains an open question how to handle disjunction effectively in the vector space. Here we present QUERY2BOX, an embedding-based framework for reasoning over KGs that is capable of handling arbitrary Existential Positive First-order (EPFO) logical queries (i.e., queries that include any set of ∧, ∨, and ∃) in a scalable manner. First, to accurately model a set of entities, our key idea is to use a closed region rather than a single point in the vector space. Specifically, we use a box (axis-aligned hyper-rectangle) to represent a query (Fig. 1(D) ). This provides three important benefits: Boxes naturally model sets of entities they enclose; Logical operators (e.g., set intersection) can naturally be defined over boxes similarly as in Venn diagrams (Venn, 1880); Executing logical operators over boxes in new boxes, which means that the operations are closed; thus, logical reasoning can be efficiently performed in QUERY2BOX by iteratively updating boxes according to the query computation graph (Fig. 1(B)(D) ). We show that QUERY2BOX can naturally handle conjunctive queries. We first prove a negative that embedding EPFO queries to only single points or boxes is intractable as it would require embedding dimension proportional to the number of KG entities. However, we provide an elegant solution, where we transform a given EPFO logical query into a Disjunctive Normal Form (DNF) , i.e., disjunction of conjunctive queries. Given any EPFO query, QUERY2BOX represents it as a set of individual boxes, where each box is obtained for each conjunctive query in the DNF. We then return nearest neighbor entities to any of the boxes as the answers to the query. This means that to answer any EPFO query we first answer individual conjunctive queries and then take the union of the answer entities. We evaluate QUERY2BOX on standard KG benchmarks and show: QUERY2BOX provides strong generalization as it can answer complex queries that it has never seen during training; QUERY2BOX is robust as it can answer any EPFO query with high accuracy even when relations involving answering the query are missing in the KG; QUERY2BOX provides up to 25% relative improvement in accuracy of answering EPFO queries over state-of-the-art baselines. Most related to our work are embedding approaches for multi-hop reasoning over KGs (; ; ;). Crucial difference is that we provide a way to tractably handle a larger subset of the first-order logic (EPFO queries vs. conjunctive queries) and that we embed queries as boxes, which provides better accuracy and generalization. Second line of related work is on structured embeddings, which associate images, words, sentences, or knowledge base concepts with geometric objects such as regions (; ;), densities (; ;), and orderings (; ;). While the above work uses geometric objects to model individual entities and their pairwise relations, we use the geometric objects to model sets of entities and reason over those sets. In this sense our work is also related to classical Venn Diagrams (Venn, 1880), where boxes are essentially the Venn Diagrams in vector space, but our boxes and entity embeddings are jointly learned, which allows us to reason over incomplete KGs. Here we present the QUERY2BOX, where we will define an objective function that allows us to learn embeddings of entities in the KG, and at the same time also learn parameterized geometric logical operators over boxes. Then given an arbitrary EPFO query q (Fig. 1(A) ), we will identify its computation graph (Fig. 1(B) ), and embed the query by executing a set of geometric operators over boxes (Fig. 1(D) ). Entities that are enclosed in the final box embedding are returned as answers to the query (Fig. 1(D) ). In order to train our system, we generate a set of queries together with their answers at training time and then learn entity embeddings and geometric operators such that queries can be accurately answered. We show in the following sections that our approach is able to generalize to queries and query structures never seen during training. We denote a KG as G = (V, R), where v ∈ V represents an entity, and r ∈ R is a binary function r: V × V → {True, False}, indicating whether the relation r holds between a pair of entities or not. In the KG, such binary output indicates the existence of the directed edge between a pair of entities, i.e., v Conjunctive queries are a subclass of the first-order logical queries that use existential (∃) and conjunction (∧) operations. They are formally defined as follows. where v a represents non-variable anchor entity, V 1,..., V k are existentially quantified bound variables, V? is the target variable. The goal of answering the logical query q is to find a set of entities q ⊆ V such that v ∈ q iff q[v] = True. We call q the denotation set (i.e., answer set) of query q. As shown in Fig. 1(A), the dependency graph (DG) is a graphical representation of conjunctive query q, where nodes correspond to variable or non-variable entities in q and edges correspond to relations in q. In order for the query to be valid, the corresponding DG needs to be a Directed Acyclic Graph (DAG), with the anchor entities as the source nodes of the DAG and the query target V? as the unique sink node . From the dependency graph of query q, one can also derive the computation graph, which consists of two types of directed edges that represent operators over sets of entities: • Projection: Given a set of entities S ⊆ V, and relation r ∈ R, this operator obtains • Intersection: Given a set of entity sets {S 1, S 2, . . ., S n}, this operator obtains ∩ n i=1 S i. For a given query q, the computation graph specifies the procedure of reasoning to obtain a set of answer entities, i.e., starting from a set of anchor nodes, the above two operators are applied iteratively until the unique sink target node is reached. The entire procedure is analogous to traversing KGs following the computation graph . So far we have defined conjunctive queries as computation graphs that can be executed directly over the nodes and edges in the KG. Now, we define logical reasoning in the vector space. Our intuition follows Fig. 1: Given a complex query, we shall decompose it into a sequence of logical operations, and then execute these operations in the vector space. This way we will obtain the embedding of the query, and answers to the query will be entities that are enclosed in the final query embedding box. In the following, we detail our two methodological advances: the use of box embeddings to efficiently model and reason over sets of entities in the vector space, and how to tractably handle disjunction operator (∨), expanding the class of first-order logic that can be modeled in the vector space (Section 3.3). Box embeddings. To efficiently model a set of entities in the vector space, we use boxes (i.e., axis-aligned hyper-rectangles). The benefit is that unlike a single point, the box has the interior; thus, if an entity is in a set, it is natural to model the entity embedding to be a point inside the box. Formally, we operate on R d, and define a box in R d by p = (Cen(p), Off(p)) ∈ R 2d as: where is element-wise inequality, Cen(p) ∈ R d is the center of the box, and Off(p) ∈ R d ≥0 is the positive offset of the box, modeling the size of the box. Each entity v ∈ V in KG is assigned a single vector v ∈ R d (i.e., a zero-size box), and the box embedding p models {v ∈ V : v ∈ Box p}, i.e., a set of entities whose vectors are inside the box. For the rest of the paper, we use the bold face to denote the embedding, e.g., embedding of v is denoted by v. Our framework reasons over KGs in the vector space following the computation graph of the query, as shown in Fig. 1 (D): we start from the initial box embeddings of the source nodes (anchor entities) and sequentially update the embeddings according to the logical operators. Below, we describe how we set initial box embeddings for the source nodes, as well as how we model projection and intersection operators (defined in Sec. 3.1) as geometric operators that operate over boxes. After that, we describe our entity-to-box distance function and the overall objective that learns embeddings as well as the geometric operators. Initial boxes for source nodes. Each source node represents an anchor entity v ∈ V, which we can regard as a set that only contains the single entity. Such a single-element set can be naturally modeled by a box of size/offset zero centerd at v. Formally, we set the initial box embedding as (v, 0), where v ∈ R d is the anchor entity vector and 0 is a d-dimensional all-zero vector. Geometric projection operator. We associate each relation r ∈ R with relation embedding r = (Cen(r), Off(r)) ∈ R 2d with Off(r) 0. Given an input box embedding p, we model the projection by p + r, where we sum the centers and sum the offsets. This gives us a new box with the translated center and larger offset because Off(r) 0, as illustrated in Fig. 2 (A). The adaptive box size effectively models a different number of entities/vectors in the set. Geometric intersection operator. We model the intersection of a set of box embeddings {p 1, . . ., p n} as p inter = (Cen(p inter), Off(p inter)), which is calculated by performing attention over the box centers and shrinking the box offset using the sigmoid function: where is the dimension-wise product, MLP(·): is the sigmoid function, DeepSets(·) is the permutation-invariant deep architecture , and both Min(·) and exp(·) are applied in a dimension-wise manner. , we model all the deep sets by DeepSets({x 1, . . ., where all the hidden dimensionalities of the two MLPs are the same as the input dimensionality. The intuition behind our geometric intersection is to generate a smaller box that lies inside a set of boxes, as illustrated in Fig. 2 1 Different from the generic deep sets to model the intersection , our geometric intersection operator effectively constrains the center position and models the shrinking set size. Entity-to-box distance. Given a query box q ∈ R 2d and an entity vector v ∈ R d, we define their distance as where and 0 < α < 1 is a fixed scalar, and As illustrated in Fig. 2(C), dist outside corresponds to the distance between the entity and closest corner/side of the box. Analogously, dist inside corresponds to the distance between the center of the box and its side/corner (or the entity itself if the entity is inside the box). The key here is to downweight the distance inside the box by using 0 < α < 1. This means that as long as entity vectors are inside the box, we regard them as "close enough" to the query center (i.e., dist outside is 0, and dist inside is scaled by α). When α = 1, dist box reduces to the ordinary L 1 distance, i.e., Cen(q) − v 1, which is used by the conventional TransE as well as prior query embedding methods . Training objective. Our next goal is to learn entity embeddings as well as geometric projection and intersection operators. Given a training set of queries and their answers, we optimize a negative sampling loss to effectively optimize our distance-based model : where γ represents a fixed scalar margin, v ∈ q is a positive entity (i.e., answer to the query q), and v i / ∈ q is the i-th negative entity (non-answer to the query q) and k is the number of negative entities. So far we have focused on conjunctive queries, and our aim here is to tractably handle in the vector space a wider class of logical queries, called Existential Positive First-order (EPFO) queries that involve ∨ in addition to ∃ and ∧. We specifically focus on EPFO queries whose computation graphs are a DAG, same as that of conjunctive queries (Section 3.1), except that we now have an additional type of directed edge, called union defined as follows: • Union: Given a set of entity sets {S 1, S 2, . . ., S n}, this operator obtains A straightforward approach here would be to define another geometric operator for union and embed the query as we did in the previous sections. An immediate challenge for our box embeddings is that boxes can be located anywhere in the vector space, so their union would no longer be a simple box. In other words, union operation over boxes is not closed. Theoretically, we can prove a general negative for any embedding-based method that maps query q into q such that dist(v; q) ≤ β iff v ∈ q. Here, dist(v; q) is the distance between entity and query embeddings, e.g., dist box (v; q) or v − q 1, and β is a fixed threshold. Theorem 1. Consider any M conjunctive queries q 1,..., q M whose denotation sets q 1,..., q M are disjoint with each other, ∀ i = j, q i ∩ q j = ∅. Let D be the VC dimension of the function class {sign(β − dist(·; q)): q ∈ Ξ}, where Ξ represents the query embedding space and sign(·) is the sign function. Then, we need D ≥ M to model any EPFO query, i.e., dist(v; q) ≤ β ⇔ v ∈ q is satisfied for every EPFO query q. The proof is provided in Appendix A, where the key is that the introduction of the union operation forces us to model the powerset {∪ q i ∈S q i : S ⊆ {q 1, . . ., q M}} in a vector space. For a real-world KG, there are M ≈ |V| conjunctive queries with non-overlapping answers. For example, in the commonly-used FB15k dataset , derived from the Freebase , we find M = 13,365, while |V| is 14,951 (see Appendix B for the details). Theorem 1 shows that in order to accurately model any EPFO query with the existing framework, the complexity of the distance function measured by the VC dimension needs to be as large as the number of KG entities. This implies that if we use common distance functions based on hyper-plane, Enclidian sphere, or axis-aligned rectangle, 2 their parameter dimensionality needs to be Θ(M), which is Θ(|V|) for real KGs we are interested in. In other words, the dimensionality of the logical query embeddings needs to be Θ(|V|), which is not low-dimensional; thus not scalable to large KGs and not generalizable in the presence of unobserved KG edges. To rectify this issue, our key idea is to transform a given EPFO query into a Disjunctive Normal Form (DNF) , i.e., disjunction of conjunctive queries, so that union operation only appears in the last step. Each of the conjunctive queries can then be reasoned in the low-dimensional space, after which we can aggregate the by a simple and intuitive procedure. In the following, we describe the transformation to DNF and the aggregation procedure. Transformation to DNF. Any first-order logic can be transformed into the equivalent DNF . We perform such transformation directly in the space of computation graph, i.e., moving all the edges of type "union" to the last step of the computation graph. Let G q = (V q, E q) be the computation graph for a given EPFO query q, and let V union ⊂ V q be a set of nodes whose in-coming edges are of type "union". For each v ∈ V union, define P v ⊂ V q as a set of its parent nodes. We first generate N = v∈Vunion |P v | different computation graphs G q,..., G q (N) as follows, each with different choices of v parent in the first step. 1. For every v ∈ V union, select one parent node v parent ∈ P v. 2. Remove all the edges of type'union.' 3. Merge v and v parent, while retaining all other edge connections. We then combine the obtained computation graphs G q,..., G q (N) as follows to give the final equivalent computation graph. 1. Convert the target sink nodes of all the obtained computation graphs into the existentially quantified bound variables nodes. 2. Create a new target sink node V?, and draw directed edges of type "union" from all the above variable nodes to the new target node. An example of the entire transformation procedure is illustrated in Fig. 3. By the definition of the union operation, our procedure gives the equivalent computation graph as the original one. Furthermore, as all the union operators are removed from G q,..., G q (N), all of these computation graphs represent conjunctive queries, which we denote as q,..., q (N). We can then apply existing framework to obtain a set of embeddings for these conjunctive queries as q,..., q (N). Aggregation. Next we define the distance function between the given EPFO query q and an entity v ∈ V. Since q is logically equivalent to q ∨ · · · ∨ q (N), we can naturally define the aggregated distance function using the box distance dist box: where dist agg is parameterized by the EPFO query q. When q is a conjunctive query, i.e., N = 1, dist agg (v; q) = dist box (v; q). For N > 1, dist agg takes the minimum distance to the closest box as the distance to an entity. This modeling aligns well with the union operation; an entity is inside the union of sets as long as the entity is in one of the sets. Note that our DNF-query rewriting scheme is general and is able to extend any method that works for conjunctive queries (e.g., ) to handle more general class of EPFO queries. Computational complexity. The computational complexity of answering an EPFO query with our framework is equal to that of answering the N conjunctive queries. In practice, N might not be so large, and all the N computations can be parallelized. Furthermore, answering each conjuctive query is very fast as it requires us to execute a sequence of simple box operations (each of which takes constant time) and then perform a range search in the embedding space, which can also be done in constant time using techniques based on Locality Sensitive Hashing . Our goal in the experiment section is to evaluate the performance of QUERY2BOX on discovering answers to complex logical queries that cannot be obtained by traversing the incomplete KG. This means, we will focus on answering queries where one or more missing edges in the KG have to be successfully predicted in order to obtain the additional answers. Figure 4: Query structures considered in the experiments, where anchor entities and relations are to be specified to instantiate logical queries. Naming for each query structure is provided under each subfigure, where'p','i', and'u' stand for'projection','intersection', and'union', respectively. Models are trained on the first 5 query structures, and evaluated on all 9 query structures. Table 2: Number of training, validation, and test queries generated for different query structures. We perform experiments on standard KG benchmarks, FB15k and FB15k-237 . Both are subsets of Freebase , a large-scale KG containing general facts. Dataset statistics are shown in Table 1. We follow the standard evaluation protocol in KG literture: Given the standard split of edges into training, test, and validation sets (Table 1), we first augment the KG to also include inverse relations and effectively double the number of edges in the graph. We then create three graphs: G train, which only contains training edges and we use this graph to train node embeddings as well as box operators. We then also generate two bigger graphs: G valid which contains G train plus the validation edges, and G test, which includes G valid as well as the test edges. We consider 9 kinds of diverse query structures shown and named in Fig. 4. We use 5 query structures for training and then evaluate on all the 9 query structures. Given a query q, let q train, q val, and q test denote a set of answer entities obtained by running subgraph matching of q on G train, G valid, and G test, respectively. (We refer the reader to Appendix C for full details on query generation.) At the training time, we use q train as positive examples for the query and other random entities as negative examples (Table 2). However, at the test/validation time we proceed differently. Note that we focus on answering queries where generalization performance is crucial and at least one edge needs to be imputed in order to answer the queries. Thus, rather than evaluating a given query on the full validation (or test) set q val (q test) of answers, we validate the method only on non-trivial answers that include missing relations. Given how we constructed G train ⊆ G valid ⊆ G test, we have q train ⊆ q val ⊆ q test and thus we evaluate the method on q val \ q train to tune hyper-parameters and then report identifying answer entities in q test \ q val. This means we always evaluate on queries/entities that were not part of the training set and the method has not seen them before. Given a test query q, for each of its non-trivial answers v ∈ q test \ q val, we use dist box in Eq. 3 to rank v among V\ q test. Denoting the rank of v by Rank(v), we then calculate evaluation metrics for answering query q, such as Mean Reciprocal Rank (MRR) and Hits at K (H@K): where f metrics (x) = We then average Eq. 6 over all the queries within the same query structure, 3 and report the separately for different query structures. The same evaluation protocol is applied to the validation stage except that we evaluate on q val \ q train rather than q test \ q val. We compare our framework QUERY2BOX against the state-of-the-art GQE . GQE embeds a query to a single vector, and models projection and intersection operators as translation and deep sets , respectively. The L 1 distance is used as the distance between query and entity vectors. For a fair comparison, we also compare with GQE-DOUBLE (GQE with doubled embedding dimensionality) so that QUERY2BOX and GQE-DOUBLE have the same amount of parameters. Although the original GQE cannot handle EPFO queries, we apply our DNF-query rewriting strategy and in our evaluation extend GQE to handle general EPFO queries as well. Furthermore, we perform extensive ablation study by considering several variants of QUERY2BOX (abbreviated as Q2B). We list our method as well as its variants below. • Q2B (our method). The box embeddings are used to model queries, and the attention mechanism is used for the intersection operator. • Q2B-AVG. The attention mechanism for intersection is replaced with averaging. • Q2B-DEEPSETS. The attention mechanism for intersection is replaced with the deep sets. • Q2B-AVG-1P. The variant of Q2B-AVG that is trained with only 1p queries (see Fig. 4); thus, logical operators are not explicitly trained. • Q2B-SHAREDOFFSET. The box offset is shared across all queries (every query is represented by a box with the same size). We use embedding dimensionality of d = 400 and set γ = 24, α = 0.2 for the loss in Eq. 4. We train all types of training queries jointly. In every iteration, we sample a minibatch size of 512 queries for each query structure (details in Appendix C), and we sample 1 answer entity and 128 negative entities for each query. We optimize the loss in Eq. 4 using Adam Optimizer with learning rate = 0.0001. We train all models for 250 epochs, monitor the performance on the validation set, and report the test performance. We start by comparing our Q2B with state-of-the-art query embedding method GQE on FB15k and FB15k-237. As listed in Table 3 and Table 4, our method significantly and consistently outperforms the state-of-the-art baseline across all the query structures, including those not seen during training as well as those with union operations. On average, we obtain Table 6: H@3 on test set for QUERY2BOX vs. several of its variants on FB15k. 9.8% (25% relative) and 3.8% (15% relative) higher H@3 than the best baselines on FB15k and FB15k-237, respectively. Notice that naïvely increasing embedding dimensionality in GQE yields limited performance improvement. Our Q2B is able to effectively model a set of entities by the box embedding, and achieves a large performance gain compared with GQE-DOUBLE (with same number of parameters) that represents queries as point vectors. Also notice that Q2B performs well on new queries with the same structure as the training queries as well as on new query structures never seen during training. We also conduct extensive ablation studies, which are summarized in Tables 5 and 6: Importance of attention mechanism. First, we show that our modeling of intersection using the attention mechanism is important. Given a set of box embeddings {p 1, . . ., p n}, Q2B-AVG is the most naïve way to calculate the center of the ing box embedding p inter while Q2B-DEEPSETS is too flexible and neglects the fact that the center should be a weighted average of Cen(p 1),..., Cen(p n). Compared with the two methods, Q2B achieves better performance in answering queries that involve intersection operation, e.g., 2i, 3i, pi, ip. Specifically, on FB15k-237, Q2B obtains more than 4% and 2% absolute gain in H@3 compared to Q2B-AVG and Q2B-DEEPSETS, respectively. Necessity of training on complex queries. Second, we observe that explicitly training on complex logical queries beyond one-hop path queries (1p in Fig. 4) improves the reasoning performance. Although Q2B-AVG-1P is able to achieve strong performance on 1p and 2u, where answering 2u is essentially answering two 1p queries with an additional minimum operation (see Eq. 5 in Section 3.3), Q2B-AVG-1P fails miserably in answering other types of queries involving logical operators. On the other hand, other methods (Q2B, Q2B-AVG, and Q2B-DEEPSETS) that are explicitly trained on the logical queries achieve much higher accuracy, with up to 10% absolute average improvement of H@3 on FB15k. Adaptive box size for different queries. Third, we investigate the importance of learning adaptive offsets (box size) for different queries. Q2B-SHAREDOFFSET is a variant of our Q2B where all the box embeddings share the same learnable offset. Q2B-SHAREDOFFSET does not work well on all types of queries. This is possibly because different queries have different numbers of answer entities, and the adaptive box size enables us to better model it. In this paper we proposed a reasoning framework called QUERY2BOX that can effectively model and reason over sets of entities as well as handle EPFO queries in a vector space. Given a logical query, we first transform it into DNF, embed each conjunctive query into a box, and output entities closest to their nearest boxes. Our approach is capable of handling all types of EPFO queries scalably and accurately. Experimental on standard KGs demonstrate that QUERY2BOX significantly outperforms the existing work in answering diverse logical queries. Proof. To model any EPFO query, we need to at least model a subset of EPFO queries Q = {∨ q i ∈S q i : S ⊆ {q 1, . . ., q M}}, where the corresponding denotation sets are {∪ q i ∈S q i : S ⊆ {q 1, . . ., q M}}. For the sake of modeling Q, without loss of generality, we consider assigning a single entity embedding v q i to all v ∈ q i, so there are M kinds of entity vectors, v q1,..., v q M. To model all queries in Q, it is necessary to satisfy the following. where q S is the embedding of query ∨ q i ∈S q i. Eq. 7 means that we can learn the M kinds of entity vectors such that for every query in Q, we can obtain its embedding to model the corresponding set using the distance function. Notice that this is agnostic to the specific algorithm to embed query ∨ q∈S q into q S; thus, our is generally applicable to any method that embeds the query into a single vector. Crucially, satisfying Eq. 7 is equivalent to {sign(β − dist(·; q)): q ∈ Ξ} being able to shutter {v q1, . . ., v q M}, i.e., any binary labeling of the points can be perfectly fit by some classifier in the function class. To sum up, in order to model any EPFO query, we need to at least model any query in Q, which requires the VC dimension of the distance function to be larger than or equal to M. Given the full KG G test for the FB15k dataset, our goal is to find conjunctive queries q 1,..., q M such that q 1,..., q M are disjoint with each other. For conjunctive queries, we use two types of queries:'1p' and'2i' whose query structures are shown in Figure 4. On the FB15k, we instantiate 308,006 queries of type'1p', which we denote by S 1p. Out of all the queries in S 1p, 129,717 queries have more than one answer entities, and we denote such a set of the queries by S 1p. We then generate a set of queries of type'2i' by first randomly sampling two queries from S 1p and then taking conjunction; we denote the ing set of queries by S 2i. Now, we use S 1p and S 2i to generate a set of conjunctive queries whose denotation sets are disjoint with each other. First, we prepare two empty sets V seen = ∅, and Q = ∅. Then, for every q ∈ S 1p, if V seen ∩ q = ∅ holds, we let Q ← Q ∪ {q} and V seen ← V seen ∪ q. This procedure already gives us Q, where we have 10, 812 conjunctive queries whose denotation sets are disjoint with each other. We can further apply the analogous procedure for S 2i, which gives us a further increased Q, where we have 13, 365 conjunctive queries whose denotation sets are disjoint with each other. Therefore, we get M = 13, 365. Given G train, G valid, and G test as defined in Section 4.1, we generate training, validation and test queries of different query structures. During training, we consider the first 5 kinds of query structures. For evaluation, we consider all the 9 query structures in Fig. 4, containing query structures that are both seen and unseen during training time. We instantiate queries in the following way. Given a KG and a query structure (which is a DAG), we use pre-order traversal to assign an entity and a relation to each node and edge in the DAG of query structure to instantiate a query. Namely, we start from the root of the DAG (which is the target node), we sample an entity e uniformly from the KG to be the root, then for every node connected to the root in the DAG, we choose a relation r uniformly from the in-coming relations of e in the KG, and a new entity e from the set of entities that reaches e by r in the KG. Then we assign the relation r to the edge and e to the node, and move on the process based on the pre-order traversal. This iterative process stops after we assign an entity and relation to every node and edge in DAG. The leaf nodes in the DAG serve as the anchor nodes. Note that during the entity and relation assignment, we specifically filter out all the degenerated queries, as shown in Fig. C. Then we perform a post-order traversal of the DAG on the KG, starting from the anchor nodes, to obtain a set of answer entities to this query. All of our generated datasets will be made publicly available. Figure 5: Example of the degenerated queries, including r and r −1 appear along one path and same anchor node and relation in intersections. When generating validation/test queries, we explicitly filter out trivial queries that can be fully answered by subgraph matching on G train /G valid. We perform additional experiments on NELL995, which is presented in. Query generation and statistics. , we first combine the validation and test sets with the training set to create the whole knowledge graph for NELL995. Then we create new validation and test set splits by randomly selecting 20,000 triples each from the whole knowledge graph. Note that we filter out all the entities that only appear in the validation and test sets but not in the training set. The statistics of NELL995 are shown in Table 11. Based on the new splits, we sample queries in the same way as in FB15k and FB15k-237. The statistics of the queries are listed in Table 12. Results. The comparing query2box and its baselines (GQE and the query2box variants) are shown in Tables 13, 14, 15, 16. Overall, we see the follow the similar trend as the two FB15k datasets; our query2box outperforms GQE as well as its variants by a large margin.

Answering a wide class of logical queries over knowledge graphs with box embeddings in vector space

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Driven by the need for parallelizable hyperparameter optimization methods, this paper studies open loop search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampling distributions. In particular, we propose the use of k-determinantal point processes in hyperparameter optimization via random search. Compared to conventional uniform random search where hyperparameter settings are sampled independently, a k-DPP promotes diversity. We describe an approach that transforms hyperparameter search spaces for efficient use with a k-DPP. In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from k-DPPs defined over any space from which uniform samples can be drawn, including spaces with a mixture of discrete and continuous dimensions or tree structure. Our experiments show significant benefits in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel. Hyperparameter values-regularization strength, model family choices like depth of a neural network or which nonlinear functions to use, procedural elements like dropout rates, stochastic gradient descent step sizes, and data preprocessing choices-can make the difference between a successful application of machine learning and a wasted effort. To search among many hyperparameter values requires repeated execution of often-expensive learning algorithms, creating a major obstacle for practitioners and researchers alike. In general, on iteration (evaluation) k, a hyperparameter searcher suggests a d-dimensional hyperparameter configuration x k ∈ X (e.g., X = R d but could also include discrete dimensions), a worker trains a model using x k, and returns a validation loss of y k ∈ R computed on a hold out set. In this work we say a hyperparameter searcher is open loop if x k depends only on {x i} k−1 i=1; examples include choosing x k uniformly at random BID3, or x k coming from a low-discrepancy sequence (c.f., BID13). We say a searcher is closed loop if x k depends on both the past configurations and validation losses {(x i, y i)} k−1 i=1; examples include Bayesian optimization BID23 and reinforcement learning methods BID30. Note that open loop methods can draw an infinite sequence of configurations before training a single model, whereas closed loop methods rely on validation loss feedback in order to make suggestions. While sophisticated closed loop selection methods have been shown to empirically identify good hyperparameter configurations faster (i.e., with fewer iterations) than open loop methods like random search, two trends have rekindled interest in embarrassingly parallel open loop methods: 1) modern deep learning model are taking longer to train, sometimes up to days or weeks, and 2) the rise of cloud resources available to anyone that charge not by the number of machines, but by the number of CPU-hours used so that 10 machines for 100 hours costs the same as 1000 machines for 1 hour. This paper explores the landscape of open loop methods, identifying tradeoffs that are rarely considered, if at all acknowledged. While random search is arguably the most popular open loop method and chooses each x k independently of {x i} k−1 i=1, it is by no means the only choice. In many ways uniform random search is the least interesting of the methods we will discuss because we will advocate for methods where x k depends on {x i} k−1 i=1 to promote diversity. In particular, we will focus on k i=1 from a k-determinantal point process (DPP) BID18. We introduce a sampling algorithm which allows DPPs to support real, integer, and categorical dimensions, any of which may have a tree structure, and we describe connections between DPPs and Gaussian processes (GPs).In synthetic experiments, we find our diversity-promoting open-loop method outperforms other open loop methods. In practical hyperparameter optimization experiments, we find that it significantly outperforms other approaches in cases where the hyperparameter values have a large effect on performance. Finally, we compare against a closed loop Bayesian optimization method, and find that sequential Bayesian optimization takes, on average, more than ten times as long to find a good , for a gain of only 0.15 percent accuracy on a particular hyperparameter optimization task. Open source implementations of both our hyperparameter optimization algorithm (as an extension to the hyperopt package BID4) and the MCMC algorithm introduced in Algorithm 2 are available. 1 2 RELATED WORK While this work focuses on open loop methods, the vast majority of recent work on hyperparameter tuning has been on closed loop methods, which we briefly review. Much attention has been paid to sequential model-based optimization techniques such as Bayesian optimization BID5 BID23 which sample hyperparameter spaces adaptively. These techniques first choose a point in the space of hyperparameters, then train and evaluate a model with the hyperparameter values represented by that point, then sample another point based on how well previous point(s) performed. When evaluations are fast, inexpensive, and it's possible to evaluate a large number of points (e.g. k = Ω(2 d) for d hyperparameters) these approaches can be advantageous, but in the more common scenario where we have limited time or a limited evaluation budget, the sequential nature of closed loop methods can be cumbersome. In addition, it has been observed that many Bayesian optimization methods with a moderate number of hyperparameters, when run for k iterations, can be outperformed by sampling 2k points uniformly at random, indicating that even simple open loop methods can be competitive. Parallelizing Bayesian optimization methods has proven to be nontrivial, though many agree that it's vitally important. While many algorithms exist which can sample more than one point at each iteration BID7 BID8 BID9 BID14, the sequential nature of Bayesian optimization methods prevent the full parallelization open loop methods can employ. Even running two iterations (with batches of size k/2) will take on average twice as long as fully parallelizing the evaluations, as you can do with open loop methods like grid search, sampling uniformly, or sampling according to a DPP.One line of research has examined the use of k-DPPs for optimizing hyperparameters in the context of parallelizing Bayesian optimization BID15 BID27. At each iteration within one trial of Bayesian optimization, instead of drawing a single new point to evaluate from the posterior, they define a k-DPP over a relevance region from which they sample a diverse set of points. They found their approach to beat state-of-the-art performance on a number of hyperparameter optimization tasks, and they proved that generating batches by sampling from a k-DPP has better regret bounds than a number of other approaches. They show that a previous batch sampling approach which selects a batch by sequentially choosing a point which has the highest posterior variance BID7 is just approximating finding the maximum probability set from a k-DPP (an NP-hard problem BID18), and they prove that sampling (as opposed to maximization) has better regret bounds for this optimization task. We use the work of BID15 as a foundation for our exploration of fully-parallel optimization methods, and thus we focus on k-DPP sampling as opposed to maximization. So-called configuration evaluation methods have been shown to perform well by adaptively allocating resources to different hyperparameter settings BID26 BID19. They initially choose a set of hyperparameters to evaluate (often uniformly), then partially train a set of models for these hyperparameters. After some fixed training budget (e.g., time, or number of training examples observed), they compare the partially trained models against one another and allocate more resources to those which perform best. Eventually, these algorithms produce one (or a small number) of fully trained, high-quality models. In some sense, these approaches are orthogonal to open vs. closed loop methods, as the diversity-promoting approach we advocate can be used as a drop-in replacement to the method used to choose the initial hyperparameter assignments. GPs have long been lauded for their expressive power, and have been used extensively in the hyperparameter optimization literature. BID10 show that drawing a sample from a k-DPP with kernel K is equivalent to sequentially sampling k times proportional to the (updated) posterior variance of a GP defined with covariance kernel K. This sequential sampling is one of the oldest hyperparameter optimization algorithms, though our work is the first to perform an in-depth analysis. Additionally, this has a nice information theoretic justification: since the entropy of a Gaussian is proportional to the log determinant of the covariance matrix, points drawn from a DPP have probability proportional to exp(information gain), and the most probable set from the DPP is the set which maximizes the information gain. With our MCMC algorithm presented in Algorithm 2, we can draw samples with these appealing properties from any space for which we can draw uniform samples. The ability to draw k-DPP samples by sequentially sampling points proportional to the posterior variance grants us another boon: if one has a sample of size k and wants a sample of size k + 1, only a single additional point needs to be drawn, unlike with the sampling algorithms presented in BID18. Using this approach, we can draw samples up to k = 100 in less than a second on a machine with 32 cores. As discussed above, recent trends have renewed interest in open loop methods. While there exist many different batch BO algorithms, analyzing these in the open loop regime (when there are no from function evaluations) is often rather simple. As there is no information with which to update the posterior mean, function evaluations are hallucinated using the prior or points are drawn only using information about the posterior variance. For example, in the open loop regime, BID14's approach without hallucinated observations is equivalent to uniform sampling, and their approach with hallucinated observations (where they use the prior mean in place of a function evaluation, then update the posterior mean and variance) is equivalent to sequentially sampling according to the posterior variance (which is the same as sampling from a DPP). Similarly, open loop optimization in SMAC BID12 ) is equivalent to first Latin hypercube sampling to make a large set of diverse candidate points, then sampling k uniformly among these points. Recently, uniform sampling was shown to be competitive with sophisticated closed loop methods for modern hyperparameter optimization tasks like optimizing the hyperparameters of deep neural networks, inspiring other works to explain the phenomenon BID0. BID3 offer one of the most comprehensive studies of open loop methods to date, and focus attention on comparing random search and grid search. A main takeaway of the paper is that uniform random sampling is generally preferred to grid search 2 due to the frequent observation that some hyperparameters have little impact on performance, and random search promotes more diversity in the dimensions that matter. Essentially, if points are drawn uniformly at random in d dimensions but only d < d dimensions are relevant, those same points are uniformly distributed (and just as diverse) in d dimensions. Grid search, on the other hand, distributes configurations aligned with the axes so if only d < d dimensions are relevant, many configurations are essentially duplicates. However, grid search does have one favorable property that is clear in just one dimension. If k points are distributed on on a grid, the maximum spacing between points is equal to 1 k−1. But if points are uniformly at random drawn on, the expected largest gap between points scales as DISPLAYFORM0 ) is a point on the grid for ij = 0, 1,..., m for all j, with a total number of grid points equal to (m + 1) d.bad luck, the optimum islocated in this largest gap, this difference could be considerable; we attempt to quantify this idea in the next section. Quantifying the spread of a sequence x = (x 1, x 2, . . ., x k) (or, similarly, how well x covers a space) is a well-studied concept. In this section we introduce discrepancy, a quantity used by previous work, and dispersion, which we argue is more appropriate for optimization problems. Perhaps the most popular way to quantify the spread of a sequence is star discrepancy. One can interpret the star discrepancy as a multidimensional version of the Kolmogorov-Smirnov statistic between the sequence x and the uniform measure; intuitively, when x contains points which are spread apart, star discrepancy is small. We include a formal definition in Appendix A.Star discrepancy plays a prominent role in the numerical integration literature, as it provides a sharp bound on the numerical integration error through the the Koksma-Hlawka inequality (given in Appendix B) BID11. This has led to wide adoption of low discrepancy sequences, even outside of numerical integration problems. For example, BID3 analyzed a number of low discrepancy sequences for some optimization tasks and found improved optimization performance over uniform sampling and grid search. Additionally, low discrepancy sequences such as the Sobol sequence 3 are used as an initialization procedure for some Bayesian optimization schemes BID23. Previous work on open loop hyperparameter optimization focused on low discrepancy sequences BID3 BID6, but optimization performance-how close a point in our sequence is to the true, fixed optimum-is our goal, not a sequence with low discrepancy. As discrepancy doesn't directly bound optimization error, we turn instead to dispersion DISPLAYFORM0 where ρ is a distance (in our experiments L 2 distance). Intuitively, the dispersion of a point set is the radius of the largest Euclidean ball containing no points; dispersion measures the worst a point set could be at finding the optimum of a space. Following BID21, we can bound the optimization error as follows. Let f be the function we are aiming to optimize (maximize) with domain B, m(f) = sup x∈B f (x) be the global optimum of the function, and m k (f ; x) = sup 1≤i≤k f (x i) be the best-found optimum from the set x. Assuming f is continuous (at least near the global optimum), the modulus of continuity is defined as DISPLAYFORM1 Theorem 1. BID21 For any point set x with dispersion d k (x), the optimization error is bounded as DISPLAYFORM2 Dispersion can be computed efficiently (unlike discrepancy, D k (x), which is NP-hard BID29 ), and we give an algorithm in Appendix C. Dispersion is at least Ω(k −1/d), and while low discrepancy implies low BID3 found that the Niederreiter and Halton sequences performed similarly to the Sobol sequence, and that the Sobol sequence outperformed Latin hypercube sampling. Thus, our experiments include the Sobol sequence (with the Cranley-Patterson rotation) as a representative low-discrepancy sequence. not hold. 4 Therefore we know that the low-discrepancy sequences evaluated in previous work are also low-dispersion sequences in the big-O sense, but as we will see they may behave quite differently. Samples drawn uniformly are not low dispersion, as they have rate (ln(k)/k) 1/d BID29. DISPLAYFORM3 Optimal dispersion in one dimension is found with an evenly spaced grid, but it's unknown how to get an optimal set in higher dimensions.5 Finding a set of points with the optimal dispersion is as hard as solving the circle packing problem in geometry with k equal-sized circles which are as large as possible. Dispersion is bounded from below with DISPLAYFORM4 it is unknown if this bound is sharp. In FIG4 we plot the dispersion of the Sobol sequence, samples drawn uniformly at random, and samples drawn from a k-DPP, in one and two dimensions. To generate the k-DPP samples, we sequentially drew samples proportional to the (updated) posterior variance (using an RBF kernel, with σ = √ 2/k), as described in Section 2.2. When d = 1, the regular structure of the Sobol sequence causes it to have increasingly large plateaus, as there are many "holes" of the same size. For example, the Sobol sequence has the same dispersion for 42 ≤ k ≤ 61, and 84 ≤ k ≤ 125. Samples drawn from a k-DPP appear to have the same asymptotic rate as the Sobol sequence, but they don't suffer from the plateaus. When d = 2, the k-DPP samples have lower average dispersion and lower variance. One other natural surrogate of average optimization performance is to measure the distance from a fixed point, say 2 ) or from the origin, to the nearest point in the length k sequence. Our experiments (in Appendix D) on these metrics show the k-DPP samples bias samples to the corners of the space, which can be beneficial when the practitioner defined the search space with bounds that are too small. Note, the low-discrepancy sequences are usually defined only for the d hypecrube, so for hyperparameter search which involves conditional hyperparameters (i.e. those with tree structure) they are not appropriate. In what follows, we study the k-DPP in more depth and how it performs on real-world hyperparameter tuning problems. We begin by reviewing DPPs and k-DPPs. Let B be a domain from which we would like to sample a finite subset. (In our use of DPPs, this is the set of hyperparameter assignments.) In general, B could be discrete or continuous; here we assume it is discrete with N values, and we define Y = {1, . . ., N} to be a a set which indexes B (this index set will be particularly useful in Algorithm 1). In Section 4.2 we address when B has continuous dimensions. A DPP defines a probability distribution over 2 Y (all subsets of Y) with the property that two elements of Y are more (less) likely to both be chosen the more dissimilar (similar) they are. Let random variable Y range over finite subsets of Y.There are several ways to define the parameters of a DPP. We focus on L-ensembles, which define the probability that a specific subset is drawn (i.e., P (Y = A) for some A ⊂ Y) as: DISPLAYFORM0 As shown in BID18, this definition of L admits a decomposition to terms representing the quality and diversity of the elements of Y. For any y i, y j ∈ Y, let: DISPLAYFORM1 4 Discrepancy is a global measure which depends on all points, while dispersion only depends on points near the largest "hole".5 In two dimensions a hexagonal tiling finds the optimal dispersion, but this is only valid when k is divisible by the number of columns and rows in the tiling.6 By construction, each individual dimension of the d-dimensional Sobol sequence has these same plateaus. where q i > 0 is the quality of y i, φ i ∈ R d is a featurized representation of y i, and K: DISPLAYFORM2 is a similarity kernel (e.g. cosine distance). (We will discuss how to featurize hyperparameter settings in Section 4.3.)Here, we fix all q i = 1; in future work, closed loop methods might make use of q i to encode evidence about the quality of particular hyperparameter settings to adapt the DPP's distribution over time. DPPs have support over all subsets of Y, including ∅ and Y itself. In many practical settings, one may have a fixed budget that allows running the training algorithm k times, so we require precisely k elements of Y for evaluation. k-DPPs are distributions over subsets of Y of size k. Thus, DISPLAYFORM0 Sampling from k-DPPs has been well-studied. When the base set B is a set of discrete items, exact sampling algorithms are known which run in O(N k 3) BID18. When the base set is a continuous hyperrectangle, a recent exact sampling algorithm was introduced, based on a connection with Gaussian processes (GPs), which runs in O(dk 2 + k 3) BID10. We are unaware of previous work which allows for sampling from k-DPPs defined over any other base sets. BID1 present a Metropolis-Hastings algorithm (included here as Algorithm 1) which is a simple and fast alternative to the exact sampling procedures described above. However, it is restricted to discrete domains. We propose a generalization of the MCMC algorithm which preserves relevant computations while allowing sampling from any base set from which we can draw uniform samples, including those with discrete dimensions, continuous dimensions, some continuous and some discrete dimensions, or even (conditional) tree structures (Algorithm 2). To the best of our knowledge, this is the first algorithm which allows for sampling from a k-DPP defined over any space other than strictly continuous or strictly discrete, and thus the first algorithm to utilize the expressive capabilities of the posterior variance of a GP in these regimes. set Y = Y ∪ {v} \ {u} with probability p: Y = Y 7: Return B Y Algorithm 1 proceeds as follows: First, initialize a set Y with k indices of L, drawn uniformly. Then, at each iteration, sample two indices of L (one within and one outside of the set Y), and with some probability replace the item in Y with the other. When we have continuous dimensions in the base set, however, we can't define the matrix L, so sampling indices from it is not possible. We propose Algorithm 2, which samples points directly from the base set B instead (assuming continuous dimensions are bounded), and computes only the principal minors of L needed for the relevant computations on the fly. Algorithm 2 Drawing a sample from a k-DPP defined over a space with continuous and discrete dimensions Input: A base set B with some continuous and some discrete dimensions, a quality function Ψ: compute the quality score for each item, q i = Ψ(β i), ∀i, and DISPLAYFORM0 DISPLAYFORM1 with probability p: β = β 10: Return β Even in the case where the dimensions of B are discrete, Algorithm 2 requires less computation and space than Algorithm 1 (assuming the quality and similarity scores are stored once computed, and retrieved when needed). Previous analyses claimed that Algorithm 1 should mix after O(N log(N)) steps. There are O(N 2) computations required to compute the full matrix L, and at each iteration we will compute at most O(k) new elements of L, so even in the worst case we will save space and computation whenever k log(N) < N. In expectation, we will save significantly more. Let φ i be a feature vector for y i ∈ Y, a modular encoding of the attribute-value mapping assigning values to different hyperparameters, in which fixed segments of the vector are assigned to each hyperparameter attribute (e.g., the dropout rate, the choice of nonlinearity, etc.). For a hyperparameter that takes a numerical value in range [h min, h max], we encode value h using one dimension (j) of φ and project into the range:, and hence label our approach k-DPP-RBF. Values for σ 2 lead to models with different properties; when σ 2 is small, points that are spread out interact little with one another, and when σ 2 is large, the increased repulsion between the points encourages them to be as far apart as possible. DISPLAYFORM0 Many real-world hyperparameter search spaces are tree-structured. For example, the number of layers in a neural network is a hyperparameter, and each additional layer adds at least one new hyperparameter which ought to be tuned (the number of nodes in that layer). For a binary hyperparameter like whether or not to use regularization, we use a one-hot encoding. When this hyperparameter is "on," we set the associated regularization strength as above, and when it is "off" we set it to zero. Intuitively, with all other hyperparameter settings equal, this causes the off-setting to be closest to the least strong regularization. One can also treat higher-level design decisions as hyperparameters BID17, such as whether to train a logistic regression classifier, a convolutional neural network, or a recurrent neural network. In this construction, the type of model would be a categorical variable (and thus get a one-hot encoding), and all child hyperparameters for an "off" model setting (such as the convergence tolerance for logistic regression, when training a recurrent neural network) would be set to zero. In this section we present our hyperparameter optimization experiments. Our experiments consider a setting where hyperparameters have a large effect on performance: a convolutional neural network for text classification BID16. The task is binary sentiment analysis on the Stanford sentiment treebank BID25. On this balanced dataset, random guessing leads to 50% accuracy. We use the CNN-non-static model from BID16, with skip-gram BID20 vectors. The model architecture consists of a convolutional layer, a max-over-time pooling layer, then a fully connected layer leading to a softmax. All k-DPP samples are drawn using Algorithm 2. We begin with a search over three continuous hyperparameters and one binary hyperparameter, with a simple tree structure: the binary hyperparameter indicates whether or not the model will use L 2 regularization, and one of the continuous hyperparameters is the regularization strength. We assume a budget of k = 20 evaluations by training the convolutional neural net. L 2 regularization strengths in the range [e −5, e −1] (or no regularization) and dropout rates in [0.0, 0.7] are considered. We consider three increasingly "easy" ranges for the learning rate:• Hard: [e −5, e 5], where the majority of the range leads to accuracy no better than chance.• Medium: [e −5, e −1], where half of the range leads to accuracy no better than chance.• Easy: [e −10, e −3], where the entire range leads to models that beat chance. FIG7 shows the accuracy (averaged over 50 runs) of the best model found after exploring 1, 2,..., k hyperparameter settings. We see that k-DPP-RBF finds better models with fewer iterations necessary than the other approaches, especially in the most difficult case. FIG7 compares the sampling methods against a Bayesian optimization technique using a tree-structured Parzen estimator (BO-TPE; BID5 . This technique evaluates points sequentially, allowing the model to choose the next point based on how well previous points performed (a closed loop approach). It is state-of-the-art on tree-structured search spaces (though its sequential nature limits parallelization). Surprisingly, we find it performs the worst, even though it takes advantage of additional information. We hypothesize that the exploration/exploitation tradeoff in BO-TPE causes it to commit to more local search before exploring the space fully, thus not finding hard-to-reach global optima. Note that when considering points sampled uniformly or from a DPP, the order of the k hyperparameter settings in one trial is arbitrary (though this is not the case with BO-TPE as it is an iterative algorithm). In all cases the variance of the best of the k points is lower than when sampled uniformly, and the differences in the plots are all significant with p < 0.01. BID28 analyzed the stability of convolutional neural networks for sentence classification with respect to a large set of hyperparameters, and found a set of six which they claimed had the largest impact: the number of kernels, the difference in size between the kernels, the size of each kernel, dropout, regularization strength, and the number of filters. We optimized over their prescribed "Stable" ranges for three open loop methods and one closed loop method; average accuracies with 95 percent confidence intervals from 50 trials of hyperparameter optimization are shown in Figure 3, across k = 5, 10, 15, 20 iterations. We find that even when optimizing over a space for which all values lead to good models, k-DPP-RBF outperforms the other methods. Our experiments reveal that, while the hyperparameters proposed by BID28, can have an effect, the learning rate, which they do not analyze, is at least as impactful. Here we compare our approach against Spearmint BID23, perhaps the most popular Bayesian optimization package. Figure 4 shows wall clock time and accuracy for 25 runs on the "Stable" search space of four hyperparameter optimization approaches: k-DPP-RBF (with k = 20), batch Spearmint with 2 iterations of batch size 10, batch Spearmint with 10 iterations of batch size 2, and sequential Spearmint 7. Each point in the plot is one hyperparameter assignment evaluation. The vertical lines represent how long, on average, it takes to find the best in one run. We see that all evaluations for k-DPP-RBF finish quickly, while even the fastest batch method (2 batches of size 10) takes nearly twice as long on average to find a good . The final average best-found accuracies are 82.61 for k-DPP-RBF, 82.65 for Spearmint with 2 batches of size 10, 82.7 for Spearmint with 10 batches of size 2, and 82.76 for sequential Spearmint. Thus, we find it takes on average more than ten times as long for sequential Spearmint to find its best solution, for a gain of only 0.15 percent accuracy. We have explored open loop hyperparameter optimization built on sampling from a k-DPP. We described how to define a k-DPP over hyperparameter search spaces, and showed that k-DPPs retain the attractive parallelization capabilities of random search. In synthetic experiments, we showed k-DPP samples perform well on a number of important metrics, even for large values of k. In hyperprameter optimization experiments, we see k-DPP-RBF outperform other open loop methods. Additionally, we see that sequential methods, even when using more than ten times as much wall clock time, gain less than 0.16 percent accuracy on a particular hyperparameter optimization problem. An open-source implementation of our method is available. A STAR DISCREPANCY DISPLAYFORM0 It is well-known that a sequence chosen uniformly at random from d has an expected star discrepancy of at least 1 k (and is no greater than DISPLAYFORM1) BID22 whereas sequences are known to exist with star discrepancy less than BID24, where both bounds depend on absolute constants. DISPLAYFORM2 Comparing the star discrepancy of sampling uniformly and Sobol, the bounds suggest that as d grows large relative to k, Sobol starts to suffer. Indeed, BID2 notes that the Sobol rate is not even valid until k = Ω(2 d) which motivates them to study a formulation of a DPP that has a star discrepancy between Sobol and random and holds for all k, small and large. They primarily approached this problem from a theoretical perspective, and didn't include experimental . Their work, in part, motivates us to look at DPPs as a solution for hyperparameter optimization. B KOKSMA-HLAWKA INEQUALITY Let B be the d-dimensional unit cube, and let f have bounded Hardy and Krause variation V ar HK (f) on B. Let x = (x 1, x 2, . . ., x k) be a set of points in B at which the function f will be evaluated to approximate an integral. The Koksma-Hlawka inequality bounds the numerical integration error by the product of the star discrepancy and the variation: DISPLAYFORM3 We can see that for a given f, finding x with low star discrepancy can improve numerical integration approximations. Find a (bounded) voronoi diagram over the search space for a point set X k. For each vertex in the voronoi diagram, find the closest point in X k. The dispersion is the max over these distances. One natural surrogate of average optimization performance is to define a hyperparameter space on d and measure the distance from a fixed point, say is motivated by a quadratic Taylor series approximation around the minimum of the hypothetical function we wish to minimize. In the first columns of Figure 5 we plot the smallest distance from the center 1 2 1, as a function of the length of the sequence (in one dimension) for the Sobol sequence, uniform at random, and a DPP. We observe all methods appear comparable when it comes to distance to the center. of what low discrepancy sequences attempt to do. While Sobol and uniformly random sequences will not bias themselves towards the corners, a DPP does. This happens because points from a DPP are sampled according to how distant they are from the existing points; this tends to favor points in the corners. This same behavior of sampling in the corners is also very common for Bayesian optimization schemes, which is not surprise due to the known connections between sampling from a DPP and Gaussian processes (see Section 2.2). In the second column of Figure 5 we plot the distance to the origin which is just an arbitrarily chosen corner of hypercube. As expected, we observe that the DPP tends to outperform uniform at random and Sobol in this metric. Figure 5: Comparison of the Sobol sequence, samples a from k-DPP, and uniform random for two metrics of interest. These log-log plots show uniform sampling and k-DPP-RBF performs comparably to the Sobol sequence in terms of distance to the center, but on another (distance to the origin) k-DPP-RBF samples outperform the Sobol sequence and uniform sampling.

We address fully parallel hyperparameter optimization with Determinantal Point Processes.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.