Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Semantic parsing which maps a natural language sentence into a formal machine-readable representation of its meaning, is highly constrained by the limited annotated training data. Inspired by the idea of coarse-to-fine, we propose a general-to-detailed neural network(GDNN) by incorporating cross-domain sketch(CDS) among utterances and their logic forms. For utterances in different domains, the General Network will extract CDS using an encoder-decoder model in a multi-task learning setup. Then for some utterances in a specific domain, the Detailed Network will generate the detailed target parts using sequence-to-sequence architecture with advanced attention to both utterance and generated CDS. Our experiments show that compared to direct multi-task learning, CDS has improved the performance in semantic parsing task which converts users' requests into meaning representation language(MRL). We also use experiments to illustrate that CDS works by adding some constraints to the target decoding process, which further proves the effectiveness and rationality of CDS. Recently many natural language processing (NLP) tasks based on the neural network have shown promising and gained much attention because these studies are purely data-driven without linguistic prior knowledge. Semantic parsing task which maps a natural language sentence into a machine-readable representation BID6 ), as a particular translation task, can be treated as a sequence-to-sequence problem BID3 ). Lately, a compositional graph-based semantic meaning representation language (MRL) has been introduced BID14 ), which converts utterance into logic form (action-object-attribute), increasing the ability to represent complex requests. This work is based on MRL format for semantic parsing task. Semantic parsing highly depends on the amount of annotated data and it is hard to annotate the data in logic forms such as Alexa MRL. Several researchers have focused on the area of multi-task learning and transfer learning BID10, BID6, BID15 ) with the observation that while these tasks differ in their domains and forms, the structure of language composition repeats across domains BID12 ). Compared to the model trained on a single domain only, a multi-task model that shares information across domains can improve both performance and generalization. However, there is still a lack of interpretations why the multi-task learning setting works BID26 ) and what the tasks have shared. Some NLP studies around language modeling BID18, BID29, BID2 ) indicate that implicit commonalities of the sentences including syntax and morphology exist and can share among domains, but these commonalities have not been fully discussed and quantified. To address this problem, in this work, compared to multi-task learning mentioned above which directly use neural networks to learn shared features in an implicit way, we try to define these cross-domain commonalities explicitly as cross-domain sketch (CDS). E.g., Search weather in 10 days in domain Weather and Find schedule for films at night in domain ScreeningEvent both have action SearchAction and Attribute time, so that they share a same MRL structure like SearchAction(Type(time@?)), where Type indicates domain and? indicates attribute value which is copying from the original utterance. We extract this domain general MRL structure as CDS. Inspired by the research of coarse-to-fine BID4 ), we construct a two-level encoder-decoder by using CDS as a middle coarse layer. We firstly use General Network to get the CDS for every utterance in all domains. Then for a single specific domain, based on both utterance and extracted CDS, we decode the final target with advanced attention while CDS can be seen as adding some constraints to this process. The first utterance-CDS process can be regarded as a multi-task learning setup since it is suitable for all utterances across the domains. This work mainly introducing CDS using multi-task learning has some contributions listed below: 1) We make an assumption that there exist cross-domain commonalities including syntactic and phrasal similarity for utterances and extract these commonalities as cross-domain sketch (CDS) which for our knowledge is the first time. We then define CDS on two different levels (action-level and attribute-level) trying to seek the most appropriate definition of CDS.2) We propose a general-to-detailed neural network by incorporating CDS as a middle coarse layer. CDS is not only a high-level extraction of commonalities across all the domains, but also a prior information for fine process helping the final decoding.3) Since CDS is cross-domain, our first-level network General Network which encodes the utterance and decodes CDS can be seen as a multi-task learning setup, capturing the commonalities among utterances expressions from different domains which is exactly the goal of multi-task learning. Traditional spoken language understanding (SLU) factors language understanding into domain classification, intent prediction, and slot filling, which proves to be effective in some domains BID9 ). Representations of SLU use pre-defined fixed and flat structures, which limit its expression skills like that it is hard to capture the similarity among utterances when the utterances are from different domains BID15 ). Due to SLU's limited representation skills, meaning representation language (MRL) has been introduced which is a compositional graph-based semantic representation, increasing the ability to represent more complex requests BID14 ). There are several different logic forms including lambda-calculus expression BID17 ), SQL BID33 ), Alexa MRL BID14 ). Compared to fixed and flat SLU representations, MRL BID14 ) based on a large-scale ontology, is much stronger in expression in several aspects like cross-domain and complex utterances. Mapping a natural language utterance into machine interpreted logic form (such as MRL) can be regarded as a special translation task, which is treated as a sequence-to-sequence problem BID28 ). Then BID1 and BID22 advance the sequence-tosequence network with attention mechanism learning the alignments between target and input words, making great progress in the performance. BID23 explore the attention mechanism with some improvements by replacing attention function with attention sparsity. Besides, to deal with the rare words, BID8 incorporate the copy mechanism into the encoder-decoder model by directly copying words from inputs. Lately, many researchers have been around improving sequence-to-sequence model itself, in interpreting the sentence syntax information. BID5 encode the input sentence recursively in a bottom-up fashion. BID30 generate the target sequence and syntax tree through actions simultaneously. Another aspect which has caught much attention is constrained decoding. BID16 and BID25 add some constraints into decoding process, making it more controllable. BID3 use the recurrent network as encoder which proves effective in sequence representation, and respectively use the recurrent network as decoder and tree-decoder. BID16 employ the grammar to constrain the decoding process. BID4, believe utterance understanding is from high-level to low-level and by employing sketch, improve the performance. For semantic parsing task especially in MRL format, it is expensive and time-consuming to annotate the data, and it is challenging to train semantic parsing neural models. Multi-task learning aims to use other related tasks to improve target task performance. BID20 deal with traditional SLU piper-line network by jointly detecting intent and doing slot filling. BID27 share parameters among various tasks, according to the low-level and high-level difference. BID11 divide the representation network into task-specific and general which is shared during multi-task learning. BID6 and BID12 directly share the encoder or decoder neural layers (model params) through different semantic parsing tasks. In BID15, multi-task learning also mainly acts sharing the params of the network. For human language expressions, especially task-oriented requests, there exist commonalities across sentences including linguistic syntax and phrase similarity. They can be seen with general sentence templates. TAB1 Since sentence templates are too many, we try to leverage these common regularities more abstractly. We extract these invariant commonalities which are implicit across domains, and call them as crossdomain sketch (CDS) in a canonical way. We define CDS in meaning representation language (MRL) format (action-object-attribute) and on two levels (action-level and attribute-level). Action-level CDS means to acquire the same action for utterances from different domains while the attribute-level CDS means to extract more detailed information. See examples in TAB1. Instead of extracting CDS from utterance directly, we try converting from logic form into CDS reversely, because it is easier to deal with structural logic form than utterance in natural language form. We analyze the dataset Snips and use a rule-based method to obtain CDS. We strip logic forms from domain-specific components and preserve domainindependent parts including general actions and attributes. We do some statistics on the dataset Snips BID7 ) used in this paper. We convert attributes [object type, movie type, restaurant type] into {object type}, [object name, restaurant name, movie name] into {object name}, [year, timeRange] into {time}, [location name, current location] into {object location}, [country, city] into {place}. All those attributes account for 55% of all attributes which indicate the existence and feasibility of CDS. Figure 1 shows our overall network, which contains a two-level encoder-decoder. The General Network encodes utterance and decodes cross-domain sketch (CDS). Since this process is domaingeneral, it can be done to all domains, which is a multi-task setup. The Detailed Network firstly encodes the CDS and the utterance, then it decodes the target based on both utterance and CDS. This process is domain-dependent, so that it is a fine-tuning process in a specific domain. For an input utterance u = u 1, u 2,...u |u|, and its middle cross-domain sketch (CDS) c = c 1, c 2,...c |c|, and its final logic form y = y 1, y 2,...y |y|, the conditional probability is: Figure 1: Overall Network. General Network (red dashed box below) encodes the utterance with bi-directional LSTM and decodes cross-domain sketch (CDS) using unidirectional LSTM with attention to utterance in all domains. For identical encoding, general utterance encoding and specific utterance encoding share the same encoder while for separate encoding, they are not (see Section 3.2.2). Then Detailed Network, in one specific domain, encodes CDS and utterance using bi-directional LSTM, decodes the final target with advanced attention to both utterance and CDS. DISPLAYFORM0 where y <t = y 1, y 2,...y |t−1|, and c <t = c 1, c 2,...c |t−1|. The neural encoder of our model is similar to neural machine translation (NMT) model, which uses a bi-directional recurrent neural network. Firstly each word of utterance is mapped into a vector u t ∈ R d via embedding layer and we get a word sequence u = (u 1, ..., u |u|). Then we use a bi-directional recurrent neural network with long short-term memory units (LSTM) BID13 ) to learn the representation of word sequence. We generate forward hidden state DISPLAYFORM0 The t-th word will be h DISPLAYFORM1 We construct two kinds of utterance encoders, general utterance encoder for General Network and specific utterance encoder for Detailed Network (see in Figure 1), so as to extract different information for different purposes. The general utterance encoder, meant to pay more attention to cross-domain commonalities of utterances, is used by all utterances from all domains. The specific utterance encoder, which is domain-dependent, belongs to one specific domain and is more sensitive to details. We call encoder outputs h ug t from general utterance encoder and h us t from specific utterance encoder. When the two encoders share the same parameters that is h ug t = h us t, we call it identical encoding and when they are not, we call it separate encoding, inspired by BID27; BID16; BID21; BID0 ) which explore the sharing mechanisms of multi-task learning and propose some improvements. The General Network is meant to obtain cross-domain sketch (CDS) c conditioned on utterance u, using an encoder-decoder network. After encoding utterance by general utterance encoder for all domains, we obtain h ug t (see Section 3.2.2). Then we start to decode CDS.The decoder is based on a unidirectional recurrent neural network, and the output vector is used to predict the word. The cd represents CDS decoder. DISPLAYFORM0 where c t is the previously predicted word embedding. The LuongAttention BID22 ) between decoding hidden state d t and encoding sequence e i (i = 1, 2, ...|e|) at time step t is computed as: DISPLAYFORM1 DISPLAYFORM2 Based on equations FORMULA4 and FORMULA5, we compute the attention a The t-th predicted output token will be: DISPLAYFORM0 DISPLAYFORM1 where W, b are parameters. After decoding CDS words c = (c 1, ..., c |c|), we use an encoder to represent its meaning and due to words' relation with forward and backward contexts, we choose to use a bi-directional LSTM. We generate forward hidden state − → h Through specific utterance encoder and cross-domain sketch (CDS) encoder, we acquired t-th word representation h us t and h ce t. Finally with advanced attention to both encoded utterance u and CDS c, we decode the final target y. The decoder is based on a unidirectional recurrent neural network, and the output vector is used to predict the word. The y represents target decoder. DISPLAYFORM0 where y t is the previously predicted word embedding. During target decoding process and at time step t, we not only compute the attention to utterance encoding outputs h us but also compute the attention to CDS encoding outputs h ce. The attention between target hidden state and utterance is a 1, 2,...|c|) in the same way. Then the t-th predicted output token will be based on the advanced two-aspect attention: DISPLAYFORM1 For training process, the objective is: DISPLAYFORM0 T is the training corpus. For inference process, we firstly obtain cross-domain sketch (CDS) via c = argmax p(c|u) then we get the final target logic form via y = argmax p(y|u, c). For both decoding processes, we use greedy search to generate words one by one. Existed semantic parsing datasets, e.g., GEO BID32 ), ATIS BID31 ), collect data only from one domain and have a very limited amount, which can not fully interpret the effectiveness of cross-domain sketch (CDS) since it needs large dataset among different domains. In this case, we mainly consider the semantic parsing task Snips (Goo et al. FORMULA0) based on MRL format (action-object-attribute). Snips collects users' requests from a personal voice assistant. The original dataset is annotated in spoken language understanding (SLU) format (intent-slot). It has 7 intent types and 72 slot labels, and more statistics are shown in TAB4. Based on the format (intent-slot), we pre-process this dataset into MRL format by some pre-defined rules, then we regard the intent as domain/task and share CDS among them. The details are shown in Target SearchAction (ScreeningEventType ( object type @ 2, movie type @ 4, timeRange @ 6, location name @ 8 9 10) ) TAB3: Several examples of Snips. Utterance is the user's request which is a natural language expression. Intent and slots are in formats from original dataset. Cross-domain sketch (CDS) has two levels (action-level and attribute-level). Target is the final logic form with numbers indicating copying words from utterance (index starting from 0). We use Tensorflow in all our experiments, with LuongAttention BID22 ) and copy mechanism. The embedding dimension is set to 100 and initialized with GloVe embeddings (Pennington et al. FORMULA0). The encoder and decoder both use one-layer LSTM with hidden size 50. We apply the dropout selected in {0.3,0.5}. Learning rate is initialized with 0.001 and is decaying during training. Early stopping is applied. The mini-batch size is set to 16. We use the logic form accuracy as the evaluation metric. Firstly, in order to prove the role of the cross-domain sketch (CDS) in helping to guide decoding process with multi-tasking learning setup, we do several experiments, and the are shown in Table 4. For joint learning, we apply several multi-task architectures from BID6 ), including one-to-one, one-to-many and one-to-shareMany. One-to-one architecture applies a single sequence-to-sequence model across all the tasks. One-to-many only shares the encoder across all the tasks while the decoder including the attention parameters is not shared. In one-to-shareMany model, tasks share encoder and decoder (including attention) params, but the output layer of decoder is task-independent. From the Table 4, in general, joint learning performs better than single task learning. In joint learning, one-to-one is the best and performs way better than one-to-many and one-to-shareMany, probably limited by the dataset's size and similarity among tasks. By incorporating CDS, our GDNN (general-to-detailed neural network) models have all improved the performance to different degrees. The CDS is defined on two levels (action-level and attribute-level, see examples in TAB3) and attribute-level CDS improves greater than action-level CDS, which is in our expectation since it offers more information for tasks to share. We also experiment on different utterance encoding setups with identical encoding and separate encoding (see Section 3.2.2). The separate encoding setup performs better than sharing the same encoder for utterance, which integrates the fact that different encoders pay different attention to the utterances due to different purposes which means one is more general and the other is more specific detailed. Method Snips Accuracy Single Seq2Seq BID28 62.3 Joint Seq2Seq (one-to-many) BID6 62.0 Joint Seq2Seq (one-to-shareMany) BID6 64.2 Joint Seq2Seq (one-to-one) BID6 71.4 GDNN with Action-level CDS (identical encoding) 74.9 GDNN with Action-level CDS (separate encoding) 75.1 GDNN with Attribute-level CDS (identical encoding) 76.7 GDNN with Attribute-level CDS (separate encoding) 78.1 Table 4: Multi-task Results. Single Seq2Seq means each task has a sequenece-to-sequence model. Joint Seq2Seq show with three multi-task mechanisms. Our include GDNN (generalto-detailed neural network) models with different levels of CDS (action-level/attribute level) and different utterance encoding mechanisms (identical encoding/separate encoding).We also list the full of GDNN in TAB6 Moreover, we compare our experiments with traditional models which regard the task as intent classification and slot filling (IC SF). The are shown in TAB7 below. From TAB7, we can see compared to IC SF models (based on sequence labeling format), Seq2Seq perform worse (71.4% compared to 73.2%) due to its fewer assumptions and larger decode size as well as its difficulty of training, which is usual in comparing IC SF models and sequence-tosequence models. Through using CDS, the performance has significantly improved. On the one Method Snips Accuracy Joint Seq. BID10 73.2 Atten.-Based BID19 74.1 Slot.-Gated (Intent Atten.) BID7 74.6 Slot.-Gated (Full Atten.) BID7 75.5 Joint Seq2Seq BID6 71.4 GDNN with Action-level CDS 73.2 GDNN with Attribute-level CDS 74.6 hand, CDS extract the cross-domain commonalities among tasks helping to make the multi-task learning more specific, which can be seen as an advance to multi-task learning. On the other hand, CDS can be seen adding some constraints to the final target decoding process which has offered more information for the decoding process, compared to direct joint Seq2Seq. To better prove and explain this idea, we do some experiments according to constraint decoding aspect. We try to compare the sub-process of converting utterance to CDS through different models, e.g., IC SF, Seq2Seq. From the Table 7, we can see that Seq2Seq achieve the comparable (87.7%) to IC SF model (84.9%) for generating CDS from utterance, which further explains that, the fact joint seq2seq performs worse (71.4%, see TAB7) than IC SF model (73.2%) is owing to the lack of guidance and constraints during the follow-up decoding process. By incorporating CDS, we add some constraints to this decoding process thus obtaining better performance. Table 7: Results of CDS generation in dataset Snips by two methods. IC SF is using intent classification and slot filling with evaluation metric (intent accuracy, slot labelling accuracy and final accuracy). Seq2Seq generates CDS based on utterance using an encoder-decoder. In this paper, we propose the concept of cross-domain sketch (CDS) which extracts some shared information across domains, trying to fully utilize the cross-domain commonalities such as syntactic and phrasal similarity in human expressions. We try to define CDS on two levels and give some examples to illustrate our idea. We also present a general-to-detailed neural network (GDNN) for converting an utterance into a logic form based on meaning representation language (MRL) form. The general network, which is meant to extract cross-domain commonalities, uses an encoderdecoder model to obtain CDS in a multi-task setup. Then the detailed network generates the final domain-specific target by exploiting utterance and CDS simultaneously via attention mechanism. Our experiments demonstrate the effectiveness of CDS and multi-task learning. CDS is able to generalize over a wide range of tasks since it is an extraction to language expressions. Therefore, in the future, we would like to perfect the CDS definition and extend its' ontology to other domains and tasks. Besides, in this paper, we use attention mechanism to make use of CDS which is still a indirect way. We would like to explore more effective ways such as constraint decoding to further enhance the role of CDS.

General-to-detailed neural network(GDNN)  with Multi-Task Learning by incorporating cross-domain sketch(CDS) for semantic parsing

represent the tldr for finding the corresponding scientific article

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the article for finding a single-sentence TLDR

Question: The learnability of different neural architectures can be characterized directly by computable measures of data complexity. In this paper, we reframe the problem of architecture selection as understanding how data determines the most expressive and generalizable architectures suited to that data, beyond inductive bias. After suggesting algebraic topology as a measure for data complexity, we show that the power of a network to express the topological complexity of a dataset in its decision boundary is a strictly limiting factor in its ability to generalize. We then provide the first empirical characterization of the topological capacity of neural networks. Our empirical analysis shows that at every level of dataset complexity, neural networks exhibit topological phase transitions and stratification. This observation allowed us to connect existing theory to empirically driven conjectures on the choice of architectures for a single hidden layer neural networks. Deep learning has rapidly become one of the most pervasively applied techniques in machine learning. From computer vision BID15 ) and reinforcement learning BID18 ) to natural language processing BID27 ) and speech recognition ), the core principles of hierarchical representation and optimization central to deep learning have revolutionized the state of the art; see BID10. In each domain, a major difficulty lies in selecting the architectures of models that most optimally take advantage of structure in the data. In computer vision, for example, a large body of work BID24, BID25, BID12, etc.) focuses on improving the initial architectural choices of BID15 by developing novel network topologies and optimization schemes specific to vision tasks. Despite the success of this approach, there are still not general principles for choosing architectures in arbitrary settings, and in order for deep learning to scale efficiently to new problems and domains without expert architecture designers, the problem of architecture selection must be better understood. Theoretically, substantial analysis has explored how various properties of neural networks, (eg. the depth, width, and connectivity) relate to their expressivity and generalization capability, BID6, BID11 ). However, the foregoing theory can only be used to determine an architecture in practice if it is understood how expressive a model need be in order to solve a problem. On the other hand, neural architecture search (NAS) views architecture selection as a compositional hyperparameter search BID23, BID9, BID31 ). As a NAS ideally yields expressive and powerful architectures, but it is often difficult to interperate the ing architectures beyond justifying their use from their emperical optimality. We propose a third alternative to the foregoing: data-first architecture selection. In practice, experts design architectures with some inductive bias about the data, and more generally, like any hyperparameter selection problem, the most expressive neural architectures for learning on a particular dataset are solely determined by the nature of the true data distribution. Therefore, architecture selection can be rephrased as follows: given a learning problem (some dataset), which architectures are suitably regularized and expressive enough to learn and generalize on that problem?A natural approach to this question is to develop some objective measure of data complexity, and then characterize neural architectures by their ability to learn subject to that complexity. Then given some new dataset, the problem of architecture selection is distilled to computing the data complexity and chosing the appropriate architecture. For example, take the two datasets D 1 and D 2 given in FIG0 (ab) and FIG0 (cd) respectively. The first dataset, D 1, consists of positive examples sampled from two disks and negative examples from their compliment. On the right, dataset D 2 consists of positive points sampled from two disks and two rings with hollow centers. Under some geometric measure of complexity D 2 appears more'complicated' than D 1 because it contains more holes and clusters. As one trains single layer neural networks of increasing hidden dimension on both datasets, the minimum number of hidden units required to achieve zero testing error is ordered according to this geometric complexity. Visually in FIG0, regardless of initialization no single hidden layer neural network with ≤ 12 units, denoted h ≤12, can express the two holes and clusters in D 2. Whereas on the simpler D 1, both h 12 and h 26 can express the decision boundary perfectly. Returning to architecture selection, one wonders if this characterization can be extrapolated; that is, is it true that for datasets with'similar' geometric complexity to D 1, any architecture with ≥ 12 hidden learns perfectly, and likewise for those datasets similar in complexity to D 2, architectures with ≤ 12 hidden units can never learn to completion? In this paper, we formalize the above of geometric complexity in the language of algebraic topology. We show that questions of architecture selection can be answered by understanding the'topological capacity' of different neural networks. In particular, a geometric complexity measure, called persistent homology, characterizes the capacity of neural architectures in direct relation to their ability to generalize on data. Using persistent homology, we develop a method which gives the first empirical insight into the learnability of different architectures as data complexity increases. In addition, our method allows us to generate conjectures which tighten known theoretical bounds on the expressivity of neural networks. Finally, we show that topological characterizations of architectures are possible and useful for architecture selection in practice by computing the persistent homology of CIFAR-10 and several UCI datasets. In order to more formally describe notions of geometric complexity in datasets, we will turn to the language of topology. Broadly speaking, topology is a branch of mathematics that deals with characterizing shapes, spaces, and sets by their connectivity. In the context of characterizing neural networks, we will work towards defining the topological complexity of a dataset in terms of how that dataset is'connected', and then group neural networks by their capacity to produce decision regions of the same connectivity. BID14 ).In topology, one understands the relationships between two different spaces of points by the continuous maps between them. Informally, we say that two topological spaces A and B are equivalent (A ∼ = B) if there is a continuous function f: A → B that has an inverse f −1 that is also continuous. When f exists, we say that A and B are homeomorphic and f is their homeomorphism; for a more detailed treatment of general topology see BID2. Take for example, the classic example of the coffee cup and the donut in FIG1. They are homeomorphic because one can define a continuous deformation of one into the other which shrinks, twists, and morphs without tearing or gluing, as in FIG1. Note that if the donut had two holes, it would no longer be equivalent to the mug. Likewise, in an informal way, FIG0 since if there were a homeomorphism f: D 1 → D 2 at least one of the clusters in D 1 would need to be split in order to produce the four different regions in D 2. DISPLAYFORM0 The power of topology lies in its capacity to differentiate sets (topological spaces) in a meaningful geometric way that discards certain irrelevant properties such as rotation, translation, curvature, etc. For the purposes of defining geometric complexity, non-topological properties 1 like curvature would further fine-tune architecture selection-say if D 2 had the same regions but with squigly (differentially complex) boundaries, certain architectures might not converge-but as we will show, grouping neural networks by'topological capacity' provides a powerful minimality condition. That is, we will show that if a certain architecture is incapable of expressing a decision region that is equivalent in topology to training data, then there is no hope of it ever generalizing to the true data. Algebraic topology provides the tools necessary to not only build the foregoing notion of topological equivalence into a measure of geometric complexity, but also to compute that measure on real data (Betti, BID7 BID2 ). At its core, algebraic topology takes topological spaces (shapes and sets with certain properties) and assigns them algebraic objects such as groups, chains, and other more exotic constructs. In doing so, two spaces can be shown to be topologically equivalent (or distinct) if the algebraic objects to which they are assigned are isomorphic (or not). Thus algebraic topology will allow us to compare the complexity of decision boundaries and datasets by the objects to which they are assigned. Although there are many flavors of algebraic topology, a powerful and computationally realizable tool is homology. Definition 2.1 . If X is a topological space, then H n (X) = Z βn is called the nth homology group of X if the power β n is the number of'holes' of dimension n in X. Note that β 0 is the number of separate connected components. We call β n (X) the nth Betti number of X. Finally, the homology 2 of X is defined as DISPLAYFORM0 Immediately homology brings us closer to defining the complexity of D 1 and D 2. If we assume that D 1 is not actually a collection of N datapoints, but really the union of 2 solid balls, and likewise that D 2 is the union of 2 solid balls and 2 rings, then we can compute the homology directly. In this case H 0 (D 1) = Z 2 since there are two connected components 3; H 1 (D 1) = {0} since there are no circles (one-dimensional holes); and clearly, H n (D 1) = {0} for n ≥ 2. Performing the same computation in the second case, we get H 0 (D 2) = Z 4 and H 1 (D 2) = Z 2 as there are 4 seperate clusters and 2 rings/holes. With respect to any reasonable ordering on homology, D 2 is more complex than D 1. The measure yields non-trivial differentiation of spaces in higher dimension. For example, the homology of a hollow donut is DISPLAYFORM1 Surprisingly, the homology of a space contains a great deal of information about its topological complexity 1. The following theorem suggests the absolute power of homology to group topologically similar spaces, and therefore neural networks with topologically similar decision regions. DISPLAYFORM2 Intuitively, Theorem 2.2 states that number of'holes' (and in the case of H 0 (X), connected components) are topologically invariant, and can be used to show that two shapes (or decision regions) are different. In order to compute the homology of both D 1 and D 2 we needed to assume that they were actually the geometric shapes from which they were sampled. Without such assumptions, for any dataset DISPLAYFORM0 where N is the number of data points. This is because, at small enough scales each data point can be isolated as its own connected component; that is, as sets each pair of different positive points d 1, d 2 ∈ D are disjoint. To properly utilize homological complexity in better understanding architecture selection, we need to be able to compute the homology of the data directly and still capture meaningful topological information. Persistent homology, introduced in BID30, avoids the trivialization of computation of dataset homology by providing an algorithm to calculate the homology of a filtration of a space. Specifically, a filtration is a topological space X equipped with a sequence of subspaces X 0 ⊂ X 1 ⊂ · · · ⊂ X. In FIG2 one such particular filtration is given by growing balls of size centered at each point, and then letting X be the ing subspace in the filtration. Define β n (X) to be the nth Betti number of the homology H(X) of X. Then for example at = 1.5, β 0 (X) = 19 and β 1 (X) = 0 as every ball is disjoint. At = 5.0 some connected components merge and β 0 (X) = 12 and β 1 (X) = 0. Finally at = 7, the union of the balls forms a hole towards the center of the dataset and β 1 (X) > 0 with β 0 (X) = 4.All together the change in homology and therefore Betti numbers for X as changes can be summarized succinctly in the persistence barcode diagram given in FIG2. Each bar in the section β n (X) denotes a'hole' of dimension n. The left endpoint of the bar is the point at which homology detects that particular component, and the right endpoint is when that component becomes indistinguishable in the filtration. When calculating the persistent homology of datasets we will frequently use these diagrams. With the foregoing algorithms established, we are now equipped with the tools to study the capacity of neural networks in the language of algebraic topology. In the forthcoming section, we will apply persistent homology to emperically characterize the power of certain neural architectures. To understand why homological complexity is a powerful measure for differentiating architectures, we present the following principle. Suppose that D is some dataset drawn from a joint distribution F with continuous CDF on some topological space X × {0, 1}. Let X + denote the support of the distribution of points with positive labels, and X − denote that of the points with negative labels. Then let H S (f):= H[f −1 ((0, ∞))] denote the support homology of some function f: X → {0, 1}. Essentially H S (f) is homology of the set of x such that f (x) > 0. For a binary classifier, f, H S (f) is roughly a characterization of how many'holes' are in the positive decision region of f. We will sometimes use β n (f) to denote the nth Betti number of this support homology. Finally let F = {f : X → {0, 1}} be some family of binary classifiers on X. Theorem 3.1 (The Homological Principle of Generalization). If X = X − X + and for all f ∈ F with H S (f) = H(X +), then for all f ∈ F there exists A ⊂ X + so f misclassifies every x ∈ A.Essential Theorem 3.1 says that if an architecture (a family of models F) is incapable of producing a certain homological complexity, then for any model using that architecture there will always be a set A of true data points on which the model will fail. Note that the above principle holds regardless of how f ∈ F is attained, learned or otherwise. However, the principle does imply that no matter how well some F learns to correctly classify D there will always be a counter examples in the true data. In the context of architecture selection, the foregoing minimality condition significantly reduces the size of the search space by eliminating smaller architectures which cannot even express the'holes' (persistent homology) of the data H(D). This allows us to return to our original question of finding suitably expressive and generalizeable architectures but in the very computable language of homological complexity: Let F A the set of all neural networks with'architecture' A, then Given a dataset D, for which architectures A does there exist a neural network f ∈ F A such that H S (f) = H(D)? We will resurface a contemporary theoretical view on this question, and thereafter make the first steps towards an emperical characterization of the capacity of neural architectures in the view of topology. Theoretically, the homological complexity of neural network can be framed in terms of the sum of the number of holes expressible by certain architectures. In particular, BID1 gives an analysis of how the maximum sum of Betti numbers grows as F A changes. The , summarized in TAB0, show that the width, depth, and activation of a fully connected architecture effect its topological expressivity to varying polynomial and exponential degrees. What is unclear from this analysis is how these bounds describe expressivity in terms of individual Betti numbers. For example, with a tanh activation function, n inputs, layers, and h hidden units, there is no description of what the number of connected components max f ∈F A β 0 (f) or 1-dimensional holes max f ∈F A β 1 (f) actually is. With regards to tighter bounds BID1 stipulate that improvements to their are deeply tied to several unsolved problems in algebraic topology. BID1.) DISPLAYFORM0 Figure 4: Topological phase transitions in low dimensional neural networks as the homological complexity of the data increases. The upper right corner of each plot is a dataset on which the neural networks of increasing hidden dimension are trained. To understand how the homology of data determines expressive architectures we turn to an empirical characterization of neural networks. In this setting, we can tighten the bounds given in TAB0 by training different architectures on datasets with known homologies and then recording the decision regions observed over the course of training. In the most basic case, one is interested in studying how the number of hidden units in a single hidden layer neural network affect its homological capacity. The of BID1 say for certain activation functions we should expect a polynomial dependence on the sum of Betti numbers β n, but is this true of individual numbers? Having an individual characterization would allow for architecture selection by computing the homology of the dataset, and then finding which architectures meet the minimal criterion for each Betti number β n. Restricting 5 our analysis to the case of two inputs, n = 2, we characterize the capacities of architectures with an increasing number of hidden units to learn on datasets with homological complexities ranging from {Z 1, 0} to {Z 20, Z 20}. In our experiment, we generate datasets of each particular homological complexity by sampling different combinations of balls, rings, and scaling, twisting, and gluing them at random. After generating the foregoing datasets with N ≈ 90000 samples we train 100 randomly (truncated normal) initialized single hidden layer architectures with hidden units h ∈ {1, . . ., 255} and tanh activation functions for 10 6 minibatches of size 128. During training, every 2500 batches we sample the decision boundary of each neural network over a grid of 500 × 500 samples, producing 1.02 × 10 6 recorded decision boundaries. Using the ing data, we not only characterize different architectures but observed interesting topological phenomena during learning. First, neural networks exhibit a statistically significant topological phase transition in their convergence which depends directly on the homological complexity of the data. For any dataset in the experiment and any random homeomorphism applied thereto, the best test error of architectures with h hidden units is strictly ordered in magnitude and convergence time for h < h phase where h phase is a number of hidden units required to express the homology of the data. In Figure 4 we plot the best performing test error of architectures h ∈ {1, . . ., 15} on some example datasets DISPLAYFORM0 In this example h phase (D 0) = 4, h phase (D 1) = 6, and h phase (D 2) = 10. Surprisingly, leading up to the phase transition point, each different architecture falls into its own band of optimal convergence. This suggests that additional hidden units do in fact add to the topological capacity of an architecture in a consistent way. Using topological phase transitions we now return to the original question of existence of expressive architectures. In FIG3, we accumulate the probabilities that neural networks of varying hidden dimension train to zero-error on datasets of different homological complexities. The table gives different views into how expressive an architecture need be in order to converge, and therefore we are able to conjecture tighter bounds on the capacity of hidden units. Extrapolating from the first view, if H 0 (D) = Z m then there exists a single hidden layer neural network with h = m + 2 that converges to zero error on D. Likewise we claim that if H 0 (D) = Z m and H 1 (D) = 1 then the same holds with h ≥ 3m − 1. Further empirical analysis of convergence probabilities yields additional conjectures. However, claiming converse conjectures about a failure to generalize in the view of Theorem 3.1 requires exhaustive computation of decision boundary homologies. By applying persistent homology to the decision boundaries of certain networks during training, we observe that given sufficient data, neural networks exhibit topological stratification. For example, consider the homologies of different architecture decision regions as training progresses in FIG4. At the beginning of training every model captures the global topological information of the dataset and is homologically correlated with one another. However as training continues, the architectures stratify into two groups with homological complexities ordered by the capacities of the models. In this example, h 3, h 4, and h 5 are unable to express as many holes as the other architectures and so never specialize to more complex and local topological properties of the data. FIG4 (b) depicts topological stratification in terms of the correlation between Betti numbers. Topologically speaking, networks with less than 6 hidden units are distinct from those with more for most of training. Furthermore, this correlative view shows that stratification is consistent with topological phase transition; that is, across all decision boundary homologies recorded during the experiment stratification occurs just when the number of hidden units is slightly less than h phase. We have thus far demonstrated the discriminatory power of homological complexity in determining the expressivity of architectures. However, for homological complexity to have any practical use in architecture selection, it must be computable on real data, and more generally real data must have non-trivial homology; if all data were topologically simple our characterization would have no predictive power. In the following section we will compute the persistent homologies up to dimension 2 of different real world datasets. CIFAR-10. We compute the persistent homology of several classes of CIFAR-10 using the Python library Dionysus. Currently algorithms for persistent homology do not deal well with high dimensional data, so we embed the entire dataset in R 3 using local linear embedding (LLE;) with K = 120 neighbors. After embedding the dataset, we take a sample of 1000 points from example class'car' and build a persistent filtration by constructing a Vietoris-Rips complex on the data. The ing complex has 20833750 simplices and took 4.3 min. to generate. UCI Datasets. We further compute the homology of three low dimensional UCI datasets and attempt to assert the of non-trivial, h phase. Specifically, we compute the persistent homology of the majority classes in the Yeast Protein Localization Sites, UCI Ecoli Protein Localization Sites, and HTRU2 datasets. For these datasets no dimensionality reduction was used. In FIG5 (left), the persistence barcode exhibits two seperate significant loops (holes) at ∈ [0.19, 0.31] and ∈ [0.76, 0.85], as well as two major connected components in β 0 (D). The Other persistence diagrams are relegated to the appendix. Existing Data. Outside of the primary machine learning literature, topological data analysis yields non-trivial computations in wide variety of fields and datasets. Of particular interest is the work of BID3, which computes the homological complexity of collections of n × n patches of natural images. Even in these simple collections of images, the authors found topologies of Klein Bottles (H(·) = {Z, Z 2 /2Z, 0 . . .}) and other exotic topological objects. Other authors have calculated non-trivial dataset homologies in biological BID26 ), natural language BID17 ), and other domains BID28, BID29 ). We will place this work in the context of deep learning theory as it relates to expressivity. Since the seminal work of BID5 which established standard universal approximation for neural networks, many theorists have endeavored to understand the expressivity of certain neural architectures. BID19 BID16 provided the first analysis relating the depth and width of architectures to the complexitity of the sublevel sets they can express. Motivated therefrom, BID1 expressed this theme in the language of Pfefferian functions, thereby bounding the sum of Betti numbers expressed by sublevel sets. gave an account of how topological assumptions on the input data lead to optimally expressive architectures. In parallel, BID8 presented the first analytical minimality in expressivity theory; that is, the authors show that there are simple functions that cannot be expressed by two layer neural networks with out exponential dependence on input dimension. This work spurred the work ofPoole et al., which reframed expressivity in a differential geometric lense. Our work presents the first method to derive expressivity empirically. Our topological viewpoint sits dually with its differential geometric counterpart, and in conjunction with the work of and BID1, this duallity implies that when topological expression is not possible, exponential differential expressivity allows networks to bypass homological constaints at the cost of adversarial sets. Furthermore, our work opens a practical connectio nbetween the foregoing theory on neural expressivity and architecture selection, with the potential to drastically improve neural architecture search BID31 ) by directly computing the capacities of different architectures. Architectural power is deeply related to the algebraic topology of decision boundaries. In this work we distilled neural network expressivity into an empirical question of the generalization capabilities of architectures with respect to the homological complexity of learning problems. This view allowed us to provide an empirical method for developing tighter characterizations on the the capacity of different architectures in addition to a principled approach to guiding architecture selection by computation of persistent homology on real data. There are several potential avenues of future research in using homological complexity to better understand neural architectures. First, a full characterization of neural networks with many layers or convolutional linearities is a crucial next step. Our empirical suggest that the their are exact formulas describing the of power of neural networks to express decision boundaries with certain properties. Future theoretical work in determining these forms would significantly increase the efficiency and power of neural architecture search, constraining the search space by the persistent homology of the data. Additionally, we intend on studying how the topological complexity of data changes as it is propagated through deeper architectures. A.1 HOMOLOGY Homology is naturally described using the language of category theory. Let T op 2 denote the category of topological spaces and Ab the category of abelian groups. A → X and j: X → (X, A) the sequence sequence of inclusions and connecting homomorphisms are exact.3. Given the pair (X, A) and an open set U ⊂ X such that cl(U) ⊂ int(A) then the inclusion k: (X − U, A − U) → (X, A) induces an isomorphism k *: H * (X − U, A − U) → H * (X, A)4. For a one point space P, H i (P) = 0 for all i = 0.5. For a topological sum X = + α X α the homomorphism DISPLAYFORM0 is an isomorphism, where i α: X α → X is the inclusion. For related definitions and requisite notions we refer the reader to.A.2 PROOF OF THEOREM 3.1 Theorem A.2. Let X be a topological space and X + be some open subspace. If F ⊂ 2 X such that f ∈ F implies H S (f) = H(X +), then for all f ∈ F there exists A ⊂ X so that f (A ∩ X +) = {0} and f (A ∩ (X \ X +)) = {1}.Proof. Suppose the for the sake of contraiction that for all f ∈ F, H S (f) = H(X +) and yet there exists an f such that for all A ⊂ X, there exists an x ∈ A such that f (x) = 1. Then take A = {x} x∈X, and note that f maps each singleton into its proper partition on X. We have that for any open subset of V ⊂ X +, f (V) = {1}, and for any closed subset W ⊂ X \ X +, f (W) = {0}. Therefore X + = A∈τ X + ∩X A ⊂ supp(f) as the subspace topology τ X + ∩X = τ X + ∩ τ X where τ X + = {A ∈ τ X | A ⊂ X +} and τ X denotes the topology of X. Likewise, int(X −) ⊂ X \supp(F) under the same logic. Therefore supp(f) has the exact same topology as X + and so by Theorem 2.2 H(X +) = H(supp(f)) but this is a contradiction. This completes the proof.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show that the learnability of different neural architectures can be characterized directly by computable measures of data complexity.

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.


Question: represent the article for finding a single-sentence tldr

Question: Challenges in natural sciences can often be phrased as optimization problems. Machine learning techniques have recently been applied to solve such problems. One example in chemistry is the design of tailor-made organic materials and molecules, which requires efficient methods to explore the chemical space. We present a genetic algorithm (GA) that is enhanced with a neural network (DNN) based discriminator model to improve the diversity of generated molecules and at the same time steer the GA. We show that our algorithm outperforms other generative models in optimization tasks. We furthermore present a way to increase interpretability of genetic algorithms, which helped us to derive design principles The design of optimal structures under constraints is an important problem spanning multiple domains in the physical sciences. Specifically, in chemistry, the design of tailor-made organic materials and molecules requires efficient methods to explore the chemical space. Purely experimental approaches are often time consuming and expensive. Reliable computational tools can accelerate and guide experimental efforts to find new materials faster. We present a genetic algorithm (GA) (; ; ;) for molecular design that is enhanced with two features: Inverse design is the systematic development of structures with desired properties. In chemistry , the challenge of inverse design has been tackled as an optimization problem, among others in the form of variational autoencoders (VAEs), generative adversarial networks (GANs) and genetic algorithms. Variational autoencoders and generative adversarial networks VAEs are a widely used method for direct generation of molecular string or graph representations (Gómez-). They encode discrete representations into a continuous (latent) space. Molecules resembling a known structure can be found by searching around the region of the encoded point. Making using of the continuous latent representation, it is possible to search via gradients or Bayesian Optimization (BO). However, the generation of semantically and syntactically valid molecules is a challenging task. Thus, several follow up works to the VAE have been proposed for inverse design in chemistry. Among them, CVAE (Gómez-), GVAE and SD-VAE work directly on string molecular representations. Alternatively, JT-VAE (a) as well as work on molecular graphs. Unlike latent space property optimization, the policy network (PN) based GCPN model proposes a reinforcement learning (RL)-based method for direct optimization on molecular graphs. ORGAN demonstrate training string-based generative adversarial networks (GANs) via RL. Another method based on adversarial training is VJTNN (b). first introduced molecule generating models based on language models and reinforcement learning, where actions in an environment are taken to construct a molecule, receiving reward from an external scoring function. This model has also shown strong performance in the GuacaMol benchmark . In all the approaches mentioned above, generative models are trained to mimic the reference data set distributions, thus limiting the exploration ability of VAEs and GANs. There exist several examples of GA based molecule optimization algorithms in literature (; ; ; ; ;) ). While some of these examples pre-define mutations on a SMILES level to ensure validity of the molecules, other approaches use fragmentbased assembly of molecules. GAs are likely to get trapped in regions of local optima . Thus, for selection of the best molecules, report multiple restarts upon stagnation. We include the aforementioned models as baselines for numerical comparison. 3 GA-D ARCHITECTURE 3.1 OVERVIEW Our approach is illustrated in Figure 1. Our generator is a genetic algorithm with a population of molecules m. In each generation, the fitness of all molecules is evaluated as a linear combination of molecular properties J(m) and the discriminator score D(m): Random mutations of high fitness (best performing) molecules replace inferior members, while bestperforming molecules continue to a subsequent generation. The probability of replacing a molecule is evaluated using a smooth logistic function based on a ranking of fitness among the molecules of a generation. At the end of each generation, a neural network based discriminator is trained jointly on molecules generated by the GA and a reference data set. The fitness evaluation accounts for the discriminator predictions for each molecule. Therefore, the discriminator plays a role in the selection of the subsequent population. Mutation of molecules to populate subsequent generations is an important element of the GA. A low degree of mutation can lead to a slow exploration of the chemical space, causing stagnation Figure 1: Overview of our hybrid structure, which augments genetic algorithms with ML based neural networks. of the fitness function. Robustness of SELFIES allows us to do random mutations to molecular strings while preserving their validity. Thus, our mutations only include 50% insertions or 50% replacements of single SELFIES characters. To accelerate the exploration of the GA, we add one domain-specific mutation rule -the direct addition of phenyl groups in approximately 4% of cases. Character deletion is implicitly taken into account in SELFIES mutations, and we do not use crossover rules. The fundamental role of the discriminator is to increase molecular diversity by removing longsurviving molecules. Consider the realistic scenario in which a GA has found a molecule close to a local optimum, where all mutations are lowering the fitness. As a , this molecule survives for multiple generations while occupying a large fraction of the population. During such periods, the GA has limited view of the chemical space as it repeatedly explores mutations of the same high-fitness molecule. A straightforward solution would be the addition of a linear penalty in the fitness function that accounts for the number of successive generations a molecule survives. However, this method assigns independent scores to similar-looking molecules, which again in less variety. Our solution is the addition of an adaptive penalty (in our case a neural network based discriminator), thus resolving the problem of stagnation. Molecules with similar representation receive similar classification scores. Furthermore, long-surviving molecules are trained longer and receive weaker scores, ing in decreasing fitness -reducing the chance of long periods of stagnation (illustrated in Figure 2). The task of the discriminator thus is to memorize families of high performing molecules and penalize their fitness to force the GA to explore different regions in chemical space. Comparing to the literature standard, we aim at maximizing the penalized logP objective J(m) proposed by (Gómez-). For molecule m, the penalized logP function is defined as where logP indicates the water-octanol partition coefficient, SA represents the synthetic accessibility and prevents the formation of chemically unfeasible molecules, and RingPenalty linearly penalizes the presence of rings of size larger than 6. Our reference data set consists of 250,000 commercially available molecules extracted from the ZINC database . All three quantities in the above equation are normalized based on this data set. 5.6 ± 0.5 RNN GCPN 12.61 ± 0.81 GA GA + D (here) 13.31 ± 0.63 20.72 ± 3.14) GA + DNN 1 average of three best molecules after performing Bayesian optimization on the latent representation of a trained VAE 2 two best molecules of a single run 3 averaged over 10 runs with 20 molecules per generation with a molecular weight (excl. hydrogen) smaller than 39.15 ± 3.50 g/mol run for 50 generations 4 averaged over 10 runs with 500 molecules up to 81 characters per generation and 100 generations 5 averaged over 5 runs with 1000 generations each We define our fitness function according to Eq. 1 and 2 with β = 0 and 10. The algorithm is run for 100 generations with a population size of 500. All generated molecules are constrained to a canonical smile length of 81 characters (as in ). We train the discriminator (fully connected neural network with ReLU activation and sigmoid output layer, the input is a vector of chemical and geometrical properties characterizing the molecules) at the end of each generation for 10 epochs on 500 molecules proposed by the GA and 500 randomly drawn molecules from the ZINC data set. We report maximum J(m) scores, averaged over 10 independent runs. The highest J(m) achieved by our approach are 13.31 ± 0.63 (β = 10) and 12.61 ± 0.81 (β = 0), respectively, which is almost twice as high es the highest literature value of 7.87±0.07 (See Table 1). Furthermore, we compare to 50,000 random valid SELFIES strings, which surprisingly outperforms some existing generative models. The GA-D(t) are explained in the next section. In Figure 3, we show the of runs where we use a time-dependent adaptive penalty. During periods of saturation, the weight of the discriminator predictions is switched from 0 to 1000 until stagnation is overcome. The genetic algorithm is hence forced to propose new families of molecules to increase J(m). As it can be observed in Figure 3, even after steep decreases in max J(m), the scores recover and potentially reach values higher than in previous plateaus. We observe that this approach significantly outperforms all previous methods in maximizing the objective J(m). Visual inspection of the highest performing molecules shows that the GA is exploiting deficiencies of the (penalized) logP metric by generating chemically irrelevant motifs such as sulfur chains. While being of limited relevance for application, this nonetheless shows the that the GA is very efficient in exploring the chemical space and finding solutions for a given task. Analysis of these solutions will help us to better understand and eventually improve objective functions. At the same time, surprising solutions found by an unbiased algorithm can lead to unexpected discovery or boost human creativity . While exploitative tasks that follow some reference database are significant for questions involving drug design, more explorative behaviour could be beneficial in domains such as solar cell design , flow battery design , and in particular for questions where datas sets are not available, such as design of targets in molecular beam interferometry . Figure 4 shows classes of molecules explored by the GA in a trajectory with 1000 generations and a generation size of 500 (see trajectories in Figure 3). A K-means clustering analysis based in the RDKit fingerprint bit-vectors with 20 clusters was used to automatically generate labels for the 50 best performing molecules (in terms of their properties J(m)) in each generation. We find that the algorithm starts with a class of relatively small molecules (≈ 40 generations, see class 10), after which the algorithm explores several different classes of large molecules with e.g. aromatic rings and conjugates carbon chains (see class 15) and long linear sulfur chains (see class 1). Class 1 includes the molecules with the highest J(m) scores found in this work, exceeding values of 20. The clustering analysis furthermore helps to analyze the large number of generated molecules. This allows us to learn from the data and derive design rules to achieve high target properties. In case of the penalized logP score, we find that representative examples (short distance to the cluster center) of high performing classes contain aromatic rings, conjugated carbon chains and linear sulfur chains, which can act as design rules for molecules with high penalized logP scores. Using these design rules to construct molecules of the maximally allowed length consisting of only sulfur chains, conjugated carbon chains and chains of benzene rings with sulfur bridges yields J(m) scores of 31.79, 8.58 and 10.02, respectively. The J(m) score reached by the linear sulfur chain outperforms the best scores found by the GA and other generative models. To visualize the explorative behaviour of the GA, we did a two-dimensional principal component analysis (PCA) of the molecular fingerprint vectors of all molecules generated by the GA-D(t) in the trajectory shown in Figure 4. Five snapshots of the trajectory are shown in Figure 5, colored according to their chemical family. The property score J(m) of the trajectory is visualized in the images. We find that the GA sequentially explores different parts of the 2D projection of the chemical space, finding different families of high performing molecules, while the discriminator prevents the algorithm from searching in the same space repeatedly. In the previous sections, we aimed to maximize the penalized logP objective. Here, we consider two different tasks: firstly, generating molecules of specific chemical interest and secondly, modifying existing molecules to increase penalized logP scores. We used β = 0 throughout Section 4.4. We run 250 instances of randomly selected sets of target properties (logP in the interval from -5 to 10, SA scores in the range from 1 to 5 and ring penalty for 0 to 3). The fitness function is modified to minimize the summed squared difference between the actual and desired properties. We compute the number of experiments that successfully proposed molecules with a squared difference less than 1.0. Each run is constrained to run for 100 generations with a maximum canonical SMILES length of 81 characters. In 90.0% of the cases, our approach proposes the desired molecules. Improvement Success Improvement Success JT-VAE (a) 0.84 ± 1.45 83.6% 0.21 ± 0.71 46.4% GCPN 2.49 ± 1.30 100.0% 0.79 ± 0.63 100.0% MMPA (b) 3.29 ± 1.12 -1.65 ± 1.44 -DEFactor 3.41 ± 1.67 85.9% 1.55 ± 1.19 72.6% VJTNN (b) 3.55 ± 1.67 -2.33 ± 1.17 -GA (here) 5.93 ± 1.41 100.0% 3.44 ± 1.09 99.8% Figure 6: Distributions of logP and QED for a) ZINC and b) GuacaMol data set compared to molecules generated using the GA. c) Examples of molecules generated by the GA with high logP and QED scores. In this experiment, we follow the experimental setup proposed by. We optimize the penalized logP score of 800 low-scoring molecules from the ZINC data set. Our genetic algorithm is initiated with a molecule from the data set, and we run each experiment for 20 generations and a population size of 500 without the discriminator. For each run, we report the molecule m that increases the penalized logP the greatest, while possessing a similarity sim(m, m) > δ with the respective reference molecules m. We calculate molecular similarity based on Morgan Fingerprints of radius 2. To ensure generation of molecules possessing a certain similarity, for molecule m we modify the fitness to: Here, SimilarityPenalty(m) is 0 if sim(m, m) > δ and −10 6 otherwise. In Table 2, we report the average improvement for each molecule. Success is determined when the GA successfully improve upon the penalized logP score, while not violating the similarity constraint. Figure S1 shows examples of improved molecular structures. To show the performance of the GA on a drug-discovery task , we modified the objective function to include the drug-likeness score QED . Solubility metric logP and drug-likeness cannot be maximized at the same time, which is shown in Figure 6 at the example of the ZINC and the GuacaMol data set . The experiment shows that the GA is able to efficiently and densely sample the edge of the property distributions of both data sets. Example molecules that simultaneously maximize logP and QED are shown in Figure 6c. The definition of the fitness function used in this work (see Equation 1) has a free parameter (β), that balances the relative importance of molecular target properties J(m) and discriminator score Figure 7 illustrates the of this test, where each curve is obtained from averaging five independent runs with equal settings. In Figure 7a, we show the average property score J(m) for 11 different values β and observe, as expected, that low values of β on average lead to higher J(m) score while very high values of β limit the algorithm to stay inside the reference data set distribution, which, per definition, has a mean J(m) score of 0. Figure 7b illustrates the low average discriminator scores encountered after a few generations in case of low values of β. The curves shown in Figure 7a and b suggest that β allows us to interpolate between high and low J(m) and high and low resemblance with the reference data set. However, as shown in Figure 7c, this is not the case. At values of β ≈ 20, there is a rather fast transition from molecules with high values of J(m) of approximately 12-13 to a J(m) distribution centered around 0 which is similar to the J(m) distribution of the reference data set. The same abrupt change is visible in the distributions of the discriminator scores. Only at β ≈ 18, we find a slightly wider distribution of intermediate J(m) values, which we assume is partially related to the fact that the β = 18 curve shown in Figure 7a is not converged yet. Figure 7d shows examples of molecules generated with β = 0 (upper panel) and β = 50 (lower panel). While in the first case, the GA finds large molecules with many aromatic rings and long linear chains, that do not resemble the molecules in the reference data set, the latter case shows molecules with structures and properties resembling the reference data set. In this case, the distribution of J(m) is comparable to that of the data set (with 0 mean and unit standard deviation), while the discriminator encounters difficulty in correctly classification of the GA molecules, indicated by a mean discriminator score varying around 0.5 across many generations. We presented a hybrid GA and ML-based generative model and demonstrated its application in molecular design. The model outperforms literature approaches in generating molecules with desired properties. A detailed analysis of the data generated by the genetic algorithm allowed us to interpret the model and learn rules for the design of high performing molecules. This human expert design inspired from GA molecules outperformed all molecules created by generative models. For computationally more expensive property evaluations, we will extend our approach by the introduction of an on-the-fly trained ML property evaluation method, which will open new ways of solving the inverse design challenge in chemistry and materials sciences. Our approach is independent of domain knowledge, thus applicable to design questions in other scientific disciplines beyond chemistry. We therefore plan to generalize the GA-D approach to make it a more general concept of generative modelling. 6 SUPPLEMENTARY INFORMATION Figure S1 shows examples of the molecules optimized in Section 4.4. Figure S1: Molecular modifications ing in increased penalized logP scores under similarity constraint sim(m, m) > 0.4, 0.6. We show the molecules that ed in largest score improvement. Figures S2-S4 show comparisons between the property distributions observed in molecule data sets such as the ZINC and the GuacaMol data set and property distributions of molecules generated using random SELFIES (Figure S2), GA generated molecules with the penalized logP objective (Figure S3) and GA generated molecules with an objective function which includes logP and QED (Figure S4). While the average logP scores of average SELFIES are low, the tail of the distribution reaches to high values, explaining the surprisingly high penalized logP scores shown in Table 1. The QED and weight distributions of molecules optimized using the penalized logP objective significantly differ from the distributions of the ZINC and the GuacaMol data set (see Figure S3). As soon as the QED score is simultaneously optimized, the distributions of GA generated molecules and molecules from the reference data sets become more similar (see Figure S4). Figure 6 shows that the GA can simultaneously optimize logP and QED. Figure S2: Distributions of a) logP, b) SA, c) QED and d) molecular weight for randomly generated SELFIES, molecules from the ZINC data set and molecules from the GuacaMol data set. Figure S3: Distributions of a) logP, b) SA, c) QED and d) molecular weight for GA generated SELFIES (penalized logP objective function), molecules from the ZINC data set and molecules from the GuacaMol data set. Figure S4: Distributions of a) logP, b) SA, c) QED and d) molecular weight for GA generated SELFIES (logP and QED as objective function), molecules from the ZINC data set and molecules from the GuacaMol data set.

represent the tldr for finding the corresponding scientific article

Tackling inverse design via genetic algorithms augmented with deep neural networks.

We use a hypernetwork to predict optimal weights given hyperparameters, and jointly train everything together.