Question: Represent the article for finding a single-sentence TLDR

Question: Learning tasks on source code (i.e., formal languages) have been considered recently, but most work has tried to transfer natural language methods and does not capitalize on the unique opportunities offered by code's known syntax. For example, long-range dependencies induced by using the same variable or function in distant locations are often not considered. We propose to use graphs to represent both the syntactic and semantic structure of code and use graph-based deep learning methods to learn to reason over program structures. In this work, we present how to construct graphs from source code and how to scale Gated Graph Neural Networks training to such large graphs. We evaluate our method on two tasks: VarNaming, in which a network attempts to predict the name of a variable given its usage, and VarMisuse, in which the network learns to reason about selecting the correct variable that should be used at a given program location. Our comparison to methods that use less structured program representations shows the advantages of modeling known structure, and suggests that our models learn to infer meaningful names and to solve the VarMisuse task in many cases. Additionally, our testing showed that VarMisuse identifies a number of bugs in mature open-source projects. The advent of large repositories of source code as well as scalable machine learning methods naturally leads to the idea of "big code", i.e., largely unsupervised methods that support software engineers by generalizing from existing source code BID4. Currently, existing deep learning models of source code capture its shallow, textual structure, e.g. as a sequence of tokens BID15 BID22 BID3, as parse trees BID18, or as a flat dependency networks of variables BID23. Such models miss out on the opportunity to capitalize on the rich and well-defined semantics of source code. In this work, we take a step to alleviate this by including two additional signal sources in source code: data flow and type hierarchies. We do this by encoding programs as graphs, in which edges represent syntactic relationships (e.g. "token before/after") as well as semantic relationships ("variable last used/written here", "formal parameter for argument is called stream", etc.). Our key insight is that exposing these semantics explicitly as structured input to a machine learning model lessens the requirements on amounts of training data, model capacity and training regime and allows us to solve tasks that are beyond the current state of the art. We explore two tasks to illustrate the advantages of exposing more semantic structure of programs. First, we consider the VARNAMING task BID1 BID23, in which given some source code, the "correct" variable name is inferred as a sequence of subtokens. This requires some understanding of how a variable is used, i.e., requires reasoning about lines of code far var clazz=classTypes ["Root"].Single as JsonCodeGenerator. ClassType; Assert. NotNull(clazz); var first=classTypes ["RecClass"].Single as JsonCodeGenerator. ClassType; Assert. NotNull(clazz); Assert. Equal("string", first. Properties ["Name"].Name); Assert. False(clazz. Properties ["Name"].IsArray); Figure 1: A snippet of a detected bug in RavenDB an open-source C# project. The code has been slightly simplified. Our model detects correctly that the variable used in the highlighted (yellow) slot is incorrect. Instead, first should have been placed at the slot. We reported this problem which was fixed in PR 4138. apart in the source file. Secondly, we introduce the variable misuse prediction task (VARMISUSE), in which the network aims to infer which variable should be used in a program location. To illustrate the task, Figure 1 shows a slightly simplified snippet of a bug our model detected in a popular open-source project. Specifically, instead of the variable clazz, variable first should have been used in the yellow highlighted slot. Existing static analysis methods cannot detect such issues, even though a software engineer would easily identify this as an error from experience. To achieve high accuracy on these tasks, we need to learn representations of program semantics. For both tasks, we need to learn the semantic role of a variable (e.g., "is it a counter? ", "is it a filename? "). Additionally, for VARMISUSE, learning variable usage semantics (e.g., "a filename is needed here") is required. This "fill the blank element" task is related to methods for learning distributed representations of natural language words, such as Word2Vec BID20 and GLoVe BID21. However, we can learn from a much richer structure such as data flow information. This work is a step towards learning program representations, and we expect them to be valuable in a wide range of other tasks, such as code completion ("this is the variable you are looking for") and more advanced bug finding ("you should lock before using this object").To summarize, our contributions are: (i) We define the VARMISUSE task as a challenge for machine learning modeling of source code, that requires to learn (some) semantics of programs (cf. section 3).(ii) We present deep learning models for solving the VARNAMING and VARMISUSE tasks by modeling the code's graph structure and learning program representations over those graphs (cf. section 4). (iii) We evaluate our models on a large dataset of 2.9 million lines of real-world source code, showing that our best model achieves 32.9% accuracy on the VARNAMING task and 85.5% accuracy on the VARMISUSE task, beating simpler baselines (cf. section 5). (iv) We document practical relevance of VARMISUSE by summarizing some bugs that we found in mature open-source software projects (cf. subsection 5.3). Our implementation of graph neural networks (on a simpler task) can be found at https://github.com/Microsoft/gated-graph-neural-network-samples and the dataset can be found at https://aka.ms/iclr18-prog-graphs-dataset. Our work builds upon the recent field of using machine learning for source code artifacts BID4. For example, BID15 BID7 model the code as a sequence of tokens, while BID18; model the syntax tree structure of code. All works on language models of code find that predicting variable and method identifiers is one of biggest challenges in the task. Closest to our work is the work of BID2 who learn distributed representations of variables using all their usages to predict their names. However, they do not use data flow information and we are not aware of any model that does so. BID23 and BID8 use conditional random fields to model a variety of relationships between variables, AST elements and types to predict variable names and types (resp. to deobfuscate Android apps), but without considering the flow of data explicitly. In these works, all variable usages are deterministically known beforehand (as the code is complete and remains unmodified), as in BID1.Our work is remotely related to work on program synthesis using sketches BID27 and automated code transplantation. However, these approaches require a set of specifications (e.g. input-output examples, test suites) to complete the gaps, rather than statistics learned from big code. These approaches can be thought as complementary to ours, since we learn to statistically complete the gaps without any need for specifications, by learning common variable usage patterns from code. Neural networks on graphs BID13 BID17 BID11 BID16 BID12 ) adapt a variety of deep learning methods to graph-structured input. They have been used in a series of applications, such as link prediction and classification BID14 and semantic role labeling in NLP BID19. Somewhat related to source code is the work of BID28 who learn graph-based representations of mathematical formulas for premise selection in theorem proving. Detecting variable misuses in code is a task that requires understanding and reasoning about program semantics. To successfully tackle the task one needs to infer the role and function of the program elements and understand how they relate. For example, given a program such as Fig. 1, the task is to automatically detect that the marked use of clazz is a mistake and that first should be used instead. While this task resembles standard code completion, it differs significantly in its scope and purpose, by considering only variable identifiers and a mostly complete program. Task Description We view a source code file as a sequence of tokens t 0... t N = T, in which some tokens t λ0, t λ1... are variables. Furthermore, let V t ⊂ V refer to the set of all type-correct variables in scope at the location of t, i.e., those variables that can be used at t without raising a compiler error. We call a token tok λ where we want to predict the correct variable usage a slot. We define a separate task for each slot t λ: Given t 0... t λ−1 and t λ+1,..., t N, correctly select t λ from V t λ. For training and evaluation purposes, a correct solution is one that simply matches the ground truth, but note that in practice, several possible assignments could be considered correct (i.e., when several variables refer to the same value in memory). In this section, we discuss how to transform program source code into program graphs and learn representations over them. These program graphs not only encode the program text but also the semantic information that can be obtained using standard compiler tools. Gated Graph Neural Networks Our work builds on Gated Graph Neural Networks BID17 (GGNN) and we summarize them here. A graph G = (V, E, X) is composed of a set of nodes V, node features X, and a list of directed edge sets E = (E 1, . . ., E K) where K is the number of edge types. We annotate each v ∈ V with a real-valued vector x (v) ∈ R D representing the features of the node (e.g., the embedding of a string label of that node).We associate every node v with a state vector h (v), initialized from the node label x (v). The sizes of the state vector and feature vector are typically the same, but we can use larger state vectors through padding of node features. To propagate information throughout the graph, "messages" of type k are sent from each v to its neighbors, where each message is computed from its current state vector as m DISPLAYFORM0 Here, f k can be an arbitrary function; we choose a linear layer in our case. By computing messages for all graph edges at the same time, all states can be updated at the same time. In particular, a new state for a node v is computed by aggregating all incoming messages as DISPLAYFORM1 k | there is an edge of type k from u to v}). g is an aggregation function, which we implement as elementwise summation. Given the aggregated messagem (v) and the current state vector h (v) of node v, the state of the next time step DISPLAYFORM2, where GRU is the recurrent cell function of gated recurrent unit (GRU) BID10 (a) Simplified syntax graph for line 2 of Fig. 1, where blue rounded boxes are syntax nodes, black rectangular boxes syntax tokens, blue edges Child edges and double black edges NextToken edges. Program Graphs We represent program source code as graphs and use different edge types to model syntactic and semantic relationships between different tokens. The backbone of a program graph is the program's abstract syntax tree (AST), consisting of syntax nodes (corresponding to nonterminals in the programming language's grammar) and syntax tokens (corresponding to terminals). We label syntax nodes with the name of the nonterminal from the program's grammar, whereas syntax tokens are labeled with the string that they represent. We use Child edges to connect nodes according to the AST. As this does not induce an order on children of a syntax node, we additionally add NextToken edges connecting each syntax token to its successor. An example of this is shown in FIG1.To capture the flow of control and data through a program, we add additional edges connecting different uses and updates of syntax tokens corresponding to variables. For such a token v, let D R (v) be the set of syntax tokens at which the variable could have been used last. This set may contain several nodes (for example, when using a variable after a conditional in which it was used in both branches), and even syntax tokens that follow in the program code (in the case of loops). Similarly, let D W (v) be the set of syntax tokens at which the variable was last written to. Using these, we add LastRead (resp. LastWrite) edges connecting v to all elements of D R (v) (resp. D W (v)). Additionally, whenever we observe an assignment v = expr, we connect v to all variable tokens occurring in expr using ComputedFrom edges. An example of such semantic edges is shown in FIG1.We extend the graph to chain all uses of the same variable using LastLexicalUse edges (independent of data flow, i.e., in if (...) {... v ...} else {... v ...}, we link the two occurrences of v). We also connect return tokens to the method declaration using ReturnsTo edges (this creates a "shortcut" to its name and type). Inspired by BID25, we connect arguments in method calls to the formal parameters that they are matched to with FormalArgName edges, i.e., if we observe a call Foo(bar) and a method declaration Foo(InputStream stream), we connect the bar token to the stream token. Finally, we connect every token corresponding to a variable to enclosing guard expressions that use the variable with GuardedBy and GuardedByNegation edges. For example, in if (x > y) {... x ...} else {... y ...}, we add a GuardedBy edge from x (resp. a GuardedByNegation edge from y) to the AST node corresponding to x > y. Finally, for all types of edges we introduce their respective backwards edges (transposing the adjacency matrix), doubling the number of edges and edge types. Backwards edges help with propagating information faster across the GGNN and make the model more expressive. We assume a statically typed language and that the source code can be compiled, and thus each variable has a (known) type τ (v). To use it, we define a learnable embedding function r(τ) for known types and additionally define an "UNKTYPE" for all unknown/unrepresented types. We also leverage the rich type hierarchy that is available in many object-oriented languages. For this, we map a variable's type τ (v) to the set of its supertypes, i.e. τ * (v) = {τ : τ (v) implements type τ } ∪ {τ (v)}. We then compute the type representation r * (v) of a variable v as the element-wise maximum of {r(τ): τ ∈ τ * (v)}. We chose the maximum here, as it is a natural pooling operation for representing partial ordering relations (such as type lattices). Using all types in τ * (v) allows us to generalize to unseen types that implement common supertypes or interfaces. For example, List<K> has multiple concrete types (e.g. List<int>, List<string>). Nevertheless, these types implement a common interface (IList) and share common characteristics. During training, we randomly select a non-empty subset of τ * (v) which ensures training of all known types in the lattice. This acts both like a dropout mechanism and allows us to learn a good representation for all types in the type lattice. Initial Node Representation To compute the initial node state, we combine information from the textual representation of the token and its type. Concretely, we split the name of a node representing a token into subtokens (e.g. classTypes will be split into two subtokens class and types) on camelCase and pascal_case. We then average the embeddings of all subtokens to retrieve an embedding for the node name. Finally, we concatenate the learned type representation r * (v), computed as discussed earlier, with the node name representation, and pass it through a linear layer to obtain the initial representations for each node in the graph. Programs Graphs for VARNAMING Given a program and an existing variable v, we build a program graph as discussed above and then replace the variable name in all corresponding variable tokens by a special <SLOT> token. To predict a name, we use the initial node labels computed as the concatenation of learnable token embeddings and type embeddings as discussed above, run GGNN propagation for 8 time steps 2 and then compute a variable usage representation by averaging the representations for all <SLOT> tokens. This representation is then used as the initial state of a one-layer GRU, which predicts the target name as a sequence of subtokens (e.g., the name inputStreamBuffer is treated as the sequence [input, stream, buffer]). We train this graph2seq architecture using a maximum likelihood objective. In section 5, we report the accuracy for predicting the exact name and the F1 score for predicting its subtokens. Program Graphs for VARMISUSE To model VARMISUSE with program graphs we need to modify the graph. First, to compute a context representation c(t) for a slot t where we want to predict the used variable, we insert a new node v <SLOT> at the position of t, corresponding to a "hole" at this point, and connect it to the remaining graph using all applicable edges that do not depend on the chosen variable at the slot (i.e., everything but LastUse, LastWrite, LastLexicalUse, and GuardedBy edges). Then, to compute the usage representation u(t, v) of each candidate variable v at the target slot, we insert a "candidate" node v t,v for all v in V t, and connect it to the graph by inserting the LastUse, LastWrite and LastLexicalUse edges that would be used if the variable were to be used at this slot. Each of these candidate nodes represents the speculative placement of the variable within the scope. Using the initial node representations, concatenated with an extra bit that is set to one for the candidate nodes v t,v, we run GGNN propagation for 8 time steps.2 The context and usage representation are then the final node states of the nodes, i.e., c(t) = h (v<SLOT>) and u(t, v) = h (vt,v). Finally, the correct variable usage at the location is computed as arg max v W [c(t), u(t, v)] where W is a linear layer that uses the concatenation of c(t) and u(t, v). We train using a max-margin objective. Using GGNNs for sets of large, diverse graphs requires some engineering effort, as efficient batching is hard in the presence of diverse shapes. An important observation is that large graphs are normally very sparse, and thus a representation of edges as an adjacency list would usually be advantageous to reduce memory consumption. In our case, this can be easily implemented using a sparse tensor representation, allowing large batch sizes that exploit the parallelism of modern GPUs efficiently. A second key insight is to represent a batch of graphs as one large graph with many disconnected components. This just requires appropriate pre-processing to make node identities unique. As this makes batch construction somewhat CPU-intensive, we found it useful to prepare minibatches on a separate thread. Our TensorFlow BID0 implementation scales to 55 graphs per second during training and 219 graphs per second during test-time using a single NVidia GeForce GTX Titan X with graphs having on average 2,228 (median 936) nodes and 8,350 (median 3,274) edges and 8 GGNN unrolling iterations, all 20 edge types (forward and backward edges for 10 original edge types) and the size of the hidden layer set to 64. The number of types of edges in the GGNN contributes proportionally to the running time. For example, a GGNN run for our ablation study using only the two most common edge types (NextToken, Child) achieves 105 graphs/second during training and 419 graphs/second at test time with the same hyperparameters. Our (generic) implementation of GGNNs is available at https://github.com/Microsoft/ gated-graph-neural-network-samples, using a simpler demonstration task. Dataset We collected a dataset for the VARMISUSE task from open source C # projects on GitHub. To select projects, we picked the top-starred (non-fork) projects in GitHub. We then filtered out projects that we could not (easily) compile in full using Roslyn 3, as we require a compilation to extract precise type information for the code (including those types present in external libraries). Our final dataset contains 29 projects from a diverse set of domains (compilers, databases, . . .) with about 2.9 million non-empty lines of code. A full table is shown in Appendix D.For the task of detecting variable misuses, we collect data from all projects by selecting all variable usage locations, filtering out variable declarations, where at least one other type-compatible replacement variable is in scope. The task is then to infer the correct variable that originally existed in that location. Thus, by construction there is at least one type-correct replacement variable, i.e. picking it would not raise an error during type checking. In our test datasets, at each slot there are on average 3.8 type-correct alternative variables (median 3, σ = 2.6).From our dataset, we selected two projects as our development set. From the rest of the projects, we selected three projects for UNSEENPROJTEST to allow testing on projects with completely unknown structure and types. We split the remaining 23 projects into train/validation/test sets in the proportion 60-10-30, splitting along files (i.e., all examples from one source file are in the same set). We call the test set obtained like this SEENPROJTEST.Baselines For VARMISUSE, we consider two bidirectional RNN-based baselines. The local model (LOC) is a simple two-layer bidirectional GRU run over the tokens before and after the target location. For this baseline, c(t) is set to the slot representation computed by the RNN, and the usage context of each variable u(t, v) is the embedding of the name and type of the variable, computed in the same way as the initial node labels in the GGNN. This baseline allows us to evaluate how important the usage context information is for this task. The flat dataflow model (AVGBIRNN) is an extension to LOC, where the usage representation u(t, v) is computed using another two-layer bidirectional RNN run over the tokens before/after each usage, and then averaging over the computed representations at the variable token v. The local context, c(t), is identical to LOC. AVGBIRNN is a significantly stronger baseline that already takes some structural information into account, as the averaging over all variables usages helps with long-range dependencies. Both models pick the variable that maximizes c(t)T u(t, v).For VARNAMING, we replace LOC by AVGLBL, which uses a log-bilinear model for 4 left and 4 right context tokens of each variable usage, and then averages over these context representations (this corresponds to the model in BID2). We also test AVGBIRNN on VARNAMING, which essentially replaces the log-bilinear context model by a bidirectional RNN. TAB1 shows the evaluation of the models for both tasks. 4 As LOC captures very little information, it performs relatively badly. AVGLBL and AVGBIRNN, which capture information from many variable usage sites, but do not explicitly encode the rich structure of the problem, still lag behind the GGNN by a wide margin. The performance difference is larger for VARMISUSE, since the structure and the semantics of code are far more important within this setting. Generalization to new projects Generalizing across a diverse set of source code projects with different domains is an important challenge in machine learning. We repeat the evaluation using the UNSEENPROJTEST set stemming from projects that have no files in the training set. The right side of TAB1 shows that our models still achieve good performance, although it is slightly lower compared to SEENPROJTEST. This is expected since the type lattice is mostly unknown in UNSEENPROJTEST.We believe that the dominant problem in applying a trained model to an unknown project (i.e., domain) is the fact that its type hierarchy is unknown and the used vocabulary (e.g. in variables, method and class names, etc.) can differ substantially. Ablation Study To study the effect of some of the design choices for our models, we have run some additional experiments and show their in TAB2. First, we varied the edges used in the program graph. We find that restricting the model to syntactic information has a large impact on performance on both tasks, whereas restricting it to semantic edges seems to mostly impact performance on VARMISUSE. Similarly, the ComputedFrom, FormalArgName and ReturnsTo edges give a small boost on VARMISUSE, but greatly improve performance on VARNAMING. As evidenced by the experiments with the node label representation, syntax node and token names seem to matter little for VARMISUSE, but naturally have a great impact on VARNAMING. Figure 3 illustrates the predictions that GGNN makes on a sample test snippet. The snippet recursively searches for the global directives file by gradually descending into the root folder. Reasoning about the correct variable usages is hard, even for humans, but the GGNN correctly predicts the variable 3 http://roslyn.io 4 Sect. A additionally shows ROC and precision-recall curves for the GGNN model on the VARMISUSE task..TrimEnd(Path. DirectorySeparatorChar); } path 13 = null; return false; } 1: path:59%, baseDirectory:35%, fullPath:6%, GlobalDirectivesFileName:1% 2: baseDirectory:92%, fullPath:5%, GlobalDirectivesFileName:2%, path:0.4% 3: fullPath:88%, baseDirectory:9%, GlobalDirectivesFileName:2%, path:1% 4: directivesDirectory:86%, path:8%, baseDirectory:2%, GlobalDirectivesFileName:1%, fullPath:0.1% 5: directivesDirectory:46%, path:24%, baseDirectory:16%, GlobalDirectivesFileName:10%, fullPath:3% 6: baseDirectory:64%, path:26%, directivesDirectory:5%, fullPath:2%, GlobalDirectivesFileName:2% 7: path:99%, directivesDirectory:1%, GlobalDirectivesFileName:0.5%, baseDirectory:7e-5, fullPath:4e-7 8: fullPath:60%, directivesDirectory:21%, baseDirectory:18%, path:1%, GlobalDirectivesFileName:4e-4 9: GlobalDirectivesFileName:61%, baseDirectory:26%, fullPath:8%, path:4%, directivesDirectory:0.5% 10: path:70%, directivesDirectory:17%, baseDirectory:10%, GlobalDirectivesFileName:1%, fullPath:0.6% 11: directivesDirectory:93%, path:5%, GlobalDirectivesFileName:1%, baseDirectory:0.1%, fullPath:4e-5% 12: directivesDirectory:65%, path:16%, baseDirectory:12%, fullPath:5%, GlobalDirectivesFileName:3% 13: path:97%, baseDirectory:2%, directivesDirectory:0.4%, fullPath:0.3%, GlobalDirectivesFileName:4e-4 Figure 3: VARMISUSE predictions on slots within a snippet of the SEENPROJTEST set for the ServiceStack project. Additional visualizations are available in Appendix B. The underlined tokens are the correct tokens. The model has to select among a number of string variables at each slot, where all of them represent some kind of path. The GGNN accurately predicts the correct variable usage in 11 out of the 13 slots reasoning about the complex ways the variables interact among them. usages at all locations except two (slot 1 and 8). As a software engineer is writing the code, it is imaginable that she may make a mistake misusing one variable in the place of another. Since all variables are string variables, no type errors will be raised. As the probabilities in Fig. 3 suggest most potential variable misuses can be flagged by the model yielding valuable warnings to software engineers. Additional samples with comments can be found in Appendix B.Furthermore, Appendix C shows samples of pairs of code snippets that share similar representations as computed by the cosine similarity of the usage representation u(t, v) of GGNN. The reader can notice that the network learns to group variable usages that share semantic similarities together. For example, checking for null before the use of a variable yields similar distributed representations across code segments (Sample 1 in Appendix C). We have used our VARMISUSE model to identify likely locations of bugs in RavenDB (a document database) and Roslyn (Microsoft's C # compiler framework). For this, we manually reviewed a sample of the top 500 locations in both projects where our model was most confident about a choosing a variable differing from the ground truth, and found three bugs in each of the projects. Figs. 1,4,5 show the issues discovered in RavenDB. The bug in Fig. 1 was possibly caused by copy-pasting, and cannot be easily caught by traditional methods. A compiler will not warn about if (IsValidBackup(backupFilename) == false) {output("Error:"+ backupLocation +" doesn't look like a valid backup"); throw new InvalidOperationException(backupLocation + " doesn't look like a valid backup");Figure 5: A bug found (yellow) in the RavenDB open-source project. Although backupFilename is found to be invalid by IsValidBackup, the user is notified that backupLocation is invalid instead.unused variables (since first is used) and virtually nobody would write a test testing another test. FIG3 shows an issue that, although not critical, can lead to increased memory consumption. Fig. 5 shows another issue arising from a non-informative error message. We privately reported three additional bugs to the Roslyn developers, who have fixed the issues in the meantime (cf. https://github.com/dotnet/roslyn/pull/23437). One of the reported bugs could cause a crash in Visual Studio when using certain Roslyn features. Finding these issues in widely released and tested code suggests that our model can be useful during the software development process, complementing classic program analysis tools. For example, one usage scenario would be to guide the code reviewing process to locations a VARMISUSE model has identified as unusual, or use it as a prior to focus testing or expensive code analysis efforts. Although source code is well understood and studied within other disciplines such as programming language research, it is a relatively new domain for deep learning. It presents novel opportunities compared to textual or perceptual data, as its (local) semantics are well-defined and rich additional information can be extracted using well-known, efficient program analyses. On the other hand, integrating this wealth of structured information poses an interesting challenge. Our VARMISUSE task exposes these opportunities, going beyond simpler tasks such as code completion. We consider it as a first proxy for the core challenge of learning the meaning of source code, as it requires to probabilistically refine standard information included in type systems. A PERFORMANCE CURVES FIG4 shows the ROC and precision-recall curves for the GGNN model. As the reader may observe, setting a false positive rate to 10% we get a true positive rate 5 of 73% for the SEENPROJTEST and 69% for the unseen test. This suggests that this model can be practically used at a high precision setting with acceptable performance. Below we list a set of samples from our SEENPROJTEST projects with comments about the model performance. Code comments and formatting may have been altered for typesetting reasons. The ground truth choice is underlined. The model predicts correctly all usages except from the one in slot #3. Reasoning about this snippet requires additional semantic information about the intent of the code. var response = ResultsFilter(typeof(TResponse), #1, #2, request);#1 httpMethod: 99%, absoluteUrl: 1%, UserName: 0%, UserAgent: 0% #2 absoluteUrl: 99%, httpMethod: 1%, UserName: 0%, UserAgent: 0%The model knows about selecting the correct string parameters because it matches them to the formal parameter names. #1 n: 100%, MAXERROR: 0%, SYNC_MAXRETRIES: 0% #2 MAXERROR: 62%, SYNC_MAXRETRIES: 22%, n: 16% It is hard for the model to reason about conditionals, especially with rare constants as in slot #2. Here we show pairs of nearest neighbors based on the cosine similarity of the learned representations u(t, v). Each slot t is marked in dark blue and all usages of v are marked in yellow (i.e. variableName). This is a set of hand-picked examples showing good and bad examples. A brief description follows after each pair. HasAddress is a local function, seen only in the testset. For this work, we released a large portion of the data, with the exception of projects with a GPL license. The data can be found at https://aka.ms/iclr18-prog-graphs-dataset. Since we are excluding some projects from the data, below we report the , averaged over three runs, on the published dataset: Accuracy (%) PR AUC SEENPROJTEST 84.0 0.976 UNSEENPROJTEST 74.1 0.934

Represent the short summary for finding the corresponding scientific article (long).

Programs have structure that can be represented as graphs, and graph neural networks can learn to find bugs on such graphs

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Overfitting is an ubiquitous problem in neural network training and usually mitigated using a holdout data set. Here we challenge this rationale and investigate criteria for overfitting without using a holdout data set. Specifically, we train a model for a fixed number of epochs multiple times with varying fractions of randomized labels and for a range of regularization strengths. A properly trained model should not be able to attain an accuracy greater than the fraction of properly labeled data points. Otherwise the model overfits. We introduce two criteria for detecting overfitting and one to detect underfitting. We analyze early stopping, the regularization factor, and network depth. In safety critical applications we are interested in models and parameter settings which perform well and are not likely to overfit. The methods of this paper allow characterizing and identifying such models. Deep neural networks have shown superior performance for a wide range of machine learning task such as speech recognition BID4 ), image classification BID8 ), playing board games BID13 ); machine translation ); beating previous methods by orders of magnitudes. To apply neural networks to safety critical problems such as autonomous driving it is necessary to evaluate their performance on new previously unseen data. One of the major problems of neural networks is their vulnerability to adversarial attacks. It has been shown that tiny unrecognizable changes of the input can fool the network to predict any class the attacker has chosen. One way to interpret this vulnerability is that the neural network overfits to the training data, with the output varying rapidly around each training point and thus slight changes of the input can lead to big changes in the output. It is thus highly desirable to prevent the network from overfitting during training. Previously reported methods reduce the chance of overfitting by evaluating the neural network on some holdout set, or by penalizing the complexity of the model class. This has the disadvantage that a holdout set can only be used once. By using design choices proven to be successful in the past the model becomes dependent on the holdout set. Penalizing the model class is only a heuristic remedy to overfitting. In the present paper we devise a method which prevents overfitting by relying on the training data only. We motivate l 1 -regularization of the kernel weights as a preferable choice to control the network complexity. Using no holdout set requires an alternative notion of overfitting. In the paper, we say that a model overfits if it is able to learn noise. Heuristics. There exists several well known heuristics which reduce the chance of overfitting. Typically one reduces the hypothesis space, or one increases the data set. We can directly control the size of the hypothesis space by the number of parameters of the model. Typical choices are the width, the depth and the filter size of the network. Dropout introduced by BID14 at each training step, individual nodes and their incoming and outgoing edges are ignored (dropped out) with probability p. This reduces the dependency of the model on individual nodes. In early The complexity of a neural network is controlled by its hyper parameter and the hyper parameter of the training algorithm. We propose to control the model class by adding the l 1 norm of the kernel weights ||W j || 1 multiplied with a regularization factor λ to the loss function. Some additional for this section is provided in the appendix. Notation. This paper considers feed forward networks f: R d0 → R k which maps the input space V 0 = R d0 to some target space V L = R k. This is followed by an arg max function which picks as output the coordinate of the largest value. The margin measures the gap between the output for the correct label and the other labels, γ = f (x) y − max j =y f (x) j. A positive margin means a correct classification, whereas a negative margin means an incorrect classification. Each layer of the network consists of a linear block A k: V k−1 → W k and a non linearity φ k: W k → V k. The networks are thus written as: DISPLAYFORM0 Here of course φ • A(x) is just another way of writing φ(A(x)). The concatenation η k = φ k • A k will be called a block of the network. In this paper only the standard relu nonlinearity φ: R → R +, which is defined by φ(x) = x ∨ 0 = max{x, 0} = x +, is considered. The input data is denoted by x 0 ∈ R d0. Further, the output of layer k of the network is denoted by y k = f k (x):= φ k • A k (· · · (φ 1 • A 1 (x))) ∈ R d k and the network which outputs the k-th layer by f k: DISPLAYFORM1 Finally, the width of the network is defined by d = max{d 0, . . ., d L}. In the paper we will call the data y k, passing through the layers, signal. Finally, we arrange all data points and signals in matrices, denoted by X ∈ R d0×n and Y k ∈ R d k ×n. So we write by slightly abusing notation, DISPLAYFORM2 l 1 -regularization A typical convolutional kernel W is determined by the filter size, and the number of incoming and outgoing features. If the convolution is written as matrix operation y = Ax and zero padding is being assumed, then the matrix A can be arranged as a vertically stacked block matrix each subblock A i representing one outgoing feature. The entries of each these blocks are determined by the weights of the i-th filter. Due to zero padding and weight sharing, each weight occurs precisely once in each row and each column of A i. It follows that the filter matrix A contains in each column each weight precisely once. Lemma 3.1.1. The spectral norm of a convolution matrix A is bounded by the l 1 -norm of its kernel weights. DISPLAYFORM3 Proof. The inequality follows as the spectral norm can be bounded by the row and columns norms of A, which can be estimated by the weight matrix W. DISPLAYFORM4 Recently shown generalization bounds are dominated by BID2 ) see also Theorem A.1.1. Here γ > 0 is the margin, X denotes the training data arranged in a matrix, and R A is the spectral complexity. We will use a simplified version of R A defined by DISPLAYFORM5 DISPLAYFORM6 Lemma 3.1.2. The spectral complexity can be bounded by the l 1 -norm of the kernel weights W j. DISPLAYFORM7 Here the first inequality holds because of inequalities between the generalized 2/3-mean and 1-mean. The second inequality follows as the spectral norm can be bounded by ||W || 1 by Lemma 3.1.1.Both lemmas show that it is beneficial to l 1 -norm of the kernel weights. In fact Lemma 3.1 shows by looking at DISPLAYFORM8 that the margin scales as ||W || 1. Since we do not use a bias in our model, we may rescale each kernel matrix by ||W || −11. This is compensated by decreasing the margin accordingly. So in order to achieve better generalization bounds we can penalize the kernel weights by λ||W || 1. Here λ > 0 is a regularization parameter to be determined. Assumptions. To simplify the analysis we make three assumptions. First, we assume that the data is independent and identically distributed. This implies that the with an increasing level of randomness the complexity of the data also increases. In dependent data, this is not necessarily the case. As in that case correlation in the data can be destroyed by the introduction of randomness making the data faster to learn. Second, we assume that the model complexity is controlled by a regularization parameter in such a way that an increase of the regularization parameter implies a strict decrease of the complexity of the model. Third, we assume that the regularization parameter and the randomness are on a similar scale. To explain this, note that the accuracy is a function of the regularization parameter and randomness. The assumption simply means that the norm of the gradient of the accuracy taking in direction of the regularization is of the same order as the norm of the gradient taken in direction of randomness. Creating randomized data. In this paper we consider a classification problem. Each data point consists of a pair (x, y) which is typically an image and a corresponding class. For a fixed level of randomness p ∈ we split the training data Z = (X, Y) in two parts. The first split Z p contains a random sample of p-percent of each class of the training data. The second part Z 1−p contains the rest of the training data. The classes of the first set are randomly permuted to give the setZ p. The randomized data is obtained by joining D p =Z p ∪ Z p−1. With this D 0 is equal to the original data set Z and D 1 is obtained by randomly permuting all class labels Y of the data set Z.Accuracy curves. Central to our paper are accuracy plots. Here we plot the accuracy as computed on the training data over the randomness of the training data. Alternatively the plot of the accuracy as computed on the training data over the regularization parameter. In the paper we call such curves in the plots accuracy over randomness curves and accuracy over regularization curves. To generate the plots we keep everything fixed except the randomness of the training data or the regularization parameter. Monotony. Let us assume that we successfully trained our model on the unperturbed data set D 0 = Z. This means that the accuracy over randomness curve starts in the left upper corner at some point close to 1. As we increase the level of randomness the training data D p becomes more complex and it is thus more difficult to learn, which means our algorithm will take longer to achieve the same training error. Thus, we expect that the accuracy drops. In other words the accuracy curve is strictly monotonically decreasing for increasing randomness. Further, if we increase the regularization parameter the model complexity drops. Thus we also expect that accuracy drops if the regularization of the model is increased. This shows that our assumption imply that the accuracy is strictly monotonically decreasing as a function of randomness and regularization. Figure 1 shows the qualitative behavior of accuracy over randomness curves which follows the assumption we made. Real accuracy curves To compare these idealized curves with accuracy curves of real data we computed the accuracy curves for different data sets, see Figure 2. In each subfigure we trained a neural network on either mnist (a), cifar10 (b), and patched-noise (c) -a generated data set. For each curve we varied the l 1 regularization parameter. Furthermore, for each randomness value on the x-axis, the network was trained for five different randomization of the labels. More details on the networks, the data sets and more plots can be found in the appendix. Criterion 1 -Convexity of accuracy curve. In Figure 1 (a) three types of accuracy curves can be seen: dashed concave curves, dotted convex curves, and a full straight line. If the accuracy curve of our model is above the straight line, the model is able to learn noise. In other words the model overfits. Analogously, an accuracy curve below the straight line, shows underfitting. The model class has not enough capacity to learn the data. The figures show the accuracy curves for three different data sets. Each curve represents a different l 1 -regularization. The curves start with no regularization, depicted in blue, to strong regularization depicted in red. As the regularization is increase the curve tend to pushed below the straight line, confirming our intuition. The model overfits if the accuracy computed on the true data set is close to one and the accuracy over randomness curve of the model is strictly concave. The model underfits if the accuracy curve is strictly convex. This criterion can be computed by measuring the squared distance of the accuracy curve to the straight line connecting the points and (1, 1 number of classes). So assuming the r 1,..., r n parametrize the randomization of the labels, a(r i) denotes the accuracy at r i the criterion can be computed by: DISPLAYFORM0 The criterion if met if crit1 is small. Criterion 2 -Steep decrease in accuracy. Following our criterion 1 we want to determine if the accuracy curves are convex. Let us recall the accuracy curves are both strictly monotone decreasing in randomness and regularization. And that we are assuming that randomness and regularization are on a similar scale. If we look at the point in the upper left of Figure 1 (a) we see that the curves are convex if the accuracy drops sharply as the randomness increases. As the accuracy curve is also monotone decreasing with increasing regularization we will also detect the convexity by a steep drop in accuracy as depicted by the marked point in the Figure 1 (b).The model overfits if the accuracy on the training data is close to one and the accuracy over regularization curve (plotted in log-log space) is constant. Otherwise it underfits. This criterion can be detected by approximating the derivative crit2 = ∂ ∂λ a(λ) of the accuracy a as the regularization parameter λ increases. If the derivative becomes larger than a threshold, the optimal value is found. Criterion 3 -Two modes in margin histograms. Finally we derive a criterion based on the margin histograms of the fully randomized training data. Looking again at 1 we see that the accuracy of the underfitting curves remains constant if we decrease the randomness just a tiny bit. While training our model several things happen simultaneously. At starting time the network outputs random noise. If the parameter settings leads to a successful training, the model typically has a phase in which it outputs one class for all inputs. Looking at the margins of this phase, the distribution has two modes, one negative and a positive one containing the mass of 1 number of classes examples. Once we train further the two modes combine to one and the network starts to converge. Our third criterion looks for these two mode, because the accuracy will remain constant for a tiny decrease in randomness, as the two modes have to collapse before the accuracy can increase, we are thus in the underfitting regime. The model overfits if the margin histograms computed with fully random data D 1 respectively with true data D 0 are both positive. The model underfits if the margin histogram computed with fully randomized training data D 1 has two mode. This criterion can also be evaluated, by simply computing the quotient of positive margins to negative margins. So if i denotes the index of a training sample, and h i its margin, then the criterion is computed by DISPLAYFORM1 where χ denotes the indicator function. The criterion is fulfilled if crit3 is close to The plots show the accuracy curves of the network trained on cifar10 over different degrees of randomness with increasing degree of l 1 -regularization, after 19999 iterations. We select λ such that the blue accuracy curve stays below the optimal green line and is convex. Following our convexity criterion we choose λ * 1 = 0.00011 as regularization factor. Cifar10. We tested our criteria on cifar-10. This data set consists of 50000 images, equally divided in ten classes. As described above we generated random samples of the training data which we denote by cifar-10 0.0, cifar-10 0.1,...,cifar-10 1.0. A superscript stands for the fraction of randomized samples per class. So cifar-10 0.0 stands for the original data. In cifar-10 0.5 half of the labels of each class are randomly permuted while the other half remains fixed. Finally, in cifar-10 1.0 all class labels are randomly permuted. Details on the architecture and training parameter can be found in the appendix. Mnist. The data sets were created similar to cifar10.Noise. This data set consists of 50000 randomly generated rgb noise, equally divided in ten classes. For each class a set of 100 random 5x5x3 patches were generated. Each image sampled 36 patches from the set of patches to build a 30x30x3 rgb image. In this way 50000 training and 50000 test images were generated. The randomization was similar to cifar10. The capacity of a neural network is controlled by the number of parameters and (optionally) by a regularization term. In this section we show that the techniques of the paper can be used to tune the regularization parameter in order to avoid overfitting while still performing well. Convexity criterion. For the convexity criterion, we compute the accuracy curves over increasingly randomized data for different accuracy parameters as shown in FIG1. We expect that our algorithm achieves zero training error for true data. Furthermore for p-percent of randomized training data we expect the algorithm to achieve an error of p-percent. In other word we expect the algorithm to stay below a straight line starting at 1 for data 0.0 and going down to 1/number of classes for data 1.0. Let us call this line the optimal line. We pick the smallest λ for which the training error stays below the optimal line and is convex.l 1 -regularization. In these set of experiments we varied the regularization factor of the l 1 regularization of our Alexnet-type network on cifar10. To get more reliable we run the experiments for five different random samples. From the samples we computed the mean and the standard deviation, which we indicated in the plots by shading. Our experiments show that all criteria lead to a similar regularization factor λ *. With increasing regularization the curves approach the straight line, from which the optimal regularization parameter 0.0023 can be determined. In (b) the same network was trained on mnist with varying dropout rate and no other regularization. With decreasing drop out rates the curves approach the straight line. In the setting of the experiment a drop out rate of 0.1 is optimal. For (c) we trained an Alexnet-type neural network with l 2 regularization with parameter 0.0012, and computed the accuracy curves for the steps 1000, 2000,..., 60000. For 1000 training steps the curve is the lowest, for 60000 training steps the curve approaches the constant 1.In FIG1 the optimal line is depicted in green. According to our criterion we choose λ 1 = 0.00011. Both criteria (C2) and (C3) lead to a similar regularization parameter, details can be found in appendix B.1. l 2 -regularization. In these set of experiments we varied the regularization factor of the l 2 regularization of a (different) Alexnet-type network on mnist. Again we run the experiments for five different random samples and computed the mean and the standard deviation, which we indicated in the plots by shading. The ing accuracy curves are shown in Figure (a) 4, the ing parameter is λ 2 = 0.0023.Dropout. Dropout is another way to regularize a neural network. To analyze its effect on overfitting we trained a Alexnet-type network on mnist with different probabilities of drop out ranging from 0.1 to 1.0. The ing accuracy curves are shown in Figure (b) 4, in the setting of the experiment the optimal drop out value is 0.1.Early stopping. Early stopping follows the rational less training time leads to better generalization properties. In Figure (c) 4 we can observe this rational. The more we train the network the more the network overfits. The curves are a bit more bumpy as we trained the models only once. In this section we compare the training accuracy with the test accuracy. The plots of Figure show nine panels with different regularization factors. In each panel the accuracy as computed on the training data is plotted in blue, and the accuracy of the test data in red. So each blue dot represents the accuracy of the different training sets cifar10 0.0... cifar10 1.0. Each red point is computed for the same test set. With no regularization the model highly overfits. The model learns randomized data rather well as shown in λ 1 = 0.0 in FIG6. We further observe that it is easier for the network to learn more random data. With no l 1 -regularization the accuracy curves decreases with increasing randomness in the data and then starts to increase again. We attribute this to correlation in the data set which make the training data more complex for lower noise levels. Higher noise levels destroy these correlations and the data complexity of the data reduces. Recall that the small animal classes and also the car / truck class are correlated in cifar10. Finally, we note that the variance for learning entirely random data is very high. As the regularization parameter increases the blue accuracy show that the network is less able to learn random data. For λ 1 = 0.00011 the curve is convex, showing the optimal regularization parameter. In λ 1 > 0.00011 the model underfits. Looking again at λ 1 = 0.00011 we see that the model is able to learn from noisy data with lots of label noise. This confirms that l 1 -regularization is a good parameter to adjust the model complexity. Plots similar to Figure 5 can be used to analyze early stopping and overfitting. Due to lack of space we will only describe the verbally. Early in the training, at 19999 steps, we see that almost all curve are convex, hence the models underfit. Once we train the network to 59999 iterations, the model trained without any regularization begins to overfit with the others still underfitting. Training the networks further the more and more models begin to overfit. Flipping through the plots in the appendix illustrates this nicely. We also looked at models with different filter sizes in the first convolutional layer. We trained several networks with filter sizes starting from 2 × 2 to 9 × 9 and a regularization parameter of λ 1 = 0.00011. We observed that all networks showed underfitting, revealed by the convexity of the accuracy over randomness curves. This hints that l 1 regularization of the kernel weights is more important to overfitting than the number of parameters. Experiments with different network depths showed a similar behavior. In the paper we measure the capacity of a neural network by injecting different noise levels in the training data. The criteria we introduced in the paper are based on the assumption that the network should only be able to achieve a training accuracy corresponding to the injected noise level. This advances previous method in the neural network setting as they rely on either a hold out set, heuristics, or generalization theory. All of which are not mature enough to detect overfitting at present. In our experiments we saw that the hyper parameters fall in two classes, one which has no effect on overfitting (kernel size) and another which controls overfitting (regularization factor, number of iterations). In other experiments on mnist and cifar10 we observed the dominance of l 1 regularization for overfitting, while structural parameters such as network width, depth did not had an effect. The convexity criterion is the most reliable, as outliers and high variance are easily detected. On the downside it requires the most training runs. The steep decrease criterion only requires to train the model on the real data and and the fully random data. It can be used to narrow the parameter range. On the down side correlation between the classes are not easily detected by the steep decrease criterion. The mode criterion, is the most easiest to use as only the totally randomized training data is used. On the downside the margin plots are not always easy to interpret. Either the entire margin is positive, then the model clearly overfits, or two modes are observed in the plots, then the model clearly underfits. Yet most of the time, the margin is somewhere in between, which makes it hard to make a judgment based on the margin histograms alone. Let us put criteria (C2) and (C3) in perspective. Criterion (C2) comes close to what has been done before. We basically train a network on true and randomly shuffled labels, and analyze the attained accuracies. An analysis of the margin histograms for networks trained on true labels and random labels has been explored before. For example in BID2 margin histograms are used to conclude that regularization only seems to bring minor benefits to test error, BID10 use the margin histograms of networks trained on fully randomized labels and true labels to discuss normalization effects. Our contribution is to show that the regularization parameter can be set such that network does train on true labels, but is unable to do so for random labels. Both criteria are able to note this effect. All criteria can be numerically evaluated and put into an automated parameter search. At present it seems that the number of parameters do not contribute to overfitting. Thus to use the criteria of this paper one would proceed in two steps: search for an architecture which achieves zero training error, and then reducing the complexity of the model by regularizing it such that it does not overfit. So the additional burden is not that much.Analyzing neural networks with randomized training data has been done before BID15 ). In the paper the authors show that a neural network is able to train random labels, and they note that regularization... is neither necessary nor by itself sufficient for controlling generalization error. In the paper we argued that l 1 -normalization of the kernel weights is a good measure to control the capacity of a network. In the experiment we saw that adjusting l 1 -normalization leads to models which do not overfit and hence we expect them to generalize better. Using an l 1 regularization (the LASSO) is one of the popular choices for regularization. The rational is typically to enforce sparsity of the network weights. Our Lemma 3.1.1 adds another reason to the list why it might be a good choice for convolutional networks. We want to highlight another unexpected illustrative . By tuning the hyper parameter to pass our overfitting tests, we see that the test accuracy of the model is much higher than the training accuracy. This shows that our criteria can also be used to learn from noisy data and that a generalization gap does not need to be a bad thing. Although the paper focused on neural networks the methods can be applied for other machine learning algorithms as well. For example it would be interesting to apply our criteria for a systematic architecture search. Another line of research could investigate whether the criteria make adversarial attacks more difficult. Figure 5: The plots shows the accuracy of the network trained on cifar10 over different degrees of randomness with increasing degree of l 1 -regularization. The network trained for 199999 iterations. For the error curves five different samples were sampled for each data point. The network was evaluated on the training set (depicted in blue) and on the test set (depicted in red). We observe that the model does not overfit for λ = 0.00011. Furthermore, we note that with this choice of λ the model is able to learn from noise data, as the red curve is clearly above the green noise level curve. Matrix norms. A matrix A: V → W can be viewed as a linear operator between two normed spaces (V, || · || p) and (W, || · || q). We equip these normed spaces with a p-norms. So for x ∈ V we set ||x|| p = (i |x i | p) 1 p and for y ∈ W we set ||y|| q = (i |y i | q) 1 q. These vector space norms induce a matrix norm for A: DISPLAYFORM0 Special cases of this norm include the spectral norm ||A|| σ = ||A|| 2→2, ||A|| 1→1 = max 1≤j≤n m i=1 |a i j| and ||A|| ∞→∞ = max 1≤j≤m n i=1 |a i j|. In the paper we use the following fact: DISPLAYFORM1 A definition of these norms can be found in books about matrix analysis see for example §2.3.1 BID3 for a definition of the ||A|| p→q norm (in a slightly different notation). Equation FORMULA12 can be found in Corollary 2.3.2 of the same reference. Generalized mean. For a non zero real number p and positive reals x 1,..., x n we define the generalized mean by DISPLAYFORM2 We will use the following inequality which holds true for all real p < q and positive x y 1 ),..., (x n, y n) drawn iid from some probability distribution over R d0 × {1, . . ., k}, with probability at least 1 − δ over ((x i, y i)) n i=1, every margin γ > 0 and admissible network f: In our steep descent criterion we propose to detect the first change point at which the accuracy is not constant anymore but falls linearly. In the figure this is depicted by the green curves: around 10 −4 the accuracy begins to fall. As a measure of how much the net learned about the data, we also provide the accuracy curves for random data. We conclude from the gap between the red and the blue curve that the net learned something meaningful about the data, instead of just memorizing the data. DISPLAYFORM3 DISPLAYFORM4 In this section we empirically show that criteria (C2) and (C3) lead to similar than criterion (C1). Regularization. In these set of experiments we varied the regularization factor of the l 1 regularization of our Alexnet-type network. To get more reliable we run the experiments for five different random samples. From the samples we computed the mean and the standard deviation, which we indicated in the plots by shading. The following experiments show that all criteria lead to a similar regularization factor λ *.Steep decrease criterion. To test our steep decrease criterion we computed the accuracy over regularization curve. As the regularization increases we expect the accuracy to drop. This is shown in FIG5. Following our criterion the point of interest λ * occurs, at which the curve is not constant anymore. This occurs around λ 1 = 0.0001.Mode criterion. To test our mode criterion we computed the margin histograms of cifar-10 1.0 after training. As the regularization increases we expect the distribution to split up in two modes. This can be seen in FIG6. Following our criterion the point of interest occurs around λ 1 = 0.00011. Architecture and training parameter. Figure 8 shows a sketch of the architecture used in most experiments. We start with 5 × 5 convolutional filters, followed by 3 × 3-convolutional filters and two fully connected layers. In all layers the linear part is followed by a relu nonlinearity. We did not use a bias. In addition we apply three non overlapping 2 × 2-max-pooling. Further we used drop-out between the first and second fully connected layer. Everything was coded in Tensorflow, with SGD, a fixed learning rate of 0.01, a batch size of 32 and l 1 normalization of all weights. The networks were trained for 199999 steps. The input images were normalized to zero mean and standard deviation using one of tensorflows build in function. Additionally, we used some standard data augmentation. Here we provide additional plots of our l 1 regularization experiments, showing that all criterion have their uses. Figure shows how we would detect overfitting with the margin based criterion. Let us recall that a positive margin corresponds to a correct classification and a negative margin corresponds to an incorrect classification. In (a) and (e) of Figure D.1 the model clearly overfits, as it is able to learn random data (e) and true data (a). In (c) and (g) of Figure D.1, we clearly see underfitting the model neither able to learn random data nor true data. Based on this observation we would select λ = 0.0001 as our regularization parameter. Here we report similar plot for our early stopping experiments. Flipping through the plots we see that initial the regularization factor does not matter at 19999 steps all curves are convex. At later iterations the models begin to memorize the data. Figure 17: The plots shows the accuracy of the network trained on cifar10 over different degrees of randomness with increasing degree of l 1 -regularization. The network trained for 159999 iterations. For the error curves five different samples were sampled for each data point. The network was evaluated on the training set (depicted in blue) and on the test set (depicted in red). Figure 18: The plots shows the accuracy of the network trained on cifar10 over different degrees of randomness with increasing degree of l 1 -regularization. The network trained for 179999 iterations. For the error curves five different samples were sampled for each data point. The network was evaluated on the training set (depicted in blue) and on the test set (depicted in red). Figure 19: The plots shows the accuracy of the network trained on cifar10 over different degrees of randomness with increasing degree of l 1 -regularization. The network trained for 199999 iterations. For the error curves five different samples were sampled for each data point. The network was evaluated on the training set (depicted in blue) and on the test set (depicted in red).

Represent the short summary for finding the corresponding scientific article (long)

We introduce and analyze several criteria for detecting overfitting.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.