Question:
represent the scientific article to find a short tldr

Fewshots:
'We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval. Humans are capable of learning visual concepts by jointly understanding vision and language BID12 BID8 BID15. Consider the example shown in Figure 1 -I. Imagine that someone with no prior knowledge of colors is presented with the images of the red and green cubes, paired with the questions and answers. They can easily identify the difference in objects' visual appearance (in this case, color), and align it to the corresponding words in the questions and answers (Red and Green). Other object attributes (e.g., shape) can be learned in a similar fashion. Starting from there, humans are able to inductively learn the correspondence between visual concepts and word semantics (e.g., spatial relations and referential expressions, Figure 1 -II), and unravel compositional logic from complex questions assisted by the learned visual concepts (Figure 1 -III, also see BID0).Motivated by this, we propose the neuro-symbolic concept learner (NS-CL), which jointly learns visual perception, words, and semantic language parsing from images and question-answer pairs. NS-CL has three modules: a neural-based perception module that extracts object-level representations from the scene, a visually-grounded semantic parser for translating questions into executable programs, and a symbolic program executor that reads out the perceptual representation of objects, classifies their attributes/relations, and executes the program to obtain an answer. Figure 1: Humans learn visual concepts, words, and semantic parsing jointly and incrementally. I. Learning visual concepts (red vs. green) starts from looking at simple scenes, reading simple questions, and reasoning over contrastive examples BID12. II. Afterwards, we can interpret referential expressions based on the learned object-based concepts, and learn relational concepts (e.g., on the right of, the same material as). III Finally, we can interpret complex questions from visual cues by exploiting the compositional structure. NS-CL learns from natural supervision (i.e., images and QA pairs), requiring no annotations on images or semantic programs for sentences. Instead, analogical to human concept learning, it learns via curriculum learning. NS-CL starts by learning representations/concepts of individual objects from short questions (e.g., What's the color of the cylinder?) on simple scenes (≤3 objects). By doing so, it learns object-based concepts such as colors and shapes. NS-CL then learns relational concepts by leveraging these object-based concepts to interpret object referrals (e.g., Is there a box right of a cylinder?). The model iteratively adapts to more complex scenes and highly compositional questions. NS-CL's modularized design enables interpretable, robust, and accurate visual reasoning: it achieves state-of-the-art performance on the CLEVR dataset (a). More importantly, it naturally learns disentangled visual and language concepts, enabling combinatorial generalization w.r.t. both visual scenes and semantic programs. In particular, we demonstrate four forms of generalization. First, NS-CL generalizes to scenes with more objects and longer semantic programs than those in the training set. Second, it generalizes to new visual attribute compositions, as demonstrated on the CLEVR-CoGenT (a) dataset. Third, it enables fast adaptation to novel visual concepts, such as learning a new color. Finally, the learned visual concepts transfer to new tasks, such as image-caption retrieval, without any extra fine-tuning. Our model is related to research on joint learning of vision and natural language. In particular, there are many papers that learn visual concepts from descriptive languages, such as image-captioning or visually-grounded question-answer pairs (; ; ; ; BID14, dense language descriptions for scenes , video-captioning BID10 and video-text alignment .Visual question answering (VQA) stands out as it requires understanding both visual content and language. The state-of-the-art approaches usually use neural attentions (; BID6 ;). Beyond question answering, Johnson et al. (2017a) proposed the CLEVR (VQA) dataset to diagnose reasoning models. CLEVR contains synthetic visual scenes and questions generated from latent programs. Table 1 compares our model with state-of-the-art visual reasoning models BID2; ) along four directions: visual features, semantics, inference, and the requirement of extra labels. explored an interpretable, object-based visual representation for visual reasoning. It performs well, but requires fully-annotated scenes during training. Our model also adopts an object-based visual representation, but the representation is learned only based on natural supervision (questions and answers). BID1 Figure 2: We propose to use neural symbolic reasoning as a bridge to jointly learn visual concepts, words, and semantic parsing of sentences.networks and answers the questions by question-conditioned attention over the object features. In contrast, NS-CL parses question inputs into programs and executes them on object features to get the answer. This makes the reasoning process interpretable and supports combinatorial generalization over quantities (e.g., counting objects). Our model also learns general visual concepts and their association with symbolic representations of language. These learned concepts can then be explicitly interpreted and deployed in other vision-language applications such as image caption retrieval. There are two types of approaches in semantic sentence parsing for visual reasoning: implicit programs as conditioned neural operations (e.g., conditioned convolution and dual attention) and explicit programs as sequences of symbolic tokens BID2 b; ). As a representative, BID2 build modular and structured neural architectures based on programs for answering questions. Explicit programs gain better interpretability, but usually require extra supervision such as groundtruth program annotations for training. This restricts their application. We propose to use visual grounding as distant supervision to parse questions in natural languages into explicit programs, with zero program annotations. Given the semantic parsing of questions into programs, proposed a purely symbolic executor for the inference of the answer in the logic space. Compared with theirs, we propose a quasi-symbolic executor for VQA.Our work is also related to learning interpretable and disentangled representations for visual scenes using neural networks. proposed convolutional inverse graphics networks for learning and inferring pose of faces, while learned disentangled representation of pose of chairs from images. proposed the neural scene de-rendering framework as an inverse process of any rendering process.; learned disentangled representations using deep generative models. In contrast, we propose an alternative representation learning approach through joint reasoning with language. We present our neuro-symbolic concept learner, which uses a symbolic reasoning process to bridge the learning of visual concepts, words, and semantic parsing of sentences without explicit annotations We treat attributes such as Shape and Color as neural operators. The operators map object representations into a visual-semantic space. We use similarity-based metric to classify objects.for any of them. We first use a visual perception module to construct an object-based representation for a scene, and run a semantic parsing module to translate a question into an executable program. We then apply a quasi-symbolic program executor to infer the answer based on the scene representation. We use paired images, questions, and answers to jointly train the visual and language modules. Shown in Figure 2, given an input image, the visual perception module detects objects in the scene and extracts a deep, latent representation for each of them. The semantic parsing module translates an input question in natural language into an executable program given a domain specific language (DSL). The generated programs have a hierarchical structure of symbolic, functional modules, each fulfilling a specific operation over the scene representation. The explicit program semantics enjoys compositionality, interpretability, and generalizability. The program executor executes the program upon the derived scene representation and answers the question. Our program executor works in a symbolic and deterministic manner. This feature ensures a transparent execution trace of the program. Our program executor has a fully differentiable design w.r.t. the visual representations and the concept representations, which supports gradient-based optimization during training. Visual perception. Shown in Figure 2, given the input image, we use a pretrained Mask R-CNN to generate object proposals for all objects. The bounding box for each single object paired with the original image is then sent to a ResNet-34 to extract the region-based (by RoI Align) and image-based features respectively. We concatenate them to represent each object. Here, the inclusion of the representation of the full scene adds the contextual information, which is essential for the inference of relative attributes such as size or spatial position. Concept quantization. Visual reasoning requires determining an object's attributes (e.g., its color or shape). We assume each visual attribute (e.g., shape) contains a set of visual concept (e.g., Cube). In NS-CL, visual attributes are implemented as neural operators, mapping the object representation into an attribute-specific embedding space. FIG1 shows an inference an object's shape. Visual concepts that belong to the shape attribute, including Cube, Sphere and Cylinder, are represented as vectors in the shape embedding space. These concept vectors are also learned along the process. We measure the cosine distances ·, · between these vectors to determine the shape of the object. Specifically, we compute the probability that an object o i is a cube by σ ShapeOf(o i), v Cube − γ τ, where ShapeOf(·) denotes the neural operator, v Cube the concept embedding of Cube and σ the Sigmoid function. γ and τ are scalar constants for scaling and shifting the values of similarities. We classify relational concepts (e.g., Left) between a pair of objects similarly, except that we concatenate the visual representations for both objects to form the representation of their relation. DSL and semantic parsing. The semantic parsing module translates a natural language question into an executable program with a hierarchy of primitive operations, represented in a domain-specific language (DSL) designed for VQA. The DSL covers a set of fundamental operations for visual reasoning, such as filtering out objects with certain concepts or query the attribute of an object. The operations share the same input and output interface, and thus can be compositionally combined to form programs of any complexity. We include a complete specification of the DSL used by our framework in the Appendix A. Our semantic parser generates the hierarchies of latent programs in a sequence to tree manner BID11. We use a bidirectional GRU BID7 to encode an input question, which outputs a fixed-length embedding of the question. A decoder based on GRU cells is applied to the embedding, and recovers the hierarchy of operations as the latent program. Some operations takes concepts their parameters, such as Filter(Red) and Query(Shape). These concepts are chosen from all concepts appeared in the input question. Figure 4 (B) shows an example, while more details can be found in Appendix B.Quasi-symbolic program execution. Given the latent program recovered from the question in natural language, a symbolic program executor executes the program and derives the answer based on the object-based visual representation. Our program executor is a collection of deterministic functional modules designed to realize all logic operations specified in the DSL. Figure 4 (B) shows an illustrative execution trace of a program. To make the execution differentiable w.r.t. visual representations, we represent the intermediate in a probabilistic manner: a set of objects is represented by a vector, as the attention mask over all objects in the scene. Each element, Mask i ∈ denotes the probability that the i-th object of the scene belongs to the set. For example, shown in Figure 4 (B), the first Filter operation outputs a mask of length 4 (there are in total 4 objects in the scene), with each element representing the probability that the corresponding object is selected out (i.e., the probability that each object is a green cube). The output "mask" on the objects will be fed into the next module (Relate in this case) as input and the execution of programs continues. The last module outputs the final answer to the question. We refer interested readers to Appendix C for the implementation of all operators. Optimization objective. The optimization objective of NS-CL is composed of two parts: concept learning and language understanding. Our goal is to find the optimal parameters Θ v of the visual perception module Perception (including the ResNet-34 for extracting object features, attribute operators. and concept embeddings) and Θ s of the semantic parsing module SemanticParse, to maximize the likelihood of answering the question Q correctly: DISPLAYFORM0 where P denotes the program, A the answer, S the scene, and Executor the quasi-symbolic executor. The expectation is taken over P ∼ SemanticParse(Q; Θ s).Recall the program executor is fully differentiable w.r.t. the visual representation. We compute the gradient w.r.t. DISPLAYFORM1 We use RE-INFORCE to optimize the semantic parser Θ s: DISPLAYFORM2, where the reward r = 1 if the answer is correct and 0 otherwise. We also use off-policy search to reduce the variance of REINFORCE, the detail of which can be found in Appendix D.Curriculum visual concept learning. Motivated by human concept learning as in Figure 1, we employ a curriculum learning approach to help joint optimization. We heuristically split the training samples into four stages (Figure 4 (A)): first, learning object-level visual concepts; second, learning relational questions; third, learning more complex questions with perception modules fixed; fourth, joint fine-tuning of all modules. We found that this is essential to the learning of our neuro-symbolic concept learner. We include more technical details in Appendix E. We demonstrate the following advantages of our NS-CL. First, it learns visual concepts with remarkable accuracy; second, it allows data-efficient visual reasoning on the CLEVR dataset (a); third, it generalizes well to new attributes, visual composition, and language domains. We train NS-CL on 5K images (<10% of CLEVR's 70K training images). We generate 20 questions for each image for the entire curriculum learning process. The Mask R-CNN module is pretrained on 4K generated CLEVR images with bounding box annotations, following. Classification-based concept evaluation. Our model treats attributes as neural operators that map latent object representations into an attribute-specific embedding space (FIG1). We evaluate the concept quantization of objects in the CLEVR validation split. Our model can achieve near perfect classification accuracy (∼99%) for all object properties, suggesting it effectively learns generic concept representations. The for spatial relations is relatively lower, because CLEVR does not have direct queries on the spatial relation between objects. Thus, spatial relation concepts can only be learned indirectly. Count-based concept evaluation. The SOTA methods do not provide interpretable representation on individual objects (a; ;). To evaluate the visual concepts learned by such models, we generate a synthetic question set. The diagnostic question set contains simple questions as the following form: "How many red objects are there?". We evaluate the performance on all concepts appeared in the CLEVR dataset. Table 2 summarizes the compared with strong baselines, including methods based on convolutional features (b) and those based on neural attentions . Our approach outperforms IEP by a significant margin (8%) and attention-based baselines by >2%, suggesting object-based visual representations and symbolic reasoning helps to interpret visual concepts. NS-CL jointly learns visual concepts, words and semantic parsing by watching images and reading paired questions and answers. It can be directly applied to VQA. Table 2: We also evaluate the learned visual concepts using a diagnostic question set containing simple questions such as "How many red objects are there?". NS-CL outperforms both convolutional and attentional baselines. The suggested object-based visual representation and symbolic reasoning approach perceives better interpretation of visual concepts. Table 3: We compare different variants of baselines for a systematic study on visual features and data efficiency. Using only 10% of the training images, our model is able to achieve a comparable with the baselines trained on the full dataset. See the text for details. TAB5 summarizes on the CLEVR validation split. Our model achieves the state-of-theart performance among all baselines using zero program annotations, including MAC and FiLM . Our model achieves comparable performance with the strong baseline TbD-Nets , whose semantic parser is trained using 700K programs in CLEVR (ours need 0). The recent NS-VQA model from Yi et al. FORMULA0 achieves better performance on CLEVR; however, their system requires annotated visual attributes and program traces during training, while our NS-CL needs no extra labels. Here, the visual perception module is pre-trained on ImageNet BID9. Without pretraining, the concept learning accuracies drop by 0.2% on average and the QA accuracy drops by 0.5%. Meanwhile, NS-CL recovers the underlying programs of questions accurately (> 99.9% accuracy). NS-CL can also detect ambiguous or invalid programs and indicate exceptions. Please see Appendix F for more details. NS-CL can also be applied to other visual reasoning testbeds. Please refer to Appendix G.1 for our on the Minecraft dataset .For a systematic study on visual features and data efficiency, we implement two variants of the baseline models: TbD-Object and MAC-Object. Inspired by BID1, instead of the input image, TbD-Object and MAC-Object take a stack of object features as input. TbD-Mask and MAC-Mask integrate the masks of objects by using them to guide the attention over the images. Table 3 summarizes the . Our model outperforms all baselines on data efficiency. This comes from the full disentanglement of visual concept learning and symbolic reasoning: how to execute program instructions based on the learned concepts is programmed. TbD-Object and MAC-Object demonstrate inferior in our experiments. We attribute this to the design of model architectures and have a detailed analysis in Appendix F.3. Although TbD-Mask and MAC-Mask do not perform better than the originals, we find that using masks to guide attentions speeds up the training. Besides achieving a competitive performance on the visual reasoning testbeds, by leveraging both object-based representation and symbolic reasoning, out model learns fully interpretable visual concepts: see Appendix H for qualitative on various datasets. Generalizing to new visual compositions. The CLEVR-CoGenT dataset is designed to evaluate models' ability to generalize to new visual compositions. It has two splits: Split A only contains gray, blue, brown and yellow cubes, but red, green, purple, and cyan cylinders; split B imposes the opposite color constraints on cubes and cylinders. If we directly learn visual concepts on split A, it overfits to classify shapes based on the color, leading to a poor generalization to split B.Our solution is based on the idea of seeing attributes as operators. Specifically, we jointly train the concept embeddings (e.g., Red, Cube, etc.) as well as the semantic parser on split A, keeping pretrained, frozen attribute operators. As we learn distinct representation spaces for different attributes, our model achieves an accuracy of 98.8% on split A and 98.9% on split B. Figure 6: Samples collected from four splits in Section 4.3 for illustration. Models are trained on split A but evaluated on all splits for testing the combinatorial generalization. Generalizing to new visual concepts. We expect the process of concept learning can take place in an incremental manner: having learned 7 different colors, humans can learn the 8-th color incrementally and efficiently. To this end, we build a synthetic split of the CLEVR dataset to replicate the setting of incremental concept learning. Split A contains only images without any purple objects, while split B contains images with at least one purple object. We train all the models on split A first, and finetune them on 100 images from split B. We report the final QA performance on split B's validation set. All models use a pre-trained semantic parser on the full CLEVR dataset. Our model performs a 93.9% accuracy on the QA test in Split B, outperforming the convolutional baseline IEP (b) and the attentional baseline TbD by 4.6% and 6.1% respectively. The acquisition of Color operator brings more efficient learning of new visual concepts. Having learned visual concepts on small-scale scenes (containing only few objects) and simple questions (only single-hop questions), we humans can easily generalize the knowledge to larger-scale scenes and to answer complex questions. To evaluate this, we split the CLEVR dataset into four parts: Split A contains only scenes with less than 6 objects, and questions whose latent programs having a depth less than 5; Split B contains scenes with less than 6 objects, but arbitrary questions; Split C contains arbitrary scenes, but restricts the program depth being less than 5; Split D contains arbitrary scenes and questions. Figure 6 shows some illustrative samples. As VQA baselines are unable to count a set of objects of arbitrary size, for a fair comparison, all programs containing the "count" operation over > 6 objects are removed from the set. ForCaption: There is a big yellow cylinder in front of a gray object.(a) An illustrative pair of image and caption in our synthetic dataset. Table 5: We introduce a new simple DSL for image-caption retrieval to evaluate how well the learned visual concepts transfer. Due to the difference between VQA and caption retrieval, VQA baselines are only able to infer the on a partial set of data. The learned object-based visual concepts can be directly transferred into the new domain for free. methods using explicit program semantics, the semantic parser is pre-trained on the full dataset and fixed. Methods with implicit program semantics learn an entangled representation for perception and reasoning, and cannot trivially generalize to more complex programs. We only use the training data from the Split A and then quantify the generalization ability on other three splits. Shown in Table 5, our NS-CL leads to almost-perfect generalization to larger scenes and more complex questions, outperforming all baselines by at least 4% in QA accuracy. The learned visual concepts can also be used in other domains such as image retrieval. With the visual scenes fixed, the learned visual concepts can be directly transferred into the new domain. We only need to learn the semantic parsing of natural language into the new DSL.We build a synthetic dataset for image retrieval and adopt a DSL from scene graph-based image retrieval . The dataset contains only simple captions: "There is an <object A> <relation> <object B>." (e.g., There is a box right of a cylinder). The semantic parser learns to extract corresponding visual concepts (e.g., box, right, and cylinder) from the sentence. The program can then be executed on the visual representation to determine if the visual scene contains such relational triples. For simplicity, we treat retrieval as classifying whether a relational triple exists in the image. This functionality cannot be directly implemented on the CLEVR VQA program domain, because questions such as "Is there a box right of a cylinder" can be ambiguous if there exist multiple cylinders in the scene. Due to the entanglement of the visual representation with the specific DSL, baselines trained on CLEVR QA can not be directly applied to this task. For a fair comparison with them, we show the in Table 5b on a subset of the generated image-caption pairs where the underlying programs have no ambiguity regarding the reference of object B. A separate semantic parser is trained for the VQA baselines, which translates captions into a CLEVR QA-compatible program (e.g., Exist(Filter(Box, Relate(Right, Filter(Cylinder))). Table 5c compares our NS-CL against typical image-text retrieval baselines on the full image-caption dataset. Without any annotations of the sentence semantics, our model learns to parse the captions into the programs in the new DSL. It outperforms the CNN-LSTM baseline by 30%. We further conduct experiments on MS-COCO images. Results are presented on the VQS dataset BID13. VQS contains a subset of images and questions from the original VQA 1.0 dataset BID3. All questions in the VQS dataset can be visually grounded: each question is associated with multiple image regions, annotated by humans as essential for answering the question. Figure 7 illustrates an execution trace of NS-CL on VQS.We use a syntactic dependency parser to extract programs and concepts from language BID2 ). The object proposals and features are extracted from models pre-trained on the MS-COCO dataset and the ImageNet dataset, respectively. Illustrated in Figure 7, our model FIG2 shows examples of the learned visual concepts, including object categories, attributes, and relations. Experiment setup and implementation details are in Appendix G.2.In this paper, we focus on a neuro-symbolic framework that learns visual concepts about object properties and relations. Indeed, visual question answering requires AI systems to reason about more general concepts such as events or activities . We leave the extension of NS-CL along this direction and its application to general VQA datasets BID3 as future work. We presented a method that jointly learns visual concepts, words, and semantic parsing of sentences from natural supervision. The proposed framework, NS-CL, learns by looking at images and reading paired questions and answers, without any explicit supervision such as class labels for objects. Our model learns visual concepts with remarkable accuracy. Based upon the learned concepts, our model achieves good on question answering, and more importantly, generalizes well to new visual compositions, new visual concepts, and new domain specific languages. The design of NS-CL suggests multiple research directions. First, constructing 3D object-based representations for realistic scenes needs further exploration BID1 BID5. Second, our model assumes a domain-specific language for describing formal semantics. The integration of formal semantics into the processing of complex natural language would be meaningful future work BID4 ). We hope our paper could motivate future research in visual concept learning, language learning, and compositionality. Our framework can also be extended to other domains such as video understanding and robotic manipulation. Here, we would need to discover semantic representations for actions and interactions (e.g., push) beyond static spatial relations. Along this direction, researchers have studied building symbolic representations for skills and learning instruction semantics from interaction in constrained setups. Applying neuro-symbolic learning frameworks for concepts and skills would be meaningful future work toward robotic learning in complex interactive environments. We first introduce the domain-specific language (DSL) designed for the CLEVR VQA dataset (a). Table 6 shows the available operations in the DSL, while TAB9 explains the type system. Scene −→ ObjectSet Return all objects in the scene. Filter (ObjectSet, ObjConcept) −→ ObjectSet Filter out a set of objects having the object-level concept (e.g., red) from the input object set. DISPLAYFORM0 Filter out a set of objects that have the relational concept (e.g., left) with the input object. AERelate (Object, Attribute) −→ ObjectSet (Attribute-Equality Relate) Filter out a set of objects that have the same attribute value (e.g., same color) as the input object. Intersection (ObjectSet, ObjectSet) −→ ObjectSet Return the intersection of two object sets. DISPLAYFORM1 Return the union of two object sets. Query (Object, Attribute) −→ ObjConcept Query the attribute (e.g., color) of the input object. AEQuery (Object, Object, Attribute) −→ Bool (Attribute-Equality Query) Query if two input objects have the same attribute value (e.g., same color). DISPLAYFORM2 Query if the set is empty. DISPLAYFORM3 Query the number of objects in the input set. CLessThan (ObjectSet, ObjectSet) −→ Bool (Counting LessThan) Query if the number of objects in the first input set is less than the one of the second set. CGreaterThan (ObjectSet, ObjectSet) −→ Bool (Counting GreaterThan) Query if the number of objects in the first input set is greater than the one of the second set. CEqual (ObjectSet, ObjectSet) −→ Bool (Counting Equal) Query if the number of objects in the first input set is the same as the one of the second set. Table 6: All operations in the domain-specific language for CLEVR VQA.We note that some function takes Object as its input instead of ObjectSet. These functions require the uniqueness of the referral object. For example, to answer the question " What's the color of the red object?", there should be one and only one red object in the scene. During the program execution, the input object set will be implicitly cast to the single object (if the set is non-empty and there is only one object in the set). Such casting is named Unique in related works (b The type system of the domain-specific language for CLEVR VQA. As shown in Appendix A, a program can be viewed as a hierarchy of operations which take concepts as their parameters. Thus, NS-CL generates the hierarchies of latent programs in a sequence to tree manner BID11 . The semantic parser adopts an encoder-decoder architecture, which contains four neural modules: a bidirectional GRU encoder IEncoder BID7 to encode an input question into a fixed-length embedding, an operation decoder OpDecoder that determines the operation tokens, such as Filter, in the program based on the sentence embedding, a concept decoder ConceptDecoder that selects concepts appeared in the input question as the parameters for certain operations (e.g., Filter takes an object-level concept parameter while Query takes an attribute), and a set of output encoders {OEncoder i} which encode the decoded operations by OpDecoder and output the latent embedding for decoding the next operation. The operation decoder, the concept decoder, and the output encoders work jointly and recursively to generate the hierarchical program layout. Algorithm 1 illustrates the algorithmic outline of the semantic parser. Algorithm 1: The String-to-Tree Semantic Parser. DISPLAYFORM0 The function parse takes two inputs: the current decoding state f and all concepts appeared in the question, as a set {c i}. The parsing procedure begins with encoding the input question by IEncoder as f 0, extracting the concept set {c i} from the input question, and invoking parse(f 0, {c i}).The concept set {c i} is extracted using hand-coded rules. We assume that each concept (including object-level concepts, relational concepts, and attributes) is associated with a single word in the question. For example, the word "red" is associated with the object-level concept Red, while the word "shape" is associated with the attribute Shape. Informally, we call these words concept words. For a given question Q, the corresponding concept set {c i} is composed of all occurrences of the concept words in Q. The set of concept words is known for the CLEVR dataset. For natural language questions, one could run POS tagging to find all concept words BID2 ). We leave the automatic discovery of concept words as a future work BID15. We use the word embedding of the concept words as the representation for the concepts {c i}. Note that, these "concept embeddings" are only for the program parsing. The visual module has separate concept embeddings for aligning object features with concepts in the visual-semantic space. We now delve into the main function parse(f, {c i}): we first decode the root operation op of the hierarchy by OpDecoder(f). If op requires a concept parameter (an object-level concept, a relational concept, or an attribute), ConceptDecoder will be invoked to choose a concept from all concepts {c i}. Assuming op takes two non-concept inputs (e.g., the operation Intersection takes two object sets as its input), there will be two branches for this root node. Thus, two output encoders OEncoder 0 and OEncoder 1 will be applied to transform the current state f into two sub-states f 1 and f 2. parse will be recursively invoked based on f 1 and f 2 to generate the two branches respectively. In the DSL, the number of non-concept inputs for any operation is at most 2.In our implementation, the input encoder IEncoder first maps each word in the question into an embedding space. The word embeddings are composed of two parts: a randomly initialized word embedding of dimension 256 and a positional embedding of dimension 128 . For a concept word, its word embedding only depends on which type it belongs to (i.e. object-level, relational or attribute). Thus, after being trained on a fixed dataset, the semantic parser can parse questions with novel (unseen) concept words. The sequence of word embeddings is then encoded by a two-layer GRU with a hidden dimension of 256 * 2 (bidirectional). The function parse starts from the last hidden state of the GRU, and works recursively to generate the hierarchical program layout. Both OpDecoder and ConceptDecoder are feed-forward networks. ConceptDecoder performs attentions over the representations of all concepts {c i} to select the concepts. Output encoders OEncoder 0 and OEncoder 1 are implemented as GRU cells. Another pre-processing of the sentence is to group consecutive object-level concept words into a group and treat them together as a single concept, inspired by the notion of "noun phrases" in natural languages. The computational intuition behind this grouping is that, the latent programs of CLEVR questions usually contain multiple consecutive Filter tokens. During the program parsing and execution, we aim to fuse all such Filters into a single Filter operation that takes multiple concepts as its parameter. A Running Example As a running example, consider again the question " What is the color of the cube right of the red matte object?". We first process the sentence (by rules) as: "What is the <Attribute 1 (color)> of the <(ObjConcept 1 (cube)> <RelConcept 1 (right)> of the <ObjConcept 2 (red matte object)>?". The expected parsing of this sentence is: DISPLAYFORM1 The semantic parser encode the word embeddings with IEncoder. The last hidden state of the GRU will be used as f 0. The word embeddings of the concept words form the set {c i} = {Attribute 1, ObjConcept 1, RelConcept 1, ObjConcept 2}. The function parse is then invoked recursively to generate the hierarchical program layout. Table 8 illustrates the decoding process step-by-step. In this section, we present the implementation of all operations listed in Table 6. We start from the implementation of Object-typed and ObjectSet-typed variables. Next, we discuss how to classify objects by object-level concepts or relational concept, followed by the implementation details of all operations. Object-typed and ObjectSet-typed variables. We consider a scene with n objects. An Objecttyped variable can be represented as a vector Object of length n, where Object i ∈ and i Object i = 1. Object i can be interpreted as the probability that the i-th object of the scene is being referred to. Similarly, an ObjectSet-typed variable can be represented as a vector ObjectSet of length n, where ObjectSet i ∈. ObjectSet i can be interpreted as the probability that the i-the object is in the set. To cast an ObjectSet-typed variable ObjectSet as an Object-typed variableStep Inputs Outputs Recursive Invocation DISPLAYFORM0 (End of branch.) Table 8: A step-by-step running example of the recursive parsing procedure. The parameter {c i} is omitted for better visualization. Object (i.e., the Unique operation), we compute: Object = softmax(σ −1 (ObjectSet)), where σ −1 (x) = log(x/(1 − x)) is the logit function. Denote o i as the visual representation of the i-th object, OC the set of all object-level concepts, and A the set of all object-level attributes. Each object-level concept oc (e.g., Red) is associated with a vector embedding v oc and a L1-normalized vector b oc of length |A|. b oc represents which attribute does this object-level concept belong to (e.g., the concept Red belongs to the attribute Color). All attributes a ∈ A are implemented as neural operators, denoted as u a (e.g., uColor). To classify the objects as being Red or not, we compute: DISPLAYFORM0 where σ denotes the Sigmoid function, ·, · the cosine distance between two vectors. γ and τ are scalar constants for scaling and shifting the values of similarities. By applying this classifier on all objects we will obtain a vector of length n, denoted as ObjClassify(Red). Similarly, such classification can be done for relational concepts such as Left. This will in an n × n matrix RelClassify(Left), where RelClassify(Left) j,i is the probability that the object i is left of the object j. To classify whether two objects have the same attribute (e.g., have the same Color), we compute: DISPLAYFORM1 We can obtain a matrix AEClassify(Color) by applying this classifier on all pairs of objects, where AEClassifier(Color) j,i is the probability that the object i and j have the same Color. Quasi-symbolic program execution. Finally, Table 9 summarizes the implementation of all operators. In practice, all probabilities are stored in the log space for better numeric stability. To tackle the optimization in a non-smooth program space, we apply an off-policy program search process to facilitate the learning of the semantic parser. Denote P(s) as the set of all valid programs in the CLEVR DSL for the input question s. We want to compute the gradient w.r.t. Θ s, the parameters of the semantic parser: DISPLAYFORM0 i ) DISPLAYFORM1 i | + γc)/(γc · τc) Table 9: All operations in the domain-specific language for CLEVR VQA. γ c = 0.5 and τ c = 0.25 are constants for scaling and shift the probability. During inference, one can quantify all operations as.where P ∼ SemanticParse(s; Θ s). In REINFORCE, we approximate this gradient via Monte Carlo sampling. An alternative solution is to exactly compute the gradient. Note that in the definition of the reward r, only the set of programs Q(s) leading to the correct answer will contribute to the gradient term. With the perception module fixed, the set Q can be efficiently determined by an off-policy exhaustive search of all possible programs P(s). In the third stage of the curriculum learning, we search for the set Q offline based on the quantified of concept classification and compute the exact gradient ∇Θ s. An intuitive explanation of the off-policy search is that, we enumerate all possible programs, execute them on the visual representation, and find the ones leading to the correct answer. We use Q(s) as the "groundtruth" program annotation for the question, to supervise the learning, instead of running the Monte Carlo sampling-based REINFORCE.Spurious program suppression. However, directly using Q(s) as the supervision by computing = p∈Q(S) − log Pr(p) can be problematic, due to the spuriousness or the ambiguity of the programs. This comes from two aspects: 1) intrinsic ambiguity: two programs are different but equivalent. For example P1: AEQuery(Color, Filter(Cube), Filter(Sphere)) and P2: Exist(Filter(Sphere, AERelate(Color, Filter(Cube)))) are equivalent. 2) extrinsic spuriousness: one of the program is incorrect, but also leads to the correct answer in a specific scene. For example, P1: Filter(Red, Relate(Left, Filter(Sphere))) and P2: Filter(Red, Relate(Left, Filter(Cube))) may refer to the same red object in a specific scene. Motivated by the REINFORCE process, to suppress such spurious programs, we use the loss function: DISPLAYFORM2 The corresponding gradient ∇ Θs is, DISPLAYFORM3 The key observation is that, given a sufficiently large set of scenes, a program can be identified as spurious if there exists at least one scene where the program leads to a wrong answer. As the training goes, spurious programs will get less update due to the sampling importance term Pr[p] which weights the likelihood maximization term. During the whole training process, we gradually add more visual concepts and more complex question examples into the model. Summarized in Figure 4 (A), in general, the whole training process is split into 3 stages. First, we only use questions from lesson 1 to let the model learn object-level visual concepts. Second, we train the model to parse simple questions and to learn relational concepts. In this step, we freeze the neural operators and concept embeddings of object-level concepts. Third, the model gets trained on the full question set (lesson 3), learning to understand questions of different complexities and various format. For the first several iterations in this step, we freeze the parameters in the perception modules. In addition, during the training of all stages, we gradually increase the number of objects in the scene: from 3 to 10.We select questions for each lesson in the curriculum learning by their depth of the latent program layout. For eaxmple, the program "Query(Shape, Filter(Red, Scene))" has the depth of 3, while the program "Query(Shape, Filter(Cube, Relate(Left, Filter(Red, Scene))))" has the depth of 5. Since we have fused consecutive Filter operations into a single one, the maximum depth of all programs is 9 on the CLEVR dataset. We now present the detailed split of our curriculum learning lessons:For lesson 1, we use only programs of depth 3. It contains three types of questions: querying an attribute of the object, querying the existence of a certain type of objects, count a certain type of objects, and querying if two objects have the same attribute (e.g., of the same color). These questions are almost about fundamental object-based visual concepts. For each image, we generate 5 questions of lesson 1.For lesson 2, we use programs of depth less than 5, containing a number of questions regarding relations, such as querying the attribute of an object that is left of another object. We found that in the original CLEVR dataset, all Relate operations are followed by a Filter operation. This setup degenerates the performance of the learning of relational concepts such as Left. Thus, we add a new question template into the original template set: Count(Relate( ·, Filter( ·, Scene))) (e.g., "What's the number of objects that are left of the cube?"). For each image, we generate 5 questions of lesson 2.For lesson 3, we use the full CLEVR question set. Curriculum learning is crucial for the learning of our neuro-symbolic concept learner. We found that by removing the curriculum setup w.r.t. the number of object in the scenes, the visual perception module will get stuck at an accuracy that is similar to a random-guess model, even if we only use stage-1 questions. If we remove the curriculum setup w.r.t. the complexity of the programs, the joint training of the visual perception module and the semantic parser can not converge. We conduct ablation studies on the accuracy of semantic parsing, the impacts of the ImageNet pretraining of visual perception modules, the data efficiency of our model, and the usage of object-based representations. F.1 SEMANTIC PARSING ACCURACY.We evaluate how well our model recovers the underlying programs of questions. Due to the intrinsic equivalence of different programs, we evaluate the accuracy of programs by executing them on the ground-truth annotations of objects. Invalid or ambiguous programs are also considered as incorrect. Our semantic parser archives > 99.9% QA accuracy on the validation split. The only extra supervision of the visual perception module comes from the pre-training of the perception modules on ImageNet BID9. To quantify the influence of this pre-training, we conduct ablation experiments where we randomly initialize the perception module following. The classification accuracies of the learned concepts almost remain the same except for Shape. The classification accuracy of Shape drops from 98.7 to 97.5 on the validation set while the overall QA accuracy on the CLEVR dataset drops to 98.2 from 98.9. We speculate that large-scale image recognition dataset can provide prior knowledge of shape. In this section, we study whether and how the number of training samples and feature representations affect the overall performance of various models on the CLEVR dataset. Specifically, we compare the proposed NS-CL against two strong baselines: TbD and MAC .Baselines. For comparison, we implement two variants of the baseline models: TbD-Object and MAC-Object. Inspired by BID1, instead of using a 2D convolutional feature map, TbD-Object and MAC-Object take a stack of object features as inputs, whose shape is k × d obj. k is the number of objects in the scene, and d obj is the feature dimension for a single object. In our experiments, we fix k = 12 as a constant value. If there are fewer than 12 objects in the scene, we add "null" objects whose features are all-zero vectors. We extract object features in the same way as NS-CL. Features are extracted from a pre-trained ResNet-34 network before the last residual block for a feature map with high resolution. For each object, its feature is composed of two parts: region-based (by RoI Align) and image-based features. We concatenate them to represent each object. As discussed, the inclusion of the representation of the full scene is essential for the inference of relative attributes such as size or spatial position on the CLEVR domain. TbD and MAC networks are originally designed to use image-level attention for reasoning. Thus, we implement two more baselines: TbD-Mask and MAC-Mask. Specifically, we replace the original attention module on images with a mask-guided attention. Denotes the union of all object masks as M. Before the model applies the attention on the input image, we multiply the original attention map computed by the model with this mask M. The multiplication silences the attention on pixels that are not part of any objects. Results. Table 3 summarizes the . We found that TbD-Object and MAC-Object approach show inferior compared with the original model. We attribute this to the design of the network architectures. Take the Relate operation (e.g., finds all objects left of a specific object x) as an example. TbD uses a stack of dilated convolutional layers to propagate the attention from object x to others. In TbD-Object, we replace the stack of 2D convolutions by several 1D convolution layers, operating over the k × d obj object features. This ignores the equivalence of objects (the order of objects should not affect the ). In contrast, MAC networks always use the attention mechanism to extract information from the image representation. This operation is invariant to the order of objects, but is not suitable for handling quantities (e.g., counting objects).As for TbD-Mask and MAC-Mask, although the mask-guided attention does not improve the overall performance, we have observed noticeably faster convergence during model training. TbD-Mask and MAC-Mask leverage the prior knowledge of object masks to facilitate the attention. Such prior has also been verified to be effective in the original TbD model: TbD employs an attention regularization during training, which encourages the model to attend to smaller regions. In general, NS-CL is more data-efficient than MAC networks and TbD. Recall that NS-CL answers questions by executing symbolic programs on the learned visual concepts. Only visual concepts (such as Red and Left) and the interpretation of questions (how to translate questions into executable programs) need to be learned from data. In contrast, both TbD and MAC networks need to additionally learn to execute (implicit or explicit) programs such as counting. For the experiments on the full CLEVR training set, we split 3,500 images (5% of the training data) as the hold-out validation set to tune the hyperparameters and select the best model. We then apply this model to the CLEVR validation split and report the testing performance. Our model reaches an accuracy of 99.2% using the CLEVR training set. We also extend the experiments to a new reasoning testbed: Minecraft worlds . The Minecraft reasoning dataset differs from CLEVR in both visual appearance and question types. FIG3 gives an example instance from the dataset. Besides different 3D visual appearance and image contexts, the Minecraft reasoning dataset introduces two new types of reasoning operations. We add them to our domain-specific language:1. FilterMost(ObjectSet, Concept) → ObjectSet: Given a set of objects, finds the "most" one. For example, FilterMost(Closest, set) locates the object in the input set that is cloest to the camera (e.g., what is the direction of the closest animal?) 2. BelongTo(Object, ObjectSet) → Bool: Query if the input object belongs to a set. Results. TAB11 summarizes the and FIG5 shows sample execution traces. We compare our method against the NS-VQA baseline , which uses strong supervision for both scene representation (e.g., object categories and positions) and program traces. In contrast, our method learns both by looking at images and reading question-answering pairs. NS-CL outperforms NS-VQA by 5% in overall accuracy. We attribute the inferior of NS-VQA to its derendering module. Because objects in the Minecraft world usually occlude with each other, the detected object bounding boxes are inevitably noisy. During the training of the derendering module, each detected bounding box is matched with one of the ground-truth bounding boxes and uses its class and pose as supervision. Poorly localized bounding boxes lead to noisy labels and hurt the accuracy of the derendering module. This further influences the overall performance of NS-VQA. We conduct experiments on the VQS dataset BID13. VQS is a subset of the VQA 1.0 dataset BID3 On the right, we show the original question and answer in natural language, as well as the latent program recovered by our parser. To answer this question, models are expected to attend to the man and his pen in the pocket. Setup. All models are trained on the first 63,509 images of the training set, and tested on the test split. For hyper-parameter tuning and model selection, the rest 5,000 images from the training set are used for validation. We use the multiple-choice setup for VQA: the models choose their most confident answer from 18 candidate answers for each question. To obtain the latent programs from natural languages, we use a pre-trained syntactic dependency parser BID2 ) for extracting programs and concepts that need to be learned. A sample question and the program obtained by our parser is shown in FIG4. The concept embeddings are initialized by the bag of words (BoW) over the GloVe word embeddings .Baselines. We compare our model against two representative baselines: MLP and MAC .MLP is a standard baseline for visual-question answering, which treats the multiple-choice task as a ranking problem. For a specific candidate answer, a multi-layer perceptron (MLP) model is used to encode a tuple of the image, the question, and the candidate answer. The MLP outputs a score for each tuple, and the answer to the question is the candidate with the highest score. We encode the image with a ResNet-34 pre-trained on ImageNet and use BoW over the GloVe word embeddings for the question and option encoding. We slightly modify the MAC network for the VQS dataset. For each candidate answer, we concatenate the question and the answer as the input to the model. The MAC model outputs a score from 0 to 1 and the answer to the question is the candidate with the highest score. The image features are extracted from the same ResNet-34 model. Results. TAB9 summarizes the . NS-CL achieves comparable with the MLP baseline and the MAC network designed for visual reasoning. Our model also brings transparent reasoning over natural images and language. Example execution traces generated by NS-CL are shown in FIG1. Besides, the symbolic reasoning process helps us to inspect the model and diagnose the error sources. See the caption for details. Another appealing benefit is that our reasoning model enjoys full interpretability. Figure 11, FIG5, and FIG1 show NS-CL's execution traces on CLEVR, Minecraft, and VQS, respectively. As a side product, our system detects ambiguous and invalid programs and throws out exceptions. As an example (Figure 11), the question "What's the color of the cylinder?" can be ambiguous if there are multiple cylinders or even invalid if there are no cylinders. Figure 14 and FIG6 include qualitative visualizations of the concepts learned from the CLEVR and Minecraft datasets, including object categories, attributes, and relations. We choose samples from the validation or test split of each dataset by generating queries of the corresponding concepts. We set a threshold to filter the returned images and objects. For quantitative evaluations of the learned concepts on the CLEVR dataset, please refer to Table 2 and Table 5 Figure 11: Visualization of the execution trace generated by our Neuro-Symbolic Concept Learner on the CLEVR dataset. Example A and B are successful executions that generate correct answers. In example C, the execution aborts at the first operator. To inspect the reason why the execution engine fails to find the corresponding object, we can read out the visual representation of the object, and locate the error source as the misclassification of the object material. Example D shows how our symbolic execution engine can detect invalid or ambiguous programs during the execution by performing sanity checks. FIG1: Illustrative execution trace generated by our Neuro-Symbolic Concept Learner on the VQS dataset. Execution traces A and B shown in the figure leads to the correct answer to the question. Our model effectively learns visual concepts from data. The symbolic reasoning process brings transparent execution trace and can easily handle quantities (e.g., object counting in Example A). In Example C, although NS-CL answers the question correctly, it locates the wrong object during reasoning: a dish instead of the cake. In Example D, our model misclassifies the sport as frisbee.' == 'We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them.' != 'We design incremental sequence-to-action parsers for text-to-SQL task and achieve SOTA results. We further improve by using non-deterministic oracles to allow multiple correct action sequences. '

Question: We study the statistical properties of the endpoint of stochastic gradient descent (SGD). We approximate SGD as a stochastic differential equation (SDE) and consider its Boltzmann Gibbs equilibrium distribution under the assumption of isotropic variance in loss gradients.. Through this analysis, we find that three factors – learning rate, batch size and the variance of the loss gradients – control the trade-off between the depth and width of the minima found by SGD, with wider minima favoured by a higher ratio of learning rate to batch size. In the equilibrium distribution only the ratio of learning rate to batch size appears, implying that it’s invariant under a simultaneous rescaling of each by the same amount. We experimentally show how learning rate and batch size affect SGD from two perspectives: the endpoint of SGD and the dynamics that lead up to it. For the endpoint, the experiments suggest the endpoint of SGD is similar under simultaneous rescaling of batch size and learning rate, and also that a higher ratio leads to flatter minima, both findings are consistent with our theoretical analysis. We note experimentally that the dynamics also seem to be similar under the same rescaling of learning rate and batch size, which we explore showing that one can exchange batch size and learning rate in a cyclical learning rate schedule. Next, we illustrate how noise affects memorization, showing that high noise levels lead to better generalization. Finally, we find experimentally that the similarity under simultaneous rescaling of learning rate and batch size breaks down if the learning rate gets too large or the batch size gets too small. Despite being massively over-parameterized BID13, deep neural networks (DNNs) have demonstrated good generalization ability and achieved state-of-the-art performances in many application domains such as image BID13 and speech recognition BID1. The reason for this success has been a focus of research recently but still remains an open question. Our work provides new theoretical insights and useful suggestions for deep learning practitioners. The standard way of training DNNs involves minimizing a loss function using SGD and its variants BID4. In SGD, parameters are updated by taking a small discrete step depending on the learning rate in the direction of the negative loss gradient, which is approximated based on a small subset of training examples (called a mini-batch). Since the loss functions of DNNs are highly non-convex functions of the parameters, with complex structure and potentially multiple minima and saddle points, SGD generally converges to different regions of parameter space depending on optimization hyper-parameters and initialization. Recently, several works BID2 BID0 BID28 have investigated how SGD impacts generalization in DNNs. It has been argued that wide minima tend to generalize better than sharp minima BID15 BID28. This is entirely compatible with a Bayesian viewpoint that emphasizes targeting the probability mass associated with a solution, rather than the density value at a solution BID21. Specifically, BID28 find that larger batch sizes correlate with sharper minima. In contrast, we find that it is the ratio of learning rate to batch size which is correlated with sharpness of minima, not just batch size alone. In this vein, while BID9 discuss the existence of sharp minima which behave similarly in terms of predictions compared with wide minima, we argue that SGD naturally tends to find wider minima at higher noise levels in gradients, and such wider minima seem to correlate with better generalization. In order to achieve our goal, we approximate SGD as a continuous stochastic differential equation BID3 BID22 BID19. Assuming isotropic gradient noise, we derive the Boltzmann-Gibbs equilibrium distribution of this stochastic process, and further derive the relative probability of landing in one local minima as compared to another in terms of their depth and width. Our main finding is that the ratio of learning rate to batch-size along with the gradient's covariances influence the trade-off between the depth and sharpness of the final minima found by SGD, with a high ratio of learning rate to batch size favouring flatter minima. In addition, our analysis provides a theoretical justification for the empirical observation that scaling the learning rate linearly with batch size (up to a limit) leads to identical performance in DNNs BID18 BID12.We verify our theoretical insights experimentally on different models and datasets. In particular, we demonstrate that high learning rate to batch size ratio (due to either high learning rate or low batchsize) leads to wider minima and correlates well with better validation performance. We also show that a high learning rate to batch size ratio helps prevent memorization. Furthermore, we observe that multiplying each of the learning rate and the batch size by the same scaling factor in similar training dynamics. Extending this observation, we validate experimentally that one can exchange learning rate and batch size for the recently proposed cyclic learning rate (CLR) schedule BID31, where the learning rate oscillates between two levels. Finally, we discuss the limitations of our theory in practice. The relationship between SGD and sampling a posterior distribution via stochastic Langevin methods has been the subject of discussion in a number of papers BID8 BID33; BID27 BID26. In particular, BID22 describe the dynamics of stochastic gradient descent (SGD) as a stochastic process that can be divided into three distinct phases. In the first phase, weights diffuse and move away from the initialization. In the second phase the gradient magnitude dominates the noise in the gradient estimate. In the final phase, the weights are near the optimum. BID29 ) make related observations from an information theoretic point of view and suggest the diffusion behaviour of the parameters in the last phase leads to the minimization of mutual information between the input and hidden representation. We also relate the SGD dynamics to the stationary distribution of the stochastic differential equation. Our derivation bears similarity with BID22. However, while BID22 study SGD as an approximate Bayesian inference method in the final phase of optimization in a locally convex setting, our end goal is to analyze the stationary distribution over the entire parameter space reached by SGD. Further, our analysis allows us to compare the probability of SGD ending up in one minima over another (in terms of width and depth), which is novel in our case. We discuss the Fokker-Planck equation which has appeared before in the machine learning literature though the exact form and solution we consider we believe is novel. For example, in the online setting BID14 ) derive a Gibbs distribution from the Fokker-Planck equation, but the relation there does not give the temperature of the Gibbs distribution in terms of the learning rate, batch size and gradient covariance. Our work is also closely related to the ongoing discussion about the role of large batch size and the sharpness of minima found in terms of generalization BID28. BID28 showed that SGD ends up in sharp minimum when using large batch size. BID12 BID16 empirically observed that scaling up the learning rate, and training for more epochs, leads to good generalization when using large batch size. Our novelty is in explaining the importance of the ratio of learning rate to batch size. In particular, our theoretical and empirical show that simultaneously rescaling the batch size and learning rate by the same amount leads SGD to minima having similar width despite using different batch sizes. Concurrent with this work, BID32 have both analyzed SGD approximated as a continuous time stochastic process and stressed the importance of the learning rate to batch size ratio. BID32 focused on the training dynamics while explored the stationary non-equilibrium solution for the stochastic differential equation for non-isotropic gradient noise, but assuming other conditions on the covariance and loss to enforce the stationary distribution to be path-independent. Their solution does not have an explicit solution in terms of the loss in this case. In contrast to other work, we strictly focus on the explicitly solvable case of the Boltzmann-Gibbs equilibrium distribution with isotropic noise. This focus allows us to relate the noise in SGD, controlled by the learning rate to batch size ratio, with the width of its endpoint. We empirically verify that the width and height of minima correlates with the learning rate to batch size ratio in practice. Our work continues the line of research on the importance of noise in SGD BID4; BID23 BID22. Our novelty is in formalizing the impact of batch size and learning rate (i.e. noise level) on the width and depth of the final minima, and empirical verifications of this. Our focus in this section is on finding the relative probability with which we end optimization in a region near a minimum characterized by a certain loss value and Hessian determinant. We will find that the relative probability depends on the local geometry of the loss function at each minimum along with batch size, learning rate and the covariance of the loss gradients. To reach this , we first derive the equilibrium distribution of SGD over the parameter space under a stochastic differential equation treatment. We make the assumption of isotropic covariance of the loss gradients, which allows us to write down an explicit closed-form analytic expression for the equilibrium distribution, which turns out to be a Boltzmann-Gibbs distribution. We follow a similar (though not identical) theoretical setup to BID22, approximating SGD with a continuous-time stochastic process, which we now outline. Let us consider a model parameterized by θ = {θ 1, . . ., θ q}. For N training examples x i, i ∈ {1, ..., N}, the loss function, L(θ), and the corresponding gradient g(θ), are defined based on the sum over the loss values for all training examples. Stochastic gradients g (S) (θ) arise when we consider a batch B of size S < N of random indices drawn uniformly from {1, ..., N} and form an (unbiased) estimate of loss and gradient based on the corresponding subset of training examples DISPLAYFORM0 We consider stochastic gradient descent (SGD) with learning rate η, as defined by the update rule DISPLAYFORM1 We now make the following assumptions: By the central limit theorem (CLT), we assume the noise in the stochastic gradient is Gaussian with covariance matrix DISPLAYFORM2 We note that the covariance is symmetric positive-semidefinite, and so can be decomposed into the product of two matrices C(θ) = B(θ)B (θ). We assume the discrete process of SGD can be approximated by the continuous time limit of the following stochastic differential equation (known as a Langevin equation) DISPLAYFORM3 where f (t) is a normalized Gaussian time-dependent stochastic term. Note that the continuous time approximation of SGD as a stochastic differential equation has been shown to hold in a weak approximation on the condition that the learning rate is small BID19.Note that we have not made Assumption 4 of BID22, where they assume the loss can be globally approximated by a quadratic. Instead, we allow for a general loss function, which can have many local minima. The Langevin equation is a stochastic differential equation, and we are interested in its equilibrium distribution which gives insights into the behavior of SGD and the properties of the points it converges to. Assuming isotropic noise, the Langevin equation is well known to have a GibbsBoltzmann distribution as its equilibrium distribution. This equilibrium distribution can be derived by finding the stationary solution of the Fokker-Planck equation, with detailed balance, which governs the evolution of the probability density of the value of the parameters with time. The FokkerPlanck equation and its derivation is standard in the statistical physics literature. In Appendix A we give the equation in the machine learning context in Eq. FORMULA12 and for completeness of presentation we also give its derivation. In Appendix C we restate the standard proofs of the stationary distribution of a Langevin system, and provide the ing Gibbs-Boltzmann equilbirium distribution here, using the notation of this paper: Theorem 1 (Equilibrium Distribution). Assume 1 that the gradient covariance is isotropic, i.e. C(θ) = σ 2 I, where σ 2 is a constant. Then the equilibrium distribution of the stochastic differential equation 1 is given by DISPLAYFORM0 where n ≡ η S and P 0 is a normalization constant, which is well defined for loss functions with L 2 regularization. Discussion: Here P (θ) defines the density over the parameter space. The above says that if we run SGD for long enough (under the assumptions made regarding the SGD sufficiently matching the infinitesimal limit), then the probability of the parameters being in a particular state asymptotically follows the above density. Note, that n ≡ η S is a measure of the noise in the system set by the choice of learning rate η and batch size S. The fact that the loss is divided by n emphasizes that the higher the noise n, the less granular the loss surface appears to SGD. The gradient variance C(θ) on the other hand is determined by the dataset and model priors (e.g. architecture, model parameterization, batch normalization etc.). This reveals an important area of investigation, i.e., how different architectures and model parameterizations affect the gradient's covariance structure. We note that in the analysis above, the assumption of the gradient covariance C(θ) being fixed and isotropic in the parameter space is unrealistic. However it is a simplification that enables straightforward insights regarding the relationship of the noise, batch size and learning rate in the Gibbs-Boltzmann equilibrium. We empirically show that various predictions based on this relationship hold in practice. Returning to SGD as an optimization method, we can ask, given the probability density P (θ) can we derive the probability of ending at a given minimum, θ A, which we will denote by lowercase p A =p A C, where C is a normalization constant which is the same for all minima (the unnormalized probabilityp A is all we are interested in when estimating the relative probability of finishing in a given minimum compared to another one). This probability is derived in Appendix D, and given in the following theorem, which is the core insight from our theory. Theorem 2 (Probability of ending in region near minima θ A). Assume the loss has a series of separated local minima. Consider one such minima, with Hessian H A and loss L A at a minimum θ A. Then the unnormalized probability of ending in a region near minima θ A is DISPLAYFORM1 where n = η S is the noise used in the SGD process to reach θ A. Discussion: For this analysis, we qualitatively categorize a minima θ A by its loss L A (depth) and the determinant of the Hessian det H A (a larger determinant implies a sharper minima). The above 1 Here we also assume a weak regularity condition that the loss L(θ) includes the regularization term τ θ 2 2for some τ > 0. shows that the probability of landing in a specific minimum depends on three factors -learning rate, batch-size and covariance of the gradients. The two factors that we directly control only appear in the ratio given by the noise n = η/S. Note that the proof of this utilizes a Laplace Approximation in which the loss near a given minimum can be approximated using a second order Taylor series in order to evaluate an integral. We emphasize this is not the same as globally treating the loss as a quadratic. To study which kind of minima are more likely if we were to reach equilibrium, it is instructive to consider the ratio of probabilities p A and p B at two distinct minima θ A and θ B respectively given by DISPLAYFORM2 To highlight that towards the equilibrium solution SGD favors wider rather than sharper minima, let's consider the special case when L A = L B, i.e., both minima have the same loss value. Then, DISPLAYFORM3 This case highlights that in equilibrium, SGD favors the minimum with lower determinant of the Hessian (i.e. the flatter minima) when all other factors are identical. On the flip side, it can be seen that if two minima have the same curvature (det H A = det H B), then SGD will favor the minima with lower loss. Finally in the general case when L A ≥ L B, it holds that p A ≥ p B if and only if DISPLAYFORM4 That is, there is an upper bound on the inverse of the noise for θ A to be favored in the case that its loss is higher than at θ B, and this upper bound depends on the difference in the heights compared to the ratio of the widths. In particular we can see that if det H B < det H A, then Y < 0, and so no amount of noise will in θ A being more probable than θ B. In words, if the minimum at θ A is both higher and sharper than the minimum at θ B, it is never reached with higher probability than θ B, regardless of the amount of noise. However, if det H B > det H A then Y > 0, and there is a lower bound on the noise DISPLAYFORM5 to make θ A more probable than θ B. In words, if the minimum at θ A is higher but flatter than the minimum at θ B, it is favored over θ B, as long as the noise is large enough, as defined by eq..To summarize, the presented theory shows that the noise level in SGD (which is defined by the ratio of learning rate to batch size) controls the extent to which optimization favors wider over deeper minima. Increasing the noise by increasing the ratio of learning rate to batch size increases the probability of wider compared to deeper minima. For a discussion on the relative probabilities of critical points that are not strictly minima, see appendix D. In this section, we empirically study the impact of learning rate η and batch size S on the local minimum that SGD finds. We first focus on a 4-layer Batch Normalized ReLU MLP trained on Fashion-MNIST . We study how the noise ratio n = η S leads to minima with different curvatures and validation accuracy. To measure the curvature at a minimum, we compute the norm of its Hessian (a higher norm implies higher sharpness of the minimum) using the finite difference method . In Figure 1a, we report the norm of the Hessian for local minima obtained by SGD for different n = set. As n grows, we observe that the norm of the Hessian at the minima also decreases, suggesting that higher To further illustrate the behavior of SGD with different noise levels, we train three Resnet56 models on CIFAR10 using SGD (without momentum) with different BID11 by investigating the loss on the line interpolating between the parameters of two models. More specifically, let θ 1 and θ 2 be the final parameters found by SGD using different Results indicate that models with large batch size (Fig. 2 -left) or low learning rate (Fig. 2-middle ; both having a lower η S than the baseline) end up in a sharper minimum relative to the baseline model. These plots are consistent with our theoretical analysis that higher n = η/S gives preference to wider minima over sharper minima. On the other hand, figure 2 (right) shows that models trained with roughly the same level of noise end up in minima of similar quality. The following experiment explores this aspect further. We train VGG-11 models BID30 on CIFAR-10, such that all the models are trained with the same noise level but with different values of learning rate and batch size. Specifically, we use DISPLAYFORM0 S=50×β, where we set β = 0.25, 1, 4. We then interpolate between the model parameters found when training with β = 1 and β = 4 (Fig. 3-left), and β = 1 and β = 0.25 S=50×β, but different η and S values determined by β. As predicted by our theory, the minima for models with identical noise levels should be qualitatively similar as can be seen by these plots.: Learning rate schedule can be replaced by an equivalent batch size schedule. The ratio of learning rate to batch size is equal at all times for both red and blue curves in each plot. Above plots show train and test accuracy for experiments involving VGG-11 architecture on CIFAR10 dataset. Left: cyclic batch size schedule (blue) in range 128 to 640, compared to cyclic learning rate schedule (red) in range 0.001 to 0.005. Right: constant batch size 128 and constant learning rate 0.001 (blue), compared to constant batch size 640 and constant learning rate 0.005 (red).(Fig. 3-right). The interpolation indicate that all the minima have similar width and depth, qualitatively supporting our theoretical observation that for the same noise ratio SGD ends up in minima of similar quality. In this section we look at two experimental phenomena: firstly, the equilibrium endpoint of SGD and secondly the dynamical evolution of SGD. The former, was theoretically analysed in the theory section, while the latter is not directly addressed in the theory section, but we note that the two are related -the endpoint is the of the intermediate dynamics. We experimentally study both phenomena in the following four experiments involving the VGG-11 architecture on the CIFAR10 dataset, shown in Regarding the first phenomena, of the endpoint of SGD, the test accuracy when training with a cyclic batch size and cyclic learning rate is 89.39% and 89.24%, respectively, and we emphasize that these are similar scores. For a constant learning rate to batch-size ratio of Figure 5: Impact of η S on memorization of MNIST when 25% and 50% of labels in the training set are replaced with random labels, using no momentum (on the right) or a momentum with parameter 0.9 (on the left). We observe that for a specific level of memorization, high η S leads to better generalization. Red has higher value of η S than blue. test accuracy is 87.25% and 86.92%, respectively, and we again emphasize that these two scores are similar to each other. That in each of these experiments the endpoint test accuracies are similar shows exchangability of learning rate for batch size for the endpoint, and is consistent with our theoretical calculation which says characteristics of the minima found at the endpoint are determined by the ratio of learning rate to batch-size, but not individually on learning rate or batch size. Additional exploring cyclical learning rate and batch size schedule are reported in Appendix F.4.Regarding the second phenomena of the dynamical evolution, we note the similarity of the training and test accuracy curves for each pair of same-noise curves in each experiment. Our theoretical analysis does not explain this phenomena, as it does not determine the dynamical distribution. Nonetheless, we report it here as an interesting observation, and point to Appendix B for some intuition on why this may occur from the Fokker-Planck equation. In Appendix F.2, FIG4 we show in more detail the loss curves. While the epoch-averaged loss curves match well when exchanging batch size for learning rate, the per-iteration loss is not invariant to switching batch size for learning rate. In particular, we note that each run with smaller batch-size has higher variance in per-iteration loss than it's same-noise pair. This is expected, since from one iteration to the next, the examples will have higher variance for a smaller batch-size. The take-away message from this section is that the endpoint and dynamics of SGD are approximately invariant if the batch size and learning rate are simultaneously rescaled by the same amount. This is in contrast to a commonly used heuristic consisting of scaling the learning rate with the square root of the batch size, i.e. of keeping the ratio η/ √ S constant. This is used for example by BID16 as a way of keeping the covariance matrix of the parameter update step the same for any batch size. However, our theory and experiments suggest changing learning rate and batch size in a way that keeps the ratio n = η/S constant instead, since this in the same equilibrium distribution. To generalize well, a model must identify the underlying pattern in the data instead of simply perfectly memorizing each training example. An empirical approach to test for memorization is to analyze how good a DNN can fit a training set when the true labels are partly replaced by random labels BID13 BID2. The experiments described in this section highlight that SGD with a sufficient amount of noise improves generalization at a given level of memorization. Experiments are performed on the MNIST dataset with an MLP similar to the one used by BID2, but with 256 hidden units. We train the MLP with different amounts of random labels in the training set. For each level of label noise, we evaluate the impact of In each experiment, we multiply the learning rate (η) and batch size (S) with β such that the ratio β×(η=0.1) β×(S=50) is fixed. We observe that for the same ratio, increasing the learning rate and batch size yields similar performance up to a certain β value, for which the performance drops significantly. (d)-Breaking point analysis when half the noise level β×(η=0.05) β×(S=50) is used. The breaking point happens for much larger β when using a smaller noise. All experiments are repeated 5 times with different random seeds. The graphs denote the mean validation accuracies and the numbers in the brackets denote the mean and standard deviation of the maximum validation accuracy across different runs. The * denotes at least one seed lead to divergence.highlights that SGD with low noise n = η S steers the endpoint of optimization towards a minimum with low generalization ability. While Fig 5 reports the generalization at the endpoint, we observe that SGD with larger noise continuously steers away from sharp solutions throughout the dynamics. We also reproduce the observation reported by BID2: that memorization roughly starts after reaching maximum generalization. For runs with momentum we exclude learning rates higher than 0.02 as they lead to divergence. Full learning curves are reported in FIG5 included in Appendix F.3. Our analysis relies on the assumption that the gradient step is sufficiently small to guarantee that the first order approximation of a Taylor's expansion is a good estimate of the loss function. In the case where the learning rate becomes too high, this approximation is no longer suitable, and the continuous limit of the discrete SGD update equation will no longer be valid. In this case, the stochastic differential equation doesn't hold, and hence neither does the Fokker-Planck equation, and so we don't expect our theory to be valid. In particular, we don't expect to arrive at the same stationary distribution as indicated by a fixed ratio η/S, if the learning rate gets too high. This is exemplified by the empirical reported in FIG7, where similar learning dynamics and final performance can be observed when simultaneously multiplying the learning rate and batch size by a factor β up to a certain limit. This is done for different training set sizes to investigate if the breaking point depends on this factor FIG7. The plots suggest that the breaking point happens for smaller β values if the dataset size is smaller. We also investigate the influence of β when half the noise level is used, due to halving the learning rate, in (FIG7 . These experiments strongly suggest that the reason behind breaking point is the use of a high learning rate because the performance drops at much higher β when the base learning rate is halved. A similar experiment is performed on Resnets (for see Fig 7 in the appendix). We highlight other limitations of our theory in appendix E. In the theoretical section of this work we treat the learning rate as fixed throughout training. However, in practical applications, the learning rate is annealed to a lower value, either gradually or in discrete jumps. When viewed within our framework, at the beginning with high noise, SGD favors width over depth of a region, then as the noise decreases, SGD prioritizes the depth more stronglythis can be seen from Theorem 3 and the comments that follow. In the theoretical section we made the additional assumption that the covariance of the gradients is isotropic, in order to be able to derive a closed form solution for the equilibrium distribution. We do not expect this assumption to hold in practice, but speculate that there may be mechanisms which drive the covariance towards isotropy, for example one may be able to tune learning rates on a per-parameter basis in such a way that the combination of learning rate and covariance matrix is approximately isotropic -this may lead to improvements in optimization. Perhaps some existing mechanisms such as batch normalization or careful initialization give rise to more equalized covariance -we leave study of this for future work. We note further that our theoretical analysis considered an equilibrium distribution, which was independent of the intermediate dynamics. However, this may not be the case in practice. Without the isotropic covariance, the system of partial differential equations in the late time limit will in general have a solution which will depend on the path through which optimization occurs, unless other restrictive assumptions are made to force this path dependence to disappear. Despite this simplifying assumption, our empirical are consistent with the developed theory. We leave study of path dependence and dynamics to future work. In experiments investigating memorization we explored how the noise level changes the preference of wide minima over sharp ones. BID2 argues that SGD first learns true labels, before focusing on random labels. Our insight is that in the second phase the high level of noise maintains generalization. This illustrates the trade-off between width of minima and depth in practice. When the noise level is lower, DNNs are more likely to fit random labels better, at the expense of generalizing less well on true ones. We shed light on the role of noise in SGD optimization of DNNs and argue that three factors (batch size, learning rate and gradient variance) strongly influence the properties (loss and width) of the final minima at which SGD converges. The learning rate and batch size of SGD can be viewed as one effective hyper-parameter acting as a noise factor n = η/S. This, together with the gradient covariance influences the trade-off between the loss and width of the final minima. Specifically, higher noise favors wider minima, which in turn correlates with better generalization. Further, we experimentally verify that the noise n = η/S determines the width and height of the minima towards which SGD converges. We also show the impact of this noise on the memorization phenomenon. We discuss the limitations of the theory in practice, exemplified by when the learning rate gets too large. We also experimentally verify that η and S can be simultaneously rescaled as long as the noise η/S remains the same. In this appendix we derive the Fokker-Planck equation. For any stochastic differential equation, since the evolution is noisy we can't say exactly where in parameter space the parameter values will be at any given time. But we can talk about the probability P (θ, t|θ 0, t 0) that a parameter takes a certain value θ at a certain time t, given that it started at θ 0, t 0. That is captured by the Fokker-Planck equation, which reads DISPLAYFORM0 In this appendix we will derive the above equation from the stochastic differential equation FORMULA13. We will not be interested in pure mathematical rigour, but intend the proof to add intuition for the machine learning audience. For brevity we will sometimes write the probability just as P (θ, t). We will sometimes make use of tensor index notation, where a tensor is denoted by its components, (for example θ i are the components of the vector θ), and we will use the index summation convention, where a repeated index is to be summed over. We start with the stochastic differential equation DISPLAYFORM1 A formal expression for the probability for a given noise function f is given by P (θ, t) = δ(θ − θ f), but since we don't know the noise, we instead average over all possible noise functions to get DISPLAYFORM2 While this is just a formal solution we will make use of it later in the derivation. We now consider a small step in time δt, working at first order in δt, and ask how far the parameters move, denoted δθ, which is given by integrating DISPLAYFORM3 where we've assumed that δθ is small enough that we can evaluate g at the original θ. We now look at expectations of δθ. Using the fact that the noise is normalized Gaussian, E(f (t)) = 0, we get (switching to index notation for clarity) DISPLAYFORM4 and using that the noise is normalized Gaussian again, we have that E(f (t)f (t)) = Iδ(t − t), leading to DISPLAYFORM5 If at time t we are at position θ and end up at position θ = θ + δθ at time t + δt, then we can take with θ f = θ and Taylor expand it in δθ i DISPLAYFORM6 We deal with the derivatives of the delta functions in the following way. We will use the following identity, called the Chapman-Kolmogorov equation, which reads DISPLAYFORM7 for any t, such that t 0 ≤ t ≤ t + δt. This identity is an integral version of the chain rule of probability, stating there are multiple paths of getting from an initial position θ 0 to θ at time t + δt and one should sum all of these paths. We will now substitute into the first term on the right hand side of FORMULA3 with t set to be t, and apply integration by parts (assuming vanishing boundary conditions at infinity) to move the derivatives off of the delta functions and onto the other terms. We end up with DISPLAYFORM8 We can then take the first term on the right hand side of FORMULA3 to the other side, insert FORMULA16 and FORMULA3, divide by δt and take the limit δt → 0, getting a partial derivative with respect to time on the left hand side, leading directly to the Fokker-Planck equation quoted in the text (where we have reverted back to vector notation from index notation for conciseness) DISPLAYFORM9 In this appendix we will give some supplementary comments about the intuition we can gain from the Fokker-Planck equation FORMULA12. If the learning rate, batch size are constant and the covariance is proportional to the identity C = σ 2 I and σ 2 is constant, then we can rewrite the Fokker-Planck equation in the following form DISPLAYFORM0 where we have rescaled the time coordinate to be t η ≡ tη. One can now see that in terms of this rescaled time coordinate, the ratio between the drift and diffusion terms is governed by the following ratio ησ DISPLAYFORM1 In terms of the balance between drift and diffusion, we see that a higher value of this ratio gives rise to a more diffusive evolution, while a lower value allows the potential drift term to dominate. In the next section we will see how this ratio controls the stationary distribution at which SGD converges to. For now we highlight that in terms of this rescaled coordinate, only this ratio controls the evolution towards this stationary distribution (not just its endpoint), in terms of the rescaled time t η. That is, learning rate and batch size are interchangable in the sense that the ratio is invariant under transformations S → aS, η → aη, for a > 0. But note that the time it takes to reach the stationary distribution depends on η as well, because of the rescaled time variable. For example, for a higher learning rate, but constant ratio, one arrives at the same stationary distribution, but in a quicker time by a factor of 1/η. However, a caution here is necessary. The first order SGD update equation only holds for small enough η that a first order approximation to a Taylor expansion is valid, and hence we expect the first order approximation of SGD as a continuous stochastic differential equation to break down for high η. Thus, we expect learning rate and batch size to be interchangable up to a maximum value of η at which the approximation breaks. In this appendix we will prove equation quoted in the main text, which claims that for an isotropic covariance, C(θ) = σ 2 I, with σ constant, the equilibrium solution of the Fokker-Planck equation has the following form DISPLAYFORM0 where n ≡ η S and P 0 is a normalization constant, which is well defined for loss functions with L 2 regularization. In order to prove this is the equilibrium distribution, we need to solve the Fokker-Planck equation FORMULA12 with the left hand side set equal to zero (which would give a stationary distribution) and further we require for equilibrium that detailed balance holds. To do this, we begin by writing the FokkerPlanck equation in a slightly different form, making use of a probability current J, defined DISPLAYFORM1 in which the Fokker-Planck equation becomes DISPLAYFORM2 At this point we use the assumption that C(θ) = σ 2 I to get DISPLAYFORM3 The stationary solution has ∂P (θ,t) ∂t = 0 and hence ∇ θ · J = 0. But we require the equilibrium solution, which is a stronger demand than just the stationary solution. The equilibrium solution is a particular stationary solution in which detailed balance occurs. Detailed balance means that in the stationary solution, each individual transition balances precisely with its time reverse, ing in zero probability currents, see §5.3.5 of BID10, i.e. that J = 0. Detailed balance is a sufficient condition for having entropy increasing with time. Non-zero J with ∇ θ · J = 0 would correspond to a non-equilibrium stationary distribution, which we don't consider here. For the equilibrium solution, J = 0, we get the for the probability distribution DISPLAYFORM4 which is the desired stationary solution we intended to find. Finally, to ensure that P 0 is a finite value, we note that the loss function L(θ) can be decomposed as, DISPLAYFORM5 where τ is some non-negative constant controlling L 2 regularization. Then we see that, DISPLAYFORM6 DISPLAYFORM7 Since L 0 (θ) ≥ 0 and DISPLAYFORM8 and we now note that the right hand side of FORMULA4 is finite because it is just a multidimensional Gaussian integral. Thus, P 0 has a finite value and hence the stationary distribution P (θ) is well defined. In this appendix we derive the discrete set of probabilities of ending at each minima, as given in. Essentially, we will use Laplace's method to approximate the integral of the probability density in the region near a minimum. This is a common approach used to approximate integrals, appearing for example in (a) and BID17. We work locally near θ A and take the following approximation of the loss function, since it is near a minimum, DISPLAYFORM0 where H A is the Hessian, and is positive definite for a minimum, and where the subscript R A indicates that this approximation only holds in the region R A near the minimum θ A. We emphasize this is not the same as Assumption 4 of BID22, where they assume the loss can be globally approximated by a quadratic. In contrast, we allow for a general loss, with multiple minima, and locally to each minima approximate the loss by its second-order Taylor expansion in order to evaluate the integral in the Laplace method. The distribution P A (θ) is a probability density, while we are interested in the discrete set of probabilities of ending at a given minimum, which we will denote by lowercase p A. To calculate this discrete set of probabilities, for each minimum we need to integrate this stationary distribution over an interval containing the minimum. Integrating FORMULA3 in some region R A around θ A and using DISPLAYFORM1 DISPLAYFORM2 where in the last line we assume the region is large enough that an approximation to the full Gaussian integral can be used (i.e. that the tails don't contribute, which is fine as long as the region is sufficiently larger than det H A) -note that the region can't be too large, otherwise we would invalidate our local assumption. The picture is that the minima are sufficiently far apart that the region can be taken sufficiently large for this approximation to be valid. Note thatP 0 is different to P 0 and includes the normalization factors from performing the Gaussian integral, which are the same for all minima. Since we are interested in relative probabilities between different minima, so we can consider the unnormalized probability, dropping factors that are common amongst all minima we get DISPLAYFORM3 This is the required expression given in the main text. We note that the derivation used above talks about strict minima, i.e., minima with a positive definite Hessian. In practice however, for deep neural networks with a large number of parameters, it is unrealistic to expect the endpoint of training to be in a strict minimum. Instead, it is more likely to be a point at which the Hessian has positive eigenvalues in a few directions, while the other eigenvalues are (approximately) zero. In such cases, to understand which minima SGD favors, we can consider the fact that at equilibrium, the iterate of SGD follows the distribution, DISPLAYFORM4 By definition this means that at any time during equilibrium, the iterate is more likely to be found in a region of higher probability. In the restrictive case of strict minimum, we model it as a Gaussian and characterize the probability of landing in a minimum depending on the curvature and depth of the loss around that minimum. In the general case of minima with degenerate (flat) directions, we can say that a minimum with more volume is a more probable one. Figure 7: Experiments involving Resnet56 architecture on CIFAR10 dataset. In each curve, we multiply the η S ratio by a given factor (increasing both batch size and learning rate). We observe that multiplying the ratio by a factor up to 5 in similar performances. However, the performance degrades for factor superior to 5. Due to our assumptions, we expect the theory to become unreliable when the discrete to continuous approximation fails, when the covariance in gradients is non-isotropic, when the batch size becomes comparable to the finite size of the training set and when momentum is considered. We discussed the limits of the discrete to continous approximation in 4.4 (further illustrated in 7).When the covariance in gradients is highly non-isotropic, the equilibrium solution to the FokkerPlanck equation is the solution to a complicated partial differential equation, and one can't easily spot the solution by inspection as we do in Appendix C. We expect this approximation to break down especially in the case of complicated architectures where different gradient directions will be have very different gradient covariances. Our theory does not involve the finite size of the training set, which is a drawback of the theory. This may be especially apparent when the batch size becomes large, compared to the training set size, and we expect the theory to break down at this point. Finally, we mention that momentum is used in practical deep learning optimization algorithms. Our theory does not consider momentum, which is a drawback of the theory and we expect it to break down in models in which momentum is important. We can write a Langevin equation for the case of momentum, with momentum damping coefficient µ, DISPLAYFORM0 where the velocity is v = dθ/dt. where P is now a function of velocity v as well as θ and t, and we've again assumed the gradient covariance varies slowly compared to the loss. From this more complicated Fokker-Planck equation, it is hard to spot the equilibrium distribution. We leave the study of this for further work. For now, we just note that a factor of η can be taken from the right hand side, and then the ratio between the diffusion and drift terms will again be the ratio ησ 2 /(2S). In this appendix we look at other experiments exploring the correlation between learning rate to batch-size ratio and sharpness of minima, and to validation performance. In FIG9, we report the validation accuracy for Resnet56 models trained on CIFAR10 with different learning rate to batchsize ratio. We notice there is a peak validation accuracy at a learning rate to batch-size ratio of around 2 × 10 −3. In particular, this example emphasizes that higher learning rate to batch-size ratio doesn't necessarily lead to higher validation accuracy. Instead, it acts as a control on the validation accuracy, and there is an optimal learning rate to batch-size ratio. FIG10 show the of a variant of 4 layer ReLU experiment, where we use 20 layers and remove Batch Normalization. The inspiration is to test predictions of theory in the more challenging setup. We again observe a correlation of Hessian norm with learning rate to batch-size ratio (though weaker), and similarly between validation performance and learning rate to batch-size ratio. Figures 10,11, 12 report additional line interpolation plots between models that uses different learning rate to batch size ratio but similar learning rate decay throught training. We repeat the experiment several time to ensure robustness with respect to the model random initializations. Results indicate that models with large batch size or low learning rate end up in a sharper minimum relative to the baseline model. In this appendix we show in more detail the experiments of Section 4.2 which show the exchangeability of learning rate and batch size. In FIG4 we show the log cross entropy loss, the epochaveraged log cross entropy loss, train, validation and test accuracy as well as the batch size schedule, learning rate schedule. We see in the learning rate to batch-size ratio plot that the orange and blue Table 1: Comparison between different cyclical training schedules (cycle length and learning rate are optimized using a grid search). Discrete schedules perform similarly, or slightly better than triangular. Additionally, discrete S schedule leads to much wider minima for similar loss. Hessian norm is approximated on 1 out of 6 repetitions and measured at minimum value (usually endpoint of training).lines (cyclic schedules) have the same ratio of learning rate to batch-size as each other throughout, and that their dynamics are very similar to each other. The same holds for the green and red lines, of constant batch size and learning rate. This supports the theoretical of this paper, that the ratio S/η governs the stationary distribution of SGD.We note that it is not just in the stationary distribution that the exchangeability holds in these plots -it appears throughout training, highlighted especially in the cyclic schedules. We postulate that due to the scaling relation in the Fokker-Planck equation, the exchangeability holds throughout learning as well as just at the end, as long as the learning rate does not get so high as to ruin the approximations under which the Fokker-Planck equation holds. We note similar behaviour occurs also for standard learning rate annealing schedules, which we omit here for brevity. We report learning curves from memorization experiment with 0.0 momentum, see Fig. 14. We additionally confirm similar to the previous experiments and show correlation between batch size and learning rate ratio and norm of Hessian, see Fig. 15 without momentum (left) and with momentum 0.9 (right). It has been observed that a cyclic learning rate (CLR) schedule leads to better generalization BID31. In Sec. 4.2 we demonstrated that one can exchange cyclic learning rate schedule (CLR) with cyclic batch size (CBS) and approximately preserve the practical benefit of CLR. This exchangeability shows that the generalization benefit of CLR must come from the varying noise level, rather than just from cycling the learning rate. To explore why this helps generalization, we run VGG-11 on CIFAR10 using 4 training schedules: we compared two discrete schedules (where either η or S switches discretely from one value to another between epochs) and two baseline schedules, one constant (η is constant) and one triangle (η is interpolated linearly between its maximum and minimum value). We track the norm of the Hessian and the training loss throughout training. Each experiment is repeated six times. For each schedule we optimize η in [1e − 3, 5e − 2] and cycle length in {5, 10, 15} on a validation set. In all cyclical schedules the maximum value (of η or S) is 5× larger than the minimum value. First, we observe that cyclical schemes oscillate between sharp and wide regions of the parameter space, see FIG7. Next, we empirically demonstrate that a discrete schedule varying either S or η performs similarly, or slightly better than triangular CLR schedule, see Tab. 1. Finally, we observe that cyclical schemes reach wider minima at the same loss value, see FIG7 and Tab. 1. All of the above suggest that by changing noise levels cyclical schemes reach different endpoints than constant learning rate schemes with same final noise level. We leave the exploration of the implications and a more thorough comparison with other learning schedules for future work. 8.0E+041.0E+05 Train loss FIG7: Cyclical schemes oscillate between sharp and wide regions. Additionally, cyclical schemes find wider minima than baseline run for same level of loss, which might explain their better generalization. All cyclical schedules use base η = 0.005 and cycle length 15 epochs, which approximate convergence at the end of each cycle. Plots from left to right: discrete S, discrete η, triangle η, constant learning rate η = 0.001, constant learning rate η = 0.005. On vertical axis we report loss (red) and approximated norm of Hessian (blue).

Represent the TLDR for finding the corresponding scientific article

Three factors (batch size, learning rate, gradient noise) change in predictable way the properties (e.g. sharpness) of minima found by SGD.

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.