Question:
represent the text to find a very short summary (1-2 sentences max)

E.g.
'People ask questions that are far richer, more informative, and more creative than current AI systems. We propose a neural program generation framework for modeling human question asking, which represents questions as formal programs and generates programs with an encoder-decoder based deep neural network. From extensive experiments using an information-search game, we show that our method can ask optimal questions in synthetic settings, and predict which questions humans are likely to ask in unconstrained settings. We also propose a novel grammar-based question generation framework trained with reinforcement learning, which is able to generate creative questions without supervised data. People can ask rich, creative questions to learn efficiently about their environment. Question asking is central to human learning yet it is a tremendous challenge for computational models. There is always an infinite set of possible questions that one can ask, leading to challenges both in representing the space of questions and in searching for the right question to ask. Machine learning has been used to address aspects of this challenge. Traditional methods have used heuristic rules designed by humans , which are usually restricted to a specific domain. Recently, neural network approaches have also been proposed, including retrieval methods which select the best question from past experience and encoder-decoder frameworks which map visual or linguistic inputs to questions (; ; ;). While effective in some settings, these approaches do not consider settings where the questions are asked about partially unobservable states. Besides, these methods are heavily data-driven, limiting the diversity of generated questions and requiring large training sets for different goals and contexts. There is still a large gap between how people and machines ask questions. Recent work has aimed to narrow this gap by taking inspiration from cognitive science. For instance, incorporates aspects of "theory of mind" in question asking by simulating potential answers to the questions, but the approach relies on imperfect agents for natural language understanding which may lead to error propagation. Related to our approach, proposed a powerful question-asking framework by modeling questions as symbolic programs, but their algorithm relies on hand-designed program features and requires expensive calculations to ask questions. We use "neural program generation" to bridge symbolic program generation and deep neural networks, bringing together some of the best qualities of both approaches. Symbolic programs provide a compositional "language of thought" for creatively synthesizing which questions to ask, allowing the model to construct new ideas based on familiar building blocks. Compared to natural language, programs are precise in their semantics, have clearer internal structure, and require a much smaller vocabulary, making them an attractive representation for question answering systems as well (; ;). However, there has been much less work using program synthesis for question asking, which requires searching through infinitely many questions (where many questions may be informative) rather than producing a single correct answer to a question. Deep neural networks allow for rapid question-synthesis using encoder-decoder modeling, eliminating the need for the expensive symbolic search and feature evaluations in. Together, the questions can be synthesized quickly and evaluated formally for quality groundtruth board partly revealed board example questions How long is the red ship? (size Red) Is purple ship horizontal? (== (orient Purple) H) Do all three ships have the same size? (=== (map (λ x (size x)) (set AllShips))) Figure 1: The Battleship task. Blue, red, and purple tiles are ships, dark gray tiles are water, and light gray tiles are hidden. The agent can see a partly revealed board, and should ask a question to seek information about the hidden board. Example questions and translated programs are shown on the right. We recommend viewing the figures in color. (e.g. the expected information gain), which as we show can be used to train question asking systems using reinforcement learning. In this paper, we develop a neural program generation model for asking questions in an informationsearch game similar to "Battleship" used in previous work (; ; . The model uses a convolutional encoder to represent the game state, and a Transformer decoder for generating questions. Building on the work of , the model uses a grammar-enhanced question asking framework, such that questions as programs are formed through derivation using a context free grammar. Importantly, we show that the model can be trained from human demonstrations of good questions using supervised learning, along with a data augmentation procedure that leverages previous work to produce additional human-like questions for training. Our model can also be trained without such demonstrations using reinforcement learning. We evaluate the model on several aspects of human question asking, including reasoning about optimal questions in synthetic scenarios, density estimation based on free-form question asking, and creative generation of genuinely new questions. To summarize, our paper makes three main contributions: 1) We propose a neural network for modeling human question-asking behavior, 2) We propose a novel reinforcement learning framework for generating creative human-like questions by exploiting the power of programs, and 3) We evaluate different properties of our methods extensively through three different experiments. Question generation has attracted attention from the machine learning community. Early research mostly explored rule-based methods which strongly depend on human-designed rules . Recent methods for question generation adopt deep neural networks, especially using the encoder-decoder framework, and can generate questions without handcrafted rules. These methods are mostly data-driven, which use pattern recognition to map inputs to questions. Researchers have worked on generating questions from different types of inputs such as knowledge base facts , pictures , and text for reading comprehension . However aspects of human question-asking remain beyond reach, including the goal-directed and flexible qualities that people demonstrate when asking new questions. This issue is partly addressed by some recent papers which draw inspiration from cognitive science. Research from and generate questions by sampling from a candidate set based on goal-oriented metrics. This paper extends the work of to overcome the limitation of the candidate set, and generate creative, goal-oriented programs with neural networks. Our work also builds on neural network approaches to program synthesis, which have been applied to many different domains (; ;). Those approaches often draw inspiration from computer architecture, using neural networks to simulate stacks, memory, and controllers in differentiable form . Other models incorporate Deep Reinforcement Learning (DRL) to optimize the generated programs in a goal oriented environment, such as generating SQL queries which can correctly perform a specific database processing task , translating strings in Microsoft Excel sheets , understanding and constructing 3D scenes and objects . Recent work has also proposed ways to incorporate explicit grammar information into the program synthesis process. design a special module to capture the grammar information as a prior, which can be used during generation. Some recent papers (; Figure (a) shows the network architecture. The board is represented as a grid of one-shot vectors and is embedded with a convolutional neural network. The board embedding and a sequence of symbols are inputted to a Transformer decoder to generate output vectors (details in section 4). PE means positional embeddings, and WE means word embeddings. (b) shows the derivation steps for program "(> (size Blue) 3)" using CFG. Non-terminals are shown as bold-faced, and terminals are shown in italic. The production rules used are shown next to each arrow. ) encode grammar with neural networks and use DRL to explicitly encourage the generation of semantically correct programs. Our work differs from these in two aspects. First, our goal is to generate informative human-like questions in the new domain instead of simply correct programs. Second, we more deeply integrate grammar information in our framework, which directly generates programs based on the grammar. In this paper, we work with a task used in previous work for studying human information search as well as question asking . The task is based on an information search game called "Battleship", in which a player aims to resolve the hidden layout of the game board based on the revealed information (Figure 1). There are three ships with different colors (blue, red, and purple) placed on a game board consisting of 6 × 6 grid of tiles. Each ship can be either horizontal or vertical, and takes 2, 3 or 4 tiles long. All tiles are initially turned over (light grey in Figure 1), and the player can flip one tile at a time to reveal an underlying color (either a ship color, or dark grey for water). The goal of the player is to determine the configuration of the ships (positions, sizes, orientations) in the least number of flips. In the modified version of this task studied in previous work (;, the player is presented with a partly revealed game board, and is required to ask a natural language question to gain information about the underlying configuration. As shown in Figure 1, the player can only see the partly revealed board, and might ask questions such as "How long is the red ship?" In this paper, we present this task to our computational models, and ask the models to generate questions about the game board. designed a powerful context free grammar (CFG) to describe the questions in the Battleship domain. The grammar represents questions in a LISP program-like format, which consists of a set of primitives (like numbers, colors, etc.) and a set of functions over primitives (like arithmetic operators, comparison operators, and other functions related to the game board). Another research by shows that it captures the full range of questions that people asked in an extensive dataset, mainly because the majority of this grammar is general functions which make it flexible enough. The grammar is able to generate an infinite set of other possible questions beyond collected human questions, capturing key notions of compositionality and computability. Figure 1 provides some examples of produced programs. The full grammar is provided in Appendix C. The neural network we use includes a Convolutional Neural Network (CNN) for encoding the input board, and a Transformer decoder for estimating the symbol distribution or selecting actions in different settings described below. The input x ∈ {0, 1} 6×6×5 is a binary representation of the 6x6 game board with five channels, one for each color to be encoded as a one-hot vector in each grid location. A simple CNN maps the input x to the encoder output e ∈ R 6×6×M, where M is the length of encoded vectors. Then a Transformer decoder takes e and a sequence of length L as input, and outputs a sequence of vectors where N o is the output size. As shown later, the input sequence and output vectors can be interpreted differently in different settings. The model is shown in Figure 2 (a), and details are described in Appendix A. Our model is compatible with both supervised and reinforcement training. Supervised training. In the supervised setting, the goal is to model the distribution of questions present in the training set. Each output y i ∈ R No is a symbol at position i in the program, where N o is the number of different symbols in the grammar. The model is trained with symbol-level cross entropy loss, and can be used to calculate the log-likelihood of a given sequence, or to generate a question symbol-by-symbol from left to right. Generation works as follows. Suppose at step t, a sequence of length t along with the encoded board is presented to the decoder. The model predicts the vector y t which represents the probability of each symbol to be chosen as next. Then we sample a symbol at location t + 1 and execute the decoder again with the new sequence, until an <eos> symbol is generated or maximum length is reached. Sequence-based RL. The framework can be adapted to generate a sequence of symbols without stepwise supervision, such that reward is provided only after the entire question is generated. Grammar-enhanced RL. Finally, the framework can be used with a novel grammar-enhanced RL training procedure. Figure 2 (b) illustrates the process of generating a program from the context-free grammar specified in. Beginning from the start symbol "A", at each step a production rule is chosen and applied to one of the non-terminals in the current string. The choice of rule is modeled as a Markov Decision Process, and we solve it with DRL. Each state is a partially derived string passed to the decoder, and we use the first output y 1 ∈ R No to represent the probability of selecting each production rule from all possible N o rules. After the rule is applied, the new string is passed back into the decoder, repeating until only terminals are contained in the sequence. We adopt the leftmost derivation here to avoid the ambiguity of parsing order, so at each step the left-most non-terminal will be replaced. In the first experiment, we designed three tasks to evaluate whether the model can learn simple compositional rules and reasoning strategies. These tasks include counting the number of visible ship tiles, locating a missing ship tile, and generalizing both strategies to unseen scenario types. Figure 3 illustrates the three tasks we designed in this experiment by providing examples of each task. Counting task. Models must select the ship color with the least number of visible tiles on the board. Each board has a unique answer, and models respond by generating a program "(topleft (coloredTiles X))" where X is a ship color. 4000 examples are used for training, and another 1000 examples are used for testing. Missing tile task. Models must select the ship that is missing a tile and identify which tile is missing. All ships are completely revealed except one, which is missing exactly one tile. Models respond by generating "(== (color Y) X)" where X is a color and Y is a location on the board. The number of training and test examples are the same as the counting task. Compositionality task. Models must combine both of the above strategies to find the missing tile of the ship with the least visible tiles. Outputs are produced as "(Z (coloredTiles X))" where X is a color and Z is either topleft or bottomright. Each board has a unique answer. This task further evaluates compositionality by withholding question types from training. With three values for X and two for Z, there are six possible question types and one is picked as the "held out" type. The other five "known" question types have 800 training examples. For the held out question type, the number of training examples is varied from 0 to 800, to test how much data is needed for generalization. Another 200 new boards of each question type is used for evaluation. More information about the model hyperparameters and training procedures are provided in Appendix B.1. We train our model in a fully supervised fashion. Accuracy for the counting and missing tile tasks is shown in Figure 3. The full neural program generation model shows strong reasoning abilities, achieving high accuracy for both the counting and missing tile tasks, respectively. We also perform ablation analysis of the encoder filters of the model, and provide the in Appendix D. The for the compositionality task are summarized in Table 1. When no training data regarding the held out question type is provided, the model cannot generalize to situations systematically different from training data, exactly as pointed out in previous work on the compositional skills of encoder-decoder models . However, when the number of additional training data increases, the model quickly incorporates the new question type while maintaining high accuracy on the familiar question tasks. On the last row of Table 1, we compare our model with another version where the decoder is replaced by two linear transformation operations which directly classify the ship type and location (details in Appendix B.1). This model has 33.0% transfer accuracy on compositional scenarios never seen during training. This suggests that the model has the potential to generalize to unseen scenarios if the task can be decomposed to subtasks and combined together. In this experiment, we examine if the neural network has the capability of capturing the distribution of human questions as a conditioned language model. To train the model, we need to construct a training set of many paired game boards and questions. Instead of laboriously collecting a large number of real human questions, and translating them into programs by hand, we construct the dataset by sampling from a previous computational model of human question asking . More precisely, we randomly generate a large number of game boards and sample K questions given each board. For generating the boards, we first uniformly sample the configuration of three ships, and randomly cover arbitrary number of tiles, with the restriction that at least one ship tile is observable. Next we randomly sample K programs for each board with importance sampling based on the cognitive model proposed by , which models the probability of a question under a given context as where ε(·) is a parameterized energy function for estimating the likelihood of a question being asked by human, which considers multiple features such as question informativeness, complexity, answer type, etc. Z is a normalization constant. We also randomly generate a larger set of questions to pretrain the decoder component of the model as a "language model" over questions, enabling it to better capture the grammatical structure of possible questions. Details regarding the model hyperparameters, training procedure, and pre-training procedure are provided in Appendix B.2. We evaluate the log-likelihood of reference questions generated by our full model as well as some lesioned variants of the full model, including a model without pretraining, a model with the Transformer decoder replaced by an LSTM decoder, a model with the convolutional encoder replaced by a simple MLP encoder, and a model that only has a decoder (unconditional language model). Though the method from also works on this task, here we cannot compare with their method for two reasons. One is that our dataset is constructed using their method, so the likelihood of their method should be an upper bound in our evaluation setting. Additionally, they can only approximate the log-likelihood due to an intractable normalizing constant, and thus it difficult to directly compare with our methods. Two different evaluation sets are used, one is sampled from the same process on new boards, the other is a small set of questions collected from human annotators. In order to calculate the log-likelihood of human questions, we use translated versions of these questions that were used in previous work , and filtered some human questions that score poorly according to the generative model used for training the neural network (Appendix B.2). A summary of the is shown in Table 2a. The full model performs best on both datasets, suggesting that pretraining, the Transformer decoder, and the convolutional encoder are all important components of the approach. However, we find that the model without an encoder performs reasonably well too, even out-performing the full model with a LSTM-decoder on the human-produced questions. This suggests that while contextual information from the board leads to improvements, it is not the most important factor for predicting human questions. To further investigate the role of contextual information and whether or not the model can utilize it effectively, we conduct another analysis. Intuitively, if there is little uncertainty about the locations of the ships, observing the board is critical since there are fewer good questions to ask. To examine this factor, we divide the scenarios based on the entropy of the hypothesis space of possible ship locations into a low entropy set (bottom 30%), medium entropy set (40% in the middle), and high entropy set (top 30%). We evaluate different models on the split sets of sampled data and report the in Table 2b. When the entropy is high, it is easier to ask a generally good question like "how long is the red ship" without information of the board, so the importance of the encoder is reduced. If entropy is low, the models with access to the board has substantially higher log-likelihood than the model without encoder. Also, the first experiment (section 5.1) would be impossible without an encoder. Together, this implies that our model can capture important context-sensitive characteristics of how people ask questions. In this experiment, we evaluate our reinforcement learning framework proposed in Section 4 on its ability of generating novel questions from scratch, without providing a large training set. The reward for training the reinforcement agent is calculated based on the energy value of the generated question q. We transform the energy value to a proper range for reward by −ε(q)/10 and clamp it between −1 and 1. The model is optimized with the REINFORCE algorithm . A baseline for REINFORCE is established simply as the average of the rewards in a batch. In order to produce higher-quality questions, we manually tune the information-related parameter of the energy function from to make it more information-seeking in this experiment. This process is described in Appendix B.2. We compare the models on their ability to generate diverse questions with high expected information gain (EIG), which is defined as the expected reduction in entropy, averaged across all possible answers to a question x. where I(·) is the Shannon entropy. The terms p(h) and p(h|d; x) are the prior and posterior distribution of a possible ship configuration h given question x and answer d ∈ A x. We compare our program-based framework with a simple text-based model, which has the same architecture but is trained with supervision on the text-based question dataset collected by . We also compare with the supervised program-based model from the last experiment. Finally, we implement a sequence-based reinforcement learning agent that specifies the program without direct access to the grammatical rules. For this alternative RL agent, we find it necessary to pretrain for 500 epochs with stepwise supervision. The models are evaluated on 2000 randomly sampled boards, and the are shown in Table 3. Note that any ungrammatical questions are excluded when we calculate the number of unique questions. First, when the text-based model is evaluated on new contexts, 96.3% of the questions it generates were included in the training data. We also find that the average EIG and the ratio of EIG>0 is worse than the supervised model trained on programs. Some of these deficiencies are due to the very limited text-based training data, but using programs instead can help overcome these limitations. With the program-based framework, we can sample new boards and questions to create a much larger dataset with executable program representations. This self-supervised training helps to boost performance, especially when combined with grammar-enhanced RL. From the table, the grammar-enhanced RL model is able to generate informative and creative questions. It can be trained from scratch without examples of human questions, and produces many novel questions with high EIG. In contrast, the supervised model rarely produces new questions beyond the training set. The sequence-level RL model is also comparatively weak at generating novel questions, perhaps because it is also pre-trained on human questions. It also more frequently generates ungrammatical questions. We also provide examples in Figure 4 to show the diversity of questions generated by the grammar enhanced model, and more in the supplementary materials. Figure 4a shows novel questions the model produces, which includes clever questions such as " Where is the bottom right of all the purple and blue tiles?" or "What is the size of the blue ship minus the purple ship?", while it can also sometimes generates meaningless questions such as "Is the blue ship shorter than itself?" Additional examples of generated questions are provided in Appendix B. Is any ship two tiles long? (> (++ (map (lambda x (== (size x) 2)) (set AllShips))) 0) Are there any ships in row 1? (> (++ (map (lambda y (and (== (rowL y) 1) (not (== (color y) Water)))) (set AllTiles))) 0) Is part of a ship on tile 4-6? (not (== (color 4-6) Water)) What is the size of the blue ship? (setSize (coloredTiles Blue)) What is the size of the purple ship? (size Purple) Which column is the first part of the blue ship? (colL (topleft (coloredTiles Blue))) What is the orientation of the blue ship? With the grammar enhanced framework, we can also guide the model to ask different types of questions, consistent with the goal-directed nature and flexibility of human question asking. The model can be queried for certain types of questions by providing different start conditions to the model. Instead of starting derivation from the start symbol "A", we can start derivation from a intermediate state such as "B" for Boolean questions or a more complicated "(and B B)" for composition of two Boolean questions. In Figure 4b, we show examples where the model is asked to generate four specific types of questions: true/false questions, number questions, location-related questions, and compositional true/false questions. We see that the model can flexibly adapt to new constraints and generate meaningful questions. In Figure 4c, we compare the model generated questions with human questions, each randomlysampled from the model outputs and the human dataset. These examples again demonstrate that our model is able to generate clever and human-like questions. However, we also find that people sometimes generate questions with quantifiers such as "any" and "all", which are operationalized in program form with lambda functions. These questions are complicated in representation and not favored by our model, showing a current limitation in our model's capacity. We introduce a neural program generation framework for question asking task under partially unobservable settings, which is able to generate creative human-like questions with human question demonstrations by supervised learning or without demonstrations by grammar-enhanced reinforcement learning. Programs provide models with a "machine language of thought" for compositional thinking, and neural networks provide an efficient means of question generation. We demonstrate the effectiveness of our method in extensive experiments covering a range of human question asking abilities. The current model has important limitations. It cannot generalize to systematically different scenarios, and it sometimes generates meaningless questions. We plan to further explore the model's compositional abilities in future work. Another promising direction is to model question asking and question answering jointly within one framework, which could guide the model to a richer sense of the question semantics. Besides, allowing the agent to iteratively ask questions and try to win the game is another interesting future direction. We would also like to use our framework in dialog systems and open-ended question asking scenarios, allowing such systems to synthesize informative and creative questions. Encoder. A simple CNN with one layer of filters is used to encode the board. Intuitively, many questions are related to specific positions, thus the position information should be recoverable from the encoding. On the other hand, some features of the board are translation-invariant, such as whether a ship is blocked by another ship. In order to capture the position-sensitive information as well as the translation-invariant patterns, three convolution operations with different filter sizes (1 × 1, 3 × 3, and 5 × 5) are performed in parallel on the same input. The inputs are padded accordingly to make sure the feature maps have the same width and height. Then three feature maps are concatenated together along the dimension of output channels, and passed through a linear projection. Formally, the outputs of the convolutions c can be obtained by where Conv k denotes a convolution operation on a k × k filter, ReLU(·) means applying a ReLU activation, and [A; B] means the concatenation of matrices A and B. Then c ∈ R 6×6×3Cout is projected to the encoder output e ∈ R 6×6×M by matrix W e o ∈ R 3Cout,M, where C out is the number of out channels of each convolution, and M is the length of encoded vectors. Decoder. We use the decoder from the Transformer model . With an input sequence of length L, the decoder computes the hidden states through several stacked Decoder Attention Layers. Each layer is composed by three sub-layers, a self-attention module, an attention over the encoded board, and a fully connected feed-forward network. Residual connections are employed around each sub-layer, followed by a layer normalization . After N layers of attention modules, a final output layer transforms the hidden states to the output vectors y i ∈ R No at every position i from 1 to L, where N o is the output size. Given the output from encoder e, and the hidden representation h n−1 from Decoder Attention Layer n − 1, each layer computes the hidden representation as where LN(·) means layer normalization , FC(·) is a fully connected layer, ATT(·) and Self-ATT(·) are multi-head attention mechanisms, which computes the attention over the output of encoder e, and the attention over the input h n−1 itself, respectively. They are defined as follows Multi-ATT(·) is the multi-head attention mechanism described in the paper by , which is a concatenation of multiple standard attention mechanisms with inputs projected using different matrices. A multi-head attention with n heads is defined as where is the scaled dot-product attention operation. Q, K, V are a set of vectors called queries, keys, and values, respectively, and d k is the dimension of queries and keys. After N layers, we apply a linear projection and a softmax activation to h N to get the output vectors y 1,..., y L. In this experiment, we use C out = 10, L = 50 for the model encoder. Each word is embedded with 50 dimension vectors in the decoder. The decoder has 2 layers, each multi-head attention module has 4 heads, and The model is trained for 500 epochs using Adam optimizer with initial learning rate at 0.001 and a batch size is set as 32. To further examine the model's ability on compositionality task, we evaluate another version of the model which replaces the decoder with two linear transformations to directly predict y l ∈{topleft, bottomright}, and y c ∈{Blue, Red, Color}. With the hidden representation of the encoder c in equation 1, y l and y c are calculated as where W l ∈ R |c|×2, W c ∈ R |c|×3, andc is the flattened vector of c. In this experiment, the model encoder has the same hyper-parameters as in the first experiment. We increase the size of the decoder by setting number of layers to 3, number of heads to 6, and set The model is also trained for 500 epochs using Adam optimizer with the same initial learning rate at 0.001. In order to better familiarize the model with grammar, we also pretrain the decoder for 500 epochs on a larger set of question programs. This pretraining corpus is first uniformly sampled from the PCFG grammar defining questions, then we calculate the average energy value of each program on 10, 000 boards, and keep the top 2500 unique questions. For the evaluation set of human questions, we found that some simple questions become complicated in program form. For example, question "How many ship pieces are there in the second column?" will be translated to "(++ (map (lambda y (and (== (colL y) 2) (not (== (color y) Water)))) (set All_Tiles)))". Such complicated programs score very poorly according to the energy function, so they do not appear in the training set. As a , the average log-likelihood is extremely skewed by these unseen questions. For a more robust evaluation, we remove the last 10% human questions with low energy values according to. The neural model for this experiment has the same hyper-parameters as in the last experiment, and is optimized by REINFORCE algorithm with initial learning rate 0.0001 and batch size 24. To encourage the exploration of the model, we also apply an -greedy strategy with set to 0.5 at the beginning, and gradually decreased to 0.01 as training continues. This model is trained for 500 epochs, within each epoch the model passes 10, 000 different boards. From some preliminary experiments, we find that the models have a strong preference of generating similar programs of relatively low complexity, with the original energy values as rewards. Thus, we tune two parameters of the energy model as mentioned in section 5.3, which are the two parameters corresponding to information seeking features (denoted as f EIG and f EIG=0 in the original paper). We increase this two parameters from 0.1 until the reinforcement learning models are able to generate a diverse set of questions. The sequence RL baseline which directly generates sequence with the decoder is trained with MIXER algorithm , which is a variant of REINFORCE algorithm widely used in sequence generation tasks. MIXER provides a smooth transition from supervised learning to reinforcement learning. This model is pretrained for 500 epochs, and trained with RL for 300 epochs. The grammar is provided in table 4 and 5. Table 4: Part 1 of the grammatical rules. Rules marked with b have a reference to the Battleship game board (e.g., during evaluation the function orient looks up the orientation of a ship on the game board) while all other rules are domain-general (i.e., can be evaluated without access to a game board). True if all elements in set of numbers are equal B → (any setB) True if any element in set of booleans is True B → (all setB) True if all elements in set of booleans are True True if the two ships are touching (diagonal does not count) B → (isSubset setL setL) True if the first set of locations is subset of the second set of locations Number of True elements in set of booleans Column number of location L N → (setSize setL) Number of elements in set of locations The most left of the most top location in the set of locations L → (bottomright setL) The most right of the most bottom location in the set of locations All locations with color C setL → (setDifference setL setL) Remove second set from first set setL → (union setL setL) Combine both sets setL → (intersection setL setL) Elements that exist in both sets setL → (unique setL) Unique elements in set We also perform an ablation test on the neural network in the first experiment (section 5.1). Accuracy for the counting and missing tile tasks is summarized in Table 6a for the full model and lesioned variants. The full neural program generation model shows strong reasoning abilities, achieving an accuracy of 99.80% and 95.50% for the counting and missing tile tasks, respectively. The full model is compared to weakened variants with only one filter size in the encoder, either "3x3" and "1x1 conv only," and the performance of the weakened models drop dramatically on the missing tile task. To better understand the role of different filter sizes, Table 6b breaks down the errors in the missing tile task on whether the question can pick the right ship (color acc.) and whether it can select the right location (location acc.). The 3 × 3 convolution filters can accurately select the correct color, but often fail to choose the right tile. The model with 1 × 1 convolution filters has poor performance for both color and location. In the current architecture, predicting the correct location requires precise information that seems to be lost without filters of different sizes. For the experiment on estimating the distribution of human questions (Experiment 5.2), Table 7 provides a full table of log-likelihood of different models on evaluation set of different uncertainty level. Here we provide more examples of questions generated by our models in the generation experiment (Experiment 5.3). Figure 5, 6 and 7 provides more examples for the same settings as shown in figure 4 in the main text. Figure 8 shows generated examples of the text-based model. What is the size of blue plus the number of the row which contains the beginning of the red ship? (+ (size Blue) (colL (topleft (coloredTiles Red)))) 2.46 Energy: -5.11 Figure 5: Novel questions generated by the grammar enhanced model.' == 'We introduce a model of human question asking that combines neural networks and symbolic programs, which can learn to generate good questions with or without supervised examples.' != 'We use reinforcement learning to train an agent to solve a set of visual arithmetic tasks using provided pre-trained perceptual modules and transformations of internal representations created by those modules.'

Question: Compression is a key step to deploy large neural networks on resource-constrained platforms. As a popular compression technique, quantization constrains the number of distinct weight values and thus reducing the number of bits required to represent and store each weight. In this paper, we study the representation power of quantized neural networks. First, we prove the universal approximability of quantized ReLU networks on a wide class of functions. Then we provide upper bounds on the number of weights and the memory size for a given approximation error bound and the bit-width of weights for function-independent and function-dependent structures. Our reveal that, to attain an approximation error bound of $\epsilon$, the number of weights needed by a quantized network is no more than $\mathcal{O}\left(\log^5(1/\epsilon)\right)$ times that of an unquantized network. This overhead is of much lower order than the lower bound of the number of weights needed for the error bound, supporting the empirical success of various quantization techniques. To the best of our knowledge, this is the first in-depth study on the complexity bounds of quantized neural networks. Various deep neural networks deliver state-of-the-art performance on many tasks such as object recognition and natural language processing using new learning strategies and architectures BID11;; ). Their prevalence has extended to embedded or mobile devices for edge intelligence, where security, reliability or latency constraints refrain the networks from running on servers or in clouds. However, large network sizes with the associated expensive computation and memory consumption make edge intelligence even more challenging BID2 ).In response, as will be more detailed in Section 2, substantial effort has been made to reduce the memory consumption of neural networks while minimizing the accuracy loss. The memory consumption of neural networks can be reduced by either directly reducing the number of weights or decreasing the number of bits (bit-width) needed to represent and store each weight, which can be employed on top of each other BID3. The number of weights can be reduced by pruning BID9, weight sparsifying , structured sparsity learning BID14 and low rank approximation BID5. The bit-width is reduced by quantization that maps data to a smaller set of distinct levels . Note that while quantization may stand for linear quantization only (; BID7 or nonlinear quantization only BID8 BID3 in different works, our discussion will cover both cases. However, as of today quantization is still only empirically shown to be robust and effective to compress various neural network architectures (; BID20 BID22 . Its theoretical foundation still remains mostly missing. Specifically, many important questions remain unanswered. For example:• Why even binarized networks, those most extremely quantized with bit-width down to one, still work well in some cases?• To what extent will quantization decrease the expressive power of a network? Alternatively, what is the overhead induced by weight quantization in order to maintain the same accuracy?In this paper, we provide some insights into these questions from a theoretical perspective. We focus on ReLU networks, which is among the most widely used in deep neural networks BID15 . We follow the idea from BID16 to prove the complexity bound by constructing a network, but with new and additional construction components essential for quantized networks. Specifically, given the number of distinct weight values λ and a target function f, we construct a network that can approximate f with an arbitrarily small error bound to prove the universal approximability. The memory size of this network then naturally serves as an upper bound for the minimal network size. The high-level idea of our approach is to replace basic units in an unquantized network with quantized sub-networks 1 that approximate these basic units. For example, we can approximate a connection with any weight in an unquantized network by a quantized sub-network that only uses a finite number of given weight values. Even though the approximation of a single unit can be made arbitrarily accurate in principle with unlimited resources (such as increased network depth), in practice, there exists some inevitable residual error at every approximation, all of which could propagate throughout the entire network. The challenge becomes, however, how to mathematically prove that we can still achieve the end-to-end arbitrary small error bound even if these unavoidable residual errors caused by quantization can be propagated throughout the entire network. This paper finds a solution to solve the above challenge. In doing so, we have to propose a number of new ideas to solve related challenges, including judiciously choosing the proper finite weight values, constructing the approximation sub-networks as efficient as possible (to have a tight upper bound), and striking a good balance among the complexities of different approximation steps. Based on the bounds derived, we compare them with the available on unquantized neural networks and discuss its implications. In particular, the main contributions of this paper include:• We prove that even the most extremely quantized ReLU networks using two distinct weight values are capable of representing a wide class of functions with arbitrary accuracy.• Given the number of distinct weights and the desired approximation error bound, we provide upper bounds on the number of weights and the memory size. We further show that our upper bounds have good tightness by comparing them with the lower bound of unquantized ReLU networks established in the literature.• We show that, to attain the same approximation error bound, the number of weights needed by a quantized network is no more than O log 5 (1/) times that of an unquantized network. This overhead is of much lower order compared with even the lower bound of the number of weights needed for the error bound. This partially explains why many state-ofthe-art quantization schemes work well in practice.• We demonstrate how a theoretical complexity bound can be used to estimate an optimal bit-width, which in turn enables the best cost-effectiveness for a given task. The remainder of the paper is organized as follows. Section 2 reviews related works. Section 3 lays down the models and assumptions of our analysis. We prove the universal approximability and the upper bounds with function-independent structure in Section 4 and extend it to function-dependent structure in Section 5. We analyze the bound-based optimal bit-width in Section 6. Finally, Section 7 discusses the and gets back to the questions raised above. Quantized Neural Networks: There are rich literatures on how to obtain quantized networks, either by linear quantization or nonlinear quantization BID19; ). Linear quantization does mapping with a same distance between contiguous quantization levels and is usually implemented by storing weights as fixed-point numbers with reduced bit-width (; BID7 . Nonlinear quantization maps the data to quantization levels that are not uniformly distributed and can be either preselected or learned from training. Then the weights are stored using lossless binary coding (the index to a lookup table) instead of the actual values BID8 BID3. It is reported that a pruned AlexNet can be quantized to eight bits and five bits in convolutional layers and fully connected layers, respectively, without any loss of accuracy. Similar are also observed in LENET-300-100, LENET-5, and VGG-16 (a). One may argue that some of these benchmark networks are known to have redundancy. However, recent works show that quantization works well even on networks that are designed to be extremely small and compact. SqueezeNet, which is a state-of-the-art compact network, can be quantized to 8-bit while preserving the original accuracy BID7 ). There are some representative works that can achieve little accuracy loss on ImageNet classification even using binary or ternary weights BID4; BID14 BID21. More aggressively, some works also reduce the precision of activations, e.g. (; ; BID6 . Although the classification accuracy loss can be minimized, the universal approximation property is apparently lost, as with limited output precision the network cannot achieve arbitrary accuracy. Accordingly, we do not include them in the discussion of this paper. The limit of quantization is still unknown while the state-of-the-art keeps getting updated. For example, VGG-16 is quantized to 3-bit while maintaining the original accuracy . Motivated by the great empirical success, the training of quantized neural networks has been analyzed theoretically, but not the network capacity (; BID3 .Universal Approximability and Complexity Bounds: The universal approximability of ReLU networks is proved in and revisited in ( Throughout this paper, we define ReLU networks as feedforward neural networks with the ReLU activation function σ(x) = max(0, x). The ReLU network considered includes multiple input units, a number of hidden units, and one output unit. Without loss of generality, each unit can only connect to units in the next layer. Our on ReLU networks can be extended to any other networks that use piecewise linear activation functions with finite breakpoints such as leaky ReLU and ReLU-6 immediately, as one can replace a ReLU network by an equivalent one using these activation functions while only increasing the number of units and weights by constant factors BID16.We denote the finite number of distinct weight values as λ (λ ∈ Z + and λ ≥ 2), for both linear quantization and nonlinear quantization. For linear quantization, without loss of generality, we assume the finite number of distinct weight values are given as {−1, λ} are uniformly spaced (hence called "linear") in and −1 is used to obtain the negative weight values. For nonlinear quantization, we assume the finite number of distinct weight values are not constrained to any specific values, i.e., they can take any values as needed. To store each weight, we only need log(λ) 2 bits to encode the index, i.e. the bit-width is log(λ). The overhead to store sparse structures can be ignored because it varies depending on the implementation and can be easily reduced to the same order as the weight storage using techniques such as compressed sparse row (CSR) for nonlinear quantization. The number of bits needed to store the codebook can also be ignored because it has lower order of complexity. We consider any function f in the Sobolev space: DISPLAYFORM0 The space W n,∞ consists of all locally integrable function f: DISPLAYFORM1, where |n| ≤ n and Ω is an open set in R d. We denote this function space as F d,n in this paper. Note that we only assume weak derivatives up to order n exist where n can be as small as 1 where the function is non-differentiable. We also only assume the Lipschitz constant to be no greater than 1 for the simplicity of the derivation. When the Lipschitz constant is bigger than 1, as long as it is bounded, the whole flow of the proof remains the same though the bound expression will scale accordingly. When constructing the network to approximate any target function f, we consider two scenarios for deriving the bounds. The first scenario is called function-dependent structure, where the constructed network topology and their associated weights are all affected by the choice of the target function. In contrast, the second scenario is called function-independent structure, where the constructed network topology is independent of the choice of the target function in f ∈ F d,n with a given. The principle behind these design choices (the network topology constructions and the choice of weights) is to achieve a tight upper bound as much as possible. One might consider that we can transform an unquantized network within the error bound to a quantized one in a straightforward way by approximating every continuous-value weight with a combination of discrete weights with arbitrary accuracy. However, the complexity of such approximation (number of discrete weights needed) depends on the distribution of those continuous-value weights (e.g., their min and max), which may vary depending on the training data and network structure and a closed-form expression for the upper bounds is not possible. As such, a more elegant approach is needed. Below we will establish a constructive approach which allows us to bound the approximation analytically. We start our analysis with function-independent structure, where the network topology is fixed for any f ∈ F d,n and a given. We first present the approximation of some basic functions by subnetworks in Section 4.1. We then present the sub-network that approximates any weight in Section 4.2, and finally the approximation of general functions and our main are in Section 4.3. Proposition 1. Denote the design parameter that determines the approximation error bound as r. The proof and the details of the sub-network constructed are included in Appendix A.1. Once the approximation to squaring function is obtained, we get Proposition 2 by the fact that 2xy = (x + y) 2 − x 2 − y 2. Proposition 2. Denote the design parameter that determines the approximation error bound as r. Given x ∈ [−1, 1], y ∈ [−1, 1], and only two weight values 1 2 and − 1 2, there is a ReLU sub-network with two input units that implements a function ×: R 2 → R, such that (i) if x = 0 or y = 0, then × (x, y) = 0; (ii) for any x, y, the error × = | × (x, y) − xy| ≤ 6 · 2 −2(r+1); (iii) the depth is O (r); (iv) the width is a constant; (v) the number of weights is O (r).Proof. Build three sub-networks f r s as described in Proposition 1 and let DISPLAYFORM0 Then the statement (i) is followed by property (i) of Proposition 1. Using the error bound in Proposition 1 and Equation, we get the error bound of ×: DISPLAYFORM1 Since a sub-network B abs that computes σ(x) + σ(−x) can be constructed to get the absolute value of x trivially, we can construct × (x, y) as a linear combination of three parallel f r s and feed them with Proposition 3. Denote the design parameter that determines the approximation error bound as t. A connection with any weight w ∈ [−1, 1] can be approximated by a ReLU sub-network that has only λ ≥ 2 distinct weights, such that (i) the sub-network is equivalent to a connection with weight w while the approximation error is bounded by 2 −t i.e., |w − w| < 2 −t; (ii) the depth is O λt DISPLAYFORM0 Proof. Consider that we need a weight w to feed the input x to a unit in the next layer as wx. With a limited number of distinct weight values, we can construct the weight we need by cascade and combination. For clarity, we first consider w ≥ 0 and x ≥ 0, and relax these assumptions later. The connections with w = 0 can be seen as an empty sub-network while w = 1 can be easily implemented by 4 units with weight 1 2. Now we show how to represent all integral multiples of 2 −t from 2 −t to 1 − 2 −t, which will lead to the statement (i) by choosing the nearest one from w as w. Without loss of generality, we assume t 1 λ−1 is an integer. We use λ weights that include − 1 2 and W: DISPLAYFORM1 We first construct all w from W c which is defined as DISPLAYFORM2 Similar to a numeral system with radix equal to t 1 λ−1, any w i ∈ W c can be obtained by concatenating weights from W while every weights in W is used no greater than t 1 λ−1 − 1 times. After that, all integral multiples of 2 −t from 2 −t to 1−2 −t can be represented by a binary expansion on W c. Note that connections in the last layer for binary expansion use weight 1 2, thus additional 2 −1 is multiplied to scale the resolution from 2 −(t−1) to 2 −t. Since for any weight in W c we need to concatenate no more than λ t 1 λ−1 − 1 weights in a straight line, the sub-network has no greater than λ t 1 λ−1 − 1 + 1 layers, and no greater than 4tλ t 1 λ−1 − 1 + 8t + 4 weights. We now relax the assumption w ≥ 0. When w < 0, the sub-network can be constructed as w = |w|, while we use − 1 2 instead of 1 2 in the last layer. To relax the assumption x ≥ 0, we can make a duplication of the sub-network. Let all the weights in the first layer of the sub-network be 1 2 for one and − 1 2 for the other. Here we are utilizing the gate property of ReLU. In this way, one sub-network is activated only when x > 0 and the other is activated only when x < 0. The sign of the output can be adjusted by flipping the sign of weights in the last layer. Note that the configuration of the sub-network is solely determined by w and works for any input x. The efficiency of the weight approximation is critical to the overall complexity. Compared with the weight selection as {2 −1, 2 DISPLAYFORM3 λ−1}, our approximation reduces the number of weights by a factor of t λ−2 λ−1. With the help of Proposition 2 and Proposition 3, we are able to prove the upper bound for general functions. Theorem 1. For any f ∈ F d,n, given λ distinct weights, there is a ReLU network with fixed structure that can approximate f with any error ∈, such that (i) the depth is DISPLAYFORM0 the number of bits needed to store the network is O λ log (λ) log DISPLAYFORM1 The complete proof and the network constructed can be found in Appendix A.2. We first approximate f by f 2 using the Taylor polynomial of order n − 1 and prove the approximation error bound. Note that even when f is non-differentiable (only first order weak derivative exists), the Taylor polynomial of order 0 at x = m N can still be used, which takes the form of P m = f (m N). Then we approximate f 2 by a ReLU network that is denoted as f with bounded error. After that, we present the quantized ReLU network that implements f and the complexity. The discussion above focuses on nonlinear quantization which is a more general case compared to linear quantization. For linear quantization, which strictly determines the available weight values once λ is given, we can use the same proof for nonlinear quantization except for a different subnetwork for weight approximation with width t and depth t log λ +1. Here we give the theorem and the proof is included in Appendix A.3. Theorem 2. For any f ∈ F d,n, given weight maximum precision 1 λ, there is a ReLU network with fixed structure that can approximate f with any error ∈, such that (i) the depth is O (log (1/)); (ii) the number of weights is O log (1/) + DISPLAYFORM2; (iii) the number of bits needed to store the network is O log(λ) log (1/) + log DISPLAYFORM3 The network complexity can be reduced if the network topology can be set according to a specific target function, i.e. function-dependent structure. In this section, we provide an upper bound for function-dependent structure when d = 1 and n = 1, which is asymptotically better than that of a fixed structure. Specifically, we first define an approximation to f (x) as f (x) that has special properties to match the peculiarity of quantized networks. Then we use piecewise linear interpolation and "cached" functions BID16 to approximate f (x) by a ReLU network. While simply using piecewise linear interpolation at the scale of can satisfy the error bound with O (1/) weights, the complexity can be reduced by first doing interpolation at a coarser scale and then fill the details in the intervals to make the error go down to. By assigning a "cached" function to every interval depending on specific function and proper scaling, the number of weights is reduced to O log −1 (1/) 1/ when there is no constraint on weight values BID16.The key difficulty in applying this approach to quantized ReLU networks is that the required linear interpolation at i T exactly where i = 1, 2, · · ·, T is not feasible because of the constraint on weight selection. To this end, we transform f (x) to f (x) such that the approximation error is bounded; the Lipschitz constant is preserved; f i T are reachable for the network under the constraints of weight selection without increasing the requirement on weight precision. Then we can apply the interpolation and cached function method on f (x) and finally approximate f (x) with a quantized ReLU network. Formally, we get the following proposition and the proof can be found in Appendix A.4.Proposition 4. For any f ∈ F 1,1, t ∈ Z +, and T ∈ Z +, there exists a function f (x) such that DISPLAYFORM0 With the help of Proposition 4 and the weight construction method described in Section 4.2, we are able to apply the interpolation and cached function approach. Denoting the output of the network as f (x), we have |f (x)−f (x)| = |f (x)− f (x)|+| f (x)−f (x)| ≤ by choosing appropriate hyperparameters which are detailed in Appendix A.5 and the network complexity is obtained accordingly. Theorem 3. For any f ∈ F 1,1, given λ distinct weights, there is a ReLU network with function-dependent structure that can approximate f with any error ∈, such that (i) the depth is O λ (log log (1/)) 1 λ−1 + log (1/); (ii) the number of weights is O λ (log log (1/)) 1 λ−1 +1 + (1/) (iii) the number of bits needed to store the network is O log λ λ (log log (1/)) DISPLAYFORM0 Using the different weight construction approach as in the case of function-independent structure, we have the for linear quantization: Theorem 4. For any f ∈ F 1,1, given weight maximum precision 1 λ, there is a ReLU network with function-dependent structure that can approximate f with any error ∈, such that (i) the depth is O (log (1/)); (ii) the number of weights is O (1/); (iii) the number of bits needed to store the network is O (log(λ)/ ). In this section, we first introduce the optimal bit-width problem and then show how a theoretical bound could potentially be used to estimate the optimal bit-width of a neural network. Because of the natural need and desire of comparison with competitive approaches, most quantization techniques are evaluated on some popular reference networks, without modification of the network topology. On the one hand, the advancement of lossless quantization almost stalls at a bit-width between two and six BID8 BID3; BID0; BID6. A specific bit-width depends on the compactness of the reference network and the difficulty of the task. On the other hand, the design space, especially the different combinations of topology and bit-width, is largely underexplored because of the complexity, ing in sub-optimal . A recent work by empirically validates the benefit of exploring flexible network topology during quantization. That work adds a simple variable of network expanding ratio, and shows that a bit-width of four achieves the best cost-accuracy trade-off among limited options in {1, 2, 4, 8, 16, 32}. Some recent effort on using reinforcement learning to optimize the network hyper-parameters BID12 could potentially be used to address this issue. But the current design space is still limited to a single variable per layer (such as the pruning ratio based on a reference network). How to estimate an optimal bit-width for a target task without training could be an interesting research direction in the future. The memory bound expression as derived in this paper helps us to determine whether there is an optimal λ that would lead to the lowest bound and most compact network (which can be translated to computation cost in a fully connected structure) for a given target function. For example, by dropping the lower-order term and ignoring the rounding operator, our memory bound can be simplified as DISPLAYFORM0 where θ 1 is a constant determined by, n, and d. We can find an optimal λ that minimizes M (λ): DISPLAYFORM1 As is detailed in Appendix B, we prove that there exists one and only one local minimum (hence global minimum) in the range of [2, ∞) whenever < 1 2. We also show that λ opt is determined by log 3n2 d /, which can be easily dominated by d. Based on such , we quantitatively evaluate the derivative of M (λ), and based on which the optimal bit-width log(λ opt) under various settings in FIG3 and FIG3, respectively. In FIG3, we also mark the input dimension of a few image data sets. It is apparent to see that the optimal bit width derived from M (λ) is dominated by d and lies between one and four for a wide range of input size. This observation is consistent with most existing empirical research , hence showing the potential power of our theoretical bound derivation. Since the bounds are derived for fully connected networks and depend on the construction approach, the interesting proximity between log(λ opt) and the empirical cannot be viewed as a strict theoretical explanation. Regardless, we show that the complexity bound may be a viable approach DISPLAYFORM2 is a positive monotonically increasing function and thus does not affect the trends too much. Note that λ is the number of distinct weight values and thus log(λ) is the corresponding bit-width. It can be seen that and n only affect log(λ opt) when d is small (< 10 2). We also mark the input dimension d of various image data set and their corresponding log(λ opt). It shows that the optimal bit-width increases very slowly with d.to understand the optimal bit-width problem, thus potentially accelerating the hyper-parameter optimization of deep neural networks. We defer such a thorough investigation of the optimal bit-width or optimal hybrid bit-width configuration across the network to our future work. In this section, we further discuss the bound of nonlinear quantization with a function-independent structure as the generality of nonlinear quantization. The availability of unquantized functionindependent structures in literature also makes it an excellent reference for comparison. Comparison with the Upper Bound: The quality of an upper bound lies on its tightness. Compared with the most recent work on unquantized ReLU networks BID16, where the upper bound on the number of weights to attain an approximation error is given by O log(1/) (1/) d n, our for a quantized ReLU network is given by O λ log DISPLAYFORM0, which translates to an increase by a factor of λ log 1 λ−1 (1/). Loosely speaking, this term reflects the loss of expressive power because of weight quantization, which decreases quickly as λ increases. We also compare our bound with the lower bound of the number of weights needed to attain an error bound of to have a better understanding on the tightness of the bound. We use the lower bound for unquantized ReLU networks from BID16, as it is also a natural lower bound for quantized ReLU networks. Under the same growth rate of depth, the lower bound is given by Ω(log −3 (1/) (1/) d/n ), while our upper bound is, within a polylog factor when λ is a constant, O(λ log DISPLAYFORM0 The comparison validates the good tightness of our upper bound. The Upper Bound of Overhead: More importantly, the above comparison yields an upper bound on the possible overhead induced by quantization. By comparing the expressions of two bounds while treating λ as a constant, we can show that, to attain the same approximation error bound, the number of weights needed by a quantized ReLU network is no more than O(log 5 (1/)) times that needed by an unquantized ReLU network. Note that this factor is of much lower order than the lower bound Ω(log −3 (1/) (1/) d/n ). This little overhead introduced by weight quantization explains in part the empirical success on network compression and acceleration by quantization and also answers in part the questions as raised in Section 1. Given the significant benefits of quantization in term of memory and computation efficiency, we anticipate that the use of quantization networks will continue to grow, especially on resource-limited platforms. Future Work: There remain many other avenues for future investigation. For example, although we derived the first upper bound of quantized neural networks, the lower bound is still missing. If a tight lower bound of the network size is established, it could be combined with the upper bound to give a much better estimation of required resources and the optimal bit-width. We believe the trends associated with the bounds can also be useful and deserve some further investigation. For example, the trend may help hardware designers in their early stage of design exploration without the need of lengthy training. While we assume a uniform bit-width across all layers, another area of research is to allow different bit-widths in different layers, which could achieve better efficiency and potentially provide theoretical justifications on the emerging trend of hybrid quantization BID17 DISPLAYFORM1 DISPLAYFORM2 where DISPLAYFORM3 is the i-th iterate of g(x). Since g(x) can be implemented by a ReLU sub-network DISPLAYFORM4 •r (x) can be obtained by concatenating such implementation of g(x) for r times. Now, to implement f r s (x) based on g•r (x), all we need are weights {2 −2, 2 −4, · · ·, 2 −2(r−1), 2 −2r }, which can be easily constructed with additional 2r layers and the weight 1 2. Note that a straightforward implementation will have to scale g •i (x) separately (multiply by different numbers of 1 2) before subtracting them from x because each g•i (x) have a different coefficient. Then the width of the network will be Θ(r). Here we use a "pre-scale" method to reduce the network width from Θ(r) to a constant. The network constructed is shown in FIG6. The one-layer sub-network that implements g(x) and the one-layer sub-network that scales the input by 4 are denoted as B g and B m respectively. Some units are copied to compensate the scaling caused by •i (x) are scaled by 2 2(r−i) respectively. As a , we obtain 2 2r x − r i=1 2 2(r−i) g •i (x) after the last B m. Then it is scaled by 2 −2r in the later 2r layers to get f r s (x). In this way, we make all g •i (x) sharing the same scaling link and a constant width can be achieved. A.2 THE PROOF OF THEOREM 1 Theorem 1. For any f ∈ F d,n, given λ distinct weights, there is a ReLU network with fixed structure that can approximate f with any error ∈, such that (i) the depth is DISPLAYFORM5 the number of bits needed to store the network is O λ log (λ) log DISPLAYFORM6 Proof. The proof is composed of four steps. We first approximate f by f 2 using the Taylor polynomial of order n − 1 and prove the approximation error bound. Note that even when f is nondifferentiable (only first order weak derivative exist), the Taylor polynomial of order 0 at x = m N can still be used, which takes the form of P m = f (m N). Then we approximate f 2 by a ReLU network that is denoted as f with bounded error. After that, we present the quantized ReLU network that implements the network f and the complexity of the network. We use a partition of unity on DISPLAYFORM7, and h(x) is defined as follows: DISPLAYFORM8 where N is a constant and DISPLAYFORM9 Note that supp ψ m ⊂ {x : DISPLAYFORM10 For all m, we have the order n − 1 Taylor polynomial for the function f at x = m N as DISPLAYFORM11 To get a more realizable approximation for quantized networks, we define P m (x) = n−|n|. Then we get an approximation to f using P m and ψ m as f 2 m∈{0,···,N} d ψ m P m. Then the approximation error of f 2 is bounded by Equation. DISPLAYFORM12 The second step follows ψ m (x) = 0 when x / ∈ suppψ m. In the third step we turn the sum to multiplication, because for any x there are up to 2 d terms ψ m (x) that are not equal to zero. The fourth step uses a Lagrange's form of the Taylor remainder. The fifth step follows different round precision of β m,n in different order and the fact that the number of terms with order i is not greater than d i.We rewrite f 2 as DISPLAYFORM13 where DISPLAYFORM14 Note that β m,n is a constant and thus f 2 is a linear combination of at most d n (N + 1) d terms of f m,n (x). Note that when d = 1, the number of terms should be n(N + 1) d instead; but for simplicity of presentation we loosely use the same expression as they are on the same order. We define an approximation to f m,n (x) as f m,n (x). The only difference between f m,n (x) and f m,n (x) is that all multiplication operations are approximated by × as discussed in Proposition 2. Consider that if we construct our function × with | × (x, y) − xy| < × = 2 −2(r+1), then DISPLAYFORM15 Applying Equation FORMULA0 to |f m,n (x) − f m,n (x)| repeatedly, we bound it to Equation FORMULA0. DISPLAYFORM16 Finally, we define our approximation to f (x) as f (x): DISPLAYFORM17 Using Equation FORMULA0, we get the error bound of the approximation to f 2 (x) as in Equation FORMULA0. DISPLAYFORM18 The second line follows again the support property and statement (i) of Proposition 2. The third line uses the bound |β m,n | ≤ 1. The fourth line is obtained by inserting Equation.Then the final approximation error bound is as follows: DISPLAYFORM19 Using statement (ii) of Proposition 2 and choosing r as r = log(6N n (d+n−1)) 2 − 1, the approximation error turns to DISPLAYFORM20 Figure 3: A qunatized ReLU network that implements f (x). The connections of all B fmn are the same. Every connection from B N to other blocks has no greater than two weights. DISPLAYFORM21 Therefore, for any f ∈ F d,n and ∈, there is a ReLU network f that approximate f with error bound if we choose DISPLAYFORM22 We now present the construction of the network for f (x). If every f m,n (x) can be computed by a sub-network, then f (x) is simply a weighted sum of all outputs of f m,n (x). By Proposition 3, we can implement the needed weights β m,n by choosing t = log. As discussed in Proposition 3, we build a weight construction network B w in the way that all integral multiples of the minimal precision can be obtained. Therefore, all mi N can be obtained in the same way as β m,n, except that we need to concatenate two weight construction sub-networks. Now we analyze the complexity of the network. The implementation of f (x) is shown in Figure 3. The function and size of blocks are listed in TAB2. Then we are able to obtain the complexity of the network. While we can write the complexity of the network in an explicit expression, here we use the O notation for clarity. Let N d, N w, N b be the depth, the number of weights, and the number of bits required respectively. The weight construction blocks B w have the highest order of number of weights and we have N w = O λt DISPLAYFORM23 Inserting t = log A.3 THE PROOF OF THEOREM 2Theorem 2. For any f ∈ F d,n, given weight maximum precision 1 λ, there is a ReLU network with fixed structure that can approximate f with any error ∈, such that (i) the depth is O (log (1/)); (ii) the number of weights is O log (1/) + DISPLAYFORM24; (iii) the number of bits needed to store the network is O log(λ) log (1/) + log 2 (1/) (1/) DISPLAYFORM25 With λ distinct values, a linearly quantized network has a minimal resolution of 1 λ. The proof for the approximability of linear quantization can be done in the same way as Theorem 1 except for a different sub-network for weight approximation. We still construct W c in Proposition 3 first and any weight value from W c can be obtained by multiply at most t log λ weights. Thus the width and depth of the weight approximation network will be t and t log λ + 1 respectively. Updating the B w in TAB2, we obtain the complexity accordingly. DISPLAYFORM26 This contradicts the fact that f (T by Equation. Before the intersection, if f < f +, DISPLAYFORM27, apply the same logic and we obtain 0 ≤ f DISPLAYFORM28 T. This implies statement (iii) and concludes the proof. Theorem 3. For any f ∈ F 1,1, given λ distinct weights, there is a ReLU network with function-dependent structure that can approximate f with any error ∈, such that (i) the depth is O λ (log log (1/)) 1 λ−1 + log (1/); (ii) the number of weights is O λ (log log (1/)) 1 λ−1 +1 + (1/) (iii) the number of bits needed to store the network is O log λ λ (log log (1/)) 1 λ−1 +1 + (1/).Proof. We first transform f to f with Proposition 4. Then we apply the interpolation and cached function method from while using the weight construction method described in Proposition 3. Denoting the output of the network as f (x), we have |f (The approximation network is shown in FIG15 . The sizes of blocks are given in TAB3 where f T is the uniform linear interpolation function of f with T − 1 breakpoints, f * is the sum of the selected cached functions, Φ(x) is a filtering function. The inputs connections to B f * 1 and the connections inside B m have higher order to the number of weights than others. Then the complexity can be obtained accordingly. Theorem 4. For any f ∈ F 1,1, given weight maximum precision 1 λ, there is a ReLU network with function-dependent structure that can approximate f with any error ∈, such that (i) the

This paper proves the universal  approximability of quantized ReLU neural networks and puts forward the complexity bound given arbitrary error.

Represent the TLDR for finding the corresponding scientific article.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.