1

arXiv:1709.06182v1 [cs.SE] 18 Sep 2017

A Survey of Machine Learning for Big Code and Naturalness
MILTIADIS ALLAMANIS, Microsoft Research
EARL T. BARR, University College London
PREMKUMAR DEVANBU, University of California, Davis
CHARLES SUTTON, University of Edinburgh and The Alan Turing Institute
Research at the intersection of machine learning, programming languages, and software engineering has
recently taken important steps in proposing learnable probabilistic models of source code that exploit code’s
abundance of patterns. In this article, we survey this work. We contrast programming languages against
natural languages and discuss how these similarities and diﬀerences drive the design of probabilistic models.
We present a taxonomy based on the underlying design principles of each model and use it to navigate the
literature. Then, we review how researchers have adapted these models to application areas and discuss crosscutting and application-speciﬁc challenges and opportunities.
CCS Concepts: • Computing methodologies → Machine learning; Natural language processing; • Software and its engineering → Software notations and tools; • General and reference → Surveys and
overviews;
Additional Key Words and Phrases: Big Code, Code Naturalness, Software Engineering Tools, Machine Learning

1 INTRODUCTION
Software is ubiquitous in modern society. Almost every aspect of life, including healthcare, energy, transportation, public safety, and even entertainment, depends on the reliable operation of
high-quality software. Unfortunately, developing software is a costly process: software engineers
need to tackle the inherent complexity of software while avoiding bugs, and still delivering highly
functional software products on time. There is therefore an ongoing demand for innovations in
software tools that help make software more reliable and maintainable. New methods are constantly sought, to reduce the complexity of software and help engineers construct better software.
Research in this area has been dominated by the formal, or logico-deductive, approach. Practitioners of this approach hold that, since software is constructed in mathematically well-deﬁned
programming languages, software tools can be conceived in purely formal terms. The design of
software tools is to be approached using formal methods of deﬁnition, abstraction, and deduction.
Properties of tools thus built should be proven using rigorous proof techniques such as induction
over discrete structures. This logico-deductive approach has tremendous appeal in programming
languages research, as it holds the promise of proving facts and properties of the program. Many
elegant and powerful abstractions, deﬁnitions, algorithms, and proof techniques have been developed, which have led to important practical tools for program veriﬁcation, bug ﬁnding, and
refactoring [21, 39, 42]. It should be emphasized that these are theory-ﬁrst approaches. Software
constructions are viewed primarily as a mathematical objects, and when evaluating software tools
This work was supported by Microsoft Research Cambridge through its PhD Scholarship Programme. M. Allamanis, E. T.
Barr, and C. Sutton are supported by the Engineering and Physical Sciences Research Council [grant numbers EP/K024043/1,
EP/P005659/1, and EP/P005314/1]. P. Devanbu is supported by the National Research Foundation award number 1414172.
Author’s addresses: M. Allamanis, Microsoft Research, Cambridge, UK; E.T. Barr, University College London, UK; P. Devanbu, UC Davis, CA, USA; C. Sutton, University of Edinburgh, UK and Alan Turing Institute, London, UK; .

2

Allamanis et al.

built using this approach, the elegance and rigor of deﬁnitions, abstractions, and formal proofs-ofproperties are of dominant concern. The actual varieties of use of software constructs, in practice,
become relevant later, in case studies, that typically accompany presentations in this line of work.
Of late, another valuable resource has arisen: the large and growing body of successful, widely
used, open-source software systems. Open-source software systems such as Linux, MySQL, Django,
Ant, and OpenEJB have become ubiquitous. These systems publicly expose not just source code,
but also meta-data concerning authorship, bug-ﬁxes, and review processes. The scale of available
data is massive: billions of tokens of code and millions of instances of meta-data, such as changes,
bug-ﬁxes, and code reviews. This abundance of “code-in-the-wild” suggests a new, data-driven approach to developing software tools: why not let the statistical distributional properties, estimated
over representative software corpora, also inﬂuence the design of development tools? Thus rather
than performing well in the worst case, or in case studies, our tools can perform well in most cases,
thus delivering greater advantages in expectation. The appeal of this approach echoes that of earlier work in computer architecture: Amdahl’s law, for example, tells us to focus on the common
case. This motivates a similar hope for development tools, that tools for software development and
program analysis can be improved by focusing on the common cases using a ﬁne-grained estimate
of the statistical distribution of code. Essentially, the hope is that analyzing the text of thousands
of well-written software projects can uncover patterns that partially characterize software that is
reliable, easy to read, and easy to maintain.
The promise and power of machine learning rests on its ability to generalize from examples and
handle noise. To date, software engineering (SE) and programming languages (PL) research has
largely focused on using machine learning (ML) techniques as black boxes to replace heuristics and
ﬁnd features, sometimes without appreciating the subtleties of the assumptions these techniques
make. A key contribution of this survey is to elucidate these assumptions and their consequences.
Just as NLP research changed focus from rule-based expert systems to statistical methods [90],
SE/PL should make the same transition, augmenting traditional methods that consider only the
formal structure of programs with information about the statistical properties of code.
This vision raises a fundamental question. Many aspects of code, such as names, formatting, the
lexical order of methods, and so on, have no impact on program semantics. This is precisely why
we abstract them away as an early step of many program analyses. But then, why should statistical
properties of code tokens matter at all for development tools? To explain this, we suggest a hypothesis which has recently emerged in the software engineering literature, called the naturalness
hypothesis. The inspiration for the naturalness hypothesis can be traced back to the “literate programming” concept of Don Knuth [94], which draws from the insight that programming is a form
of human communication: “Let us change our traditional attitude to the construction of programs:
Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather
on explaining to human beings what we want a computer to do...” [94] The naturalness hypothesis,
then, holds that
The naturalness hypothesis. Software is a form of human communication; software corpora have similar statistical properties to natural language corpora; and these
properties can be exploited to build better software engineering tools.
The exploitation of the statistics of human communication is a mature and eﬀective technology,
with numerous applications [90]. Large corpora of human communication, viz. natural language
corpora, have been extensively studied, and highly reﬁned statistical models of these corpora have
been used to great eﬀect in speech recognition, translation, error-correction, etc..

A Survey of Machine Learning for Big Code and Naturalness

3

The naturalness hypothesis holds that, because coding is an act of communication, one might
expect large code corpora to be statistically similar to natural language corpora, thus allowing software engineering tools to exploit probabilistic ML models. The ﬁrst actual empirical evidence of
this hypothesis, showing that models originally developed for natural language were surprisingly
eﬀective for source code, was presented by Hindle et al. [80, 81]. More evidence and numerous applications of this approach have followed, which are the subject of this review. Interestingly, recent
cognitive science evidence, based on fMRI [54], compares the mechanisms in the brain associated
with reading code and reading natural text, ﬁnding that the brain mechanisms are less distinct for
more experienced programmers than less experienced ones, leading the authors to conclude: “. . .
programming languages are treated more like natural languages with greater expertise” [54].
The naturalness hypothesis, then, inspires the general goal to apply machine learning and statistical approaches to create probabilistic source code models that learn how developers naturally
use code. These models can be used to augment existing tools with statistical information and enable new machine learning-based software engineering tools, such as recommender systems and
program analyses. At a high level, statistical methods allow a system to make hypotheses, along
with probabilistic conﬁdence values, of what a developer might want to do next or what formal
properties might be true of a chunk of code. Probabilistic methods also provide natural ways of
learning correspondences between code and other types of documents, such as requirements, blog
posts, comments, etc.— such correspondences will always be uncertain, because natural language
is ambiguous, and so the quantitative measure of conﬁdence provided by probabilities is especially
natural. As we discuss in Section 4, one could go so far as to claim that almost every area of software engineering and programming language research has potential opportunities for exploiting
statistical properties .
The “naturalness hypothesis” implies that source code and its properties are predictable and recent research has exploited this fact [19, 61, 81, 126, 168]. Although this might not seem surprising,
one should appreciate the root cause of “naturalness”. Naturalness of code seems to have a strong
connection with the fact that developers prefer to write [5] and read [78] code that is conventional,
idiomatic, and familiar because it helps understanding and maintaining software systems. Code
that takes familiar forms is more transparent, in that its meaning is more readily apparent to an
experienced reader. Thus, the naturalness hypothesis leads seamlessly to a “code predictability”
notion, suggesting that code artifacts — from simple token sequences to formal veriﬁcation statements — contain useful recurring and predictable patterns that can be exploited. “Naturalness” and
“big code” should be viewed as instances of a more general concept that there is exploitable regularity across human-written code that can be “absorbed” and generalized by a learning component
that can transfer its knowledge and probabilistically reason about new code.
This article reviews the emerging area of machine learning and statistical natural language processing methods applied to source code. We focus on probabilistic models of code, that is, methods
that estimate a distribution over all possible source ﬁles. Machine learning in probabilistic models
has seen wide application throughout artiﬁcial intelligence, including natural language processing,
robotics, and computer vision, because of its ability to handle uncertainty and to learn in the face
of noisy data. One might reasonably ask why it is necessary to handle uncertainty and noise in
software development tools, when in many cases the program to be analyzed is known (there is no
uncertainty about what the programmer has written) and is deterministic. In fact, there are several
interesting motivations for incorporating probabilistic modeling into machine learning methods
for software development. First, probabilistic methods oﬀer a principled method for handling uncertainty and fusing multiple, possibly ambiguous, sources of information. Second, probabilistic

4

Allamanis et al.

models provide a natural framework for connecting prior knowledge to data — providing a natural framework to design methods based on abstractions of statistical properties of code corpora.
In particular, we often wish to infer relationships between source code and natural language text,
such as comments, bug reports, requirements documents, documentation, search queries, and so
on. Because natural language text is ambiguous, it is useful to quantify uncertainty in the correspondence between code and text. Finally, when predicting program properties, probabilities
provide a way to relax strict requirements on soundness: we can seek unsound methods that predict program properties based on statistical patterns in the code, using probabilities as a way to
quantify the method’s conﬁdence in its predictions.
In this paper, we review recent work on machine learning methods for analyzing source code, including probabilistic models, n-gram language models, and deep learning methods. We also touch
on other types of machine learning-based source code models, aiming to give a broad overview
of the area, to explain the core methods and techniques, and to discuss applications in programming languages and software engineering. We focus on work that goes beyond a “bag of words”
representation of code, modeling code using sequences, trees, and continuous representations. We
describe a wide range of emerging applications, ranging from recommender systems, readability,
debugging, program analysis, and program synthesis. This review is structured as follows. We ﬁrst
discuss the diﬀerent characteristics of natural language and source code to motivate the design decisions involved in machine learning models of code (Section 2). We then introduce a taxonomy
of probabilistic models of source code (Section 3). Then we describe the software engineering and
programming language applications of probabilistic source code models (Section 4). Finally, we
mention a few overlapping research areas (Section 6), and we discuss challenges and interesting
future directions (Section 5).
Related Reviews and other Resources. There have been short reviews summarizing the progress
and the vision of the research area, from both software engineering [48] and programming languages perspectives [25, 181]. However, none of these articles can be considered extensive literature reviews, which is the purpose of this work. Ernst [52] discusses promising areas of applying natural language processing to software development, including error messages, variable
names, code comments, and user questions. Some resources, datasets and code can be found at
http://learnbigcode.github.io/. An online version of the work reviewed here — which we will keep
up-to-date by accepting external contributions — can be found at https://ml4code.github.io.
2

TEXT, CODE AND MACHINE LEARNING

Programming languages bridge the gap between computers and the human mind: they construct
palatable abstractions out of a multitude of minute state transitions. Thus, source code has two
audiences and is inherently bimodal: it communicates along two channels: one with humans, and
one with computers. Humans must understand code to read and write it; computers must be able
to execute it. The bimodality of code drives the similarities and diﬀerences between it and text.
Below, we discuss these similarities and diﬀerences with forward pointers to how they have been
exploited, handled, or remain open. Where they are similar, code is a new problem domain for
existing ML and NLP techniques. Hindle et al. [80] not only showed that exploitable similarity
exists between the two via an n-gram language model, but that code is even less surprising than
text. Although it may seem manifestly obvious that code and text have many diﬀerences, it is useful
to enumerate these diﬀerences carefully, as this allows us to gain insight into when techniques
from NLP need to be modiﬁed to deal with code. Perhaps the most obvious diﬀerence is that
code is executable and has formal syntax and semantics. We close by discussing how source code

A Survey of Machine Learning for Big Code and Naturalness

5

bimodality manifests itself as synchronization points between the algorithmic and explanatory
channels.
Executability. All code is executable; text rarely is. So code is often semantically brittle — small
changes (e.g. swapping function arguments) can drastically change the meaning of code; whereas
natural language is more robust in that readers can often understand text even if it contains mistakes. Despite the bimodal nature of code and its human-oriented modality, the sensitivity of code
semantics to “noise” necessitates the combination of probabilistic and formal methods. For example, existing work builds probabilistic models then applies strict formal constraints to ﬁlter their
output (Section 3.1) or uses them to guide formal methods (Section 4.7). Nevertheless, further research on bridging formal and probabilistic methods is needed (Section 5.1).
Whether it is possible to translate between natural languages in a way that completely preserves meaning is a matter of debate. Programming languages, on the other hand, can be translated between each other exactly, as all mainstream programming languages are Turing-complete.
(That said, porting real-world programs to new languages and platforms remains challenging in
practice [29]). ML techniques have not yet tackled these problems (Section 5.1). Programming languages diﬀer in their expressivity and intelligibility, ranging from Haskell to Malbolge1 , with some
especially tailored for certain problem domains; in contrast, natural languages are typically used
to communicate across a wide variety of domains. Executability of code induces control and data
ﬂows within programs, which have only weak analogs in text. Finally, executability gives rise to
additional modalities of code — its static and dynamic views (e.g. execution traces), which are not
present in text. Learning over traces or ﬂows are promising directions (Section 5.1).
Formality. Programming languages are formal languages, whereas formal languages are only
mathematical models of natural language. As a consequence, programming languages are designed top-down by a few designers for many users. Natural languages, in contrast, emerge, bottom up, “through social dynamics” [43]. Natural languages change gradually, while programming
languages exhibit punctuated change: new releases, like Python 3.0, sometimes break backward
compatibility. Text is relatively free-form, while formatting can be meaningful in code: Python’s
whitespace sensitivity is the canonical example. Long-range dependencies (e.g. library imports) are
longer in code than text. Text is “grounded” and this environmental dependence is robust. In contrast, code suﬀers from bit rot because all its explicit environmental interactions must be speciﬁed
upfront and execution environments evolve much more quickly than natural languages.
Source code’s formality facilities reuse. Solving a problem algorithmically is cognitively expensive, so developers actively try to reuse code [87], moving common functionality into libraries
to facilitate reuse. As a result, most functions are unique, with the exception of coding competitions or undergraduate projects. In contrast, one can ﬁnd thousands of news articles describing
an important global event. On the other hand, Gabel and Su [61] have found that locally, code is
more pattern dense than text (Section 3.1). This has led to important performance improvements
on some applications, such as code completion (Section 4.1).
Because programming languages are automatically translated into machine code, they must be
syntactically, even to a ﬁrst approximation semantically, unambiguous2 . In contrast to NLP models, which must always account for textual ambiguity, probabilistic models of code can and do
take advantage of the rich and unambiguous code structure. Although it is less pervasive, ambiguity remains a problem in the analysis of code, because of issues like polymorphism and aliasing.
1 https://en.wikipedia.org/wiki/Malbolge

2 Exceptions exist, like http://po-ru.com/diary/ruby-parsing-ambiguities/

but these rarely matter in practice.

and https://stackoverﬂow.com/questions/38449606/ambiguity-in

6

Allamanis et al.

Section 3.2 and Section 4.4 discuss particularly notable approaches to handling it. Co-reference
ambiguities can arise when viewing code statically, especially in dynamically typed languages
(Section 5.1). The undeﬁned behavior that some programming languages permit can cause semantic ambiguity and, in the ﬁeld, syntactic problems can arise due to nonstandard compilers [21];
however, the dominance of a handful of compilers/interpreters for most languages ameliorates
both problems.
Synchronizing the Channels. Code’s two channels interact through their semantic units, but mapping code units to textual units remains an open problem. Natural semantic units in code are identiﬁers, statements, blocks, and functions. None of these universally maps to textual semantic units.
Identiﬁers, even verbose function names that seek to describe their function, carry less information
than words like “Christmas” or “set”, which has the longest entry in the Oxford English Dictionary3 .
Statements in code tend to be semantically impoverished, usually nearly meaningless in isolation.
Blocks vary greatly in length and semantics richness and often lack clear boundaries to a human
reader. Functions are clearly delimited and semantically rich, but long. In NL, a sentence is the
natural multiword semantic unit and usually contains fewer than 50 words (tokens). Intuitively,
action sentences in text usually describe changes that are roughly comparable in their eﬀect on the
world to the eﬀect that functions have in code, yet functions usually contains hundreds of tokens
and often require multiple sentences (i.e. a paragraph) to describe.
Additionally, parse trees in text tend to be diverse, short, and shallow compared to abstract
syntax trees for code, which are usually much deeper with repetitive internal structure. Code bases
are multilingual (i.e. contain code in more than one programming language, e.g. Java and SQL)
with diﬀerent tasks described in diﬀerent languages, more frequently than text corpora; this can
drastically change the shape and frequency of its semantic units. Code has a higher neologism rate
than text. Almost 70% of all characters are identiﬁers and a developer must choose a name for each
one [47]; when writing text, an author rarely names new things but usually chooses an existing
word to use. Existing work handles code’s neologism rate by introducing cache mechanisms or
decomposing identiﬁers at a subtoken level (Section 3.1).
Determining which semantic code unit is most useful for which task is an open question. Consider the problem of automatically generated comments that describe code, which can be formalized as a machine translation problem from code to text. Statistical machine translation approaches
learn from an aligned corpus. Statement-granular alignment yields redundant comments, while
function granular alignment has saliency issues (Section 4.5). As another example, consider code
search, where search engines must map queries into semantic code units. Perhaps the answer will
be in maps from code to text whose units vary by granularity or context (Section 5.1).
3 PROBABILISTIC MODELS OF CODE
In this section, we turn our attention to probabilistic machine learning models of source code. As
all models do, probabilistic machine learning models make simplifying assumptions about the modeled domain. These assumptions make the models tractable to learn and use, but introduce error.
Since each model makes diﬀerent assumptions, each model has its own strengths and weaknesses
and is more suitable for some applications. In this section, we group existing work into families of
models and discuss their assumptions. We also discuss how and why these models diﬀer from their
natural language counterparts. In Section 4, we discuss applications of these models in software
engineering and programming languages.
3 http://public.oed.com/history-of-the-oed/dictionary-facts/

A Survey of Machine Learning for Big Code and Naturalness

7

A probabilistic model of source code is a probability distribution over code artifacts. To group
these family of models in terms of shared design choices, we separate these models into three
categories, based on the form of the equation of the modeled probability distribution and their
inputs and outputs, with the caveat that some models fall into multiple categories.
Code-generating Models deﬁne a probability distribution over code by stochastically modeling the generation of smaller and simpler parts of code, e.g. tokens or AST nodes.
Representational Models of Code take an abstract representation4 of code as input. Example representations include token contexts or data ﬂow. The resulting model yields a
conditional probability distribution over code element properties, like the types of variables, and can predict them.
Pattern Mining Models infer, without supervision, a likely latent structure within code.
These models are an instantiation of clustering in the code domain; they can ﬁnd reusable
and human-interpretable patterns.
Code-generating models ﬁnd analogues in generative models of text, such as language models,
machine translation and natural language generation systems. Code representational models are
analogous to systems for named entity recognition, text classiﬁcation, and sentiment analysis in
NLP. Finally, code pattern mining models are analogous to probabilistic topic models and ML techniques for mining structured information (e.g. knowledge-bases) from text. To simplify notation
below, we use c to denote an arbitrary code abstraction, like an AST.
3.1

Code-generating Probabilistic Models of Source Code

Code-generating probabilistic models of code are probability distributions that describe a stochastic process for generating valid code, i.e. they model how code is written. Given training data D,
an output code representation c, and a possibly empty context C(c), these models learn the probability distribution P D (c|C(c)) and sample P D to generate code. When C(c) = , the probability
distribution P D is a language model of code, i.e. it models how code is generated when no context
information is available. When C(c) is a non-code modality (e.g. natural language), P D describes
a code-generative multimodal model of code. When C(c) is also code, the probability distribution
P D is a transducer model of code. In addition to generating code, by deﬁnition, code generating
probabilistic models of code score code, assigning a non-zero probability to every possible snippet
of code. This score, sometimes referred to as “naturalness” of the code, suggests how probable the
code is under a learned model.
Since code-generating models predict the complex structure of code, they make simplifying assumptions about the generative process and iteratively predict elements of the code to generate a
full code unit, e.g. code ﬁle or method. Because of code’s structural complexity and the simplifying
assumptions these models make to cope with it, none of the existing models in the literature generate code that always parses, compiles, and executes. Some of the models do, however, impose
constraints that take code structure into account to remove some inconsistencies; for instance,
[113] only generate variables declared within each scope.
We structure the discussion about code-generating models of code as follows. We ﬁrst discuss
how models in this category generate code and then we show how the three diﬀerent types of
models (language, transducer and multimodal models) diﬀer.
3.1.1 Representing Code in Code-Generating Models. Probabilistic models for generating structured objects are widely in use in machine learning and natural language processing with a wide
4

In the machine learning literature, representation, applied to code, is roughly equivalent to abstraction in programming
language research: a lossy encoding that preserves a semantic property of interest.

8

Allamanis et al.

range of applications. Machine learning research is considering a wide range of structures from
natural language sentences to chemical structures and images. Within the source code domain, we
can broadly ﬁnd three category of models based on the type of the predicted structure.
Sequence-based Models. Sequence-based models are commonly used because of their simplicity.
They view code as a sequence of elements, usually code tokens or characters, i.e. c = t 1 . . . t M .
Predicting a large sequence in a single step is infeasible due to the exponential number of possible sequences; for a set of V elements, there are |V | N sequences of length N . Therefore, most
sequence-based models predict sequences by sequentially generating each element, i.e. they model
the probability distribution P(tm |t 1 . . . tm−1 , C(c)). However, directly modeling this distribution is
impractical and all models make diﬀerent simplifying assumptions.
The n-gram model is the most widely used sequence-based model and most commonly used with
language models (i.e. C(c) = ). It is an eﬀective and practical LM for capturing local and simple
statistical dependencies in sequences. N -gram models assume that tokens are generated sequentially, left-to-right and that the next token can be predicted using only the previous n − 1 tokens.
The consequence of capturing a short context is that n-gram models cannot handle long-range
dependencies, notably scoping information. Formally, the probability of a token tm , is conditioned
on the context C(c) (if any) and the generated sequence so far t 1 . . . tm−1 , which is assumed to
depend on only the previous n − 1 tokens. Under this assumption, we write
P D (c|C(c)) = P(t 1 . . . t M |C(c)) =

M
Ö

P(tm |tm−1 . . . tm−n+1 , C(c)).

(1)

m=1

To use this equation, we need to know the conditional probabilities P(tm |tm−1 . . . tm−n+1 , C(c)) for
each possible n-gram and context. This is a table of |V |n numbers for each context C(c). These are
the parameters of the model that we learn from the training corpus. The simplest way to estimate
the model parameters is to set P(tm |tm−1 . . . tm−n+1 ) to the proportion of times that tm follows
tm−1 . . . tm−n+1 . In practice, this simple estimator does not work well, because it assigns zero probability to n-grams that do not occur in the training corpus. Instead, n-gram models use smoothing
methods [37] as a principled way for assigning probability to unseen n-grams by extrapolating
information from m-grams (m < n). Furthermore, considering n-gram models with non-empty
contexts C(c) exacerbates sparsity rendering these models impractical. Because of this, n-grams
are predominantly used as language models. The use of n-gram LMs in software engineering originated with the pioneering work of Hindle et al. [81] who used an n-gram LM with Kneser-Ney
[37] smoothing. Most related research has followed this practice.
In contrast to text, code tends to be more verbose [81] and much information is lost within the
n − 1 tokens of the context. To tackle this problem, Nguyen et al. [132] extended the standard
n-gram model by annotating the code tokens with parse information that can be extracted from
the currently generated sequence. This increases the available context information allowing the
n-gram to achieve better predictive performance. Following this trend, but using concrete and abstract semantics of code, Raychev et al. [152] create a token-level model that treats code generation
as a combined synthesis and probabilistic modeling task.
Tu et al. [166] and later Hellendoorn and Devanbu [77] noticed that code has a high degree of
localness, where identiﬁers (e.g. variable names) are repeated often within close distance. In their
work, they adapted work in speech and natural language processing [97] adding a cache mechanism that assigns higher probability to tokens that have been observed most recently, achieving
signiﬁcantly better performance compared to other n-gram models. Modeling identiﬁers in code
is challenging [6, 10, 26, 113]. The agglutinations of multiple subtokens (e.g. in getFinalResults)

A Survey of Machine Learning for Big Code and Naturalness

9

when creating identiﬁers is one reason. Following recent NLP work that models subword structure
(e.g. morphology) [160], explicitly modeling subtoken in identiﬁers may improve the performance
of generative models. Existing token-level code-generating models do not produce syntactically
valid code. Raychev et al. [152] added additional context in the form of constraints — derived from
program analysis — to avoid generating some incorrect code.
More recently, sequence-based code models have turned to deep recurrent neural network (RNN)
models to outperform n-grams. These models predict each token sequentially, but loosen the ﬁxedcontext-size assumption, instead representing the context using a distributed vector representation
(Section 3.2). Following this trend, Karpathy et al. [93] and Cummins et al. [45] use characterlevel LSTMs [83]. Similarly, White et al. [175] and Dam et al. [46] use token-level RNNs. Recently,
Bhoopchand et al. [23] used a token sparse pointer-based neural model of Python that learns to
copy recently declared identiﬁers to capture very long-range dependencies of identiﬁers, outperforming standard LSTM models5 .
Although neural models usually have superior predictive performance, training them is signiﬁcantly more costly compared to n-gram models usually requiring orders of magnitude more data.
Intuitively, there are two reasons why deep learning methods have proven successful for language
models. First, the hidden state in an RNN can encode longer-range dependencies beyond the short
context of n-gram models. Second, RNN language models have a much richer notion of similarity
across contexts. For example, consider an 13-gram model over code, in which we are trying to
estimate the distribution following the context for(int i=N; i>=0; i--). In a corpus, few examples of this pattern may exist because such long contexts occur rarely. A simple n-gram model
cannot exploit the fact that this context is very similar to for(int j=M; j>=0; j--). But a neural
network can exploit it, by learning to assign these two sequences similar vectors.
Syntactic Models. Syntactic (or structural) code-generating models model code at the level of
abstract syntax trees (ASTs). Thus, in contrast to sequence-based models, they describe a stochastic process of generating tree structures. Such models make simplifying assumptions about how
a tree is generated, usually following generative NLP models of syntactic trees: they start from
a root node, then sequentially generate children top-to-bottom and left-to-right. Syntactic models generate a tree node conditioned on context deﬁned as the forest of subtrees generated so far.
In contrast to sequence models, these models — by construction — generate syntactically correct
code. In general, learning models that generate tree structures is hard: it is computationally expensive, especially for neural models. Additionally, modeling the context of the tree generation
process, represented by a more complicated data structure (compared to sequence models), is an
open problem with no apparent consensus in existing research. In contrast to their wide application in NLP, probabilistic context free grammars (PCFG) have been found to be unsuitable as
language models of code [113, 150]. This may seem surprising, because most parsers assume that
programming languages are context-free. But the problem is that the PCFGs are not a good model
of statistical dependencies between code tokens, because close-by tokens may be far away in the
AST. So it is not that PCFGs do not capture long-range dependencies (n-gram-based models do
not either), but that they do not even capture close-range dependencies that matter. Further, code
ASTs tend to be signiﬁcantly deeper and wider than text parse trees due to the highly compositional nature of the code.
Allamanis et al. [12], Maddison and Tarlow [113] increase the size of the context considered
by creating a non-context-free log-bilinear neural network grammar, using a distributed vector
5 This

work diﬀers from the rest from the fact that it anonymizes/normalizes identiﬁers, creating a less sparse problem.
Because of the anonymization, the results are not directly comparable with other models.

10

Allamanis et al.

representation for the context. Additionally, Maddison and Tarlow [113] restricts the generation to
generate variables that have been declared. To achieve this, they use the deterministically-known
information and ﬁlter out invalid output. This simple process always produces correct code, even
when the network does not learn to produce it. In contrast, Amodio et al. [14] create a signiﬁcantly
more complex model that aims to learn to enforce deterministic constraints of the code generation,
rather than enforcing them on the directly on the output. We further discuss the issue of embedding
constraints and problem structure in models vs. learning the constraints in Section 5.
Bielik et al. [26], Raychev et al. [150] increase the context by annotating PCFGs with a learned
program that uses features from the code. Although the programs can, in principle, be arbitrary,
they limit themselves to synthesizing decision tree programs. Similarly, Wang et al. [172], Yin and
Neubig [182] use an LSTM over AST nodes to achieve the same goal. Allamanis and Sutton [11]
also create a syntactic model learning Bayesian TSGs [40, 144].
Semantic Language Models. Semantic code-generating models view code as a graph hose nodes
represent code elements and whose edges represent the relations among them. this family of models is the more general then the aforementioned models, potentially allowing both syntactic and
semantic structure of source code to be directly modeled. Learning to generate complex graphs
with a rich set of properties is hard, since there is no natural “starting” point. The limited number
of probabilistic generative models of graphs in the literature reﬂects this challenge. Similarly, to
our knowledge there are no generative models that directly generate graph representations of code
(e.g. data-ﬂow models). Nguyen and Nguyen [127] propose a type of generative model, related to
graph generative models in NLP, to suggest API completions. They train their model over API
protocols usages projected out of code. They predict entire graphs as completions and perform
no smoothing, so they assign no probability to unseen API protocol graphs. In these ways, their
model diﬀers from graph generating models in NLP, which predict nodes and use smoothing.
3.1.2 Types of Code Generating Models. We use external context C(c) to reﬁne code generating
models into three subcategories.
Language Models. Language models model the language itself, without using any context, i.e.
C(c) = . Although LMs learn the high-level structure and constraints of programming languages
fairly easily, predicting and generating source code identiﬁers (e.g. variable and method names),
long-range dependencies and taking into account code semantics makes the language modeling
of code a hard and interesting research area. We discuss these and other diﬀerences and their
implications for probabilistic modeling in Section 5
Code LMs are evaluated like LMs in NLP, using perplexity, cross-entropy and word error rate.
Cross-entropy H is the most common measure. Language models — as most predictive machine
learning models — can be seen as compression algorithms where the model predicts the full output
(i.e. decompresses) using extra information. Cross entropy measures the average number of extra
bits of information per token of code that a model needs to decompress the correct output using a
perfect code (in the information-theoretic sense)
H (c, P D ) = −

1
log2 P D (c)
M

(2)

where M is the number of tokens within c. By convention, the average is reported per-token, even
for non-token models. Thus, a “perfect” model, correctly predicting all tokens with probability 1,
would require no additional bits of information because, in a sense, already “knows” everything.
Cross-entropy allows comparisons across diﬀerent models. Other, application-speciﬁc measures,
are used when the LM was trained for a speciﬁc task, such as code completion (Section 4.1).

A Survey of Machine Learning for Big Code and Naturalness

11

Code Transducer Models. Inspired by statistical machine translation (SMT), transducer models
translate/transduce code from one format into another (i.e. C(c) is also code), such as translating
code from one source language into another, target language. They have the form P D (c|s), where
c is the target source code that is generated and C(c) = s is the source source code. Most work
in code transducer models is based on phrase-based machine translation. Intuitively, phrase-based
models assume that small chunks from the source input can directly be mapped to chunks in
the output. Although this assumption is reasonable in NLP and many source code tasks, these
models present challenges in capturing long-range dependencies within the source and target.
For example, as we will mention in the next section, transducing code from an imperative source
language to a functional target is not currently possible because of the source and target are related
with a signiﬁcantly more complicated relation that simply matching “chunks” of the input code to
“chunks” in the output. Naturally, code transducer models are related with language models since
they both need to generate code.
These types of models have found application within code migration [2, 92, 129], pseudocode
generation [134] and code ﬁxing [147]. Traditionally transducer models have followed a noisy
channel model, in which they combine a language model P D (c) of the target language with a translation/transduction model P D (s|c) to match elements between the source and the target. These
methods pick the optimal transduction c ∗ such that c∗ = arg max P D (c|s) = arg max P D (s|c)P D (c),
where the second equality derives from the Bayes equation. Again, these probabilistic generative
models of code do not necessarily produce valid code, due to the simplifying assumptions they
make in both P D (c) and P D (s|c). More recently, machine translation methods based on phrasebased models and the noisy channel model have been outperformed by neural network-based
methods that directly model P D (c|s).
Transducer models can be evaluated with SMT evaluation measures, such as BLEU [138] —
commonly used in machine translation as an approximate measure of translation quality — or
programming-related measures (e.g. “does the translated code parse/compile?”).
Multimodal Models. Code-generating multimodal models correlate code with one or more noncode modalities, such as comments, speciﬁcations, or search queries. These models have the form
P D (c|m) i.e. C(c) = m is a representation of one or more non-code modalities. Multimodal models
are closely related to representational models (discussed in Section 3.2): multimodal code-generating
models learn an intermediate representation of the non-code modalities m and use it to generate
code. In contrast, code representational models create an intermediate representation of the code
but are not concerned with code generation.
Multimodal models of code have been used for code synthesis, where the non-code modalities
are leveraged to estimated a conditional generative model of code, e.g. synthesis of code given a natural language description by Gulwani and Marron [71] and more recently by Yin and Neubig [182].
The latter model is a syntactic model that accepts natural language. Recently, Beltramelli [20] designed a multimodal model that converts a screenshot of a GUI (the non-code modality) into a DSL
describing how it is constructed. Another use of these models is to score the co-appearance of the
modalities, e.g. in code search, to score the probability of some text given a textual query [12].
This stream of research is related to work in NLP and computer vision where one seeks to generate a natural language description for an image. These models are closely related to the other
code generating models, since they generate code. These models make also assume that the input
modality conditions the generation process. Multimodal models combine an assumption with a design choice. Like language models, these models assume that probabilistic models can capture the
process by which developers generate code; unlike language models, they additionally bias code
generation using information from the input modality m. The design choice is how to transform

12

Allamanis et al.

the input modality into an intermediate representation. For example, Allamanis et al. [12] use a
bag-of-words assumption averaging the words’ distributed representations. However, this limits
the expressivity of the models because the input modality has to ﬁt in whole within the distributed
representation. To address this issue, Ling et al. [107] and Yin and Neubig [182] use neural attention mechanisms to selectively attend to information within the input modality without the need
to “squash” all the information into a single representation. This area is closely related to semantic
parsing in NLP, where the input modality m is natural language text and the other modality c is
some form of executable instructions. Because it is restricted to carefully crafted domain-speciﬁc
languages rather than general purpose programming languages, semantic parsing is out-of-scope
for this survey. Neubig [125] surveys relevant publications.
3.2 Representational Models of Source Code
Generative models recapitulate the process of generating source code, but cannot explicitly predict
facts about the code that may be directly useful to engineers or useful for other downstream tasks,
such as static analyses. To solve this problem, researchers have built models to learn intermediate,
not necessarily human-interpretable, encodings of code, like a vector embedding. These models
predict the probability distribution of properties of code snippets, like variable types. We call them
representational code models. They learn the conditional probability distribution of a code property π as P D (π | f (c)), where f is a function that transforms the code c into a target representation.
These models use a diverse set of machine learning methods and are often application-speciﬁc.
Table 2 lists representational code model research.
3.2.1 Distributed Representations. Distributed representations [82] are widely used in NLP to
encode natural language elements. For example, Mikolov et al. [118] learn distributed representations of words, showing that such representations can learn useful semantic relationships and
Le and Mikolov [100] extend this idea to sentences and documents. Distributed representations
refer to arithmetic vectors or matrices where the meaning of an element is distributed across multiple components (e.g. the “meaning” of a vector is distributed in its components). This contrasts
with local representations, where each element is uniquely represented with exactly one component. Distributed representations are commonly used in machine learning and NLP because they
tend to generalize better and have recently become extremely common due to their omnipresence
in deep learning. Models that learn distributed representations assume that the elements being
represented and their relations can be encoded within a multidimensional real-valued space and
that the relation (e.g. similarity) between two representations can be measured within this space.
Probabilistic code models widely use distributed representations. For example, models that use distributed vector representations learn a function of the form c → RD that maps code elements to
a D-dimensional vector. Such representations are usually the (learned) inputs or output of (deep)
neural networks.
Allamanis et al. [6] learn distributed vector representations for variable and methods usage
contexts and use them to predict a probability distribution over their names. Such distributed
representations are quite similar to those produced by word2vec [118]; the authors found that the
distributed vector representations of variables and methods learn common semantic properties,
implying that some form of the distributional hypothesis in NLP also holds for code.
Gu et al. [70] use a sequence-to-sequence deep neural network [163], originally introduced for
SMT, to learn intermediate distributed vector representations of natural language queries which
they use to predict relevant API sequences. Mou et al. [119] learn distributed vector representations
using custom convolutional neural networks to represent features of snippets of code, then they

A Survey of Machine Learning for Big Code and Naturalness

13

Table 1. Research on Source Code Generating Models P D (c|C(c)) (sorted alphabetically), describing the
process of generating source code. References annotated with ∗ are also included in other categories.
Reference

Type

Representation

PD

Application

Aggarwal et al. [2]
Allamanis and Sutton [10]
Allamanis et al. [5]
Allamanis and Sutton [11]∗
Allamanis et al. [12]∗
Amodio et al. [14]
Barone and Sennrich [18]
Beltramelli [20]
Bhatia and Singh [22]
Bhoopchand et al. [23]
Bielik et al. [26]
Campbell et al. [32]
Cerulo et al. [34]

P D (c|s)
P D (c)
P D (c)
P D (c)
P D (c|m)
P D (c)
P D (c|m)
P D (c|m)
P D (c)
P D (c)
P D (c)
P D (c)
P D (c)

Token
Token
Token + Location
Syntax
Syntax
Syntax + Constraints
Token
Token
Token
Token
Syntax
Token
Token

Migration
—
Coding Conventions
—
Code Search/Synthesis
—
Documentation
GUI Code Synthesis
Syntax Error Correction
Code Completion
Code Completion
Syntax Error Detection
Information Extraction

Cummins et al. [45]
Dam et al. [46]
Gulwani and Marron [71]
Gvero and Kuncak [75]
Hellendoorn et al. [78]
Hellendoorn and Devanbu [77]
Hindle et al. [81]
Hsiao et al. [85]
Lin et al. [106]
Ling et al. [107]
Liu [108]
Karaivanov et al. [92]
Karpathy et al. [93]
Kushman and Barzilay [98]
Maddison and Tarlow [113]
Menon et al. [116]
Nguyen et al. [128]
Nguyen et al. [132]
Nguyen et al. [129]
Nguyen and Nguyen [127]
Oda et al. [134]
Patra and Pradel [141]
Pham et al. [142]

P D (c)
P D (c)
P D (c|m)
P D (c)
P D (c)
P D (c)
P D (c)
P D (c)
P D (c|m)
P D (c|m)
P D (c)
P D (c|s)
P D (c)
P D (c|m)
P D (c)
P D (c|m)
P D (c|s)
P D (c)
P D (c|s)
P D (c)
P D (c|s)
P D (c)
P D (c)

Character
Token
Syntax
Syntax
Token
token
Token
PDG
Tokens
Token
Token
Token
Characters
Token
Syntax with scope
Syntax
Token
Token + parse info
Token + parse info
Partial PDG
Syntax + Token
Syntax
Bytecode

Pu et al. [147]
Rabinovich et al. [148]∗
Raychev et al. [152]
Ray et al. [149]
Raychev et al. [150]
Saraiva et al. [159]
Sharma et al. [161]
Tu et al. [166]
Vasilescu et al. [167]
Wang et al. [172]
White et al. [175]
Yadid and Yahav [180]
Yin and Neubig [182]

P D (c|s)
P D (c|m)
P D (c)
P D (c)
P D (c)
P D (c)
P D (c)
P D (c)
P D (c|s)
P D (c)
P D (c)
P D (c)
P D (c|m)

Token
Syntax
Token + Constraints
Token
Syntax
Token
Token
Token
Token
Syntax
Token
Token
Syntax

Phrase
n-gram
n-gram
Grammar (pTSG)
Grammar (NN-LBL)
RNN
Neural MT
NN (Encoder-Decoder)
RNN (LSTM)
NN (Pointer Net)
PCFG + annotations
n-gram
Graphical
Model
(HMM)
NN (LSTM)
NN (LSTM)
Phrase Model
PCFG + Search
n-gram
n-gram (cache)
n-gram
n-gram
NN (Seq2seq)
RNN + Attention
n-gram
Phrase
RNN (LSTM)
Grammar (CCG)
NN
PCFG + annotations
Phrase
n-gram
Phrase SMT
n-gram
Tree-to-String + Phrase
Annotated PCFG
Graphical
Model
(HMM)
NN (Seq2seq)
NN (LSTM-based)
n-gram/ RNN
n-gram (cache)
PCFG + annotations
n-gram
n-gram
n-gram (cache)
Phrase SMT
NN (LSTM)
NN (RNN)
n-gram
NN (Seq2seq)

Benchmark Synthesis
—
Text-to-Code
Code Synthesis
Code Review
–
Code Completion
Program Analysis
Synthesis
Code Synthesis
Obfuscation
Migration
—
Code Synthesis
—
Code Synthesis
Migration
Code Completion
Migration
Code Completion
Pseudocode Generation
Fuzz Testing
Code Completion
Code Fixing
Code Synthesis
Code Completion
Bug Detection
Code Completion
—
Information Extraction
Code Completion
Deobfuscation
Code Completion
—
Information Extraction
Synthesis

14

Allamanis et al.

assume that student solutions to various coursework problems have been intermixed and seek to
recover the solution-to-problem mapping via classiﬁcation.
Li et al. [103] learn distributed vector representations for the nodes of a memory heap and
use the learned representations to synthesize candidate formal speciﬁcations for the code that
produced the heap. Li et al. [103] exploit heap structure to deﬁne graph neural networks, a new
machine learning model based on GRUs (a type of RNN) to directly learn from heap graphs. Piech
et al. [143] and Parisotto et al. [139] learn distributed representations of source code input/output
pairs and use them to assess and review student assignments or to guide program synthesis from
examples.
Neural code-generative models of code also use distributed representations to capture context,
a common practice in NLP. For example, the work of Maddison and Tarlow [113] and other neural
language models (e.g. LSTMs in Dam et al. [46]) describe context distributed representations while
sequentially generating code. Ling et al. [107] and Allamanis et al. [12] combine the code-context
distributed representation with a distributed representations of other modalities (e.g. natural language) to synthesize code. While all of these representations can, in principle, encode unbounded
context, handling all code dependencies of arbitrary length is an unsolved problem. Some neural architectures, such as LSTMs [83], GRUs [38] and their variants, have made progress on this
problem and handle moderately long-range dependencies.
3.2.2 Structured Prediction. Structured prediction refers to the problem of joint prediction of
a set of interrelated variables where an underlying structure, e.g. characterized by a graphical
model, deﬁnes the relationships among the variables. Such problems have been widely studied
within machine learning and NLP and are omnipresent in code. In contrast to text, since code is
unambiguous6 , the internal structure of code and its elements can give rise to signiﬁcantly richer
structures and relations that are rarely present in text.
Structured prediction is a very general framework that usually involves predicting multiple
variables and graphical modeling is one common way of representing such problems. These models
make assumptions about the underlying structure and relations of the properties being modeled.
Raychev et al. [151] represent code as a variable dependency network, represent each JavaScript
variable as a single node, and model their pairwise interactions as a conditional random ﬁeld (CRF).
They train the CRF to jointly predict the types and names of all variables within a snippet of code.
Proksch et al. [146] use a directed graphical model to represent the context of an (incomplete)
usage of an object to suggest a method invocation (viz. constructor) autocompletion in Java.
Structured prediction, such as predicting a sequence of elements, can be combined with distributed representations. For example, Allamanis et al. [9] and Allamanis et al. [6] use distributed
representations to predict sequences of identiﬁer sub-tokens to build a single token and Gu et al. [70]
predict the sequence of API calls. Li et al. [103] learn distributed representations for the nodes
of a ﬁxed heap graph by considering its structure and the interdependencies among the nodes.
Kremenek et al. [96] use a factor graph to learn and enforce API protocols, like the resource usage
speciﬁcation of the POSIX ﬁle API, as do Livshits et al. [109] for information ﬂow problems. Allamanis and Brockschmidt [7] predict the data-ﬂow graph of code by learning to paste snippets of
code into existing code and adapting the variables used.
3.3 Pattern Mining Models of Source Code
Pattern mining models aim to discover a ﬁnite set of human-interpretable patterns from source
code, without annotation or supervision, and present the mined patterns to software engineers.
6 With

usual caveats, like ignoring undeﬁned behavior.

A Survey of Machine Learning for Big Code and Naturalness

15

Table 2. Research on Representational Models of Source Code P D (π | f (c)) (sorted alphabetically). References
annotated with ∗ are also included in other categories. GM refers to graphical models
Reference

Input Code Representation (c)

Target (π )

Intermediate Representation (f )

Application

Allamanis et al. [6]
Allamanis et al. [12]∗

Token Context
Natural
Language
Tokens
Data Flow
Dependency Net
Partial
Object
Use
Data Flow Graph
Tokens
Tokens

Identiﬁer Name
LM (Syntax)

Distributed
Distributed

Naming
Code Search

Method Name
Variable Allocation
Identiﬁer Name
Invoked Method

Distributed
Distributed
CRF (GM)
Localized

Naming
Contextualization
Deobfuscation
Code Completion

Static Analysis
Feature Location
Optimization Flags

Localized
Distributed
Distributed

LM (Tokens)
API Calls

Distributed
Distributed

Program Analysis
Feature Location
Optimization Heuristics
—
API Search

Traceability link
Code Fix
Natural Language
Natural Language
Natural Language
False Positives
Ownership
Alignment
Separation Logic
Natural Language
LM (AST)
Prob. Analysis
Code Comments
Classiﬁcation
API Calls
Expressions

Distributed
Distributed
Distributed
Distributed
Distributed
Distributed
Factor (GM)
Distributed
Distributed
Distributed
Distributed
MaxSAT
Directed GM
Distributed
Distributed
Directed GM

Traceability
Code Fixing
Summarization
Summarization
Commit Message
Program Analysis
Pointer Ownership
Decompiling
Veriﬁcation
Explain code changes
—
Program Analysis
Comment Prediction
Task Classiﬁcation
Migration
Code Completion

Analysis Params
Student Feedback
Object Usage
LM (AST)
Types + Names
Defects
LM (Tokens)
—
Execution Trace

Static Analysis
Distributed
Directed GM
Distributed
CRF (GM)
LM (n-gram)
Distributed
Distributed
Distributed

Program Analysis
Student Feedback
Code Completion
Code Synthesis
Types + Names
Bug Detection
—
Clone Detection
—

Allamanis et al. [9]
Allamanis and Brockschmidt [7]
Bichsel et al. [24]
Bruch et al. [30]
Chae et al. [35]
Corley et al. [41]
Cummins et al. [44]
Dam et al. [46]∗
Gu et al. [70]
Guo et al. [73]
Gupta et al. [74]
Hu et al. [86]
Iyer et al. [88]
Jiang et al. [89]
Koc et al. [95]
Kremenek et al. [96]
Levy and Wolf [102]
Li et al. [103]
Loyola et al. [111]
Maddison and Tarlow [113]∗
Mangal et al. [114]
Movshovitz-Attias and Cohen [120]
Mou et al. [119]
Nguyen et al. [130]
Omar [136]
Oh et al. [135]
Piech et al. [143]
Proksch et al. [146]
Rabinovich et al. [148]∗
Raychev et al. [151]
Wang et al. [170]
White et al. [175]∗
White et al. [174]∗
Zaremba and Sutskever [183]

Token Context
Natural
Language
Tokens
Tokens
Linearized AST
Tokens
Tokens (Diﬀ)
Bytecode
Partial PDG
Statements
Memory Heap
Tokens (Diﬀ)
LM AST Context
Logic + Feedback
Tokens
Syntax
API Calls
Syntactic
Context
Features
Syntax + State
Inc. Object Usage
LM AST Context
Dependency Net
Tokens
Tokens
Token + AST
Characters

Broadly, these models cluster source code into a ﬁnite set of groups. Probabilistic pattern mining
models of code infer the likely latent structure of a probability distribution
Õ
P D (f (c)) =
P D (f (c)|l)P(l)
(3)
l

where f is a deterministic function that returns a (possibly partial, e.g. API calls only) view of the
code and l represents a set of latent variables that the model introduces and aims to infer. Applications of such models are common in the mining software repositories community and include
documentation (e.g. API patterns), summarization, and anomaly detection. Table 3 lists this work.

16

Allamanis et al.

Table 3. Research on Pattern Mining Probabilistic Models of Source Code (sorted alphabetically). These models have the general form P D (f (c)). References annotated with ∗ are also included in other categories.
Reference

Code
(c)

Representation

Allamanis and Sutton [11]∗
Allamanis et al. [4]
Fowkes and Sutton [57]
Murali et al. [122]
Murali et al. [123]
Movshovitz-Attias and Cohen [121]
Nguyen et al. [131]
Fowkes et al. [56]
Wang et al. [171]
White et al. [174]∗

Syntax
Abstracted AST
API Call Sequences
Sketch Synthesis
API Usage Errors
Tokens
API Usage
Tokens
Serialized ASTs
Token & Syntax

Representation (f )

Application

Graphical Model
Graphical Model
Graphical Model
Graphical Model
Graphical Model
Graphical Model
Distributed
Graphical Model
Distributed
Distributed

Idiom Mining
Semantic Idiom Mining
API Mining
Sketch Mining
Defect Prediction
Knowledge-Base Mining
API Mining
Code Summarization
Defect Prediction
Clone Detection

Unsupervised learning is one of the most challenging areas in machine learning. This hardness
stems from the need to automatically distinguish important patterns in the code from spurious
patterns that may appear to be signiﬁcant because of limited and noisy data. When designing unsupervised models, the core assumption lies in the objective function being used and often we
resort to using a principle from statistics, information theory or a proxy supervised task. Like all
machine learning models, they require assumptions about how the data is represented. An important issue with unsupervised methods is the hardness of evaluating the output, since the quality of
the output is rarely quantiﬁable. A vast literature on non-probabilistic methods exploits data mining
methods, such as frequent pattern mining and anomaly detection. We do not discuss these models
here, since they are not probabilistic models of code. Classic probabilistic topic models [27], which
usually views code (or other software engineering artifacts) as a bag-of-words, have also been
heavily investigated. Since these models and their strengths and limitations are well-understood,
we omit them here.
Allamanis and Sutton [11] learn a tree substitution grammar (TSG) using Bayesian nonparametrics, a technique originally devised for natural language grammars. TSGs learn to group commonly
co-appearing grammar productions (tree fragments). Although TSGs have been used in NLP to improve parsing performance (which is ambiguous in text), Allamanis and Sutton [11] observe that
the inferred fragments represent common code usage conventions and name them idioms. Later,
Allamanis et al. [4] extend this technique to mine semantic code idioms by modifying the input
code representation and adapting the inference method.
In a similar fashion, Fowkes and Sutton [57] learn the latent variables of a graphical model to
infer common API usage patterns. Their method automatically infers the most probable grouping
of API elements. This is in stark contrast to frequency-based methods [178] that suﬀer from ﬁnding
frequent but not necessarily “interesting” patterns. Finally, Movshovitz-Attias and Cohen [121]
infer the latent variables of a graphical model that models a software ontology.
As in NLP and machine learning in general, evaluating pattern mining models is hard, since the
quality of the discovered latent structure is subjective. Thus, researchers often resort to extrinsic,
application-speciﬁc measures.
4

APPLICATIONS

Probabilistic models of source code have found a wide range of applications in software engineering and programming language research. These models enable the principled use of probabilistic

A Survey of Machine Learning for Big Code and Naturalness

17

reasoning to handle uncertainty. Common sources of uncertainty are underspeciﬁed or inherently
ambiguous data (such as natural language text). In some domains, probabilistic source code models also simplify or accelerate analysis tasks that would otherwise be too computationally costly
to execute. In this section, our goal is to explain the use of probabilistic models in each area, not
review them in detail. We describe each area’s goals and key problems, then explain how they can
beneﬁt from probabilistic, machine learning based methods, and how the methods are evaluated.
4.1

Recommender Systems

Software engineering recommender systems [155, 156] make recommendations to assist software
engineering tasks. Many of these systems employ data mining and machine learning approaches
on various software engineering artifacts. Probabilistic models of code ﬁnd application in source
code-based recommender systems [117], such as those that aid developers write or maintain code.
Modeling developer intent is a challenge: even if there were an agreed upon way to formalize
intent, developers are reluctant to formalize their intent separately from their code itself. Probabilistic reasoning is well-suited for inferring intent, since it allows us to quantify the uncertainty
that is inherent to inferring any latent variable. Probabilistic recommender systems extract information from the context of (partial) code and use it to probabilistically reason about developer
intent.
The most prominent recommender system and a feature commonly used in integrated development environment (IDEs) is code completion. All widely used IDEs, such as Eclipse, IntelliJ and
Visual Studio, have some code completion features. According to Amann et al. [13], code completion is the most used IDE feature. However, code completion tools typically return suggestions
in alphabetic order, rather than in relative order of predicted relevance to the context. Statistical
code completion aims to improve suggestion accuracy by learning probabilities over the suggestions and providing to the users a ranked list. Some systems focus on automatically completing
speciﬁc constructs (e.g. method calls and parameters); others try to complete all code tokens. In all
cases, probabilistic code completion systems use existing code as their training set.
Statistical code completion was ﬁrst studied by Bruch et al. [30] who extracted features from
code context to suggest completions for method invocations and constructors. Later, Proksch
et al. [146] used Bayesian graphical models (structured prediction) to improve accuracy. This
context-based model captures all usages of an object and models the probability distribution for
the next call. A version of this research is integrated into the Eclipse IDE under Eclipse Recommenders [55].
Source code language models have implicitly and explicitly been used for code completion. Hindle et al. [81] were the ﬁrst to use a token-level n-gram LM for this purpose, using the previous
n − 1 tokens to represent the completion context at each location. Later, Franks et al. [58], Tu
et al. [166] used a cache n-gram LM and further improved the completion performance, showing
that a local cache acts as a domain adapted n-gram. Nguyen et al. [132] augment the completion context with semantic information, improving the code completion accuracy of the n-gram
LM. Raychev et al. [152] exploit formal properties of the code in context to limit incorrect (but
statistically probable) API call suggestions. Their method is the ﬁrst to depart from simple statistical token completion towards statistical program synthesis of single statements. Apart from
token-level language models for code completion, Bielik et al. [26] and Maddison and Tarlow [113]
create AST-level LMs that can be used for suggestion.
In contrast to work that predicts source code, Movshovitz-Attias and Cohen [120] create a recommender system to assist comment completion given a source code snippet, using a topic-like
graphical model to model context information. Similarly, the work of Allamanis et al. [5, 6, 9] can

18

Allamanis et al.

be seen as a recommender systems for suggesting names for variables, methods, and classes that
uses relevant code tokens as the context.

Evaluation Measures. To date, research in recommender systems has used these measures: (1) Accuracy for rank is the percentage of suggestion points whose top k suggestions contain the correct
prediction; (2) Mean reciprocal rank is the average reciprocal rank of the correct recommendation
at each suggestion point; (3) Cross entropy quantiﬁes the divergence between two probability distributions; (4) Keystrokes saved compute the number of keystrokes saved by using a code completion;
and (5) Memory and speed measure the memory requirements and the time needed for computing
(real-time) recommendations.
These measures estimate the predictive ability of recommender systems. Unfortunately, they
are imprecise. For instance, probabilistic recommender systems deﬁne a probability density over
suggestions whose cross-entropy can be computed against the empirical distribution in test data.
Although cross entropy is correlated with suggestion accuracy and conﬁdence, small improvements in cross entropy may not improve accuracy. Sometimes the imprecision is due to unrealistic
use case assumptions. For example, the measures for LM-based code completion tend to assume
that code is written sequentially, from the ﬁrst token to the last one. However, developers rarely
write code in such a simple and consistent way [145]. Context-based approaches assume that the
available context (e.g. other object usages in the context) is abundant, which is not true in a real
editing scenarios. Researchers reporting keystrokes saved have usually assumed that code completion suggestions are continuously presented to the user as she is typing. When the top suggestion
is the target token, the user presses a single key (e.g. return) to complete the rest of the target.

4.2

Inferring Coding Conventions

Coding conventions are syntactic constraints on code beyond those imposed by the grammar of
a programming language. They govern choices like formatting (brace and newline placement) or
Hungarian notation vs. Camel case naming. They seek to prevent some classes of bugs and make
code easier to comprehend, navigate, and maintain [5]. In massive, open, online courses, coding
conventions help teachers identify and understand common student errors [65]. Enforcing coding
conventions is tedious. Worse, it is sometimes diﬃcult to achieve consensus on what they should
be, a prerequisite for their codiﬁcation in rule-based systems. Inferring coding conventions with
machine learning solves this problem by learning emergent conventions directly from a codebase.
This can help software teams to determine the coding conventions a codebase uses without the
need to deﬁne rules rule upfront or conﬁgure existing convention enforcing tools.
Machine learning models of source code that look at the surface structure (e.g. tokens, syntax)
are inherently well-suited for this task. Using the source code as data, they can infer the emergent
conventions while quantifying uncertainty over those decisions. An important challenge in this
application domain is the sparsity of the code constructs, caused by the diverse and non-repeatable
form of source code within projects and domains. Allamanis et al. [5, 6, 9] exploit the statistical
similarities of code’s surface structure to learn and suggest variable, method, and class naming
conventions, while Allamanis and Sutton [11] and Allamanis et al. [4] mine conventional syntactic
and semantic patterns of code constructs that they call idioms. They show that these idioms are
useful for documentation and can help software engineering tool designers achieve better coverage
of their tools. To handle code formatting conventions, Parr and Vinju [140] learn a source code
formatter from data by using a set of hand-crafted features from the AST and a k-NN classiﬁer.

A Survey of Machine Learning for Big Code and Naturalness

19

4.3 Code Defects
Probabilistic models of source code assign high probability to code that appears often in practice,
i.e. is natural. Therefore, code considered very improbable may be buggy. This is analogous to
anomaly detection using machine learning [36]. Finding defects is a core problem in software
engineering and programming language research. The challenge in this domain rests in correctly
characterizing source code that contains a defects with high precision and recall. This is especially
diﬃcult because of the rarity of defects and the extreme diversity of (correct) source code.
Preliminary work suggests that the probability assigned by language models can indicate code
defects. Allamanis and Sutton [10] suggest that n-gram LMs can be seen as complexity measures
and Ray et al. [149] present evidence that buggy code tends to have lower probability (is less
“natural”) than correct code and show that LMs ﬁnd defects as well as popular tools like FindBugs.
Wang et al. [171] use deep belief networks to automatically learn token-level source code features that predict code defects. Fast et al. [53] and Hsiao et al. [85] learn statistics from large
numbers of code to detect potentially erroneous code and perform program analyses while Wang
et al. [170] learn coarse-grained n-gram language models to detect uncommon usages of code.
These models implicitly assume that a simple set of statistics or an LM can capture anomalous/unusual
contexts. Recently, Murali et al. [123] use a combination of topic models to bias a recurrent neural
network that models the sequences of API calls in a learned probabilistic automaton. They use the
model to detect highly improbable sequences of API calls detecting real-world bugs in Android
code.
Because of the sparsity of source code, work on detecting code defects uses diﬀerent abstraction levels of source code. For example, Wang et al. [170] create coarse-grained n-grams, while
Murali et al. [123] focus on possible paths (that remove control ﬂow dependencies) over API calls.
Therefore, each model captures a limited family of defects, determined by the model designers’
choice of abstraction to represent. Pu et al. [147] and Gupta et al. [74] create models for detecting
and ﬁxing defects but only for student submissions where data sparsity is not a problem. Other
data-mining based methods (e.g. Wasylkowski et al. [173]) also exist, but are out-of-scope from
this review since they do not employ probabilistic methods.
Also related is the work of Campbell et al. [32] and Bhatia and Singh [22]. These researchers
use source code LMs to identify and correct syntax errors. Detecting syntax errors is an easier and
more well deﬁned task. The goal of these models is not to detect the existence of such an error
(that can be deterministically found) but to eﬃciently localize the error and suggest a ﬁx.
The earlier work of Liblit et al. [105], Zheng et al. [185] use traces for statistical bug isolation.
Kremenek et al. [96] learn factor graphs (structured prediction) to model resource-speciﬁc bugs
by modeling resource usage speciﬁcations. These models use an eﬃcient representation to capture bugs, but can fail on interprocedural code that requires more complex graph representations.
Finally, Patra and Pradel [141] use an LM of source code to generate input for fuzz testing browsers.
Not all anomalous behavior is a bug, but bugs in useful code almost certainly is [51]. Thus,
probabilistic models of source code seem a natural ﬁt for ﬁnding defective code. They have not,
however, seen much industrial uptake. One possible cause is their imprecision. The vast diversity of
code constructs entails sparsity, from which all anomaly detection methods suﬀer. Methods based
on probabilistic models are no exception: they tend to consider rare, but correct, code anomalous.

4.4

Code Migration, Porting, and Clones

Code migration is the movement of code from one context to another. This includes cloning code,
translating code from one (programming) language into another, and porting code across systems.

20

Allamanis et al.

The success of machine translation among (natural) languages has inspired researchers to use
machine learning to translate code from one source language (e.g. Java) to another (e.g. C#). Although rule-based rewriting systems can be (and have been) used, it is tedious to create and maintain these rules. Statistical machine translation models are well suited for this task, although they
tend to produce invalid code. To reduce these errors during translation Karaivanov et al. [92] and
Nguyen et al. [129] add semantic constraints to the translation process.
Existing research has applied machine learning models widely used in the machine translation
of text. Although these models learn mappings between diﬀerent language constructs such as APIs,
they have only been used for translating between programming languages of similar paradigms
and structure (C# and Java are both object-oriented languages with managed memory). This is an
important limitation; machine learning innovations are required to translate between languages
of diﬀerent types (e.g. Java to Haskell or assembly to C) or even languages with diﬀerent memory
management (e.g. Java to C). Existing per-statement translation from Java to C would not track
memory allocations and therefore fail to emit memory de-allocations that C’s lack of a garbage
collection requires. Similarly, translating object-oriented code to functional languages would require learning the conceptual diﬀerences of the two paradigms while preserving semantics, such
as learning to translate a loop to a map-reduce functional. Researchers evaluate translation models
by scoring exact matches, measuring the syntactic or semantic correctness of the translated code,
or using BLEU [138]. We discuss this and other measure-related issues in Section 5.
Code porting and cloning have also been tackled with machine learning techniques. Allamanis
and Brockschmidt [7] use structured prediction and distributed representations to adapt a pasted
(ported) snippet’s variables into the existing context. Their method allows them to probabilistically
represent semantic information about variable use, which allows to predict the correct adaptation
without any external information (e.g. tests). Code clones, a lightweight but common form of code
migration, have also been studied by White et al. [174]. The authors use autoencoders [66] and
recurrent neural networks to detect code clones by ﬁnding snippets of code that share similar
distributed representations. Using distributed vector representations allows them to learn a continuous similarity metric between diﬀerent code locations, rather than using measures that rely
on matching code constructs exactly.

4.5

Code to Text Translation and Traceability

Linking natural language text to source code has many useful applications, such as program synthesis, search and documentation. However, the diversity of text and code, the ambiguity of text,
the compositional nature of code and the layered abstractions in software make interconnecting
text and code a hard problem. Probabilistic machine learning models provide a principled method
for probabilistically modeling and resolving ambiguities both in text and in code.
Generating source code using natural language and natural language given source code is one
of the core problems in this application area. Oda et al. [134] translate Python code to pseudocode
(in natural language) and use the BLEU score to evaluate their approach. Recently, Yin and Neubig [182] presented a neural architecture for general-purpose code generation. Neubig [125] provides an additional survey of code-generation methods. For the task of converting source code to
text, Iyer et al. [88] design a neural attention model that summarizes code as text. They evaluate
their summaries with BLEU and their model as a code search model with MRR. Gulwani and Marron [71] synthesize Excel formulas from natural language and evaluate on the rank of the correctly
synthesized program. Movshovitz-Attias and Cohen [120] create a comment-generative model and
evaluate its suggestion performance.

A Survey of Machine Learning for Big Code and Naturalness

21

This area is closely related to semantic parsing in NLP. Semantic parsing is the task of converting
a natural language sentence into a representation of its meaning, often a DSL. A classic example
of a semantic parsing problem is to translate a natural language question into an SQL query that
would return an answer to the query if applied to an appropriate database. This has been the focus
of a large body of work in NLP [90]. Because it focuses on parsing and extracting meaning from
text (not software artifacts) and generates code only as output, we do not discuss it further. In NLP,
grounded language refers to the notion that language must ground its referents to objects in the
real world, for example connecting the word “glass” to a physical glass in a speaker’s visual ﬁeld.
Programming languages provide an ideal setting in which to study language in a grounded setting,
because the semantics of code is deﬁned precisely, to a ﬁrst approximation.
Code search — a common activity for software engineers [13, 158] — can employ natural language queries. Software engineering researchers have focused on the code search problem using
information retrieval (IR) methods [62, 84, 115, 164]. Niu et al. [133] has used learning-to-rank
methods but with manually extracted features. Within the area of statistical models of source
code, Gu et al. [70] train a sequence-to-sequence neural network to map natural language into
API sequences. Allamanis et al. [12] learn a bimodal, generative model of code, conditioned on
natural language text and use it to rank code search results. All of these methods use rank-based
measures (e.g. mean reciprocal rank) to evaluate their performance.
Documentation is text that captures requirements, speciﬁcations, and descriptions of code. Engineers turn to it to prioritize new features and to understand code during maintenance. Searching, formalizing, reasoning about, and interlinking code to (i.e. the traceability problem of Gotel et al. [68]) documentation are seminal software engineering problems. Mining common API
patterns is a recurring theme and there is a large literature of non-probabilistic methods (e.g.
frequency-based) for mining and synthesizing API patterns [31, 178], which are out-of-scope of
this review. Also out-of-scope is work that combines natural language information with APIs. For
example, Treude and Robillard [165] extract phrases from StackOverﬂow using heuristics (manually selected regular expressions) and use oﬀ-the-self classiﬁers on a set of hand-crafted features.
We refer the reader to Robillard et al. [155] for all probabilistic and non-probabilistic recommender
systems. Within this domain, there are a few probabilistic code models that mine API sequences.
Gu et al. [70] map natural language text to commonly used API sequences, Allamanis and Sutton [11] learn ﬁne-grained source code idioms, that may include APIs. Fowkes and Sutton [57]
uses a graphical model to mine interesting API sequences.
Documentation is also related to information extraction from (potentially unstructured) documents. Cerulo et al. [34] use a language model to detect code “islands” in free text. Sharma
et al. [161] use a language model over tweets to identify software-relevant tweets.

4.6 Program Synthesis
Program synthesis is concerned with generating full or partial programs from a speciﬁcation [72].
Traditionally, a speciﬁcation is a formal statement in an appropriate logic. More recently, researchers
have considered partial or incomplete speciﬁcations, such as input/output pairs [99] or a natural
language description. Program synthesis generates full or partial programs from a speciﬁcation.
When the speciﬁcation is a natural language description, this is the semantic parsing task (see
Section 4.5). Program synthesis (e.g. from examples or a speciﬁcation) has received a great deal
of attention in programming language research. The core challenge is searching the vast space of
possible programs to ﬁnd one that complies with the speciﬁcation. Probabilistic machine learning
models help guiding the search process to more probable programs.

22

Allamanis et al.

Research on programming by example (PBE) leverages machine learning methods to synthesize
code. Liang et al. [104] use a graphical model to learn commonalities of programs across similar
tasks with the aim to improve program synthesis search. Menon et al. [116] use features from
the input/output examples to learn a parameterized PCFG to speed up synthesis. Singh and Gulwani [162] extract features from the synthesized program to learn a supervised classiﬁer that can
predict the correct program and use it to re-rank synthesis suggestions. The recent work of Balog
et al. [17] and Parisotto et al. [139] combine ideas from existing enumerative search techniques
with learned heuristics to learn to eﬃciently synthesize code, usually written within a DSL. Neelakantan et al. [124] and Reed and de Freitas [153] introduce neural diﬀerentiable architectures for
program induction. This is an interesting emerging area of research, but does not yet scale to the
types of problems considered by the programming language and software engineering community
[63]; also see Devlin et al. [49] for a comparison of neural program induction and program synthesis methods. Finally, the code completion work of Raychev et al. [152] can be seen as a limited
program synthesis of method invocations at speciﬁc locations.
Although program synthesis is usually referred in the context of generating a program that complies to some form of speciﬁcation, probabilistic models have been used to synthesize random —
but functioning — programs for benchmarks and compiler fuzzing. Cummins et al. [45] synthesize
automatically a large number of OpenCL benchmarks by learning a character-level LSTM over
valid OpenCL code. Their goal is to generate reasonable-looking code, rather than synthesize a
program that complies with a speciﬁcation. To ease their task, they normalize the code by consistently alpha-renaming variables and method names. Finally, they ﬁlter invalid intermediate output
so they, in the end, generate only valid programs. In a similar manner, Patra and Pradel [141] synthesize JavaScript programs for fuzz testing JavaScript interpreters.
4.7 Program Analysis
Program analysis is an important area that seeks to soundly extract semantic properties, like correctness, from programs. Sound analyses, especially those that scale, can be imprecise and return
unacceptably large numbers of false positives. Probabilistic models of code use probabilistic reasoning to alleviate these problems. Three distinct program analysis approaches have exploited
probabilistic models of source code.
First, a family of models relaxes the soundness requirement, yielding probabilistic results instead.
Raychev et al. [151] use a graphical model to predict the probability distribution of JavaScript
variable types by learning statistical patterns of how code is used. Oh et al. [135] and Mangal
et al. [114] use machine learning models to statistically parameterize program analyses to reduce
false positive ratio while maintaining high precision. Chae et al. [35] reduce automatically (without
machine learning) a program to a set of data-ﬂow graphs, manually extract features from them.
Using these features they then learn the appropriate parametrization of a static analysis, using a
traditional classiﬁer. Koc et al. [95] train a classiﬁer, using LSTMs, to predict if a static analysis
warning is a false positive. The neural network learns common patterns that can discriminate
between false and true positives. The second paradigm that has been explored is to use machine
learning to create models that produce plausible hypotheses of formal veriﬁcation statements that
can be proved. Brockschmidt et al. [28] and Li et al. [103] propose a set of models that generate
separation logic expressions from the heap graph of a program, suitable for formally verifying its
correctness. The third paradigm — although not yet applied directly to source code but to other
forms of formal reasoning — learns heuristics to speed-up the search for a formal automated proof.
The goal of such methods is to replace hard-coded heuristics with a learnable and adaptive module
that can prioritize search tactics per-problem without human intervention. Alemi et al. [3] and

A Survey of Machine Learning for Big Code and Naturalness

23

Loos et al. [110] take a ﬁrst step towards this direction by learning heuristic for automated theorem
proving for mathematical expressions.
5 CHALLENGES AND FUTURE DIRECTIONS
The development and analysis of code must contend with uncertainty, in many forms: “What is
the purpose of this code?”, “What does functionality does this unit test test?”, “From this program,
can we infer any of its speciﬁcation?”, “What is this program’s intended (or likely) input domain?”
or “Why did this program crash here?”. This contrasts with traditional program analysis which is
conservative: it deems that a program has a property, like a bug, if that property is possible, independent of likelihood. In contrast, machine learning studies robust inference under uncertainty
and noise. Thus, the application of machine learning to code and its development, is an emerging
research topic with the potential to inﬂuence programming language and software engineering
research. Here we list topics where principled probabilistic reasoning promises new advances, focusing on probabilistic models of code. We also list open challenges and problems, some quite
longstanding like long range dependency, and speculate about directions for making progress on
their resolution.
For each open problem, machine learning is introduced to handle uncertainty, ambiguity, or
avoid hard-coded heuristics. Probabilistic learning systems model these as noise which they handle robustly by training statistical principled models. This in turn has allowed the creation of new
previously impossible systems (e.g. text-to-code systems) or replaced existing, hard-coded heuristics with machine learning systems that promise to be more robust and generalize better. This
course of direction highly resembles that of NLP, where hard-coded “expert” systems have been
successfully replaced by sophisticated machine learning methods.
5.1 The Third Wave of Machine Learning
The ﬁrst wave of machine learning for source code applied oﬀ-the-shelf machine learning tools
with hand-extracted features. The second wave, reviewed here, avoids manual feature extraction
and uses the source code itself within machine learning heavily drawing inspiration from existing
machine learning methods in NLP and elsewhere. The third wave promises new machine learning
models informed by programming language semantics. What form will it take?
At the time of this writing, machine learning, and deep learning in particular, is enjoying rockstar status among research ﬁelds. Despite its current (perhaps ephemeral) popularity, it is not
a panacea. In some cases, a machine learning model may not be required (e.g. when the problem is deterministic) and, in other cases, simple models can outperform advanced, oﬀ-the-self
deep learning methods, designed for non-code domains [59, 77]. Furthermore, over-engineering
or under-engineering machine learning models usually leads to suboptimal results. Selecting a machine learning model for a speciﬁc problem necessitates questioning if a speciﬁc model is ﬁt for the
target application. Strong baseline methods are needed to estimate if a performance improvement
justiﬁes the added complexity of a model. In short, the right tools should be used for the right job,
and machine learning is no exception.
Bridging Representations and Communities. Programming language research and practice use
a well deﬁned and widely useful set of representations, usually in a symbolic form. In contrast,
machine learning research customarily works with continuous representations. Bridging the gap
between these representations by allowing machine learning systems to reason with programming
language representations, is an important challenge. Handling unambiguous code has already led
to a combination of probabilistic methods with “formal” constraints, that limit the probabilistic

24

Allamanis et al.

model to valid code. For example, Maddison and Tarlow [113] limit their model to generate syntactically correct and scope-respecting code while Raychev et al. [151] easily create a highly-structured
model of a program taking advantage of its unambiguous form. Introducing better representations
that bridge the gap between machine learning and source code will allow the probabilistic models
of code to reason about the rich structure and semantics of code, without resorting to proxies. The
core problem in this area is the lack of understanding of machine learning from the programming
language community and vica-versa. At the same time, new machine learning methods that can
handle programming language structures in its full complexity at scale need to be researched.
A major obstacle here is engineering systems that eﬃciently and eﬀectively combine the probabilistic world of machine learning and the formal, logic-based world of code analysis. One approach, taken by several authors [8, 114, 151, 157], is to relax formal systems into probabilistic.
Such systems, however, lack the guarantees formal systems (e.g. soundness) often provide. Alemi
et al. [3], Balog et al. [17] and Loos et al. [110] follow a second approach that maintains soundness:
they learn input and context-speciﬁc heuristics that eﬃciently guide search-based methods, such
as theorem proving and code synthesis.
All of approaches to source code modeling must decide whether to explicitly model the structure
and constraints of source code or to rely on general methods with adequate capacity. One one hand,
using well-known, domain-generic machine learning methods has the advantage that the models
are well-understood and rarely require signiﬁcant eﬀort or expertise to apply. On the other hand,
designing models with built-in inductive biases for the problem domain usually performs better
with less data, at the cost of manually designing and debugging domain (even problem-speciﬁc)
networks. One such promising direction is modular neural network architectures. Such architectures decompose the network into components that are combined based on the problem instance.
For source code models, such architectures can derive its structure through static analyses. These
architectures have been useful for visual question answering in NLP. Andreas et al. [15] create
neural networks by composing them from “neural modules”, based on the input query structure.
Similarly, we believe that such architectures will be useful within probabilistic models of source
code. An early example is the work of Allamanis and Brockschmidt [7] who design a neural network based on the output of data ﬂow analysis. Such architectures should not only be eﬀective
for bridging representations among communities but — as we will discuss next — can combat issues with compositionality, sparsity and generalization. Nevertheless, issues that arise in static
analyses, such as path explosion, will still need to be addressed.
Data Sparsity, Compositionality and Strong Generalization. The principle of reusability in software engineering creates a form of sparsity in the data, where it is rare to ﬁnd multiple source
code elements that perform exactly the same tasks. For example, it is rare to ﬁnd hundreds of
database systems, whereas one can easily ﬁnd thousands of news articles on a popular piece of
news. The exceptions, like programming competitions and student solutions to programming assignments, are quite diﬀerent from industrial code. This suggests that there are many opportunities
for researching machine learning models and inference methods that can handle and generalize
from the highly-structured, sparse and composable nature of source code data. Do we believe in
the unreasonable eﬀectiveness of data [76]? Yes, but we do not have suﬃcient data.
Although code and text are both intrinsically extensible, code pushes the limit of existing machine learning methods in terms of representing composition. This is because most natural language methods rarely deﬁne novel, previously unseen, terms, with the possible exception of legal
and scientiﬁc texts. In contrast, source code is inherently extensible, with developers constantly
creating new terms (e.g. by deﬁning new functions and classes) and combining them in still higher

A Survey of Machine Learning for Big Code and Naturalness

25

level concepts. In machine learning, generating novel terms and phrases in called compositionality, because it constructs novel terms by composing elements into new ones. Recent work [79]
has shown that deep learning architecture can learn some aspects of compositionality in text. Machine learning for highly compositional objects remains challenging, because it has proven hard
to capture relations between objects, especially across abstraction levels. Such challenges arise
even when considering simple code-like expressions [8]. However, if suﬃcient progress is to be
made, representing source code artifacts in machine learning will improve signiﬁcantly, positively
aﬀecting other downstream tasks. For example, learning composable models that can combine
meaningful representations of variables into meaningful representations of expressions and functions will lead to much stronger generalization performance.
Data sparsity is still an important and unsolved problem. Although ﬁnding a reasonably large
amount of source code is relatively easy, it is increasingly hard to retrieve sophisticated representations of source code. Indeed, it is infeasible even to compile all of the projects in a corpus of
thousands of projects, because compiling a project requires understanding how the project handles
external dependencies, which can sometimes be idiosyncratic. Furthermore, computing or acquiring semantic properties of existing, real-world code (e.g. purity of a function, pre/post-conditions
of a method) is hard to do at a truly large scale. The same problem also applies to run-time data
from real-life programs where it is challenging to acquire large amounts of such data even for a
single project. A promising direction to tackle this issue is by learning to extrapolate from runtime data (e.g. collected via instrumentation of a test-suite) to static properties of the code [4].
Although this is inherently a noisy process, achieving high accuracy may be suﬃcient, thanks to
the inherent ability of machine learning to handle small amounts of noise.
Strong generalization also manifests as a deployability problem. Machine learning models, especially when they have become eﬀective, are often so large that they are too large for a developer’s
machine, but using the cloud raises privacy7 concerns and prevents oﬄine coding. When under
development and tooling is needed, code evolves quickly, subjecting models to constant concept
drift and necessitating frequent retraining which can be extremely slow and costly. Addressing
this deployability concern is an open problem and requires advances in machine learning areas
such as transfer learning and one-shot learning. For example, say a program P uses libraries A and
B, which have been shipped with the models M A and M B . Could we save time training a model for
P by transferring knowledge from M A and M B ?
Finally, source code representations are multifaceted. For example, the token-level “view” of
source code is quite diﬀerent from a data ﬂow view of code. Learning to exploit multiple views
simultaneously can help machine learning models generalize and tackle issues with data sparsity.
Multi-view [179] and multi-modal learning (e.g. Gella et al. [64]), areas actively explored in machine learning, aim to achieve exactly this. By combining multiple representations of data, they
aim to improve upon the performance on various tasks, learning to generalize using multiple input signals. We believe that this is a promising future direction that may allow us to combine
probabilistic representations of code to achieve better generalization.
Metrics. To train and evaluate machine learning models, we need to easily compute indicators
about their performance. These measures of performance, although by deﬁnition approximate,
allow direct comparison of diﬀerent models and have already lead to improvements in multiple
areas, such as in code completion.
However, some metrics that are widely used in NLP are not suited for source code. For example, BLEU score is not suitable for measuring the quality of output source code (e.g. in transducer
7 https://www.theregister.co.uk/2017/07/25/kite_ﬂies_into_a_fork/

26

Allamanis et al.

models) because it fails to account for the fact that the syntax is known in all programming languages, so the BLEU score may be artiﬁcially “inﬂated” for predicting deterministic syntax. Second,
the granularity where BLEU is computed over (e.g. per-statement vs. per-token) is a controversial
choice. Finally, syntactically diverse answers may be semantically equivalent, yielding a low BLEU
score while being correct. Finding new widely-accepted measures for various tasks will allow the
community to build reliable models with well-understood performance characteristics.
5.2 New Domains
Here, we visit a number of domains to which machine learning has not yet been systematically
applied and yet suﬀer from uncertainty problems that machine learning is particularly well suited
to address, promising new advances.
Debugging. Debugging is a common task for software engineers [177]. Debugging is like trying
to ﬁnd a needle in the haystack; a developer has to recognize relevant information from the deluge
of available information. A multitude of tools exist in this area whose main goal is to visualize a
program’s state during its execution. Probabilistic models of source code could help developers,
such as by ﬁltering highly-improbable program states. Statistical debugging models, such as the
work of Zheng et al. [184, 185] and Liblit et al. [105] are indicative of the possibilities within this
area. Further adding learning within debugging models may allow further advances in statistical debugging. However, progress in this area is impeded by the combination of lack of data at a
large scale and the inherent diﬃculty of pattern recognition in very high-dimensional spaces. Defects4J [91] — a curated corpus of bugs — could further prove useful within machine learning for
fault prediction. Furthermore, collecting and ﬁltering execution traces to aid debugging is another
challenge for which machine learning is well-suited. Collection requires expensive instrumentation, which can introduce Heisenbugs. Here the question is “Can machine learning identify probe
points or reconstruct more complete traces from partial traces?” Concerning ﬁltering traces, machine learning may be able to ﬁnd interesting locations, like the root cause of bugs. Future methods
should also be able to generalize across diﬀerent programs, or even diﬀerent revisions of the same
program, a diﬃcult task for existing machine learning methods.
Traceability. Traceability is the study of links among software engineering artifacts. Examples
include links that connect code to its speciﬁcation, the speciﬁcation to requirements, and ﬁxes
to bug reports. Developers can exploit these links to better understand and maintain their code.
Usually, these links must be recovered. Information retrieval has dominated link recovery. The
work of Guo et al. [73] and Le et al. [101] suggests that learning better (semantic) representations
of artifacts can successfully, automatically solve important traceability problems.
Two major obstacles impede progress: lack of data and a focus on generic text. Tracing discussions in email threads, online chat rooms (e.g. Slack), documents and source code would be
extremely useful, but no publicly available and annotated data exists. Additionally, to date, NLP
research has mostly focused on modeling generic text (e.g. from newspapers); technical text in conversational environments (e.g. chatbots) has only begun to be researched. StackOverﬂow presents
one such interesting target. Although there are hundreds of studies that extract useful artifacts (e.g.
documentation) from StackOverﬂow, NLP methods — such as dependency parsing, co-reference
analysis and other linguistic phenomena — have not been explored.
Code Completion and Synthesis. Code completion and synthesis using machine learning are two
heavily researched and interrelated areas. Despite this fact, to our knowledge, there has been no

A Survey of Machine Learning for Big Code and Naturalness

27

full scale comparison between LM-based [80, 132, 152] and structured prediction-based autocompletion models [30, 146]. Although both types of systems target the same task, the lack of a wellaccepted benchmark, evaluation methodology and metrics has lead to the absence of a quantitative
comparison that highlights the strengths and weaknesses of each approach. This highlights the necessity of widely accepted, high-quality benchmarks, shared tasks, and evaluation metrics that can
lead to comparable and measurable improvements to tasks of interest. NLP and computer vision
follow such a paradigm with great success8 .
Omar et al. [137] discuss the challenges that arise from the fact that program editors usually
deal with incomplete, partial programs. Although they discuss how formal semantics can extend
to these cases, inherently any reasoning about partial code requires reasoning about the programmer’s intent. Lu et al. [112] used information-retrieval methods for synthesizing code completions
showing that simply retrieving snippets from “big code” can be useful when reasoning about code
completion, even without a learnable probabilistic component. This suggests a fruitful area for
probabilistic models of code that can assist editing tools when reasoning about incomplete code’s
semantics, by modeling how code could be completed.
Education. Software engineering education is one area that is already starting to be aﬀected by
this ﬁeld. The work of Campbell et al. [32] and Bhatia and Singh [22] already provide an automated
method for ﬁxing syntax errors in student code, whereas Piech et al. [143], Wang et al. [169] suggest
advancements towards giving richer feedback to students. Achieving reasonable automation can
help provide high-quality computer science education to many more students than is feasible today.
However, there are important challenges associated with this area. This includes the availability of
highly-granular data where machine learning systems can be trained, diﬃculty with embedding
semantic features of code into machine learning methods and the hardness of creating models
that can generalize to multiple and new tasks. Student coursework submissions are potentially a
ripe application area for machine learning, because here we have available many programs, from
diﬀerent students, which are meant to perform the same tasks. An especially large amount of
such data is available in Massive Open Online Courses (MOOCs). This opens exciting possibilities,
such as providing granular and detailed feedback, curriculum customization and other intelligent
tutoring systems can signiﬁcantly change computer science education.
Assistive Tools. Probabilistic models have allowed computer systems to handle noisy inputs such
as speech, and handwritten text input. In the future, probabilistic models of source code may enable novel assistive IDEs, creating inclusive tools that improve upon conventional methods of
developer-computer interaction and provide inclusive coding experiences.
6

RELATED RESEARCH AREAS

A variety of related research areas within software engineering and programming languages overlap with the area of statistical modeling of code. One of the most closely related research areas is
mining software repositories (MSR) and big code. MSR is a well-established, vibrant and active ﬁeld;
the idea is to mine the large amounts of source code data and meta-data available in open-source
(and commercial) repositories to gain valuable information, and use this information to enhance
both tools and processes. The eponymous ﬂagship conference is now in its 15th iteration. “Big
Code” is a synonymous neologism, created by DARPA’s MUSE program9 to borrow some branding
shine from the well-marketed term “Big Data”. The MSR ﬁeld’s early successes date back to work
by Zimmermann et al. [186], Williams and Hollingsworth [176], Gabel and Su [60] and Acharya
8 See

https://qz.com/1034972/ for a popular account of the eﬀect of large-scale datasets in computer vision.

9 http://science.dodlive.mil/2014/03/21/darpas-muse-mining-big-code/

28

Allamanis et al.

et al. [1] on mining API protocols from source code bases. These approaches used pragmatic counting techniques, such as frequent item-set mining. Mining software repositories is in one sense a
broader ﬁeld than statistical models of code, as rather than focusing on code alone, MSR considers
the full spectrum of software engineering data that can be derived from the software engineering
process, such as process measures, requirement traceability, commit logs. Additionally, research
in malware detection is related to probabilistic models of code [16, 33].
Another active area at the intersection machine learning and programming language research
is probabilistic programming [67]. This might appear to be related to statistical models of code,
but in fact there is a fundamental diﬀerence; essentially, probabilistic programming works in the
reverse direction. Probabilistic programming seeks to deploy programming language concepts to
make it easier for developers to write new machine learning algorithms. Statistical code models
seek to apply machine learning concepts to make it easier for developers to write new programs.
In some sense, the two areas are dual to each other. That being said, one can certainly imagine completing the cycle and attempting to develop statistical code models for probabilistic programming
languages. This could be a fascinating endeavor as probabilistic programming grows in popularity
to the extent that large corpora of probabilistic programs become available.
In software engineering, the term modeling often refers to formal speciﬁcations of program
behavior, which are clearly a very diﬀerent kind of model than those described here. Combining
formal models of semantics with statistical models of source code described in this review would
be an interesting area for future research. There is also some work on probabilistic models of code
that do not have a learning component, such as Liblit et al. [105]. Within machine learning, there
has been an interesting recent line of work on neural abstract machines [69, 153, 154], which extend
deterministic automata from computer science, such as pushdown automata and Turing machines,
to represent diﬀerentiable functions, so that the functions can be estimated by techniques from
machine learning. To the best of our knowledge, this intriguing line of work does not yet consider
source code, unlike the work described in this review. Semantic parsing is a vibrant research area
of NLP that is closely related to the idea of program synthesis from natural language. The research
in this area is extensive. We direct the interested reader to Jurafsky [90].
7 CONCLUSIONS
Probabilistic models of source code have exciting potential to support new tools in almost every
area of program analysis and software engineering. We reviewed existing work in the area, presenting a taxonomy of probabilistic machine learning source code models and their applications.
The reader may appreciate that most of the research contained in this review was conducted within
the past few years, indicating a growth of interest in this area among the machine learning, programming languages and software engineering communities. Probabilistic models of source code
raise the exciting opportunity of learning from existing code, probabilistically reasoning about
new source code artifacts and transferring knowledge between developers and projects.
A SUPPLEMENTARY MATERIALS
REFERENCES
[1] Mithun Acharya, Tao Xie, Jian Pei, and Jun Xu. 2007. Mining API patterns as partial orders from source code: from
usage scenarios to speciﬁcations. In Proceedings of the Joint Meeting of the European Software Engineering Conference
and the Symposium on the Foundations of Software Engineering (ESEC/FSE).
[2] Karan Aggarwal, Mohammad Salameh, and Abram Hindle. 2015. Using machine translation for converting Python 2
to Python 3 code. Technical Report.
[3] Alex A Alemi, Francois Chollet, Geoﬀrey Irving, Christian Szegedy, and Josef Urban. 2016. DeepMath–Deep Sequence
Models for Premise Selection. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS).

A Survey of Machine Learning for Big Code and Naturalness

29

Table 4. Some datasets, available online, used in research of Probabilistic Models of Source Code (sorted
alphabetically). Links are clickable in the digital version.
Reference

Short Description

Link

Allamanis and Sutton [10]
Allamanis et al. [9]
Barone and Sennrich [18]
Cerulo et al. [34]
Dyer et al. [50]
Iyer et al. [88]
Kushman and Barzilay [98]
Ling et al. [107]
Raychev et al. [150]
Raychev et al. [150]
Oda et al. [134]

Deduplicated snapshot of all Java GitHub projects with least one fork.
Parsed source code for 11 highly-ranked Java GitHub projects.
Parallel corpus of 150k Python function declarations, docstrings and bodies.
Free text data with source code “islands”.
800k+ code repositories, software to support queries and mining
Source code snippets with their StackOverﬂow title.
Dataset for generating regular expressions from natural language.
Text descriptions and code for card game.
A dataset of deduplicated JavaScript ﬁles and their ASTs extracted from GitHub.
A dataset of deduplicated Python ﬁles and their ASTs extracted from GitHub.
Parallel corpus of Python code and pseudocode in English and Japanese.

link
link
link
link
link
link
link
link
link
link
link

[4] Miltiadis Allamanis, Earl T. Barr, Christian Bird, Premkumar Devanbu, Mark Marron, and
Charles Sutton. 2016.
Mining Semantic Loop Idioms from Big Code.
Technical Report.
https://www.microsoft.com/en-us/research/publication/mining-semantic-loop-idioms-big-code/
[5] Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2014. Learning Natural Coding Conventions. In
Proceedings of the International Symposium on Foundations of Software Engineering (FSE).
[6] Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2015. Suggesting accurate method and class
names. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the Symposium on the
Foundations of Software Engineering (ESEC/FSE).
[7] Miltiadis Allamanis and Marc Brockschmidt. 2016. SmartPaste: Learning to Adapt Source Code. arXiv preprint
arXiv:1705.07867 (2016).
[8] Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, and Charles Sutton. 2017. Learning Continuous
Semantic Representations of Symbolic Expressions. In Proceedings of the International Conference on Machine Learning
(ICML).
[9] Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. In Proceedings of the International Conference on Machine Learning (ICML).
[10] Miltiadis Allamanis and Charles Sutton. 2013. Mining source code repositories at massive scale using language
modeling. In Proceedings of the Working Conference on Mining Software Repositories (MSR).
[11] Miltiadis Allamanis and Charles Sutton. 2014. Mining idioms from source code. In Proceedings of the International
Symposium on Foundations of Software Engineering (FSE).
[12] Miltiadis Allamanis, Daniel Tarlow, Andrew Gordon, and Yi Wei. 2015. Bimodal modelling of source code and natural
language. In Proceedings of the International Conference on Machine Learning (ICML).
[13] Sven Amann, Sebastian Proksch, Sarah Nadi, and Mira Mezini. 2016. A study of Visual Studio usage in practice. In
Proceedings of the International Conference on Software Analysis, Evolution, and Reengineering (SANER).
[14] Matthew Amodio, Swarat Chaudhuri, and Thomas Reps. 2017. Neural Attribute Machines for Program Generation.
arXiv preprint arXiv:1705.09231 (2017).
[15] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to Compose Neural Networks for
Question Answering. In Proceedings of NAACL-HLT.
[16] Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. 2014. DREBIN:
Eﬀective and Explainable Detection of Android Malware in Your Pocket. In Network and Distributed System Security
Symposium (NDSS).
[17] Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder:
Learning to Write Programs. In Proceedings of the International Conference on Learning Representations (ICLR).
[18] Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of Python functions and documentation
strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275 (2017).
[19] Earl T Barr, Yuriy Brun, Premkumar Devanbu, Mark Harman, and Federica Sarro. 2014. The plastic surgery hypothesis.
In Proceedings of the International Symposium on Foundations of Software Engineering (FSE).
[20] Tony Beltramelli. 2017. pix2code: Generating Code from a Graphical User Interface Screenshot. arXiv preprint
arXiv:1705.07962 (2017).
[21] Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott

30

[22]
[23]
[24]
[25]
[26]
[27]
[28]
[29]
[30]

[31]
[32]
[33]
[34]
[35]
[36]
[37]
[38]
[39]
[40]
[41]
[42]
[43]
[44]
[45]
[46]
[47]
[48]
[49]
[50]

Allamanis et al.
McPeak, and Dawson Engler. 2010. A few billion lines of code later: using static analysis to ﬁnd bugs in the real
world. Commun. ACM (2010).
Sahil Bhatia and Rishabh Singh. 2016. Automated Correction for Syntax Errors in Programming Assignments using
Recurrent Neural Networks. arXiv preprint arXiv:1603.06129 (2016).
Avishkar Bhoopchand, Tim Rocktäschel, Earl T. Barr, and Sebastian Riedel. 2016. Learning Python Code Suggestion
with a Sparse Pointer Network. (2016). https://openreview.net/pdf?id=r1kQkVFgl.
Benjamin Bichsel, Veselin Raychev, Petar Tsankov, and Martin Vechev. 2016. Statistical Deobfuscation of Android
Applications. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security.
Pavol Bielik, Veselin Raychev, and Martin Vechev. 2015. Programming with “Big Code”: Lessons, Techniques and
Applications. In LIPIcs-Leibniz International Proceedings in Informatics.
Pavol Bielik, Veselin Raychev, and Martin Vechev. 2016. PHOG: Probabilistic Model for Code. In Proceedings of the
International Conference on Machine Learning (ICML).
David M Blei. 2012. Probabilistic Topic Models. Commun. ACM 55, 4 (2012), 77–84.
Marc Brockschmidt, Yuxin Chen, Pushmeet Kohli, Siddharth Krishna, and Daniel Tarlow. 2017. Learning Shape
Analysis. In International Static Analysis Symposium. Springer, 66–87.
Peter John Brown. 1979. Software Portability: an advanced course. CUP Archive.
Marcel Bruch, Martin Monperrus, and Mira Mezini. 2009. Learning from examples to improve code completion
systems. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the Symposium on
the Foundations of Software Engineering (ESEC/FSE).
Raymond PL Buse and Westley Weimer. 2012. Synthesizing API usage examples. In Proceedings of the International
Conference on Software Engineering (ICSE).
Joshua Charles Campbell, Abram Hindle, and José Nelson Amaral. 2014. Syntax errors just aren’t natural: improving
error reporting with language models. In Proceedings of the Working Conference on Mining Software Repositories (MSR).
Lei Cen, Christoher S Gates, Luo Si, and Ninghui Li. 2015. A probabilistic discriminative model for android malware
detection with decompiled source code. IEEE Transactions on Dependable and Secure Computing (2015).
Luigi Cerulo, Massimiliano Di Penta, Alberto Bacchelli, Michele Ceccarelli, and Gerardo Canfora. 2015. Irish: A
Hidden Markov Model to detect coded information islands in free text. Science of Computer Programming (2015).
Kwonsoo Chae, Hakjoo Oh, Kihong Heo, and Hongseok Yang. 2016. Automatically generating features for learning
program analysis heuristics. arXiv preprint arXiv:1612.09394 (2016).
Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM Computing Surveys
(CSUR) 41, 3 (2009), 15.
Stanley F Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling.
Computer Speech and Language 13 (1999), 359–394.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the Properties of Neural
Machine Translation: Encoder–Decoder Approaches. Syntax, Semantics and Structure in Statistical Translation (2014).
Edmund Clarke, Daniel Kroening, and Karen Yorav. 2003. Behavioral consistency of C and Verilog programs using
bounded model checking. In Proceedings of the 40th annual Design Automation Conference. ACM, 368–371.
Trevor Cohn, Phil Blunsom, and Sharon Goldwater. 2010. Inducing tree-substitution grammars. Journal of Machine
Learning Research (JMLR) (2010).
Christopher S Corley, Kostadin Damevski, and Nicholas A Kraft. 2015. Exploring the use of deep learning for feature
location. In Software Maintenance and Evolution (ICSME), 2015 IEEE International Conference on.
Patrick Cousot, Radhia Cousot, Jerôme Feret, Laurent Mauborgne, Antoine Miné, David Monniaux, and Xavier Rival.
2005. The ASTRÉE analyzer. In ESPO. Springer.
William Croft. 2008. Evolutionary linguistics. Annual Review of Anthropology (2008).
Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. 2017a. End-to-end Deep Learning of Optimization Heuristics. In International Conference on Parallel Computing Technologies.
Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. 2017b. Synthesizing benchmarks for predictive
modeling. In IEEE/ACM International Symposium on Code Generation and Optimization (CGO).
Hoa Khanh Dam, Truyen Tran, and Trang Pham. 2016. A deep language model for software code. arXiv preprint
arXiv:1608.02715 (2016).
Florian Deißenböck and Markus Pizka. 2006. Concise and consistent naming. Software Quality Journal (2006).
Premkumar Devanbu. 2015. New initiative: the naturalness of software. In Proceedings of the International Conference
on Software Engineering (ICSE).
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel rahman Mohamed, and Pushmeet Kohli. 2017.
RobustFill: Neural Program Learning under Noisy I/O. In International Conference on Machine Learning (ICML).
Robert Dyer, Hoan Anh Nguyen, Hridesh Rajan, and Tien N. Nguyen. 2013. Boa: A Language and Infrastructure for

A Survey of Machine Learning for Big Code and Naturalness

31

Analyzing Ultra-Large-Scale Software Repositories. In International Conference on Software Engineering (ICSE).
[51] Dawson Engler, David Yu Chen, Seth Hallem, Andy Chou, and Benjamin Chelf. 2001. Bugs as deviant behavior: A
general approach to inferring errors in systems code. In ACM SIGOPS Operating Systems Review.
[52] Michael D Ernst. 2017. Natural language is a programming language: Applying natural language processing to
software development. In LIPIcs-Leibniz International Proceedings in Informatics.
[53] Ethan Fast, Daniel Steﬀee, Lucy Wang, Joel R Brandt, and Michael S Bernstein. 2014. Emergent, crowd-scale programming practice in the IDE. In Proceedings of the Annual ACM Conference on Human Factors in Computing Systems.
[54] Benjamin Floyd, Tyler Santander, and Westley Weimer. 2017. Decoding the representation of code in the brain: An
fMRI study of code review and expertise. In Proceedings of the International Conference on Software Engineering (ICSE).
[55] Eclipse Foundation. Code Recommenders. www.eclipse.org/recommenders. (????). Visited June 2017.
[56] Jaroslav Fowkes, Pankajan Chanthirasegaran, Razvan Ranca, Miltos Allamanis, Mirella Lapata, and Charles Sutton.
2017. Autofolding for Source Code Summarization. IEEE Transactions on Software Engineering (TSE) (2017).
[57] Jaroslav Fowkes and Charles Sutton. 2015. Parameter-Free Probabilistic API Mining at GitHub Scale. In Proceedings
of the International Symposium on Foundations of Software Engineering (FSE).
[58] Christine Franks, Zhaopeng Tu, Premkumar Devanbu, and Vincent Hellendoorn. 2015. Cacheca: A cache language
model based code suggestion tool. In Proceedings of the International Conference on Software Engineering (ICSE).
[59] Wei Fu and Tim Menzies. 2017. Easy over Hard: A Case Study on Deep Learning. In Proceedings of the International
Symposium on Foundations of Software Engineering (FSE).
[60] Mark Gabel and Zhendong Su. 2008. Javert: fully automatic mining of general temporal properties from dynamic
traces. In Proceedings of the International Symposium on Foundations of Software Engineering (FSE).
[61] Mark Gabel and Zhendong Su. 2010. A study of the uniqueness of source code. In Proceedings of the International
Symposium on Foundations of Software Engineering (FSE).
[62] Rosalva E Gallardo-Valencia and Susan Elliott Sim. 2009. Internet-scale code search. In Proceedings of the 2009 ICSE
Workshop on Search-Driven Development-Users, Infrastructure, Tools and Evaluation.
[63] Alexander L Gaunt, Marc Brockschmidt, Rishabh Singh, Nate Kushman, Pushmeet Kohli, Jonathan Taylor, and Daniel
Tarlow. 2016. TerpreT: A probabilistic programming language for program induction. arXiv preprint arXiv:1608.04428
(2016).
[64] Spandana Gella, Mirella Lapata, and Frank Keller. 2016. Unsupervised Visual Sense Disambiguation for Verbs using
Multimodal Embeddings. In Proceedings of NAACL-HLT.
[65] Elena L Glassman, Jeremy Scott, Rishabh Singh, Philip J Guo, and Robert C Miller. 2015. OverCode: Visualizing
variation in student solutions to programming problems at scale. ACM Transactions on Computer-Human Interaction
(TOCHI) (2015).
[66] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. www.deeplearningbook.org.
[67] Andrew D Gordon, Thomas A Henzinger, Aditya V Nori, and Sriram K Rajamani. 2014. Probabilistic programming.
In Proceedings of the International Conference on Software Engineering (ICSE). ACM, 167–181.
[68] Orlena Gotel, Jane Cleland-Huang, Jane Huﬀman Hayes, Andrea Zisman, Alexander Egyed, Paul Grünbacher, Alex
Dekhtyar, Giuliano Antoniol, Jonathan Maletic, and Patrick Mäder. 2012. Traceability fundamentals. In Software and
Systems Traceability. Springer.
[69] Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural Turing Machines. arXiv preprint arXiv:1410.5401 (2014).
[70] Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep API Learning. In Proceedings of the
International Symposium on Foundations of Software Engineering (FSE).
[71] Sumit Gulwani and Mark Marron. 2014. NLyze: Interactive programming by natural language for spreadsheet data
analysis and manipulation. In Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data.
[72] Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, and others. 2017. Program Synthesis. Foundations and Trends®
in Programming Languages (2017).
[73] Jin Guo, Jinghui Cheng, and Jane Cleland-Huang. 2017. Semantically enhanced software traceability using deep
learning techniques. In Proceedings of the International Conference on Software Engineering (ICSE).
[74] Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. DeepFix: Fixing common C language errors by
deep learning. In Proceedings of the Conference of Artiﬁcial Intelligence (AAAI).
[75] Tihomir Gvero and Viktor Kuncak. 2015. Synthesizing Java expressions from free-form queries. In Proceedings of the
Conference on Object-Oriented Programming, Systems, Languages & Applications (OOPSLA).
[76] Alon Halevy, Peter Norvig, and Fernando Pereira. 2009. The unreasonable eﬀectiveness of data. IEEE Intelligent
Systems (2009).
[77] Vincent J Hellendoorn and Premkumar Devanbu. 2017. Are deep neural networks the best choice for modeling source
code?. In Proceedings of the International Symposium on Foundations of Software Engineering (FSE). ACM, 763–773.
[78] Vincent J Hellendoorn, Premkumar T Devanbu, and Alberto Bacchelli. 2015. Will they like this?: Evaluating code

32

Allamanis et al.

contributions with language models. In Proceedings of the Working Conference on Mining Software Repositories (MSR).
[79] Felix Hill, KyungHyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to Understand Phrases by Embedding the Dictionary. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) (2016).
[80] Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016. On the naturalness of software.
Commun. ACM 59, 5 (2016), 122–131.
[81] Abram Hindle, Earl T Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. 2012. On the naturalness of software.
In Proceedings of the International Conference on Software Engineering (ICSE).
[82] Geoﬀrey E Hinton. 1984. Distributed representations. (1984).
[83] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation (1997).
[84] Reid Holmes, Robert J Walker, and Gail C Murphy. 2005. Strathcona example recommendation tool. In ACM SIGSOFT
Software Engineering Notes.
[85] Chun-Hung Hsiao, Michael Cafarella, and Satish Narayanasamy. 2014. Using web corpus statistics for program
analysis. In ACM SIGPLAN Notices.
[86] Xing Hu, Yuhan Wei, Ge Li, and Zhi Jin. 2017. CodeSum: Translate Program Language to Natural Language. arXiv
preprint arXiv:1708.01837 (2017).
[87] Andrew Hunt and David Thomas. 2000. The pragmatic programmer: from journeyman to master. Addison-Wesley
Professional.
[88] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing Source Code using a
Neural Attention Model. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
[89] Siyuan Jiang, Ameer Armaly, and Collin McMillan. 2017. Automatically Generating Commit Messages from Diﬀs
using Neural Machine Translation. arXiv preprint arXiv:1708.09492 (2017).
[90] Dan Jurafsky. 2000. Speech & Language Processing (3 ed.). Pearson Education.
[91] René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled
testing studies for Java programs. In Proceedings of the International Symposium on Software Testing and Analysis
(ISSTA).
[92] Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. 2014. Phrase-based statistical translation of programming
languages. In International Symposium on New Ideas, New Paradigms, and Reﬂections on Programming & Software.
[93] Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2015. Visualizing and understanding recurrent networks. arXiv
preprint arXiv:1506.02078 (2015).
[94] Donald E Knuth. 1984. Literate programming. Comput. J. 27, 2 (1984), 97–111.
[95] Ugur Koc, Parsa Saadatpanah, Jeﬀrey S Foster, and Adam A Porter. 2017. Learning a classiﬁer for false positive
error reports emitted by static code analysis tools. In Proceedings of the 1st ACM SIGPLAN International Workshop on
Machine Learning and Programming Languages.
[96] Ted Kremenek, Andrew Y Ng, and Dawson R Engler. 2007. A Factor Graph Model for Software Bug Finding. In
Proceedings of the International Joint Conference on Artiﬁcal intelligence (IJCAI).
[97] Roland Kuhn and Renato De Mori. 1990. A cache-based natural language model for speech recognition. Pattern
Analysis and Machine Intelligence, IEEE Transactions on (1990).
[98] Nate Kushman and Regina Barzilay. 2013. Using Semantic Uniﬁcation to Generate Regular Expressions from Natural
Language. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies (NAACL-HLT).
[99] Tessa Lau. 2001. Programming by demonstration: a machine learning approach. Ph.D. Dissertation. University of
Washington.
[100] Quoc V Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. In Proceedings of
the International Conference on Machine Learning (ICML).
[101] Tien-Duy B Le, Mario Linares-Vásquez, David Lo, and Denys Poshyvanyk. 2015. Rclinker: Automated linking of
issue reports and commits leveraging rich contextual information. In Proceedings of the International Conference on
Program Comprehension (ICPC).
[102] Dor Levy and Lior Wolf. 2017. Learning to Align the Source Code to the Compiled Object Code. In Proceedings of
the International Conference on Machine Learning (ICML).
[103] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2016. Gated Graph Sequence Neural Networks.
Proceedings of the International Conference on Learning Representations (ICLR) (2016).
[104] Percy Liang, Michael I Jordan, and Dan Klein. 2010. Learning programs: A hierarchical Bayesian approach. In
Proceedings of the International Conference on Machine Learning (ICML).
[105] Ben Liblit, Mayur Naik, Alice X Zheng, Alex Aiken, and Michael I Jordan. 2005. Scalable statistical bug isolation. In
ACM SIGPLAN Notices.

A Survey of Machine Learning for Big Code and Naturalness

33

[106] Xi Victoria Lin, Chenglong Wang, Deric Pang, Kevin Vu, and Michael D Ernst. 2017. Program synthesis from natural language using recurrent neural networks. Technical Report. Technical Report UW-CSE-17-03-01, University of
Washington Department of Computer Science and Engineering, Seattle, WA, USA.
[107] Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Andrew Senior, Fumin Wang, and Phil
Blunsom. 2016. Latent Predictor Networks for Code Generation. In Proceedings of the Annual Meeting of the Association
for Computational Linguistics (ACL).
[108] Han Liu. 2016. Towards better program obfuscation: optimization via language models. In Proceedings of the 38th
International Conference on Software Engineering Companion.
[109] Benjamin Livshits, Aditya V. Nori, Sriram K. Rajamani, and Anindya Banerjee. 2009. Merlin: Speciﬁcation Inference
for Explicit Information Flow Problems. In Conference on Programming Language Design and Implementation (PLDI).
[110] Sarah Loos, Geoﬀrey Irving, Christian Szegedy, and Cezary Kaliszyk. 2017. Deep Network Guided Proof Search.
arXiv preprint arXiv:1701.06972 (2017).
[111] Pablo Loyola, Edison Marrese-Taylor, and Yutaka Matsuo. 2017. A Neural Architecture for Generating Natural
Language Descriptions from Source Code Changes. arXiv preprint arXiv:1704.04856 (2017).
[112] Yanxin Lu, Swarat Chaudhuri, Chris Jermaine, and David Melski. 2017. Data-Driven Program Completion. arXiv
preprint arXiv:1705.09042 (2017).
[113] Chris Maddison and Daniel Tarlow. 2014. Structured Generative Models of Natural Source Code. In Proceedings of
the International Conference on Machine Learning (ICML).
[114] Ravi Mangal, Xin Zhang, Aditya V Nori, and Mayur Naik. 2015. A user-guided approach to program analysis. In
Proceedings of the International Symposium on Foundations of Software Engineering (FSE).
[115] Collin Mcmillan, Denys Poshyvanyk, Mark Grechanik, Qing Xie, and Chen Fu. 2013. Portfolio: Searching for relevant
functions and their usages in millions of lines of code. ACM Transactions on Software Engineering and Methodology
(TOSEM) (2013).
[116] Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai. 2013. A machine learning framework
for programming by example. In Proceedings of the International Conference on Machine Learning (ICML).
[117] Kim Mens and Angela Lozano. 2014. Source code-based recommendation systems. In Recommendation Systems in
Software Engineering. Springer.
[118] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeﬀrey Dean. 2013. Eﬃcient estimation of word representations in
vector space. arXiv preprint arXiv:1301.3781 (2013).
[119] Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional Neural Networks over Tree Structures for
Programming Language Processing. In Proceedings of the Conference of Artiﬁcial Intelligence (AAAI).
[120] Dana Movshovitz-Attias and William W. Cohen. 2013. Natural Language Models for Predicting Programming Comments. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
[121] Dana Movshovitz-Attias and William W. Cohen. 2015. KB-LDA: Jointly learning a knowledge base of hierarchy,
relations, and facts. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
[122] Vijayaraghavan Murali, Swarat Chaudhuri, and Chris Jermaine. 2017a. Bayesian Sketch Learning for Program Synthesis. arXiv preprint arXiv:1703.05698 (2017).
[123] Vijayaraghavan Murali, Swarat Chaudhuri, and Chris Jermaine. 2017b. Finding Likely Errors with Bayesian Speciﬁcations. arXiv preprint arXiv:1703.01370 (2017).
[124] Arvind Neelakantan, Quoc V Le, and Ilya Sutskever. 2015. Neural Programmer: Inducing latent programs with
gradient descent. In Proceedings of the International Conference on Learning Representations (ICLR).
[125] Graham Neubig. 2016.
Survey of Methods to Generate Natural Language from Source Code.
http:// www.languageandcode.org/ nlse2015/ neubig15nlse-survey.pdf (2016).
[126] Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N. Nguyen. 2016a. A large-scale study on repetitiveness, containment, and composability of routines in open-source projects. In Proceedings of the Working Conference on Mining
Software Repositories (MSR).
[127] Anh Tuan Nguyen and Tien N. Nguyen. 2015. Graph-based statistical language model for code. In Proceedings of the
International Conference on Software Engineering (ICSE).
[128] Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2013. Lexical statistical machine translation for
language migration. In Proceedings of the International Symposium on Foundations of Software Engineering (FSE).
[129] Anh T Nguyen, Tung Thanh Nguyen, and Tien N. Nguyen. 2015. Divide-and-Conquer Approach for Multi-phase
Statistical Migration for Source Code. In Proceedings of the International Conference on Automated Software Engineering
(ASE).
[130] Trong Duc Nguyen, Anh Tuan Nguyen, and Tien N Nguyen. 2016b. Mapping API elements for code migration with
vector representations. In Proceedings of the International Conference on Software Engineering (ICSE).
[131] Trong Duc Nguyen, Anh Tuan Nguyen, Hung Dang Phan, and Tien N Nguyen. 2017. Exploring API embedding for

34

Allamanis et al.

API usages and applications. In Proceedings of the International Conference on Software Engineering (ICSE).
[132] Tung Thanh Nguyen, Anh Tuan Nguyen, Hoan Anh Nguyen, and Tien N Nguyen. 2013. A statistical semantic
language model for source code. In Proceedings of the Joint Meeting of the European Software Engineering Conference
and the Symposium on the Foundations of Software Engineering (ESEC/FSE).
[133] Haoran Niu, Iman Keivanloo, and Ying Zou. 2016. Learning to rank code examples for code search engines. Empirical
Software Engineering (ESEM) (2016).
[134] Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura.
2015. Learning to Generate Pseudo-Code from Source Code Using Statistical Machine Translation. In Proceedings of
the International Conference on Automated Software Engineering (ASE).
[135] Hakjoo Oh, Hongseok Yang, and Kwangkeun Yi. 2015. Learning a strategy for adapting a program analysis via
Bayesian optimisation. In Proceedings of the Conference on Object-Oriented Programming, Systems, Languages & Applications (OOPSLA).
[136] Cyrus Omar. 2013. Structured statistical syntax tree prediction. In Proceedings of the Conference on Systems, Programming, Languages and Applications: Software for Humanity (SPLASH).
[137] Cyrus Omar, Ian Voysey, Michael Hilton, Joshua Sunshine, Claire Le Goues, Jonathan Aldrich, and Matthew A
Hammer. 2017. Toward Semantic Foundations for Program Editors. (2017).
[138] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of
machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
[139] Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. 2017.
Neuro-Symbolic Program Synthesis. In Proceedings of the International Conference on Learning Representations (ICLR).
[140] Terence Parr and Jurgin Vinju. 2016. Technical Report: Towards a Universal Code Formatter through Machine
Learning. arXiv preprint arXiv:1606.08866 (2016).
[141] Jibesh Patra and Michael Pradel. 2016. Learning to Fuzz: Application-Independent Fuzz Testing with Probabilistic,
Generative Models of Input Data. (2016).
[142] Hung Viet Pham, Phong Minh Vu, Tung Thanh Nguyen, and others. 2016. Learning API usages from bytecode: a
statistical approach. In Proceedings of the International Conference on Software Engineering (ICSE).
[143] Chris Piech, Jonathan Huang, Andy Nguyen, Mike Phulsuksombati, Mehran Sahami, and Leonidas J Guibas. 2015.
Learning Program Embeddings to Propagate Feedback on Student Code. In Proceedings of the International Conference
on Machine Learning (ICML).
[144] Matt Post and Daniel Gildea. 2009. Bayesian learning of a tree substitution grammar. In Proceedings of the Annual
Meeting of the Association for Computational Linguistics (ACL).
[145] Sebastian Proksch, Sven Amann, Sarah Nadi, and Mira Mezini. 2016. Evaluating the evaluations of code recommender systems: a reality check. In Proceedings of the International Conference on Automated Software Engineering
(ASE).
[146] Sebastian Proksch, Johannes Lerch, and Mira Mezini. 2015. Intelligent code completion with Bayesian networks.
ACM Transactions on Software Engineering and Methodology (TOSEM) (2015).
[147] Yewen Pu, Karthik Narasimhan, Armando Solar-Lezama, and Regina Barzilay. 2016. sk_p: a neural program corrector for MOOCs. In Proceedings of the Conference on Systems, Programming, Languages and Applications: Software for
Humanity (SPLASH).
[148] Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract Syntax Networks for Code Generation and Semantic Parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
[149] Baishakhi Ray, Vincent Hellendoorn, Saheel Godhane, Zhaopeng Tu, Alberto Bacchelli, and Premkumar Devanbu.
2016. On the naturalness of buggy code. In Proceedings of the International Conference on Software Engineering (ICSE).
[150] Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. 2016. Learning programs from noisy data. In
Proceedings of the Symposium on Principles of Programming Languages (POPL).
[151] Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting Program Properties from “Big Code”. In
Proceedings of the Symposium on Principles of Programming Languages (POPL).
[152] Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In Proceedings of the Symposium on Programming Language Design and Implementation (PLDI).
[153] Scott Reed and Nando de Freitas. 2016. Neural Programmer-Interpreters. In Proceedings of the International Conference on Learning Representations (ICLR).
[154] Sebastian Riedel, Matko Bošnjak, and Tim Rocktäschel. 2016. Programming with a Diﬀerentiable Forth Interpreter.
arXiv preprint arXiv:1605.06640 (2016).
[155] Martin Robillard, Robert Walker, and Thomas Zimmermann. 2010. Recommendation systems for software engineering. Software, IEEE (2010).
[156] Martin P Robillard, Walid Maalej, Robert J Walker, and Thomas Zimmermann. 2014. Recommendation systems in

A Survey of Machine Learning for Big Code and Naturalness

35

software engineering. Springer.
[157] Tim Rocktäschel and Sebastian Riedel. 2017. End-to-end Diﬀerentiable Proving. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS).
[158] Caitlin Sadowski, Kathryn T Stolee, and Sebastian Elbaum. 2015. How developers search for code: a case study. In
Proceedings of the International Symposium on Foundations of Software Engineering (FSE).
[159] Juliana Saraiva, Christian Bird, and Thomas Zimmermann. 2015. Products, developers, and milestones: how should
I build my N-Gram language model. In Proceedings of the International Symposium on Foundations of Software Engineering (FSE).
[160] Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword
units. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
[161] Abhishek Sharma, Yuan Tian, and David Lo. 2015. NIRMAL: Automatic identiﬁcation of software relevant tweets
leveraging language model. In Proceedings of the International Conference on Software Analysis, Evolution, and Reengineering (SANER).
[162] Rishabh Singh and Sumit Gulwani. 2015. Predicting a correct program in programming by example. In International
Conference on Computer Aided Veriﬁcation.
[163] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS).
[164] Suresh Thummalapenta and Tao Xie. 2007. Parseweb: a programmer assistant for reusing open source code on the
web. In Proceedings of the International Conference on Automated Software Engineering (ASE).
[165] Christoph Treude and Martin P Robillard. 2016. Augmenting API documentation with insights from Stack Overﬂow.
In Proceedings of the International Conference on Software Engineering (ICSE).
[166] Zhaopeng Tu, Zhendong Su, and Premkumar Devanbu. 2014. On the localness of software. In Proceedings of the
International Symposium on Foundations of Software Engineering (FSE).
[167] Bogdan Vasilescu, Casey Casalnuovo, and Premkumar Devanbu. 2017. Recovering clear, natural identiﬁers from
obfuscated JS names. In Proceedings of the International Symposium on Foundations of Software Engineering (FSE).
ACM, 683–693.
[168] Martin Velez, Dong Qiu, You Zhou, Earl T Barr, and Zhendong Su. 2015. A Study of “Wheat” and “Chaﬀ” in Source
Code. arXiv preprint arXiv:1502.01410 (2015).
[169] Lisa Wang, Angela Sy, Larry Liu, and Chris Piech. 2017. Deep Knowledge Tracing On Programming Exercises. In
Conference on Learning @ Scale.
[170] Song Wang, Devin Chollak, Dana Movshovitz-Attias, and Lin Tan. 2016a. Bugram: bug detection with n-gram
language models. In Proceedings of the International Conference on Automated Software Engineering (ASE).
[171] Song Wang, Taiyue Liu, and Lin Tan. 2016b. Automatically learning semantic features for defect prediction. In
Proceedings of the International Conference on Software Engineering (ICSE).
[172] Xin Wang, Chang Liu, Richard Shin, Joseph E. Gonzalez, and Dawn Song. 2016c. Neural Code Completion. (2016).
https://openreview.net/pdf?id=rJbPBt9lg.
[173] Andrzej Wasylkowski, Andreas Zeller, and Christian Lindig. 2007. Detecting object usage anomalies. In Proceedings
of the Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software
Engineering (ESEC/FSE).
[174] Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. 2016. Deep Learning Code Fragments
for Code Clone Detection. In Proceedings of the International Conference on Automated Software Engineering (ASE).
[175] Martin White, Christopher Vendome, Mario Linares-Vásquez, and Denys Poshyvanyk. 2015. Toward deep learning
software repositories. In Proceedings of the Working Conference on Mining Software Repositories (MSR).
[176] Chadd C Williams and Jeﬀrey K Hollingsworth. 2005. Automatic mining of source code repositories to improve bug
ﬁnding techniques. IEEE Transactions on Software Engineering (TSE) (2005).
[177] W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization.
IEEE Transactions on Software Engineering (TSE) (2016).
[178] Tao Xie and Jian Pei. 2006. MAPO: Mining API usages from open source repositories. In Proceedings of the Working
Conference on Mining Software Repositories (MSR).
[179] Chang Xu, Dacheng Tao, and Chao Xu. 2013. A survey on multi-view learning. arXiv preprint arXiv:1304.5634 (2013).
[180] Shir Yadid and Eran Yahav. 2016. Extracting code from programming tutorial videos. In Proceedings of the 2016 ACM
International Symposium on New Ideas, New Paradigms, and Reﬂections on Programming and Software.
[181] Eran Yahav. 2015. Programming with “Big Code”. In Asian Symposium on Programming Languages and Systems.
Springer, 3–8.
[182] Pengcheng Yin and Graham Neubig. 2017. A Syntactic Neural Model for General-Purpose Code Generation. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL) (2017).

36

Allamanis et al.

[183] Wojciech Zaremba and Ilya Sutskever. 2014. Learning to execute. arXiv preprint arXiv:1410.4615 (2014).
[184] Alice X Zheng, Michael I Jordan, Ben Liblit, and Alex Aiken. 2003. Statistical debugging of sampled programs. In
Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS).
[185] Alice X Zheng, Michael I Jordan, Ben Liblit, Mayur Naik, and Alex Aiken. 2006. Statistical debugging: simultaneous
identiﬁcation of multiple bugs. In Proceedings of the International Conference on Machine Learning (ICML).
[186] Thomas Zimmermann, Andreas Zeller, Peter Weissgerber, and Stephan Diehl. 2005. Mining version histories to
guide software changes. IEEE Transactions on Software Engineering (TSE) (2005).

