Search Based Code Generation for Machine Learning Programs
Muhammad Zubair Malik, Muhammad Nawaz,
Nimrah Mustafa

Junaid Haroon Siddiqui
LUMS School of Science and Engineering
Lahore, Pakistan
junaid.siddiqui@lums.edu.pk

Information Technology University of the Punjab
Lahore, Pakistan
{zubair.malik,muhammad.nawaz,nimrah.mustafa}@itu.
edu.pk

arXiv:1801.09373v2 [cs.SE] 6 Feb 2018

ABSTRACT
Machine Learning (ML) has revamped every domain of life as it
provides powerful tools to build complex systems that learn and
improve from experience and data. Our key insight is that to solve
a machine learning problem, data scientists do not invent a new
algorithm each time, but evaluate a range of existing models with
different configurations and select the best one. This task is laborious, error-prone, and drains a large chunk of project budget and
time. In this paper we present a novel framework inspired by programming by Sketching[8] and Partial Evaluation[4] to minimize
human intervention in developing ML solutions. We templatize
machine learning algorithms to expose configuration choices as
holes to be searched. We share code and computation between different algorithms, and only partially evaluate configuration space
of algorithms based on information gained from initial algorithm
evaluations. We also employ hierarchical and heuristic based pruning to reduce the search space. Our initial findings indicate that our
approach can generate highly accurate ML models. Interviews with
data scientists show that they feel our framework can eliminate
sources of common errors and significantly reduce development
time.

1

INTRODUCTION

Machine Learning (ML) is a set of techniques that give computers
the ability to learn specific tasks from the data without being explicitly programmed. We are living in the golden age of machine
learning—ML algorithms have defeated humans in chess, learned to
drive autonomously, beat humans in Jeopardy, and out performed
humans in fundamentally innate tasks of image recognition as well
as speech understanding. While most of these battles were won by
well funded teams of highly trained engineers, machine learning
tools remain prohibitively expensive for average users and domain
experts. In this paper we present software engineering techniques
that make machine learning algorithms more usable by reducing
the time to specify them and making them less costly to build.
Sculley et al. [7], based on their experience at Google, observe
that software engineers developing ML code often have less time
to write high quality code or explore the best solutions. FernándezDelgad et al. [3] in a large scale study of various classifiers and
datasets note that even well trained data scientists choose only
classifiers from a range of familiar classifiers and do not necessarily
pick the best ones. Moreover, the developers are susceptible to
various machine learning pitfalls that existing APIs and frameworks
do not protect against [2]. As more and more applications move
to incorporate machine learning components in them, there is a
need to provide better software engineering and tool support to
developers.

Solar-Lezama [8] presented the Sketching framework to synthesize and generate code automatically from partial implementation
i.e. a high level specifications with holes. To generally express the
models as templates, we use a similar approach used by sketches.
Search mechanisms fill the holes to specify the models for specific
data and hyperparameters. Sketching uses SAT-based inductive
synthesis but our framework uses heuristics guided searching to
fill the holes. To reduce the search space and runtime, we exploit
partial evaluation [4]. While searching optimal hyperparameters,
we can treat a previously trained model as partial evaluated and
can update it for new combinations of hyperparameters. Thornton
et al. used Bayesian optimization in Auto-Weka [9] to search for
models and hyperparameters for a given dataset and integrated
it with Weka [5]. The focus of their technique is to optimize and
automate the model searching mechanism using a statistical approach i.e. Bayesian optimization. By contrast, we show that we can
develop a comprehensive machine learning framework using software engineering techniques such as sketching, partial evaluation,
and searching using rule based decision.
In this paper we present a proof of concept implementation of
our machine learning pipeline. We have selected a set of supervised1
learning algorithms and stored their templates, similar to sketches
with holes. The user can set up the problem by providing a data
source, its format, and select the column with class labels. Our
approach preprocesses the data, normalizes it, and analyzes the
data for its classes and features. We use a systematic search to find
the best hyperparameter configuration that fills the "holes" for each
classification algorithm. We only partially evaluate the search space
and employ heuristics to prune our search space. Our approach
ranks the algorithms based on their accuracy and returns the top
candidate.
Contributions: We make various novel contributions in this
work. We are the first to employ code synthesis and compiler optimization techniques to generate machine learning code. Existing
model selection techniques [9] do not prune search space and simply attempt to find an optimum value based on a given criteria, and
they do not work across various algorithms. We borrow meticulous
engineering steps traditionally used in compiler optimization to
prune the search space as well as transfer computation and information across algorithms.

2

ILLUSTRATIVE EXAMPLES

In this section we demonstrate how our prototype classifiers real
life datasets by selecting the right model with optimal hyperparameters and pruning search space using partial-evaluation and
1 Supervised machine learning algorithms assume that the training data has class labels

available

3

rules. Suppose we want to determine labels of ‘Glass Identification’
dataset from UCI repository [1]. We call the program as:
metaClassifier(format=’CSV’, source=’./glass.csv’,
verbose=True, hasHeader=True)
Some arguments like data source are mandatory while others like
data format are optional which are either inferred like data format or can be set as default such as default verbose=False. The
arguments are parsed and the program is initialized. The holes of
data acquisition sketches are filled by determining the data source
and format and the potential search space is built i.e. classifiers
and their hyperparameters’ candidate space. By default, the size of
search space for classification is more than a thousand i.e. the sum
of the combinations of hyperparameters of all classifiers under consideration. In case of the SVC, the hyperparameters’ candidate space
is given below having forty combinations of hyperparameters:

 kernel


C


kernel

CSVcandSpace =  


C



 deдree


APPROACH

Our approach to develop the framework is similar to fundamental
ML pipeline. Every module of pipeline automates the work to mitigate the human intervention and reduces the search space wherever
possible. Figure 1 demonstrates our pipeline approach. Module A
acquires the datasets from given source provided by a user as arguments. The test and training datasets are fed to Module B for
data preparation without any human intervention to standardize
the data like predicting the missing values and data transformation.
After common preprocessing, the framework inspects data in Module D to get its size i.e. the number of instances, features, number of
classes, and also run some simple tests like linear separability test.
Using the results of inspection and heuristics, Module E updates
the search space by including or excluding the linear models and
their hyperparameters—building the list of classifiers 𝒞 and the list
of their corresponding hyperparameters ℋ.
After preparation, the labeled (training) datasets is used for
model-selection using stratified K-fold cross validation where
the framework selects the best model without user intervention. In
Module C, labeled dataset is split in k stratified folds i.e. Dl abel ed =
{D 1 , D 2 , ..., D k }. Module F picks the classifier sketch C and its hyperparameters’ candidate space ℋ from search space and generates
code by filling the holes of sketch and the generated model is trained
using train set i.e. Dl abel ed \D i , 1 < i < k. In Module G, the labels
are predicted for test features i.e. D i using model testing. Accuracy
is computed using predicted labels and actual labels in Module H,
Model Evaluation. After each model evaluation, heuristics are
gathered to update and prune the search space for example if accuracy is not increased by increasing the value of max_iter in
Logistic Regression, we can remove all combinations of hyperparameters with higher values of max_iter. When the search space
exhausts, our framework selects the model with highest accuracy
to predict the labels of unlabeled (test) dataset in Module I and the
predicted labels are output to user.
Algorithm 1 outlines our framework to solve classification problems automatically. The framework takes minimum information
like data source and format, labels’ information, time budget, etc.
as arдs. It parses the arдs and sets the fields of options (line 2); a
dictionary holding global variables and behavior of the framework.


∈ {linear , rb f , siдmoid } 

∈ {1, 10, 100, 1000, 10000} 


∈ {poly}


 

∈ {1, 10, 100, 1000, 10000} 




∈ {3, 4, 5, 6, 7}


The dataset is analyzed to determine that it has ten features,
two hundreds and fourteen instances and six classes implying it
is a multiclass dataset and prune out all classifiers and/or hyperparameters from candidate space specific to binary-class datasets
like for Logistic Regression, multi_class ∈ {multinomial } instead
of multi_class ∈ {multinomial, ovr }, because ovr repeatedly uses
a binary approach to fit multiclass datasets. Next, linearly separable
test’ using perceptron is carried out. Its code is generated by filling
the holes of perceptron with default parameters (penalty=None,
alpha=0.0001 etc.) to classify the dataset and the accuracy is computed which is 43% (< 50%). Concluding that the data is not linearly
separable, the search space is updated by removing linear classifiers
such as linearSVC, and hyperparameters such as liblinear from
solvers for Logistic Regression and linear from kernels of SVC.
This reduces the search space to about 80% of the original.
After preliminary pruning, the program begins to search the best
model and optimal hyperparameters while dynamically pruning
the remaining search space. It starts from the sketch of SVC:
SVC(C=??, kernel=??, degree=??)
by filling the holes from the above given candidate space e.g. C=1,
kernel=rbf (linear kernel is pruned and degree is needed only if
kernel=poly). Next it tries kernel=rbf, C={10, 100} and determines that by increasing C beyond 10, the accuracy is not increasing
so rules out higher values of C. Same rule applies to max_iter and
some other numeric hyperparameters to reduce search space. Further models and their hyperparameters are tried while pruning
out the search space. Search space reduces to about 60% of the
original and when it exhausts, the model configuration with highest accuracy is selected to predict the labels, which is SVC(C=10,
kernel=rbf) in this case. Consider another example of ‘Wisconsin
Breast Cancer Dataset‘ from UCI dataset repository which is linearly
separable binary class dataset. Passing the linear separability test
i.e. running perceptron, LinearSVC on this dataset, we can rule
out all non-linear options (both classifiers and hyperparameters).

3.1

Model Selection, Hyperparameters
Optimization and Feature Selection

Our framework prototype treats the classification problem as a
meta-search — the combined selection of classifier, its best hyperparameters and features selection. Suppose we have a set of classifiers,
𝒞 = {C1 , C2 , ..., Cn } and their respective hyperparameters spaces
as ℋ = {H1 , H2 , ..., Hn }. Algorithm 1 from line 8 to 17 iterates
over the search space while updating and reducing. The search continues until interruption due to resource budget constraints or the
Í
search space 𝒮 = i |Hi | exhausts. In time constraints, where 𝒮
does not exhaust, to find the potentially best model, the searching
algorithm takes some decisions based on previous observations
(explained in §3.2). In a given dataset, each feature/attribute doesn’t
contribute to a solution equally. Line 9 selects best features of the
dataset for C. From line 10 to 15, the algorithm iterates over all feasible combinations of hyperparameters of Hi for a given classifier
2

Figure 1: Framework Pipeline
Algorithm 1 Meta Classifier
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:

models is generic enough to synthesize (generate) the final code
with specific data and hyperparameters.
Partial Evaluation: While searching the best hyperparameters
of a model, for two consecutive combinations (where usually, the
value of one hyperparameter is changed), we can use the previously
fitted model as a partially evaluated module and update it. For example, finding the optimal number of epochs for stochastic gradient
descent used by many ML models, instead of restarting iteration
and initializing coefficients to zeros, we can resume iterations by
retaining the coefficients. Moreover, different hyperparameters of
a model usually control independent properties of an underlying
algorithm.
Heuristics: Previous execution of a model with a combination of
hyperparameters may, in some cases, omit other combinations. For
example, while searching for the best hyperparameters for logistic
regression, if the accuracy does not improve with the increase of
max_iter (but keeping all other hyperparameters constant), we
can skip all combinations of hyperparameters with higher values of
max_iter. Similarly, the observations derived from the executions
of one model can lead to the prioritization of other models. This
raises the probability that the best model is selected despite interruption due to time constraints. For example, if we run SGDClassifier
and its accuracy with loss=perceptron is very low, we should try
classifier Perceptron at the end. We continue updating our search
space i.e. the list 𝒞 and hyperparameters ℋ based on the observations to reduce the search space and prioritize models.
Rule based Optimization: The knowledge and insights of the data
scientists guiding the search for the best performing model can be,
in fact, translated into a long rule-based list, thereby automating
the human decision making process, as in rule based expert systems.
Following are a few examples:

procedure metaClassifier(arдs)
options ← parse(arдs)
▷ options is global dictionary
Dt r ain , Dt est = acqreData()
preprocess(Dt r ain , Dt est )
▷ Common preprocessing
inspectData(Dt r ain , Dt est ) ▷ Inspect and run basic test
Build list 𝒞 and corresponding ℋ ▷ based on data tests
results ← ϕ
▷ To hold results of cross validation
while not interrupted And C ← 𝒞.removeHead() do
Dt r ain , f eatures = SelectFeatures(C, Dt r ain )
while ℋC not exhausts do
▷ ℋC is the grid of C
j
Initialize C with HC
▷ Set j t h combination
acc, std = evaluateModel(C, Dt r ain )
j
results.Add(acc, std, C, HC , f eatures)
Update ℋC
end while
Update 𝒞 and ℋ
end while
Get appropriate model i.e with max(accu) and min(std)
y = model .predict(Dt est )
return y
end procedure

Ci and using stratified cross validation, the scores (accuracy and std.
deviation) are calculated and recorded in results.

3.2

Optimizations

The naïve searching mechanism can end up owing limited time
budget and in those cases, sometime the searched model may not be
the best one. To mitigate these situations, we used different software
engineering techniques. In this subsection, we describe the nonexhaustive list of the optimization tactics used in our prototype
framework.
Sharing of Code and Computation: Machine learning models
solving similar problems like classification have tendency to share
code and computation to a reasonable extent. For example, instead
of loading data from either local storage or remote server for every
model separately, an obvious optimization is to load data once and
perform some common preprocessing before executing some model
on it.
Templatization and Code Generation: For generalization of the
models, we define them as templates in the same fashion as of the
sketches for program synthesis [8]. The partial implementation of

• Linearly Separable Test: We should start the search using
SVM classifier (i.e C 1 = SV M) without some kernel or with
linear kernel. If the score/accuracy is reasonably high, we
can conclude that data is potentially linearly separable and
we should first try other linear classifiers such as SVM with
fine tuning.
• Each hyperparameter is not used in every combination. Thus,
we can ignore these to reduce the search space. For instance,
in case of scikit-learn’s logistic regression, n_jobs and
warm_start hyperparameters are ignored if solver is set
to liblinear. Similarly, in case of sklearn.svm.SVC, the
hyperparameter degree is ignored for all kernels except
kernel=poly.
3

Dataset
Best Hyperparameters
Breast Cancer LR solver=liblinear, max_iter=10, penalty=l1, C=1
solver=newton-cg, max_iter=100,
Iris
LR multi_class=multinomial, pen=l2, C=100
Glass
SVC kernel=rbf, C=10
Ionosphere SVC kernel=rbf, C=10
Diabetes
SVC kernel=linear, C=1
Sonar
SVC kernel=rbf, C=1

Accu
0.98

parameters and hyperparameters based on earlier runs of other
algorithms or prior runs on a smaller dataset. This transfer of knowledge is the key to reducing the state space. In this paper we have
used heuristics to transfer information, but in the future we are
working on analytical solutions to this problem. The key advantage of an analytical solution is that it allows us to initialize the
parameters of the algorithms optimally thus reducing the time for
algorithm convergence.
Searching for Algorithm Architecture: We also see the possibility of extending our work to directly search the architecture of
ML algorithm, chaining different algorithms and composing them
like software modules. A simple example of compositional search
would be to have a DNN that combines DNNs to find nose, ears,
lips and eyes to find a face. This creates a possibility of building
Wide Neural Networks where parts are searched in parallel.
Optimizing for Commodity Hardware: MapReduce and Spark
have demonstrated the power of distributed system in exploring
large search spaces in fraction of seconds. We believe that search
based ML will become main stream as we port our search to exploit
large scale distributed systems.

1
0.76
0.88
0.78
0.82

Table 1: Best Models with Hyperparameters and Accuracy

4

IMPLEMENTATION AND EVALUATION

In order to evaluate our proposed framework, we implemented a
prototype having the following classifiers: Logistic Regression, Perceptron, Support Vector Machine (SVM) Classifier and Linear SVM
Classifier. Instead of implementing these classifiers from scratch, we
used scikit-learn [6] as the underlying models along with some
basic preprocessing and grid search to find the optimal hyperparameters. Our framework is designed generically to further incorporate
other classifiers such as Naive Bayes, as well as preprocessing techniques and optimization insights. In the same fashion, we can scale
it for other machine learning problems such as regression, clustering
etc.
We evaluated the prototype on some common datasets from UCI
Machine Learning Repository [1]. Table 1 shows the best model
with optimal hyperparameters, accuracies and execution time of
the test suit. In order to evaluate usability, we are interviewing ML
practitioners to gather data on the time, effort and cost splurged in
writing and debugging code. So far, we have anecdotal evidence that
even experienced ML users fall into common pitfalls such as not
using stratification, not identifying outliers, forgetting to normalize,
or using different scales on train and test sets resulting in loss of
precious time and effort. We plan to carry out a full scale qualitative
and quantitative usability study and anticipate that ML users and
particularly domain experts would benefit from the implementation
support and ML insights provided by the framework.

5

6

CONCLUSION

In this work we have demonstrated that software engineering techniques can make machine learning accessible to average users. We
are able to represent ML algorithms with holes that our approach
can efficiently and optimally fill with rules guided systematic search.
We have also implemented a machine learning pipeline to search
across different ML algorithms and transfer knowledge between
models being evaluated to minimize the human effort. The ultimate
objective of our research is to build an end-to-end machine learning
solution that can find the best model without any human intervention. The work presented here is a very promising first step in this
direction.

REFERENCES

FUTURE DIRECTIONS

[1] A. Asuncion and D.H. Newman. 2007. UCI Machine Learning Repository. (2007).
http://archive.ics.uci.edu/ml
7 common mistakes when doing Machine
[2] Cheng-Tao Chu. 2015.
Learning.
(March 2015).
http://www.kdnuggets.com/2015/03/
machine-learning-data-science-common-mistakes.html
[Online; posted
01-March-2015].
[3] Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim.
2014. Do We Need Hundreds of Classifiers to Solve Real World Classification
Problems? J. Mach. Learn. Res. 15, 1 (Jan. 2014), 3133–3181. http://dl.acm.org/
citation.cfm?id=2627435.2697065
[4] Yoshihiko Futamura. 1983. Partial computation of programs. In RIMS Symposia
on Software Science and Engineering. Springer, 1–35.
[5] Geoffrey Holmes, Andrew Donkin, and Ian H Witten. 1994. Weka: A machine
learning workbench. In Intelligent Information Systems, 1994. Proceedings of the
1994 Second Australian and New Zealand Conference on. IEEE, 357–361.
[6] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel,
Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss,
Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal
of Machine Learning Research 12, Oct (2011), 2825–2830.
[7] D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar
Ebner, Vinay Chaudhary, and Michael Young. 2014. Machine Learning: The
High Interest Credit Card of Technical Debt. In SE4ML: Software Engineering for
Machine Learning (NIPS 2014 Workshop).
[8] Armando Solar-Lezama. 2008. Program Synthesis by Sketching. Ph.D. Dissertation.
Berkeley, CA, USA. Advisor(s) Bodik, Rastislav. AAI3353225.
[9] Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2013.
Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference
on Knowledge discovery and data mining. ACM, 847–855.

Search based software engineering can revolutionize the field of
machine learning. We strongly believe that work presented in this
paper will be foundational in a massive effort to democratize machine learning and enable average users and domain experts to take
full advantage of the technology. We have abstracted and templatized ML algorithms thus exposing interfaces for searching it. We
foresee many promising future direction arising from our work:
Self Application of Machine Learning: One key observation
of our work is that we can apply machine learning to discover and
find better machine learning solutions. The curse of dimensionality
and search space explosion have been a key hindrance in applying
machine learning to itself. Initial approaches attempted to explore
the entire space or approach it with random searching. We have
presented a more systematic method that prunes the space based on
human insights, which is comparable to how data scientists make
decision and produce equivalent results. We believe this provides
an avenue to explore other ML techniques to pick the best ML
algorithm. For example, a Deep Neural Network (DNN) which
decides which ML approach will work best for the given data and
classification problem.
Transfer Learning among ML Algorithms: When data scientist run a machine learning algorithm, they naturally select the
4

