1

Software Engineers vs.
Machine Learning Algorithms:
An Empirical Study Assessing
Performance and Reuse Tasks
arXiv:1802.01096v2 [cs.SE] 7 Feb 2018

Nathalia Nascimento, Carlos Lucena, Paulo Alencar and Donald Cowan
Abstract—Several papers have recently contained reports on applying machine learning (ML) to the
automation of software engineering (SE) tasks, such as project management, modeling and development.
However, there appear to be no approaches comparing how software engineers fare against machine-learning
algorithms as applied to specific software development tasks. Such a comparison is essential to gain insight
into which tasks are better performed by humans and which by machine learning and how cooperative work or
human-in-the-loop processes can be implemented more effectively. In this paper, we present an empirical study
that compares how software engineers and machine-learning algorithms perform and reuse tasks. The
empirical study involves the synthesis of the control structure of an autonomous streetlight application. Our
approach consists of four steps. First, we solved the problem using machine learning to determine specific
performance and reuse tasks. Second, we asked software engineers with different domain knowledge levels to
provide a solution to the same tasks. Third, we compared how software engineers fare against
machine-learning algorithms when accomplishing the performance and reuse tasks based on criteria such as
energy consumption and safety. Finally, we analyzed the results to understand which tasks are better
performed by either humans or algorithms so that they can work together more effectively. Such an
understanding and the resulting human-in-the-loop approaches, which take into account the strengths and
weaknesses of humans and machine-learning algorithms, are fundamental not only to provide a basis for
cooperative work in support of software engineering, but also, in other areas.
Index Terms—Machine learning, human-in-the-loop, software engineer, automatic software engineering,
internet of things, empirical study

F

1

I NTRODUCTION

S

OFTWARE engineering processes can be very
complex, costly and time-consuming [1].
They typically consist of a collection of related
tasks [2] such as designing, implementing, maintaining, testing and reusing software applications
[3]. In addition, as software has become embedded in systems of all kinds, millions of computer
programs have to be corrected, adapted, and

•

•
•

Laboratory of Software Engineering (LES) at the Department of Informatics, Pontifical Catholic University of Rio de
Janeiro, Brazil. CAPES scholarship/Program 194/Process:
88881.134630/2016-01
E-mail: nnascimento,lucena@inf.puc-rio.br
See http://www.inf.puc-rio.br/∼nnascimento/
David R. Cheriton School of Computer Science. University
of Waterloo, Canada.
E-mail: palencar,dcowan@csg.uwaterloo.ca

enhanced [2]. As a result, the field of software
engineering requires millions of skilled Information Technology (IT) professionals to create
millions of lines of code, which must be installed,
configured, tuned, and maintained. According to
Kephart (2005) [4], in the near future, it will be extremely challenging to manage IT environments,
even for the most skilled IT professionals.
Several researchers have proposed the use of
artificial intelligence, especially machine-learning
(ML) techniques, to automate different software
engineering (SE) tasks [3], [5]–[20]. For example, Zhang has extensively studied this theme
recently and in [3] he stated that:
”The field of software engineering turns
out to be a fertile ground where many
software development and maintenance

2

tasks could be formulated as learning
problems and approached in terms of
learning algorithms.”
However, there is a lack of approaches to
compare how software engineers fare against
machine-learning algorithms for specific software development tasks. This comparison is critical in order to evaluate which S.E. tasks are
better performed by automation and which require human involvement or human-in-the-loop
approaches [21], [22]. In practice, because there
are no explicit comparisons between the tasks
performed by engineers and automated procedures, including machine learning, it is often not
clear when to use automation in a specific setting.
For example, a Brazilian company acquired a
software system to select petroleum exploration
models automatically, but the engineers decided
they could provide a better solution manually.
However, when there was a comparison of the
manual solution with the one provided automatically by the system, it became clear that the
automated solution was better. This illustrates
that a lack of comparisons makes choosing a
manual or an automated solution or a combined
human-in-the-loop approach difficult.
This paper, contains an empirical study [23] to
compare how software engineers and machinelearning algorithms achieve performance and
reuse tasks. The empirical study uses a case study
involving the creation of a control structure for
an autonomous streetlight application. The approach consists of four steps. First, the problem
was solved using machine learning to achieve
specific performance and reuse of tasks. Second, we asked software engineers with different
domain-knowledge levels to provide a solution
to achieve the same tasks. Third, we compared
how software engineers compare with machinelearning algorithms when accomplishing the performance and reuse tasks based on criteria such
as energy consumption and safety. Finally, the
results were analyzed to understand which tasks
are better performed by either humans or algorithms so that they can work together more
effectively.
Such an understanding is essential in realizing novel human-in-the-loop approaches in
which machine-learning procedures assist software developers in achieving tasks. Such humanin-the-loop approaches, which take into account
the strengths and weaknesses of humans and
machine-learning algorithms, are fundamental
not only to provide a basis for cooperative work

in software engineering, but also in other application areas.
This paper is organized as follows: Section
2 presents the empirical study describing research questions, hypotheses and the objective
of the study. Section 3 presents the method selected to collect our empirical data. Sections 4
and 5 present the experimental results. Section
6 presents the threats to the validity of our experiment. Section 7 presents the related work.
The paper ends with concluding remarks and
suggestions for future work.
1.1

Motivation

The theme of this paper, namely whether artificial intelligence such as machine learning, can
benefit software engineering, has been investigated since 1986, when Hebert A Simon published a paper entitled “Whether software engineering needs to be artificially intelligent” [24].
In this paper, Simon discussed “the roles that
humans now play versus the roles that could be
taken over by artificial intelligence in developing
computer systems.” Notwithstanding, in 1993,
Ian Sommerville raised the following question
[25]: “What of the future - can Artificial Intelligence make a contribution to system engineering?” In this paper [25], Sommervile performed a
literature review in applications of artificial intelligence to software engineering, and concluded
that:
“the contribution of AI will be in supporting...activities that are characterized
by solutions to problems which are
neither right nor wrong but which are
more or less appropriate for a particular situation...For example, requirements specification and analysis which
involves extensive consultation with domain experts and in project management.”
Several papers have since investigated the use
of Machine Learning (ML) [26] in solving different software engineering (SE) tasks [5]–[20], [27]–
[103]. These investigations include approaches
to: i) project management [27]–[49], dealing with
problems related to cost, time, quality prediction,
and resource management; ii) defect prediction
[50]–[84]; iii) requirements management, focusing on problems of classifying or representing requirements [85]–[88], or generating requirements
[89]; iv) software development, such as code generation [20], [68], [90]–[96], synthesis [97]–[101],
and code evaluation [102], [103].

3

Most of these papers present successful applications of machine learning in software engineering, showing that ML techniques can provide
correct automatic solutions to some SE problems.
However, very few papers discuss whether or
not a domain expert could propose a manual
solution more appropriate for the particular
situation. “More appropriate”, means a solution
that provides better performance or increases another quality that is important to a particular application scenario, such as user preference [104].
For example, in the medical and aviation engineering fields, trust [105] in a solution provided
to the end-user is an important factor to consider
for a solution to be more appropriate. However,
although many authors [106]–[109] have been
promoting the use of neural networks [110] in
medicine, Abbas et al. [105] and Castelvecchi
[111] are among the few authors who questioned:
“what is the trustworthiness of a prediction made
by an artificial neural network?”
In other application scenarios, such as many
of those related to the Internet of Things (IoT)
[112], [113], numerous authors [93], [95], [101],
[114] consider the reuse of a solution as an important quality. They agree that to achieve the
goal of billions of things connected to the Internet
over the next few years [112], it is necessary to
find ways to reduce time to market. For example,
it is desirable that the solution or parts of the
solution to design autonomous streetlights [96]
for a specific scenario could be reused to design
streetlights for another scenario.
In particular, the Internet of Things has considerably increased the number of approaches
that propose the use of machine learning to
automate software development [93]–[96], [101],
[115]. None of this research contains a comparison of their results to experiments designed by
IoT experts. For example, do Nascimento and Lucena [101], [116] developed a hybrid framework
that uses learning-based and manual program
synthesis for the Internet of Things (FIoT). They
generated four instances of the framework [101],
[108], [117], [118] and used learning techniques
to synthesize the control structure automatically.
These authors stated that the use of machine
learning made feasible the development of these
applications. However, they did not present any
experiment without using learning techniques. In
contrast, most of the solutions released for the
Internet of Things, such as Apple’s HomeKit’s
approach [119] and Samsung Smart Things, [120]
consider a software developer synthesizing the

control structure for each thing manually.

1.2

Objective

In this context, we decided to ask the following
question: “How do software engineers compare
with machine-learning algorithms?” To explore
this question, we selected the Internet of Things
as our application domain and then, compared
a solution provided by a skilled IoT professional
with a solution provided by a learning algorithm
with respect to performance and reuse tasks.
In short, Figure 1 depicts the theory [121] that
we investigate in this paper. According to the
theory, the variables that we intend to isolate
and measure are the performance and reusability achieved from three kinds of solutions: i)
solutions provided by learning techniques; ii)
solutions provided by software engineers with
IoT skills; and iii) solutions provided by software
engineers without IoT skills.
To evaluate the relationship among these variables, we performed an empirical study, using
FIoT [101]. As shown in Figure 1, we raised
four research questions (RQx) to investigate our
theory’s propositions (e.g hypotheses (H-RQx)).
We present these questions and hypotheses in
Section 2. To collect and analyze our empirical data, we performed a controlled experiment.
To perform this experiment, we reproduced the
problem of synthesizing the control structure
of autonomous streetlights using neuroevolution
(i.e. “a learning algorithm which uses genetic
algorithms to train neural networks” [122]) presented in [118]. Then, we invited 14 software
engineers to provide a solution for the same
problem using the same architecture and environment. Lastly, we compared the solution provided by the learning algorithm against the solutions provided by the software engineers. In
this application of autonomous streetlights, we
are considering a “more appropriate” solution
as one that presents a better performance in
the main scenario [118] or can be satisfactorily
reused in a new scenario, based on criteria such
as minimal energy consumption and safety
(that is, maximum visual comfort in illuminated
areas).

4

Actor

SE Tasks

Technology

Synthesize and reuse
the control structure of
autonomous things

Machine Learning
Techniques
Software Engineers

with IoT skills

without IoT skills

produce solutions
better than
(H-RQ1, H-RQ2)

Software System
Internet of Things

produce solutions better
than (H-RQ3, H-RQ4)
Improve (H-RQ1, H-RQ3)
Increase (H-RQ2, H-RQ4)

Performance
Reusability

Fig. 1. Theory [121]: Machine Learning can create solutions more appropriate than software engineers in the context of
the Internet of Things.

2

H OW DO SOFTWARE ENGINEERS COM WITH MACHINE - LEARNING ALGO RITHMS ? A N E MPIRICAL S TUDY A D DRESSING P ERFORMANCE AND R EUSE IN
THE I OT D OMAIN

RQ4. ...software engineers without IoT
skills with respect to their re-usability?

PARE

The experimental goal, based on the GoalQuestion-Metric (GQM) method [123] is to use
the Framework for the Internet of Things (FIoT)
for the purpose of comparing the use of an
automated approach against a manual approach
when synthesizing the control of autonomous
things with respect to their performance and
reuse.
For this purpose, we asked four research
questions (RQs) and performed a controlled experiment [23] (section 3) to investigate them.

2.2

Hypotheses

Each RQ is based on one or more hypotheses,
which are described next.
H - RQ1.
•

•

H - RQ2.
•

2.1

Questions

In terms of synthesizing the control structure of
autonomous things, how does the result from
a machine learning-based solution differ from
solutions provided by...
RQ1. ...software engineers with IoT
skills with respect to their performance?
RQ2. ...software engineers with IoT
skills with respect to their re-usability?
RQ3. ...software engineers without IoT
skills with respect to their performance?

H0. An ML-based approach does not improve the performance of autonomous
things compared to solutions provided by
IoT expert software engineers.
HA. An ML-based approach improves the
performance of autonomous things compared to solutions provided by IoT expert
software engineers.

•

H0. An ML-based approach does not increase the reuse of autonomous things
compared to solutions provided by IoT
expert software engineers.
HA. An ML-based approach increases the
reuse of autonomous things compared to
solutions provided by IoT expert software
engineers.

H - RQ3.
•

H0. An ML-based approach does not improve the performance of autonomous
things compared to solutions provided by
software engineers without experience in
IoT development.

5
•

HA. An ML-based approach improves the
performance of autonomous things compared to solutions provided by software
engineers without experience in IoT development.

TABLE 1
Implementing FIoT flexible points to synthesize streetlight
controllers using a ML-based approach.
FIoT Framework
Controller

H - RQ4.
•

•

H0. An ML-based approach does not increase the reuse of autonomous things
compared to solutions provided by software engineers without experience in IoT
development.
HA. An ML-based approach increases the
reuse of autonomous things compared to
solutions provided by software engineers
without experience in IoT development.

2.3 The object of the study: The Framework
for the Internet of Things (FIoT)
The Framework for the Internet of Things (FIoT)
[101] is a hybrid software framework that allows
the developer to generate controller structures for
autonomous things through learning or procedural algorithms.
If a researcher develops an application using
FIoT, the application will contain a Java software
component already equipped with modules for
detecting autonomous things in an environment,
assigning a controller to a specific thing, creating
software agents, collecting data from devices and
supporting the communication structure among
agents and devices.
Some features are variable and may be selected/developed according to the application type,
as follows: (i) a control module such as “if-else”,
neural network or finite state machine; (2) an
adaptive technique to synthesize the controller
at design-time, such as reinforcement learning
[124] or genetic algorithm; and (iii) an evaluation
process to evaluate the behavior of autonomous
things that are making decisions based on the
controller.
For example, Table 1 summarizes how the
“Streetlight Control” application will adhere
to the proposed framework using a machine
learning-based approach, while extending the
FIoT flexible points.
Table 2 summarizes how the “Streetlight Control” application will adhere to the proposed
framework using a solution provided by a software engineer, while extending the FIoT flexible
points.
Our goal is to provide both solutions to the
same application and compare the results based
on the same evaluation process.

Making Evaluation

Controller Adaptation
at design time

Light Control Application
Three Layer Neural Network
Collective Fitness Evaluation:
the solution is evaluated
based on the energy
consumption, the number of
people that finished their
routes after the
simulation ends, and the
total time spent by people
to move during their trip
Evolutionary Algorithm:
Generate a pool of
candidates to represent the
neural network parameters

TABLE 2
Implementing FIoT flexible points to synthesize streetlight
controllers using a solution provided by a Software
Engineer
FIoT Framework
Controller

Making Evaluation

Controller Adaptation
at design time

3

Light Control Application
if-else module provided by a
software engineer
Collective Fitness Evaluation:
the solution is evaluated
based on the energy
consumption, the number of
people that finished their
routes after the
simulation ends, and the
total time spent by people
to move during their trip
None

C ONTROLLED E XPERIMENT

The first step of the experiment was to reproduce
the experiment presented in [118] by using a not
supervised learning approach. Then, we invited
14 software engineers to provide a solution for
the same problem. Finally, we compared the solution provided through the learning algorithm
against solutions provided by the participants.
3.1

Participant Analysis

As we have described previously, the knowledge in the application domain is an important
variable in our empirical study. Therefore, before
performing the controlled experiment, we asked
participants to describe their experience with the
development of applications based on the Internet of Things, that is, developing distributed
systems with embedded characteristics, such as
providing each element of the system with sensors and actuators. As shown in Figure 2, 43% of

6

Medium
High
14%
Low

22%
21%
43%

None
Fig. 2. Experience of participants in developing applications based on the Internet of Things.

participants have never developed an application
based on the Internet of Things and 57% have
developed at least one application.
3.2

Experiment: Streetlight Application

In short, our experiment involves developing
autonomous streetlights. The overall goal of this
application is to reduce the energy consumption while maintaining appropriate visibility in
illuminated areas [118]. For this purpose, we
provided each streetlight with ambient brightness and motion sensors, and an actuator to
control light intensity. In addition, we also provided streetlights with wireless communicators
as shown in Figure 3. Therefore, the streetlights
are able to cooperate with each other to establish
the most likely routes for passers-by and thus
achieve the goal of minimizing energy consumption.

Lighting
(Dark/DIM/Light)
Presence
(Yes/No)
Data collected from
the closest street light
(0.0/0.5/1.0)
Previous Listening
Decision
(Yes/No)

Each streetlight has to execute three tasks
every second: data collection, decision-making
and action enforcement. The first task consists
of receiving data related to people flow, ambient brightness, data from the neighboring streetlights and current light status (activation level
of sensors and the previous output value of
listeningDecision). The second task consists of
analyzing collected data and making decisions
about actions to be enforced. The last task is
the action enforcement, which consists of setting
the value of three output variables: (i) listeningDecision, that enables the streetlight to receive
signals from neighboring streetlights in the next
cycle; (ii) wirelessTransmitter, a signal value to
be transmitted to neighboring streetlights; and
(iii) lightDecision, that activates the light’s OFF/DIM/ON functions.
The interested reader may consult a more
extensive paper about the application scenario
[118] 1 .
3.2.1 The Challenge
As we explained to the participants, the tasks
of collecting data and enforcing actions have
already been implemented. The challenge was
to provide a solution for the task of making
decisions, as depicted in Figure 4.

Light Decision
(OFF/DIM/ON)

Wireless
Transmitter
(0.0/0.5/1.0)
Listening
Decision
(Yes/No)

Fig. 3. Variables collected and set by streetlights.

Each streetlight in the simulation has a microcontroller that is used to detect the proximity
of a person, and control the closest streetlight. A
streetlight can change the status of its light to ON,
OFF or DIM.

Fig. 4. The challenge: how does a streetlight make decisions based on collected data?

We provided pseudocode that considered all
possible combinations of input variables. Then,
participants decided how to set output variables
according to the collected data. Part 2 of this
pseudocode is depicted in Figure 5.
Each participant provided a different solution. Therefore, we conducted the experiment by
1. All documents that we prepared to explain this application scenario to participants are available at
http://www.inf.puc-rio.br/.̃nnascimento/projects.html
2. The pseudocode that we provided to participants is
available at:
http://www.inf.puc-rio.br/.̃nnascimento/projects.html

7

Fig. 5. Small portion of the pseudocode of the decision module that was filled by participants.

if (lighting_sensor = Medium AND detected_person =
NO AND data_collected = 0.0 AND
previous_listening_decision = YES) then {
light_decision = __X_OFF____DIM____ON
wireless_transmitter = __X_0.0 ___0.5____1.0
listening_decision = _X__YES ____NO
} if (lighting_sensor = Medium AND detected_person =
YES AND data_collected = 0.0 AND
previous_listening_decision = YES) then {
light_decision = ___OFF__X__DIM____ON
wireless_transmitter = _X__0.0 ___0.5____1.0
listening_decision = __X_YES ____NO
} if (lighting_sensor = Dark AND detected_person = NO
AND data_collected = 0.0 AND
previous_listening_decision = NO) then {
light_decision = __X_OFF____DIM____ON
wireless_transmitter = ___0.0 __X_0.5____1.0
listening_decision = ___YES __X__NO
}

Fig. 6. Small portion of the rule decisions that was synthesized according to the learning-based approach.

using each one. In addition, we also considered
a “zeroed” solution, which always sets all values
to zero. This zeroed solution is supposed to be
the worst solution, since streetlights will always
switch their lights to OFF.
3.2.2 The solution generated by a machinelearning algorithm
We compared the results from all of these approaches to the result produced using the machine learning approach. As do Nascimento and
Lucena explain in [118], the learning approach
uses a three-layer feedforward neural network
combined with an evolutionary algorithm to generate decision rules automatically. Figure 6 depicts some of the rules that were generated by the
evolved neural network. The interested reader
can consult more extensive papers [101], [118] or
read Nascimento’s dissertation [116] (chap. ii, sec.
iii).
Based on the generated rules and the system
execution, we observe that using the solution
provided by the neural network, only the street-

lights with broken lamps emit “0.5” from their
wireless transmitters.
In addition, we also observed that a streetlight
that is not broken switches its lamp ON if it detects a persons proximity or receives “0.5” from a
wireless transmitter.
3.2.3

Scenario constraints

Before starting a solution, each participant
should consider the following constraints:
•

•
•

Do not take light numbering into account,
since your solution may be used in different scenarios (see an example of a scenario
in Figure 7).
Three streetlights will go dark during the
simulation.
People walk along different paths starting
at random departure points. Their role is
to complete their routes by reaching a destination point. The number of people that
finished their routes after the simulation
ends, and the total time spent by people

8

•

•
•
•

moving during their trip are the most
important factors for a good solution.
A person can only move if his current and
next positions are not completely dark.
In addition, we also consider that people
walk slowly if the place is partially devoid
of light.
The energy consumption also influences
the solution evaluation.
The energy consumption is proportional
to the light status (OFF/DIM/ON).
We also consider the use of the wireless
transmitter to calculate energy consumption (if the streetlight emits something
different from “0.0”, it consumes 0.1 of
energy).

Therefore, each solution is evaluated after the
simulation ends based on the energy consumption, the number of people that finished their
routes after the simulation ends, and the total
time spent by people moving during their trip.

Figure 7 depicts the elements that are part of the
application namely, streetlights, people, nodes
and edges.

Execution: 12 seconds
- A person moves from
one point to another in
one second or a second
and a half.
- Street lights execute
cycles of 1 second
ON
DIM
OFF
Broken Lamps
departure points
target points

Fig. 7. Simulated Neighborhood.

Nascimento and Lucena [118] modeled the
scenario as a graph, in which a node represents
a streetlight position and an edge represents
the smallest distance between two streetlights.
(completedP eople × 100)
The graph representing the streetlight network
pP eople =
(1)
consists of 18 nodes and 34 edges. Each node
totalP eople
represents a streetlight. In the graph, the yellow,
(totalEnergy × 100)
pEnergy = 11×(timeSimulation×totalSmartLights) gray, black and red triangles represent the street) light status (ON/DIM/OFF/Broken Lamp). Each
(
10
(2) edge is two-way and links two nodes. In addi(totalT imeT rip × 100)
tion, each edge has a light intensity parameter
pT rip = 3×timeSimulation
((
) × totalP eople)
that is the sum of the environmental light and the
(2)
(3) brightness from the streetlights in its nodes. Their
f itness = (1.0 × pP eople) − (0.6 × pT rip)−
goal is to simulate different lighting in different
(0.4 × pEnergy)
neighborhood areas.
(4)

Equations (1) through (4) show the values to
be calculated for the evaluation in which pP eople
is the percentage of the number of people that
completed their routes before the end of the
simulation out of the total number of people
in the simulation; pEnergy is the percentage of
energy that was consumed by streetlights out of
the maximum energy value that could be consumed during the simulation. We also considered
the use of the wireless transmitter to calculate
energy consumption; pT rip is the percentage of
the total duration time of peoples trips out of the
maximum time value that their trip could spend;
and f itness is the fitness of each candidate that
encodes the proposed solution.
3.2.4 Example - Simulating the environment
We showed participants the same simulated
neighborhood scenario that was used by the genetic algorithm to evolve the neural network.

1 second/1 second and a half

Fig. 8. Person moving in the simulated Neighborhood.

As depicted in Figure 8, only one person was
started in the scenario that we showed to participants. For instance, the person starting at point
0 has point 11 as a target. We ask participants to
provide a solution to streetlights to assure that

9

this person will conclude his route before the
simulation ends after 12 seconds.
3.2.5

New Scenario: Unknown environment

The second step of the experiment consists of
executing solutions from participants and the
learning approach in a new scenario, but with the
same constraints. This scenario, that is depicted
in Figure 9 was not used by the learning algorithm and was not presented to participants.
The goal of this new part of the experiment
is to verify if the decision module that was designed to control streetlights in the first scenario
can be reused in another scenario.

the solution provided by each one of the 14
participants 3 .
To provide a controlled experiment and be
able to compare the different solutions, we
started with only one person in the scenario and
manually we set the parameters that were supposed to be randomly selected, such as departure
and target points and broken lamps.
Each experiment execution consists of executing the simulated scenario three times: (i) night
(environmental light is equal to 0.0); (ii) late
afternoon (environmental light is equal to 0.5);
and (iii) morning (environmental light is equal
to 1.0). The main idea is to determine how the
solution behaves during different parts of the
day. Figure 10 depicts the percentage of energy
that was spent according to the environmental
light for each one of the 16 different solutions. As
we described previously, we also considered the
use of the wireless transmitter to calculate energy
consumption. As expected, as streetlights using
the zeroed decision never switch their lights ON
and never emit any signal, the energy consumed
using this solution is always zero. It is possible
to observe that only the solutions provided by
the learning algorithm and by the 5th and 11th
participants do not expend energy when the environmental light is maximum. In fact, according
to the proposed scenario, there is no reason to
turn ON streetlights during the period of the day
with maximum illumination.
100
90

.

Fig. 9. Simulating a new neighborhood.

80

0

70
60

(

50

.)1

In this new scenario, we also only started
one person, who has the point 18 (yellow point)
as departure and the point 8 as target. As the
scenario is larger, we established a simulation
time of 30 seconds.

40
30
20
10
0

.

4

E XPERIMENT - PART 1 - R ESULTS

We executed the experiment 16 times, only
changing the decision solution of the autonomous streetlights. In the first instance, we
set all outputs to zero (the zeroed solution) during the whole simulation, which is supposed to
be the worst solution. For example, streetlights
never switch their lights ON. In the second instance, we executed the experiment using the
best solution that was found by the learning
algorithm, according to the experiment presented
in [118]. Then, we executed the simulation for

.)1

).5

.)1

).5

%

.)1

).5

Fig. 10. Scenario1: Percentage of energy spent in different
parts of the day according to the participant solutions.

Figure 11 depicts the percentage of time that
was spent by the unique person in each one of
the simulations. As shown, the higher difference
between solutions occurs at night. If the time is
3. All files that were generated during the development
of this work, such as executable files and participants’
solutions results, are available at
http://www.inf.puc-rio.br/.̃nnascimento/projects.html

10

100%, it means that the person did not complete
the route, thus the solution did not work

4.1 Discussion: Participants Knowledge in
IoT Versus Results

100
90
80

) (
).

70
60
40

simulation time did not allow the person to finish
the route.

After executing the solution proposed by each
participant, we connect that solution’s results
with the participant’s knowledge in the IoT domain, as shown in Table 3.

50
30
20
10
0

TABLE 3
Correlation between participants expertises in the Internet
of Things with their solution results.

1) 0
). (

).

(

%

).

(

Fig. 11. Scenario1: Percentage of time spent by person
to conclude his route based on different parts of the day
according to the participant solutions.

Besides presenting the results of the different
solutions in different parts of the day, the best
solution must be the one that presents the best
result for the whole day. Thus, we calculated the
average of each one of the parameters (energy,
people, trip and fitness) that was achieved by
solutions in different parts of the day. Figure 12
depicts a common average. We also calculated a
weighted average, taking into account the duration of the parts of the day (we considered 12
hours for the night period, 3h for dim and 9h for
the morning), but the results were very similar.
100

Software
Engineer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Learning
zeroed

Experience
with IoT
Development
(None/Low/
Medium/
High)
High
None
High
Low
None
Low
Medium
None
High
None
None
Medium
None
Low

Solution
Performance
(Fitness
Average)

Does
the
solution
work?

55.48
26.99
62.88
62.49
30.50
51.09
54.37
16.59
28.62
61.60
29.67
47.81
30.32
56.91
59.53
28.33

Y
N
Y
Y
N
Y
Y
N
N
Y
N
Y
N
Y
Y
N

90
80
70

%

60

62.88
55.48

61.60

62.49
51.09

56.91 59.53

54.37

47.81

50
40
30
20

26.99

30.50

28.62

29.67

30.32

28.33

16.59

10
0

Fitness

We observe a significant difference between
results from software engineers with any experience in IoT development and results from
software engineers without experience in IoT
development. Participant 10 is the only individual without knowledge of IoT that provided a
solution that works and participant 9 is the only
individual with any knowledge of IoT that did
not provide a working solution.

Fig. 12. Scenario1: Average of energy, trip and fitness
calculated for the different parts of the day according to the
participant solutions.

4.2

As shown in Figure 12, based on the fitness
average, three participants namely 3, 4 and 10
provided a solution slightly better than the solution provided by the learning algorithm. Five
other participants provided a solution that works
and the remaining six provided a solution that
does not work. As explained earlier, we have
been considering an incorrect solution as one
in which the person did not finish the route
before the simulation ends. Even increasing the

In this section, we investigate the hypotheses
related to the solutions’ performance evaluation
(i.e H-RQ1 and H-RQ3), as presented in subsection 2.2. Thus, we performed statistical analyses,
as described by Peck and Devore [125], of the
measures presented in Table 3.
As shown in Table 4, we separated the results
of the experiments into two groups: i) software
engineers with IoT knowledge and ii) software
engineers without IoT knowledge. Then, we calculated the mean and the standard deviation of

Hypothesis Testing

11

the results achieved by each group performing
the experiment and compare each result against
the value achieved using the ML-based solution.
4.2.1 How does the evaluation result from a machine learning-based solution differ from solutions
provided by IoT expert software engineers with
respect to their performance?
H - RQ1.
•

•

H0. An ML-based approach does not improve the performance of autonomous
things compared to solutions provided by
IoT expert software engineers.
HA. An ML-based approach improves the
performance of autonomous things compared to solutions provided by IoT expert
software engineers.

The first null hypothesis H0 claims that there
is no difference between the mean performance
for IoT expert software engineers’ solutions and
the ML-based approach solution. The alternative
hypothesis claims that the ML-based approach
solution improves the performance of the application in comparison to IoT expert software engineers’ solutions. Thus, the claim is that the true
IoT expert software engineers’ solutions mean
is below the performance achieved by the MLbased approach, that is = 59.53.
Therefore, we used the ML performance as
our hypothesizedvalue to test the following onesided hypothesis:
H0: µs e = 59.53
H1: µs e < 59.53
where µs e denotes the true mean performance for all IoT expert software engineers’ solutions.
For instance, we restricted the population
sample to the number of software engineers that
confirmed having experience with developing
applications for the Internet of Things. As shown
in Table 4, the performance mean (x) of the IoT
expert software engineers’ solutions is 52.46 and
the standard deviation (σ ) is 10.91. To verify
if the data that we have is sufficient to accept
the alternative hypothesis, we need to verify the
probability of rejecting the null hypothesis [125].
Assuming that the H0 is true, and using a statistical significance level [125] of 0.01 (the chance of
one in 100 of making an error), we computed the
test statistic (t − statistic), as follows [125]:

t − statistic : t =

According to t-statistic theory, we can safely
reject our null hypothesis if the t − statistic
value is below the negative t − criticalvalue
(threshold) [125]. This negative t − criticalvalue
bounds the area of rejection of a T-distribution,
as shown in Figure 13. In our experiment, as we
specified a statistical significance level of 0.01,
the probability of getting a T-value less or equal
than the negative t − criticalvalue is 1%. We calculated the tcriticalvalue of this T-distribution
according to the T-table presented in Peck and
Devore (2011, pg 791) [125]. Accordingly, for a
distribution with 7 degrees of freedom (see Table
4) and a confidence level of 99%, the negative
tcriticalvalue is -3.00. As we depicted in Figure
13, the test statistic of our sample is higher than
the tcriticalvalue.
norm.php 850×350 pixels

12/11/17, 6:12 PM

Fig. 13. Hypothesis H - RQ1 Test Graph.

As the test statistic does not fall in the critical
region, we cannot safely reject this null hypothesis. Based on a t-value of -1.83 and a degree of
freedom of 7, we could reject our null hypothesis
only if we had reduced the precision of our
experiment to 85%. Thus, we would fail to reject
the null hypothesis and would not accept the
alternative hypothesis. Therefore, we cannot
state that an ML-based approach improves the
performance of autonomous things compared
to solutions provided by IoT expert software
engineers.
http://www.imathas.com/stattools/norm.php

Page 1 of 1

4.2.2 How does the evaluation result from a machine learning-based solution differ from solutions
provided by software engineers without IoT skills
with respect to their performance?
H - RQ3.
•

(x − hypothesizedvalue)
t − statistic : t =
(5)
( √σn )

(52.46 − 59.53)
= −1.83 (6)
√ )
( 10.91
8

H0. An ML-based approach does not improve the performance of autonomous
things compared to solutions provided by

12

TABLE 4
Data to perform test statistic.

Variable
Software
Engineers
Software
Engineers
with IoT
knowledge
Software
Engineers
without IoT
knowledge
Machinelearning
based
approach

•

Standard
deviation
σ

Degrees
of
freedom
(n-1)

t
critical
value
(.99%)

49.45

16.00

13

2.65

52.46

54.92

10.91

7

3.00

32.61

30.00

15.15

5

3.37

n
samples

Highest
value

Mean
x

Median

14

62.88

43.95

8

62.88

6

61.60

1

59.53

software engineers without experience in
IoT development.
HA. An ML-based approach improves the
performance of autonomous things compared to solutions provided by software
engineers without experience in IoT development.

For instance, we restricted the population
sample to the number of software engineers that
confirmed not having experience with developing applications for the Internet of Things. As
shown in Table 4, the performance mean (x) of
the solutions from software engineers without
experience in IoT development is 32.61 and the
standard deviation (σ ) is 15.15. Thus, the

t − statistic : t =

(32.61 − 59.53)
= −4.35 (7)
√ )
( 15.15
6

As shown in Table 4, this T-distribution has 5
degrees of freedom. Thus, for a confidence level
of 99%, the negative tcriticalvalue is -3.37. As
we depicted in Figure 14, the test statistic of our
sample is below the tcriticalvalue .
norm.php 850×350 pixels

Fig. 14. Hypothesis H - RQ3 Test Graph.

12/11/17, 6:14 PM

As the t − statistic value is below the
negative tcriticalvalue (-4.35 < −3.37), we can
safely reject the null hypothesis, assuring that
the error chancing of making an error is lower
than 1%. Therefore, we accepted the alternative
hypothesis: An ML-based approach improves the
performance of autonomous things compared to
solutions provided by software engineers with no
experience in IoT development.

5

E XPERIMENT - PART 2 - R ESULTS

As explained previously, the second part of the
experiment consists of translating the solution
provided by machine learning and participants
to an unknown environment. In this second part
of the experiment, we also executed the simulation 16 times: for each one of the participants’
solutions, for the machine-learning solution and
for the zeroed solution.
Table 5 shows the results that were achieved
by the different solutions at night in a simulation
of 30 seconds. As shown, most of the solutions
did not work. The person in these simulations
did not finish the route even when we increased
the simulation time. Only the solution provided
by the machine-learning algorithm and by participant 12 worked. Remember, this scenario was
not used by the machine-learning algorithm during the training process. This solution was provided through machine learning for the first scenario and it was just reused in this new scenario.
In other words, we did not restart the machinelearning process.
We selected only those solutions that worked
and verified their results for the other periods of

13

the day (morning and late afternoon). As shown
in Table 6, when considering the whole day, the
machine-learning approach presented the best
result. Because the average time for the trip
was a little higher using the machine-learning
approach, the difference in energy consumption
between the two solutions is considerably higher.

of 14 engineers, only one participant, who has
experience with IoT development, provided a
solution that worked.

Bad
93%

TABLE 5
Using the same solution in a different environment - only
at night.
Software
Engineer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Learning
zeroed

Energy%

People%

Trip%

Fitness

6.50
2.77
6.62
4.30
2.58
6.88
8.33
2.33
3.77
3.78
11.36
50.56
2.77
4.50
24.44
0

0
0
0
0
0
0
0
0
0
0
0
100
0
0
100
0

100
100
100
100
100
100
100
100
100
100
100
42.22
100
100
61.11
100

-42.60
-41.11
-42.65
-41.72
-41.03
-42.75
-43.33
-60.26
-41.51
-60.18
-44.54
54.43
-41.11
-41.80
53.55
-40

TABLE 6
Using the same solution in a different environment - day
average.

Average
Participant
12
Average
Learning

5.1

Energy%

People%

Trip%

Fitness

50.52

100

38.14

56.90

8.46

100

46.29

68.83

0% Work better than ML
7%
Work

Fig. 15. Participants’ solution results in the second scenario.

Therefore, we can safely reject the null hypothesis and accept both alternative hypotheses:
1)

2)

6

H-RQ2: H1: An ML-based approach increases the reuse of autonomous things
compared to solutions provided by IoT
expert software engineers.
H-RQ4: H1: 2) An ML-based approach
increases the reuse of autonomous things
compared to solutions provided by software engineers with no experience with
IoT development.

D ISCUSSION

In this section, we analyze the empirical experimental results to understand which tasks are
better performed by humans and which by algorithms. This is important for selecting whether
software engineers or machine learning can accomplish a specific task better.
In our empirical study, in which we have assessed performance and reuse tasks, we accepted
three alternative hypotheses and rejected one:

Hypothesis Testing

In this section, we investigate the hypotheses related to the solutions’ reuse evaluation, that is HRQ2 and H-RQ4, as presented in subsection 2.2.
Their alternative hypotheses state that an MLbased approach improves the performance of autonomous things compared to solutions provided
by software engineers, software engineers with
experience in IoT development, and software engineers without experience in IoT development,
respectively. We planned to perform a statistical development to evaluate these hypotheses.
However, as depicted in Figure 15, in the new
scenario, 0% of participants provided a result
better than the result provided by the machinelearning solution. In addition, from the group

Accepted:
1)

2)

3)

An ML-based approach improves the
performance of autonomous things compared to solutions provided by software
engineers without experience with IoT
development.
An ML-based approach increases the
reuse of autonomous things compared
to solutions provided by IoT expert software engineers.
An ML-based approach increases the
reuse of autonomous things compared
to solutions provided by software engineers without experience with IoT development.

14

Based on these results, we have found evidence that the use of machine-learning techniques can perform some SE tasks better than
software engineers, considering solutions that
improve performance and increase reuse. As illustrated in the experimental results, only one
of the 14 software engineers provided a solution
that could be reused in a new scenario. Further
none of those software engineers provided a solution that works better than the ML’s solution
in this new scenario. If the flexibility of the application is the most important factor, based on
our results, we can safely recommend the use of
machine learning.
However, if we had considered performance
as the only important factor to evaluate the quality of these solutions, we have found evidence
that software engineers can perform SE tasks better than machine learning, considering “better”
as a solution that improves performance. As described in our experiments, we cannot state that
ML improves the performance of an application
in comparison to solutions provided by domain
expert software engineers. This is also an interesting result as many researchers, especially in the
IoT domain, have strictly focused on automating
software development.
In brief, our experiment indicates that in some
cases, software engineers outperform machinelearning algorithms, whereas in other cases, they
do not. The evidence shows that it is important to
know which one performs better in different situations in order to determine ways for software
engineers to work cooperatively and effectively
with automated machine-learning procedures.

(2016) [127] report the diversity of participants as
another possible threat. Therefore, in our study,
we needed to be aware of at least two threats
to validity namely: we have selected a sample of
only 14 participants, which may not be enough
to achieve conclusive results; and our sample
consisted of only graduate students from two
Brazilian universities. Such a group may not be
representative of all software engineers, who may
have substantially more professional experience
and background.
To mitigate the problems of the number of
participants and their diversity, we selected our
participants carefully. All of them have at least
two years of experience with software development. In addition, we allowed participants to
solve the problem by manipulating a pseudocode
version, thereby avoiding gaps in the participants’ knowledge, such as experience with a particular programming language or architecture.
Note that a survey was used to select participants
and they all indicated a level of experience with
pseudocode. The pseudocode provided by each
participant was carefully translated into Java as
this is the language supported by the Framework
for the Internet of Things.
Oizumi et al. (2017) reported a third threat
to validity in [126], namely, possible misunderstandings during the study. To mitigate this problem of misunderstandings, we asked all participants to write about their understanding of the
problem both before and after applying the solution. All participants indicated that they understood the task completely. We also asked them
about their confidence in their proposed solution. Most of them evaluated their own solution
with the highest grade, allowing us to increase
our confidence in the experimental results. In
addition, we assisted the participants during the
entire study, making sure they understood the
experimental task

7

8

Rejected:
1) An ML-based approach improves the
performance of autonomous things compared to solutions provided by IoT expert software engineers.

T HREATS TO VALIDITY

Although we have designed and conducted the
experiments carefully, there are always factors
that can challenge the experiments validity. Some
threats to validity as described in [123] could
indeed limit the legitimacy of our results. In this
section, we present the actions taken to mitigate
their impact of these factors on the research results.
As Oizumi et al. (2017) report in [126], the
number of participants in the study can be a
threat to validity. In addition, Fernandes et al

R ELATED W ORK

Comparing intelligent machines to the ability of
a person to solve a particular problem is not
a new approach. This kind of discussion has
been promoted since the beginning of Artificial
Intelligence. For example, in 1997, an important
moment in the history of technology happened
with Garry Kasparov’s 1997 chess match against
the IBM supercomputer Deep Blue [128].
Recently, Silver et al. (2016, 2017) [129], [130]
published a paper in the Nature Science Journal,
comparing the performance of a ML technique

15

against the results achieved by the world champion in the game of Go. In [130], Silver et al.
(2017) state that their program “achieved superhuman performance.”
Whiteson et al. [122] indirectly performed this
comparison, by evaluating the use of three different approaches of the neuroevolution learning
algorithm to solve the same tasks: (i) coevolution,
that is mostly unassisted by human knowledge;
(ii) layered learning, that is highly assisted; and
(iii) concurrent layered learning, that is a mixed
approach. The authors state that their results
“demonstrate that the appropriate level of human assistance depends critically on the difficulty of the problem.”
Furthermore, there is also a new approach
in machine learning, called Automatic Machine
Learning (Auto-ML) [100], which uses learning
to set the parameters of a learning algorithm automatically. In a traditional approach, a software
engineer with machine learning skills is responsible for finding a good configuration for the
algorithm parameters. Zoth and Lee [100] present
an Auto-ML-based approach to design a neural
network to classify images of a specific dataset.
In addition, they compared their results with
the previous state-of-the-art model, which was
designed by an ML expert engineer. According
to Zoth and Lee [100] , their AutoML-based approach “can design a novel network architecture
that rivals the best human-invented architecture
in terms of test set accuracy.” Zoth and Lee
also showed that a machine-learning technique
is capable of beating a software engineer with
ML skills in a specific software engineering task,
but the authors do not discuss this subject in the
paper.
Our paper appears to be the first to provide an empirical study to investigate the use
of a machine-learning techniques to solve a
problem in the field of Software Engineering,
by comparing the solution provided by a MLbased approach against solutions provided by
software engineers.

9

C ONCLUSION AND F UTURE W ORK

Several researchers have proposed the use of
machine-learning techniques to automate software engineering tasks. However, most of these
approaches do not direct efforts toward asking
whether ML-based procedures have higher success rates than current standard and manual
practices. A relevant question in this potential

line of investigation is: “Could a software engineer solve a specific development task better than
an ML algorithm?”. Indeed, it is fundamental to
evaluate which tasks are better performed by engineers or ML procedures so that they can work
together more effectively and also provide more
insight into novel human-in-the-loop machinelearning approaches to support SE tasks.
This paper appears to be the first to provide an empirical study comparing how software engineers and machine-learning algorithms
achieve performance and reuse tasks. In brief,
as a result of our experiment, we have found
evidence that in some cases, software engineers
outperform machine-learning algorithms, and in
other cases, they do not. Further, as is typical in
experimental studies, although we have designed
and conducted the experiment carefully, there are
always factors that can threaten the experiment’s
validity. For example, some threats include the
number and diversity of the software engineers
involved in our experiment.
Understanding how software engineers fare
against ML algorithms is essential to support
new methodologies for developing human-inthe-loop approaches in which machine learning
automated procedures assist software developers
in achieving their tasks. For example, methodologies to define which agent (engineers or automated ML procedure) should execute a specific
task in a software development set. Based on
this understanding, these methodologies can provide a basis for software engineers and machine
learning algorithms to cooperate in Software Engineering development more effectively.
Future work to extend the proposed experiment includes: (i) conducting further empirical
studies to assess other SE tasks, such as design,
maintenance and testing; (ii) experimenting with
other machine-learning algorithms such as reinforcement learning and backpropagation; and
(iii) using different criteria to evaluate task execution.
Possible tasks that could be investigated (refer
to (i)) include programming tasks, in which case
tasks performed by software development teams
and ML algorithms are compared. For example,
we could invite software developers from the
team with the highest score in the last ACM International Collegiate Programming Contest [131],
which is one of the most important programming
championships in the world, to be involved in
this comparison. This competition evaluates the
capability of software engineers to solve complex

16

software problems. Software engineers are classified according to the number of right solutions,
performance of the solutions and development
time.
Another line of investigation could address
the use of different qualitative or quantitative
methodologies. For example, the task execution comparison could rely on reference performances, such as the performance of highly
successful performers [100], [129], [130] . This
research work can also be extended by proposing, based on the comparison between the performance of engineers and ML algorithms, a
methodology for more effective task allocation.
This methodology could, in principle, lead to
more effective ways to allocate tasks such as software development in cooperative work involving humans and automated procedures. Such
human-in-the-loop approaches, which take into
account the strengths and weaknesses of humans
and machine learning algorithms, are fundamental to provide a basis for cooperative work in
software engineering and possibly in other areas.

ACKNOWLEDGMENTS
This work has been supported by the Laboratory of Software Engineering (LES) at PUC-Rio.
Our thanks to CAPES, CNPq, FAPERJ and PUCRio for their support through scholarships and
fellowships. We would also like to thank the
software engineers who participated in our experiment.

R EFERENCES
[1]
[2]

[3]

[4]

[5]

[6]

F. Brooks and H. Kugler, No silver bullet.
April, 1987.
R. S. Pressman, Software engineering: a practitioner’s approach.
Palgrave Macmillan,
2005.
Q. Zhang, “Software developments,” Engineering Automation for Reliable Software, p.
292, 2000.
J. O. Kephart, “Research challenges of autonomic computing,” in Software Engineering,
2005. ICSE 2005. Proceedings. 27th International Conference on. IEEE, 2005, pp. 15–22.
J. Mostow, “Foreword what is ai? and what
does it have to do with software engineering?” IEEE Transactions on Software Engineering, vol. 11, no. 11, p. 1253, 1985.
D. Barstow, “Artificial intelligence and software engineering,” in Proceedings of the 9th

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

international conference on Software Engineering. IEEE Computer Society Press, 1987,
pp. 200–211.
D. Partridge, “Artificial intelligence and
software engineering: a survey of possibilities,” Information and Software Technology,
vol. 30, no. 3, pp. 146–152, 1988.
L. C. Cheung, S. Ip, and T. Holden, “Survey
of artificial intelligence impacts on information systems engineering,” Information
and Software Technology, vol. 33, no. 7, pp.
499–508, 1991.
D. Partridge, Artificial Intelligence in Software Engineering. Wiley Online Library,
1998.
A. Van Lamsweerde and L. Willemet, “Inferring declarative requirements specifications from operational scenarios,” IEEE
Transactions on Software Engineering, vol. 24,
no. 12, pp. 1089–1114, 1998.
G. D. Boetticher, “Using machine learning to predict project effort: Empirical case
studies in data-starved domains,” in Model
Based Requirements Workshop.
Citeseer,
2001, pp. 17–24.
F. Padberg, T. Ragg, and R. Schoknecht,
“Using machine learning for estimating the
defect content after an inspection,” IEEE
Transactions on Software Engineering, vol. 30,
no. 1, pp. 17–28, 2004.
D. Zhang, “Applying machine learning algorithms in software development,” in The
Proceedings of 2000 Monterey Workshop on
Modeling Software System Structures, 2000,
pp. 275–285.
——, “Machine learning in value-based
software test data generation,” in Tools with
Artificial Intelligence, 2006. ICTAI’06. 18th
IEEE International Conference on.
IEEE,
2006, pp. 732–736.
D. Zhang and J. J. Tsai, Machine learning
applications in software engineering. World
Scientific, 2005, vol. 16.
D. Zhang, “Machine learning and valuebased software engineering: a research
agenda.” in SEKE, 2008, pp. 285–290.
T. M. Khoshgoftaar, “Introduction to the
special issue on quality engineering with
computational intelligence,” 2003.
D. Zhang, “Machine learning and valuebased software engineering,” in Software
Applications: Concepts, Methodologies, Tools,
and Applications.
IGI Global, 2009, pp.
3325–3339.

17

[19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28]

[29]

D. Zhang and J. J. Tsai, “Machine learning
and software engineering,” in Tools with
Artificial Intelligence, 2002.(ICTAI 2002). Proceedings. 14th IEEE International Conference
on. IEEE, 2002, pp. 22–29.
M. D. Kramer and D. Zhang, “Gaps: a
genetic programming system,” in Computer
Software and Applications Conference, 2000.
COMPSAC 2000. The 24th Annual International. IEEE, 2000, pp. 614–619.
A. Holzinger, M. Plass, K. Holzinger, G. C.
Crişan, C.-M. Pintea, and V. Palade, “Towards interactive machine learning (iml):
applying ant colony algorithms to solve
the traveling salesman problem with the
human-in-the-loop approach,” in International Conference on Availability, Reliability,
and Security. Springer, 2016, pp. 81–95.
A. Holzinger, “Interactive machine learning for health informatics: when do we
need the human-in-the-loop?” Brain Informatics, vol. 3, no. 2, pp. 119–131, 2016.
S. Easterbrook, J. Singer, M.-A. Storey, and
D. Damian, “Selecting empirical methods
for software engineering research,” Guide to
advanced empirical software engineering, pp.
285–311, 2008.
H. A. Simon, “Whether software engineering needs to be artificially intelligent,” IEEE
Transactions on Software Engineering, no. 7,
pp. 726–732, 1986.
I. Sommerville, “Artificial intelligence and
systems engineering,” Prospects for Artificial Intelligence: Proceedings of AISB’93,
29 March-2 April 1993, Birmingham, UK,
vol. 17, p. 48, 1993.
R. S. Michalski, J. G. Carbonell, and T. M.
Mitchell, Machine learning: An artificial intelligence approach.
Springer Science &
Business Media, 2013.
A. Marchetto and A. Trentini, “Evaluating
web applications testability by combining
metrics and analogies,” in Information and
Communications Technology, 2005. Enabling
Technologies for the New Knowledge Society:
ITI 3rd International Conference on. IEEE,
2005, pp. 751–779.
S. Bouktif, F. Ahmed, I. Khalil, and G. Antoniol, “A novel composite model approach
to improve ality prediction,” Information
and Software Technology, vol. 52, no. 12, pp.
1298–1311, 2010.
L. Radlinski, “A survey of bayesian net
models for software development effort

[30]

[31]

[32]

[33]

[34]

[35]

[36]

[37]

[38]

[39]

prediction,” International Journal of Software
Engineering and Computing, vol. 2, no. 2, pp.
95–109, 2010.
W. Zhang, Y. Yang, and Q. Wang, “Handling missing data in software effort prediction with naive bayes and em algorithm,” in Proceedings of the 7th International
Conference on Predictive Models in Software
Engineering. ACM, 2011, p. 4.
Ł. Radliński, “A framework for integrated software quality prediction using
bayesian nets,” Computational Science and
Its Applications-ICCSA 2011, pp. 310–325,
2011.
P. O. O. Sack, M. Bouneffa, Y. Maweed,
and H. Basson, “On building an integrated
and generic platform for software quality
evaluation,” in Information and Communication Technologies, 2006. ICTTA’06. 2nd, vol. 2.
IEEE, 2006, pp. 2872–2877.
M. Reformat and D. Zhang, “Introduction to the special issue on:“software
quality improvements and estimations
with intelligence-based methods”,” Software Quality Journal, vol. 15, no. 3, pp. 237–
240, 2007.
B. Twala, M. Cartwright, and M. Shepperd,
“Applying rule induction in software prediction,” in Advances in Machine Learning
Applications in Software Engineering. IGI
Global, 2007, pp. 265–286.
V. U. Challagulla, F. B. Bastani, and I.-L.
Yen, “High-confidence compositional reliability assessment of soa-based systems using machine learning techniques,” in Machine Learning in Cyber Trust.
Springer,
2009, pp. 279–322.
R. C. Veras, S. R. Meira, A. L. Oliveira,
and B. J. Melo, “Comparative study of clustering techniques for the organization of
software repositories,” in Hybrid Intelligent
Systems, 2007. HIS 2007. 7th International
Conference on. IEEE, 2007, pp. 372–377.
I. Birzniece and M. Kirikova, “Interactive
inductive learning service for indirect analysis of study subject compatibility,” in Proceedings of the BeneLearn, 2010, pp. 1–6.
D. B. Hanchate, “Analysis, mathematical modeling and algorithm for software
project scheduling using bcga,” in Intelligent Computing and Intelligent Systems
(ICIS), 2010 IEEE International Conference on,
vol. 3. IEEE, 2010, pp. 1–7.
Z. Xu and B. Song, “A machine learning ap-

18

[40]

[41]

[42]

[43]

[44]

[45]

[46]

[47]

[48]

[49]

plication for human resource data mining
problem,” Advances in Knowledge Discovery
and Data Mining, pp. 847–856, 2006.
J. Wen, S. Li, Z. Lin, Y. Hu, and C. Huang,
“Systematic literature review of machine
learning based software development effort estimation models,” Information and
Software Technology, vol. 54, no. 1, pp. 41–
59, 2012.
E. Rashid, S. Patnayak, and V. Bhattacherjee, “A survey in the area of machine learning and its application for software quality
prediction,” ACM SIGSOFT Software Engineering Notes, vol. 37, no. 5, pp. 1–7, 2012.
H. A. Al-Jamimi and M. Ahmed, “Machine
learning-based software quality prediction
models: state of the art,” in Information
Science and Applications (ICISA), 2013 International Conference on. IEEE, 2013, pp. 1–4.
Ł. Radliński, “Enhancing bayesian network
model for integrated software quality prediction,” in Proc. Fourth International Conference on Information, Process, and Knowledge
Management, Valencia. Citeseer, 2012, pp.
144–149.
F. Pinel, P. Bouvry, B. Dorronsoro, and S. U.
Khan, “Savant: Automatic parallelization
of a scheduling heuristic with machine
learning,” in Nature and Biologically Inspired
Computing (NaBIC), 2013 World Congress on.
IEEE, 2013, pp. 52–57.
D. Novitasari, I. Cholissodin, and W. F.
Mahmudy, “Optimizing svr using local
best pso for software effort estimation,”
Journal of Information Technology and Computer Science, vol. 1, no. 1, 2016.
Ł. Radliński, “Towards expert-based modelling of integrated software quality,” Journal of Theoretical and Applied Computer Science, vol. 6, no. 2, pp. 13–26, 2012.
T. Rongfa, “Defect classification method
for software management quality control
based on decision tree learning,” in Advanced Technology in Teaching-Proceedings
of the 2009 3rd International Conference on
Teaching and Computational Science (WTCS
2009). Springer, 2012, pp. 721–728.
R. Rana and M. Staron, “Machine learning approach for quality assessment and
prediction in large software organizations,”
in Software Engineering and Service Science
(ICSESS), 2015 6th IEEE International Conference on. IEEE, 2015, pp. 1098–1101.
H. Wang, M. Kessentini, W. Grosky, and

[50]

[51]

[52]

[53]

[54]

[55]
[56]

[57]

[58]

[59]

H. Meddeb, “On the use of time series
and search based software engineering for
refactoring recommendation,” in Proceedings of the 7th International Conference on
Management of computational and collective
intElligence in Digital EcoSystems. ACM,
2015, pp. 35–42.
V. U. Challagulla, F. B. Bastani, I.-L.
Yen, and R. A. Paul, “Empirical assessment of machine learning based software defect prediction techniques,” in
Object-Oriented Real-Time Dependable Systems, 2005. WORDS 2005. 10th IEEE International Workshop on. IEEE, 2005, pp. 263–
270.
K. Kaminsky and G. Boetticher, “Building a genetically engineerable evolvable
program (geep) using breadth-based explicit knowledge for predicting software
defects,” in Fuzzy Information, 2004. Processing NAFIPS’04. IEEE Annual Meeting of the,
vol. 1. IEEE, 2004, pp. 10–15.
——, “How to predict more with less, defect prediction using machine learners in
an implicitly data starved domain,” in The
8th world multiconference on systemics, cybernetics and informatics, Orlando, FL. Citeseer,
2004.
K. Kaminsky and G. D. Boetticher, “Better
software defect prediction using equalized
learning with machine learners,” Knowledge
Sharing and Collaborative Engineering, 2004.
O. Kutlubay and A. Bener, “A machine
learning based model for software defect
prediction,” working paer, Boaziçi University,
Computer Engineering Department, 2005.
X. Ren, “Learn to predict “affecting
changes” in software engineering,” 2003.
E. Ceylan, F. O. Kutlubay, and A. B. Bener,
“Software defect identification using machine learning techniques,” in Software Engineering and Advanced Applications, 2006.
SEAA’06. 32nd EUROMICRO Conference on.
IEEE, 2006, pp. 240–247.
Y. Kastro and A. B. Bener, “A defect prediction method for software versioning,”
Software Quality Journal, vol. 16, no. 4, pp.
543–562, 2008.
O. Kutlubay, B. Turhan, and A. B. Bener, “A
two-step model for defect density estimation,” in Software Engineering and Advanced
Applications, 2007. 33rd EUROMICRO Conference on. IEEE, 2007, pp. 322–332.
A. S. Namin and M. Sridharan, “Bayesian

19

[60]

[61]

[62]

[63]

[64]

[65]

[66]

[67]

[68]

[69]

reasoning for software testing,” in Proceedings of the FSE/SDP workshop on Future of
software engineering research. ACM, 2010,
pp. 349–354.
C. Murphy and G. Kaiser, “Metamorphic runtime checking of non-testable programs,” Columbia University Dept of Computer Science Tech Report cucs-042-09, p.
9293, 2009.
W. Afzal, R. Torkar, R. Feldt, and
T. Gorschek, “Genetic programming for
cross-release fault count predictions in
large and complex software projects,” Evolutionary Computation and Optimization Algorithms in Software Engineering, pp. 94–126,
2010.
C. Murphy et al., “Using metamorphic testing at runtime to detect defects in applications without test oracles,” 2008.
D. Qiu, S. Fang, and Y. Li, “A framework to
discover potential deviation between program and requirement through mining object graph,” in Computer Application and System Modeling (ICCASM), 2010 International
Conference on, vol. 4. IEEE, 2010, pp. V4–
110.
C. Murphy, G. E. Kaiser et al., “Automatic
detection of defects in applications without
test oracles,” Dept. Comput. Sci., Columbia
Univ., New York, NY, USA, Tech. Rep. CUCS027-10, 2010.
W. Afzal, “Search-based approaches to software fault prediction and software testing,” Ph.D. dissertation, Blekinge Institute
of Technology, 2009.
M. K. Taghi, B. Cukic, and N. Seliya, “An
empirical assessment on program moduleorder models,” Quality Technology & Quantitative Management, vol. 4, no. 2, pp. 171–
190, 2007.
J. H. Wang, N. Bouguila, and T. Bdiri, “Empirical evaluation of selected algorithms for
complexity-based classification of software
modules and a new model,” in Intelligent
Systems: From Theory to Practice. Springer,
2010, pp. 99–131.
H. Jin, Y. Wang, N.-W. Chen, Z.-J. Gou,
and S. Wang, “Artificial neural network for
automatic test oracles generation,” in Computer Science and Software Engineering, 2008
International Conference on, vol. 2. IEEE,
2008, pp. 727–730.
J. Ferzund, S. N. Ahsan, and F. Wotawa,
“Automated classification of faults in pro-

[70]

[71]

[72]

[73]

[74]

[75]

[76]

[77]

[78]

[79]

[80]

[81]

gramms using machine learning techniques,” in Artificial Intelligence Techniques
in Software Engineering Workshop, 2008.
O. Maqbool and H. Babri, “Bayesian learning for software architecture recovery,” in
Electrical Engineering, 2007. ICEE’07. International Conference on. IEEE, 2007, pp. 1–6.
A. Okutan, “Software defect prediction using bayesian networks and kernel methods,” Ph.D. dissertation, ISIK UNIVERSITY, 2012.
D. Cotroneo, R. Pietrantuono, and S. Russo,
“A learning-based method for combining
testing techniques,” in Proceedings of the
2013 International Conference on Software Engineering. IEEE Press, 2013, pp. 142–151.
D. Zhang, “A value-based framework for
software evolutionary testing,” in Advances
in Abstract Intelligence and Soft Computing.
IGI Global, 2013, pp. 355–373.
A. Okutan and O. T. Yıldız, “Software defect prediction using bayesian networks,”
Empirical Software Engineering, vol. 19, no. 1,
pp. 154–181, 2014.
S. Agarwal and D. Tomar, “A feature selection based model for software defect
prediction,” assessment, vol. 65, 2014.
G. Abaei and A. Selamat, “Important issues in software fault prediction: A road
map,” in Handbook of Research on Emerging
Advancements and Technologies in Software
Engineering. IGI Global, 2014, pp. 510–539.
A. Okutan and O. T. Yildiz, “A novel kernel
to predict software defectiveness,” Journal
of Systems and Software, vol. 119, pp. 109–
121, 2016.
X.-d. Mu, R.-h. Chang, and L. Zhang,
“Software defect prediction based on competitive organization coevolutionary algorithm,” Journal of Convergence Information
Technology (JCIT) Volume7, Number5, 2012.
J. Cahill, J. M. Hogan, and R. Thomas,
“Predicting fault-prone software modules
with rank sum classification,” in Software
Engineering Conference (ASWEC), 2013 22nd
Australian. IEEE, 2013, pp. 211–219.
R. Rana, M. Staron, C. Berger, J. Hansson, M. Nilsson, and W. Meding, “The
adoption of machine learning techniques
for software defect prediction: An initial
industrial validation,” in Joint Conference
on Knowledge-Based Software Engineering.
Springer, 2014, pp. 270–285.
T. Schulz, Ł. Radliński, T. Gorges, and

20

[82]

[83]

[84]

[85]

[86]

[87]

[88]

[89]

[90]

[91]

W. Rosenstiel, “Predicting the flow of defect correction effort using a bayesian network model,” Empirical Software Engineering, vol. 18, no. 3, pp. 435–477, 2013.
E. Rashid, “R4 model for case-based reasoning and its application for software
fault prediction,” International Journal of
Software Science and Computational Intelligence (IJSSCI), vol. 8, no. 3, pp. 19–38, 2016.
——, “Improvisation of case-based reasoning and its application for software fault
prediction,” International Journal of Services
Technology and Management, vol. 21, no. 4-6,
pp. 214–227, 2015.
J. K. Chhabra and A. Parashar, “Prediction of changeability for object oriented
classes and packages by mining change
history,” in Electrical and Computer Engineering (CCECE), 2014 IEEE 27th Canadian
Conference on. IEEE, 2014, pp. 1–6.
G. Spanoudakis, A. S. d. Garcez, and
A. Zisman, “Revising rules to capture requirements traceability relations: A machine learning approach.” in SEKE, 2003,
pp. 570–577.
M. Shin and A. Goel, “Modeling software component criticality using a machine learning approach,” Artificial Intelligence and Simulation, pp. 440–448, 2005.
J. S. Shirabad, “Predictive techniques in
software engineering,” in Encyclopedia of
Machine Learning. Springer, 2011, pp. 782–
789.
A. A. Araújo, M. Paixao, I. Yeltsin, A. Dantas, and J. Souza, “An architecture based
on interactive optimization and machine
learning applied to the next release problem,” Automated Software Engineering, pp.
1–49, 2016.
T. Tourwé, J. Brichau, A. Kellens, and
K. Gybels, “Induced intentional software
views,” Computer Languages, Systems &
Structures, vol. 30, no. 1, pp. 35–47, 2004.
J. S. Di Stefano and T. Menzies, “Machine
learning for software engineering: Case
studies in software reuse,” in Tools with
Artificial Intelligence, 2002.(ICTAI 2002). Proceedings. 14th IEEE International Conference
on. IEEE, 2002, pp. 246–251.
J. Fu, F. B. Bastani, and I.-L. Yen, “Automated ai planning and code pattern based
code synthesis,” in Tools with Artificial Intelligence, 2006. ICTAI’06. 18th IEEE International Conference on. IEEE, 2006, pp. 540–

546.
J. Fu, F. B. Bastani, I.-L. YEN et al.,
“Semantic-driven component-based automated code synthesis,” Semantic Computing, pp. 249–283, 2010.
[93] A. Katasonov, O. Kaykova, O. Khriyenko,
S. Nikitin, and V. Y. Terziyan, “Smart
semantic middleware for the internet of
things.” ICINCO-ICSO, vol. 8, pp. 169–178,
2008.
[94] L. Baresi, S. Guinea, and A. Shahzada,
“Short paper: Harmonizing heterogeneous
components in sesame,” in Internet of
Things (WF-IoT), 2014 IEEE World Forum on.
IEEE, 2014, pp. 197–198.
[95] L. Zhu, H. Cai, and L. Jiang, “Minson: A
business process self-adaptive framework
for smart office based on multi-agent,” in
e-Business Engineering (ICEBE), 2014 IEEE
11th International Conference on. IEEE, 2014,
pp. 31–37.
[96] J. F. De Paz, J. Bajo, S. Rodrı́guez, G. Villarrubia, and J. M. Corchado, “Intelligent
system for lighting control in smart cities,”
Information Sciences, vol. 372, pp. 241–255,
2016.
[97] I. Birzniece, “The use of inductive learning
in information systems,” in Proceedings of
the 16th International Conference on Information and Software Technologies, 2010, pp. 95–
101.
[98] D. Alrajeh, A. Russo, and S. Uchitel, “Inferring operational requirements from scenarios and goal models using inductive learning,” in Proceedings of the 2006 international
workshop on Scenarios and state machines:
models, algorithms, and tools. ACM, 2006,
pp. 29–36.
[99] A. M. Sharifloo, A. Metzger, C. Quinton,
L. Baresi, and K. Pohl, “Learning and evolution in dynamic software product lines,”
in Proceedings of the 11th International Symposium on Software Engineering for Adaptive
and Self-Managing Systems. ACM, 2016, pp.
158–164.
[100] B. Zoph and Q. V. Le, “Neural architecture
search with reinforcement learning,” arXiv
preprint arXiv:1611.01578, 2016.
[101] N. M. do Nascimento and C. J. P. de Lucena, “Fiot: An agent-based framework for
self-adaptive and self-organizing applications based on the internet of things,” Information Sciences, vol. 378, pp. 161–176, 2017.
[102] F. Jacob and R. Tairas, “Code template in[92]

21

[103]

[104]

[105]

[106]

[107]

[108]

[109]

[110]

[111]

[112]

[113]

ference using language models,” in Proceedings of the 48th Annual Southeast Regional
Conference. ACM, 2010, p. 104.
B. Amal, M. Kessentini, S. Bechikh, J. Dea,
and L. B. Said, “On the use of machine
learning and search-based software engineering for ill-defined fitness function: a
case study on software refactoring,” in International Symposium on Search Based Software Engineering. Springer, 2014, pp. 31–
45.
Y. Peng, G. Wang, and H. Wang, “User
preferences based software defect detection
algorithms selection using mcdm,” Information Sciences, vol. 191, pp. 3–13, 2012.
H. A. Abbass, E. Petraki, K. Merrick, J. Harvey, and M. Barlow, “Trusted autonomy
and cognitive cyber symbiosis: Open challenges,” Cognitive computation, vol. 8, no. 3,
pp. 385–408, 2016.
W. G. Baxt, “Use of an artificial neural
network for the diagnosis of myocardial
infarction,” Annals of internal medicine, vol.
115, no. 11, pp. 843–848, 1991.
M. A. Mazurowski, P. A. Habas, J. M.
Zurada, J. Y. Lo, J. A. Baker, and G. D.
Tourassi, “Training neural network classifiers for medical decision making: The
effects of imbalanced datasets on classification performance,” Neural networks, vol. 21,
no. 2, pp. 427–436, 2008.
N. M. do Nascimento, M. L. Viana, and
C. J. P. de Lucena, “An iot-based tool for
human gas monitoring,” in IXV Congresso
Brasileiro de Informatica em Saude (CBIS),
vol. 1. SBIS, 2016, pp. 96–98.
R. Morejón, M. Viana, and C. Lucena,
“Generating software agents for data mining: An example for the health data area,”
in International Conference on Software Engineering & Knowledge Engineering-SEKE,
2017.
S. Haykin, Neural Networks: A Comprehensive Foundation. Macmillan, 1994.
[Online]. Available: http://books.google.
com.br/books?id=PSAPAQAAMAAJ
D. Castelvecchi, “Can we open the black
box of ai?” Nature News, vol. 538, no. 7623,
p. 20, 2016.
L. Atzori, A. Iera, and G. Morabito, “The
internet of things: A survey,” Computer networks, vol. 54, no. 15, pp. 2787–2805, 2010.
M. A. Salahuddin, A. Al-Fuqaha,
M. Guizani, K. Shuaib, and F. Sallabi,

[114]

[115]

[116]

[117]

[118]

[119]

[120]

[121]

[122]

[123]

[124]

[125]

“Softwarization of internet of things
infrastructure for secure and smart
healthcare,” Computer, vol. 50, no. 7, pp.
74–79, 2017.
I. Ayala, M. Amor, L. Fuentes, and J. M.
Troya, “A software product line process to
develop agents for the iot,” Sensors, vol. 15,
no. 7, pp. 15 640–15 660, 2015.
J.-P. Briot, N. M. de Nascimento, and C. J. P.
de Lucena, “A multi-agent architecture for
quantified fruits: Design and experience,”
in 28th International Conference on Software Engineering & Knowledge Engineering
(SEKE’2016). SEKE/Knowledge Systems
Institute, PA, USA, 2016, pp. 369–374.
N. M. Nascimento, “FIoT: An agentbased framework for self-adaptive and
self-organizing internet of things applications,” Master’s thesis, PUC-Rio, Rio de
Janeiro, Brazil, August 2015.
N. M. d. Nascimento, C. J. P. d. Lucena,
and H. Fuks, “Modeling quantified things
using a multi-agent system,” in IEEE / WIC
/ ACM International Conference on Web Intelligence and Intelligent Agent Technology (WIIAT), vol. 1. IEEE, 2015, pp. 26–32.
N. M. NASCIMENTO and C. J. P. LUCENA, “Engineering cooperative smart
things based on embodied cognition,” in
NASA/ESA Conference on Adaptive Hardware
and Systems (AHS 2017). IEEE, 2017.
Apple,
“Homekit,”
https://developer.apple.com/homekit/,
March 2017.
Samsung, “Samsung smart things,”
https://www.smartthings.com,
March
2017.
D. I. Sjøberg, T. Dybå, B. C. Anda, and
J. E. Hannay, “Building theories in software
engineering,” Guide to advanced empirical
software engineering, pp. 312–336, 2008.
S. Whiteson, N. Kohl, R. Miikkulainen, and
P. Stone, “Evolving soccer keepaway players through task decomposition,” Machine
Learning, vol. 59, no. 1-2, pp. 5–30, 2005.
C. Wohlin, P. Runeson, M. Höst, M. C.
Ohlsson, B. Regnell, and A. Wesslén, Experimentation in software engineering. Springer
Science & Business Media, 2012.
R. S. Sutton and A. G. Barto, Reinforcement
learning: An introduction. MIT press Cambridge, 1998, vol. 1, no. 1.
R. Peck and J. Devore, Statistics: The Exploration & Analysis of Data. Nelson Educa-

22

tion, 2011.
[126] W. Oizumi, L. Sousa, A. Garcia, R. Oliveira,
A. Oliveira, O. Agbachi, and C. Lucena,
“Revealing design problems in stinky code:
a mixed-method study,” in Proceedings of
the 11th Brazilian Symposium on Software
Components, Architectures, and Reuse. ACM,
2017, p. 5.
[127] E. Fernandes, F. Ferreira, J. A. Netto, and
E. Figueiredo, “Information systems development with pair programming: An academic quasi-experiment,” in Proceedings of
the XII Brazilian Symposium on Information
Systems on Brazilian Symposium on Information Systems: Information Systems in the
Cloud Computing Era-Volume 1. Brazilian
Computer Society, 2016, p. 64.
[128] G. Kasparov, Deep Thinking: Where Machine
Intelligence Ends and Human Creativity Begins. Hachette UK, 2017.
[129] D. Silver, A. Huang, C. J. Maddison,
A. Guez, L. Sifre, G. Van Den Driessche,
J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., “Mastering the
game of go with deep neural networks and
tree search,” Nature, vol. 529, no. 7587, pp.
484–489, 2016.
[130] D. Silver, J. Schrittwieser, K. Simonyan,
I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton et al.,
“Mastering the game of go without human
knowledge,” Nature, vol. 550, no. 7676, pp.
354–359, 2017.
[131] A. Trotman and C. Handley, “Programming contest strategy,” Computers & Education, vol. 50, no. 3, pp. 821–837, 2008.

