Towards a graphical language for quadrotor missions

arXiv:1412.1961v1 [cs.RO] 5 Dec 2014

Benjamin Schwartz1 , Ludwig Nägele1 , Andreas Angerer1 , and Bruce A. MacDonald2

Abstract— This paper presents an approach for defining
Unmanned Aerial Vehicle (UAV) missions on a high level.
Current methods for UAV mission specification are evaluated
and their deficiencies are analyzed. From these findings, a new
graphical specification language for UAV missions is proposed,
which is targeted towards typical UAV users from various
domains rather than computer science experts. The research
is ongoing, but a first prototype is presented.

I. I NTRODUCTION
Quadrotors or in general Unmanned Aerial Vehicles
(UAV) are becoming increasingly popular and affordable for
a broad range of activities. Researchers and developers are
exploring the use of UAVs in application domains such as
agriculture, archaeology or infrastructure monitoring. Patel et
al. investigate how to detect possible crop diseases on fields
using a quadrotor equipped with an infrared camera [1]. Employing quadrotors for overhead pictures of an exhibition site
to reduce the cost for archaeological research is discussed by
Heiermann et al. [2]. The German companies Thyssengas and
microdrones are working on a system to inspect gas pipelines
with quadrotors [3]. The plans of the company Amazon to
use UAVs to deliver packages [4] to end customers went
through the international press some months ago.
To execute such tasks reliably, UAVs must be increasingly
equipped with more autonomy. The routes the UAV should
take might be predefined or may depend on parameters of the
task, additional actions might be necessary at certain points
of the route (such as taking pictures, activating sensors)
and unforeseen situations must be handled (e.g. avoiding
obstacles). Above all this, the operational state, including
the battery lifetime, and environment conditions, such as
wind and weather conditions, should be checked constantly
to safely abort the mission if necessary.
The predominant way of defining and creating such a UAV
mission is hard-coding it in a general-purpose programming
language. However, the typical users of UAV systems – e.g.,
farmers, archeologists or pipeline technicians – do not have
the skill to design software of this complexity by hand. This
paper presents a novel approach to describe UAV missions
with a graphical Domain Specific Language based on an
analysis of use cases and grouping of concepts, and with
an extendable model. Sect. II presents existing approaches
and motivates the decision to design a new domain specific
language (DSL). Concepts and challenges are introduced in
1 Institute of Software & Systems Engineering, University of Augsburg,
Germany
2 CARES, Department of Electrical and Computer Engineering, University of Auckland, New Zealand

Sect. III. The work is ongoing, however, the current prototype
and first results are presented in Sect. IV.
II. A NALYSIS
The first part of this section examines existing tools for defining UAV missions in a user-friendly way and motivates the
choice to create a new language. In the second part, different
approaches for realizing such a language are discussed.
A. Related work
MissionLab developed in ’99 by Arkin et al. [5] allows
specifying missions for autonomous robots in a graphical
way. The interface, however, is quite restricted. In ’04 a
mission planning wizard was added to improve usability.
A usability evaluation [6] showed it improved and sped
the mission planning process. However, MissonLab lacks
flexibility for more complex, branched workflows.
MAVLink based robots use the APM Planner [7] mission
planning tool, which has a Graphical User Interface (GUI) to
configure both waypoints and tasks to be carried out at each
waypoint. This tool does not support conditional waypoints
or specifying simultaneous tasks such as avoiding obstacles.
The APM Planner supports communication with a ground
station during the flight, which enables manual reactions
by the operator of the station to specific changes. This
communication is mandatory and must be maintained during
the whole flight, which limits missions to a specific area. The
GUI is highly developed and can also show the waypoints
of a mission in a virtual map. However, the planned mission
cannot be changed after generation and cannot be adapted
to the user needs. DJI [8] provides a similar proprietary tool
for their quadrotors. An additional feature of this tool is the
ability to define non-fly zones.
Graphical State Space Programming (GSSP) [9] is another
framework for developing quadrotor missions. GSSP has
a graphical and a textual interface. The graphical part is
used to declare waypoints and the workflow of the mission.
Special tasks, which are carried out during the execution,
are specified in the textual interface in the programming
language python. Thus, it is possible to perform operations
during the flight simultaneously or introduce branches in the
workflow. However, even the simplest of such special tasks
requires writing code.
The analysis showed some major deficiencies. Many existing approaches (e.g. APM Planner and DJI PC Ground
Station) have a proprietary character and can only be used
with certain types of UAVs. Moreover, though providing
graphical interfaces, all approaches focus on the main, sequential workflow of a UAV mission and are very limited

when it comes to branches and alternative flows. In some
approaches, additional flexibility is achievable by adding new
functionality using some host programming language. This is
however not optimal for end users that lack computer science
background. Finally, similar deficiencies exist with respect to
simultaneous tasks that have to be executed during the whole
mission or parts of it. Existing approaches provide little to
no graphical support for this.
To overcome the identified problems and provide a flexible
and at the same time easy-to-use description mechanism,
the authors decided to design a new graphical language
that should incorporate advantages of existing work, but add
much more flexibility for describing UAV missions while
still being intuitive to use also for non-experts.
B. Graphical programming languages
When creating a graphical language, a straightforward
approach is to use an existing general-purpose visual programming tool. A prominent example is Scratch [10], which
is used to help people learn to program textual generalpurpose programming languages. It provides a drag and drop
mechanism to put together an application. Since its purpose
is to help people learn a general-purpose programming
language it focuses on traditional programming paradigms.
A related project is MIT’s App Inventor [11], which uses
Scratch for rapidly prototyping Android applications without
programming knowledge. Another powerful general-purpose
visual programming tool is Microsoft Visual Programming
Language (VPL) [12], which employs a dataflow-based programming model. Developing a ’new’ language based on
Scratch or VPL would be possible with moderate effort.
However, some advanced domain specific concepts would
likely require modifications to be made in the respective
editor, which is not possible since VPL is a closed source.
Otherwise, all domain-specific concepts would need to be
encoded based on existing syntactic elements.
A second approach is to employ a toolkit for constructing
a new DSL. The advantages of a custom DSL are the higher
abstraction level, which enables domain experts to create
programs in terms of familiar domain concepts. A DSL can
furthermore provide domain specific validations. A drawback
is the effort to create a DSL, which only pays off if the
DSL can be used frequently. Often different DSLs exist in
one domain and there is seldom a standard DSL, which
can lead to confusion for the user. An executable DSL can
also produce code, which can be used as a starting point
for a quadrotor mission. Existing frameworks to support
the development of a DSL include the Eclipse Modeling
Framework [13], [14].
Even more freedom is achievable with a new custom
designed editor implemented in a general-purpose language.
APM Planner and PC Ground Station of DJI are examples.
The advantage is the broad range of possible designs. A
drawback of this freedom is that the resulting tool will
be more complicated for other developers to extend and
enhance, as well as development time and cost will be higher
than with both other methods.

The DSL toolkit approach is more flexible and allows good
tailoring of the developed language to the specific needs of
the domain.
III. P ROPOSED GRAPHICAL LANGUAGE
The conceptual model of the developed language is based
on a literature research, identifying numerous use cases
whose purposes vary significantly. A common set of concepts
was identified and used as basis for the language. The first
part of this section will introduce those concepts, whereas the
second part will present the graphical syntax of the proposed
language.
A. Basic concepts
Routing elements are the central concept to describe
quadrotor missions. They are used to move the quadrotor
from some location to another one. Examples for routing
elements are taking off, touching down or flying to a certain
position or area.
While routing elements provide means for building the
structure of a mission, actions can be used to give a mission a
purpose. In the analyzed use cases, a broad range of activities
was carried out at various points during a mission. Examples
include taking a picture, using a laser scanner, or scanning
for a signal such as wireless LAN. The idea of actions
is similar as in GSSP. However, in contrast to GSSP the
implementation is always hidden from the user.
To introduce flexibility in the mission flow, conditional
branches are available. They allow for specifying alternative
mission flows depending on runtime conditions. This concept
is adapted from general programming language theory and
allows expressing a broad range of quadrotor missions.
For conditional branching, input data is needed as well
as some kind of processing of this data. This led to the
introduction of processing actions, e.g. to recognize an
image or interpret a laser scan.
For enhancing routing elements with individual movement
strategies, the concept of filters is introduced. This is based
on the observation that quadrotor movements frequently need
to respect certain constraints, e.g. maintaining a specified
velocity or avoiding obstacles on-line. Thus, filters may
influence routing elements.
Finally, to support activities that are meant to run simultaneously to the main flow, a parallel block is provided.
This can be employed e.g. to monitor sensor outputs, record
a video or take a picture every minute. In conjunction with
conditional branches, parallel blocks are able to influence the
program flow, e.g. if a sensor value exceeds some threshold.
B. Graphical syntax
To gain a consistent visual language all concepts mentioned above and the way they interact need to obtain
meaningful graphical representations. In order to keep the
language clear and intuitive, different colors are used to
easily distinguish between different element types.
The routing elements Take off and Touch down are the
starting and ending points of every mission and thus form an

Fig. 1.

Example quadrotor mission program which demonstrates the invented language constructs

invariant surrounding of each program. Within this surrounding, a chronological order of routing elements (e.g. Fly home,
Fly in area, ...) is possible (cf. Fig. 1). As discussed above,
routing elements should be augmentable with actions that
are to be performed at various points of the mission. Instead
of connecting actions to (possibly many) routing elements
by using (many) links, the design of our language embeds
instances of actions directly in routing elements (Fig. 1, A).
Conditional branches are symbolised by diamonds (Fig. 1,
B) having two outgoing transitions, a green one to be used in
case of true and a red one for false. For validation purpose,
conditional branches reference the result of one previously
executed action (of the previous block or element), processes
it in some way (using processing actions) and compares the
output to the given reference value. In the given example
(Fig. 1, C), the infrared picture taken in the previous parallel
block is processed by an image recognition action, which
may produce the result “disease found”. Also cycles and
iterations can be modeled with conditional branches (Fig. 1,
D). Filters have to be defined by specifying a list of atomic
filter actions. The order of those actions specifies their
priority. In the example (Fig. 1, E) the filter monitors a
certain speed value but always guarantees collision freedom.
Once defined, a filter can be attached to routing elements
(Fig. 1, F).
Parallel blocks (Fig. 1, G) can be attached via links to
one or more routing elements to which the embedded actions
should run concurrently. For periodically triggered actions,
a period time can be specified. As parallel blocks should be
able to alter the mission workflow in some cases, conditional
branches can be attached to them. However, in order to
preserve a clean main mission flow, this combination implies
a particular semantics: If a parallel block is followed by a
conditional branch, the false-case implicitly returns control
to the parallel block. Thus, the execution context leaves the
parallel block only in the true case. To illustrate this special
semantics, a dashed line to the conditional branch is used
instead of an arrow (Fig. 1, H). In case of evaluation to true,
the green arrow indicates the next active routing element of
the main flow.

IV. P ROTOTYPE
A prototype (see Fig. 2) has been implemented based on
the Eclipse Modeling Project (EMP) [13], which is a set
of different model-based development tools and frameworks.
A meta-model of the designed quadcopter mission language
has been defined and a graphical editor could be generated
by using EMP’s Graphical Modelling Framework (GMF).
The extension by new actions was identified as an important
feature of the language, because not all possibly desired
functionality of quadrotor missions can be predicted beforehand and considered in the meta-model. Since it would be
highly impractical to adapt the meta-model and recompile
the editor for each new action, another way of enhancing
the editor by additional actions has been developed. In the
following first section the method how to add new concrete
action implementations is presented. A brief look onto the
execution of developed quadrotor programs is given in the
subsequent section.
A. Extensions mechanism for user-specific actions
For extending the program with own actions, a super class
GenericAction is provided. It is part of the meta-model and
needs to be inherited programmatically by all action implementations by declaring special information such as a name
for describing the action (e.g. “TakePicture”), the return type
of the action and its logical behaviour. An action implementation can then graphically be used - maybe more than
once - within the graphical program in the editor. For each
usage, a new instance of the action type is created. In case
the action is capable of being parameterised, the super class
ensures that each instance can be given individual specific
configurations. Technically, this is a map of variable name
and value pairs. For the “TakePicture”-example, additional
parameters might be resolution and compression quality. The
properties view (see Fig. 2, bottom middle) shows these
variables and allows for their modification. The possibility of
parameterising makes actions reusable. The extension by new
processing actions works analogously. A specific action can
be implemented, graphically referenced and parameterised
within the same single development environment.

Fig. 2.

Example of the invented language in the developed editor

Regarding different result types of action definitions, the
prototype ensures type consistency when actions are to
be referenced in processing actions or comparisons within
conditional branches.
B. Execution
Different techniques to execute the model were considered. In general interpreting code is less efficient than
executing previously generated code. Furthermore, an interpreter typically needs more hardware resources which might
not be available on limited quadrotor hardware. Thus, code
generation was chosen for the prototype. In order to have
a first evaluation of the invented language using a realworld application and to be able to rapidly prototype and test
graphical programs, Morse [15] and its quadrotor extension is
used as standalone application. Python code, which controls
the Morse simulator, is generated out of the editor by using
Xtend [16] template expressions.
V. C ONCLUSION & F UTURE WORK
The paper showed there is need for a higher-level and
platform-independent approach for specifying quadrotor missions. A graphical DSL was identified to have the highest
potential to solve the given requirements and issues, and a
set of concepts was discussed to represent the structure of the
new language. The implemented prototype showed that the
identified theoretical domain concepts can be automatically
translated into working applications.
A future step is to conduct a user-study to see how
the chosen graphical representation and the editor are used
and to identify possible improvements in order to simplify
usage. Such an improvement would be a virtual map for
showing waypoints. Another further development could be
the extension by new translation templates for supporting
code translation to real quadrotor platforms. The model could
also be enriched by more environment information such as
where the quadrotor is flying. This could result in a different

collection of rules, for example touch-down is not allowed
over water.
R EFERENCES
[1] P. N. Patel, M. A. Patel, R. M. Faldu, and Y. R. Dave, “Quadcopter
for Agricultural Surveillance,” Advance in Electronic and Electric
Engineering, vol. 3, no. 4, pp. 427–432, 2013.
[2] Medienservice Sachsen: Landesamt für Archäologie, “Drohne im
Einsatz für die Archäologie.” [Online]. Available: http://www.
archaeologie.sachsen.de/4178.htm [Accessed: 15/05/14]
[3] Microdrones GmbH, “Thyssengas: Aerial Survey Monitoring.”
[Online]. Available: http://microdrones.com/references/case-study/
microdrones-thyssengas-aerial-survey-gas4energy-en.pdf [Accessed:
15/05/14]
[4] Amazon.com, Inc., “Amazon Prime Air.” [Online]. Available:
http://www.amazon.com/b?node=8037720011 [Accessed: 25/08/14]
[5] R. R. C. Arkin, T. R. T. Collins, and Y. Endo, “Tactical mobile robot
mission specification and execution,” Mobile Robots XIV, vol. 3838,
no. 150, pp. 150–163, 1999.
[6] Y. Endo, “Usability evaluation of high-level user assistance for robot
mission specification,” IEEE Transactions on Systems, Man, and
Cybernetics, Part C, vol. 34, no. 2, 2004.
[7] Ardupilot Mega Project, “ArduCopter | Multirotor UAV.” [Online].
Available: http://copter.ardupilot.com/ [Accessed: 23/02/14]
[8] DJI, “PC Ground Station.” [Online]. Available: http://www.dji.com/
product/pc-ground-station [Accessed: 14/05/14]
[9] J. Li, A. Xu, and G. Dudek, “Graphical State Space Programming:
A visual programming paradigm for robot task specification,” in
Proceedings of 2011 IEEE International Conference on Robotics and
Automation. IEEE, May 2011, pp. 4846–4853.
[10] M. Resnick, B. Silverman, Y. Kafai, J. Maloney, A. MonroyHernández, N. Rusk, E. Eastmond, K. Brennan, A. Millner, E. Rosenbaum, and J. Silver, “Scratch,” Communications of the ACM, vol. 52,
no. 11, p. 60, Nov. 2009.
[11] D. Wolber, “App inventor and real-world motivation,” in Proc. 42nd
ACM Techn. Symposium on Computer Science Education. New York,
New York, USA: ACM Press, Mar. 2011, p. 601.
[12] Microsoft, “Visual Programming Language.” [Online]. Available:
http://msdn.microsoft.com/en-us/library/bb964572.aspx
[Accessed:
05/06/14]
[13] R. C. Gronback, Eclipse Modeling Project: A Domain-Specific Language (DSL) Toolkit. Addison-Wesley Professional, Mar. 2009.
[14] The Eclipse Foundation, “Eclipse Modeling Project.” [Online].
Available: http://www.eclipse.org/modeling/ [Accessed: 15/02/14]
[15] G. Echeverria, N. Lassabe, A. Degroote, and S. Lemaignan, “Modular
open robots simulation engine: MORSE,” in 2011 IEEE Intl. Conf. on
Robotics and Automation. IEEE, May 2011, pp. 46–51.
[16] L. Bettini, Implementing Domain-Specific Languages with Xtext and
Xtend, ser. EBL-Schweitzer. Packt Publishing, Limited, 2013.

