Sim-to-real Transfer of Visuo-motor Policies for Reaching in Clutter:
Domain Randomization and Adaptation with Modular Networks*

Abstract— A modular method is proposed to learn and
transfer visuo-motor policies from simulation to the real world
in an efficient manner by combining domain randomization
and adaptation. The feasibility of the approach is demonstrated
in a table-top object reaching task where a 7 DoF arm is
controlled in velocity mode to reach a blue cuboid in clutter
through visual observations. The learned visuo-motor policies
are robust to novel (not seen in training) objects in clutter and
even a moving target, achieving a 93.3% success rate and 2.2 cm
control accuracy.

In-hand Camera

I. I NTRODUCTION

*This research was conducted by the Australian Research Council Centre
of Excellence for Robotic Vision (project number CE140100016). Computational resources & services used in this work were partially provided by the
HPC and Research Support Group, Queensland University of Technology.
1 The authors are with the Australian Centre for Robotic Vision
(ACRV), Queensland University of Technology (QUT), Brisbane, Australia

fangyi.zhang@hdr.qut.edu.au

B
Joint Angles
Perception Module (CNN)

C

Control Modue (FC)

Bottleneck (object position, 3 DoF)

Joint Velocities

A

x1,x2,x3

The advent of large datasets and sophisticated machine
learning models, commonly referred to as deep learning,
has in recent years created a trend away from hand-crafted
solutions towards more data-driven ones. Learning techniques have shown significant improvements in robustness
and performance [1], particularly in the computer vision
field. Traditionally robotic reaching approaches have been
based on crafted controllers that combine (heuristic) motion
planners with the use of hand-crafted features to localize the
target visually. Recently learning approaches to tackle this
problem have been presented [2]–[5], however a consistent
issue faced by most approaches is the reliance on large
amounts of data to train these models. For example, Google
researchers addressed this problem by developing an "arm
farm" with 6 to 14 robots collecting data in parallel [3].
Generalization forms another challenge: many current systems are brittle when learned models are applied to robotic
configurations that differ from those used in training. This
leads to the question: Is there a better way to learn and
transfer visuo-motor policies on robots for tasks such as
reaching?
Various approaches have been proposed to address these
problems in a robot learning context: (i) the use of simulators
and synthetic data [4], [6]–[8]; (ii) methods that transfer the
learned models to real-world scenarios [9], [10]; (iii) directly
learning real-world tasks by collecting large amounts of data
[3], [11].
The contribution of this paper is a method that connects
these three, usually separately considered, approaches. We
propose a modular deep learning approach to efficiently
learn and transfer visuo-motor policies from simulated to real

RGB Image

arXiv:1709.05746v1 [cs.RO] 18 Sep 2017

Fangyi Zhang, Jürgen Leitner, Michael Milford, Peter Corke1

Fig. 1. A robot (Baxter) learns visuo-motor policies to control its left arm
(7 DoF) to reach a target blue cuboid in clutter on a table. Baxter visually
observes the table-top environments through the monocular camera in its
right hand. A modular network (Fig. 1C) is used to efficiently learn and
transfer visuo-motor policies from simulation (Fig. 1A) to the real world
(Fig. 1B). The modular network consists of perception and control modules,
connected by a bottleneck layer (see Section III).

environments, and benchmark with a visually-guided tabletop object reaching task for a 7 DoF robotic arm (Fig. 1).
Vision and kinematics data is gathered in simulation (cheap)
to decrease the amount of real world collection necessary
(costly). Significantly, by introducing a modular approach
the perception skill and the controller can be transferred
individually to a robotic platform, while retaining the ability
to fine-tune them in an end-to-end fashion to further improve
hand-eye coordination on a real robot (in this research a
Baxter).
The use of both simulated (with domain randomization)
and real (domain adaptation) data makes this approach able
to transfer perception to the real world (3.2 cm accuracy)
with only 3000 simulated and 186 real images. Benefiting
from the modular structure and weighted end-to-end finetuning, the learned visuo-motor policy can achieve a reaching
accuracy of 2.2 cm with only 333 trajectories (30225 statevelocity pairs). The learned visuo-motor policy is not only
able to reach a target object in clutter with seen distractor
objects but also for the cases with novel (not seen in training)
distractor objects and even when the target object is moving.
II. R ELATED W ORK
Data-driven learning approaches have become popular in
computer vision and are starting to replace hand-crafted so-

lutions also in robotic applications. Especially robotic vision
tasks – robotic tasks based directly on real image data – such
as navigation [12], object grasping and manipulation [3],
[11], [13] have seen increased interest. The lack of largescale real-world datasets, which are expensive and slow to acquire, limit the general applicability of the approach and has
so far limited the broader application. Collecting the datasets
required for deep learning has been sped up by using many
robots operating in parallel [3]. With over 800,000 grasp
attempts recorded, a deep network was trained to predict
the success probability of a sequence of motions aiming at
grasping using a 7 DoF robotic manipulator with a 2-finger
gripper. Combined with a simple derivative-free optimization
algorithm the grasping system achieved a success rate of
80%. Another example of dataset collection for grasping is
the approach to self-supervised grasp learning in the real
world where force sensors were used to autonomously label
samples [11]. After training with 50,000 real-world trials
using a staged leaning method, a deep convolutional neural
network (CNN) achieved a grasping success rate around
70%. These are impressive results but achieved at high cost
in terms of dollars, space and time.
DeepMind showed that a deep reinforcement learning
system is able to synthesize control actions for computer
games directly from vision data [14]. While this result is
an important and exciting breakthrough it does not transfer
directly to real robots with real cameras observing real
scenes [2]. In fact very modest image distortions in the simulation environment (small translations, Gaussian noise and
scaling of the RGB color channels) caused the performance
of the system to fall dramatically. Introducing a real camera
observing the game screen was even worse [15].
There has been increasing interest to create robust visuomotor policies for robotic applications, especially in reaching
and grasping. Levine et al. introduced a CNN-based policy representation architecture with an added guided policy
search (GPS) to learn visuo-motor policies (from joint angles
and camera images to joint torques) [16], which allows
reduction in the number of real world training examples
by providing an oracle (or expert’s initial condition to start
learning). Impressive results were achieved in complex tasks,
such as hanging a coat hanger, inserting a block into a toy,
and tightening a bottle cap. Recently it has been proposed to
simulated depth images to learn and then transfer grasping
skills to real-world robotic arms [17], yet no adaptation in
the real-world has been performed.
Transfer learning attempts to develop methods to transfer
knowledge between different tasks (scenarios) [18], [19].
To reduce the amount of data collected in the real world
(expensive), transferring skills from simulation to the real
world is an attractive alternative. Progressive neural networks
are leveraged to improve transfer and avoid catastrophic
forgetting when learning complex sequences of tasks [20].
Their effectiveness has been validated on reinforcement
learning tasks, such as Atari and 3D maze game playing.
Modular reinforcement learning approaches have shown skill
transfer capabilities in simulation [21]. However, methods for

real-world robotic applications are still scarce and require
manually designed mapping information, e.g. similaritybased approach to skill transfer for robots [9]. To reduce the
number of real-world images required, a method of adapting
visual representations from simulated to real environments
was proposed, achieving a success rate of 79.2% in a
“hook loop” task, with 10 times less real-world images [10].
Domain randomization has also been successfully used to
transfer deep neural networks from simulation to the real
world [7], [8].
III. M ETHODOLOGY
Domain randomization has been successfully used to
transfer deep neural networks from simulation to the real
world for object position recognition [7] and also visuomotor control [8]. In [8], auxiliary tasks such as cube pose
estimation are used in training to improve performance in
an end-to-end manner. In this paper, we propose a modular
approach to take use of the object position estimation task for
more efficient learning and transfer of visuo-motor policies.
A. Modular Deep Networks
Studies of deep visuo-motor policies indicate that the convolutional layers focus on perception, i.e., extracting useful
information from visual inputs, while the fully connected
(FC) layers perform control [23]. To make the learning and
transfer of visuo-motor policies more efficient, we propose
to separate a deep neural network into perception and control
modules connected by a bottleneck layer (Fig. 2). The
bottleneck forces the network to learn a low-dimensional
representation, not unlike Auto-encoders [24]. The difference
is that we explicitly equate the bottleneck layer with the
object position (x∗ ).
With the bottleneck, the perception module learns how to
estimate the object position x∗ from a raw-pixel image I;
the control module learns to determine the most appropriate
joint velocities v given the object position and joint angles
q (defined as scene configuration Θ = [x∗ , q]). The values
of x∗ and q are normalized to the interval [0, 1].
B. Training Method
1) Perception: The perception module is trained using
supervised learning – first using simulated images, then finetuned with a small number of real samples for skill transfer
– with the quadratic loss function
m
1 X
2
yp (Ij ) − x∗j ,
(1)
Lp =
2m j=1
where yp (Ij ) is the prediction of x∗j for Ij ; m is the number
of samples. The physical meaning of x∗ guarantees the
convenience of collecting labelled training data.
2) Control: The control module is also trained using
supervised learning with only simulated data:
m
1 X
2
Lc =
kyc (sj ) − vj k ,
(2)
2m j=1
where yc (sj ) is the prediction of vj for state sj . Here, s =
Θ.

fully connected + ReLU

max pooling (2×2)

fully connected

Joint Angles

16×16×128

256×256×3

128×128×64
256×256×64

64×64×128

16×16×64

32×32×128

8×8×64

4×4×64

v1,v2, ..., v7

LcBN

300 units

400 units

x1,x2,x3

64 units

256 units

2×2×64

Joint Velocities

convolutional (3×3) + ReLU

64×64×64

x*
Bottleneck

Perception Loss (Lp)

Perception Module

Control Loss (Lc)

Control Module

Fig. 2. The modular network consists of perception and control modules, connected by a bottleneck layer representing target object position x∗ . The
perception module architecture is customized from VGG16 [22] with its first convolutional layer initialized with weights from pre-trained VGG16. The
control module consists of 3 fully connected layers, it determines joint velocities according to target position and joint angles. The perception and control
modules are first trained separately, then fine-tuned in an end-to-end fashion using weighted losses (Section III-B).

3) End-to-end fine-tuning using weighted losses: To further improve hand-eye coordination, an end-to-end finetuning is conducted for the combined network (perception +
control) after their separate training, using weighted control
(Lc ) and perception (Lp ) losses. Note that s = I in
Equation 2 for the end-to-end fine-tuning, rather than Θ.
The control module is updated using only Lc , while the
perception module is updated using the weighted loss

A

B

Top Centre

L = βLp + (1 − β)LBN
c ,

(3)

Bottom Centre

LBN
c

is a pseudo-loss which reflects the loss of Lc
where
in the bottleneck; β ∈ [0, 1] is a balancing weight. From
the backpropagation algorithm [25], we can infer that δL =
βδLp + (1 − β)δLBN
, where δL is the gradients resulting
c
are the gradients resulting respectively
from L; δLp and δLBN
c
BN
from Lp and Lc (equivalent to that resulting from Lc in
the perception module).
IV. B ENCHMARK : ROBOTIC R EACHING
We use a canonical target reaching task as a benchmark to
evaluate the feasibility of the proposed approach. The task
is defined as controlling a robot arm so that its end-effector
position x in operational space moves to the position of a
target x∗ ∈ Rm (object position introduced in Section IIIA). The robot’s joint configuration is represented by its joint
angles q ∈ Rn . The two spaces are related by the forward
kinematics, i.e., x = K(q). The reaching controller adjusts
the robot configuration in velocity mode to minimize the
error between the robot’s current and target position, i.e.,
kx − x∗ k. We consider a 7 DoF robotic arm (Fig. 1), i.e.,
q ∈ R7 steering its end-effector position in 3D, i.e., x ∈ R3
– ignoring orientation.
A. Task Setup
The real-world task employs a Baxter robot’s left arm
(7 DoF) to reach a blue cuboid in clutter. All objects are
arbitrarily placed in the operation area (50×60 cm) as shown
in Fig. 3A. The blue cuboid has a side length of 6.5 cm. The
robot observes environments through a monocular camera
in its right hand (Fig. 1A), providing RGB images with a
resolution of 256×256. The left arm is controlled in velocity mode. A reach is deemed successful, if the Euclidean

Operation Area (50×60cm)

Fig. 3. A Baxter robot controls its left arm in velocity mode to reach a
blue cuboid (6.5×6.5×6.5 cm) in clutter arbitrarily placed in the operation
area. The “Top Centre” and “Bottom Centre” are the top centre of the target
cuboid and the bottom center of the suction gripper. Fig. 3B shows the left
arm in its reference initial configuration.

distance between the the top centre of the target cuboid and
the bottom center of the suction gripper (“Top Centre” and
“Bottom Centre” in Fig. 3) is smaller than 4.6 cm (half of
the diagonal length of any side of the cuboid). In the task,
the left arm is randomly initialized to a configuration with a
normal distribution around the reference configuration shown
in Fig. 3B.
B. Network Architecture
In this work, we used a network with the architecture
shown in Fig. 2. The perception module has an architecture
customized from VGG16 [22] for lower computational cost
but without losing too much performance for this task. It
consists of twelve convolutional layers with 3×3 filters and
seven 2×2 max pooling layers, followed by three fully
connected layers. Simulated or real RGB images are cropped
and down-sampled to 256 × 256 as inputs to the perception
module. The control module consists of 3 fully connected
layers, with 400 and 300 units in the two hidden layers.
Input to the control module is the scene configuration Θ
(target position and joint angles), its outputs are the estimates for joint velocities v (7 DoF). Networks with a
first convolutional layer initialized with weights from pretrained VGG16 [22] are observed to converge faster and

Simulated Images

...
Real Images

...
Fig. 4.
Simulated and real images for training perception modules.
Simulated images were collected from a V-REP simulator using domain
randomization [7]. Real images were collected for domain adaptation on a
real Baxter (Fig. 1B).

achieve higher accuracy, particularly when the training sets
are relatively small. The other parts of the networks are
initialized with random weights.
C. Datasets Collection
Perception datasets contain a number of image-position
(I-x∗ ) pairs. In this work, we label the position of the target
cuboid top centre as the target position x∗ rather than its
mass-centre. Aiming to get a good balance between domain
randomization and adaptation, we collected both simulated
and real data as shown in Fig. 4 for investigation. The
simulated data was collected using V-REP [26] (a robotic
simulation platform) through domain randomization [7] in
the following aspects:
• number of distractor objects in clutter: random in [0, 9];
• shape of distractor objects in clutter: random in 9
primitive shapes with different geometries (5 cuboids,
2 spheres, 2 cylinders);
• pose of distractor objects: random position in the operation area and random orientation of the axis pointing
upwards;
• color of distractor objects: random RGB values;
• left arm configuration: random in joint space, excluding
the ones with self-collision;
• color of the table, floor, robot body and target cuboid:
random changes based on reference colors (±10%);
• camera pose: random changes of each right arm joint
angle based on reference angles (±1%);
• camera field of view (FoV): random changes based on
a reference FoV (±2%);
• table pose: random changes based on a reference posi-

tion ([±1.5%, ±5%, ±1%]) and a reference orientation
of the axis pointing upwards (±7%);
All the above randomization is uniformly distributed. The
reference colors , FoV and table pose were tuned manually to
approximate the real scene. The reference joint angles were
tuned in the real world, making sure the in-hand camera
can see the entire operation area. The parameters for the
aspects randomized based on references were manually tuned
to simulate possible variations in the real scene.
The real images shown in Fig. 4 were collected on a
real Baxter robot (Fig. 1B) with random objects and left
arm configurations. The ground-truth position of the target
blue cuboid was collected by putting the end-effector bottom
centre on the cuboid top centre and recording the left arm
configuration (target configuration q∗ ) for forward kinematics, i.e., x∗ = K(q∗ ). The ground-truth position collected
in this way is accurate enough for the benchmark task,
although some errors might be caused by manually matching
the end-effector with the cuboid. This ground-truth position
collection method was also used in the control performance
evaluation in Section V.
In training, to increase the training data diversity, data
augmentation is done on-the-fly for both simulated and real
images by varying image brightness (±80% for simulated
images and ±40% for real images) and white balance
(±2.5%) in a post-processing manner. These augmentation
parameters were empirically determined.
Control datasets contain a number of sceneconfiguration-velocity (Θ-v) pairs (i.e., trajectories) as
well as image-velocity (I-v) and image-position (I-x∗ )
pairs. Θ-v pairs are for training control modules separately
(Section III-B.2); I-v and I-x∗ pairs are for end-to-end
fine-tuning to obtain δLc and δLp (Section III-B.3).
We collected control datasets in simulation using V-REP.
Trajectories were generated to control the left arm with a
random initial configuration (excluding the ones with selfcollision) to reach a target arbitrarily placed in the operation
area, regardless of obstacle avoidance. As introduced in
Section IV-A, the random initial configuration has a normal
distribution around the reference configuration shown in
Fig. 3B; the random targets are uniformly distributed in the
operation area. When generating the trajectories, the pseudo
inverse method (V-REP internal implementation) was used to
calculate the desired arm configuration to reach a target, i.e.,
q∗ = K−1 (x∗ ). Then a simple proportional controller was
used to control the left arm to reach the desired configuration
from its initial configuration with a control frequency of
20 Hz. In the process, the target cuboid position, joint angles
and velocity commands were recorded, accompanying with
the images from the camera in the right hand. Experiments
(Section V-B) show that simulated control training data is
sufficient to achieve good performance alone – there is no
need to collect real control datasets.
V. E XPERIMENTS AND R ESULTS
We evaluated the performance of the proposed approach
in three aspects: perception accuracy, control performance

[cm]
55

279

3.7
'1.8

3.2
'1.3

2.9
'1.7

31.2
'16.2

4.5
'4.0

3.7
'1.9

3.2
'1.7

50
45

PP
186

35
30
25

93

35.1
'21.8

33.3
'22.4

32.8
'23.9

6.9
'8.4

20
15

0

NaN

44.5
'18.9

44.6
'17.4

340

750

50.2
'23.0

10
5
0

A. Perception Accuracy

0

3000

Number of Real Images
Fig. 5. Object position estimation error map. The numbers in the map
show the mean and standard deviation of the Euclidean distances between
predicted and ground-truth positions. “NaN” means no result for that case.
6

Euclidean Distanc Error d [cm]

To investigate the influence of the numbers of simulated
and real images on perception accuracy, we evaluated 15 different perception modules. They were trained with different
combinations of images:
• the number of simulated images from 0 to 3000;
• the number of real images from 0 to 279.
As introduced in Section III-B.1, all the 15 perception
modules were first trained using simulated images then finetuned with real images. The training was from scratch, except
that the first convolutional layer was initialized with weights
from pre-trained VGG16 [22]. In training, we used a minibatch size of 32 and a learning rate of 0.01. RmsProp [27]
was adopted, the same in the training of control modules
(Section V-B) and end-to-end fine-tuning (Section V-C).
The pixel values in images were normalized to [−1, 1] per
image. The mean and standard deviation of their perception
errors for a test set are shown in Fig. 5. The test set has
48 real images where the target is uniformly distributed in
the operation area, with random distractor objects that have
appeared in training.
From Fig. 5, we can see that the perception modules
trained with only simulated (the bottom row) or real images
(the left most column) have very large errors. For the ones
trained with both simulated and real images, increasing the
number of either simulated or real images helped reduce the
error. Fine-tuning (adaptation) with as few as 93 real images
can make a perception module work in the real world with an
average error of 6.9 cm. The one trained with 3000 simulated
and 279 real images (the top right one) achieved the smallest
average error (2.9 cm). However, trading off the accuracy and
the used number of real images, we pick the one trained with
3000 simulated and 186 real images for further experiments,
labelled as PP. It has an average error of 3.2 cm which is
slightly larger than the best one, but needs only 67% of the
real images.
To study how much the on-the-fly data augmentation
method (Section IV-C) can help improve the perception accuracy. We trained a perception module using 3000 simulated
and 186 real images without data augmentation. It achieved
an average error of 4.1 cm (±3.2 cm), which is 28% larger

32.5
'16.1

40

Number of Simulated Images

and hand-eye coordination. The performance was evaluated
in the real world using the metrics of:
• Perception Error: the Euclidean distance between the estimated and ground-truth object positions;
• Control Error: the Euclidean distance between the
target cuboid top centre and end-effector bottom centre
(“Top Centre” and “Bottom Centre” in Fig. 3A);
• Success Rate: the percentage of successful reaching among all trials, where a reach is deemed successful if the final Euclidean distance between the target
and end-effector is smaller than 4.6 cm as defined in
Section IV-A;
• Time Cost: the number of control steps used in an
entire reaching process.

5
4

3.1 '2.6
80.0%
(118)

3
1.2 '0.8
100.0%
(333)

2
1
0
10 2

0.9 '0.6
100.0%
(2964)

CC
10 2.5

10 3

10 3.5

Number of Trajectories

Fig. 6. Control performance curve which shows the mean and standard
deviation of the Euclidean distances between the target and end-effector.
Three control modules were evaluated in 15 trials. They were trained with
different numbers of trajectories (the numbers in the brackets).

than PP. This shows that the data augmentation method did
help improve the perception accuracy.
B. Control Performance
To investigate how many trajectories are sufficient for
training a control module, we evaluated three control modules trained with different control datasets which have
varying numbers of trajectories: 118, 333, and 2964. As
introduced in Section IV-C, the trajectories in each dataset
were all collected in simulation (therefore cheap) for targets
uniformly distributed in the operation area. In training, we
used a mini-batch size of 64 and a learning rate decreasing
from 0.01 to 0.001. The metrics of control error and success
rate were used in the evaluation. Their performance in 15
real-world reaching trials are shown in Fig. 6. The 15
trials were for targets uniformly distributed in the operation
area, with random initial left arm configurations (normally
distributed around the reference configuration in Fig. 3B).
From Fig. 6, we can see that a control module trained
with more trajectories is able to achieve a better control
performance in terms of both control error and success rate.
The one trained with 118 trajectories has a success rate of
80%; the others are 100%. It also has a much larger control
error than the other two. This indicates that 118 trajectories

TABLE I
E ND - TO - END C ONTROL P ERFORMANCE
Test Condition

Network
EE0: PP + CC
EE1: naively fine-tuned EE0
EE2: EE0 fine-tuned using our approach

Single object (Fig. 7A)
Clutter with novel objects (Fig. 7B)
With occlusions (Fig. 7C)

A

B

Novel Objects

EE2

C

Occlusion

D

Fig. 7. Test cases for end-to-end performance. A: reaching the blue cuboid
without distractor objects; B: reaching with seen and novel (not seen in
training) objects as distractors; C: reaching with occlusion(s); D: reaching
when the target is moving.

are too few to get a good control module. The one trained
with 2964 trajectories just achieved a slightly smaller control
error (0.9 cm) than the one (1.2 cm) trained with much
fewer trajectories (333). This shows that 333 trajectories are
sufficient to get a reasonably good control module. Trading
off the performance and number of trajectories, we pick the
one trained with 333 trajectories to compose the network for
end-to-end reaching in Section V-C, labelled as CC.
C. Hand-eye Coordination
To further improve hand-eye coordination, we proposed an
end-to-end fine-tuning approach using weighted losses. To
evaluate the feasibility of the approach, we compare three
combined networks:
• EE0: composed by PP and CC, directly connected after
separate training without end-to-end fine-tuning;
• EE1: EE0 end-to-end fine-tuned naively, only using the
control loss Lc ;
• EE2: EE0 fine-tuned using the proposed approach with
weighted losses.
In the weighted end-to-end fine-tuning for EE2, β = 0.9
(empirically determined), we used a learning rate of 0.01
and a mini-batch size of 8 and 64 for control and perception
losses respectively. 30225 image-velocity pairs in the control
dataset (333 trajectories) for CC were used in the fine-tuning
to obtain δLc ; its image-position pairs were used to obtain
δLp . To make sure that the perception module remembers
the adaptation to the real scene, the 186 real images for PP
were also used to obtain δLp . In a mini-batch for δLp , 87.5%
samples were real ones, i.e., at each weight updating step,
56 real and 8 simulated images were used.
In the naive fine-tuning for EE1, we used the same
learning rate and a mini-batch size of 64 for Lc . Similarly,
87.5% samples in a mini-batch were real ones, labelled with
velocities obtained using the same method in Section IV-C.
The evaluation of fine-tuning was first done in the real
world without distractor objects on the table as shown in
Fig. 7A, using the metrics of control error, success rate and
time cost. Their results in 15 real-world reaching trials are
listed in Table I. The 15 trials were for the same targets and
initial left arm configurations used in Section V-B.

Control Error [cm]
3.0±1.5
6.1±3.8
2.2±1.2
2.4±1.1
5.4±3.8

Success Rate [%]
80.0
33.3
93.3
93.3
46.7

Average Time Cost [step]
39.6
61.2
29.1
34.9
43.1

From the results for the single object case (Fig. 7A),
we can see that, after the weighted end-to-end fine-tuning,
EE2 achieved a better performance compared to EE0: 27%
smaller control error; 17% higher success rate; 27% fewer
average steps. In comparison, EE1 has a much worse performance than EE0. This shows that the proposed approach of
end-to-end fine-tuning with weighted losses is able to significantly improve the performance of a combined network, but
a naive approach could make the performance even worse.
In addition, EE2 even has a smaller control error than
the perception error of PP. If we individually evaluate the
perception module in EE2, its perception error increased
from 3.2±1.7 cm to 4.2±3.5 cm. This indicates that the endto-end fine-tuning did improve the coordination between the
perception and control modules (hand-eye coordination) in
EE2, rather than improving them individually.
To further evaluate the performance of EE2 in more
challenging cases with novel distractor objects in clutter and
the target cuboid partially occluded, we conducted more
experiments in settings as shown in Fig. 7B-C. From the
results in Table I, we can see that EE2 achieved a comparable
performance to that in Case A (Fig. 7A) even when there
were multiple novel distractor objects in clutter (Fig. 7B). For
the case with the target cuboid partially occluded (Fig. 7C),
we observed a performance drop, but it was still able to reach
half of the targets. We also tested EE2 in the case when the
target cuboid was moving (Fig. 7D). It was able to adapt to
target position changes in real time and performed well in
most cases as shown in the attached video1 .
VI. D ISCUSSION
This work leads us to the following observations:
a) Value of a modular structure and end-to-end finetuning: The significant performance improvement between
EE2 and EE0 after end-to-end fine-tuning with weighted
losses shows the feasibility of the modular approach. Benefiting from the modular structure as well as domain randomization and adaptation, visuo-motor policies for a table-top
reaching task can be learned and transferred from simulation
to the real world with just 33225 simulated (including the
30225 ones for end-to-end fine-tuning) and 186 real samples,
achieving a comparable performance to [8] (the reaching
stage of the multi-stage task) but with fewer training data.
The modular approach can be used in more general ways.
Although we explicitly equated the bottleneck layer with the
target object position in this work, the bottleneck in general
could be any explicit or latent low-dimensional features,
like Auto-encoders. The perception and control modules can
1 The

video is also on the project page https://goo.gl/P1c354.

also be trained with other methods such as unsupervised
learning and reinforcement learning. The feasibility of the
modular approach for reinforcement learning (DQN) has
been validated in a planar reaching task [23], [28].
b) Domain randomization and adaptation: In Section V-A, the perception module trained with 3000 simulated
images has a large error (50.2±23.0 cm), much higher than
expected from [7]. Apart from the experiments in Section VA, we also trained a number of perception modules using
simulated images with random RGB values for the table,
floor and robot body rather than ±10% changes around
the reference colors. However, this did not bring obvious
accuracy improvement. Possible reasons include: too simple
textures (only random RGB values); too simple randomization for light conditions; no simulated shadows; or sensitivity
to domain randomization parameters and tuning.
Nevertheless, the adaptation with just a few real images
(as few as 93) is able to transfer a network from simulation
to the real world, and needs fewer simulated images to reach
a comparable accuracy to [7]. The combination of domain
randomization and adaptation is promising for more efficient
deep neural network transfer.
VII. C ONCLUSION
In this paper, we proposed a modular approach to learn
and transfer visuo-motor policies from simulation to the
real world through domain randomization and adaptation.
The feasibility was demonstrated in the task of reaching a
table-top object amongst clutter with a 7 DoF robotic arm
in velocity mode. By using weighted losses to fine-tune a
combined network in an end-to-end fashion, its performance
was significantly improved (27%), achieving a success rate
of 93.3% with an average control error of 2.2 cm. The
learned policies are robust to novel distractor objects in
clutter and even a moving target. The modular approach is
promising for more efficient transfer of visuo-motor policies
from simulation to the real world.
R EFERENCES
[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in Neural
Information Processing Systems (NIPS), 2012, pp. 1097–1105.
[2] F. Zhang, J. Leitner, M. Milford, B. Upcroft, and P. Corke, “Towards
vision-based deep reinforcement learning for robotic motion control,”
in Australasian Conference on Robotics and Automation (ACRA),
December 2015. [Online]. Available: https://arxiv.org/abs/1511.03791
[3] S. Levine, P. P. Sampedro, A. Krizhevsky, and D. Quillen, “Learning
hand-eye coordination for robotic grasping with deep learning and
large-scale data collection,” in International Symposium on Experimental Robotics (ISER), 2016.
[4] Q. Bateux, E. Marchand, J. Leitner, F. Chaumette, and P. Corke,
“Visual servoing from deep neural networks,” in New Frontiers
for Deep Learning in Robotics Workshop at Robotics: Science
and Systems Conference (RSS), 2017. [Online]. Available: https:
//arxiv.org/abs/1705.08940
[5] K. Katyal, I.-J. Wang, and P. Burli, “Leveraging deep reinforcement
learning for reaching robotic tasks,” in The IEEE Conference on
Computer Vision and Pattern Recognition (CVPR) Workshops, 2017.
[6] A. D’Innocente, F. M. Carlucci, M. Colosi, and B. Caputo, “Bridging
between computer and robot vision through data augmentation: a case
study on object recognition,” in International Conference on Computer
Vision Systems (ICVS), 2017.

[7] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel,
“Domain randomization for transferring deep neural networks from
simulation to the real world,” in IEEE/RSJ International Conference
on Intelligent Robots and Systems (IROS), 2017.
[8] S. James, A. J. Davison, and E. Johns, “Transferring end-to-end
visuomotor control from simulation to real world for a multi-stage
task,” in 1st Annual Conference on Robot Learning (CoRL), 2017.
[9] T. Fitzgerald, A. Goel, and A. Thomaz, “A similarity-based approach
to skill transfer,” in Women in Robotics Workshop at Robotics: Science
and Systems Conference (RSS), 2015.
[10] E. Tzeng, C. Devin, J. Hoffman, C. Finn, P. Abbeel, S. Levine,
K. Saenko, and T. Darrell, “Adapting deep visuomotor representations
with weak pairwise constraints,” in Workshop on the Algorithmic
Foundations of Robotics (WAFR), 2016.
[11] L. Pinto and A. Gupta, “Supersizing self-supervision: Learning to
grasp from 50k tries and 700 robot hours,” in IEEE International
Conference on Robotics and Automation (ICRA), 2016.
[12] L. Tai, S. Li, and M. Liu, “A deep-network solution towards modelless obstacle avoidance,” in IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS), 2016.
[13] I. Lenz, R. Knepper, and A. Saxena, “Deepmpc: Learning deep
latent features for model predictive control,” in Robotics: Science and
Systems (RSS), 2015.
[14] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.
Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski
et al., “Human-level control through deep reinforcement learning,”
Nature, vol. 518, no. 7540, pp. 529–533, 2015.
[15] A. W. Tow, S. Shirazi, J. Leitner, N. Sünderhauf, M. Milford, and
B. Upcroft, “A robustness analysis of deep q networks,” in Australasian Conference on Robotics and Automation (ACRA), 2016.
[16] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training
of deep visuomotor policies,” Journal of Machine Learning Research,
vol. 17, no. 39, pp. 1–40, 2016.
[17] U. Viereck, A. t. Pas, K. Saenko, and R. Platt, “Learning a visuomotor
controller for real world robotic grasping using simulated depth
images,” in 1st Annual Conference on Robot Learning (CoRL), 2017.
[18] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE
Transactions on Knowledge and Data Engineering, vol. 22, no. 10,
pp. 1345–1359, 2010.
[19] M. E. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey,” The Journal of Machine Learning Research,
vol. 10, pp. 1633–1685, 2009.
[20] A. A. Rusu, M. Vecerik, T. Rothörl, N. Heess, R. Pascanu, and
R. Hadsell, “Sim-to-real robot learning from pixels with progressive
nets,” in 1st Annual Conference on Robot Learning (CoRL), 2017.
[21] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine, “Learning
modular neural network policies for multi-task and multi-robot transfer,” in IEEE International Conference on Robotics and Automation
(ICRA), 2017.
[22] K. Simonyan and A. Zisserman, “Very deep convolutional networks
for large-scale image recognition,” in International Conference on
Learning Representations (ICLR), 2015.
[23] F. Zhang, J. Leitner, M. Milford, and P. Corke, “Modular deep q
networks for sim-to-real transfer of visuo-motor policies,” Queensland
University of Technology, Tech. Rep., 2017. [Online]. Available:
https://arxiv.org/abs/1610.06781
[24] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality
of data with neural networks,” science, vol. 313, no. 5786, pp. 504–
507, 2006.
[25] Y. LeCun, “A theoretical framework for back-propagation,” in Proceedings of the 1988 Connectionist Models Summer School, D. Touretzky, G. Hinton, and T. Sejnowski, Eds. CMU, Pittsburgh, Pa: Morgan
Kaufmann, 1988, pp. 21–28.
[26] E. Rohmer, S. P. Singh, and M. Freese, “V-rep: A versatile and scalable
robot simulation framework,” in IEEE/RSJ International Conference
on Intelligent Robots and Systems (IROS), 2013, pp. 1321–1326.
[27] T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient
by a running average of its recent magnitude,” COURSERA: Neural
Networks for Machine Learning, vol. 4, no. 2, 2012.
[28] F. Zhang, J. Leitner, M. Milford, and P. I. Corke, “Tuning
modular networks with weighted losses for hand-eye coordination,”
in The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) Workshops, July 2017. [Online]. Available:
https://arxiv.org/abs/1705.05116

