abstract: In this paper, we propose a deep reinforcement learning (DRL) solution to the grasping problem using 2.5D images as the only source of information. In particular, we developed a simulated environment where a robot equipped with a vacuum gripper has the aim of reaching blocks with planar surfaces. These blocks can have different dimensions, shapes, position and orientation. Unity 3D [^1] allowed us to simulate a real-world setup, where a depth camera is placed in a fixed position and the stream of images is used by our policy network to learn how to solve the task. We explored different DRL algorithms and problem configurations. The experiments demonstrated the effectiveness of the proposed DRL algorithm applied to grasp tasks guided by visual depth camera inputs. When using the proper policy, the proposed method estimates a robot tool configuration that reaches the object surface with negligible position and orientation errors. This is, to the best of our knowledge, the first successful attempt of using 2.5D images only as of the input of a DRL algorithm, to solve the grasping problem regressing 3D world coordinates.
author:
;
bibliography: IEEEtranBST/IEEEabrv.bib; IEEEtranBST/mybibfile.bib
title: Learning to Grasp from 2.5D images: a Deep Reinforcement Learning Approach

# INTRODUCTION

In industrial environments, manipulator robots are usually designed to solve precise and predefined tasks. However, there are situations where it may be required to generalize the behaviour of the robots due to variations of size, shape, position, and orientation of the object to grasp. In these cases, the development of solutions according to mainstream standard computer vision and robotic control approaches can be complex and may lead to customized algorithms that cannot be easily generalized to different scenarios. Deep Reinforcement Learning addresses this task by merging the reinforcement learning and the deep learning domains, approximating the policy to learn with a deep neural network. The DRL framework has been applied to several tasks achieving impressive results. Although this success, to the best of our knowledge, no DRL algorithm has effectively solved the specific manipulation and grasping task.

The first contribution of this work is the development of a simulated grasping environment consisting of a manipulator robot equipped with a suction cup as a gripper. This environment is based on *Unity 3D* together with a modified version of *ML-Agents* , and has been used to train the proposed algorithm. The second contribution is the DRL system that has been trained to grasp blocks characterized by random parameters in terms of dimensions, shape, position, and rotation relying only on visual data. The reinforcement learning problem is modelled so that state corresponds to the 2.5D grabbed image and the action corresponds to the manipulator tool pose. At every trial, the algorithm receives a reward in response to each action and uses the rewards to update the policy approximated by a deep neural network. Setting-up an environment using a depth-camera is easy but exploiting all the information that a 2.5D image contains is a challenging task. Solving this task can bring a lot of flexibility at a relatively low cost. We propose an analysis based on ones of the most important DRL algorithms, varying configurations of visual inputs and rewards. We used and compared three policies: Proximal Policy Optimization (PPO), Trust Region Policy Optimization (TRPO) and Deep Deterministic Policy Gradient (DDPG). Our results suggest that DRL is a promising approach to solve grasping problems that require precision.

# RELATED WORKS

Deep reinforcement learning has been applied to solve several tasks such as learning to play video-games and robotics problems . In particular, it has been applied to grasp tasks with manipulator robots equipped with grippers, to locomotion tasks and also to humanoid robots . These problems are currently solved in a simulated environment. Several works obtained good results, like that involves the simulation of a *Kuka* robot and its working environment to develop a solution based on deep reinforcement learning algorithms, to solve the grasping problem. Another recent work is that simulate four complex tasks of dexterous manipulation based on deep reinforcement learning. It uses a policy gradient method, in particular, *Natural Policy Gradient* in combination with an imitation learning algorithm called *Behavioral Cloning* which learns a policy through supervised learning to mimic the demonstrations of an expert.

*Gibson Virtual Environment* is an open-source perceptual and physics simulator. This work can be seen as a bridge between learning from a simulator and transfer learning. The main goal of *Gibson environment* is to facilitate transferring models trained in a simulated environment to the real world.

In particular, it ensures semantic complexity of the simulated environment that aims to be a good replica of the complex real-world and it renders visual observations that are as much as possible close to what a real camera captures. It employs a *VGG-16* neural network for making rendering more like real images (forward function) and a network that makes real images look like rendering (backward function).

Only a few works   achieved the complex goal of transferring the knowledge learned in a simulated environment to the real world. The core of these approaches is *generalization*, obtained with randomization. In authors exploit the full state observability in the simulator to learn better policies which take as input partial observation like RGB-D images. They use an *actor-critic* algorithm in which the critic is trained on full states while the actor takes images as input state. Then they use domain randomization and applied transfer learning to real robot tasks such as pushing, picking and moving blocks.

The second work   uses *progressive networks* to bridge the reality gap and successfully transfer policies learned in simulation to the real world. *Progressive networks* aim to reuse everything from low-level visual features to high-level policies for transferring knowledge to new tasks. They use only a deep reinforcement learning approach with sparse rewards. RGB images are used as a state while joint velocity as actions.

*Actor-critic* algorithm with multiple asynchronous agents is used for training in simulation with a CNN (Convolutional Neural Network) followed by an LSTM (Long-Short Term Memory). They use discrete actions to control different degrees of freedom of the simulated robot. After training in the first simulated environment, a new network is initialized with lateral, non-linear connections to each convolutional and recurrent layer of the previously trained network. Then the new network is trained in real-world tasks, using the features learned by the first network in the simulated environment.

Although the previous works seem to obtain good results using deep reinforcement learning, there are a lot of other works that do not show these satisfactory results. A significant example is , which shows good results mainly on tasks with vector observations as input, while most of the tasks with visual observations perform badly.

Reward functions are difficult to design and their tuning depends on the specific problem; moreover, their design heavily affects the learning process. It is common that during the training process the *cumulative reward* given by the policy stops growing and starts swinging around a local optimum, increasing variance around these values.

Another problem in deep reinforcement learning, and deep learning in general, is hyper-parameters tuning. Deep reinforcement learning problems involve a huge number of hyper-parameters that affect training process. It is necessary to tune hyper-parameters such as the learning rate, batch size, random seed, architecture of the policy networks, together with correct reward functions, defining and normalizing them before feeding the network. In the authors introduce a large comparison of state of the art problems, algorithms and implementations to highlight how deep reinforcement learning is still heavily weak to hyper-parameters variation.

# PROBLEM SETUP

Deep reinforcement learning is a promising field to solve robotics tasks because it can solve continuous space actions problems using only raw data taken from a camera. Moreover, no manual features extraction is necessary beyond the one performed by the neural network. The goal of the project is evaluating how DRL can solve a difficult problem such as sensor-driven grasping objects with a flat gripper. As explained in section , the literature shows conflicting results in deep reinforcement learning for grasp tasks. Traditional approaches based on classical computer vision are more susceptible to camera parameters variations, blocks position and rotations, light variance etc. Moreover, it is difficult to find another approach that generalizes to a random position, rotation and shape of the blocks.

The final goal is teaching to a robot, in our case a simulated Kuka KR16, to reach flat objects placed on a support in front of the robot. It has to place its end-effector in front of the first block, in a position that leads to picking the object, with the correct orientation. As far as we know, deep reinforcement learning has never been applied to suction grasping problems, which requires accurate estimation of position and orientation. The simulated environment implemented with *Unity* is shown in .

## Depth visual observations

We simulate depth camera adding Gaussian noise to make more realistic images. In depth camera image with Gaussian noise is shown.

ML-Agents visual encoder input is modified clamping it to the minimum and maximum range of a real camera to simulate a real depth input. In this way, when a trained model is used to make inference on real images, it does not need to make any pre-processing phase (or less as possible). For depth input we scale them between $[0.4, 2]$ meters range, try to simulate a real meters coordinates, that could generate a real depth camera. In particular, we refer to *Orbbec Astra S* camera that has exactly a 0.4 to 2 meters tracking range as shown in . $$new_v = min_v + actual_v \cdot \frac{max_v - min_v}{255.0}
\label{eq:clamp}$$

We also introduce the possibility to treat differently inputs from a different type of camera: depth and RGB.

ML-Agents works with a flow of images, a stream of consecutive images that are passed through convolutional layers to extract features. Our tasks a single configuration for each episode. This makes the task solvable only looking at the first frame of the stream; there is no need for any extra information. Indeed during an episode, the agent could try different actions, but the environment configuration does not change. Moreover, the camera is positioned on the base of the robot, so images during action can show the robot that hides blocks. In such a context, frame grab can be helpful to simplify the learning process and to avoid images that show robot movements.

## Generalizations

To make the learning process general, we applied some random transformation to the environment at the beginning of every episode. If the environment is static, the learning process risks overfitting a particular configuration (blocks position and rotation), hence, an agent trained on a static environment will be only able to grasp objects in the only configuration seen during the training phase, failing in all the others. We randomize blocks position, orientation and, scale; light position and intensity and camera position, orientation and clip planes.

## Rewards design

Sparse rewards are given only when a specific event happened, for example when a goal is reached or the agent hits something in the environment. This is suitable for simple problems characterized by discrete actions (like e.g. choosing an action that leads to picking an object instead of another), substantially when actions space and solutions space are relatively small. If the problem has these properties, using sparse rewards is the best way to solve it. This is because sparse rewards are simply scalars, so it is easy to find a good reward design. However, this is not the case of many robotics tasks, especially for our problem that is characterized by huge actions space and specific target position and rotation. For this reason, dense rewards are used. Dense rewards are given at each action and give to the agent the information about how much is approaching the target. It is necessary to design functions that properly compute this measure. Rewards design is one of the challenges of reinforcement learning. With no rewards, the agent is not able to discriminate if an action is good or bad. Conversely, exceeding with rewards may lead to convergence to local minima in gradient descent. Indeed when an agent gets small positive rewards at each action, network parameters, such as *beta* and *learning rate*, decrease reducing exploration in favour of exploitation and accordingly stabilizing to a local minimum.

We implemented five functions to define rewards. Among them three function providing rewards only at the occurring of a specific event. In particular, a small positive reward of $0.1$ is given when the suction head touched the target. This reward is given because during the first steps, with random actions, it is really difficult for the agent reaching the correct position, so it can be useful for the learning process to give a small positive reward in response to an action that brings the end-effector to touch the target. A negative reward of $-0.1$ is given when undesired collisions happen. In particular, we consider undesired collisions, collisions between each axis of the robot and each block and support, suction head and the support. In this way, we can implement *obstacle avoidance* simply giving a negative reward for each undesired collision. A positive reward is given if the action chosen takes the end-effector within a range of positions, then a further positive reward is given if also rotation coordinates are between defined values. To give a correct reward for the position we define a *tooltip*, that is a 3D point position on the centre of the suction head surface. So we consider a good action an action that takes the *tooltip* position coinciding with block centroid position plus an offset along *z* axis that considers that centroid of the block is within the centre of the mesh. For rotation we check only rotation around *y* axis. In particular we set a range of $[-0.1, 0.1]$ meters around block centroid along *x* and *y* axes and a range between $[0.01, 0.02]$ meters for *z* axis, while for end-effector rotation we consider a range of $[-10, 10]$ degrees from block rotation. We set a reward of $0.5$ if the correct position is reached and another reward of $0.5$ if also rotation is reached. Having only these three rewards functions we may have episodes where no rewards are received. For this reason, we implement the other two functions providing progressive rewards after each action informing the agent on the effectiveness of the action chosen. The first function is shown in . $$r_\text{fmt} = k_{1} \cdot f_\text{mt}
\label{eq:dir_func}$$ where $$f_\text{mt} = \vec{v} \cdot \vec{n}
\label{eq:fmt}$$ with $v$ that is velocity computed considering the previous position and the current one, while $\vec{n}$ is the difference between the target position and the current end-effector position, normalized. In this way, the function is positive if the action chosen makes the end-effector moving towards the target and is negative if the action chosen moves the end-effector away from the target. A second function is implemented in . $$r_{fft} = - k_{2} \cdot f_\text{ft}
\label{eq:or_func}$$ where $$f_\text{ft}  = \vec{n} \cdot \vec{z_\text{ee}}
\label{eq:fft}$$ with $z_{\text{ee}}$ which is the normal vector to the end-effector face. A positive reward is given when direction of *z*-axis of end-effector frame is opposite to block *z*-axis, a negative one otherwise. $k_{1}$ and $k_{2}$ are constants empirically defined respectively as $0.03$ and $0.01$.

## Curriculum learning design

Curriculum learning gives the possibility to increase the task difficulty during the training process. In particular, it can be useful to restrict the area that gives large rewards when the cumulative reward overcomes a predefined threshold. Each threshold defines a step of learning. We vary the area around the block that defines the best position and rotation of the end-effector. *x* and *y* position limits, *z* position upper limit and *y* rotation limits are modified at each step, restricting progressively the goal area. 19 lessons are implemented starting form a negative threshold and increasing it at each constant and durable cumulative reward increment. The last lesson provides an area of 0.01 meters variation around *x*, *y* and *z* axes for position and 2 degrees angle around *y* for rotation.

# REINFORCEMENT LEARNING ALGORITHMS

Reinforcement learning is based on an autonomous agent observing a state $s_{t}$ from its environment at time $t$. The agent interacts with the environment by taking an action $a_{t}$, than the agent and the environment go to a new state $s_{t+1}$ based on the current state and the chosen action. At each transition to a new state, a single or multiple rewards are provided as scalars and then embodied in $r_{t+1}$ value given to the agent as feedback. The goal of the agent is to learn a control strategy called policy $\pi$ that maximize the expected cumulative rewards and/or maximizing a value function $V^\pi$ estimating the expected cumulative reward of a given state. Indeed what matter in reinforcement learning problems is expectations . Reinforcement learning can be described as a Markov Decision Process (MDP) which can be considered as episodic, so the state is reset after each episode with length $T$. These elements (states sequence, actions, rewards within an episode) form the trajectory of the policy, where the policy can be represented by . $$\pi : S \rightarrow p(\mathcal{A} = a | \mathcal{S})
    \label{eq:policy}$$ where $\mathcal{S}$ is the set of states and $\mathcal{A}$ is the set of actions. The policy can be considered a mapping between states and actions which maps a state to an action or a distribution over actions and its goal is finding an optimal mapping between them. Deep reinforcement learning consists of optimizing a policy using a deep neural network as an approximation. Optimal criteria represent how good the model fit data and this measure in DRL is *future rewards*. There are two optimal criteria to maximize future rewards: *value function*, to evaluate the state on base of the probability of future rewards and *policy*, to guide an agent in the choice of an action given the state. These criteria generate three classes of DRL algorithms: value-based, policy-based and actor-critic. In policy-based methods, policy outputs parameters for a probability distribution, usually modelled by Gaussian distributions for continuous actions. Policy parameters are updated to maximize expected return using gradient-based or gradient-free optimization. Policy gradients methods are often used in *actor-critic* implementations. Actor-critic methods combine value function methods with policy-based methods, reducing variance introduced by policy gradient algorithms and at the same time bias introduced from value function methods. Indeed actor-critic uses value function learned to optimize policy. Our focus is on *actor-critic* algorithms. In particular, three of the most important and recent ones are used: *Proximal Policy Optimization*, *Trust Region Policy Optimization*and *Deep Deterministic Policy Gradient*. We also used PPO combining with *Curriculum Learning*.

## Deep Deterministic Policy Gradient

Deep Deterministic Policy Gradient is a model-free, off-policy algorithm using deep function approximator that can learn policies in high-dimensional and continuous action space. It is based on a policy gradient algorithm that uses a stochastic behaviour policy to improve exploration. Actions are chosen to base on the stochastic policy, to estimate a deterministic target policy which is easier to learn. DDPG uses experience replay that stores a replay buffer with the experiences of the agent during training, and then randomly sample experiences to break up the temporal correlations within different training episodes. DDPG can learn using low-dimensional observations and directly from pixels. In particular, we use version that adds normal noise $N$ to exploration actor policy $\mu'$ sampled from a noise process as shown in . $$\mu'(s_{t}) = \mu(s_{t} | \theta^\mu_{t}) + N
    \label{eq:noise}$$

## Trust Region Policy Optimization

Trust Region Policy Optimization aims to minimize an objective function guaranteeing policy improvement with non-trivial step size. Then a series of approximations are done to apply a theoretical algorithm to practice . In TRPO an objective function is maximized subject to a constraint on the size of the policy update. It is formally written in . $$max_{\theta} \hat{E}_{t} \bigg [\frac{\pi_{\theta}(a_{t} | s_{t})}{\pi_{\theta_{old}}(a_{t} | s_{t})} \hat{A}_{t}\bigg]
\label{eq:trpo}$$ subject to $$\hat{E}_{t}[KL[\pi_{\theta_{old}}( \cdot | s_{t}), \pi_{\theta}( \cdot | s_{t})]] \leq \delta
\label{eq:trpo2}$$ where KL is Kullback - Leibler divergence that is a measure of how one probability distribution diverges from a second expected probability distribution. A constraint can be added in the form of penalty solving the unconstrained optimization problem. $$max_{\theta} \hat{E}_{t} \bigg [\frac{\pi_{\theta}(a_{t} | s_{t})}{\pi_{\theta_{old}}(a_{t} | s_{t})} \hat{A}_{t}\bigg] - \beta \hat{E}_{t}[KL[\pi_{\theta_{old}}( \cdot | s_{t}), \pi_{\theta}( \cdot | s_{t})]]$$ In this way, for *Lagrange multiplier* method optimal point of $\delta$-constrained point problem is also an optimal point of $\beta$-penalized problem for some $\beta$ and in practice, $\delta$ is easier to tune. Using max KL instead of mean KL lead to a pessimistic bound on policy performance. TRPO has several limitations: it is hard to use with architecture with multiple outputs and empirically performs poorly with deep CNNs and RNNs (Recurrent Neural Networks). Moreover constrained gradient computation makes implementation core complicated.

## Proximal Policy Optimization

Proximal Policy Optimization is an algorithm that reaches data efficiency and performance of TRPO, without its limitations, using only first-order optimization. It uses clipped probability ratios which form a lower bound of policy performance. It is based on an alternation between sampling data from the policy and performing subsequent epochs of optimization on the same data. Formally it can be explained introducing the probability ratio $r_{t}(\theta)$ so that $r_{t}(\theta_{\text{old}}) = 1$. PPO modify the objective to penalize change that moves $r_{t}(\theta)$ away from 1. In particular, $L^{CLI}(\theta)$, lower bound on the expected improvement of consecutive policy iteration, can be written as in . $$L^{CLI}(\theta) = \hat{E}_{t} \bigg [\frac{\pi_{\theta}(a_{t} | s_{t})}{\pi_{\theta_{old}}(a_{t} | s_{t})} \hat{A}_{t}\bigg] = \hat{E}_{t} [r_{t}(\theta) \hat{A}_{t}]
\label{eq:ppo}$$ This equation has been modified as by adding a clip normalization between $[1- \epsilon, 1 + \epsilon]$ as in . $$L^{CLIP}(\theta) =  \hat{E}_{t} [min(r_{t}(\theta) \hat{A}_{t}, clip(r_{t}(\theta), 1 - \epsilon, 1 + \epsilon)\hat{A}_{t})]
\label{eq:ppo2}$$ $L^{CLIP}(\theta)$ is a lower bound on $L^{CLI}(\theta)$ with a penalty for having too large policy updates. Another way is to use a penalty on KL divergence to achieve some target value at each policy update. Using a neural network as an approximation that shares parameters between the policy and the value function, a loss function has to be used to combine policy surrogate and a value function error term. It is done adding an entropy bonus to ensure exploration . This policy is better then TRPO in continuous control and compatible with multiple output networks, such as CNNs and RNNs.

## Curriculum learning

Curriculum learning can be seen as a general strategy for the global optimization of non-convex functions. The basic idea of curriculum learning is to augment problem difficulty during training. At an abstract level, a curriculum can also be seen as a sequence of training criteria. Each training criterion in the sequence is associated with a different set of weights on the training examples, or more generally, on a re-weighting of the training distribution. Initially, simpler examples that can be learned easily are presented to the agent. Then the next steps involve a slight change in the weighting of examples that increase the probability of sampling slightly more difficult examples. At the end of the sequence, the re-weighting of the examples is uniform and we train on the target training set or the target training distribution . In deep reinforcement learning *curriculum learning* can be implemented modifying the environment to increase the problem difficulty during training.

# EXPERIMENTS AND RESULTS

In this section, the experimental validation and parameters tuning process are presented. Even if an environment is well defined, or the reset logic is coherent to the problem and the reward design is done, the training process can lead to bad results. Then, a hard process of parameters tuning has to be performed to obtain the best configuration for the sensor-based grasping task. Final results of the work are shown, in particular, the results varying according to visual inputs dimensions and architectures. CNNs with a different number of convolutional layers and input images dimensions are tested. The same task is performed, in a scenario with two cameras. An additional RGB camera is fixed on the top of the support, to observe the objects. Also, frame grab working is tested. Eventually, an algorithms comparison is done using the most famous state of the art reinforcement learning algorithms. The task is addressed with DDPG and TRPO policies, with PPO plus curriculum learning and PPO with the addition of LSTM besides the CNN. For DDPG and TRPO *stable baselines* implementation is used.

Not all tests have been done keeping the same maximum number of training steps. The tests were interrupted when the result seemed satisfactory, or when the increase in the cumulative reward did not seem significant for a large number of steps. In the latter case, it is assumed that to obtain positive results, we should expect a much longer time (not by the purpose of this analysis) or have greater computing resources.

All the experiments have been performed using a Intel^^ Core^^ i7-7820X CPU and NVIDIA^^ GTX 1060.

## Sparse rewards

To validate the analysis we tried to solve the task of taking with sparse rewards. In this test is returned only one positive reward when the action chosen by the network leads the agent to reach the optimal area. No other rewards have been prepared. The results of subsequent tests show that this type of reward is not sufficient to solve the proposed problem. The cumulative reward remains for many steps around zero with peaks in correspondence with so-called winning actions. This does not mean that the problem can not be solved with a single positive reward, but that probably takes many more training steps compared to the configuration of the problem with dense rewards. Observing the results, it can be seen that at the same number of training steps, in the case of sparse rewards the curve of the cumulative reward remains flat on zero with peaks, while in the case of dense rewards the curve is already growing.

## Visual inputs and CNN architectures

Different input dimensions and convolutional architecture are tested with PPO. In particular we test four image dimensions with 4 layers configurations: 32x32 with 2 convolutional layers (the first one with 4 filters, kernel size \[2, 2\], strides \[4, 4\], the second one with 8 filters, kernel size \[1, 1\], strides \[2, 2\]), 80x80 with 2 convolutional layers (the first one with 16 filters, kernel size \[8, 8\], strides \[4, 4\], the second one with 32 filters, kernel size \[4, 4\], strides \[2, 2\]), 128x128 with 3 convolutional layers (the first and the second ones with 80x80 parameters with a third additional layer with 64 filters kernel size \[2, 2\], strides \[1, 1\]) and 256x256 with 4 convolutional layers (the first, the second and the third ones with 128x128 parameters with a fourth additional layer with 72 filters kernel size \[2, 2\], strides \[1, 1\]). All CNN architectures give satisfactory results in terms of cumulative reward, except for 32x32 test that led to an unstable training process.

Other visual inputs modifications are tested: using two cameras and frame grab. The using of both RGB and depth camera has resulted in a performing learning process. Cumulative reward increases during training and the agent can avoid obstacles and to reach the desired block.

Also frame grab gives results comparable to the normal process.

## Policies comparison

Most famous reinforcement learning algorithms described in Section are compared. DDPG performs poorly with our environment. This is probably due to the huge number of hyper-parameters DDPG requires to tune. Probably applying grid or random search on all of them better results could be reached. TRPO gives some positive results. Cumulative reward increase overcoming 0, but it results in an unstable learning process. PPO with curriculum learning gives the best results. The agent can reach the correct position and rotation with a little margin of error. The trend seems to be unstable, but this is due to the lesson changing after a defined value of the reward is reached. PPO with a different network architecture is also tested: adding an LSTM besides the CNN. It does not give good results, probably because LSTMs are not particularity adapt to solve this kind of tasks. It results in a cumulative reward that oscillates around $0$ for most of the training steps, with some peaks.

## Hyper-parameters tuning

Hyper-parameters tuning process is hard in all machine learning tasks and, in particular for deep reinforcement learning ones because of the huge number of parameters related both to the policy and to the CNN. Moreover, there are several factors related to task design that can affect performances such as actions space, reward values etc.. According to our analysis, parameters that mostly affect the training process are *learning rate* and *batch size*. For the learning rate, two decay types are analyzed (polynomial decay and exponential decay) and different starting values for both the actor and the critic. Depending on the task and configuration, the initial learning rate is set between $10^{-4}$ and $10^{-3}$. Despite batch size is usually set to a high value for continuous actions space, our experiments show that a small value (32) can bring to satisfactory results, speeding up the learning process. For PPO training *beta* decay is slowed down to maintain a correct balancing between exploration and exploitation for the process.

# DISCUSSION AND FUTURE WORK

With this work, we demonstrate the validity of a deep reinforcement learning approach to grasping problems with a vacuum gripper and 2.5D visual inputs. We applied the state of the art reinforcement learning algorithms to this task and compared them.

Different types of visual inputs have been tested together with the definition of several convolutional architectures to find the best configuration. We experimented how the DRL algorithms behave with different inputs such as a single 2.5D input coming from a simulated depth-camera and two visual inputs together as input to the network using a simulated depth map and an RGB camera.

Our approach to the environment definition and reward design gives good results and this suggests that it could be applied to a real-world scenario transferring the knowledge from the simulated environment to the real one, applying some transfer learning techniques. Further research directions can be the definition of a more complete environment that simulates a point cloud together with an extension to ML-Agents that allows working with this kind of data.

# ACKNOWLEDGMENT

The authors would like to thank Dario Lodi Rizzini of the University of Parma, for several fruitful discussions and valuable advice.

[^1]: https://unity3d.com


abstract: Blueprints of polyhedral cages with icosahedral symmetry made of circular DNA molecules are provided. The basic rule is that every edge of the cage is met twice in opposite directions by the DNA strand, and vertex junctions are realised by a set of admissible junction types. As nanocontainers for cargo storage and delivery, the icosidodecahedral cages are of special interest as they have the largest volume per surface ratio of all cages discussed here.

(Article contributed to 'Algorithmic Bioprocesses', A. Condon, D. Harel, J. N. Kok, A. Salomaa, and E. Winfree Editors, Springer Verlag, 2008)
**Natasha Jonoska [^1], Anne Taormina [^2] and Reidun Twarock [^3]**
${}^{1}$*Department of Mathematics
University of South Florida,
Tampa, FL 33620, USA
${}^{2}$**Department of Mathematical Sciences
University of Durham,
Durham DH1 3LE, U.K.**
${}^{2}$**Department of Mathematics and Department of Biology,
University of York,
York YO10 5DD, U.K.
***

# Introduction

Recent advances in biotechnology provide the necessary tools to engineer cage structures from nucleic acids, and open novel avenues for applications in nanotechnology. Cages with crystallographic symmetry have already been realised experimentally in the shape of a cube , a tetrahedron , an octahedron or a truncated octahedron , and one natural idea is to use such cages for cargo delivery or storage . Moreover, models for two realisations of a cage with a non-crystallographic symmetry have recently been proposed from a theoretical point of view . These studies were motivated by the hope that such mathematical considerations on the organisation of RNA or DNA in cages with icosahedral symmetry will aid the design of artificial cages inspired by nature.

We provide here a systematic comparative analysis of three polyhedral cages with icosahedral symmetry, the icosahedron, the dodecahedron and the icosidodecahedron (see Fig. ), which are – from a mathematical point of view – distinguished because they are the three smallest vertex sets realising icosahedral symmetry.

Moreover, their edges are uniformly of the same length. We expect that these properties make them easier to be realised experimentally than other polyhedra with this symmetry, and we therefore focus on these three cases here.

An interesting feature of these polyhedra is the fact that they have vertices of different connectivity: The dodecahedron has trivalent vertices and hence needs to be realised in terms of three-junctions, whilst the icosidodecahedron requires four-junctions and the icosahedron five-junctions. We investigate here possibilities of realising these polyhedra with a circular DNA molecule. Our analysis shows that the complexity of this problem increases when the polyhedra considered have even degree vertices. The strategy implemented here is to enumerate all possible inequivalent realisations of these cages with a single strand of DNA molecule, given a set of rules on the junctions. This combinatorial approach is manageable without computer help for the icosahedral and the dodecahedral cages, but becomes cumbersome for the icosidodecahedron. Our treatment of the latter in this paper is therefore computer-aided, but a more elegant mathematical framework to solve this type of problem is being developed.

Our analysis shows that in all three cases at least two circular DNA strands are needed to form the cage. From that point of view, all three polyhedra lend themselves equally well for templates of DNA cages. However, the icosidodecahedron is the polyhedron among the three with the largest volume to surface ratio (at an edge length of 1, it is approximately 0.47, compared to 0.37 for the dodecahedron, 0.25 for the icosahedron[^4]), making it perhaps the most interesting icosahedral cage for applications in nanotechnology.

We start by introducing our theoretical construction method in general terms for all polyhedra with icosahedral symmetry in Section , and then provide details for the three polyhedra in the subsequent sections: the icosahedral cage in Section , the dodecahedral cage in Section , and the icosidodecahedral cage in Section .

# Construction of cages with icosahedral symmetry: general principles

The goal is to provide blueprints of polyhedral cages with icosahedral symmetry, made of a circular DNA molecule, with the basic rule that every edge of a given polyhedral cage is met twice in opposite directions by the strand. This requirement enables hybridization of the two portions of the strand running along an edge into a double helix structure.

Mathematically, a cage is a graph whose nodes are the vertices of the corresponding polyhedron, and the connectors are the edges. As explained in , the idea is to topologically embed graphs into orientable thickened graphs as deformation retracts. Such thickened graphs are compact orientable 2-dimensional surfaces constructed out of strips and thickened $n$-junctions glued together. They can be considered as models for DNA cages as follows: The boundary curves of the thickened graph represent single stranded DNA molecules in the cage, and provide a blueprint that specifies which types of junctions have to be used to realise this graph.

The design of such templates for DNA cages must take the following factors into account:

1.  *Initial data*: Assume that the cages correspond to polyhedra with all edges of equal length $\lambda$. Then the number $\nu(\lambda)$ of half-turns in the duplex structure along each edge depends on $\lambda$. Configurations where $\nu(\lambda)$ is odd are modelled as cross-overs in the planar projective views of the polyhedral cages as shown in Fig. .

    Note that for DNA there are about 10.5 base pairs (bp) per helical turn.

2.  *Thickened $n$-junctions*: Mechanical stress may be imposed on the overall configuration if the strands of the edges cross each other at the incident vertex junctions. For example, the thickened $n$-junction shown in Fig. (a) imposes no stress on the configuration (we name it 'type A$_n$'), whilst the thickened $n$-junctions appearing in Fig. (b) and (c) accommodate one or two cross-overs (we name them 'type B$_{1n}$' and 'type B$_{2n}$') and may impose stress on the overall configuration unless extra nucleotides are introduced along the corresponding edge that compensate for it. In general, a thickened $n$-junction with $k$ cross-overs of the strands is of type B$_{kn}, k \le n$.

The number $n$ of legs in the junctions depends on the type of cage considered. In subsequent sections, cages with thickened 5-junctions (icosahedra), 4-junctions [^5] (icosidodecahedra) and 3-junctions (dodecahedra) are discussed.

## The construction procedure: Step I

The first step in the construction procedure is to identify *start configurations*, i.e., orientable thickened graphs with a maximum number of type A$_n$ thickened junctions. Such graphs are usually made of several distinct circular strands. In order to determine the start configuration, first assume that every vertex on the polyhedron is represented by a junction of type A$_n$. This is always possible if the polyhedral edges have an even number of helical half-turns. The start configuration in this case is given by $N$ separate circular strands (loops), where $N$ is the number of polyhedral faces.

However, in the presence of cross-overs which take into account the odd number of half-turns along the edges, this distribution of type A$_n$ junctions does not necessarily provide an orientable thickened graph. In particular, this is the case if faces with an odd number of edges occur in the polyhedron (as is the case for all three polyhedra considered here). In particular, if the cages have all edges of the same length as in the present analysis, and moreover, exhibit an odd number of half-turns in the double helix along each edge, the start configuration is obtained as follows: The two-dimensional surface, orientable or not, obtained by gluing the twisted strips representing the cross-overs to type A$_n$ junctions, is called the *initial data configuration* (see Fig.  for the initial data configuration of the icosahedron; note that this configuration has 6 loops, but they do not all run in opposite directions).

To this initial start configuration, we apply the *bead rule* to obtain a start configuration. The bead rule consists of placing beads on selected edges of the polyhedron to indicate that a twisted strip (cross-over) is glued to a twisted leg of a type B$_{kn}$ thickened junction, as illustrated in Fig.  for a 4-coordinated polyhedron.

Note that by addition of a "cross-over" at one vertex the orientation of the boundary components change at all neighbouring edges. For polyhedra with an odd number of half-turns on their edges a start configuration is obtained if the following rules are satisfied:

-   Each edge accommodates either a cross-over or a bead.

-   Every face of the polyhedron in the start configuration must have an *even* number of cross-overs.

-   The number of beads is minimal.

After the bead rule has been applied, one must discard configurations which are equivalent under icosahedral symmetry. The symmetry-inequivalent bead configurations correspond to the possible start configurations. In such configurations some junctions are of type A$_n$, and others of type B$_{kn}$, the latter having been introduced to provide orientability of the two-dimensional surface representing the thickened graph obtained from the polyhedron.

## The construction procedure: Step II

Start configurations are usually given in terms of more than one strand of DNA. Since focus is here on building cages out of a minimal number of strands, these need to be "merged". This can be achieved by replacing some of the junctions in the start configuration by junctions with different connectivity of the strands. Since such replacements result in a change of the overall number of strands used to form the cage and therefore depend on the overall structure, which is different in all three cases considered here, these replacements are discussed on a case by case basis in the following sections.

Note that these replacements result in a change of the number of half-turns of the DNA strands, because they add or remove a half-turn in the duplex structure . If the edges of the polyhedral cage are of equal length and are rigid duplexes, i.e., have hybridization going up to the vertices, then there is little flexibility of the strands at the vertices to cross over from one edge to another, and we assume that such replacements of the vertex structure are difficult. The replacements of one type of junction with another are therefore carried out under the assumption that the strands at the vertices of the cage are not completely hybridized or the lengths of the edges may differ by a few nucleotides. In these cases, the changes in the connection of the strands introduced by these replacements do not interfere with the embedding of the double helical structure.

# The icosahedral cage

The rules described above can be applied in a straightforward manner to the construction of icosahedral cages, and we start by considering the case where an extra twist occurs on each edge, i.e., the edges are of equal length with an odd number of half-turns. An icosahedron is a five-coordinated polyhedron with twenty triangular faces. According to the preceding section, orientability requires an odd number of beads per face, and the minimum number of 10 beads is achieved with exactly one bead per face.

Compiling an exhaustive list of start configurations in this case is effortless. For instance, the bead rule can be implemented by first considering one of the 12 five-coordinated vertices of the icosahedron, and asking how many allowed possibilities there are to place beads on the triangular faces forming a pentagon whose centre is the vertex in question.

Direct inspection shows that there are three inequivalent ways to obtain a minimal bead distribution, all illustrated in Fig. . The 3, 4 and 5-bead configurations are labelled $(p,r) = (1,2), (3, 1)$ and $(5,0)$ respectively, where $p$ is the number of beads on the pentagon perimetre, and $r$ is the number of beads on the radii.

The following strategy provides a full set of start configurations (all letters used for vertices refer to the labelling convention shown in Fig. (a)):

1.  Place a $(5,0)$ pentagon configuration at any vertex (choose A, say, and label this configuration $(5, 0)_A$). Due to the high symmetry of this bead distribution, the only bead distribution for pentagons with center vertices B,C,D,E and F is the 3-bead one, as the corresponding vertices automatically have at least two beads on radii. There is no allowed bead distribution with more than two beads on radii, so $(1,2)$ is the only possibility. A similar argument holds for the vertices G, H, I, J and K, which must be of type $(1, 2)$. Finally, vertex L is automatically of type $(5,0)$. The essential features of this icosahedral bead configuration are encoded in Fig. (b), where each vertex is labelled by the number of beads placed on its incident edges. This start configuration (without further replacements of $5$-junctions) requires four DNA single strands.

2.  Place a $(3,1)$ pentagon configuration at any vertex in any orientation. Choose vertex A, say, and place this $(3,1)_A$ configuration such that the edge linking vertices A and B hosts a bead. This fixes vertices D, E and J to have $(1, 2)$ configurations. Vertex B can be in a $(3, 1)$ or a $(1, 2)$ configuration. The former yields that that the minimal number bead distribution makes the corresponding thickened graph not orientable, as one triangular face acquires two beads. The configuration $(1, 2)$ can be placed on vertex B in two orientations: the 'left' orientation $(1, 2)_{\ell, B}$ shows a bead on the edge linking B and G, while the 'right' orientation $(1, 2)_{r, B}$ shows a bead on the edge linking B and H. Note that B lies on the axis connecting A, B and J and a bead on the edge connecting B and G is a reflection of the bead placed on the edge connecting B and H, therefore $(1, 2)_{l,B}$ and $(1, 2)_{r,B}$ yield equivalent bead configurations (see Fig. ).

3.  Place a $(1, 2)$ pentagon configuration at any vertex in any orientation. Choose vertex A, say, and place this $(1, 2)_A$ so that the edge linking vertices D and E hosts a bead. Then vertex B must be in a $(5, 0)$ or a $(3, 1)$ configuration since two edges on the corresponding pentagon perimetre host beads. If B is of type $(5, 0)$, one arrives at the same configuration as discussed in case 1 (Fig.  (b)). If B is $(3, 1)$, the configuration can be placed in two orientations: $(3, 1)_{\ell, B}$ if the edge linking B and G hosts a bead, and $(3, 1)_{r, B}$ if the edge linking B and H hosts a bead. Once the orientation is chosen, there is one more choice to make. In the 'left' configuration, vertices C and D are automatically in $(1, 2)$ configurations but vertex F can be of type $(1, 2)$ or $(3, 1)$. Each choice leads to a start configuration which requires two DNA strands. In the 'right' configuration, vertices E and F are automatically in $(1, 2)$ configurations but vertex C can be of type $(1, 2)$ or $(3, 1)$. Each choice once again leads to a start configuration which requires two DNA strands. These configurations are shown in Fig.  and Fig. . Note that two of these four start configurations are mirror images of the other two.

These considerations are summarized in Table .

```latex
\begin{tabular}{|c|c|c|c|c|c|c|c|c|} \hline
\multicolumn{9}{|c|}{{\em Systematics of minimal bead configurations}}\\ \hline
$(5, 0)_A$ & \multicolumn{3}{c| }{ $(3, 1)_A$} & \multicolumn{5}{c| } {$(1, 2)_A$}\\  \cline{5-9}
$(1, 2)_B$ &  \multicolumn{3}{c| }{ $(1, 2)_D$}&$(5, 0)_B$&\multicolumn{2}{c| }{ $(3, 1)_{\ell, B}$} &\multicolumn{2}{c| }{ $(3, 1)_{r, B}$} \\
$(1, 2)_C$ &  \multicolumn{3}{c| }{ $(1, 2)_E$}&$(1, 2)_C$&\multicolumn{2}{c| }{ $(1, 2)_C$} &\multicolumn{2}{c| }{ $(1, 2)_E$} \\
$(1, 2)_D$ &  \multicolumn{3}{c| }{ $(1, 2)_J$}&$(1, 2)_D$&\multicolumn{2}{c| }{ $(1, 2)_D$} &\multicolumn{2}{c| }{ $(1, 2)_F$} \\ \cline{2-9}
$(1, 2)_E$ &$(3, 1)_B$&$(1, 2)_{\ell, B}$&$(1, 2)_{r, B}$&$(1, 2)_E$&$(1, 2)_F$&$(3, 1)_F$&$(1, 2)_L$&$(3, 1)_C$\\
$(1, 2)_F$ & X &$(1, 2)_C$&$(3, 1)_C$&$(1, 2)_F$&$(3, 1)_E$&$(1, 2)_E$&$(3, 1)_D$&$(1, 2)_D$\\
$(1, 2)_G$ & X &$(3, 1)_F$&$(1, 2)_F$&$(1, 2)_G$&$(1, 2)_G$&$(1, 2)_G$&$(3, 1)_G$&$(1, 2)_G$\\
$(1, 2)_H$ & X &$(1, 2)_G$&$(1, 2)_G$&$(1, 2)_H$&$(3, 1)_H$&$(1, 2)_H$&$(1, 2)_H$&$(1, 2)_H$\\
$(1, 2)_I$ & X &$(1, 2)_H$&$(1, 2)_H$&$(1, 2)_I$&$(1, 2)_I$&$(3, 1)_I$&$(1, 2)_I$&$(1, 2)_I$\\
$(1, 2)_J$ & X &$(3, 1)_I$&$(1, 2)_I$&$(5, 0)_J$&$(3, 1)_J$&$(3, 1)_J$&$(3, 1)_J$&$(3, 1)_J$\\
$(1, 2)_K$ & X &$(1, 2)_K$&$(3, 1)_K$&$(1, 2)_K$&$(1, 2)_K$&$(1, 2)_K$&$(1, 2)_K$&$(3, 1)_K$\\
$(5, 0)_L$ & X &$(3, 1)_L$&$(3, 1)_L$&$(1, 2)_L$&$(1, 2)_L$&$(1, 2)_L$&$(1, 2)_L$&$(1, 2)_L$\\
\hline
1&(2)&3&4&5&6&7&8&9\\
{ 4 strds}& & { 2 strds} &{ 2 strds}&{ 4 strds}& {2 strds} &{ 2 strds}& { 2 strds} &{ 2 strds}\\
\hline
\end{tabular}
```

The result is that, up to mirror symmetry, one has four inequivalent classes. One admits four strands (configurations 1 and 5 are equivalent under a rotation of the icosahedron), and the other three require two strands. Configurations 3 and 4 are mirror-symmetric, as are configurations 6 and 8, as well as 7 and 9.

The final step in the construction of such cages is to merge the different strands by using new types of junctions. The start configuration 1 corresponds to the four-strand cage of Fig. (a).

The figure shows that the two vertices (A and L, compare with the labelling in Fig. (a)) host thickened junctions of type A$_5$, containing five strands. All other vertices have thickened junctions of type $B_{2,5}$ , i.e., 5-junctions with two cross-overs of the strands at the vertices. These junctions involve three different strands, and via a replacement of strand connections as depicted in Fig.  these strands can be merged into one.

If such a replacement takes place at vertex B say (replacing the junction of type $B_{2,5}$), then the resulting cage is made of two distinct strands. The results in show that the parity of the boundary components in every thickened graph of a given polyhedron is always the same. Therefore, two strands are the least number of strands that can be used in the construction of an icosahedron.

For the case when an icosahedron has an edge length with an even number of helical half-turns, a start configuration is given by 20 strands of DNA tracing the polyhedral faces. These can be reduced to two strands in a number of ways. One possibility is to use the replacements shown in Fig. , which result in (a) a reduction of five distinct strands meeting at a vertex to one, and (b) a reduction of three different loops to one.

Via three replacements of type (a) and two of type (b) one obtains a two-strand realisation of the cage.

# The dodecahedral cage

Next we consider the case of the dodecahedron, a three-coordinated polyhedron with 12 pentagonal faces. We start with the case where the edge length of the dodecahedral cage is such that the DNA duplex has an odd number of half-turns along each edge, and we encode this additional helical turn via a red cross-over on a dodecahedral cage in planar projection (see Fig. (a)).

Since the resulting number of twists along a pentagonal face is odd, the surface corresponding to the ribbon representing the double helical structure of the DNA is non-orientable. We follow the procedure outlined in Section to produce orientable embeddings that may serve as start configurations for our analysis. In particular, we replace $A_3$ junctions by $B_{1,3}$ junctions in order to compensate for the twist, and determine their locations via the bead rule.

As there are twelve faces in the dodecahedron, the minimal number of beads needed to provide an orientable embedding of the whole double helical structure is six. There are several non-equivalent ways to introduce six beads by tessellating the surface of the dodecahedron with the tile shown in Fig. (b). The inequivalent possibilities of placing beads on the edges of the dodecahedron such that precisely one edge per pentagonal face is decorated by a bead are obtained as follows. Without loss of generality, we place the first bead on any of the edges, because they are all equivalent by symmetry. We label this edge, and hence the bead on this edge, as 1; see Fig. .

To keep the number of beads at a minimum, no other edge of the two pentagons labelled A and B can have a bead.

Since the order in which pentagons are considered does not matter, we can next concentrate on the pentagon labelled C. Edges labelled 2 and 4 are equivalent by symmetry and hence lead to equivalent configurations. We therefore consider only the possibility of placing a bead on edge 2 or edge 3, and treat each case separately. A bead on edge 2 'covers' pentagons C and D in Fig. . Since every pentagonal face has to have a bead on one of its edges, we can, without loss of generality, look at pentagon E next. Due to the beads already present there is no symmetry, and thus all edges 5 to 8 lead to different solutions. There are 10 solutions in total, denoted S1 to S10 in Fig. .

We next concentrate on edge 3 instead of edge 2, hence pentagons A, B, C and I contain an edge with a bead. Up to symmetry arguments, there are three different ways to complete this analysis based on the bead rule, labelled S11 to S13 in Fig. .

We investigate whether any of these 13 solutions can be symmetrically mapped onto another. To do so, we observe that different bead configurations can be classified according to different *bead paths*: We call any path along edges of the decorated dodecahedron a bead path if, starting and ending on a beaded edge, it alternates along non-beaded and beaded edges. The four possible scenarios of bead paths are summarised in Table .

| *Systematics of bead paths* |  |
|:--:|:---|
| Type I | all beads lie on a closed (cyclic) bead path |
|  | (corresponding to solutions S1, S2, S4, S9) |
| Type II | all beads lie on a non-cyclic (open) bead path |
|  | (corresponding to solutions S3, S5, S7, S8, S10, S12) |
| Type III | all beads lie on three disjoint bead paths with two beads each |
|  | (corresponding to solutions S6, S13) |
| Type IV | all beads lie on six disjoint bead paths one edge in length |
|  | (corresponding to solution S11) |

*Summary of possible bead paths for the identification of the symmetry-inequivalent start configurations.*

Bead placements corresponding to a given type (I–IV) are inequivalent to arrangements of any other type, because any symmetry would map a path onto another path, and a cycle onto another cycle. Hence an equivalence may appear only within a given type. We start with type I. Solutions S1 and S9 are mirror images of each other (obtained by turning the sphere inside out, i.e., they have different helical structures but are otherwise identical). We therefore consider them as equivalent. Solutions S2 and S4 can be considered equivalent as they correspond to the closed path that divides the 12 pentagons of the dodecahedron into two identical sets of six. However, S1 and S2 are not related by symmetry, and there are hence two distinct solutions of type I, represented by S1 and S2. Solutions S3, S5 and S8 represent mutually symmetric open paths, while S7, S10 and S12 represent mutually symmetric paths, but of opposite orientation to S3, S5, and S8 (again obtained by turning the sphere inside out). Hence, we can consider all these options as equivalent, and so there is only one (up to symmetry) arrangement of beads producing a non-cyclic path, represented by S3. The two arrangements of type III corresponding to solutions S6 and S13 can be mapped one onto another. So there is only one solution of this type, represented, say by S6. Finally, S11 is the only solution of type IV. It corresponds to an equidistant distribution of beads on edges and is in that sense the 'most symmetric' solution.

Every bead configuration translates into a start configuration. The five inequivalent start configurations are depicted in Fig. .

Each of these DNA embeddings has 14 vertex configurations of type $A_3$ and 6 vertex configurations of type $B_{1,3}$. The DNA embeddings corresponding to solutions S2 and S11 result in six distinct strands (Fig. (d,e)), the embeddings corresponding to solutions S1 and S3 in four strands (Fig. (b,c)), while the embedding corresponding to solution S6 in two strands (Fig. (a)).

In order to realise the cage in terms of a minimal number of DNA strands, the configurations in Fig. (b–d) require further vertex replacements. Note that as in the case of the icosahedron, (by the results in ) two is the least number of strands that can be used to obtain the dodecahedral cage. Indeed, if a vertex configuration of type $A_3$ obtained by hybridizing three separate strands is replaced by a vertex configuration of a branch point of type $B_{1,3}$, then the number of strands meeting at the vertex reduces to one according to . In the case of six separate strands (i.e., the cases of S2 and S11) we need to choose two vertices for replacement, while for the case of four separate strands (i.e., the cases of S1 and S3) only one replacement is needed. The reader is referred to for details. In all cases, the minimal number of strands to assemble a dodecahedron using only junctions of type $A_3$ and $B_{1,3}$ is two. Fig. (a) shows the two-strand configuration corresponding to solution S6.

We now consider the case of double-stranded DNA embeddings into dodecahedral cages with *an even number of half-turns per edge*. In these cases, there are no additional twists on the edges. A start configuration is therefore easily obtained by placing 12 separate strands into the 12 faces of the dodecahedron. Via $n=5$ replacements, one again obtains a DNA embedding with two separate strands. According to the remark at the end of Section (see also details in ), these replacements can be freely chosen if flexible strand connections are used, and require particular positions in the non-flexible case.

# The icosidodecahedral cage

We now consider the icosidodecahedron, a 32-faced polyhedron with quatro-valent vertices. We again start with the case where all edges accommodate an odd number of helical half-turns, which implies additional cross-overs on the edges in the initial configuration. According to Section , the bead rule has to be applied in order to obtain a start configuration. The minimal number of beads required is easily calculated. All faces of the icosidodecahedron have an odd number of edges: there are 12 pentagons and 20 triangles. However, each face must have an even number of cross-overs to keep the orientability. Therefore, each of the 20 triangles must receive at least one bead. However, by placing a bead on each triangle, one also places at least one bead on each pentagon. The distribution of this minimum number of 20 beads should be such that pentagons receive an odd number of beads. Let $\alpha$ be the number of pentagons receiving one bead, $\beta$ be the number of pentagons receiving three beads and $\gamma$ be the number of pentagons receiving five beads. Given that there are 12 pentagons in total, we must satisfy the two equations $$\begin{array}{rcl}
\alpha+3\beta+5\gamma&=&20,\nonumber\\
\alpha+\beta+\gamma&=&12,
\end{array}$$ with $\alpha, \beta$ and $\gamma$ positive integers. There are three solutions to the problem, namely $$\label{3cases}
\begin{array}{rcl}
&&{\rm Case\, I}\qquad\qquad \,\,\,\,\alpha=8, \, \beta=4,\, \gamma=0\nonumber\\
&&{\rm Case\, II}\qquad \qquad\,\,\alpha=9, \, \beta=2,  \, \gamma=1\nonumber\\
&&{\rm Case\, III}\qquad \qquad\alpha=10,  \, \beta=0,  \, \gamma=2.
\end{array}$$ We start by considering case I. This tells us that the bead rule is fulfilled if there are four pentagons with three beads each, and if every triangle has precisely one bead. We therefore determine all symmetry-inequivalent start configurations with that property. Since this is a significant combinatorial task for the polyhedron under consideration, our analysis is computer-assisted.

In a first instance, we use the icosahedral symmetry to reduce the number of options to be considered. In particular, we determine all symmetry-inequivalent distributions of four pentagonal faces on the icosidodecahedron. Each of these four faces has three of its edges decorated by one bead, whilst all other pentagons and all triangles have only one bead on their perimeter. In order to determine all inequivalent configurations of four pentagons, we consider the equivalent problem of finding all different possibilities of colouring four of the 12 vertices of an icosahedron.

There are 9 inequivalent such configurations for case I, which we call the *partial start configurations*. We show them schematically in a projective view of the icosahedron in Fig. .

We next determine the inequivalent bead configurations for each partial start configuration. Each partial start configuration encodes several possible cage scenarios which correspond to all inequivalent ways of placing three beads on the edges of four distinguished pentagons, and one bead on one of the edges of all other faces (pentagonal or triagonal). We carry out this combinatorial task computationally via a computer programme that tests, for each start configuration, which combinations of beads are possible, given the fact that the four distinguished pentagonal faces each have three beads, while all others have one. The results for all three cases are summarised in Table .

| Loop number | Case I | Case II | Case III |
|:-----------:|:------:|:-------:|:--------:|
|     10      |   X    |         |          |
|     12      |   X    |         |          |
|     14      |   X    |    X    |    X     |
|     16      |        |         |    X     |

*The distribution of 10-, 12-, 14- and 16-loop configurations among the three different cases. A cross indicating the occurrence of this type of loop configuration for a given case.*

The computations show that the smallest number of distinct loops is 10 and the largest number 16. While there are over $10^4$ distinct 10-loop configurations, there is only one configuration with 16 loops, occurring in case III. All vertex configurations involve either four, three, or two distinct loops. As before, these can at best be reduced to two-strand configurations because the number of distinct circular DNA strands is even in all cases. Such a reduction is possible via the replacements of type I and type II shown in Fig. .

If the edge length of the icosidodecahedron is such that none of the edges has an additional cross-over, the start configuration consists of 32 loops corresponding to the 32 faces of the polyhedron. Via 30 replacements of type I or type II, a duplex cage structure realised by two circular DNA molecules is obtained.

# Discussion

We have performed a theoretical analysis of different types of icosahedral cages with minimal (two) DNA strands such that each edge is in a duplex structure. Focus was placed on the three icosahedrally symmetric polyhedra with the smallest number of vertices, the icosahedron with 12 vertices at the 5-fold axes of icosahedral symmetry, the dodecahedron with 20 vertices at the 3-fold axes, and the icosidodecahedron with 30 vertices at the 2-fold axes. These polyhedra are distinguished by the fact that they have uniform edge lengths and may therefore be easier to manufacture than other polyhedra with icosahedral symmetry.

We remark that polyhedral RNA cages have been observed also within the protein containers, called viral capsids, that encapsulate and hence provide protection for the viral genome . However, the RNA has to be unknotted for successful replication and therefore these cages usually do not appear in the form we have described here and are realised by the viral RNA in a different way .

A comparative analysis reveals interesting features of the cage structures. In particular, for each of them the minimal number of circular molecules needed to construct the cage is two. The icosidodecahedral cage is distinguished by the fact that its volume per surface ratio is the largest among the polyhedra considered here (and is also larger than those of the cages that have been realised experimentally to date). It may therefore be more suitable for nanotechnology applications in which the cages serve as containers for storage or the transport of a cargo.

We hope that the blueprints for the organisation of the cages with a non-crystallographic symmetry suggested here may assist in their experimental realisation. In particular, these blueprints suggest the structures of the junction molecules that may be used as basic building blocks for the self-assembly of those cages along the lines of .

# Acknowledgements

AT has been supported by an EPSRC Springboard Fellowship, and RT by an EPSRC Advanced Research Fellowship. Some of this work has been carried out at the University of South Florida, and both AT and RT would like to thank the Department of Mathematics for their hospitality. NJ has been supported in part by the NSF grants CCF $\#$`<!-- -->`{=html}0523928 and CCF $\#$`<!-- -->`{=html}0726396.

[^1]: E-mail: `email@example.com`

[^2]: E-mail: `firstname.lastname@example.com`

[^3]: E-mail: `firstname.lastname@example.com`

[^4]: Moreover, the volume to area ratios of all icosahedral cages considered here are larger than those of the crystallographic cages realised to date. For comparison, the value for the octahedron is approximately 0.14.

[^5]: Stable four-junctions can be assembled, see for example .


abstract: While it is known that $\mathrm{Tr}\rho^n$ can be measured directly (i.e., without first reconstructing the density matrix) by performing joint measurements on $n$ copies of the same state $\rho$, it is shown here that random measurements on single copies suffice, too. Averaging over the random measurements directly yields estimates of $\mathrm{Tr}\rho^n$, even when it is not known what measurements were actually performed (so that $\rho$ cannot be reconstructed).
author: S.J. van Enk$^{1,2}$ and C.W.J. Beenakker$^3$
bibliography: RandomU.bib
title: The power of random measurements: measuring $\mathrm{Tr}\rho^n$ on single copies of $\rho$

The standard textbook quantum measurement of an observable $\hat{O}$ on a given quantum system produces an estimate of the expectation value $\mathrm{Tr}(\rho\hat{O})$, where $\rho$ is the density matrix of the system. This expectation value is linear in $\rho$. As is well-known by now , *non*linear functions of the density matrix $\rho$, such as the purity $p_2=\mathrm{Tr}\rho^2$ and its cousins $p_n=\mathrm{Tr}\rho^n$ for $n>2$, can be measured directly, too, without first having to reconstruct the whole density matrix. For this direct measurement method to work one needs $n$ quantum systems that are all in the same state $\rho$, plus the ability to perform the appropriate joint measurement(s) on those multiple copies.

Here we point out that estimates of the same nonlinear quantities can be obtained from *random* measurements on *single* copies as well. A random measurement can be assumed to be implemented by performing a random unitary rotation on the single copy (possibly including an ancilla which starts off in a standard state), followed by a fixed measurement on the single copy (and possibly on the ancilla). By averaging the measurement results over the random unitaries, one can directly infer estimates of $\mathrm{Tr}\rho^n$ \[with $n=2\ldots M$, with $M$ the Hilbert space dimension of the system of interest\], without having to reconstruct the density matrix. One point of the averaging procedure is that one does not have to know which random unitaries were in fact applied, and as a consequence one cannot reconstruct the density matrix in that case. An example of a random measurement is furnished by intensity measurements of speckle patterns resulting from light (be it two photons, or a single photon, or a coherent laser beam) propagating through a disordered medium , and in that case the purity $p_2$ can (and was indeed) inferred directly from those measurements (see also [^1]).

There is an important difference between the known direct method and the current random method in what quantity exactly is estimated. Suppose one's source does not produce the same state every single time, but instead a state $\rho_j$ at try $j$. In this case standard quantum measurements of a given observable on $J$ instances $j=1\ldots J$ can still be described by a single density matrix, namely, the mean $\bar{\rho}=\sum_j \rho_j/J$. Since the random method only involves measurements on single copies, it produces, likewise, an estimate of $\mathrm{Tr}(\overline{\rho}^n)$. This requires no assumption about the quantum systems being uncorrelated or unentangled with each other, since $\rho_j$ is obtained by tracing out all degrees of freedom except those of system $j$.

On the other hand, a direct measurement would yield an estimate of $\mathrm{Tr}(\hat{S}\overline{\rho_{j,j+1\ldots, j+n-1}})$ instead, where $\rho_{j,j+1\ldots j+n-1}$ is the joint density matrix of $n$ systems $j,j+1,\ldots, j+n-1$, and $\hat{S}$ is the cyclical shift operator, which acts on the basis states of the $n$ quantum systems as $\hat{S}|\psi_j\rangle|\psi_{j+1}\rangle\ldots|\psi_{j+n-1}\rangle=
|\psi_{j+1}\rangle|\psi_{j+2}\rangle\ldots|\psi_{j}\rangle.$ It is only under the assumption that the states of the $n$ systems are identical and independent (*i.i.*) that the direct measurement yields $\mathrm{Tr}\rho^n$. In fact, the direct measurement is eminently suited for detecting that the states are *not* identical . Although the assumption of *i.i.* states is standard, it is only recently that precise conditions have been stated under which the approximate *i.i.* character can be inferred . The required permutation invariance is easily enforced when performing measurements on single copies, but not when performing joint measurements on multiple copies . Avoiding this difficulty is the main advantage of the random method.

An $N\times N$ random unitary matrix, distributed according to the Haar measure, can be easily constructed by the method presented in . One first constructs a matrix whose elements are independent complex Gaussian variables, and one then performs an orthogonalization of the resulting random matrix (where one small pitfall needs to be avoided ). We first consider approximate results for random unitaries, because the resulting expressions are quite simple, and subsequently we will give the more involved exact results.

If we consider an arbitrary submatrix $V$ (of size $M$) of $U$ (of size $N$), with $M\ll N$ , then the real and imaginary parts of its matrix elements can still be very well approximated by independent and normally distributed numbers if $N$ is large. With this Gaussian approximation we can compute the following *averages* (we indicate averages over the distribution of random unitaries by $\left\langle.\right\rangle$): first, we have $$\begin{aligned}
\left\langle V_{kl}V^*_{mn}\right\rangle=
\delta_{km}\delta_{ln}/N.
\end{aligned}$$ Here and in all of the following we assume we have picked some basis $\{|k\rangle\}$, and we write all matrix elements w.r.t. that basis. The normalization factor $1/N$ follows immediately from the fact that $U$, of which $V$ is a submatrix, is unitary, so that $\sum_{l=1}^{N}U_{kl}U^*_{ml}=\delta_{km}$. Higher-order averages follow from the Isserlis ("Gaussian-moments") theorem . In particular, the only nonzero averages arise from products of $2K$ factors of the form $$\begin{aligned}
\left\langle V_{k_1l_1}\ldots V_{k_K l_K}V^*_{m_1n_1}\ldots V^*_{m_K n_K}\right\rangle=
\nonumber\\\frac{
\sum_{{\rm all\,pairs}(i,j)} \delta_{k_i m_j}\delta_{l_i n_j}}{N^K}.\label{Uaver}
\end{aligned}$$ We now apply the preceding approximate results to the following scenario. Consider an "input" density matrix $\rho^{{\rm in}}$ of size $M\times M$. Embed the system in a larger Hilbert space of size $N$, by constructing a new $N\times N$ density matrix by adding zero matrix elements. Then apply a random unitary $U$ to the larger matrix. Finally, consider measurements in a fixed $M$-dimensional (sub)basis $\{\left| k \right\rangle\}$. The probability $\mathrm {Prob}(k)$ of finding measurement outcome $k$ is given by $$\mathrm {Prob}(k)=\sum_{m,n}\rho^{{\rm in}}_{mn}V_{mk} V^*_{nk}.$$ This expectation value depends on what $V$ is, of course, but its average is given simply by $$\left\langle\mathrm {Prob}(k)\right\rangle=\sum_{m,n}\rho^{{\rm in}}_{mn}\left\langle V_{mk} V^*_{nk}\right\rangle=1/N,$$ where we used that $\mathrm{Tr}(\rho^{{\rm in}})=\sum_m\rho^{{\rm in}}_{mm}=1$. Defining $P_n(k)=\left\langle\mathrm {Prob}(k)^n\right\rangle$, the following averages are obtained by using the Isserlis theorem (up to order $n=4$; subsequent orders can be easily obtained, too, but for our purposes this will do) $$\label{pk}
\begin{eqnarray}
P_2(k)&=&\left[1+p_2\right]/N^2,\\
P_3(k)&=&\left[1+3p_2+2p_3\right]/N^3,\\
P_4(k)&=&\left[1+3p_2^2+6p_2+8p_3+6p_4\right]/N^4,
\end{eqnarray}$$ where we defined $p_n=\mathrm{Tr}\left((\rho^{{\rm in}})^n\right)$. Inverting these equations gives estimates of $p_n$ in terms of the measurable quantities on the left-hand sides. We denote those estimates by an overbar, e.g., $\bar{p}_2=N^2P_2(k)-1$. We refrain from giving the other inverse relations now, as we will give the *exact* relations below in ().

We can also compute standard deviations in the (mean) estimates. For example, assuming we average the results for one value of $k$ over $N_{{\rm rand}}$ random unitaries, then the statistical error in the estimate of the purity is $$\sqrt{N_{{\rm rand}}-1}\Delta(\bar{p}_2)=
\sqrt{4p_2+2p_2^2+8p_3+6p_4}.\label{Delta}$$ This is an increasing function of $p_2, p_3, p_4$, so that the variance is largest for a pure state and smallest for the totally mixed state $\rho^{{\rm in}}=\openone/M$.

In an actual experiment one may not know exactly what the values of $N$ and/or $M$ are (for instance, this is the case in the speckle experiments of Refs. ). In such a case $N$ can be directly estimated from $P_1(k)$ through $N= 1/P_1(k)$. So, we would use $$\begin{aligned}
\tilde{p}_2&=&\frac{P_2(k)}{P_1(k)^2}-1,\label{pknum}
\end{aligned}$$ instead (such estimates we indicate by a tilde). Now this estimate $\tilde{p}_2$ has a *smaller* variance than $\bar{p}_2$ has, simply because the errors in $P_1(k)$ and $P_2(k)$ are positively correlated. It is, therefore, better to use $\tilde{p}_2$ as estimate for $p_2$, even when $N$ is in principle known. The numerical results given below will confirm this, also for the exact result for $\tilde{p}_2$. For the estimates $\tilde{p}_3$ and $\tilde{p}_4$, however, there is not much difference between the two methods.

When $N$ is not very large, equations () and hence () are not correct. The exact results, which can be extracted from Refs.  and , are still given by () upon multiplication of $P_n(k)$ by the correction factor $C_n$, where $$\begin{aligned}
C_n&=&(1+1/N)(1+2/N)\ldots(1+(n-1)/N).\label{corr}
\end{aligned}$$ Note that these factors depend only on $N$, not on $M$, and the results are valid even when $M=N$. This then leads to the inverse formulas: $$\label{pkall}
\begin{eqnarray}
\bar{p}_2&=&D_2 P_2(k)-1,\\
\bar{p}_3&=&\tfrac{1}{2}\bigl[D_3 P_3(k)-1-3p_2\bigr],\\
\bar{p}_4&=&\tfrac{1}{6}\bigl[D_4P_4(k)-1-3p_2^2-6p_2-8p_3\bigr],
\end{eqnarray}$$ with $D_n=(N+n-1)!/(N-1)!$ Taking into account the correction factors () leads to different values for the statistical errors in estimates. It is still true that pure states lead to the largest errors; for those we get $$\sqrt{N_{{\rm rand}}-1}\Delta(\bar{p}_2)=
\sqrt{\frac{24(1+1/N)}{(1+2/N)(1+3/N)}-4}.\label{Deltap}$$ The right-hand side (slowly) increases with increasing $N$, from $\sqrt{52/7}$ for $N=4$ to $\sqrt{20}$ for $N\rightarrow\infty$.

In order to illustrate the method and the meanings of $N$ and $M$, we consider the following examples here: **(i)** Suppose we have a single photon occupying one of $M$ input modes. We then apply a random linear optics transformation that involves $N-M$ ancilla modes. The photon now ends up being coherently distributed over $N$ output modes. We then estimate the probability $\mathrm {Prob}(k)$ with which the photon ends up in one of a fixed set of $M$ output modes $k=1\ldots M$. This is an example akin to that considered in .

**(ii)** Suppose our system of interest consists of 2 qubits, so that $M=4$. Suppose we have an ancilla qubit in a fixed state $|0\rangle$, and we apply a random unitary operation to the 3 qubits. In this case, $N=8$. We then perform measurements on each of the three qubits separately in the standard basis. We measure the probability $\mathrm {Prob}(k)$ of the two qubits ending up in one of the $M=4$ combinations $k=00, 01, 10, 11$ *and* the ancilla ending up in $|0\rangle$ (thus measuring only a $M$-dimensional subspace).

**(ii')** There is no need for any ancillas if dealing with a fixed and known number of qubits, say $Q$. In that case, we simply have $N=M=2^Q$. We consider only case (ii') in the following numerical results.

We assume that we run an experiment with a fixed random ("unknown") unitary of size $N$ sufficiently many times that we get a very good estimate of $\mathrm {Prob}(k)$ for each $k$ for the given unitary and the given input state (of size $M$). Subsequently we average over $N_{{\rm rand}}$ random unitaries to obtain $P_n(k)=\left\langle\mathrm {Prob}(k)^n\right\rangle$. From those results we estimate the values of $p_2, p_3, p_4$.

The first example we consider corresponds to case (ii') mentioned above, where we have two qubits. In Figs.  and we plot results for pure input states, where we use the results for just 1 value of $k$ to estimate $p_n$, in two different ways: using the exact value $N=4$ (Fig. ) or using the estimate $N\approx 1/\left\langle\mathrm {Prob}(k)\right\rangle$ (Fig. ). The results show how the latter method is more accurate for estimating purity. The *same* data are used in the two Figures, so that all differences between them are entirely due to the different analysis of those data. This different analysis reduces the statistical variation in $\tilde{p}_2$, but not in $\tilde{p}_3$ and $\tilde{p}_4$. In addition, the plots show that the statistical errors in $\tilde{p}_2, \tilde{p}_3, \tilde{p}_4$ are strongly correlated in the latter case.

In the remaining figures we perform an additional average over the $M$ different values of $k$, leading to smaller (by a factor of about $\sqrt{M}$) error bars.

Performing tomography on two qubits would require 15 independent (and known) measurements. Here we show that with just a moderate overhead one can obtain good estimates of $p_2, p_3, p_4$ for *generic* (i.e. randomly picked [^2]) states.

In Fig.  results are displayed for 200 generic two-qubit states, using $N_{{\rm rand}}=30$.

In Fig.  we show (for five qubits) that the number of random unitaries needed to obtain a fixed-size error bar does not increase with the number of qubits. For $N_{{\rm rand}}=30$ one still obtains good estimates: in fact, the error bars *decrease* (roughly as $1/\sqrt{M}$) when going to more and more qubits, just because the number $M$ of measurement results one can average over increases exponentially with the number of qubits, while the variance () increases only very slowly. This is illustrated for pure multi-qubit states in Fig. . It shows that the statistical error in the estimate of $\mathrm{Tr}\rho^n$ for $n=2,3,4$ first increases with the number of qubits before (at $\geq n$ qubits) it starts to decrease monotonically.

In conclusion then, using the ideas of random matrix theory, we showed that nonlinear functions of the density matrix such as $\mathrm{Tr}\rho^n$ can be directly obtained from appropriately averaged random measurements on single copies. No assumptions are needed on the independence of the copies, nor on their states being identical. This contrasts the random method with so-called direct measurements on $n$ identical copies .

Moreover, one does not need to know which random measurements were actually performed, because the averaging procedure keeps all information about the eigenvalues of $\rho$, which is all that is needed to estimate $\mathrm{Tr}\rho^n$. One does need to verify that the random unitaries have been drawn from the appropriate ensemble. There are two tests one could perform: first of all, the definition of the ensemble is that it is unitarily invariant. This means in our context that all averages $\left\langle\mathrm {Prob}(k)^n\right\rangle$ should be independent of $k$. This is a statistically testable property. In addition, one can apply the random measurements to *known* input states, so that the values of those $k$-independent averages are known.

Importantly, the number of unitaries over which one has to average in order to obtain a fixed error bar in the estimates of $\mathrm{Tr}\rho^n$ scales very favorably with the Hilbert space dimension of one's system: in fact, this number even tends to *decrease*. For two qubits this amounts to needing a small overhead as compared to full quantum-state tomography, but for larger systems (more than, say, four qubits) the random method requires (far) fewer resources than does full quantum-state tomography.

[^1]: A different case is the experiment of M. Munroe *et al.*, Phys. Rev. A **52**, R924 (1995) where diagonal density matrix elements in the photon-number basis (hard to measure directly) were obtained by phase-averaging more straightforward quadrature measurements.

[^2]: As only the eigenvalues of $\rho$ matter, the states were chosen according to a simple distribution, without any significance otherwise: first, $M$ uniformly distributed random numbers $(z_i)$ between 0 and 1 are picked; then $\rho$ is chosen to be the diagonal matrix diag$(z_i^E)/(\sum_i z_i^E)$, with $E$ chosen equal to 2 in Fig.  and $E=8$ in Fig. . These choices were made so as to produce a wider spread of values for $\mathrm{Tr}\rho^n$ than do other more standard ensembles.