Affordance Blending Networks
Hakan Aktas Yukie Nagai Minoru Asada Erhan Oztop Emre Ugur










Affordance Blending Networks
    Hakan Aktas^1, Yukie Nagai^2, Minoru Asada^3,4, Erhan Oztop^3,5, Emre Ugur^1
^1Hakan Aktas and Emre Ugur are with Computer Engineering Department, Bogazici University
        hakanwhitestone@gmail.com
^2 Yukie Nagai is with IRCN, the University of Tokyo
^3Minoru Asada and Erhan Oztop are with SISREC, Osaka University
^4Minoru Asada is also affiliated with the International Professional University of Technology in Osaka ^5Erhan Oztop is also affiliated with Computer Engineering Department, Ozyegin University.


    
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




Affordances, a concept rooted in ecological psychology and pioneered by James J. Gibson, have emerged as a fundamental framework for understanding the dynamic relationship between individuals and their environments. Expanding beyond traditional perceptual and cognitive paradigms, affordances represent the inherent effect and action possibilities that objects offer to the agents within a given context. As a theoretical lens, affordances bridge the gap between effect and action, providing a nuanced understanding of the connections between agents' actions on  entities and the effect of these actions. In this study, we propose a model that unifies object, action and effect into a single latent representation in a common latent space that is shared between all affordances that we call the affordance space. Using this affordance space, our system is able to generate effect trajectories when action and object are given and is able to generate action trajectories when effect trajectories and objects are given. In the experiments, we showed that our model does not learn the behavior of each object but it learns the affordance relations shared by the objects that we call equivalences. In addition to simulated experiments, we showed that our model can be used for direct imitation in real-world cases. We also propose affordances as a base for Cross Embodiment transfer to link the actions of different robots. Finally, we introduce selective loss as a solution that allows valid outputs to be generated for indeterministic model inputs. 









§ INTRODUCTION

Affordances are action possibilities provided by the objects in the environment to embodied agents. In general agents learn affordances through goal-free exploration and interaction with the environment and by observing the effects they create on objects with the available actions. Affordances generally encode the relations between objects, actions, and effects and can be represented by models that predict effects given to objects and actions. In this paper, we use an affordance formalization inspired by <cit.> to represent single and multi-agent affordances and various equivalences. We built the structure of our model based on this representation and analyzed the latent representations it forms in regard to these equivalences.

The contribution of our work is as follows

    
  * A deep LfD model that can learn affordances and form common latent representations between action-effect-object representations based on these affordances and the equivalences between the affordances
    
  * Cross-embodiment learning using agent equivalences between the actions of different actions using the same model 
    
  * Cross-Embodiment transfer of learned affordances
    
  * Direct imitation in real-world between demonstrators and robots using objects' effect trajectories
    
  * Selective Loss, a solution to mitigate the undesired consequences of indeterministic training steps and to obtain valid outputs from indeterministic model inputs




§ RELATED WORK

Learning From Demonstration (LfD) <cit.>, also known as Imitation Learning, has been a topic studied by many in recent years <cit.>. Several different LfD methods have been proposed in various studies. Some were based on statistical modeling and dynamic systems <cit.>, while others used Locally Weighted Regression to learn the environment parameters <cit.>. Locally Weighted Projection Regression <cit.>, Gaussian Mixture Models <cit.> and Hidden Markov Models <cit.> are also utilized to learn motion patterns from demonstrated actions. 

In recent years, neural networks have also been widely used in Learning from Demonstration systems to learn movement primitives using complex data <cit.>. In a previous study, the Conditional Neural Movement Primitives <cit.> was introduced as a deep LfD architecture that can be used to learn complex motion trajectories. This method is based on Conditional Neural Processes <cit.> which has the ability to construct full trajectories using any given points on arbitrary time steps. Because of this capability, in this study, we used the CNMPs as a backbone system for our model.






Cross Embodiment Transfer,
the previous works on cross-embodiment transfer aim to transfer or generalize collected robot motion trajectories to various robots with different embodiments <cit.>. These studies include transfer in and in between domains such as transfer to robots with different morphologies in simulation<cit.>, transfer to different robots in real world<cit.>, from robots in simulation to real robots<cit.>. 

Several methods have been introduced to use cross-embodiment transfer to expedite the training by using information from other agents. When the policy of the source agent is known, several studies showed that Reinforcement learning can be used to fine-tune the value function<cit.>, the visual encoder <cit.>, or the policy <cit.> of the target robot. In a previous study, we also showed that cross-embodiment transfer of learned representations can be used to bootstrap Reinforcement Learning <cit.>. We also proposed a method that can align and transfer the actions of the robots if a common task space can be established between them <cit.>. In this study, we build on that model and propose that affordances can be used as a well-defined common task space between the robots. 




§ METHOD



 §.§ Affordance Formalization

In this paper, we will use an affordance representation similar to the one used in <cit.>. However, we will be using an object-centric approach. We formalize and represent the affordances as the following,

    (effect,(object,action))

Here, we define the object as the entity the robot/agent interacts with and the effect as the resulting change caused by the action when the action is executed on the given object. Action can be any motion the robot/agent executes. Using this representation, we also define several equivalences. When the same action is applied to different objects and the same effect is observed, we state that their affordance for that action and effect are equivalent. We formalize this equivalence as the following,  

    (effect,({[ object1; object2 ]}, action)

We also define the affordance of two actions as equivalent if the effects caused by that action on the given object are the same. We formalize action equivalence as the following,

    (effect,(object,{[ action1; action2 ]} )

When multiple agents/robots are involved, the notation for the affordance representation becomes,

    (effect,(agent,(object,action)))

In this case, the actions of two different agents can also be equivalent and can be formalized differently based on the actions. If the actions executed by the agents are the same, 

    (effect,( {[ agent1; agent2 ]},(object,action ))

An example of this case would be,

    (lifted,( {[ Robot1; Robot2 ]},(cuboid,lift ))

Assuming both Robot1 and Robot2 can lift the given object (cuboid) with their lift action. On the other hand, if the actions performed by the agents are different, the formalization becomes,

    (effect, {[ agent1; agent2 ]    ,(object, [ action1; action2 ]} )

An example of this case would be,

    (moved, {[ Robot1; Robot2 ]    ,(cuboid, [ push; pull ]} )

Assuming Robot1 can move the cuboid object by using a push action and Robot2 can move it by using a pull action.



 §.§ Model

In an earlier work <cit.>, we showed that a multi-channel CNMP  model can be used to find correspondences between robots with different morphologies. In this study, we generalize that approach to object affordances and propose affordance blending networks that can be used to form discrete affordance representations from object representations and continuous effect and action trajectories. 




In our system, the objects, and continuous action and effect trajectories are encoded and learned using CNMPs, and the latent representations of the CNMPs of these trajectories are blended to form a common latent space that we call affordance space.  Let A be the dataset that contains the information related to one affordance, including the action trajectories of multiple agents, the resulting effect trajectory of the object entity, and the representation of the object entity. More formally,


    A_j = ({a_t^R_j,e_t,o,t}_t=0^t=1)_j

where e_t represents the resulting effect at time t, o represents the object/entity information, t represents time and a_t^R_j represents the sensorimotor action information of R_j at time t where R_j = [Agent_1, Agent_2, Agent_3 ... , Agent_k] represents the agents that are related to affordance A_j.  j represents the sample number and 0 <= j <= m where m is the size of the dataset.  0 ≤ t ≤ 1 is a phase variable  that represents the time where t ∈𝐑. Not all agents are related to all affordances since it is not possible to talk about grasp affordance if the agent has no means of grasping such as a generic differential drive mobile robot. Since the goal is to convert  the data to latent representations and blend them to get an affordance representation, first, the action trajectories of all the agents that are related are converted to latent representations as in <cit.>,

    L_i^a_r_j = E^a_r_j((t_i,a_t_i^r_j)|θ^a_r_j)    r_j ∈ R_j

where E^a_r_j is an action deep encoder for agent r_j with weights θ^a_r_j, and L_i^a_r_j is the action latent representation of agent r_j constructed using the given observation. Then these representations are averaged for each agent, 

    L^a_r_j = 1/n∑_i^n L_i^a_r_j   r_j ∈ R_j

where n is the number of sampled observations during this iteration. These are then combined using convex combination to obtain the common latent action representation,

    L^a = ∑_r_j^R_j p^r_j L^a_r_j   0≤ p^r_j≤ 1, ∑ p^r_j = 1, r_j ∈ R_j

Object and effect latent representations are obtained similarly to L_i^a_r_j,

    L_i^e = E^e((t_i,e_t_i)|θ^e)    L^e = 1/n∑_i^n L_i^e


    L_i^o = E^o((t_i,o_t_i)|θ^o)    L^o = 1/n∑_i^n L_i^o

where E^e and E^o  are  effect and object deep encoders respectively. The obtained latent action, effect, and object representations are then combined using convex combination,

    L^F = p^a L^a + p^e L^e + p^o L^o     0≤ p ≤ 1, p^a+p^e+p^o = 1

where L^F represents the affordance tuple that can be used to reconstruct the trajectories used to form it. After all representations are merged into one, this representation is decoded to obtain target distributions on t_target for the action of all agents, object and effect: 

    (μ_t_target^a_r_j,σ_t_target^a_r_j ) = Q^a_r_j((L^F,t_target)|ϕ^a_r_j)    r_j ∈ R_j


    (μ_t_target^e,σ_t_target^e ) = Q^e((L^F,t_target)|ϕ^e)


    (μ_t_target^o,σ_t_target^o ) = Q^e((L^F,t_target)|ϕ^o)


where Q^r is a deep decoder with weights ϕ^r that constructs distributions with mean μ_t_target^r and variance σ_t_target^r for the robot r. The learning goal of our system is to obtain more accurate distributions for given condition point(s) and the target point, and therefore the loss is defined similarly to <cit.>:

    ℒ(θ, δ) = - log P(y_j | μ_j, softmax(σ_j))

The complete model can be seen in Figure <ref>.

Our system can also be used to transfer affordance relations demonstrated only using one of the agents to other agents. After sufficient training with affordances known to all agents, the demonstration of the affordance known only by one agent can also be used for training. The system should continue to be trained using the old data alongside the new one to prevent Catastrophic Forgetting. After sufficient training with the new data, the system can also output the corresponding action of the other agent and can output the effect of that action when conditioned with the new object, if the affordance is shared.



 §.§ Selective Loss

In addition, we also introduce a modification to the system's loss function, namely the negative log-likelihood function. As stated, the system uses stochastic convex combination to combine the representations of the actions, effects, and objects. This introduces a problem if more than 2 objects with similar behavior are used (such as a cuboid and upright cylinder). If the weight of the object representation is too small, the resulting affordance representation would contain too little information about the object which makes the prediction about the object in the output non-deterministic. While this does not matter in the test time since it is not possible to deterministically infer the object given the action effect couple if more than one object with similar behavior is used, during training this causes spikes in loss of the system. This occurs when the following all happen,

    
  * the system is biased to output one of the objects over the others for the affordance that corresponds to the action-effect couple given during that training iteration
    
  * the weight of the object representation is too small to affect the affordance representation during that training iteration
    
  * there are multiple objects that have the behavior that corresponds to the action-effect couple given as input during that training iteration
    
  * the desired object output during that iteration is different from the object that the system is biased to output

In this case, the system will output the biased object as output while the desired object output is different. This behavior is not desired and hurts training performance. That's why we introduce a modification to the loss of the image output layer.
We call the new loss function selective loss. It is formally defined as the following,

    ℒ_selective = arg  _x minℒ_x(θ, δ)

Simply, the loss is calculated for all possible outputs and the minimum is chosen as the loss to use during that iteration. This also biases the system to output one of the objects when only action and effect representations are used to construct the affordance representation. This way, the system outputs one of the possible objects that have that affordance rather than outputting a combination of the depth images of all the objects that share that affordance which does not provide any information.

We also believe that this loss function can be used in single-channel (or vanilla) CNMP since the architecture shares the same issue when the following all happens during a training iteration,

    
  * the sampled condition/context point combination is not novel (all shared points are shared between more than one trajectory)
    
  * the selected target point is novel (not shared between the trajectories)
    
  * the system is biased to another trajectory than the desired one for the sampled condition points

In this case, the system predicts the wrong trajectory, and a very high loss value is obtained. In the later stages of the training, this effect significantly increases since the variance values get smaller as training progresses. If the number of non-novel points is too many in the training set, this causes the system not to learn even when novel condition points are given as input because frequent loss spikes disrupt the learning process. An example of such training iteration can be seen in the right of Figure <ref>. As can seen in the figure, if not novel points are chosen as condition points (black points on the figure) and the predicted point is different from the desired point, the loss value will be significantly high.



This problem may look trivial to fix in the case of the trajectories on the left of Figure <ref> since one can disclude the common parts of the trajectories from the training set, since they will not be used during testing. However, in cases like in the trajectories on the right of Figure <ref> where multiple trajectories overlap and/or intersect at different intervals, this problem becomes hard to overcome with simple solutions.



§ EXPERIMENTAL RESULTS












 §.§ Insertability

This experiment aims to verify that our model can encode and decode insertion affordance. A single robot (UR-10) is used in a simulated environment <cit.> . UR-10 is equipped with a fixed-sized red rod as can seen in Figure <ref>. The task is defined as inserting the rod into the opening in the planar surface on the table. The sizes of the openings vary across experiments, and if it is large enough, it is insertable by the robot (in the center of Figure <ref>), while it is not insertable if it is not large enough (on the right of Figure <ref>). The action is defined as the joint positions of the robot, and the depth image of the surface centered around the opening is used as the object. A force sensor is placed in the connector that connects the robot to the rod, and its output is used as effect input to the system. The depth images of 8 different-sized openings  are used for training (4 insertable and 4 non-insertable), and 2 are used for testing (1 insertable and 1 non-insertable) to show that our model can generalize to unseen-sized objects. The sizes of the test images are chosen in between the sizes of the training images and are not chosen in between the images that are at the boundaries (where one image is inserable and the other is not) since they are not possible to predict. Some of these images can be seen in Figure <ref>.




The effect and action results can be seen in Figure <ref>. The difference between the force readings of insert and not insert conditions can be seen at the top left and center of the figure. When the opening is insertable, the force readings start to rise at the end of the trajectory because the tip of the rod touches the table, while when the opening is not insertable, the force readings start to rise in the middle of the trajectory because the tip of the rod touches the edges of the opening. Due to this difference in motion, the action trajectories are also slightly different, as can seen at the bottom left and center of the Figure. These prediction results can be obtained by setting any two of the three weights of the system (action, effect, and object) to 0.5. There are no weights for the robots since only one robot is used. Furthermore, if the given input configuration (combination of inputs) is insertable the system outputs the depth image at the top right of the Figure and outputs the image at the bottom left of the Figure if the input configuration is not insertable  regardless of the input image given to the system (note that if the input configuration is insertable the given image has to insertable and if the configuration is not insertable the image also has to be not insertable). This shows that instead of learning different representations for each input image, our system learns a common representation that depends on the affordance. 




This formation can be seen in Figure <ref>. The plot shows how the affordance representation changes as training progresses. To construct the affordance representations, we set the weights of the system (action, effect, and object) to 0.33. We picked several novel points along the effect and action trajectories as input to the effect and action encoders. We used all ten objects used in training and testing which are shown in the figure by different markers. Principal Component Analysis (PCA) is used to decrease the dimension of each affordance representation to two.  The colorbar shows the training progress. As can seen in the figure, at the beginning of the training (on the right of the plot), the representations are closer together (due to initialization), and as the training progresses, they diverge, and two groups converge into two different points. It is hard to see all markers due to overlaps. This shows that rather than learning different affordance representations for each (effect, (object, action)) tuple, our model learns the equivalence between the objects. The two equivalences learned by our model can be shown by our notation as the following,

    
  * (inserted,(<insertable-openings>, insert)
    
  * (not- inserted,(<non-insertable-openings>, insert)

Where <insertable-openings> represents all openings that are insertable by the rod attached to the robot and <non-insertable-openings> represents all openings that are not insertable by the rod attached to the robot.












 §.§ Graspability




While the previous experiment showed that our model can encode single-agent affordances, this experiment aims to show our model can be used to encode multi-agent affordances and thus can be used for cross-embodiment learning. Furthermore, in the previous experiment, the affordance used depended primarily on the object. To address these, in this experiment, we showed that our experiment can learn multi-agent affordances that depend not only on the object but also on the action. The task is defined as grasping and lifting an object on the table. Two robots (UR-10 with BarrettHand and Baxter with Baxter gripper) are used in a simulated experiment. The actions used in this experiment can be seen in Figure <ref>. Similar to the previous experiment, varying-sized objects are used. Among the 10 objects that are used, 5 can be graspable only by UR-10, and the remaining 5 can be grasped and lifted by both robots (since the opening of the Baxter gripper is small it can not grasp large objects). Out of these 10 objects, 8 are used for training and 2 are used for testing with similar constraints to the previous experiment. The effect is defined as the change in the z coordinate of the object. If the object is grasped and lifted, the z coordinate changes; if not, it does not change.









The results of this experiment can be seen in Figure <ref>. The difference in effect output when the given inputs represent a graspable condition or non-graspable condition can be seen in the top left and center of the figure. Note that the scale of the grasp effect result plot and the non-grasp effect result plot are different, which is why the error on the left of the figure looks higher. Similar to the previous experiment, the system learned common representations for objects that behave similarly. The outputs of the model when such objects are given to the system can be seen on the right side of the figure. The depth image on the bottom is the output when the inputs include objects that can not be grasped and lifted by Baxter, and the image at the top is the output when the inputs include objects that can be grasped and lifted by both robots. The results also do not change when the images in the test set are given. The generated action trajectories can be seen at the bottom left and center of the figure.








 §.§ Rollability


Another affordance we investigated is object rollability. Two robots (KUKA LBR4+ and UR-10) with two different grippers (Robotiq85 and BarrettHand) are used as agents to conduct this experiment. The task is defined as pushing the given objects in 3 different directions (to the left, to the right, and straight). 2 different actions are used for UR-10 (one with its fingers while the gripper is closed and the other with the palm of the gripper). Push-to-left actions with the cuboid object can be seen in Figure <ref>. The other actions are realized similarly with objects in the same position. 



Objects with different rollability characteristics are used in the experiment. These objects can be seen in Figure <ref>. While the rollability of some of them does not depend on the action applied to them (cuboid, upright cylinder, cone, and sphere), the rollability of the others (cylinders standing on their side) depends on the direction of the action applied to them. All objects except the cone and the upright cylinder are used during training, and the cone and the upright cylinder are used to check whether our system can transfer affordances from one agent to another(s).




The results of this experiment can be seen in Figure <ref>. The difference between the effect trajectory of a rollable object and a non-rollable object can be seen at the top left and center of the figure. When the object is not rollable the object stops when the robot stops pushing (on the left), and when the object is rollable, the object keeps moving after the robot stops pushing (in the center). The generated effect trajectories in the figure can be obtained by giving the system a rollable or non-rollable input combination based on the desired trajectory. Unlike the experiment in Section <ref>, since the rollability of the object does not depend on the action, these effect trajectories can be generated even when only the object is given as input. Thus, our model can predict the object's rollability solely based on its depth image.


Aside from effect trajectories, our system can reproduce the action trajectories of the robots with almost negligible error, as seen from the top right, bottom left, and bottom center of the figure. However, since UR-10 has two different actions with the same outcome in this experiment when no information regarding its action is given, the system is unable to generate a valid action trajectory for UR-10. One of such trajectories can be seen in the bottom right of Figure <ref>. It can be seen that the generated trajectories are very different from the valid ones (the ones on the right and at the center). We believe this is another case where selective loss is useful. After training normally for several epochs, we continued training the model with selective loss used in the UR-10 output of the system. During testing, we saw that not only did the system continue to generate the correct action trajectories for UR-10 when information about its action was given to the system but also it was able to output one of the valid actions when no information about its action is given to the system.





In this experiment, we also checked the system's ability to transfer a learned affordance from one robot to the other. To achieve this, after training the model with cuboid, sphere, and cylinders standing on the side objects, we trained the model with the remaining objects (upright cylinder and cone) with only one robot's one of actions. For this, we devised a two-step training procedure. First, we train the system with the new data (cone and cylinder objects) for one training step. Then, to prevent catastrophic forgetting, we train the system with the old data for one training step. This procedure is repeated for several epochs. In the end, when a new object image and the action of the robot the new object is not trained for is given to the system, it is able to predict the correct effect trajectory for that object. Furthermore, the system was able to generalize to the actions of other directions. In the trials, we trained the model with the new data using a push-to-left action, and it was able to generalize to the push-to-right and push-straight actions of both robots. However, this behavior is not always desirable since if the new data contains an object that behaves differently based on the direction it was pushed (like the cylinder that stands on the side), it would generalize incorrectly.



 §.§ Real Robot Experiments

Since our system can generate action trajectories using effect trajectories and object information, it can be used for direct imitation. To test this ability, we devised an experiment using a real UR-10 with a Robotiq 3-finger adaptive gripper. The task is defined as pushing the given objects to different angles. The locations of the objects are tracked using a tabletop Intel RealSense depth camera which is also used to take the depth images of the objects used in the experiment. During testing, one of the objects is pushed to different angles by the experimenter. The final position of the object is given to the system along with the object's depth image. After putting the object back in the initial position, the action executed by the real robot was able to push the object to the position the experimenter pushed it to.



§ CONCLUSION

In this paper, we proposed and verified a model that can learn affordance representations that can be used to generate continuous action and effect trajectories and object images that constitute those affordances. For the affordances we used in the experiments, our system was able to learn common representations for equivalent representations. We analyzed the formation of these representations using a latent space analysis. We also showed that affordances could be used as a way to couple the actions of different robots, allowing cross-embodiment transfer. We also demonstrated our model's generalization capabilities in transferring affordance information between agents. Moreover, in real-world experiments, we showed that our model can be used for direct imitation. Finally, we introduced selective loss both for our system and vanilla CNMP systems and showed cases where it would be beneficial to use it.




§ ACKNOWLEDGMENTS



 IEEEtran
 



