Title: Consequentialist conditional cooperation in social dilemmas with imperfect information,
Abstract: Social dilemmas, where mutual cooperation can lead to high payoffs but
participants face incentives to cheat, are ubiquitous in multi-agent
interaction. We wish to construct agents that cooperate with pure cooperators,
avoid exploitation by pure defectors, and incentivize cooperation from the
rest. However, often the actions taken by a partner are (partially) unobserved
or the consequences of individual actions are hard to predict. We show that in
a large class of games good strategies can be constructed by conditioning one's
behavior solely on outcomes (ie. one's past rewards). We call this
consequentialist conditional cooperation. We show how to construct such
strategies using deep reinforcement learning techniques and demonstrate, both
analytically and experimentally, that they are effective in social dilemmas
beyond simple matrix games. We also show the limitations of relying purely on
consequences and discuss the need for understanding both the consequences of
and the intentions behind an action.