title: Multiagent RL for a single policy in a large-scale network of homogeneous agents
body: <p>I am having issue with finding a proper multi-agent RL (MARL) algorithm that scales well for a large number of homogenous agents in an environment. I have an environment,</p>
  <ol>
  <li>That is partially observable - agents cannot communication with
  each other and they don't know about the actions taken by the other
  agent or the state of the other agents.</li>
  <li>Has a common observation for every agent, i.e., after every time step t -
  each agent performs an action and everyone receives the same
  observation from the environment.</li>
  <li>Where agents are required have a policy (same policy for every agent)
  that can be learned in a centralized way and distributed among
  agents in a decentralized manner.</li>
  </ol>
  <p>I know parameter sharing is a one way to do it, by extending the single agent network (such as DQN or DDQP or SAC) to multiple agents but such mechanism doesn't scale well for a large number of agents especially for the above-mentioned environment.</p>
  <p>MADDPG, COMA, MAPPO are some of the choices that use centralized critic but the problem with them again, is that they are not suitable for scaling up because the dimensions of the centralized critic state grow exponentially with the number of agents.</p>
  <p><a href="https://arxiv.org/pdf/1802.05438.pdf" rel="nofollow noreferrer">Mean Field MF-MARL</a> is another choice for scaling up but in MF-MARL, each agent learns its own policy, which is not something I want.</p>
  <p>Can someone please suggest me the techniques I can use or if there are any existing algorithms out there that I can use to scale up for 100-500 number of agents, that have a single policy (or maybe 2 policies), that can be deployed at the agents in a decentralized way.</p>