How to define Agar.io state and action space?
The state space is certainly continuous, assuming that you can somehow feed that AI exact coordinates. You may have to resort to CNNs if you do not have access to this information. For the action space, you should consider how the game actually plays. Since you use a mouse to simply show the direction, you could use (x,y) positions of the mouse as an action, or even just the angle $\theta$ of the mouse cursor in a circle around the agent. If you are playing on the site, then your observations would have to be from a CNN, it should be possible to use the score as your reward, as well as the possibility for eating things or distance from opponents that are bigger or smaller as intermediate rewards. The number of nodes in your network is something that you must find experimentally, you may like to research what kind of architectures other people have used in this field. You shouldn't need to account for different numbers of players. Nothing special needs to be done to get a single node giving a continuous output to represent the angle $\theta$, or alternatively two nodes representing x and y. You can then use tanh or sigmoid to limit the output node values for eject and split actions.