What is the role of embeddings in a deep recurrent Q network?
The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step.
$(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input) $\rightarrow$ input network (embedding) $\rightarrow$ $z_t$ (task representation)

According to to section 6.1 of the paper, the input is a tuple represented as $(o^a_t,m^a′_{t−1},u^a_{t−1},a)$. Each of these  terms is described in sec 3 as:

$o_t$ - The observation. The authors assume a POMDP

Two types of actions:

$u_t$ - An environment action selects by all the agents at each time step in exchange for a team reward
$m_t$ - A communication action, observed by other agent but has no direct impact on the reward or env


$a$ - The agent (This being a multi-agent DQN algorithm)


They form the input to the Deep Recurrent Q-Network architecture.
The purpose of the embedding network is to receive a tuple of these inputs and produce a state
embedding $z$. This state embedding is then fed  into a hidden state $h^a_{t-1}$ of the RNN.
Though the authors refer to it as an input network that produces embedding, in practice, they use different embedding functions for each of these inputs. The final task/state embedding $z_t^a$ is expressed as a sum:
$$
z_t^a = \text{MLP}(o_t^a) + \text{MLP}(m_{t-1}) + \text{LookupTable}(u_{t-1}) + \text{LookupTable}(a)
$$
The summed embeddings are all of the same size.
This answer makes an assumption that by &quot;embedding layer&quot; you meant the input embedding network. Since the paper makes no reference to a single embedding layer in the model architecture.