Deep Q-learning is not performing well when there are several enemies I am playing with a deep Q-learning algorithm in my own environment. The network can perform well as long as there is only one enemy. My agent can perform the following actions:


do_nothing
prepare_for(e)
attack(e) 


where e is some enemy. 

In the case of two enemies, the action vector has 5 elements:

|   0       |      1          |      2      |        3         |     4      |
-----------------------------------------------------------------------------
|do_nothing | prepare_for(e1) |  attack(e1) |  prepare_for(e2) | attack(e2) |
-----------------------------------------------------------------------------


After a couple of episodes, the agent always starts picking the first do_nothing action, which is not desired. Changing reward for do_nothing action is not helping, even using significantly higher negative reward, than for other actions. 

There is no problem with the environment with only one enemy. (Only using columns 0, 1, 2). I feel like my action encoding can be the issue, but I can't figure it out, how to fix it. Any suggestions?
1) Vary the number of enemies during the training process.
2) Use other generalisation methods that are appropriate for neural networks such as dropout or L1 and L2 regularisation.