Unexpected results when comparing a greedy policy to a DQN policy
Your problems appear to derive from a non-Markov state description. In brief, the agent has no way to ever learn that "heavy jobs will make a server unavailable for longer" and this is further compounded by arbitrary state transitions every 10 steps whilst the agent cannot track time.

Looking at the example from Sutton &amp; Barto, you should note that they took care to model a valid MDP. Servers become available at random on each time step, and there is no hidden state. In your case you have three sources of hidden data that affects state evolution systematically. If an agent had some data about these then it might be able to use to optimise choices better:


Servers in use due to tasks assigned by the agent return deterministically n steps after the action that assigned them. The agents inability to track this is likely to be your largest problem - there is no data flow in RL that will allow the agent to associate the heavier jobs or slower servers with making a server unavailable for longer, so it never "sees" the clash of resource management that you want it to resolve.
Every 10 timesteps there is an arbitrary change to server availability. This is a problem because it is entirely predictable on which time steps it can occur, but the agent does not know the current timestep.
You are driving this from a dataset, not randomly, so the impact on this will vary depending on how the dataset evolves over time. A particularly bad case here would be if the dataset represented a "working day" so that there were clear timezones when availability followed different patterns. The agent has no data input in the state in order to associate patterns of use (such as the time of day), so the environment either becomes an online learning problem (agent must continuously learn and adapt to new patterns as they arise), or it must come up with a strategy that works in some average over all possible patterns.


In addition I can see a possible secondary effect, an "exploit" in the simulation, that may have a noticeable effect depending on how often specific servers drop out of the pool. It would appear if an agent assigns a task to a server, that you have no mechanism to unassign that task should the server drop out. So an aggressive agent that assigns greedily to the best available servers will get extra "free" time on the high capacity servers if it always claims them. As the agent doesn't know when the drop outs could happen, this effect competes with the average of leaving servers available just in case a heavy task arrives.

How to fix it? It depends on how much of your simulation is fixed by your requirements. You can either adjust the simulation to match your current state representation, or you can change the state representation to capture more system data from the simulation. 

Probably the simplest state change would be to expand the list of servers to always include all servers, and for each unavailable server to have a "ticker" to show how many time steps before it becomes available. To make the problem smaller, you could maybe just track 1-10 ticks and have a combined state for (>10), as that will expose enough of the key state information that the agent can make decisions about pushing easy tasks into medium or low capacity servers. This does not need to be done to the level server id if the servers truly are interchangeable otherwise. So instead of an array of number of free servers by type, consider that it expands to a table with number of free servers by type at one end, then a column of number of servers by type with 0.1s remaining until they are free, then a column for servers that still have 0.2s remaining etc etc