What fast loss convergence indicates on a CNN?
Ok, so the fact that you always give the same category means the network only has to output the same value for any given input. This is exceedingly easy to learn (all the weights always shift in the same directions) and this make this graph. 

Once you have more categories (try 10 for example), you'll see the graph will be closer to expected.