Why is training take so long on my GPU?
That's about expected. If you divide the number of seconds by the number of images you processed, you get 33 milliseconds per image, which seems about right for such a small network. Larger networks usually take in the ballpark of 50 to 200 milliseconds per image. 

Yes, a large dense layer is likely to hurt your performance, since that's a huge matrix (256 by 4096) and a large matrix multiplication to go along with it every time you run the network.