CNN based model poor result
Have you tried using a different learning rate?
Yours seems to be quite high, you could try 1e-4 or something in that region.
Coming from the computer vision side, you could also try to improve image readability with prior grayscaling or blurring.
Here are some repos on GTSRB:
https://github.com/Junth/Traffic-Sign-Classification-using-ConvNets
https://github.com/surmenok/GTSRB