Image classification on stock data
I like this question!

Potential pitfalls:

1. Architecture is just not sophisticated enough to catch all nuances in the time-series image. Take a look at here what they did for earthquake-time-eries-as-images approach or take a look at this paper. You could take some guidelines from there, but you notice right away that the architecture is bigger. Or this one

2. Data, you say its predicting only one class, but what part of the image is it focusing on. Use shap values for images to inspect. I suspect following: it will be some trend that is consistent only for one class, hence you can think about pre-processing steps that can point to properties on images of other classes. Or even add some hand-crafted numeric features. In any case increase your dataset if you havent already, you want to have enough information to catch all nuances.