What to do when seed has a big impact on model performance?
It's just a simple idea that I often recommend to understand what happens in cases like this:
You could try to do an ablation study, where you train the model with varying sizes of training data (e.g. 10%, 20%,..., 100% of your full training data). Given that your problem is instability, it would be even better to do every size several times with different random subsets. Then you plot the performance as a function of the data size, for example with boxplots in order to visualize the variance. The evolution of the performance (and its variance) would show you whether there is enough training data given the complexity of the model: if yes, the curve increases and then starts stabilizing around its maximum. If not, the curve is still in a phase of increase when it reaches 100%, meaning that more data would be needed to allow the model to stabilize.
Of course the disadvantage is that you need to run the training/testing many times, so this might not be practical.