Can the performance of a CNN be dependent on the train-test-val split random seed?
Yes, it can depend on it because it changes the data distribution with the network is trained.
You shouldn't consider random seed as a hyperparameter. Keep the same random seed and run comparison. Do this for at least 5 or 10 random seeds. You will definitely have a winner. If you don't go for more than 10 unless you get a winner. That will give you sufficient proof about the efficacy of the method. Otherwise since one is good in random seed or other one in other, just choose one of those random seeds, choose the method that gives best in it and continue your work.