How to choose the random seed?
TL:DR, I would suggest not to optimise over the random seed. A better investment of the time would be to improve other parts of your model, such as the pipeline, the underlying algorithms, the loss function... heck, even optimise the runtime performance!  :-)



This is an interesting question, even though (in my opinion) should not be a parameter to optimise.

I can imagine that researchers, in their struggles to beat current state-of-the-art on benchmarks such as ImageNet, may well run the same experiments many times with different random seeds, and just pick/average the best. However, the difference should not be considerable.

If your algorithms has enough data, and goes through enough iterations, the impact of the random seed should tend towards zero. Of course, as you say, it may have a huge impact. Imagine I am categorising a batch of images, into cat or dog. If I have a batch size of 1, and only 2 images that are randomly sampled, and one is correctly classified, one is not, then the random seed governing which is selected will determine whether or not I get 100% or 0% acuracy on that batch.



Some more basic information:

The use of a random seed is simply to allow for results to be as (close to) reproducible as possible. All random number generators are only pseudo-random generators, as in the values appear to be random, but are not. In essence, this can be logically deduced as (non-quantum) computers are deterministic machines, and so if given the same input, will always produce the same output. Have a look here for some more information and relative links to literature.