Are there any downsides of using a fixed seed for a neural network's weight initialization?
When you use a particular seed, it actually ceases to become a random initialization and is instead fixed. I believe the only reason to actually do this would be for reliable reproduction in research and not as a method of training production models.