Using pointers and type cast to break up integers into byte array
If the buffer size is not divisible by 4, the remaining bytes are not set. This is a bug and should be corrected.



If you just need random values in the bytes, Random.NextBytes() does what you are trying to do and will likely be faster than both of your current implementation. It's up to you to decide if you care that the values were technically int32's before being written to the buffer.



When writing a benchmark, Stopwatch will provide more precision than DateTime.Now. I also thing 100 is not a large enough number of iterations to get a good average. As you pointed out, there were outliers that took almost twice as long as the majority of values.

It is probably best to take the seed from a command line argument. This will allow you to remove one more variable between the two implementation. While I don't expect Random's execution time to change significantly based on the seed, it is best to keep everything else equal when comparing two things.



Arrays have a Length property, so you shouldn't be dealing with the buffer size independently.



As for if it is worth it to make this optimization, that isn't something we can tell you. Have you profiled the execution of the actual program? Is this where your code if spending the majority of it's time? If it's actually IO bound, ~12ms doesn't mean anything. Where as, if this chunk of code is running millions of times each time your program does a single task, ~12ms is huge and well worth optimizing.