What is the purpose of the serrated trailing edge spoilers on the B52?
The comments above by John K. and Maury Markowitz are on the right track. The serrated trailing edge of the spoilers has an effect on vortex shedding. It is called that because it is a vortex-shaped airflow that is shed from the rear of an airfoil or other surface.

On modern jet engine nacelles the serrated edges at the rear reduce noise. However, on the B-52 the most likely reason for this feature on the spoilers is to reduce the possibility of vortex shedding inducing vibrations that could cause the wing structure to resonate due to the harmonic oscillation of the vortex. This is also known as flutter. My guess is that at some point in the life of the aircraft design it was determined that this was (or could be) an issue, and this was the solution.

If you think about the last time you flew on an airliner where the pilots deployed the spoilers while still in flight, you may remember the vibration generated by them. That is the consequence of vortex shedding, and that is what the serrated edges are designed to minimize on the B-52.