FileVault encryption slow when idle: opposite expected
There could be several reasons for this type of behaviour.

If you're on a battery-powered laptop, it might be that the system was designed to optimize run time on the battery. It could be that the 5 MB/sec mode is more power efficient than the 35 MB/sec mode.

Another reason could be that the encryption process needs good quality random numbers. It is often seen (across vendors, products and operating systems) that good quality random numbers require some form of user input or other external input to the system. Thus when the system is completely idle, the source of random numbers is slow (or runs out) - while just wiggling the mouse around can make the process much faster. It's not fully clear to me which part of the FileVault process you're working with, but I think you should be over the phase where random number generation is needed.