How to get unique displacement for each particle object?
I came across ParticleInstance modifier. I'm not sure if this is the best way to do it but it seems to work.

In the particle system settings, render as object and use Empty:



Add a ParticleInstance modifier to the actual object and place it before the Displace modifier. To use the size randomness, enable the Size option:



Edit. Actually the 'Render As' option of the particle system can be None. Just temporarily set it to Object to set the Scale and Scale Randomness. Then set to None, or just don't select anything in 'Instance Object'. There's no need to multiply all the Empty objects.