Joining Rigid Body objects after baking to shorten BVH building time
If you link the objects with &#9096; CtrlL > Object data, one copy of the mesh data will be shared between all 20k objects (as opposed to 20k copies of the same data).

While not as efficient as whole-object instances (e.g. dupli objects), AFAIK this is about as close as you can get while still keeping the rigid body sim.