Should I model my animation assets to real-world scale?
I would say: stick to real world scale. At the end, in very very small scale there will be rounding errors when placing vertices.

And for your navigation difficulties in large scale: In Preferences > Interface, check Auto Depth and Zoom To Mouse Position - this really helps. And when editing an object, place it in an empty layer and switch off all others, this also improves pan and view functionality.

Update for Blender 2.82: These settings are now to be found by clicking Edit > Preferences > Navigation.