Should my models be actual size?
It won't slow things down.

However, Blender's default settings may not be good for such a large piece. 

Especially the viewport and camera clippings settings (these settings define the minimal and maximum distances rendered). The End clipping is quite small by default so if you happen to have your objects kind of cut in far distance, this is why.

You will find the viewport clipping settings in the View panel of the options bar (N) of the viewport, and the camera clipping settings are in the Camera tab (select the camera first), in the Lens panel :