How can I speed up Freestyle rendering?
Disable everything else

As you thought, ther are lots of things you can avoid to render if you are looking for just the freestyle edges.

You just have to disable all the elements you can find in the Include section of the Layer panel you can find in the Render Layers tab (except Freestyle obiuvsly, if you are using Blender Render)