Computing accurate vibrational and rotational contributions to the free energy of transition states and loosely bound complexes
Given the accuracy it sounds like you're looking for, and the systems you're studying, this indeed appears to be a Very Hard Problem. I think your intuitions are well founded.

Briefly, to your questions:

1) How to handle the low-frequency modes?

A lot of manual exploration of the PES, doing things like running normal mode scans and optimizing intermediate geometries along suspected internal rotations. This gives you an idea how the various energetic barriers to non-harmonic motions compare to a Boltzmann distribution of states.


If the barriers are high, the harmonic approximation is probably decent and you can just use Gaussian's results
If barriers are low, then you have ~free internal rotation. 
If intermediate, then you're facing hindered internal rotation.


There's literature on evaluating enthalpy/entropy contributions of free and hindered internal rotations.  Some can be found in my answer to a different question, here.  Other potentially relevant resources include:


Czaszar (WIREs Comput Molec Sci 2 273, 2012)
Overview of anharmonic force fields and their computation.
Bloino (J Chem Theor Comput 8(3) 1015, 2012)
Overview of the VPT2 anharmonic computational system implemented in recent versions of Gaussian.
Carbonniere (Chem Phys Lett 392 365, 2004)
Description of the vibrational-rotational (Coriolis) coupling anharmonic correction and its potential significance for some systems. More recent references are likely available.
Meal (J Chem Phys 24(6) 1119, 1956)
Meal (J Chem Phys 24(6) 1126, 1956)
Classic discussions of Coriolis coupling.


You should be able to turn up quite a bit more information with some quick further searches, if need be.

2) Do I need to be worried about contributions from centrifugal distortion?

Very likely. Unfortunately, I don't know of very many examples where people have attempted corrections such as this. Certainly you're looking at manually developing a suitable analysis; I can't imagine any software packages out there at present are capable of automatically chugging through this for you.