How to calculate time of arrival using room impulse response
If you do not have a very controlled room, there might be something attenuating the direct path and a reflection might then be the highest peak. In that case you should accept also a somewhat attenuated peak if it comes earlier than the strongest peak.

Coupling to structures is not that much of an issue. For example, when a sound wave reflects at the interface of wood and air, only a fraction of about 0.0007 (see Ref. 1) of the energy of the wave will be passed to the other medium. The wave will then spend a lot of time in the first medium until much of its energy is passed to the second medium, so you can imagine that the impulse response of this coupling is not very concentrated in time and doesn't disturb the peak detection much.

References:


Acoustics of wood, course in non-destructive testing of wood, ETSI Montes, ETS Arquitectura – Universidad Politécnica de Madrid, June 2005