Exclude audio from time remap
I would duplicate the layer before applying time remapping (cmd+D).
Then only apply the time remap to one of the two layers.
Switch off the eyeball on the layer for audio (and make sure the loudspeaker icon is still enabled)
Switch off the loudspeaker icon on the layer with the time-remapping applied to it.