The rattling frequencies of the noble gas atoms are determined through a detailed balance of their atomic mass and how strongly they interact with the host framework, both of which increase from He to Xe. The heavier filler atoms suppress the heat transport through the low-lying optic modes, which accounts for ~20 % of the total 𝜅 latt in pristine CoSb3, primarily by reducing the group velocity. The effect of the fillers on the phonon lifetimes depends intricately on changes in the phonon-phonon interaction strengths and changes in the number of energy-conserving scattering pathways for modes in different parts of the phonon spectrum. The heavier noble gases show comparatively weak phonon-phonon interactions, and their effect on the phonon lifetimes occurs mainly by providing additional scattering channels.