Given that the high thermoelectric performance of the chalcognides is due in part to low κ latt attributed to intrinsic phonon anharmonicity, it is of interest to understand the impact of the different crystal structure types adopted across the series on the structural dynamics and thermal transport. First-principles modelling using techniques such as the single-mode relaxation-time approximation (SM-RTA) allows for accurate prediction of the lattice thermal conductivity while providing microscopic detail at the level of individual phonon modes, and these calculations have for example played an important role in understanding the low thermal conductivity of SnSe. We recently developed a set of techniques for analysing SM-RTA calculations that can be used to attribute differences in κ latt quantitatively to differences in the phonon group velocities and lifetimes, and to quantify the contribution of the strength of the anharmonic phonon interactions to the latter. These were recently applied to a family of silicon allotropes and highlighted the important role of structural complexity in determining the κ latt . In this work, we apply these techniques to investigate the impact of the crystal structure on the lattice thermal conductivity of the group IV-VI chalcogenides GeSe, GeTe, SnSe and SnTe. We find that the link between the κ latt and different structure types results from a balance of low phonon group velocities, favoured by large primitive cells, and strong phonon anharmonicity and short lifetimes, favoured by structures where the tetrel atoms are constrained to locally symmetric environments. We also find that the shape of the phonon spectra, which determines the so-called "scattering phase space" (the number of allowed energy-and momentum-conserving scattering pathways), also has a prominent influence on the phonon lifetimes. These effects are best balanced in the high-temperature Cmcm structure of SnSe, which has the lowest κ latt across the ten structures we examine. Our results demonstrate that the crystal structure has a far larger impact on the κ latt than the atomic masses, and the microscopic insight from this analysis provides clear guidelines for optimising the heat transport in the IV-VI chalcogenides through crystalstructure engineering.