We presented a heuristic quantum approach to describe the permittivity of an excitonic system coupled to a plasmonic cavity. This way, we could reconcile theory and experimental data for core-shell particles with a metal core and an excitonic shell. Previous purely classical models predicted the occurrence of an additional peak that has not been observed in measurements. Our approach started from the semi-classical model of a two-level system that takes the power broadening into account, which is induced by the light-matter coupling. We argue that this power broadening occurs even in the low-excitation limit since in plasmonic cavities, even the intensity of the vacuum field is strong enough to noticeably alter the optical properties of the shell. This effect does not even require strong coupling, but can also arise in weakly coupled systems with lossy cavities. The power broadening suppresses the non-physical peak allowing for a remarkable match with experimental data. Using our model, we were able to recreate the extinction spectra of coreshell nanorods of various sizes. The necessity to alter the dye parameters individually vanishes and we only need to consider the varying coupling strengths. The fact that already vacuum fluctuations can cause the effect and that the power broadening occurs already in the ground state, implies that this might be a vacuum effect similar to e.g. the vacuum Rabi splitting. This would allow for lowphoton nonlinearites, even outside the strong coupling regime.