How do I make sense of the cosine wave having Fourier Transform coefficients which have infinite magnitude?
Your function is both integrable, and square integrable, because it is continuous on the finite support $[0\;2\pi]$. In this context ($L_1 \cap L_2$), its Fourier transform is pretty well defined as a classical function (not a distribution like the Dirac), and continuous. So, your formula is incorrect.

So there is no infinite magnitude. Moreover, one cannot simply say that Dirac have "infinite magnitude".

Yet, this would not imply that a perfect cosine wave can not be produced in real life; produced by whom? However, there is little chance that somebody could observe, or capture, a perfect cosine wave in real life (caveat: if real life can be described by a continuous time variable, and amplitude could be continuous as well).