Help with obtaining the power spectral density of a simple continuous cosine (using both forms of the definition for PSD)
There are several misconceptions in the question that have not been addressed in the existing answers. First of all, the signal $x(t)=A\cos(2\pi f_0t)$ is a deterministic power signal (unless $A$ or $f_0$ are modeled as random variables). For this reason several definitions in the question are inappropriate. First, the auto-correlation of a power signal is given by

$$R_x(\tau)=\lim_{T\to\infty}\frac{1}{2T}\int_{-T}^Tx^*(t)x(t+\tau)dt\tag{1}$$

The integral given in the question (with infinite limits and without division by $T$) does not exist for the given $x(t)$. With definition $(1)$, the auto-correlation of $x(t)$ is indeed obtained as

$$R_x(\tau)=\frac{A^2}{2}\cos(2\pi f_0\tau)\tag{2}$$

The Fourier transform of $(2)$ results in the power spectrum of $x(t)$.

The power spectrum can also be computed directly from $x(t)$, but the formula given in the question only applies to random signals, but not to deterministic signals. For deterministic signals, the appropriate definition is

$$S_x(f)=\lim_{T\to\infty}\frac{1}{2T}\left|\int_{-T}^{T}x(t)e^{-j2\pi ft}dt\right|^2\tag{3}$$

The computation of $(3)$ for the given signal is discussed in this question.