What is the formula for the frequency spectrum?
You are likely looking for the "Power Spectral Density" which is determined by taking the complex conjugate multiplication of the Fourier Transform. The complex conjugate process will give the expected real result and represents a power quantity (which has no phase). For the DFT this is often presented in normalized fashion such as dBFS (dB relative to full scale) or dBc (dB relative to the total power in the signal).  If the time domain signal is in volts, then scale the DFT by $1/N$ and the complex conjugate multiplication will represent the power in each bin in terms of Watts assuming a 1 ohm load (or adjust further for true power if the resistance is known). Since it is now a power quantity to convert to dB use $10Log_{10}(F(\omega)F^*(\omega))$. Where $F(\omega)$ represents the DFT or DTFT result with proper scaling, depending on which units are used.

Further as a "density" the result will be power per unit of frequency (such as dBc/Hz). The DFT and DTFT without any further windowing has an equivalent noise bandwidth that is the inverse of the time duration $T$ of the time domain signal, so therefore is equal to $1/T$ (in whatever units are used for $T$). Windowing complicates this further, but see this post for more details on that:

How to calculate resolution of DFT with Hamming/Hann window?