Unit of Energy Spectral Density
If $x(t)$ is an action signal like voltage or current, then the square of $x(t)$ is proportional to instantaneous power and the constant of proportionality depends on what kinda animal $x(t)$ is and what load $x(t)$ is connected to.

So if $\big|x(t)\big|^2$ is proportional to instantaneous power, so also is the integral of $\big|x(t)\big|^2$ over all time proportional to energy (and having the same constant of proportionality).  So you can call this "energy":

$$E_x = \int\limits_{-\infty}^{\infty}\big|x(t)\big|^2 \, dt$$

as long as you remember that it's really proportional to energy and the constant of proportionality depends on what dimension is $x(t)$ and what load that signal is connected to.

For example, if $v(t)$ is a voltage, then $|v(t)|^2$ is proportional to instantaneous power and if $v(t)$ is connected to a resistor having resistance $R$, then instantaneous power is 

$$ p(t) = \frac{1}{R}\big|v(t)\big|^2 $$

and the total energy is

$$E_v=\int\limits_{-\infty}^{\infty}p(t) \, dt$$

or

$$E_v=\frac{1}{R}\int\limits_{-\infty}^{\infty}\big|v(t)\big|^2 \, dt$$

Clearly the constant of proportionality is $\frac{1}{R}$.

If $x(t)$ is a dimensionless value in a DSP or computer, then you have to scale it with the reference voltage $V_\mathrm{ref}$ of the D/A converter to make this number a voltage that delivers power to a load.  Then the constant of proportionality becomes $\frac{V^2_\mathrm{ref}}{R}$ and the energy really is (with correct scaling):

$$E_x = \frac{V^2_\mathrm{ref}}{R} \int\limits_{-\infty}^{\infty}\big|x(t)\big|^2 \, dt$$