1/n octave smoothing
Typically "smoothing" means "replace the current value with average over the neighboring ones". Most common is energy smoothing, where the smoothing results in the energy average over the smoothing interval and the phase information is lost. Complex smoothing can be done as well but it's tricky business because of phase wrapping.

Energy smoothing can be expressed as 
$$Y(k)=\sqrt{\frac{1}{N}\cdot \sum_{i=0}^{N-1}X(i)\cdot X^{*}(i)\cdot W_{k}(i)}$$

where $W_{k}(i)$ is some suitable window function. In the case of, say, third-octave smoothing this could be derived as the the magnitude squared of the transfer function of a third octave band pass filter around frequency k. This also means that for a, say, 1024 point FFT you need to design 1024 different bandpass filters, so that's a fair bit of work.

Things can be simplified if the exact shape of the smoothing filter is flexible. Rectangular smoothing can be done as 
$$Y(k)=\sqrt{\frac{1}{b-a+1}\cdot \sum_{i=a}^{b}X(i)\cdot X^{*}(i)}$$

where
$$a = round(k*2^{-\frac{1}{2\cdot n}}), b = round(k*2^{\frac{1}{2\cdot n}})$$

are simply the indices of the band edges for $n^{th}$ octave smoothing. 

There are a few more methods that are in between in the arbitrary window and the rectangular one in terms of complexity.