What technique could I use to smooth the right amount these signals?
If you come up with a set of criteria that define what makes a local maximum become a "peak of interest", then you could test whether each local maximum meets the criteria you set. For example:

A local maximum at $X(f)$ is considered a peak of interest if for some chosen small value $\epsilon$ and some chosen height $\mu$ , we can satisfy the following

$$\exists \, \delta_1 , \delta_2 \lt \epsilon \quad s.t. \quad X(f)-X(f-\delta_1) &gt; \mu \quad \text{and} \quad X(f)-X(f+\delta_2) &gt; \mu$$

If one such criterion is not sufficient to describe your concept of a peak of interest, then you can logically combine multiple criteria.

You could approach the problem from a filtering and smoothing perspective as you mention, but notice that each filter or smoothing operation has an implicit criteria that it is enforcing.  No matter how you implement your peak detection, realize that there is no "right answer" as to what defines a peak of interest. So before you can achieve the desired result you must first decide what that result should look like.