Fourier Transformed a signal $f(t) = \cos(2\pi*4t)*5\cos(2\pi*0.4t)$ but did not get expected results. What went wrong?
Fourier Transform is linear transform. It is mean that Fourier Transforms of sum of functions will be sum of Fourier Transforms of each function. It is not valid for multiply of function.

Your example. There is very well known formula (see 7c in this link):
$$
\cos(\alpha) * cos(\beta) = 1/2 (\cos(\alpha+\beta) + \cos(\alpha-\beta))
$$
use this formula  to transform your function:
$$ 
f(t) = \cos(2\pi*4t)*5\cos(2\pi*0.4t) = 5/2*(\cos(2\pi*4t - 2\pi*0.4t) + \cos(2\pi*4t - 2\pi*0.4t)) = 5/2*(\cos(2\pi*3.6t) + \cos(2\pi*4.4t))
$$
So Matlab results is correct - you have 2 peaks equal amplitude with 3.6 and 4.4 frequency.