MIT 6.003 HW#8 Problem 4 - Fourier Coefficients of Triangle Wave
The symbol $\underset{T}{*}$ in the solution refers to ($T$-) periodic convolution, which is defined as
$$y(t)=\int_{0}^Tx_1(\tau)x_2(t-\tau)d\tau\tag{1}$$
where $x_1(t)$ and $x_2(t)$ are $T$-periodic signals. If $a_k$ and $b_k$ are the Fourier coefficients of $x_1(t)$ and $x_2(t)$, respectively, then the Fourier coefficients $c_k$ of $y(t)$ are given by
$$c_k=Ta_kb_k\tag{2}$$
Your approach to the problem is valid for $k\neq 0$, and your result is actually pretty close because
$$\frac{1-\cos\big(\frac{\pi k}{2}\big)}{4\pi^2 k^2}=\frac{2\sin^2\big(\frac{\pi k}{4}\big)}{4\pi^2 k^2}=\frac{\sin^2\big(\frac{\pi k}{4}\big)}{2\pi^2 k^2}\tag{3}$$
so there's just some scaling problem that I'm sure you'll be able to fix (check your definition of $f(x)$!). Note that you need to compute the coefficient for $k=0$ separately.