What is the frequency response function for an input through a differentiator and mixer?
The output of the differentiator is $Y(\omega)= j\omega X(\omega).$

Multiplying by the sine in the time domain is convolving with the Fourier Transform of sine in the frequency domain:

$$\mathcal F\left\{\sin(w_ct)\right\} = \frac{1}{2j}(\delta(\omega+\omega_c)-\delta(\omega-\omega_c))$$

Which results in the following expression for the output of the mixer, as a function of $Y(\omega)$:

$$\frac{1}{2j}(Y(\omega+\omega _c)- Y(\omega-\omega _c))$$

substituting back for $Y(\omega)$ in terms of $X(\omega)$: 

$$\frac{1}{2j}(j(\omega+\omega _c) X(\omega+\omega _c)- j(\omega-\omega _c) X(\omega-\omega _c))$$

which is

$$\frac{1}{2}(\omega+\omega _c) X(\omega+\omega _c)- (\omega-\omega _c) X(\omega-\omega _c))$$