Efficient computation of dot product of convolutions
Convolution is a specific type of linear operator. So in all generality, you can write your expression as $\langle F_1 s_1 | F_2 s_2 \rangle$ with two linear operators $F_{1,2}$. Using the rules of operators in bilinear forms, we can rewrite this as $\langle s_1|F_1^\dagger F_2 |s_2\rangle$ and that again as $\langle s_1|\left(F_1^\dagger F_2\right) s_2\rangle$.

In your case, the two operators have a pleasant property. They are toeplitz, and if we take your padding into account, even circulant. The product of two circulant operators is again circulant, as is the conjugate transpose. That means you can write the result of the operator product as a single convolution. Your convolution kernel is the correlation product $\mathrm{corr}({f_1}^*,f_2)$. 

If you implement this using fast convolution, you need to pad this correlation product and the size of your kernel will increase to the sum of the sizes of $f_{1,2}$. Accordingly, the required padding for $s_2$ will also increase.