Return Loss \$ (S_{11}) \$ Equation for a nth-order Butterworth Filter
There is no single answer to this question. There are numerous circuits that can implement any particular filter design, as we discussed in a recent question The return loss (\$S_{11}\$) depends on the topology you choose to implement the filter.

If you use an active topology, for example, then only the first stage will affect \$Z_{in}\$, and thus only the first stage will affect \$S_{11}\$.

If you choose a passive topology it depends if you construct the filter from pi sections or T-sections or some other topology. 

For example, if you use pi sections (with LC parallel elements in the shunt members), then \$S_{11}\$ will go to -1 in the stop-bands. If you use T sections (with LC series elements in the through members) it will go to +1.

If you use microstrip elements, the behavior will likely have some complex periodic behavior in the stop bands.

Whichever one you choose, \$S_{11}\$ should be near zero in the pass band. Whether you achieve -40 or -50 or -60 dB probably depends more on choosing very tight-tolerance parts or trimming the circuit carefully, rather than on the nominal design. Although some design choices might be more or less sensitive to component variation. So a closed form solution for reflections in the nominal design won't help as much as doing a Monte Carlo simulation accounting for likely component variations.

If \$n\$ is more than 2, I'd suggest to just simulate the design rather than try to find a closed-form solution, because the equations will get rather tedious to deal with very quickly.