Order of applying two Butterworth filters (distributivity)
The short answer: it depends.

The longer version: From MIT OpenCourseWare notes:


  Specifically, if we have several LTI systems cascaded together, the
  output generated by an input to the overall cascade combination does not depend
  on the order in which the systems are cascaded. 


In your specific case, you have a cascade of two blocks: one which is a 2nd order Butterworth low-pass filter (which can be shown to be a linear-time-invariant (LTI) system), and another which is your linear transformation function $f(a,b)$. The question then boils down to whether $f(a,b)$, in addition to being a linear transformation, also happens to be a time-invariant transformation. 

If $f(a,b)$ happens to be an LTI transformation, then the cascade $\mbox{butter(f(a,b))}$ is equivalent to $f(\mbox{butter}(a),\mbox{butter}(b))$. Thus, in that case one could say that $\mbox{butter}$ is distributive over $f(a,b)$.

If $f(a,b)$ isn't time invariant, then we can't make that claim as to whether the result depends on the order of operations.