Does bidirectional impedance matching matter?
I assume you are talking about L-Pads in your post. I am far from an expert, but I believe that bi-directional impedance matching is only important when you are concerned about issues such as reflections and standing waves, such as RF applications (or any other time transmission line effects are significant). In these cases, then yes, the primary design constraint is the matching of the impedances in both directions, and the insertion loss is often a secondary consideration.

Conversely, in situations such as audio attenuation, the impedance matching doesn't matter in the "reverse" direction nearly as much. A common use of L-pads are attenuating the output of, say, a tube amplifier that expects to be presented with a certain nominal impedance. The speaker, on the other hand, doesn't care so much what it is being driven by (consider the fact that most solid state power amplifiers are often lower impedance than the speakers they drive). The primary design consideration here is attenuation (insertion loss) and matching in one direction, whereas there often is no consideration as to what the speaker will see.