Audio Mixer Passive Attenuator Circuit
If you take a look at existing mixer designs, you'll find that they typically use a passive resistor attenuator, and that the values aren't necessarily huge. Because it's just a pad, the 20dB needn't be exact (the trim control will sort that out).

So, determine the typical source impedances that your mixer is going to be handling, and pick resistor values that give you approximately 20dB of attenuation. If it varies depending on the source, that's not a huge issue.

Most of the sources you are dealing with are going to be 600Ω +/- 300Ω (ish). Don't bother with trying to design for instrument pickups and other Hi-Z sources: that's what DIs/preamps/active pickups are for. Yes, there are microphones in the 5kΩ and up range which you may want to consider. However, any microphone past 15kΩ or so ought to be considered Hi-Z, and I would argue that microphones past 5kΩ are atypical enough to warrant a preamp of their own.

I'm a big fan of Allen &amp; Heath because lots of their mixer schematics/service manuals are easily available (also, the consoles sound great). If you take a peek at some of these, you'll get an idea for design and component values of a typical mic pre.