How can I hang pots on my brick wall with the least amount of drilling?
you definitely don't want to drill into the brick. Drilling into the mortar isn't too bad though, and easily repairable if ever desired. 

I've used Tapcons for anchoring into mortar. They work well:



The box will say what size masonry drill bit you need to pre-drill, then you just screw them in with your driver like any screw. I'd suggest longer is better (though no longer than the width of the brick, as that may be as thick as the brick is.)

Also, make sure this brick you are drilling into isn't a chimney. If it is, make sure it's either not in use, or has a metal flu inside (and don't puncture the flu)

Your idea of installing a wood cross-member into the brick and then hang something from there isn't a bad idea.  I'd suggest 3 screws, then perhaps attach peg board to that, hanging the pots from peg board ala Julia Child