Does "build up" on the bottom of copper-bottom stainless sauce and frying pans matter?
I think it depends on your cooking surface.

If you're using gaz, induction or coil, then it should be ok.

If you are using a glass top, then the contact between the pan and the glass will not be efficient, and you will loose some of the heat.

I would spend some time cleaning it up.