How thick should should wood be to hold 160 to 180 pounds over a four-foot span?
Regular framing lumber will do what you need. You will not want to place a ladder on individual 2Xs that make up your scaffolding. The deflection on one 2X will feel enormous when you are on the top of a ladder. 2X4s are cool as long as there are no large knots, and you fill the area. #2 grade 2X needs to be watched for this. Don't even use #3 grade. I have seen 2X6s with 2" knots in them that would render them useless as a walk plank. All said, please be mind full of the condition of the wood used for walk planks. Since you are putting a ladder on all this, unify the surface with a sheet of plywood (1/4" will work in a pinch with a solid fill, if you spaced 2X6s, use 1/2") cut to fit within reason, 3' wide or larger will work and as long as feasible. Believe me, you will not want the ladder to find even one gap big enough for a ladder leg to slip through.