How would I go about wiring a new light fixture where there are more wires than expected?
Personally, I would put the cap back on and back away slowly. 
IMO your only option is to call an electrician to come and replace that old iron box and strip back on the cable to expose fresh insulation (not always a sure thing though). 
Those old "boxes" give you no place to attach a modern fixture safely, and have absolutely no room for the wiring. They took into account that old fixtures had huge canopies that would contain the wiring.