Why mountain bikes had originally 26 inch wheels?
Wheel &amp; tire sizes are a source of unending confusion. The more you learn, the more confusing they get.

29" mountain-bike wheels have the same bead-seat diameter as modern road bikes: 622 mm (which are sometimes referred to as 28" wheels in Europe, which is confusing, because they're slightly smaller at the rim than 27" wheels…see what I mean?).

When mountain biking as we know it today was being pioneered on the repack ride on Mt Tamalpais, riders weren't using road bikes, and they weren't using mountain bikes because mountain bikes per se did not exist. They were using "cruiser bikes," which had fatter tires and slacker angles, making them better suited to bombing down fire trails. This is a case of people using what they had to hand and adapting it to circumstances, rather than designing something for an intended purpose.

The question of why cruisers have a different (and otherwise non-standard) BSD is another matter. I will note that a rim with a 571-mm BSD mounted with a  1¾" tire (which passed for "fat" back in the day) has a very similar outer tire diameter to a road bike using a 622-mm or 630-mm rim with a skinny tire, and sizing the rim to the intended outer tire diameter is common in bike history. Also, Schwinn, which created the cruiser, dominated the US bike market and could make de-facto standards whenever they wanted.

I can't answer whether 29ers back in the day would have conferred an advantage, although you might not have been able to squeeze those wheels into those old cruisers.