How did Wilbur Wright create and use the formulas he mentions in his letter to Octave Chanute?
I do not have first-hand knowledge of this formula, so what I write below is speculation. But the Wrights used sound physical reasoning, and I will try do do the same.

Power in physics is the product of force and speed. The formula does indeed contain a force (thrust) and a speed (pitch times rotation speed) if the terms used by Wilbur are interpreted in a certain way. Pitch today is an angle, but I would think Wilbur meant with pitch what is termed distance D in the picture below.



Propeller advance ratio (picture source)

This is consistent with his earlier sentence where he gives a length for the term pitch. The term pitch * RPM/60 is actually the flight speed in m/s.

Now to the term 5/6: I would expect that Wilbur calculated the pitch from the incidence of his propeller. Since the propeller blade needs some angle of attack to produce thrust, the actual distance traveled through air is a little smaller than what Wilbur got. Therefore, I think the term 5/6 accounts for that slippage and had been determined empirically.

Now for the revolutionary increase in efficiency: I do not know what engines Wilbur used for comparison, but the low number he cites suggests that he used exceptionally poor specimens .