Is there a mathematical formula to find the most efficient propeller?
Yes. The first good ones were published by A. Betz and L. Prandtl in 1919, and the latest substantial improvements were added by Larabee. Those formulas are the basis of Mark Drela's XROTOR. It will give you the chord and twist distribution over span for a prescribed speed, torque or thrust. It is best used for lightly loaded propellers - if you need anything with activity factors (power absorbed by the propeller per blade area) above that of purely subsonic airplanes, better consult prop manufacturer tables.

Generally, the most efficient propeller will be large, have two blades and will spin slowly. Depending on flight speed and desired thrust, you will need to reduce diameter, increase speed and add blades as required, at the price of lower efficiency.