Formula for calculating IIR filter coefficients for a high order low pass?
First, the equations you solve for the coefficients depend on what type of filter you're after. They also depend on how the equations are derived. For instance, a common case would be a Butterworth lowpass, converted from an analog prototype in the s-domain via the bilinear z transform. Converting a higher order lowpass with the bilinear transformation would yield a solution with more coefficients.

However, it's common practice to form higher order filters from second order sections in series, with a first order section if the filter order is odd. The responses of the individual filters multiply to give the overall higher order response. The reason for reducing a high-order filter to first and second order filter is that high-order filters are more sensitive to the limited precision of typical floating point fixed point math. The cascaded filters are all set to the same corner frequency, and the Q of each filter is set to a different values that, together, produce the intended overall Q.

Here's an article about how to produce higher-order Butterworth filters for first and second order sections, including a calculator that produces the Q values for filter orders up to 20:

Cascading filters

Here's a second-order filter calculator that you can use to generate coefficients for the individual filters:

Biquad calculator v2

Here is more detail on the bilinear z transform:

The bilinear z transform