How BLDC motora coils and magnets are mechanically arranged
The canonical realization is what's in the picture: three coils for every two magnets.  In this canonical realization, the magnets and coils are arranged so that the effect of each one varies in strength sinusoidally.

In the circular case, in theory, the two magnets set up a field that is constant and pointing in one direction with respect to the rotor.  Then -- again in theory -- the three coils are energized in such a way that their contribution to the field points in just one direction.  As long as you keep those fields perpendicular, the motor torque is maximized.

In a linear motor, you just "unwind" the rotor and stator, and repeat sections to get what you see (in a real motor you might have a longer strip of magnets).

The realization that I see in BLDC motors for model aircraft use is to have four magnets per three coils.  This actually doesn't work when the fields vary sinusoidally -- but if the fields generated by the magnets and the coils are roughly rectangular, with sharp changes in field strength at the edges, then not only can torque (or force, in the linear case) be had, but you get more change in magnetic field strength per degree of motor rotation -- this works out well for "outrunner" style brushless motors that are designed to rotate comparatively slowly and allow the system to work without a gearbox.

As to calculating the optimal combination of geometry and whatnot -- there is none.  Or at least, mine may be different from yours.  There's a huge tradeoff between cogging torque, motor efficiency, the sensitivity of the motor to dimensional variations over age, manufacturing variations and mechanical stress, etc., etc.

In general, the way a motor designer would go about this would be to use finite element modeling tools (FEM) to predict the performance of any given geometry with regard to the above-mentioned parameters.  They would probably do this with input from a mechanical designer on how much the dimensions of the motor could be expected to change due to manufacturing variance and environmental effects.

I can't speak to motor design per se., but it'll end up being a complicated process, and it's not one that's cut and dried -- various engineers have their own favorite issues to stress, different customers have their own needs, and various companies (and product line managers within companies) have different views of what customers want and/or need.