Magnetic flux outside a solenoid perpendicular to the direction of axis of solenoid
The Biot-Savart law allows you to calculate the flux density at any point from any shaped coil you choose to describe: -





Pictures taken from here

So, if you can mathematically derive a relationship between point p (at radius r) and the shape of the coil, you are in business. The trick, of course, is integrating all the infinitesimally short lengths of wire carrying the current for one turn then, repeating this for the length of the solenoid. It can get really tricky of course.