Image FFT radial component reduction
The two methods are equivalent over discrete images and in both of them it is "...tricky to select the correct boundary radii to really count the pixels that correspond to a certain frequency".

Whether "selecting pixels by distance" or converting from rectangular to polar coordinates, there will always be the problem of what do you do when you have to "land" between pixels. That is, when you have to "land" at an image site, within the acquired data range but between known values.

The answer here is of course "interpolation". Whether you are trying to select all harmonics that are $\approx 14.48765\ldots$ pixels from the centre of the image or you are converting $(x,y)$ pixel $(15,5)$ to its $(\rho, \theta)$ corresponding location of $(\approx 14.41479\ldots , \approx 81.36745\ldots)$, you still have the problem of "landing" between pixels.

You could of course $round(\cdot)$ everything, which would result in a nearest neighbour interpolation but that doesn't look very attractive. You are likely to have to use more complex interpolation schemes.

By the way, by convention, in digital image processing, the two dimensional discrete fourier transform's output is shifted by $\frac{N}{2}$ in both dimensions in order to bring the low frequencies to the centre of the "image". Therefore, you are likely to have to use fftshift too.

As far as the last part of your question is concerned, there should be no need for "scaling" if float or double data types are used, both to calculate the spectrum and its sum over different frequencies. If you are asking what sort of scaling you should do in order to be able to display the image (?), then, please note that you may have to use a logarithmic transformation on the spectrum values before you convert the image to something like unit8 to prepare it for display. Happy to clarify this later part more if more information emerges.

Hope this helps.