What would be the correct way to calculate saturation in this case? I'm trying to make a HSV representation of the xyY color space. To calculate hue from an $(x, y)$ color, I use the angle between that color and red (wavelength 745) on the xy chromacity diagram, with white $(\frac{1}{3}, \frac{1}{3})$ as the center.

The saturation is the ratio between the distance between white and $(x, y)$, and white and a fully-saturated version of $(x, y)$ (which is the intersection between the line between $(\frac{1}{3}, \frac{1}{3})$ and $(x, y)$ and the edge of the chromacity diagram).

xy chromacity diagram:



The problem that I'm having is that when I plot my color space (at value=1) and compare it to the HSV representation of RGB, the saturation (distance from center) doesn't seem to match how "colorful" the color actually is:

My color space (saturation seems wrong):



HSV color space of RGB:



How should I calculate saturation instead?
There is unfortunately no good answer to this question. Simply it wont work. There is no good way to define colorful, it this context. Cie is trying to capture the physical measurement. It however does not succeed very well in relating the colors to each other.

Colors on the very outer arc represent spectral distributions of close to Dirac delta function. So one could construct a model that says that a color is very colorful when it is a Dirac delta. 

There is a unforeseen consequence of this definition though. Namely the magenta colors do not exist as Dirac Deltas. As these colors do not exist in the spectrum. So they consist of mixture of 2 wavelengths only. This would mean they are less colorful than most other colors.

Other problems

Unfortunately, xyY is not perceptually uniform. So a straight line on the xyY does not represent interpolations between 2 color mixtures. Therefore making a polar transformation means you will have different color bases on same coordinates. Also precieved color do not really move over to your model. To do this properly you would need to do a extremely sophisticated transformation.

There are many problems with converting color to a polar coordinates in that that is exactly contrary how vision works. White is also a bit problematic in this context. The distance to full saturated signal is different for each of the 3 different cones in the eye. Hell, even what is while depends on the surrounding colors and ambient color conditions. So aim afraid your trying to force a worldview that does not exist.

Finally

What would this be useful for?