How to find center of the circle if the data points are in curved coordinates system(Horizontal - AltAz)
Project your points on the unit sphere and compute the best fitting plane. The normal vector of the plane points towards the center of that circle. The radius of your circle will be equal to sqrt(1-d²) if d is the distance between the plane and the origin or acos(d) if you want the angle between the center and a point of the circle (since we're doing spherical geometry).
EDIT : do an orthogonal regression because if you don't, the z-axis could be favored over the others or vice-versa.
I have 6 points with their coordinates in the Cartesian plane, XY, placed on one side of a circle. Using the least square method it is relatively easy to fit a circle to those 6 points and to find the radius and the center of the fitted circle again in XY coordinates..
However, I also have Altitude Azimuth coordinates for those 6 points, because those points are on the sky, so I was wondering is it possible to fit a curve to those curved coordinates and then find the center of that circle.