Accurately measuring relative distance between a set of fiducials (Augmented reality application) Let's say I have a set of 5 markers. I am trying to find the relative distances between each marker using an augmented reality framework such as ARToolkit. In my camera feed thee first 20 frames show me the first 2 markers only so I can work out the transformation between the 2 markers. The second 20 frames show me the 2nd and 3rd markers only and so on. The last 20 frames show me the 5th and 1st markers. I want to build up a 3D map of the marker positions of all 5 markers.

My question is, knowing that there will be inaccuracies with the distances due to low quality of the video feed, how do I minimise the inaccuracies given all the information I have gathered?

My naive approach would be to use the first marker as a base point, from the first 20 frames take the mean of the transformations and place the 2nd marker and so forth for the 3rd and 4th. For the 5th marker place it inbetween the 4th and 1st by placing it in the middle of the mean of the transformations between the 5th and 1st and the 4th and 5th. This approach I feel has a bias towards the first marker placement though and doesn't take into account the camera seeing more than 2 markers per frame.

Ultimately I want my system to be able to work out the map of x number of markers. In any given frame up to x markers can appear and there are non-systemic errors due to the image quality.

Any help regarding the correct approach to this problem would be greatly appreciated.
You can use structure from motion type of algorithm to estimate the camera pose from the environment, not from the markers and then fuse this camera pose with the marker poses in order to detect the locations of the markers accurately. Knowing the extrinsic pose of your camera (by SFM), you can triangulate all the 3D positions. 

For pose estimation, 5-point methods are usually more accurate than 8-point algorithm.

Presumably, you should do further bundle adjustment so that the overall accuracy increases.