Segmenting and labeling chessboards through computer vision With almost no experience/knowledge in computer vision, I am currently trying to segment and label the separate squares on a chessboard (i.e. a1 ... h8). My goal is to use the information to physically play chess with an AI using a webcam and a robotic arm.

I am writing the vision program in C# (using Emgu CV) for simpler programming and GUI implementation. As my first step, I attempted to detect squares by following this tutorial. 



As we can see, not all of the squares were detected and I believe some of the squares were detected more than once (as 91 boxes were created).

Questions


What are the factors to consider in determining the suitable values for Canny threshold and threshold linking? i.e gray.Canny(cannyThreshold, cannyThresholdLinking)
The squares in the original image appear as straight lines to us but to the computer, it is slightly tilted. What is the cause of this? Is it because of aliasing at the edges?
Why are most of the squares not detected? Could it be the lighting?
What are the proper techniques/methods I should investigate to solve this problem? I am only familiar with the keywords: Canny, Hough (for circles I believe) and Sobel at the moment.


Code

Image&lt;Bgr, Byte&gt; image = new Image&lt;Bgr, Byte&gt;(Emgu_CV.Properties.Resources.Chessboard).Resize(0.15, INTER.CV_INTER_LINEAR);
Image&lt;Gray, Byte&gt; gray = image.Convert&lt;Gray, Byte&gt;(); //.PyrDown().PyrUp() worsens detection

imageBox.Image = image;

double cannyThreshold = 180.0;
double cannyThresholdLinking = 150.0;   

Image&lt;Gray, Byte&gt; cannyEdges = gray.Canny(cannyThreshold, cannyThresholdLinking);

List&lt;MCvBox2D&gt; boxList = new List&lt;MCvBox2D&gt;();

using (MemStorage storage = new MemStorage())
    for (Contour&lt;Point&gt; contours = cannyEdges.FindContours(); contours != null; contours = contours.HNext)
    {
        Contour&lt;Point&gt; currentContour = contours.ApproxPoly(contours.Perimeter * 0.05, storage); // don't quite understand the purpose of *0.05
        if (contours.Area &gt; 250 &amp;&amp; contours.Area &lt; 10000)
        {
            if (currentContour.Total == 4)
            {
                bool isSquare = true;
                Point[] pts = currentContour.ToArray();
                LineSegment2D[] edges = PointCollection.PolyLine(pts, true);

                for (int i = 0; i &lt; edges.Length; i++)
                {
                    double angle = Math.Abs(edges[(i + 1) % edges.Length].GetExteriorAngleDegree(edges[i]));

                    if (angle &lt; 80 || angle &gt; 95)
                    {
                         isSquare = false;
                         break;
                    }

                    if (isSquare) boxList.Add(currentContour.GetMinAreaRect());                    
                } 
            }
        }     
    }

Image&lt;Bgr, Byte&gt; squareImage = image.CopyBlank();
foreach (MCvBox2D box in boxList)
    squareImage.Draw(box, new Bgr(Color.Yellow), 1);
outputBox.Image = squareImage;
I have once done something similar. You should give a try to OpenCV's checkerboard detection. There are also works on Sudoku square detection, which might give you a hint.

If these don't work for you, then I would suggest the following: Forget about detection the checkerboard. Initially, assume that every piece is in place and take the snapshot of the board (you can even manually place an initial board, doesn't matter). Then, whenever a piece is moved, grab a new image and apply a differencing of two images (this is somewhat like a background subtraction). Note that this way, you easily get the location of the moved piece. And since you know the piece and what was in the previous location beforehand, you get the complete information of the move.

Use a downward looking camera, just like you showed.