Algorithm to group connected graphs
I'm going to leave off suggesting anything about the algorithm for now as I don't fully understand what you want from it. Instead, I'm going to talk about your Point class in laborious depth... 

public class Point
{
    public int X { get; set; }
    public int Y { get; set; }


I would argue that a point should be immutable - i.e. it doesn't change after you've created it. I would implement with full readonly properties:

private readonly int x;
public int X { get { return x; } }

private readonly int y;
public int Y { get { return y; } }

public Point(int x, int y)
{
   this.x = x;
   this.y = y
}


However, most people would let you off with private set auto properties:

public int X { get; private set; }


Let's look at your Equals implementations

public override bool Equals(Object obj)
{
    if (ReferenceEquals(null, obj)) return false;
    if (ReferenceEquals(this, obj)) return true;
    return obj.GetType() == GetType() &amp;&amp; Equals((Point)obj);
}

protected bool Equals(Point other)
{
    return X.Equals(other.X) &amp;&amp; Y.Equals(other.Y);
}


Firstly, bool Equals(Point other) -> that's IEquatable&lt;Point&gt; if made public so your class should implement that:

public class Point : IEquatable&lt;Point&gt;


Now, simplify the object.Equals override: 

public override bool Equals(Object obj)
{
    return Equals(obj as Point);
}


And sharpen up the IEquatable&lt;Point&gt;.Equals:

public bool Equals(Point other)
{
    return other != null
            &amp;&amp; other.X == X
            &amp;&amp; other.Y == Y;
}


We should also override the == and != operators so that we're sure we are consistent with out equality:

public static bool operator ==(Point left, Point right)
{
    if (object.ReferenceEquals(left, null))
    {
        return object.ReferenceEquals(right, null);
    }

    return left.Equals(right);
}

public static bool operator !=(Point left, Point right)
{
    if (object.ReferenceEquals(left, null))
    {
        return !object.ReferenceEquals(right, null);
    }

    return !left.Equals(right);
}


Then you have a nice immutable class for your Point with consistent equality behaviour... Phew!

Edit

As EBrown pointed out in the comments, Point is a perfect fit for being a struct. It's less than 16 bytes, immutable and (0, 0) is a perfectly valid value (i.e. unitialized).