Recursively evaluate neighbors in a two dimensional grid
Avoid unnecessary calculations

It's a trivial performance improvement, but in 


        //Check south
        if(y - 1 &gt;= 0)
        {
            if(this.at(x, y - 1).getColor() == color)
                locateNeighbors(x, y - 1, color, matches);
        }

        //Check west
        if(x - 1 &gt;= 0)
        {
            if(this.at(x - 1, y).getColor() == color)
                locateNeighbors(x - 1, y, color, matches);
        }



changing to 

        //Check south
        if (y &gt; 0)
        {
            if (this.at(x, y - 1).getColor() == color)
            {
                locateNeighbors(x, y - 1, color, matches);
            }
        }

        //Check west
        if (x &gt; 0)
        {
            if (this.at(x - 1, y).getColor() == color)
            {
                locateNeighbors(x - 1, y, color, matches);
            }
        }


will be faster when the expression is false.  When it's true, it's unlikely to matter, as the compiler should optimize down to just one subtraction.  But when false, no subtractions are needed at all.  Of course, subtractions add a trivial amount of time in boards with just a few hundred squares.  

Note:  I also added brackets around the single statement form if statements.  While the compiler is fine with the other syntax, it tends to lead to a coding error that can be hard to diagnose.  For that reason, many programmers always use the block form.  

Recursion adds overhead

You could also try rewriting the recursive function iteratively.  Note that there is more state in a function call than you need to save.  The color doesn't change, and it always refers to the same matches Set.  Only the points change.  

You don't provide all the code, so I can't test it.  But it would look something like.  

        List&lt;Point&gt; candidates = new ArrayList&lt;&gt;();
        candidates.add(new Point(x, y));
        while (!candidates.isEmpty())
        {
            Point p = candidates.remove(candidates.size() - 1);
            if (matches.contains(p))
            {
                continue;
            }
            else
            {
                matches.add(p);
            }

            //Check north
            if (p.getY() + 1 &lt; columns)
            {
                if(at(p.getX(), p.getY() + 1).getColor() == color)
                {
                    candidates.add(new Point (p.getX(), p.getY() + 1));
                }
            }

            //Check east
            if (p.getX() + 1 &lt; rows)
            {
                if(at(p.getX() + 1, p.getY()).getColor() == color)
                {
                    candidates.add(new Point (p.getX() + 1, p.getY());
                }
            }

            //Check south
            if (p.getY() &gt; 0)
            {
                if (at(p.getX(), p.getY() - 1).getColor() == color)
                {
                    candidates.add(new Point (p.getX(), p.getY() - 1);
                }
            }

            //Check west
            if (p.getX() &gt; 0)
            {
                if (at(p.getX() - 1, p.getY()).getColor() == color)
                {
                    candidates.add(new Point (p.getX() - 1, p.getY());
                }
            }
        }


Note however that the compiler may optimize out the suboptimal portions of the original code.  In that case, this may not be faster.  

You also should question whether this is the place in the program where optimizations are most helpful.  I would think that removing gems, adding new ones, and redrawing the board would take more time.  Shaving off a few milliseconds here won't help you if you're stuck with extra seconds elsewhere.  As a general rule, you want to finish the program and then find where optimizations would be most useful.  

It's unnecessary to write this.at unless there is a conflict.  Simply saying at will almost always work.  

Declare as the interface

You define matches as a HashSet.  In Java, it is more common to define objects as interfaces rather than implementations.  So 


                HashSet&lt;Point&gt; matches = new HashSet&lt;Point&gt;();



would be written as 

                Set&lt;Point&gt; matches = new HashSet&lt;&gt;();


instead.  This allows you to change the implementation quickly in one place without affecting the rest of the program.  

Assuming that you are using Java 8, you don't have to write Point twice.  The compiler will figure out the appropriate type for you.