Optimizing Floodfill algorithm implementation as much as possible
Monitor.IsEntered

This call basically makes no sense. Monitor.IsEntered is used for coordination/mutual exclusion between threads. There are no threads here, and even if there were, there is no reason to lock on i: it is a counter. It isn't even a meaningful target for Monitor.IsEntered because it isn't an object.

There is no apparently threading going on here, so you can confidently remove all the locking stuff (including Interlocked.Increment). This may noticeably improve performance, since Monitor.IsEntered is non-trivial, and the call will incur boxing (so at worse this will improve the memory characteristics of your code).

If, for some reason, you are reading i from a different thread while the flood-fill is running, then you might want to use Interlocked methods to provide some notion of up-to-dateness.... but unless you are using that low-latency synchronisation (in which case it is a bad choice for doing so) you won't in practice need any manual memory barriers here. If it is not the case that you want to access i before this has finished then keep track of it as a 'normal' variable, and then set it as out at the end (if it is needed at all). out parameters can be abused, and may interfere with the compilers ability to perform optimisations.

100000

What is this?



if (i &gt; 100000)
    break;


This is a magic number threshold. This says "if I happen to look at some mysterious number of entries, then I will give up and provide a mangled result". This is terrible, especially so because it isn't documented. If you want a cut-out, then have the threshold passed as a parameter, and document it in the inline documentation. It's good that you have inline documentation, but it isn't very helpful at the moment, failing to explain what any of the parameters are beyond suggesting that target will be 'replaced' (what does that mean?).

Furthermore, if it does cut out, then the consumer should be informed. You could throw a FloodfillCutoffException, or return a bool indicating that it did not complete, or something else.

HashSet/Stack

You should probably be using a Stack. HashSet.Contain is efficient (for what it does), but you don't need it. (This is assuming that target.Equals(replacement) is always false, but if that isn't the case then your code is broken anyway, because it doesn't keep track of pixels it has dequeued)

Excepting the first pixel, if you set source[targetIndex] = replacement when you queue it then you don't need to check if it is already in the HashSet: you only need to check that source[targetIndex].Equals(target), which you have to do anyway. The probably with your non-HashSet code wasn't the Stack, it was how you were using it.

With a stack, the code will look mostly the same, only you'll be Poping from the stack (instead of using First() and Remove(T). It should be much faster, because it doesn't lean on HashSet, which while being 'kind of O(1)' isn't going to outperform a single array lookup which probably needs to be done anyway.

Indexing

I'm guessing Pn is static, and looks something like this:

static int Pn(int x, int y, int width =&gt; x + y * width;


IEquatable&lt;T&gt;

It's good that you are using this.

Naming

Most of the names are cryptic. This is not ideal. It isn't clear why you use P rather than Pn. i and _i are completely unrelated (and both poor names). target and targetIndex are mostly unrelated.

Code

Here is a (completely untested) rework of your code I threw together in a hurry incorporating some of the points above, with some added comments.

public static void FloodFill&lt;T&gt;(this T[] source, int x, int y, int width, int height, T target, T replacement, int cutoff, out int explored)
    where T : IEquatable&lt;T&gt;
{
    // code can't handle this, so throw
    if (target.Equals(replacement))
    {
        throw new ArgumentException("Target and Replacement cannot equate");
    }

    // do start manually, and exit quickly if appropriate
    int start = P(x, y, width, height);
    if (!source[start].Equals(target))
    {
        explored = 0;
        return;
    }

    // stack is not a great name, but I'm in a hurry right now...
    Stack&lt;int&gt; stack = new Stack&lt;int&gt;();
    stack.Push(start);

    int count = 1;
    while (queue.Count &gt; 0)
    {
        int _i = stack.Pop(),
            _x = _i % width,
            _y = _i / width;

        for (int offsetX = -1; offsetX &lt; 2; offsetX++)
            for (int offsetY = -1; offsetY &lt; 2; offsetY++)
            {
                // do not check origin or diagonal neighbours
                if (offsetX == 0 &amp;&amp; offsetY == 0 || offsetX == offsetY || offsetX == -offsetY || -offsetX == offsetY)
                    continue;

                int targetIndex = Pn(_x + offsetX, _y + offsetY, width); // This is already inverted that's why we don't use F.P(...)
                int _tx = targetIndex % width,
                    _ty = targetIndex / width;

                // skip out of bounds point
                if (_tx &lt; 0 || _ty &lt; 0 || _tx &gt;= width || _ty &gt;= height)
                    continue;

                // if pixel matched target, reset it and queue
                if (source[targetIndex].Equals(target))
                {
                    source[targetIndex] = replacement;

                    stack.Push(targetIndex);
                    count++;
                }
            }

        if (i &gt; cutoff)
            throw new Exception("FloodFill passed cutoff");
    }

    // report the number of cells explored
    explored = count;
}