Sweep & prune broadphase algorithm
An IEnumerable (and IOrderedEnumerable) represents a question, not an answer. axisList is not really a list -- it is a question asking &quot;What is the sequence of elements from _gameObjects ordered by X value?&quot;. So every time you do this:
            GameObject a = axisList.ElementAt(i);
            GameObject b = axisList.ElementAt(j);

You are answering the question &quot;What is the ith element in that ordering&quot;. Which means it needs to re-order the sequence on every iteration! Your outer loop ends up being hugely expensive (O(n^3 log(n)) if my math checks out); you certainly don't need to do that.
You should cache the result of the ordering and re-use it in your loop. Change this
IOrderedEnumerable&lt;GameObject&gt; axisList = _gameObjects.OrderBy(x =&gt; x.LeftRightMost.X);

to this:
var orderedByX = _gameObjects.OrderBy(x =&gt; x.LeftRightMost.X).ToList();

.ToList() will evaluate the OrderBy query and return a flattened &quot;answer&quot; that you can keep reusing, and as a bonus it has O(1) lookup so ElementAt is fast -- you could even change ElementAt(i) to [i] to make that more clear.