Finding the best combinations between items of 2 arrays in a sequential manner
Negate the Y values.  Then you are looking for the Pareto-optimal frontier of the set of combinations.

There are standard algorithms for computing the Pareto-optimal frontier of any set.  In particular, see the last part of this answer for an overview of algorithms (the simple cull algorithm, divide-and-conquer, and hybrids between those two).  Those algorithms should be much more efficient than what you were considering before.  For example, the simple cull algorithm has $O(n^2)$ running time to compute the Pareto-optimal frontier of a set of size $n$ (or $O(nm)$ time, if you know the output is of length $m$).

Here's how you can use this in your setting.  First, compute the Pareto-optimal frontier of A and the Pareto-optimal frontier of B.  Then, enumerate all combinations of those two frontiers.  Finally, compute the Pareto-optimal frontier of that list of combinations.  If you use the simple cull algorithm, the running time will be at worst $O(n^4)$, where $n$ is the length of the input lists.  If the output has size $m$, the running time will be at worst $O(n^2 m)$.  In either case, the memory consumption will be $O(n^2)$.  The algorithms might perform better than that in practice, depending on your data.

If you want the output to be in sorted order, you can sort by Y value after computing the set.

I would recommend that you try these algorithms and see what the memory consumption looks like.  If the memory consumption is still too much, then you can let us know.