Are there reasons *not* to have a copper-pour ground plane on a PCB?
Ground planes in general are almost always a good thing, but if used incorrectly can actually hurt the quality of your board.

A typical board like you have here would have 1 layer dedicated to be a ground pour only with no traces running on it. However, it sounds like you are wanting to make your top layer have a ground pour so that you don't have to remove all of that extra copper. Doing a ground pour on a layer with a lot of traces is not really a ground plane at all, rather you can think of it as a ground trace with varying sizes running all around your board. It is hard to say if it will actually hurt the signal integrity of the design, but I can say for certain that it will not provide the same benefit that a ground plane will.

Typically when I see milled boards like this, the copper will be left unconnected on the unused areas of board. This provides a benefit of knowing that if you accidentally short one line to the unused copper, you don't get a hard short to ground that can kill some ICs. This can also be a negative though as accidentally shorting to a large unused piece of copper can turn into a nice antenna and pick up noise that you may have a hard time hunting the source of.

I realize my answer may not be a direct answer to what you are wanting to know, but it is very difficult to predict what configuration will be best for you. But, if it were my design, I would go ahead and just leave the extra copper on the board, but leave it disconnected from everything.