Where did the concept of fighting Nightmares come from?
There is no canon answer to this question. The Nightmares' form and function appear derivative of the Wraiths (Wraith-lite if you will). Both are manifestations of the negative energies produced by humanity's dark emotions; the Wraiths' original generation from "curses" is now moderated to the much more benign "nightmares". 

As to whose mind they came from (Homura's or the Incubators'), there is no way to know based on the information in the movie. The question at heart goes to the underlying basis of the "world" inside Homura's soul gem, which we are never told. While most of it is likely generated directly from her subconscious, there are other elements influencing it, such as the other magical girls who are external persons who became trapped in the barrier, and Kyubei and his comrades who are manipulating the barrier. 

Both Homura and Kyubei were familiar with the workings of the real city, and with the real Wraiths, so the idea could have come from either of them.