Why are portal area lamps slowing down my renders?
Portals have limitations — it depends on how "open" your scene is

As mentioned by @Leander, the blender manual says that portals are not good for "outdoor" scenes.


portals lamps provide diminishing returns as your indoor scene becomes too "open"
in other words, if your indoor scene has several very large windows, you may be better off without using the portals (because the cost of portals starts to outweigh the benefits)
in my experience, I found portals to be unfit (by a narrow margin) for an indoor scene where I have a room with a whole wall basically missing (huge floor-to-ceiling window), and also an open doorway in one of the intact walls (think of a garage with the big door open, and also a side-door open too)