How could I address this dead space between my stair case and this wall?
I think I'd go with part of my original answer, but leave the railing as-is. Take the balusters out for all treads above the new wall. Replace the treads and risers with longer ones that extend all the way to the wall, closing the gap. Replace the balusters. Done deal. I wouldn't even mess with the framing. 

You'll have a little bit of wasted tread space along the wall, but it'll be much less odd than a deep gap. The lowest riser would notch over the wall over the new doorway. It looks like a very clean situation from here.



[original answer]

Were it my home I'd return the handrail into the wall and widen the five upper stairs to the wall, filling in as appropriate above the bathroom door casing to the first wide tread.

I can't see what's going on with the soffit/ceiling above the lower hall, but you might also be able to extend the wall above the bathroom door to meet the stairs, and either terminate the railing there or run it past/above the wall, which will either have a ledge at the top or run all the way to the high ceiling.