Is it rude to ask an unaffiliated student to leave the PhD student office?
I will mention that while Nicole Hamilton's answer is fundamentally correct—this is an issue for your advisor, as well as the other PI's in charge of the office space, not you—there is one issue worth elaborating on.

In particular, it is an issue if someone who isn't part of the department is using the department's facilities, because of the liability issue if something were to happen. For those grounds, it would be especially helpful to mention that one of the people coming into the space is not actually authorized to be there at all.