Is it possible to delete a monitor or change the name as shown in Display Preferences?
It turns out it's the dongle with which the monitors are connected. I have two different brands (Satechi and Anker) at home, and (Apple and Anker) at work. When I plug the monitors into the different dongles, it changes orientation. The reason the physical swap didn't work, was because I also unplugged the cables from the monitors in the office, and reconnected them to the different dongles. I tried unplugging the monitors from the dongles at home and swapping them, and and they indeed switched.

This is probably the most trivial of information, but it's at least answered!