Putting off PhD for a (potentially) better experience
It seems like you've laid out the pros and cons pretty well; only you can weigh them. But here is my analysis: 


  I feel my academic interests have little overlap with many others in the lab.


In my mind, this is the only valid con. It is difficult to do good work if you are not really interested in the project and don't share your advisor's goals. 


  The lab is very young (not sure if pro or con). In the start of Fall 2020, the oldest lab members will be the 2 students in their third year.


I see no real problem with this, given your positive comments about the PI.


  I fear that a rushed and partially-remote PhD will be a lower quality PhD.


By the last year(s) of your PhD, working remotely should not be a problem. 3-4 years in CS does not seem rushed if you already have a master's.


  I would like input on potential pros/cons that I am missing, other options, or perhaps even weighing in on what you think the right decision would be.


I think a missing pro is that you start right away. First, you don't know what the future holds; as they say, "A done PhD is better than a perfect PhD." And second, even if you had a guaranteed perfect PhD starting in three years, this means that you are delaying your professional life by three years. That's a long time -- only you can decide if it's worth it.