To continue or quit my PhD in physics after 4yrs？
We only have half the story. What you describe seems outrageous (particularly the dinner part!), but I imagine if your professor had described the situation, he would characterize it quite differently (I have one socially-awkward but useful student and one whiny, unproductive student...just as an example, not saying you're actually whiny or unproductive). Given this, I'm going to avoid criticizing any of you.

In any case, the current situation is clearly untenable. A couple of options:


Talk to your advisor. Send an e-mail asking for a formal meeting to discuss your status. In the meeting, you might start by saying something nice about finding his work interesting and regretting that things haven't gone as well as you would like. Then, ask him when he sees you graduating and whether he will give you interesting work going forward. If you can smooth things over and make a good plan going forward to work independently, maybe you can finish out your last two years. If not, better to find out now so you can cut your losses.
Get a new advisor. In a small group of ~10 professors, this may be difficult. I would start by talking to the department chair or a trusted professor
Transfer to a new university. I realize this may be difficult with your visa restrictions. 
Leave with an MS, if this is possible. If you are OK going into industry, an MS is not so bad (and you can always apply to PhD programs after leaving with an MS).