How do I make non-buddhists help understand I operate with a different view?
As long as what you are doing is not immoral in my case I would go along with what the others are doing.

Main point is by doing this you should not accumulate more fabrication. One way is to reduce contact through it is not very effective, which you are trying now.

The more effective way is to go along with the crowed but what every you do be mind full of the sensation and its impermanence. It is reaction of clinging or aversion to the sensation created when you come in contact with something which causes fabrications. This is what you should actively avoid. Though you might not chase sense pleases you have to go with the flow in social contexts if you want to be a householder.