Working in industry before pursuing a PhD in a different field
This answer may be valid only in the US.
The industry experience certainly won't be a plus, but as a minus it would probably be minimal. Can't predict precisely since every admissions committee is different and is composed of individuals.
But changing fields after a BS is pretty common here as is going back for a doctorate from the &quot;working world&quot;.
But, it will be essential, I think, that  you maintain contact with your current professors over that span so that they don't forget you and the (obviously) wonderful things you did as a student. You will still want good letters of recommendation from academics to pursue a doctorate. I think the supervisor of any physics research would be especially valuable to keep as a contact.
You could also, perhaps, take a graduate course or two as a &quot;non-matriculated&quot; student, which is permitted in some places. Or even some undergraduate physics courses that might be missing from your background.
But generally, look to what you will need to present in an application and work to assure that such things will still be available and valid.
And, the shorter the delay, the better.