Disable upside down rotation
Use apps like Ultimate Rotation Control - with proper settings, it seamlessly integrates with system rotation toggle, and on top of that you can disable any rotation direction you don't want (most phones don't have upside-down rotation, have to be enabled this way), force apps that don't usually rotate to rotate, and set per-app rotation behavior. In your case, just select "Forced 3/4 Auto" and "Lock Current" in rotation choices.