VIewport View Rotates Weirdly Around Certain Object's Pivot
Oh, I just solved it myself. It seems my camera was locked to that object in the N menu in the "view" section. I just removed object from there.