Is it unethical of me and can I get in trouble if a professor passes me based on an oral exam without attending class?
I feel like you're overthinking this. What you did seems an improvement to me over taking, say, bowling to fill a last credit, which isn't uncommon in undergrad. The professor gave you your grade and you're off to grad school. There's no reason to revisit it, with yourself or anyone else.

Also, it isn't your responsibility, as a student, to audit how a professor grades a class.  They may have even consulted with others before letting you do this; you don't know.  Don't make assumptions - you did what you did, and the professor decided it was enough to pass the class.

Put this behind you and focus on grad school.