Use Arduino(5V) to switch a 3.3V
I'd recommend a bistable mechanical relay. Advantages:


Electrically isolate your Arduino and HDMI switch
Holds either state without power (especially useful for noise suppression)
No voltage drop across closed contacts; exactly replicates electrical properties of the switch you'd replace


A quick search revealed a candidate (PB1618-ND) with bistable operation perfect for an Arduino; a 5V-rated coil draw of 12 mA. The contacts are rated for 1A @ 125VDC, which should be plenty for a little switch.