PF set condition for not equal to a list of ports for MacOS
You can make the inverse command logic. 

First Block all, then open just the ones you want: 

The result is the same you tried by negating all except those chosen to be opened. :)

set skip on lo
port_pass = "{ 80 66 53 22 }"
block all
pass out on en0 proto { tcp, udp } to any port $port_pass keep state