SSH with ipv6 working only if I pass %en0 with my ipv6 address
When you want to connect to an unroutable address like link-local you need to specify the interface. This is because for unroutable addresses your system can't determine the outgoing interface by looking at the routing table.

Unroutable destinations include link-local addresses (fe80::) and link-local multicast (ff02::) and more.