How can I route traffic to different gateways by protocol
If you are prepare to do a lot of research and command line work then you should be able to do it using the built in routing and firewall software.

There is a good package IPNetRouterX which makes doing it a lot easier. They also claim to be faster since they operate at the kernel level. They offer a 21 day trial so you can see if it works well for you before paying the $100 they want for it. I have had some success with it for other uses in a couple of clients.