How do I install a custom recovery on a Note 3 from OS X?
All I know is to use Odin on windows. If heimdall doesn't work, you will have no chance from osx.

But there might be one solution:
Unpack your umage until you have a recovery.img file. Put it on your rooted device.
In the device shell as root type if=/pathtorecoverymimg of=/dev/block/platform/,,,,,/by-name/recovery
Sync,  reboot

To do a backup of the partition first is always a good idea. Just do it the other way round to get a backup in a file.

Some devices automatically restore recovery on boot. Try to reboot directly into recovery ( eventually disable automated recovery recovery using cwm recovery :)