Browser Custom Protocol unsafe
It turns out this was much simpler than I thought (as usually happens in software development). Somehow I didn't catch this until user testing, which is what threw me in the direction of thinking it was a Mac issue. It is not a Mac issue.

I'm using the AngularJS framework and one of its "features" is to mark unknown protocols with an unsafe prefix. I used this answer to configure it to recognize mine and now all is well.