pic32 maximum external frequency while running from internal 8MHz RC oscillator
As @Brian Drummond suggests... don't even consider REFCLKO for an application which demands exacting timing specifications (while being over the core clock speed, AND at an odd interval, AND when running from the internal clock source!)

You likely haven't seen much about this because it just isn't done in practice. Yes, the device may boost an 8MHz reference to 40MHz internally - but as @Ignacio explains, REFCLKO can only approximate 11.2896MHz using ROTRIM - and erattically at that - so there will be jitter and lots of it. 

I know, it's tempting to want this one PIC to "do it all." But in this case, it really is a much better idea to use a $0.39 xtal, resonator, etc. for that other device, which you know will be within a specified ppm. Or better yet, use a multiple of this into the PIC directly, and feed (an even) division of it out to that device. Then everything is synched.

Also note that the spec for a typical PIC32MZ0512EFE064 CLKO stability is \$\pm0.25\$% and next page, the FRC stability is \$\pm\$5% and LPRC is even worse. So even if it could output a jitter-free 11.2896MHz, there is no guarantee it wouldn't be \$\pm\$5%, even 15% or more out if using the FRC or LPRC, and 0.5% if using a reference oscillator.