Is it permissible to install a low-voltage fixture over an existing ceiling box?
You're overthinking it

All you need here is some insulating means to separate the 120V circuit from the 12V circuit -- while they don't make box barriers in the configuration you're after, a non-metallic cover plate of the appropriate size will do just as well provided you can route the cable around it.  You'll need to use slightly longer screws than what came in the box, though, as you'll have to mount the smoke detector base through the cover plate holes and into the box mounting holes.