Feasibility of rat-race as RF mixer without active devices
some sort of active device (sometimes diodes)

This suggests that you deem diodes to be active devices, so presumably a passive device means no diodes. In that case, a structure that's passive will not yield down-conversion for you. A mixer requires a non-linearity, like a diode, in order to function.
The important part of a mixer is the non-linearity which does the down-conversion, not the phase-shifting or adding networks like rat-races that give the mixer other useful properties like balancing some of the signals to specific ports.
This can be seen from the fact that you can make a mixer from a single diode, shunted across a pair of terminals. Apply two signals, one of which should be of adequate amplitude, and sum and difference frequencies appear. It's not a particularly convenient mixer, needing filters to separate the signals. Adding transformers, baluns, transmission lines, simplifies the signal application and extraction, and adds ports with inter-port isolation, but it's the diode or diodes that do the work.