Would using compressionless brake housing for derailleur be a problem?
It's fine as long as there are no unusually tight internal routing frame fittings that will only play nice with a 4mm shift housing, rather than a 5mm (all modern brake housings, compressionless or coil, are 5mm). I can't think of anything like that I've seen offhand, but the array of housing ports and guides that frames have come with is now pretty vast, so there are probably examples out there.

I've done this many times using Odyssey Linear Slic and similar BMX cable kits for funky color scheme purposes. Also from observing how low-friction they can be even when wrapped around bars multiple times, I personally also suspect that braided (aka compressionless brake) housings work a little better than everything else in routing schemes with tight housing bends. I don't have any proof, but I've used them successfully for this purpose on at least a couple bikes with horribly designed internal routing.

The bigger cable/liner gap does not make any difference in practice. The cable is following the path of least resistance and is under tension. Remember that standard 4mm and 5mm shift housings both work totally fine, and they have different amounts of gap. It's uncommon to see 5mm compressionless shift housing on new bikes now, but it was very common for a long time.

Ferrule outside diameter is the same across all the common ferrule types (4mm shift, 5mm shift, 5mm brake, and 5mm compressionless/braided anything). Compressionless brake kits and rolls usually come with special stepped-down ferrules, like the Jagwire POP ferrule, that are intended to be used on brakes and levers that were designed to not use a brake housing ferrule at all, such as many caliper brakes and brifters. Ferrules with an appropriately thick end surface are required on all types of compressionless housings because other wires will blow through over time otherwise. If the extra length of the stepped ferrules is causing an awkward bend, and the part in question can take a normal OD ferrule, it's fine to use a standard 5mm brass or aluminum shift ferrule instead. (But not a 5mm brake ferrule, because the thin end surface won't be strong enough to keep the wire blow-through problem from happening over time.)