Magnets in Transformers Design
Ferrite has a low reluctance (basically the magnetic equivalent of resistance) which means that magnetic fields flow real easily through it. I don't know the magnetic permeability of Neodymium offhand (permeability is 1/reluctance, the magnetic equivalent of conductance) but if it's some small value (as JonRB suggested), then it'd act as if it wasn't there (high reluctance = high resistance = open circuit) and would offer no benefit over an air core. Ferrite also has the nice advantage of low hysteresis losses, hysteresis loss is the energy wasted in magnetising and demagnetising the core as the voltage/current in the coils change, NdFeB magnets magnetise real well (and like to stay magnetised, they have real good magnetic hysteresis) so that's a fail right there. There's also the slight problem of Neodymium being somewhat rarer (i.e. expensive) than the zinc/manganese/iron/nickle/cobalt ceramic composites used in making most ferrites.