Use magnetic field to estimate distance - best sensor type
Any ideas?


Maybe (just maybe) you could engineer the equivalent to a linear voltage displacement transducer (LVDT). It uses three coils and a linear moving target to estimate position: -



Conceptually it is the same as a food-industry three coil metal detector: -



The thing about the metal detector is that the aperture design (the hole in which the product travels) is big enough to allow whatever mechanism you have planned to fit in it but, bear in mind that only the "target" must be metal.

You would excite the centre coil at tens to hundreds of kHz and look for the phase of the signal on the receive coils. At dead centre the receive signal is zero and the left side would produce the opposite phase signal to right-side.


  I could attach a small magnet to the moving element


A piece of ferrite would work nicely.