Measuring 0.1mm accuracy of very short distance (< 5 or 10mm)
There is a device called a Linear Variable Differential Transformer which is used for displacement measurements on the scale which you describe. However, in order to use this method, it would be necessary to have the measurement core slide in a frictionless manner on the rotating shaft, so that the radial displacement is obtained. This particular requirement, along with the associated expense, may preclude its use in this application.