Is it possible to get higher resolutions by using high resolution encoders and custom firmware?
The mistake in your reasoning is assuming no microstepping. Most 3D printers use 16 microsteps, and in my experience with both cheap A4988 drivers and nice TMC2209 drivers, microstepping is quite accurate. As part of an answer to a question I asked, you can see a test print showing single-microstep features. My motors have 1.8° step angle, yielding 3200 steps per rotation at 16 microsteps, or 12.5 microns of linear movement per microstep. With 0.9° step angle you could get it down to half that, and you could probably halve it again going to 32 microsteps.
Even if you can't get it as good as your 4 microns with stepper motors though, at 12.5 micron positioning resolution you're already to the point where extrusion error is going to play a much bigger role in dimensional accuracy than toolhead positioning error does. Going past that with FDM requires high resolution extruder axis movement, closed-loop control with a precise filament diameter sensor, direct drive with minimal distance between the extruder gear and nozzle, etc.