Automate turning on and off Twixtor effect in After Effects
You could try making two copies of your footage, then disabling all the interpolation features within Twixtor on one of the copies - ie:


Motion Vectors = None
Image Prep  = None
Motion Sensitivity = 0.00
Frame Interp = Nearest


Keyframe the "speed %" controls on both tracks exactly the same, but then remove the appropriate track when you're running at 20% (maybe by keyframing opacity between 0% and 100%).

Let me know if it works.