How to get the transform amount from a screen vector and a direction vector?
I would not recommend to correlate the extrusion direction with its screen projection because of the asymptotic behaviour in case the axis points toward the screen or the resulting extrusion crosses the focal point of the camera.

One consistent way of doing it would be computing the extrusion height independently of the axis direction:


Find a virtual construction plane of your operation which is parallel to the screen. For example, in the real world, this plane would be a plane of your computer monitor screen reachable by hand as you sit at your computer.


Usually, you have some kind of central point which serves as a center of screen camera rotation as you navigate your viewport. You can draw your construction plane through this point.

Another way is drawing your construction plane through the intersection of the mouse cursor and your object as you start your extrusion operation.

Either way, the construction plane must be parallel to the screen and in front of your camera view for this method to work.

If you are using an orthogonal projection (camera without a Field of View) then you can draw the construction plane through any point.


Compute the length of the projection of mouse motion on the construction plane your figured out in point (1). This is your desired extrusion height.
Provide a means of moving the construction plane back and forth along the view axis, if the current viewport has FOV. You don't need to display and the move the construction plane if your viewport uses orthogonal projection.


Using this method your extrusion operation height would be visually consistent with what user expects with mouse motion, and the user would not experience any asymptotic and singularity issues as with the axis projection method.