How to jump to a location using mouse on iTerm/terminal?
At least in Terminal.app you can hold down option while clicking to jump within your current line (that actually means just the line, so if your current prompt is so long that it wraps to a second line of input you'll have to option-click to the beginning of the "second" line and then hop with the arrow keys to the "first" line where you can  option-click again.)