From How do I jump to the location of my last edit?I can press:`.to take my cursor to the position of last edit but:``doesn't take me to the last position of my cursor, as it says it should in the question / answer linked above.There is also `` which will bring you back to where the cursor was before you made your last jump. 
Cursor to last edit works, last position doesn't
cursor movement
That link addresses a different use case so it only partially applies.The single quote and single backtick  marks are not saved between sessions so there's no reason to expect `` to work its magic in your specific example.`` moves the cursor to the position before the last jump. Since you are moving your cursor with the arrows or hjkl there's no jump involved and thus no reason to expect `` to work its magic in the current session anyway.Contrary to the single quote and single backtick marks, the dot mark is saved between sessions so`.will effectively work within the current session and in future session.The move to last position mark you are looking for is the double quote. Like the dot mark it is saved between sessions and can be used at any time:`