Moving vertex moves only the vertex, not the edges it is connected to
Looks like I was not pressing G AFTER selecting the vertex I wanted to move.
I was just selecting the vertex and moving it.
Pressing G then the vertex I want to move made it work as intended.