Vertices behaving like parented bone is rotating when neighbouring bones rotate
Okay so I fixed it, I think. I just dissolved the lower thigh bone and subdivided the upper one and gave the new one the same name as the old lower thigh bone so the weights would sync up.

I still have no idea what the problem was but it's gone now.