GIF/SVG kind of compression for Video
I wonder if the original Citrix ICA or for that effect Nomachine's NX, Microsoft RDP or even VNC would be of your interest. From what I understand of ICA and NX that idea was really to be in between of window manager calls, giving it an ultimate access to 'small updates' opposed to complete framebuffers. By a bit of searching I found VNCREC. The biggest problem with the to 'vector' format is that most font rendering is very far from the original text, so you would need something very in between, maybe such as GTK Broadway which goes into HTML5.