Edge following using Hough transform
1) Matching lines across multiple views is a common research problem and is reasonably well studied:
If you have the end points for example : http://cmp.felk.cvut.cz/~werner/software/lmatch/lmatch_memo.pdf

Even tough this deals with more geometric properties, if your scenes are well conditioned, you could as well use the gradient/intensity information to match the lines. 

Here is a full source code on multiview line matching from monocular setups:
https://github.com/manhofer/Line3Dpp

You might also like to start with that.

2) Using a KALMAN filter on the lines would be a good idea. There are also more sophisticated algorithms. For example, in each frame you might want to minimize cost function to obtain the subpixel line, which has better resemblence to the line in the previous frame.

You can also track the camera pose and from there hypothesize the lines.

You try to avoid SLAM, but maybe, triangulating the lines and obtaining 3D structure would be the best in reducing the errors.