Video stabilization using Lucas-Kanade method
For video stabilization, you need to estimate motion vector with respect to a reference frame $F_{reference}$ (which let's assume it to be the first frame). Then your motion model should be determined, for simplicity let's assume it to be Translational motion. Therefore, we need to estimate motion in $x$ direction and motion in $y$ direction for each frame. Now you have two options to estimate the change. If you are interested to estimate it through optical flow you can do either these:


Estimate regular conventional optical flow from frame to frame
Estimate Optical flow of first frame to the $i$ $th$ frame.



  I accumulate during the whole process- and practically...


If you've chose the first, then Yes, you need to accumulate the optical flow, but I think it would be simpler to go for the second method.

As a hint, I suggest track a token or compute optical flow of a small patch of the image.

Edit:

I feel the problem is with your Warp function, the code below is part of video stabilization assignment in my class, try something like this sample. The translational parameters are estimated using token based tracking.

function [ ou_I ] = frame_translation_func( in_I,Tx,Ty,ou_RoWs,ou_CuLoMns,ou_layer)
ou_I = uint8(zeros(ou_RoWs,ou_CuLoMns,ou_layer));
for i2=1:ou_RoWs
    for j2=1:ou_CuLoMns
        if (i2 - Tx&lt;1) || (j2 - Ty&lt;1) || (i2 - Tx&gt;ou_RoWs) || (j2 - Ty&gt;ou_CuLoMns)
            ou_I(i2,j2,:)=0;
        else
            ou_I(i2,j2,:) = in_I(i2 - Tx,j2 - Ty,:);
        end
    end
end
end