title: Farneback optical flow - dealing with pixels out of view, pixels with wrong flow result, different size image
body: <p>I am writing my thesis and one part of the task is to <strong>interpolate between images to create intermediate images</strong>. The work has to be done in c++ using openCV 2.4.13.
  The best solution I've found so far is <strong>computing optical flow and remapping</strong>. But this solution has two problems that I am unable to solve on my own:</p>
  
  <ul>
  <li>There are pixels that should go <strong>out of view</strong> (bottom of image for example), but they do not.</li>
  <li>Some pixels do not move, creating a <strong>distorted result</strong> (upper right part of the couch)</li>
  </ul>
  
  <p>What has made the flow&amp;remap approach <strong>better</strong>:</p>
  
  <ul>
  <li>Equalizing the intensity. This i'm allowed to do. You can check the result by comparing the couch form (centre of remapped image and original).</li>
  <li>Reducing size of image. This i'm NOT allowed to do, as I need the same size output. Is there a way to <strong>rescale the optical flow</strong> result to get the bigger remapped image?</li>
  </ul>
  
  <p>Other approaches tried and <strong>failed</strong>:</p>
  
  <ul>
  <li>cuda::interpolateFrames. Creates incredible ghosting.</li>
  <li>blending images with cv::addWeighted. Even worse ghosting.</li>
  </ul>
  
  <p>Below is the code I am using at the moment. And images: <a href="https://www.dropbox.com/sh/gnppax0rrs7s0eo/AAA3OS2pGpzgu9r21iuF9nUla?dl=0" rel="nofollow noreferrer">dropbox link with input and result images</a></p>
  
  <p>int main(){</p>
  
  <pre><code>cv::Mat second, second_gray, cutout, cutout_gray, flow_n;
  second = cv::imread( "/home/zuze/Desktop/forstack/second_L.jpg", 1 );
  cutout = cv::imread("/home/zuze/Desktop/forstack/cutout_L.png", 1);
  cvtColor(second, second_gray, CV_BGR2GRAY);
  cvtColor(cutout, cutout_gray, CV_RGB2GRAY );
  
  ///----------COMPUTE OPTICAL FLOW AND REMAP -----------///
  cv::calcOpticalFlowFarneback( second_gray, cutout_gray, flow_n, 0.5, 3, 15, 3, 5, 1.2, 0 );
  cv::Mat remap_n; //looks like it's drunk.
  createNewFrame(remap_n, flow_n, 1, second, cutout );
  cv::Mat cflow_n;
  cflow_n = cutout_gray;
  cvtColor(cflow_n, cflow_n, CV_GRAY2BGR);
  drawOptFlowMap(flow_n, cflow_n, 10, CV_RGB(0,255,0));
  
  ///--------EQUALIZE INTENSITY, COMPUTE OPTICAL FLOW AND REMAP ----///
  cv::Mat cutout_eq, second_eq;
  cutout_eq= equalizeIntensity(cutout);
  second_eq= equalizeIntensity(second);
  
  cv::Mat flow_eq, cutout_eq_gray, second_eq_gray, cflow_eq;
  cvtColor( cutout_eq, cutout_eq_gray, CV_RGB2GRAY );
  cvtColor( second_eq, second_eq_gray, CV_RGB2GRAY );
  
  cv::calcOpticalFlowFarneback( second_eq_gray, cutout_eq_gray, flow_eq, 0.5, 3, 15, 3, 5, 1.2, 0 );
  cv::Mat remap_eq;
  createNewFrame(remap_eq, flow_eq, 1, second, cutout_eq );
  cflow_eq = cutout_eq_gray;
  cvtColor(cflow_eq, cflow_eq, CV_GRAY2BGR);
  drawOptFlowMap(flow_eq, cflow_eq, 10, CV_RGB(0,255,0));
  
  cv::imshow("remap_n", remap_n);
  cv::imshow("remap_eq", remap_eq);
  cv::imshow("cflow_eq", cflow_eq);
  cv::imshow("cflow_n", cflow_n);
  cv::imshow("sec_eq", second_eq);
  cv::imshow("cutout_eq", cutout_eq);
  cv::imshow("cutout", cutout);
  cv::imshow("second", second);
  
  cv::waitKey();
  
  return 0;
  </code></pre>
  
  <p>}</p>
  
  <p>Function for remapping, to be used for intermediate image creation:</p>
  
  <pre><code>void createNewFrame(cv::Mat &amp; frame, const cv::Mat &amp; flow, float shift, cv::Mat &amp; prev, cv::Mat &amp;next){
      cv::Mat mapX(flow.size(), CV_32FC1);
      cv::Mat mapY(flow.size(), CV_32FC1);
      cv::Mat newFrame;
      for (int y = 0; y &lt; mapX.rows; y++){
          for (int x = 0; x &lt; mapX.cols; x++){
              cv::Point2f f = flow.at&lt;cv::Point2f&gt;(y, x);
              mapX.at&lt;float&gt;(y, x) =  x + f.x*shift;
              mapY.at&lt;float&gt;(y, x) =  y + f.y*shift;
          }
      }
      remap(next, newFrame, mapX, mapY, cv::INTER_LANCZOS4);
      frame = newFrame;
      cv::waitKey();
  }
  </code></pre>
  
  <p>Function to display optical flow in vector form:</p>
  
  <pre><code>void drawOptFlowMap (const cv::Mat&amp; flow, cv::Mat&amp; cflowmap, int step, const cv::Scalar&amp; color) {
      cv::Point2f sum; //zz
      std::vector&lt;float&gt; all_angles;
      int count=0; //zz
      float angle, sum_angle=0; //zz
      for(int y = 0; y &lt; cflowmap.rows; y += step)
          for(int x = 0; x &lt; cflowmap.cols; x += step)
          {
              const cv::Point2f&amp; fxy = flow.at&lt; cv::Point2f&gt;(y, x);
              if((fxy.x != fxy.x)||(fxy.y != fxy.y)){ //zz, for SimpleFlow
                  //std::cout&lt;&lt;"meh"; //do nothing
              }
              else{
                  line(cflowmap, cv::Point(x,y), cv::Point(cvRound(x+fxy.x), cvRound(y+fxy.y)),color);
                  circle(cflowmap, cv::Point(cvRound(x+fxy.x), cvRound(y+fxy.y)), 1, color, -1);
                  sum +=fxy;//zz
                  angle = atan2(fxy.y,fxy.x);
                  sum_angle +=angle;
                  all_angles.push_back(angle*180/M_PI);
                  count++; //zz
              }
          }
  }
  </code></pre>
  
  <p>Function to equalize intensity of images, for better results:</p>
  
  <pre><code>cv::Mat equalizeIntensity(const cv::Mat&amp; inputImage){
      if(inputImage.channels() &gt;= 3){
          cv::Mat ycrcb;
          cvtColor(inputImage,ycrcb,CV_BGR2YCrCb);
          std::vector&lt;cv::Mat&gt; channels;
          cv::split(ycrcb,channels);
          cv::equalizeHist(channels[0], channels[0]);
          cv::Mat result;
          cv::merge(channels,ycrcb);
          cvtColor(ycrcb,result,CV_YCrCb2BGR);
          return result;
      }
      return cv::Mat();
  }
  </code></pre>
  
  <p>So to recap, <strong>my questions</strong>:</p>
  
  <ul>
  <li>Is it possible to <strong>resize Farneback optical flow</strong> to apply to 2xbigger image?</li>
  <li>How to deal with <strong>pixels that go out of view</strong> like at the bottom of my images (the brown wooden part should disappear).</li>
  <li>How to deal with <strong>distortion</strong> that is created because optical flow wasn't computed for those pixels, while many pixels around there have motion? (couch upper right, &amp; lion figurine has a ghost hand in the remapped image). </li>
  </ul>