title: Display filtered ffmpeg frame with GTK
body: <p>I want to display cropped and scaled frames using the ffmpeg api and I am using GTK+ 3 for the GUI components.  From following the <a href="http://jan.newmarch.name/LinuxSound/Diversions/Gtk/#heading_toc_j_11" rel="nofollow">this tutorial</a> and the <a href="http://ffmpeg.org/doxygen/trunk/filtering_video_8c-example.html" rel="nofollow">ffmpeg examples</a>, I am able to display unfiltered frames, though with some instability. The filtered frames does not display correctly at all. It mostly produces completely black output. I suspect that this is due to sws_scale() but I have not found out why this is happening. </p>
  
  <p>Using the "trivial" display from the ffmpeg examples I can confirm that the frame is being cropped and scaled properly.</p>
  
  <p>Running the code below I get a bunch of errors:</p>
  
  <pre><code>[swscaler @ 0x7fb58b025400] bad src image pointers
  [swscaler @ 0x7fb58b025400] bad dst image pointers
  </code></pre>
  
  <p>I also get this error:</p>
  
  <pre><code>[swscaler @ 0x7fd05c025600] Warning: data is not aligned! This can lead to a speedloss
  </code></pre>
  
  <p>I tried making a buffer that was 16 bit aligned, but it did not seem to have any effect on the result.</p>
  
  <p>This is how I decode the frames and apply the filters:</p>
  
  <pre><code>void decode(gpointer args) {
      int ret;
      AVPacket packet;
      AVFrame *frame = av_frame_alloc();
      AVFrame *filt_frame = av_frame_alloc();
      int got_frame;
  
      if(!frame || !filt_frame) {
          perror("Could not allocate frame");
          exit(1);
      }
  
      /* read all packets */
      while (1) {
          if ((ret = av_read_frame(fmt_ctx, &amp;packet)) &lt; 0)
              break;
          if (packet.stream_index == video_stream_index) {
              got_frame = 0;
              ret = avcodec_decode_video2(dec_ctx, frame, &amp;got_frame, &amp;packet);
              if (ret &lt; 0) {
                  av_log(NULL, AV_LOG_ERROR, "Error decoding video\n");
                  break;
              }
              if (got_frame) {
                  frame-&gt;pts = av_frame_get_best_effort_timestamp(frame);
                  /* push the decoded frame into the filtergraph */
                  if (av_buffersrc_add_frame_flags(buffersrc_ctx, frame, AV_BUFFERSRC_FLAG_KEEP_REF) &lt; 0) {
                      av_log(NULL, AV_LOG_ERROR, "Error while feeding the filtergraph\n");
                      break;
                  }
                  /* pull filtered frames from the filtergraph */
                  while (1) {
                      ret = av_buffersink_get_frame(buffersink_ctx, filt_frame);
                      if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
                          break;
                      if (ret &lt; 0)
                          goto end;
                      display_frame2(filt_frame, buffersink_ctx-&gt;inputs[0]-&gt;time_base);
                      av_frame_unref(filt_frame);
                  }
                  av_frame_unref(frame);
              }
          }
          av_free_packet(&amp;packet);
      }
  end:
      avfilter_graph_free(&amp;filter_graph);
      avcodec_close(dec_ctx);
      avformat_close_input(&amp;fmt_ctx);
      av_frame_free(&amp;frame);
      av_frame_free(&amp;filt_frame);
      if (ret &lt; 0 &amp;&amp; ret != AVERROR_EOF) {
          fprintf(stderr, "Error occurred: %s\n", av_err2str(ret));
          exit(1);
      }
  
  }
  </code></pre>
  
  <p>And this is how I display the frames.</p>
  
  <pre><code>void display_frame2(const AVFrame *frame, AVRational time_base) {
      GdkPixbuf *pixbuf;
      int64_t delay;
      AVFrame *filt;
      uint8_t *buffer;
      int num_bytes, i;
      buffer = NULL;
  
  
      filt = av_frame_alloc();
      num_bytes = avpicture_get_size(PIX_FMT_RGB24, dec_ctx-&gt;width, dec_ctx-&gt;height);
      buffer = (uint8_t *)av_malloc(num_bytes * sizeof(uint8_t));
      avpicture_fill((AVPicture *)filt, buffer, PIX_FMT_RGB24, dec_ctx-&gt;width, dec_ctx-&gt;height);
  
      if (frame-&gt;pts != AV_NOPTS_VALUE) {
          if (last_pts != AV_NOPTS_VALUE) {
              /* sleep roughly the right amount of time;
               * usleep is in microseconds, just like AV_TIME_BASE. */
              delay = av_rescale_q(frame-&gt;pts - last_pts,
                                   time_base, AV_TIME_BASE_Q);
              if (delay &gt; 0 &amp;&amp; delay &lt; 1000000)
                  usleep(delay);
          }
          last_pts = frame-&gt;pts;
      }
  
      sws_scale(  sws_ctx,
                  (uint8_t const * const *)frame-&gt;data,
                  frame-&gt;linesize,
                  0,
                  frame-&gt;height,
                  filt-&gt;data,
                  filt-&gt;linesize);
      pixbuf = gdk_pixbuf_new_from_data(  filt-&gt;data[0], GDK_COLORSPACE_RGB,
                                          0, 8, dec_ctx-&gt;width, dec_ctx-&gt;height,
                                          filt-&gt;linesize[0], NULL, NULL);
      gtk_image_set_from_pixbuf((GtkImage *)image, pixbuf);
      free( filt );
      free( buffer );
  
  }
  </code></pre>
  
  <p>EDIT: 
  After some more thought and experimentation I got the filtered frames to be displayed, albeit in SDL, not GTK+. I used the transcoding example from ffmpeg to see if I could re-encode the video with the filters, and that does indeed work. With that example I basically changed the filter being fed the filtergtaph and most of the work is already done. At this point all I am doing is to display the video using SDL as shown in danger's tutorial. The cropping filter creates a lot of artifacts but it is at least showing something.</p>
  
  <p>I have to do some more work to see if it will work with GTK+. I have not taken a detailed look at the differences between the above program and the one in the transcoding example, so I have not yet figured out why my old code does not display filtered frames. Both sets of code use sws_scale() but I get no errors with the new code, so that must mean that something is different. I will update this post once I make more progress.</p>
  
  <p>EDIT 2:
  Added a small compilable example that should work, as per @drahnr’s request. I have not had the chance to try out replacing GtkPixbuf.</p>
  
  <pre><code>#define _XOPEN_SOURCE 600 
  #include &lt;libavformat/avformat.h&gt;
  #include &lt;libavformat/avio.h&gt;
  #include &lt;libavfilter/avfiltergraph.h&gt;
  #include &lt;libavfilter/avcodec.h&gt;
  #include &lt;libavfilter/buffersink.h&gt;
  #include &lt;libavfilter/buffersrc.h&gt;
  #include &lt;libavcodec/avcodec.h&gt;
  #include &lt;libswscale/swscale.h&gt;
  #include &lt;libavutil/avstring.h&gt;
  #include &lt;libavutil/time.h&gt;
  #include &lt;libavutil/opt.h&gt;
  #include &lt;unistd.h&gt;
  
  #include &lt;gtk/gtk.h&gt;
  #include &lt;gdk/gdkx.h&gt;
  
  
  GtkWidget *image;
  GtkWidget *window;
  
  struct SwsContext *sws_ctx;
  char *filter_descr = "crop=100:100,scale=640:360";
  static AVFormatContext *fmt_ctx;
  static AVCodecContext *dec_ctx;
  AVFilterContext *buffersink_ctx;
  AVFilterContext *buffersrc_ctx;
  AVFilterGraph *filter_graph;
  static int video_stream_index = -1;
  
  static void open_input_file(const char *filename)
  {
      AVCodec *dec;
      avformat_open_input(&amp;fmt_ctx, filename, NULL, NULL);
      avformat_find_stream_info(fmt_ctx, NULL);
      video_stream_index = av_find_best_stream(fmt_ctx, AVMEDIA_TYPE_VIDEO, -1, -1, &amp;dec, 0);
      dec_ctx = fmt_ctx-&gt;streams[video_stream_index]-&gt;codec;
      av_opt_set_int(dec_ctx, "refcounted_frames", 1, 0);
      avcodec_open2(dec_ctx, dec, NULL);
  }
  
  
  static void init_filters(const char *filters_descr)
  {
      char args[512];
      AVFilter *buffersrc  = avfilter_get_by_name("buffer");
      AVFilter *buffersink = avfilter_get_by_name("buffersink");
      AVFilterInOut *outputs = avfilter_inout_alloc();
      AVFilterInOut *inputs  = avfilter_inout_alloc();
      AVRational time_base = fmt_ctx-&gt;streams[video_stream_index]-&gt;time_base;
      enum AVPixelFormat pix_fmts[] = { PIX_FMT_RGB24, AV_PIX_FMT_NONE };
      filter_graph = avfilter_graph_alloc();
      snprintf(args, sizeof(args),
              "video_size=%dx%d:pix_fmt=%d:time_base=%d/%d:pixel_aspect=%d/%d",
              dec_ctx-&gt;width, dec_ctx-&gt;height, dec_ctx-&gt;pix_fmt,
              time_base.num, time_base.den,
              dec_ctx-&gt;sample_aspect_ratio.num, dec_ctx-&gt;sample_aspect_ratio.den);
      avfilter_graph_create_filter(&amp;buffersrc_ctx, buffersrc, "in", args, NULL, filter_graph);
      avfilter_graph_create_filter(&amp;buffersink_ctx, buffersink, "out", NULL, NULL, filter_graph);
      av_opt_set_int_list(buffersink_ctx, "pix_fmts", pix_fmts, AV_PIX_FMT_NONE, AV_OPT_SEARCH_CHILDREN);
      outputs-&gt;name       = av_strdup("in");
      outputs-&gt;filter_ctx = buffersrc_ctx;
      outputs-&gt;pad_idx    = 0;
      outputs-&gt;next       = NULL;
      inputs-&gt;name       = av_strdup("out");
      inputs-&gt;filter_ctx = buffersink_ctx;
      inputs-&gt;pad_idx    = 0;
      inputs-&gt;next       = NULL;
      avfilter_graph_parse_ptr(filter_graph, filters_descr, &amp;inputs, &amp;outputs, NULL);
      avfilter_graph_config(filter_graph, NULL);
  }
  
  static void display_frame2(const AVFrame *frame, AVRational time_base) {
      GdkPixbuf *pixbuf;
  
      AVFrame *filt;
      uint8_t *buffer;
      int num_bytes;
      buffer = NULL;
  
      filt = av_frame_alloc();
      num_bytes = avpicture_get_size(PIX_FMT_RGB24, dec_ctx-&gt;width, dec_ctx-&gt;height);
      buffer = (uint8_t *)av_malloc(num_bytes * sizeof(uint8_t));
      avpicture_fill((AVPicture *)filt, buffer, PIX_FMT_RGB24, dec_ctx-&gt;width, dec_ctx-&gt;height);
      usleep(33670 / 4);
      sws_scale(  sws_ctx,
                  (uint8_t const * const *)frame-&gt;data,
                  frame-&gt;linesize,
                  0,
                  frame-&gt;height,
                  filt-&gt;data,
                  filt-&gt;linesize);
      pixbuf = gdk_pixbuf_new_from_data(  filt-&gt;data[0], GDK_COLORSPACE_RGB,
                                          0, 8, dec_ctx-&gt;width, dec_ctx-&gt;height,
                                          filt-&gt;linesize[0], NULL, NULL);
      gtk_image_set_from_pixbuf((GtkImage *)image, pixbuf);
      free( filt );
      free( buffer );
  
  }
  
  
  void decode(gpointer args) {
      int ret;
      AVPacket packet;
      AVFrame *frame      = av_frame_alloc();
      AVFrame *filt_frame = av_frame_alloc();
      int got_frame;
  
      while (1) {
          av_read_frame(fmt_ctx, &amp;packet);
          if (packet.stream_index == video_stream_index) {
              got_frame = 0;
              avcodec_decode_video2(dec_ctx, frame, &amp;got_frame, &amp;packet);
              if (got_frame) {
                  frame-&gt;pts = av_frame_get_best_effort_timestamp(frame);
                  if (av_buffersrc_add_frame_flags(buffersrc_ctx, frame, AV_BUFFERSRC_FLAG_KEEP_REF) &lt; 0) {
                      av_log(NULL, AV_LOG_ERROR, "Error while feeding the filtergraph\n");
                      break;
                  }
                  while (1) {
                      ret = av_buffersink_get_frame(buffersink_ctx, filt_frame);
                      if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
                          break;
                      // Display original frame
                      display_frame2(frame, buffersink_ctx-&gt;inputs[0]-&gt;time_base);
                      // Display filtered frame
                      // display_frame2(filt_frame, buffersink_ctx-&gt;inputs[0]-&gt;time_base);
                      av_frame_unref(filt_frame);
                  }
                  av_frame_unref(frame);
              }
          }
          av_free_packet(&amp;packet);
      }
  
  }
  
  static void realize_cb(GtkWidget *widget, gpointer data) {
      GThread *tid;
      tid = g_thread_new("video", decode, NULL);
  }
  
  static void destroy(GtkWidget *widget, gpointer data) {
          gtk_main_quit();
  }
  
  int main(int argc, char **argv)
  {
      av_register_all();
      avfilter_register_all();
      open_input_file(argv[1]);
      init_filters(filter_descr);
      sws_ctx = NULL;
      sws_ctx = sws_getContext(   dec_ctx-&gt;width, dec_ctx-&gt;height, dec_ctx-&gt;pix_fmt, dec_ctx-&gt;width, dec_ctx-&gt;height,
                                      PIX_FMT_RGB24, SWS_BILINEAR, NULL, NULL, NULL );
      av_dump_format( fmt_ctx, 0, argv[1], 0);
  
      gtk_init(&amp;argc, &amp;argv);
      window = gtk_window_new( GTK_WINDOW_TOPLEVEL );
      g_signal_connect(window, "realize", G_CALLBACK(realize_cb), NULL);
      g_signal_connect(window, "destroy", G_CALLBACK(destroy), NULL);
      gtk_container_set_border_width(GTK_CONTAINER(window), 10);
      image = gtk_image_new();
      gtk_widget_show(image);
      gtk_container_add(GTK_CONTAINER(window), image);
      gtk_widget_show(window);
      gtk_main();
      return 0;
  }
  </code></pre>