title: Issue with configuration of cv2.VideoWriter and GStreamer
body: <p>I am having trouble setting up a GStreamer pipeline to forward a video stream over UDP via OpenCV. I have a laptop, and an AGX Xavier connected to the same network. The idea is to forward the webcam video feed to AGX which will do some OpenCV optical flow estimation on the GPU (in Python), draw flow vectors on the original image and send it back to my laptop. Up until now, I can configure two pipelines. As a minimum example, I have made two bash scripts and a Python script that ideally would function as pass-through over OpenCV's VideoCapture and VideoWriter objects.</p>
  <p>servevideo.bash:</p>
  <pre><code>#!/bin/bash
  
  gst-launch-1.0 v4l2src device=[device-fd] \
      ! video/x-raw, width=800, height=600, framerate=24/1 \
      ! jpegenc ! rtpjpegpay ! rtpstreampay \
      ! udpsink host=[destination-ip] port=12345
  </code></pre>
  <p>receivevideo.bash:</p>
  <pre><code>#!/bin/bash
  
  gst-launch-1.0 -e udpsrc port=12344 \
      ! application/x-rtp-stream,encoding-name=JPEG \
      ! rtpstreamdepay ! rtpjpegdepay ! jpegdec \
      ! autovideosink
  </code></pre>
  <p>If I run these two scripts on either the same computer or on two different computers on the network, it works fine. When I throw my Python script (listed below) in the mix, I start to experience issues. Ideally, I would run the bash scripts on my laptop with the intended setup in mind while running the Python script on my Jetson. I would then expect to see the webcam video feed at my laptop after taking a detour around the Jetson.</p>
  <p>webcam_passthrough.py:</p>
  <p>#!/usr/bin/python3.6</p>
  <pre><code>import cv2
  
  video_in = cv2.VideoCapture(&quot;udpsrc port=12345 ! application/x-rtp-stream,encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! appsink&quot;, cv2.CAP_GSTREAMER)
  video_out = cv2.VideoWriter(&quot;appsrc ! videoconvert ! jpegenc ! rtpjpegpay ! rtpstreampay ! udpsink host=[destination-ip] port=12344&quot;, cv2.CAP_GSTREAMER, 0, 24, (800, 600), True)
  
  while True:
      ret, frame = video_in.read()
  
      if not ret: break
  
      video_out.write(frame)
      cv2.imshow('Original', frame)
  
      key = cv2.waitKey(1) &amp; 0xff
      if key == 27: break
  
  cv2.destroyAllWindows()
  video_out.release()
  video_in.release()
  </code></pre>
  <p>With the following Python script, I can visualise the frames via <code>cv2.imshow</code> received from the pipeline set up by the <code>servevideo.bash</code> script. So I think my problem is connected to how I am setting up the VideoWriter <code>video_out</code> in OpenCV. I have verified my two bash scripts are working when I am relaying the webcam video feed between those two pipelines created, and I have verified that the <code>cv2.VideoCapture</code> receives the frames. I am no expert here, and my GStreamer knowledge is almost non-existent, so there might be several misunderstandings in my minimum example. It would be greatly appreciated if some of you could point out what I am missing here.</p>
  <p>I will also happily provide more information if something is unclear or missing.</p>
  <p><strong>EDIT:</strong>
  So it seems the intention of my minimum example was not clearly communicated.</p>
  <p>The three scripts provided as a minimum example serve to relay my webcam video feed from my laptop to the Jetson AGX Xavier who then relays the video-feed back to the laptop. The <code>servevideo.bash</code> creates a GStreamer pipeline on the laptop that uses v4l2 to grab frames from the camera and relay it on to a UDP socket. The <code>webcam_passthrough.py</code> runs on the Jetson where it &quot;connects&quot; to the UDP socket created by the pipeline running on the laptop. The Python script serves a passthrough which ideally will open a new UDP socket on another port and relay the frames back to the laptop. The <code>receivevideo.bash</code> creates yet another pipeline on the laptop for receiving the frames that were passed through the Python script at the Jetson. The second pipeline on the laptop is only utilised for visualisation purpose. Ideally, this minimum example shows the &quot;raw&quot; video feed from the camera connected to the laptop.</p>
  <p>The two bash scripts are working in isolation, both running locally on the laptop and running <code>receivevideo.bash</code> remotely on another computer.</p>
  <p>The <code>cv2.VideoCapture</code> configuration in the Python script also seems to work as I can visualise the frames (with <code>cv2.imshow</code>) received over the UDP socket provided by the <code>servevideo.bash</code> script. This is working locally and remotely as well. The part that is causing me some headache (I believe) is the configuration of <code>cv2.VideoWriter</code>; ideally, that should open a UDP socket which I can &quot;connect&quot; to via my <code>receivevideo.bash</code> script. I have tested this locally and remotely but to no avail.</p>
  <p>When I run <code>receivevideo.bash</code> to connect to the UDP socket provided by the Python script I get the following output:</p>
  <pre><code>Setting pipeline to PAUSED ...
  Pipeline is live and does not need PREROLL ...
  Setting pipeline to PLAYING ...
  New clock: GstSystemClock
  </code></pre>
  <p>This does not seem wrong to me, I have tried to run the different scripts with GST_DEBUG=3 which gave some warnings, but as the pipeline configurations are basically the same in the bash scripts and for the cv2 <code>VideoCapture</code> and <code>VideoWriter</code> I do not add much value to those warnings. As an example I have included one such warning below:</p>
  <pre><code>0:00:06.595120595  8962      0x25b8cf0 WARN              rtpjpegpay gstrtpjpegpay.c:596:gst_rtp_jpeg_pay_read_sof:&lt;rtpjpegpay0&gt; warning: Invalid component
  </code></pre>
  <p>This warning is printed continuously running the Python script with <code>GST_DEBUG=3</code>. Running the <code>receivevideo.bash</code> with the same debug level gave:</p>
  <pre><code>Setting pipeline to PAUSED ...
  Pipeline is live and does not need PREROLL ...
  0:00:00.013911480  9078 0x55be0899de80 FIXME           videodecoder gstvideodecoder.c:933:gst_video_decoder_drain_out:&lt;jpegdec0&gt; Sub-class should implement drain()
  Setting pipeline to PLAYING ...
  New clock: GstSystemClock
  </code></pre>
  <p>I hope my intention is clearer now, and as I already pointed out I believe something is wrong with my <code>cv2.VideoWriter</code> in the Python script, but I am no expert and GStreamer is far from something that I use every day. Thus, I may have misunderstood something.</p>
  <p><strong>EDIT 2:</strong>
  So now I have tried to split the two pipelines into two separate processes as suggested by @abysslover. I still see the same result, and I still have no clue why that is. My current implementation of the Python script is listed below.</p>
  <p>webcam_passthrough.py:</p>
  <pre><code>#!/usr/bin/python3.6
  
  import signal, cv2
  from multiprocessing import Process, Pipe
  
  is_running = True
  
  def signal_handler(sig, frame):
      global is_running
      print(&quot;Program was interrupted - terminating ...&quot;)
      is_running = False
  
  def produce(pipe):
      global is_running
      video_in = cv2.VideoCapture(&quot;udpsrc port=12345 ! application/x-rtp-stream,encoding-name=JPEG ! rtpstreamdepay ! rtpjpegdepay ! jpegdec ! videoconvert ! appsink&quot;, cv2.CAP_GSTREAMER)
  
      while is_running:
          ret, frame = video_in.read()
          if not ret: break
          print(&quot;Receiving frame ...&quot;)
  
          pipe.send(frame)
  
      video_in.release()
  
  if __name__ == &quot;__main__&quot;:
      consumer_pipe, producer_pipe = Pipe()
  
      signal.signal(signal.SIGINT, signal_handler)
      producer = Process(target=produce, args=(producer_pipe,))
  
      video_out = cv2.VideoWriter(&quot;appsrc ! videoconvert ! jpegenc ! rtpjpegpay ! rtpstreampay ! udpsink host=[destination-ip] port=12344&quot;, cv2.CAP_GSTREAMER, 0, 24, (800, 600), True)
      producer.start()
  
      while is_running:
          frame = consumer_pipe.recv()
          video_out.write(frame)
          print(&quot;Sending frame ...&quot;)
  
      video_out.release()
      producer.join()
  </code></pre>
  <p>The pipe that I have created between the two processes is providing a new frame as expected. When I try to listen to UDP port 12344 with <code>netcat</code>, I do not receive anything that is the same thing as before. I also have a hard time understanding how differentiating the pipelines are changing much as I would expect them to already run in different contexts. Still, I could be wrong concerning this assumption.</p>