title: Docker run vs build - build gstreamer Different behaviour
body: <p>I'm trying to build a docker image that uses nvidia hardware decoding in gstreamer and have encountered a strange problem with making the image. </p>
  
  <p>The build process does not find the nvidia cuda related stuff while running <code>docker build</code> (or <code>nvidia-docker build</code>), but when I spin up the failed image as a container and do those very same steps from within the container everything works. I even saved the container as image which gave me a persistent image that works as intended.</p>
  
  <p>Has anyone experienced similar problem and can shed some light on it?</p>
  
  <pre><code>Dockerfile:
  </code></pre>
  
  <pre><code>FROM nvcr.io/nvidia/deepstream:3.0-18.11 AS base
  
  ENV DEBIAN_FRONTEND noninteractive
  
  #install some dependencies. NOTE - not removing apt cache for the MWE
  RUN apt-get update &amp;&amp; apt-get install -y --no-install-recommends \
      build-essential \
      libdc1394-22 \
      tmux \
      vim \
      libjpeg-dev \
      libpng-dev \
      libpng12-dev \
      cuda-toolkit-10-0 \
      python3-setuptools \
      python3-pip ninja-build pkg-config gobject-introspection gnome-devel bison flex libgirepository1.0-dev liborc-0.4-dev
  
  RUN pip3 install meson &amp;&amp; ldconfig
  
  FROM base
  #pull and make gstreamer:
  RUN cd /tmp &amp;&amp; mkdir gstreamer
  RUN git clone https://github.com/GStreamer/gst-build.git /tmp/gstreamer \
      &amp;&amp; cd /tmp/gstreamer \
      &amp;&amp; git checkout tags/1.16.0 \
      &amp;&amp; ./setup.py -Dgtk_doc=disabled -Dgst-plugins-bad:nvdec=enabled -Dgst-plugins-bad:nvenc=enabled -Dgst-plugins-bad:iqa=disabled -Dgst-plugins-bad:bluez=disabled --reconfigure \
      &amp;&amp; ninja -C build \
      &amp;&amp; ninja install -C build
  </code></pre>
  
  <p><strong>Testing:</strong>
  build and run the container. Inside the container:</p>
  
  <pre><code>$ gst-inspect-1.0 nvdec
    No such element or plugin 'nvdec'
  $ cd /tmp/gstreamer
  $ ./setup.py -Dgtk_doc=disabled -Dgst-plugins-bad:nvdec=enabled -Dgst-plugins-bad:nvenc=enabled -Dgst-plugins-bad:iqa=disabled -Dgst-plugins-bad:bluez=disabled --reconfigure 
  $ ninja -C build
  $ ninja install -C build
  $ gst-inspect-1.0 nvdec
    Factory Details:
    Rank                     primary (256)
    [... all plugin parameters show up]
    GObject
      +----GInitiallyUnowned
           +----GstObject
                 +----GstElement
                      +----GstVideoDecoder
                           +----GstNvDec
  </code></pre>
  
  <p><strong>EDIT1</strong><br>
  The image builds with no errors, only when I try to call gstreamer it is built with no acceleration. I noticed that in the build process the major difference is</p>
  
  <pre><code>meson.build:109:2: Exception: Problem encountered: The nvdec plugin was enabled explicitly, but required CUDA dependencies were not found.
  </code></pre>
  
  <p>which does not happen when building from within the container.</p>
  
  <p>Lack of error is related, most likely, to the ninja+meson build system which looks for compatible packages, reports the exception, but doesn't throw it and continues as if nothing wrong happened</p>
  
  <p><strong>EDIT2</strong><br>
  Answering comment:
  To build it and get the error, just build the attached docker image:</p>
  
  <p><code>sudo docker build -t gst16:latest . &gt; build.log</code></p>
  
  <p>This will dump all the output into the <code>build.log</code> file.<br>
  I don't have a docker registry that I could use for this and the docker image gets quite big by docker standards (~8 Gigs), but to produce successfully, it's fairly simple:</p>
  
  <p><code>sudo docker run --runtime="nvidia" -ti gst16:latest /bin/bash</code></p>
  
  <p>or</p>
  
  <p><code>sudo nvidia-docker run -ti gst16:latest /bin/bash</code></p>
  
  <p>which seems to work the same for me. Notice <strong>no --rm</strong> flag! From within the container:</p>
  
  <pre><code>#check if nvidia decoder plugin is there:
  gst-inspect-1.0 nvdec
  #fail!
  #now build it from within:
  cd /opt/gstreamer
  ./setup.py -Dgtk_doc=disabled -Dgst-plugins-bad:nvdec=enabled -Dgst-plugins-bad:nvenc=enabled -Dgst-plugins-bad:iqa=disabled -Dgst-plugins-bad:bluez=disabled --reconfigure
  ninja -C build
  ninja install -C build
  gst-inspect-1.0 nvdec
  #success reported
  </code></pre>
  
  <p>Now to get the image, exit the container (ctrl+d) and in the host shell:</p>
  
  <ul>
  <li><code>sudo docker container ls -a</code> to view all containers including stopped ones</li>
  <li>from gst16:latest get the CONTAINER_ID and copy it</li>
  <li><code>sudo docker commit &lt;CONTAINER_ID&gt; gst16:manual</code> and after a few seconds you should have the container saved as an image. Verify with <code>sudo docker images</code></li>
  <li>run the new image with <code>sudo docker run --runtime=`nvidia` --rm -ti gst16:manual /bin/bash</code></li>
  <li>from within the container try again the <code>gst-inspect-1.0 nvdec</code> to verify it's working</li>
  </ul>
  
  <p><strong>EDIT3</strong></p>
  
  <pre><code>$ nvidia-docker --version 
    Docker version 18.09.0, build 4d60db4 
  </code></pre>