title: How to determine the number of constrained CPUs within a Docker container? For use with make -j
body: <p>Docker supports a <code>--cpus</code> option which constrains a container to using a certain proportion of the host's computing resource. For example, <code>--cpus 2.5</code> will allocate computing resource equivalent to approximately two and a half of the host's CPU capability. This allocation seems to be somewhat &quot;logical&quot; rather than physical, in that a container running eight processes but with <code>--cpus 1.0</code> tends to run each process on a separate CPU but limited to approximately 12.5%. This is fine, I suppose.</p>
  <p>The problem is that I am not able to find a way for a process within a constrained container to determine how many equivalent CPUs it has available. This is useful specifically for build processes, for example <code>make -j &lt;n&gt;</code>, but what is <code>n</code>? Commonly used <code>nproc</code> outputs the number of CPUs on the host regardless of the <code>--cpus</code> option value, which makes it unsuitable as it is far too large. The value of <code>/proc/cpuinfo</code> is also the host's view (same as <code>nproc</code>).</p>
  <p><strong>Is there a way for a contained process to determine the number of CPUs available to it?</strong></p>
  <p>As an example, on my 16-core host, I'd like to use <code>docker run ... --cpus 4.0 ...</code> and then have <code>make -j &lt;something&gt;</code> run <em>only four</em> processes rather than, say, sixteen.</p>
  <p>FWIW, I'm using the Docker Executor with GitLab-CI and <code>gitlab-runner</code>, which is why <code>--cpuset-cpus=</code> isn't really helpful. The Executor should be able to allocate a number of CPUs to each job across all of the host's CPUs, rather than a specific subset. Although I could create separate Executors for, say, each explicit set of four CPUs, there's no obvious way I'm aware of for <code>gitlab-runner</code> to allocate these in a balanced way to incoming jobs, and I want to avoid pushing the responsibility onto the jobs themselves, perhaps using <strong>tags</strong> to select an Executor, since that would require each job to know too much about the runner.</p>
  <p>I did consider setting a corresponding environment variable like <code>NUM_CPUS=4</code> in the <code>gitlab-runner</code>'s <code>config.toml</code> so that the build job can find out how many to use with <code>make -j $NUM_CPUS</code>, however variables in <code>config.toml</code> can only be set for all runners, not individual ones:</p>
  <pre><code>concurrent = 4
  check_interval = 0
  
  [[runners]]
    name = &quot;build&quot;
    url = &quot;https://gitlab.com/&quot;
    executor = &quot;docker&quot;
    environment = [&quot;NUM_CPUS=4&quot;]
    [runners.docker]
      # ...
      cpus = &quot;4&quot;    # match NUM_CPUS!
      environment = [&quot;FOO=456&quot;]
  </code></pre>
  <p>The container's environment will see <code>NUM_CPUS=4</code> but the statement to effect <code>FOO=456</code> is invalid and of course no such value appears in the container's environment, which makes it impossible to vary the value between Executors.</p>
  <p>I should add that in the near future I want to create Executors with different numbers of CPUs - for example, one with just one CPU for simple single-process jobs, and another with lots of CPUs for intensive multi-process jobs, and the GitLab-CI jobs will be able to pick these using GitLab-CI <strong>tags</strong>. I specifically want to avoid the situation where the pipeline jobs explicitly state that they are to use exactly, say, 4 CPUs, since that requires the jobs to know too much about the build environment and makes it brittle.</p>