title: Monitor the CPU usage of an OpenFOAM simulation running on a slurm job
body: <p>I'm running an OpenFOAM simulation on a cluster. I have used <a href="https://cfd.direct/openfoam/user-guide/running-applications-parallel/" rel="nofollow noreferrer">the Scotch decomposition method</a> and my <code>decomposeParDict</code> looks like this:</p>
  
  <pre><code>FoamFile
  {
      version     2.0;
      format      ascii;
      class       dictionary;
      object      decomposeParDict;
  }
  numberOfSubdomains 6;
  method          scotch;
  </code></pre>
  
  <p><code>checkMesh</code> and <code>decomposePar</code> finish with no issues. I have assigned 6 nodes to the slurm by</p>
  
  <pre><code>srun -N6 -l sonicFoam
  </code></pre>
  
  <p>and the solver runs smoothly without any errors.</p>
  
  <p>The issue is the solution <strong>speed is not improved</strong> in comparison to the non-parallel simulation I ran before. I want to monitor the CPU usage to see if all of the 6 nodes I have assigned are similarly loaded. The <code>squeue --user=foobar</code> command return the <code>jobNumber</code> and list of nodes assigned (<code>NODELIST(REASON)</code>) which looks like this:</p>
  
  <pre><code>foo,bar[061-065]
  </code></pre>
  
  <p>from <code>sinfo</code> command these nodes are both in <code>debug</code> and <code>main*</code> <code>PARTITION</code>s  (which I have absolutely no idea what it means!).</p>
  
  <p><a href="https://stackoverflow.com/a/24087711/4999991">This post</a> says that you can use the <code>sacct</code> or <code>sstat</code> commands to monitor CPU time and memory usage of a slurm job. But when I run </p>
  
  <pre><code>sacct --format="CPUTime,MaxRSS"
  </code></pre>
  
  <p>it gives me:</p>
  
  <pre><code> CPUTime     MaxRSS
  ---------- ----------
    00:00:00
    00:00:00
    00:07:36
    00:00:56
    00:00:26
    15:26:24
  </code></pre>
  
  <p>which I can not understand. And when I specify the job number by </p>
  
  <pre><code>sacct --job=&lt;jobNumber&gt; --format="UserCPU"
  </code></pre>
  
  <p>The return is empty. So my questions are</p>
  
  <ul>
  <li>Is my simulation loading all nodes or is it running on one or two and the rest are free?</li>
  <li>am I running the right commands? if yes what those numbers mean? how they represent the CPU usage per node?</li>
  <li>If not then what are the right <code>--format="..."</code>s for <code>sacct</code> and/or <code>sstat</code> (or maybe other slurm commands) to get the CPU usage/load?</li>
  </ul>
  
  <p><strong>P.S.1.</strong> I have followed the OpenFOAM compiling following <a href="https://www.openfoam.com/code/build-guide.php" rel="nofollow noreferrer">the official instructions</a>. I did not do anything with <code>OpenMPI</code> and it's <code>mpicc</code> compiler for that matter though.</p>
  
  <p><strong>P.S.2</strong> For those of you who might end up here. Maybe I'm running the wrong command apparently one can first allocate some resources by:</p>
  
  <pre><code>srun -N 1 --ntasks-per-node=7 --pty bash
  </code></pre>
  
  <p>where 7 is the number of cores you want and bash is just a name. and then run the solver with:</p>
  
  <pre><code>mpirun -np 7 sonicFoam -parallel -fileHandler uncollated
  </code></pre>
  
  <p>I'm not sure yet though.</p>