title: Prometheus (Docker): determine available memory per node (which metric is correct?)
body: <p>We have been struggling to create a good memory monitoring for our nodes running Docker components. We use Prometheus in combination with cadvisor and node_exporter.  </p>
  
  <p>What is the best way to determine the used memory per node? </p>
  
  <p><strong>Method 1</strong>: gives in our example around 42%</p>
  
  <pre><code>(1-(node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes))*100
  </code></pre>
  
  <p><strong>Method 2</strong>: gives around 80%</p>
  
  <pre><code>(1-((node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes)/node_memory_MemTotal_bytes))*100
  </code></pre>
  
  <p>Q2: Why is this difference? What can I learn from this? </p>
  
  <p>So I digged a bit deeper on determined the individual metrics: </p>
  
  <ol>
  <li><p><strong>Free memory</strong>: in our experiment was about 5%</p>
  
  <p>(node_memory_MemFree_bytes/node_memory_MemTotal_bytes)*100</p></li>
  <li><p><strong>Buffered memory</strong>: around 0.002%</p>
  
  <p>(node_memory_Buffers_bytes/node_memory_MemTotal_bytes)*100</p></li>
  <li><p><strong>Cached memory</strong>: around 15%</p>
  
  <p>(node_memory_Cached_bytes/node_memory_MemTotal_bytes)*100</p></li>
  <li><p><strong>Availabable memory</strong>: 58%</p>
  
  <p>(node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes)*100</p></li>
  </ol>
  
  <p>I would expect that FreeMem + BufferedMem + CachedMem would be around the AvailableMem. But that is not the outcome of this simple experiment. </p>
  
  <p>Q3: Why is this not true? </p>
  
  <p>It is said that the free memory on Linux consists of free mem + buffered mem + cached mem. When there is a short of memory, the cached memory could be freed, etc. </p>