title: NodeJS memory growth - Memory leak in (system)?
body: <p>I am encountering a strange memory leak in our live-environment, where <code>(system)</code> objects in the heap keep growing.</p>
  
  <h3>Heap dump</h3>
  
  <p>Here is a memory dump where the memory usage grew to 800MB:
  <a href="https://i.stack.imgur.com/vvEpA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vvEpA.png" alt="enter image description here"></a></p>
  
  <p>Notice that this memory is retained in a <code>Generator</code> object. TBH, I have no idea what that is.</p>
  
  <p>Manually triggering garbage collection via <code>global.gc();</code> usally frees about 10MB of memory. (But repetitive triggering of gc has no affect) </p>
  
  <h3>Application crashes</h3>
  
  <p>The application crashes roughly every 12 hours with following error:</p>
  
  <pre><code>2019-11-15T08:29:01.174417825Z 08:29:01 0|. | &lt;--- Last few GCs ---&gt;
  2019-11-15T08:29:01.176095829Z 08:29:01 0|. | [47:0x55ac8cdf3dc0] 65890519 ms: Scavenge 912.3 (929.2) -&gt; 909.3 (929.2) MB, 6.7 / 0.0 ms  (average mu = 0.987, current mu = 0.987) allocation failure 
  2019-11-15T08:29:01.177260332Z 08:29:01 0|. | [47:0x55ac8cdf3dc0] 65890653 ms: Scavenge 912.9 (929.2) -&gt; 910.0 (929.2) MB, 10.1 / 0.0 ms  (average mu = 0.987, current mu = 0.987) allocation failure 
  2019-11-15T08:29:01.178027234Z 08:29:01 0|. | [47:0x55ac8cdf3dc0] 65890701 ms: Scavenge 913.6 (929.2) -&gt; 910.6 (929.2) MB, 6.3 / 0.0 ms  (average mu = 0.987, current mu = 0.987) allocation failure 
  2019-11-15T08:29:01.183406747Z 08:29:01 0|. | &lt;--- JS stacktrace ---&gt;
  2019-11-15T08:29:01.184194849Z 08:29:01 0|. | ==== JS stack trace =========================================
  2019-11-15T08:29:01.184939851Z 08:29:01 0|. |     0: ExitFrame [pc: 0x55ac88379c39]
  2019-11-15T08:29:01.185674553Z 08:29:01 0|. | Security context: 0x064ed4d408d1 &lt;JSObject&gt;
  2019-11-15T08:29:01.187183757Z 08:29:01 0|. |     1: /* anonymous */ [0x34eb4a55a171] [/usr/src/app/node_modules/express-validator/src/chain/context-runner-impl.js:~25] [pc=0x34ab58ee2e1f](this=0x3c92e4142b89 &lt;ContextRunnerImpl map = 0x13b2881d9399&gt;)
  2019-11-15T08:29:01.188904261Z 08:29:01 0|. |     2: next [0x64ed4d63621](this=0x34eb4a55a569 &lt;JSGenerator&gt;)
  2019-11-15T08:29:01.190831866Z 08:29:01 0|. |     3: /* anonymous */(aka /* anonymous */) [0x34eb4a55a201] [/usr/src/app/node_modules/expr...
  2019-11-15T08:29:01.195055876Z 08:29:01 0|. | FATAL ERROR: invalid array length Allocation failed - JavaScript heap out of memory
  2019-11-15T08:29:01.396743877Z 2019-11-15T08:29:01: PM2 log: App [.:0] exited with code [0] via signal [SIGABRT]
  2019-11-15T08:29:01.400481986Z 08:29:01 PM2      | App [.:0] exited with code [0] via signal [SIGABRT]
  2019-11-15T08:29:01.403095193Z 2019-11-15T08:29:01: PM2 log: App [.:0] starting in -fork mode-
  2019-11-15T08:29:01.406294501Z 08:29:01 PM2      | App [.:0] starting in -fork mode-
  2019-11-15T08:29:01.419367433Z 2019-11-15T08:29:01: PM2 log: App [.:0] online
  2019-11-15T08:29:01.422309441Z 08:29:01 PM2      | App [.:0] online
  
  2019-11-15T08:29:03.279070252Z 08:29:03 0|.      | server started at http://localhost:80
  </code></pre>
  
  <p>Which happens around 1.2 GB process memory usage (at which point the machine is running on ~67% memory). This is strange on its own since i run the sever with the following command:</p>
  
  <pre><code>pm2 start . --no-daemon --node-args=\"--max-old-space-size=2048 --max-semi-space-size=4 --expose_gc\" --max-memory-restart 1536M
  </code></pre>
  
  <p>so node should have 2GB available and pm2 should restart well before the process reaches the limit.</p>
  
  <h3>Application background</h3>
  
  <p>The app is a json api written in TypeScript. It uses <code>express</code> to handle requests (about 3k per minute) and uses <code>node-mssql</code> to query an MS SQL Server.
  Additionally it processes jobs (which perform additional sql queries) asynchronously via the <code>bee-queue</code> package.</p>