I need help figuring out a memory problem on a debian server. The servers main purose is hosting 30 Ruby web apps. I upgraded to stretch yesterday and a couple of hours later I started getting notifications from Monit about low system memory. The apps themselves arent using more memory than normal. I've never seen this issue on jessie or wheezy and so far Im at a loss. I've upgraded 3 other systems to stretch and not seen the same problem.I replaced the RAM chips and upgraded from 8GB to 12GB just in case, but that only delayed the system from running out of memory from 2 hours to 4 hours. After 7-8 hours the system starts killing processes. So far the only solution is to reboot the system.Looking at the output from ps aux I dont see anything that's eating up memory. In /var/log/syslog I see periodic errors about page allocation failure:Jul 15 23:30:21 odin kernel: [36637.127915] ruby: page allocation failure: order:4, mode:0x26040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK)Jul 15 23:30:22 odin kernel: [36637.127924] CPU: 8 PID: 9800 Comm: ruby Tainted: G          I     4.9.0-3-amd64 #1 Debian 4.9.30-2+deb9u2Jul 15 23:30:22 odin kernel: [36637.127925] Hardware name: HP ProLiant ML350 G6, BIOS D22 01/30/2011Jul 15 23:30:22 odin kernel: [36637.127927]  0000000000000000 ffffffff98d28414 ffffffff993feb20 ffffc22bc350fa18Jul 15 23:30:22 odin kernel: [36637.127929]  ffffffff98b84f3a 026040c0026040c0 ffffffff993feb20 ffffc22bc350f9b8Jul 15 23:30:22 odin kernel: [36637.127931]  ffffa06000000010 ffffc22bc350fa28 ffffc22bc350f9d8 fa32a9a5fc8c2ebdJul 15 23:30:22 odin kernel: [36637.127934] Call Trace:Jul 15 23:30:22 odin kernel: [36637.127940]  [<ffffffff98d28414>] ? dump_stack+0x5c/0x78Jul 15 23:30:22 odin kernel: [36637.127943]  [<ffffffff98b84f3a>] ? warn_alloc+0x13a/0x160Jul 15 23:30:22 odin kernel: [36637.127945]  [<ffffffff98b84c7a>] ? __alloc_pages_direct_compact+0x4a/0xf0Jul 15 23:30:22 odin kernel: [36637.127947]  [<ffffffff98b85264>] ? __alloc_pages_slowpath+0x294/0xbc0Jul 15 23:30:22 odin kernel: [36637.127948]  [<ffffffff98b85264>] ? __alloc_pages_slowpath+0x294/0xbc0Jul 15 23:30:22 odin kernel: [36637.127950]  [<ffffffff98b85d8e>] ? __alloc_pages_nodemask+0x1fe/0x260Jul 15 23:30:22 odin kernel: [36637.127954]  [<ffffffff98bde16d>] ? cache_grow_begin+0x9d/0x550Jul 15 23:30:22 odin kernel: [36637.127955]  [<ffffffff98bde16d>] ? cache_grow_begin+0x9d/0x550Jul 15 23:30:22 odin kernel: [36637.127956]  [<ffffffff98bde8d1>] ? fallback_alloc+0x161/0x200Jul 15 23:30:22 odin kernel: [36637.127960]  [<ffffffff98c91eea>] ? SYSC_semtimedop+0x58a/0xcd0Jul 15 23:30:22 odin kernel: [36637.127961]  [<ffffffff98bdf761>] ? __kmalloc+0x1e1/0x560Jul 15 23:30:22 odin kernel: [36637.127963]  [<ffffffff98c91eea>] ? SYSC_semtimedop+0x58a/0xcd0Jul 15 23:30:22 odin kernel: [36637.127967]  [<ffffffff98ee6de0>] ? release_sock+0x40/0x90Jul 15 23:30:22 odin kernel: [36637.127969]  [<ffffffff98f51c6f>] ? tcp_sendmsg+0x8bf/0xc00Jul 15 23:30:22 odin kernel: [36637.127971]  [<ffffffff98d31414>] ? timerqueue_add+0x54/0xa0Jul 15 23:30:22 odin kernel: [36637.127973]  [<ffffffff98ae5b98>] ? enqueue_hrtimer+0x38/0x80Jul 15 23:30:22 odin kernel: [36637.127974]  [<ffffffff98ae6464>] ? hrtimer_start_range_ns+0x194/0x360Jul 15 23:30:22 odin kernel: [36637.127976]  [<ffffffff98ae7532>] ? do_setitimer+0x1d2/0x230Jul 15 23:30:22 odin kernel: [36637.127977]  [<ffffffff98ae76b2>] ? SyS_setitimer+0x92/0x120Jul 15 23:30:22 odin kernel: [36637.127980]  [<ffffffff9900627b>] ? system_call_fast_compare_end+0xc/0x9bJul 15 23:30:22 odin kernel: [36637.127981] Mem-Info:Jul 15 23:30:22 odin kernel: [36637.127985] active_anon:531660 inactive_anon:132876 isolated_anon:61Jul 15 23:30:22 odin kernel: [36637.127985]  active_file:9801 inactive_file:9856 isolated_file:0Jul 15 23:30:22 odin kernel: [36637.127985]  unevictable:0 dirty:0 writeback:353 unstable:0Jul 15 23:30:22 odin kernel: [36637.127985]  slab_reclaimable:19576 slab_unreclaimable:15771Jul 15 23:30:22 odin kernel: [36637.127985]  mapped:7837 shmem:178 pagetables:6100 bounce:0Jul 15 23:30:22 odin kernel: [36637.127985]  free:30650 free_pcp:11 free_cma:0Jul 15 23:30:22 odin kernel: [36637.127989] Node 0 active_anon:2126640kB inactive_anon:531504kB active_file:39204kB inactive_file:39424kB unevictable:0kB isolated(anon):244kB isolated(file):0kB mapped:31348kB dirty:0kB writeback:1412kB shmem:712kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? noJul 15 23:30:22 odin kernel: [36637.127990] Node 0 DMA free:15908kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kBJul 15 23:30:22 odin kernel: [36637.127993] lowmem_reserve[]: 0 3076 11967 11967 11967Jul 15 23:30:22 odin kernel: [36637.127996] Node 0 DMA32 free:52868kB min:17352kB low:21688kB high:26024kB active_anon:27820kB inactive_anon:8080kB active_file:144kB inactive_file:524kB unevictable:0kB writepending:0kB present:3250368kB managed:3173852kB mlocked:0kB slab_reclaimable:1068kB slab_unreclaimable:5784kB kernel_stack:16kB pagetables:96kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kBJul 15 23:30:22 odin kernel: [36637.128000] lowmem_reserve[]: 0 0 8890 8890 8890Jul 15 23:30:22 odin kernel: [36637.128002] Node 0 Normal free:53824kB min:50140kB low:62672kB high:75204kB active_anon:2098820kB inactive_anon:523424kB active_file:39060kB inactive_file:38900kB unevictable:0kB writepending:1412kB present:9306108kB managed:9104276kB mlocked:0kB slab_reclaimable:77236kB slab_unreclaimable:57300kB kernel_stack:5360kB pagetables:24304kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:0kBJul 15 23:30:22 odin kernel: [36637.128006] lowmem_reserve[]: 0 0 0 0 0Jul 15 23:30:22 odin kernel: [36637.128008] Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15908kBJul 15 23:30:22 odin kernel: [36637.128017] Node 0 DMA32: 45*4kB (UME) 294*8kB (ME) 430*16kB (ME) 470*32kB (UME) 226*64kB (UME) 85*128kB (UME) 8*256kB (ME) 0*512kB 1*1024kB (E) 0*2048kB 0*4096kB = 52868kBJul 15 23:30:22 odin kernel: [36637.128026] Node 0 Normal: 1794*4kB (UME) 5789*8kB (UME) 36*16kB (UM) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54064kBJul 15 23:30:22 odin kernel: [36637.128034] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kBJul 15 23:30:22 odin kernel: [36637.128034] 93118 total pagecache pagesJul 15 23:30:22 odin kernel: [36637.128035] 73394 pages in swap cacheJul 15 23:30:22 odin kernel: [36637.128036] Swap cache stats: add 14111506, delete 14038112, find 8508247/13062492Jul 15 23:30:22 odin kernel: [36637.128037] Free swap  = 4632260kBJul 15 23:30:22 odin kernel: [36637.128037] Total swap = 5846012kBJul 15 23:30:22 odin kernel: [36637.128038] 3143117 pages RAMJul 15 23:30:22 odin kernel: [36637.128038] 0 pages HighMem/MovableOnlyJul 15 23:30:22 odin kernel: [36637.128039] 69608 pages reservedJul 15 23:30:22 odin kernel: [36637.128040] 0 pages hwpoisonedAny help on how to investigate this further would be appreciated. I'll quote some outputs below.$ topTasks: 189 total,   2 running, 187 sleeping,   0 stopped,   0 zombie%Cpu(s): 11.6 us,  3.7 sy,  0.0 ni, 55.2 id, 29.2 wa,  0.0 hi,  0.3 si,  0.0 stKiB Mem : 12294036 total,   124468 free, 12004288 used,   165280 buff/cacheKiB Swap:  5846012 total,  4392540 free,  1453472 used.    53360 avail Mem$ cat /proc/meminfoMemTotal:       12294036 kBMemFree:          120960 kBMemAvailable:      28484 kBBuffers:           27140 kBCached:            37904 kBSwapCached:       244056 kBActive:          2108308 kBInactive:         548892 kBActive(anon):    2074920 kBInactive(anon):   518556 kBActive(file):      33388 kBInactive(file):    30336 kBUnevictable:           0 kBMlocked:               0 kBSwapTotal:       5846012 kBSwapFree:        4491776 kBDirty:                 0 kBWriteback:          2148 kBAnonPages:       2443316 kBMapped:            26856 kBShmem:              1304 kBSlab:             120544 kBSReclaimable:      56712 kBSUnreclaim:        63832 kBKernelStack:        5376 kBPageTables:        25216 kBNFS_Unstable:          0 kBBounce:                0 kBWritebackTmp:          0 kBCommitLimit:    11993028 kBCommitted_AS:    4735892 kBVmallocTotal:   34359738367 kBVmallocUsed:           0 kBVmallocChunk:          0 kBHardwareCorrupted:     0 kBAnonHugePages:         0 kBShmemHugePages:        0 kBShmemPmdMapped:        0 kBHugePages_Total:       0HugePages_Free:        0HugePages_Rsvd:        0HugePages_Surp:        0Hugepagesize:       2048 kBDirectMap4k:       88252 kBDirectMap2M:    12484608 kB$ vmstat -s     12294036 K total memory     12012524 K used memory      2067832 K active memory       535264 K inactive memory       167376 K free memory        27756 K buffer memory        86380 K swap cache      5846012 K total swap      1279912 K used swap      4566100 K free swap      1741799 non-nice user cpu ticks            0 nice user cpu ticks       553504 system cpu ticks      8201481 idle cpu ticks      4685975 IO-wait cpu ticks            0 IRQ cpu ticks        42823 softirq cpu ticks            0 stolen cpu ticks     72478199 pages paged in     35891576 pages paged out      7177475 pages swapped in      8365758 pages swapped out     56148095 interrupts     69385724 CPU context switches   1500117582 boot time       173809 forks$ ps aux --sort rssUSER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND(0 mem commands removed)message+   364  0.0  0.0  45460     0 ?        Ss   13:19   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activationroot       376  0.0  0.0   4216     0 ?        Ss   13:19   0:00 /usr/sbin/acpidroot       407  0.0  0.0      0     0 ?        S<   13:19   0:00 [ttm_swap]root       522  0.0  0.0 443760     0 ?        Ssl  13:19   0:07 /usr/local/bin/junglediskserversysadmin   632  0.0  0.0  82916     0 ?        S    13:19   0:00 (sd-pam)root       764  0.0  0.0      0     0 ?        S<   13:19   0:00 [cifsiod]root       765  0.0  0.0      0     0 ?        S<   13:19   0:00 [cifsoplockd]root       786  0.0  0.0      0     0 ?        S    13:19   0:00 [cifsd]root       792  0.0  0.0  16072     0 tty1     Ss+  13:19   0:00 /sbin/agetty --noclear tty1 linuxroot      4378  0.1  0.0      0     0 ?        S    18:17   0:32 [kworker/8:2]root      5689  0.0  0.0      0     0 ?        S    22:44   0:00 [kworker/u64:3]root      7100  0.0  0.0      0     0 ?        S    22:57   0:00 [kworker/u64:1]root      9809  0.0  0.0      0     0 ?        D    23:25   0:00 [kworker/u64:0]root     11159  0.0  0.0      0     0 ?        S    23:38   0:00 [kworker/4:2]root     11396  0.0  0.0      0     0 ?        S    23:41   0:00 [kworker/u64:2]root     11398  0.0  0.0      0     0 ?        S    23:41   0:00 [kworker/u64:4]root     11592  0.0  0.0      0     0 ?        S    23:43   0:00 [kworker/4:0]root     14633  0.1  0.0      0     0 ?        S    13:54   0:48 [kworker/8:0]root     15179  0.1  0.0      0     0 ?        S    18:51   0:27 [kworker/12:1]root     30176  0.0  0.0      0     0 ?        S    21:49   0:04 [kworker/4:1]root       367  0.0  0.0 250116    20 ?        Ssl  13:19   0:00 /usr/sbin/rsyslogd -nsysadmin   624  0.0  0.0  64868    32 ?        Ss   13:19   0:00 /lib/systemd/systemd --userroot       324  0.0  0.0  49868    60 ?        Ss   13:19   0:00 /sbin/rpcbind -f -wpostfix   1780  0.0  0.0  83396    76 ?        S    13:20   0:00 qmgr -l -t unix -upostfix   9094  0.0  0.0  83228    84 ?        S    23:17   0:00 pickup -l -t unix -u -csysadmin 10337  0.0  0.0  96864    84 ?        S    23:30   0:00 sshd: sysadmin@pts/0root      1775  0.0  0.0  81164    88 ?        Ss   13:20   0:00 /usr/lib/postfix/sbin/master -wroot       372  0.0  0.0  27992    96 ?        Ss   13:19   0:00 /usr/sbin/cron -froot       674  0.0  0.0  11400    96 ?        Ss   13:19   0:00 /sbin/mount.ntfs /dev/sdb1 /mybook -o rwroot       264  0.0  0.0  46864   112 ?        Ss   13:19   0:01 /lib/systemd/systemd-udevdntp        598  0.0  0.0 102064   152 ?        Ssl  13:19   0:02 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 108:113root       368  0.0  0.0  46804   160 ?        Ss   13:19   0:00 /lib/systemd/systemd-logindroot       371  0.0  0.0  35900   172 ?        Ss   13:19   0:01 /usr/sbin/irqbalance --foregrounddaemon     365  0.0  0.0  27964   204 ?        Ss   13:19   0:00 /usr/sbin/atd -froot       427  0.0  0.0  69940   260 ?        Ss   13:19   0:00 /usr/sbin/sshd -Droot     14632  0.0  0.0  43408   264 ?        S    13:54   0:04 /usr/bin/monit -c /etc/monit/monitrcroot         1  0.0  0.0  57492   340 ?        Ss   13:19   0:01 /sbin/initroot     10303  0.0  0.0  95168   536 ?        Ss   23:30   0:00 sshd: sysadmin [priv]root       612  0.0  0.0  48180   732 ?        Ss   13:19   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;root       228  0.0  0.0  46216  1112 ?        Ss   13:19   0:01 /lib/systemd/systemd-journaldsysadmin 10338  0.0  0.0  22952  1832 pts/0    Ss   23:30   0:00 -bashsysadmin  2089  0.0  0.0 159712  2148 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app1/current/config/unicorn.rb -E productionsysadmin  2149  0.0  0.0 159284  2160 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app2/current/config/unicorn.rb -E productionsysadmin  2155  0.0  0.0 159304  2192 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app3/current/config/unicorn.rb -E productionsysadmin  2202  0.0  0.0 159408  2220 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app4/current/config/unicorn.rb -E productionsysadmin  2781  0.0  0.0 159540  2316 ?        Sl   13:22   0:00 unicorn_bin master -D -c /apps/app5/current/config/unicorn.rb -E productionsysadmin  4449  0.0  0.0 159288  2316 ?        Sl   18:17   0:00 unicorn_bin master -D -c /apps/app6/current/config/unicorn.rb -E productionsysadmin 15205  0.0  0.0 159540  2360 ?        Sl   13:56   0:00 unicorn_bin master -D -c /apps/app7/current/config/unicorn.rb -E productionsysadmin  2027  0.0  0.0 159316  2368 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app8/current/config/unicorn.rb -E productionsysadmin  2012  0.0  0.0 159400  2404 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app9/current/config/unicorn.rb -E productionsysadmin  2179  0.0  0.0 159372  2408 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app10/current/config/unicorn.rb -E productionsysadmin  2131  0.0  0.0 159396  2420 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app11/current/config/unicorn.rb -E productionsysadmin  2119  0.0  0.0 159360  2516 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app12/current/config/unicorn.rb -E productionsysadmin  2113  0.0  0.0 159352  2520 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app13/current/config/unicorn.rb -E productionsysadmin  2125  0.0  0.0 158772  2520 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app14/current/config/unicorn.rb -E productionsysadmin  2053  0.0  0.0 159420  2528 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app15/current/config/unicorn.rb -E productionsysadmin  2059  0.0  0.0 159128  2528 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app16/current/config/unicorn.rb -E productionsysadmin  2101  0.0  0.0 159448  2532 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app17/current/config/unicorn.rb -E productionsysadmin  2185  0.0  0.0 160228  2540 ?        Sl   13:20   0:06 unicorn_bin master -D -c /apps/app18/current/config/unicorn.rb -E productionsysadmin  2047  0.0  0.0 159376  2556 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app19/current/config/unicorn.rb -E productionsysadmin  2065  0.0  0.0 159432  2556 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app20/current/config/unicorn.rb -E productionsysadmin  2143  0.0  0.0 159488  2560 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app21/current/config/unicorn.rb -E productionsysadmin  2167  0.0  0.0 159368  2568 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app22/current/config/unicorn.rb -E productionsysadmin  2021  0.0  0.0 159472  2768 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app23/current/config/unicorn.rb -E productionwww-data   613  0.1  0.0  49312  2780 ?        S    13:19   0:48 nginx: worker processsysadmin  2192  0.0  0.0 159380  3036 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app24/current/config/unicorn.rb -E productionsysadmin  2041  0.0  0.0 159172  3096 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app25/current/config/unicorn.rb -E productionsysadmin  2034  0.0  0.0 159372  3220 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app26/current/config/unicorn.rb -E productionsysadmin  2107  0.0  0.0 159584  3260 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app27/current/config/unicorn.rb -E productionsysadmin  2006  0.0  0.0 159420  3264 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app28/current/config/unicorn.rb -E productionsysadmin  2083  0.0  0.0 159000  3272 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app29/current/config/unicorn.rb -E productionsysadmin  2095  0.0  0.0 159376  3284 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app30/current/config/unicorn.rb -E productionsysadmin 11801  0.6  0.0  40152  3316 pts/0    R+   23:46   0:00 ps aux --sort rsssysadmin  2071  0.0  0.0 158964  3360 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app31/current/config/unicorn.rb -E productionsysadmin  2137  0.0  0.0 159296  3396 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app32/current/config/unicorn.rb -E productionsysadmin  2161  0.0  0.0 159388  3656 ?        Sl   13:20   0:00 unicorn_bin master -D -c /apps/app33/current/config/unicorn.rb -E productionmongodb    538  0.2  0.0 2001460 4700 ?        Sl   13:19   1:52 /usr/bin/mongod --config /etc/mongodb.confsysadmin 11772  0.1  0.0 159388  6060 ?        Dl   23:45   0:00 unicorn_bin worker[0] -D -c /apps/app33/current/config/unicorn.rb -E productionsysadmin 11687  0.1  0.0 159096  9816 ?        Dl   23:45   0:00 unicorn_bin worker[0] -D -c /apps/app31/current/config/unicorn.rb -E productionsysadmin 10700  0.6  0.1 349848 13296 ?        Sl   23:34   0:04 unicorn_bin worker[0] -D -c /apps/app1/current/config/unicorn.rb -E productionsysadmin 11576  0.1  0.1 164972 19784 ?        Dl   23:43   0:00 unicorn_bin worker[0] -D -c /apps/app28/current/config/unicorn.rb -E productionsysadmin 11596  0.3  0.1 169660 23240 ?        Dl   23:43   0:00 unicorn_bin worker[0] -D -c /apps/app30/current/config/unicorn.rb -E productionsysadmin 11510  0.7  0.4 210316 49828 ?        Dl   23:42   0:01 unicorn_bin worker[0] -D -c /apps/app27/current/config/unicorn.rb -E productionsysadmin 11488  0.7  0.4 215252 53172 ?        Dl   23:42   0:01 unicorn_bin worker[0] -D -c /apps/app26/current/config/unicorn.rb -E productionsysadmin 10790  0.7  0.4 363380 53748 ?        Sl   23:35   0:05 unicorn_bin worker[0] -D -c /apps/app7/current/config/unicorn.rb -E productionsysadmin  9639  0.5  0.4 372832 56308 ?        Sl   23:23   0:06 unicorn_bin worker[1] -D -c /apps/app16/current/config/unicorn.rb -E productionsysadmin 10217  0.5  0.4 437220 57072 ?        Dl   23:29   0:05 unicorn_bin worker[0] -D -c /apps/app21/current/config/unicorn.rb -E productionsysadmin 10442  0.6  0.4 371012 59192 ?        Sl   23:31   0:06 unicorn_bin worker[0] -D -c /apps/app9/current/config/unicorn.rb -E productionsysadmin 10521  0.7  0.5 371792 63568 ?        Sl   23:32   0:06 unicorn_bin worker[0] -D -c /apps/app19/current/config/unicorn.rb -E productionsysadmin 10674  0.7  0.5 356400 65540 ?        Sl   23:34   0:05 unicorn_bin worker[0] -D -c /apps/app16/current/config/unicorn.rb -E productionsysadmin 11102  1.1  0.5 432420 65560 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app20/current/config/unicorn.rb -E productionsysadmin 10611  0.9  0.5 372992 67856 ?        Sl   23:33   0:06 unicorn_bin worker[0] -D -c /apps/app13/current/config/unicorn.rb -E productionsysadmin 11089  1.1  0.5 365200 70340 ?        Sl   23:37   0:05 unicorn_bin worker[0] -D -c /apps/app17/current/config/unicorn.rb -E productionsysadmin 11427  1.0  0.5 295068 72612 ?        Dl   23:42   0:02 unicorn_bin worker[0] -D -c /apps/app32/current/config/unicorn.rb -E productionsysadmin 11288  0.8  0.5 298980 73360 ?        Dl   23:40   0:02 unicorn_bin worker[0] -D -c /apps/app10/current/config/unicorn.rb -E productionsysadmin 11170  1.1  0.6 363160 74384 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app11/current/config/unicorn.rb -E productionsysadmin 10588  0.9  0.6 493632 75100 ?        Dl   23:33   0:07 unicorn_bin worker[0] -D -c /apps/app14/current/config/unicorn.rb -E productionsysadmin 11106  1.2  0.6 364900 81756 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app12/current/config/unicorn.rb -E productionsysadmin 11178  1.2  0.7 361828 88628 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app23/current/config/unicorn.rb -E productionsysadmin 11409  1.4  0.7 349048 88840 ?        Dl   23:41   0:03 unicorn_bin worker[0] -D -c /apps/app29/current/config/unicorn.rb -E productionsysadmin 11400  1.7  0.7 355636 93176 ?        Dl   23:41   0:04 unicorn_bin worker[0] -D -c /apps/app25/current/config/unicorn.rb -E productionsysadmin 11175  0.8  0.7 322132 95716 ?        Dl   23:38   0:03 unicorn_bin worker[0] -D -c /apps/app6/current/config/unicorn.rb -E productionsysadmin 11252  0.9  0.7 322268 95880 ?        Dl   23:39   0:03 unicorn_bin worker[0] -D -c /apps/app22/current/config/unicorn.rb -E productionsysadmin 11169  1.2  0.7 362224 97484 ?        Sl   23:38   0:05 unicorn_bin worker[0] -D -c /apps/app15/current/config/unicorn.rb -E productionsysadmin 10993  1.3  0.8 386516 103752 ?       Sl   23:37   0:07 unicorn_bin worker[1] -D -c /apps/app25/current/config/unicorn.rb -E productionsysadmin 11047  0.8  0.8 354136 106652 ?       Dl   23:37   0:04 unicorn_bin worker[0] -D -c /apps/app18/current/config/unicorn.rb -E productionsysadmin 10555  1.2  0.8 469496 109208 ?       Sl   23:32   0:10 unicorn_bin worker[0] -D -c /apps/app5/current/config/unicorn.rb -E productionsysadmin 11282  1.8  0.9 380480 114992 ?       Sl   23:40   0:06 unicorn_bin worker[0] -D -c /apps/app8/current/config/unicorn.rb -E productionsysadmin 11210  1.0  0.9 343824 117296 ?       Dl   23:39   0:04 unicorn_bin worker[0] -D -c /apps/app4/current/config/unicorn.rb -E productionsysadmin 10897  1.6  1.0 467556 123780 ?       Sl   23:36   0:10 unicorn_bin worker[0] -D -c /apps/app3/current/config/unicorn.rb -E productionsysadmin 11027  1.9  1.1 470524 141648 ?       Sl   23:37   0:10 unicorn_bin worker[0] -D -c /apps/app2/current/config/unicorn.rb -E productionsysadmin 10784  1.6  1.1 470824 145132 ?       Sl   23:35   0:11 unicorn_bin worker[0] -D -c /apps/app24/current/config/unicorn.rb -E production
Debian memory leak after stretch upgrade
debian;memory
Solved by upgrading the Unicorn HTTP server from v4.9 to v5.3. I dont know whether the update fixed the problem or if it was the act of reinstalling that solved issue. The weird thing is that I was never able to pin point the problem with the help of monitoring tools or logging.