I'm running Gentoo Hardened AMD64 using kernel 4.3.3-hardened-r4. With my system only running some basic daemons like wpa_supplicant, cron or DHCP, and with an X session with only Windowmaker, GKrellM and xterm open, Linux starts eating more and more RAM as time goes on until after some 8-12 hours it ends up running out of RAM and throws a kernel panic. This is not a matter of Linux reporting RAM used for buffers and filesystem cache as taken, because top, htop and GKrellM both account for these cases and display how much RAM is actually taken by processes. Up until recently I thought it was linked to my Bitcoin Core client, but that was not the case (I just casually happened to run that app while my Linux system was up).I was able to see in a few instances my RAM usage suddenly jumping back to normal while issuing a full world update (emerge -NDu --with-bdeps=y @world), but I haven't been able to reproduce this workaround.So far I've tried the following fixes:Compiling NUMA support on my kernel (by default not enabled by Gentoo's genkernel) and adding vm.zone_reclaim_mode=1 to my sysctl. Didn't work.Adding vm.drop_caches=1 to my sysctl. Didn't work.Checking whether a tmpfs mount was getting full. My tmpfs mounts hardly even register over 1 MB of filesystem usage.Evidence of this behavior can be seen in these screenshots:Exhibit A: In which the only memory-eating processes that are running are Firefox, GKrellm and X yet Linux is eating almost 3 GB of core.Note: I didn't have my swap space enabled here (it's on an USB 3.0 external HD because my internal HD is old and slow), but even with swap enabled I still end up with an OOM kernel panic after 8+ hours of keeping Bitcoin Core running.Exhibit B: Just in case htop and GKrellm are flawed, I double-checked with top. Same result.Exhibit C: My tmpfs mount usage statistics, my output of free and my content of /proc/meminfo available here.This post has been greatly edited to account for my most recent findings. The old post can be found on this Pastebin here.
Gentoo Hardened eats 3 GB of memory after running for ~12 hours with very few processes running
memory;gentoo
So after almost two months of puzzling over this question, I decided to ask around how to enable that vm.zone_reclaim_mode sysctl option and play a little with different values, and lo and behold -- problem solved.Solution:Enable CONFIG_NUMA on my kernel's configuration and rebuild it.Put vm.zone_reclaim_mode = 7 on sysctl.confNow my system can finally stay in one piece for over 24 hours. I'm a bit afraid of reduced performance since the kernel documentation says both NUMA and such an aggressive zone reclaim setting could probably slow things down, but for now my system finally works.