Before you get your pitchforks out, I am having trouble tracking down on where the memory is going into the caching system in Linux. I have seen http://www.linuxatemyram.com/, and https://stackoverflow.com/questions/4802481/how-to-see-top-processes-by-actual-memory-usage and Correctly determining memory usage in Linux but using those as examples, the numbers don't quite add up with the numbers I'm seeing on my system. For this system I understand that it is probably cached, or used by the programs, but the numbers aren't even coming close to adding up for me.In top I see.top - 09:04:09 up 19 days, 20:38,  2 users,  load average: 0.00, 0.10, 0.11Tasks: 160 total,   1 running, 159 sleeping,   0 stopped,   0 zombieCpu(s):  0.1%us,  0.2%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%stMem:  65974296k total, 43507804k used, 22466492k free,   305336k buffersSwap:  7548924k total,        0k used,  7548924k free,  1480836k cachedI get it, 43GB of ram used isn't really true, probably most of it is probably cached. So digging into it I ran: $ free -m             total       used       free     shared    buffers     cachedMem:         64428      38845      25582          0        298       1445-/+ buffers/cache:      37101      27326Swap:         7371          0       7371So this shows that 37GB of it is indeed used and only 1445MB is cached (this is where I would have expected to see 1445 being something like 20000). The websites I linked above show that the cached column is usually pretty high. So digging further I went and did the following to check applications that were using memory.    sudo smem -t      PID User     Command                         Swap      USS      PSS      RSS9468    21475 root     python /usr/bin/smem -t            0     6212     6234     6984     2431 root     /opt/OV/lbin/perf/coda             0     7156     8060    12068     2213 root     /opt/perf/bin/perfd                0    19056    19485    22032    20849 root     /opt/shiny-server/ext/node/        0    77244    77321    78616    21325 atpa     /usr/lib64/R/bin/exec/R --n        0  3729836  3733774  3739520    21287 atpa     /usr/lib64/R/bin/exec/R --n        0  4060136  4064074  4069820    -------------------------------------------------------------------------------       63 11                                          0  7947984  7970344  8054032So two applications of R are using ~8GB of memory. The other articles I linked above show that Linux was reserving the memory and keeping it in cache (e.g. free -m showed cache was high value on the Mem: line), whereas in my case it seemed to be actually in use but no applications actually seem to be reporting the usage of the memory and I can't seem to track down where Linux is using (caching/reserving?) the memory.Where is this memory going? I am assuming Linux is using it, but I can't determine where it is being utilized.Edit:cat /proc/meminfoMemTotal:       65974296 kBMemFree:        24191624 kBBuffers:          305320 kBCached:          1480760 kBSwapCached:            0 kBActive:          7769776 kBInactive:         215532 kBActive(anon):    6199392 kBInactive(anon):      476 kBActive(file):    1570384 kBInactive(file):   215056 kBUnevictable:           0 kBMlocked:               0 kBSwapTotal:       7548924 kBSwapFree:        7548924 kBDirty:               116 kBWriteback:             0 kBAnonPages:       6172696 kBMapped:            47400 kBShmem:               652 kBSlab:             255468 kBSReclaimable:     225620 kBSUnreclaim:        29848 kBKernelStack:        1736 kBPageTables:        18780 kBNFS_Unstable:          0 kBBounce:                0 kBWritebackTmp:          0 kBCommitLimit:    40536072 kBCommitted_AS:    6455352 kBVmallocTotal:   34359738367 kBVmallocUsed:      247288 kBVmallocChunk:   34359487760 kBHardwareCorrupted:     0 kBAnonHugePages:   2586624 kBHugePages_Total:       0HugePages_Free:        0HugePages_Rsvd:        0HugePages_Surp:        0Hugepagesize:       2048 kBDirectMap4k:       10240 kBDirectMap2M:    67098624 kB
Tracking down Linux memory usage when not showing up in cache
linux;memory;top;memory leaks
I think I found my issue...My problem seems to have been VMware's memory ballooning system. Basically this is a way for the host system to apply memory pressure to the guest OS, consuming the guests memory allocation when other hosts start using a significant amount of memory.http://www.vfrank.org/2013/09/18/understanding-vmware-ballooning/If you are using VMware, run the command vmware-toolbox-cmd stat balloonThis will show the amount of ballooned memory.For me #:vmware-toolbox-cmd stat balloon32425 MBOther sources: https://serverfault.com/questions/660080/detect-memory-ballooning-from-within-the-affected-vmTurn off ballooned memory to validate the issueUnballooning memory: https://serverfault.com/questions/528295/unballooning-ram-thats-been-ballooned-by-vmware