How to find out which process eats up memory?
Check with this command (will show more processes):

sudo ps -awxm -o %mem,rss,comm | sort -nr | head


if won't help, check with fs_usage to display system calls and page faults related to filesystem:

sudo fs_usage -f filesys,diskio


especially page in and out by appending the following to above command:

| egrep -i "page_|pgin|pgout"


Add extra | grep -v kernel to ignore kernel process or | grep -v 0.00 to show these with higher time spent.

If your swap is big, you can force disk cache to be flushed and emptied by: sudo purge.

Based on my experience, OS X have some issues with managing the memory after higher uptime (when it used all of free memory), so sometimes only full restart may help.

Check also sudo iotop or sudo vm_stat 1 commands which may help.

See also: How to investigate high kernel task memory usage?