WindowServer process takes a lot of memory
According to this article  this appears to be a bug in the system when used with a Scaled Display resolution


  This seems to be a bug with scaled screen resolution (specially with 4K monitors) in MacOS High Seirra (and perhaps in older versions of the OS as well). Using "Default for display" screen resolution seems to fix the issue (Not my ideal resolution, but a temporary fix until an MacOS update with a fix is available). More information here: https://www.tonymacx86.com/threads/windowserver-memory-leak-on-high-sierra.233510/


I am using a scaled (external) display and I am facing a similar problem which is what led me to this topic.