OS X Yosemite: System slow and using a lot of RAM
As noted in the comment it had probably something to do with the WindowServer.



Now regarding the other part of the question:

OS X presents memory usage in a different way when comparing to say MS Windows.
For example, if you quit an app in OS X, its memory is available to other apps, but it is not 'cleared' (so if you relaunch the app it would be quite faster in most cases).

My MBP is currently showing 7.88 GB ("Memory used" our of 8GB total), but everything runs as smooth as after restarting (Memory pressure is minimal). 

I suggest you read the How to use Activity Monitor, especially the section about Memory, where you can find some additional information.

Long story short, you better check the "Memory Pressure" indication graph in order to have a more accurate idea about your system's current status.