Memory usage on Mac OS, how Activity monitor works?
I suspect you probably are using Mac OS 10.7 which is very memory hungry. As noted by kccqzy, the purge command may reclaim some inactive memory if you have XCode installed. 

Something you should try is closing and reopening applications which are memory hungry or have a tendency to leak memory (Safari, Firefox). Alternatively, restarting your computer would also accomplish the same though take considerably longer.

To answer the question, are you really using 4GB of memory. Yes. Technically more if you take note of the "Swap used:" stat.