Is it possible to reduce memory usage by iTunes 11 on Mac?
If your iTunes is using this much memory, something is wrong.

Just how large your library is? My 700 GB audio + video library takes far less than 500 MB of physical RAM and I actually have my virtual memory disabled.

I will suggest you do the following,

a) Quit iTunes more often.

b) For testing, create a new clean library and monitor the memory usage

c) For testing, reinstall iTunes

Good luck. If you have any more question, comment below!