com.apple.iconServicesAgent using a lot of RAM in 10.9
Observations

You may observe natural rises and falls in usage. 

You should not treat the usage as a problem. 

Background: about icon services

For an idea of what's done by icon services (the com.apple.IconServices daemon, com.apple.IconServicesAgent processes and related files), run the following command in a Terminal window that's maybe 257 characters wide, or wider: 

sudo fs_usage -w -f filesys com.apple.IconServices | grep write

Then in Finder: 


browse a folder that has not been browsed since you started the Mac
browse away
browse back to the same folder.


You should find that:


the initial browse causes writes (to a cache)
second and subsequent browses cause no additional caching, for as long as the contents of the folder are unchanged. 


To abort the command, Control-C

For an unfiltered view of things: 

sudo fs_usage -w -f filesys com.apple.IconServices

Benefits of icon services in Mavericks

It seems to me that Apple's approach to caching icons is of particular benefit to apps/processes where primary functions include reading metadata from a variety of types of file system. Apps such as Finder, processes such as Dock, and so on. 

As part of a broader approach to maximising performance, icon services should allow presentation with minimal delay of the subset of metadata that is of greatest interest to the user. 

For the Recent Applications stack in Dock: probably names, icons and dates/times as essentials within a subset. 

For views in Finder: subsets may be entirely different, depending on what's required by the user. 

Additional thoughts

In HFS Plus (Mac OS Extended), with reference to retired documentation, we have the performance-oriented hot file B-tree, attributes file and so on. With everything else that has been done to that file system, it is now debatably over-extended. 

The performance-related benefits of icon services should be relatively file system-agnostic. This is pleasing. I expect the benefits to be realised by users of ZFS on Mavericks, and so on …