Trouble Understanding Inactive Memory in MacOSX
I have found, on my machine, that Mac OS X is a little slow to page the right things out. 

Whenever doing anything particularly memory hungry, I will often quit a few applications to help it out. Safari is a huge memory hog if it has been running for a while and is top of the list of applications to quit. In my case this was when I tried to run VMWare Fusion without quite enough actual RAM.

You probably know this but here are a couple of links to Apple documentation which describe memory usage and virtual memory on OS X:

http://support.apple.com/kb/ht1342

http://developer.apple.com/library/mac/#documentation/Performance/Conceptual/ManagingMemory/Articles/AboutMemory.html

Here is the explanation that is given: The inactive list contains pages that are currently resident in physical memory but have not been accessed recently. It's hard to get an exact definition of what this means but is seems to include a file system cache as well as actual inactive memory. 

Inactive memory is probably a difficult thing to handle on an interactive GUI driven OS where a user may choose to switch to any running application at any time and providing responsiveness in this sort of situation is important.... but the flipside of that is that in a situation like yours the OS doesn't quite know what the right thing to do is.

It is possible to purge (some of) the inactive memory by using the purge command. This may need to be installed by installing the CHUD tools but you may already have that installed. I haven't tried using the purge command so YMMV...

I've rambled on a bit and may or may not have answered your question. After writing all this out, I found another question on this site which much of the same info that might also help.