Is there a way to limit the size of the disk swap file in Mac OS X?
Yes you can configure the dynamic_pager, but for most it is simpler to just disable it and see which program crashes due to the inevitable out of memory errors.

Why would I disable swap file in Mac OS X?

Pay attention to programs that implement their own virtual memory / cache / paging systems like virtualization and the Adobe Suite in case you have inadvertently tuned them to use more RAM than is available on the system. They tend to be first on my investigation list when I see a mac with runaway swap allocation on the filesystem.