How can I increase the maximum allowed swap space?
While I could not find any more recent documentation, Singh's "Mac OS X Internals: A Systems Approach" mentions two obscure limits in /sbin/dynamic_pager (p. 919 f.):


  
  The absolute minimum and maximum swap file sizes are 64MB and 1GB, respectively.
  The maximum swap file size must not be greater than 12.5% of the free space available on the volume that contains the swap file. Moreover, the maximum swap file size must not be greater than the amount of physical memory on the system.
  At most, eight swap files can be created.
  
  
  [...]


While the book is a bit dated and the numbers are likely to have changed, the 12.5% rule sounds very suspicious to me. 

Singh implies that those are hard-coded limits in the binary and thus the property file you found is likely not to be of use. Also note that the swap system creates several, but only up to eight, distinct swap files.

/sbin/dynamic_pager is documented, as you noted, as a very rudimentary man page, and there is a conflicting statement in the current developer documentation on Virtual Memory, stating that


  To give processes access to their entire 4 gigabyte or 18 exabyte address space, OS X uses the hard disk to hold data that is not currently in use. As memory gets full, sections of memory that are not being used are written to disk to make room for data that is needed now. The portion of the disk that stores the unused data is known as the backing store because it provides the backup storage for main memory.
  
  [...]
  
  Note: Unlike most UNIX-based operating systems, OS X does not use a preallocated disk partition for the backing store. Instead, it uses all of the available space on the machine’s boot partition.


(emphasis mine)

Hence, at around this point, I'd go for some pragmatic workaround. Perhaps try increasing your disk space even further to check for a correlation in the obtainable swap size, thus checking the 12.5% limitation mentioned above.