What makes OSX say that it's low free space on the boot volume for the program memory?
First off, "system memory" does not necessarily refer to available hard disk space. It sounds like you don't have enough free hard disk space on your system disk to rebuild Mail's index. One thing to note is that an index rebuild is usually done by making a clean copy of the index, keeping the old file around until the rebuild is complete. Even if you've symlinked the file to the external drive, the index may be a separate hidden file on the SSD.

If the message truly did refer to "system memory," it's also possible that you ran out of swap space, which can only reside on the system disk.

Move as much as you can off the SSD and try one more time. You may also want to look around for the files caused by the failed index rebuilds and delete them (making sure, of course, you do not delete the real one).