Upgrading the from OS X 10.6 to OS X Server 10.6 in-place to remove the 10-user FTP limit in client?
No - server needs additional packages to be run at install time and there is no general upgrade path or migration assistant to move a client mac to server.

Even at $499 Snow Leopard Server provides a lot of bang for the buck - but if you can compile open source and don't need this NOW a free solution might make more sense.

I've never had to bypass the limit - but my first step would be turning off the mac delivered ftpd and compiling Pure-FTPd (you need to use PAM when you compile for Max OS X)

Once you confirm the new binary works as planned for more than 10 users, you can dig into launchd and set it up to call your ftpd instead of the built in one.