Recommended sequence clustering algorithm for transcriptome data
It does sound like you have a lot of data.  

I would first try Robert Edgar's other newer tool UPARSE which is faster and can handle more data using the free 32-bit version.  I think you'll mainly be limited by machine memory though, right?

Did you try CD-Hit?