Approaches to high dimension pattern matching problem My apologies in advance as I am new to this. I have searched the internet and tried various processes and nothing seems to work or address this situation. 

I have a dataset of 30,000 transactions and 500,000 items. Average item size for a transaction is 50. The dataset is sparse, so the support number must be set quite low. Furthermore, the rules become more valuable the larger the number of items in the rule. 

I have tried running this in arules and the tests fail after exceeding 64 gb of RAM (the limit of the machine). I have tried reducing items and transactions to smaller subsets, but still hit this memory limit.

Ultimately, I am looking for ways to cluster large groups of similar accounts by selection of items and generate confidence and lift of various next items selected from those clusters.

My question: are there alternative, more efficient ways to do this, or other approaches to consider?

Thank you.
According to the following discussion on StackOverflow, a situation like that you've described can occur, when one of the variables in a data set is of unexpected type (for example, a factor instead of a character): https://stackoverflow.com/q/7246412/2872891.

Also, consider using package bigmemory, recommended in the accepted answer, or similar packages for big data analysis. For the latter, please see section "Large memory and out-of-memory data" in CRAN Task View High-Performance and Parallel Computing with R.

Finally, an additional note. There is an ecosystem of R packages, built around the arules package, which includes supporting packages for algorithms (arulesNBMiner), applications (arulesSequences, arulesClassify) and visualization (arulesViz). You are likely aware of that, but I have decided to include this good-to-know fact just in case and for the sake of completeness.