Approaches to high dimension pattern matching problem
According to the following discussion on StackOverflow, a situation like that you've described can occur, when one of the variables in a data set is of unexpected type (for example, a factor instead of a character): https://stackoverflow.com/q/7246412/2872891.

Also, consider using package bigmemory, recommended in the accepted answer, or similar packages for big data analysis. For the latter, please see section "Large memory and out-of-memory data" in CRAN Task View High-Performance and Parallel Computing with R.

Finally, an additional note. There is an ecosystem of R packages, built around the arules package, which includes supporting packages for algorithms (arulesNBMiner), applications (arulesSequences, arulesClassify) and visualization (arulesViz). You are likely aware of that, but I have decided to include this good-to-know fact just in case and for the sake of completeness.