Dummy coding a column in R with multiple levels I have a dependent variable measuring the net revenue. One of the major predictor affecting this is "product" i.e. the product sold to the customer. My randomly sampled dataset contains 1.4 million entries.

Products are assigned a specific categorical value.
I feel that using dummy variables to represent the products would be apt however, there are 4481 levels of products. I do not know how to code so many levels in R.

model.matrix(~ product, data=salesdata) 
 returns an error. (Needs 38.4GB of memory)

Can someone guide me a little on how to code these categorical variables?



Dependent: Net revenue (quantitative)
Independent: Product code (quantitative but treated as qualitative since values are nominal)
You can use either sparse matrices or feature hashing.

Sparse Matrix

I suppose that using a sparse matrix is the only choice. I suspect that this line of code will work. This uses the Matrix package.

sparseProducts &lt;- sparse.model.matrix(~ product, data=salesdata)


Take my example:

sparseDiagonalMatrix &lt;- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))


each column represents a different factor, this will yield:

1  1 . . . . . . . . .
2  1 1 . . . . . . . .
3  1 . 1 . . . . . . .
4  1 . . 1 . . . . . .
5  1 . . . 1 . . . . .
6  1 . . . . 1 . . . .
7  1 . . . . . 1 . . .
8  1 . . . . . . 1 . .
9  1 . . . . . . . 1 .
10 1 . . . . . . . . 1

&gt; class(sparseDiagonalMatrix)
[1] "dgCMatrix"
attr(,"package")
[1] "Matrix"


alternatively you can remove the intercept and have all zeros represent class 1

sparseDiagonalMatrix &lt;- sparse.model.matrix(~., data.frame(V1 = as.factor(seq(1, 10))))[, -1, drop=FALSE]

10 x 9 sparse Matrix of class "dgCMatrix"
V12 V13 V14 V15 V16 V17 V18 V19 V110
1    .   .   .   .   .   .   .   .    .
2    1   .   .   .   .   .   .   .    .
3    .   1   .   .   .   .   .   .    .
4    .   .   1   .   .   .   .   .    .
5    .   .   .   1   .   .   .   .    .
6    .   .   .   .   1   .   .   .    .
7    .   .   .   .   .   1   .   .    .
8    .   .   .   .   .   .   1   .    .
9    .   .   .   .   .   .   .   1    .
10   .   .   .   .   .   .   .   .    1

&gt; class(sparseDiagonalMatrix)
[1] "dgCMatrix"
attr(,"package")
[1] "Matrix"


You will need a package that supports sparse matrices for measuring the net revenue though. Fortunately, most modern mainstream packages support sparse matrices.

Feature Hashing

Here is a great explanation of feature hashing in R (among other techniques) which is also an alternative, specially useful when you have hundreds of thousands or millions of multiple levels. 

https://amunategui.github.io/feature-hashing/