title: How to pivot a large data frame (or matrix) without hitting memory limits?
body: <p>I have a data frame of millions of rows that represent customers buying products. I need to reshape it from this long format to a wide format such that each row represents the number of times a customer bought each product.</p>
  <p>For example, this creates a mock dataset.</p>
  <pre><code># Create mock dataset of 1 million customers and thousands of products.
  df &lt;- data.frame(cust = 1:1e6)
  products = paste0(sample(letters, 1e6, replace = T), &quot; - &quot;, 
                    sample(1:1000, 1e6, replace = T))
  df$prod &lt;- products
  df$orders &lt;- floor(rpois(1e6, 2))
  </code></pre>
  <p>When I use dplyr's <code>pivot_wider()</code>, I get a memory error: <code>Error: cannot allocate vector of size 8.0 Mb</code></p>
  <p>For example:</p>
  <pre><code>wide = pivot_wider(df, 
                     id_cols = cust, 
                     names_from = prod, 
                     values_from = orders
                     )
  </code></pre>
  <p>How can I pivot a large data frame (or matrix) without hitting memory limits?</p>
  <p>The resulting data should look like this:</p>
  <pre><code>cust item1 item2 ...
  1     0     2
  2     4     0
  ...   ...   ...
  1000000 0   1
  </code></pre>
  <ul>
  <li>I do not NEED to use a data frame</li>
  <li>I do not need to use the tidyverse; base R solutions are welcome</li>
  </ul>