title: Memory Error while pickling a data frame to disk
body: <p>I have a 51K X 8.5K data frame with just binary (1 or 0) values. </p>
  
  <p>I wrote the following code: </p>
  
  <h1>Pickling the data to the disk</h1>
  
  <pre><code>outfile=open("df_preference.p", "wb")
  pickle.dump(df_preference,outfile)
  outfile.close()
  </code></pre>
  
  <p>It throws me Memory Error as below: </p>
  
  <pre><code>MemoryError                               Traceback (most recent call last)
  &lt;ipython-input-48-de66e880aacb&gt; in &lt;module&gt;()
        2 
        3 outfile=open("df_preference.p", "wb")
  ----&gt; 4 pickle.dump(df_preference,outfile)
        5 outfile.close()
  </code></pre>
  
  <p>I am assuming it means this data is huge and it can't be pickled? But it just has binary values. </p>
  
  <p>Before this, I created this dataset from another data frame which had normal counts and lot of zeros. Used the following code: </p>
  
  <pre><code>df_preference=df_recommender.applymap(lambda x: np.where(x &gt;0, 1, 0))
  </code></pre>
  
  <p>This itself took some time to create df_preference. Same size of matrix. </p>
  
  <p>My concern is if it takes time to create a data frame using applymap and ii) doesn't even pickle the data frame due to memory error, then going ahead I need to do matrix factorization of this df_prefence using SVD and Alternating Least Squares. It would then be more slow? How to tackle this slow run and solve the memory error?</p>
  
  <p>Thanks</p>