title: Can Pandas DataFrames with ~500,000 columns be processed on a personal computer?
body: <p>I'm attempting to use scikit-learn's <code>KMeans</code> algorithm to cluster a group of sets based on overlap. There are roughly 500,000 potential members of each set (though each set itself isn't too large).</p>
  
  <p>It is my understanding that to use sklearn's <code>KMeans</code> implementation, I would need to represent each set as in 500,000-space with each column representing a binary value. </p>
  
  <p>Is this too large for a personal computer to process. Is there a more efficient way to perform this operation?</p>