title: Why is there no native string folding in Pandas?
body: <p>I'm building an application to read from a SQL database into Pandas for analysis.  The data is 'medium data' - too big for one computer (8GB RAM) to hold in memory.  I really didn't want the cost and hassle of spinning up AWS instances constantly and getting beefier hardware was difficult (I work in non profit), so I looked to optimize my own data read memory cost.</p>
  
  <p>I've spent a long time implementing this solution from mobify: <a href="http://www.mobify.com/blog/sqlalchemy-memory-magic/" rel="nofollow">http://www.mobify.com/blog/sqlalchemy-memory-magic/</a></p>
  
  <p>Specifically method 3: they use a dictionary that stores all unique string values.  This avoids duplication of objects holding the same string value, by instead passing a reference to the same string.  I took their code and implemented it and the results have been very impressive (reduction of memory use by 2-10x, depending on data slice).  </p>
  
  <p>It was straightforward enough that I'm confused as to why Pandas doesn't have this natively. I'm a noob to the world of Pandas, but it seems like duplication of strings within large datasets is a given these day.  Is there any drawback to a default string folding in the DataFrames?  Am I missing something here?</p>
  
  <p><strong>TL;DR</strong> Weakness of pandas is the high memory cost.  String folding seems like a straightforward way to significantly reduce the memory overhead.  Why doesn't it have it?</p>