title: How to efficiently create a sparse vector in python?
body: <p>I have a dictionary of keys where each value should be a sparse vector of a huge size (~ 700000 elements, maybe more). How do I efficiently grow / build this data structure.
  Right now my implementation works only for smaller sizes.</p>
  
  <pre><code>myvec = defaultdict(list)
  for id in id_data:
      for item in item_data:
          if item in item_data[id]:
              myvec[id].append(item * 0.5)
          else:
              myvec[id].append(0)
  </code></pre>
  
  <p>The above code when used with huge files quickly eats up all the available memory. I tried removing the <code>myvec[id].append(0)</code> condition and store only non-zero values because the length of each <code>myvec[id]</code> list is constant. That worked on my huge test file with a decent memory consumption but I'd rather find a better way to do it.</p>
  
  <p>I know that there are different type of sparse arrays/matrices for this purpose but I have no intuition which one is better. I tried to use <code>lil_matrix</code> from numpy package instead of <code>myvec</code> dict but it turned out to be much slower than the above code.</p>
  
  <p>So the problem basically boils down to the following two questions:</p>
  
  <ol>
  <li><p>Is it possible to create a sparse data structure <em>on the fly</em> in python?</p></li>
  <li><p>How can one create such sparse data structure with decent speed?</p></li>
  </ol>