title: MemoryError, Killed
body: <p>I am trying to create two matrices key_mat and c_mat
  c_mat initially is (1,847)
  fixd_len = 847</p>
  
  <p>while the number is not 0, Im trying to pile up key_mat and c_mat</p>
  
  <p>so this could become some(n,847) Col :1 --> n</p>
  
  <p>Each element in Cols of key_mat and c_mat is either 1 or 0 int type;</p>
  
  <p>its not dot product.. specifically i need bit-wise so I am doing element-wise operation;</p>
  
  <pre><code>number = int(''.join(map(str,key_mat.tolist()[0])),2)
          #Build matrix of Keys
          #print key_mat
          #print c_mat
          while number &gt;= 0:
              #Divide integer&gt;&gt; by 2 until it covers all rows
              number &gt;&gt;= 1
              key_mat = np.concatenate((key_mat, np.matrix(list((np.binary_repr(number)).zfill(fixd_len)), dtype=int)))
              c_mat = np.concatenate((c_mat, c_mat))
  </code></pre>
  
  <p>Does, this increase memory alot?</p>
  
  <p>I suppose c_mat and key_mat are in shared RAM memory and key_mat and c_mat assignment happens over there and not cache.</p>
  
  <p>im running it in a VPS and on my machine which is core i7, 8Gb Win 8;</p>
  
  <p>PS: Consider 847 bit places..</p>
  
  <p>938439603600587528746394711938657107663969949193687942084737423845328945327403963493426274822541422606069252398088182827397836333287780407720182613329988145004965865323862822167078543736143176539997470989737828269291292380585577139908076735904949708259328L</p>
  
  <p>possible bits and that number being the highest possible decimal number;</p>
  
  <p>you start dividing >> by 2, and you re assign in the same variable.</p>
  
  <p>Notice this quick n dirty code, there is no strict type assigned and var are reused;</p>
  
  <p>large matrix var is re assigned dynamically.</p>
  
  <p>What is occupying memory??</p>
  
  <p>I am not sure.</p>
  
  <p>UPDATE: I found <a href="https://stackoverflow.com/questions/19085012/numpy-memory-error-creating-huge-matrix?lq=1">this</a> interesting. still reading it.</p>
  
  <p>UPDATE2: The above update gave good insight (esp answers and comments)</p>
  
  <p>and if someone could tell me how to approach this problem (847 is still not very big). how do you tackle memory tight situation here? Would be grateful..</p>