title: python Aggregate (groupby) 2d-matrix
body: <p>I have a square array/matrix of data (roughly 2000 rows and columns)
  and I want to aggregate it into a smaller array (roughly 1000 rows and columns)</p>
  
  <p>I have a list of main_ids, which corresponds to the rows/columns in the orriginal array.
  e.g. new_ids = [0,0,0,1,1,2,...] would send aggregate the first 3 rows/columns to new row/column 1, and then the 4th,5th row/column aggregated to the 2nd and so on...</p>
  
  <p>So far I have the following code but I would like to optomize it if possible</p>
  
  <pre><code>import numpy as np
  new_ids = np.array(new_ids)
  unew_ids = unique(new_ids)
  unew_ids.sort()
  
  #Collapse rows
  rowMat = np.zeros([len(new_ids),len(unew_ids)], dtype=np.float64)
  
  for i in range(0,len(mat[0])):
      for j in range(0, len(unew_ids)):
          rowMat[i,j] = np.sum(mat[i][new_ids == unew_ids[j]])
  
  #Collapse columns
  outMat = np.zeros([len(unew_ids),len(unew_ids)], dtype=np.float64)
  
  for i in range(0, len(unew_ids)):
      for j in range(0, len(unew_ids)):
          outMat[j,i] = np.sum(rowMat[:,i][new_ids == unew_ids[j]])
  
  return outMat
  </code></pre>
  
  <p>I have tried experimenting with matplotlib.mlab.rec_groupby, however for that I need (or I think I need) to build a new array of (new_ids, row) where row is a numpy array with a row from the orriginal matrix, however that doesn't seem to work.</p>
  
  <p>[Edit:]</p>
  
  <p>As an added problem, I'm not actually doing a sum when I aggregate, I'm doing a custom function that needs all the numbers at once (doing it by rows and then columns wasn't working) </p>
  
  <p>{roughly - my_function = (take sum of elements, if any were negative remove those from sum and then mulitply the whole thing by 1/(1+sum of negative elements))}</p>
  
  <p>I think what I need to do is:</p>
  
  <pre><code>outputmat = [ [my_function(input_mat[new_ids_arr == tuple])] for tuple in unique_arr.ravel()]
  
  outputmat = np.array(outputmat)
  outputmat.reshape(something)
  </code></pre>
  
  <p>where new_ids_arr is an array of the form [[(10,10),(10,10),...][(10,10),(10,10),(10,12),...],...] - i.e. tuples of where everything is going post aggregation.
  I'm hoping I can find a funciton (or build one) that does new_ids_arr = new_ids * new_ids^Transpose ... or similar</p>
  
  <p>unique_arr = is an array of the same size/shape as output, and with each element containing a tuple (e.g. unique_arr[0,0] = (10,10)  meaing take put all the input_matrix elements that where in a place flagged as 10,10 into this location).</p>
  
  <p>Any thoughts?
  Particularly on how I could easily build an array of tuples given a list?</p>