title: how to get a kind of "maximum" in a matrix, efficiently
body: <p>I have the following problem: I have a matrix opened with <code>pandas</code> module, where each cell has a number between -1 and 1. What I wanted to find is the maximum "posible" value in a row that is also not the maximum value in another row. </p>
  
  <p>If for example 2 rows has their maximum value at the same column, I compare both values and take the bigger one, then for the row that has its maximum value smaller that the other row, I took the second maximum value (and do the same analysis again and again).</p>
  
  <p>To explain myself better consider my code  </p>
  
  <pre><code>import pandas as pd
  
  matrix = pd.read_csv("matrix.csv") 
  # this matrix has an id (or name) for each column 
  # ... and the firt column has the id of each row
  results = pd.DataFrame(np.empty((len(matrix),3),dtype=pd.Timestamp),columns=['id1','id2','max_pos'])
  
  l = len(matrix.col[[0]]) # number of columns
  
  while next = 1:
     next = 0
     for i in range(0, len(matrix)):
         max_column = str(0)
         for j in range(1, l): # 1 because the first column is an id
             if matrix[max_column][i] &lt; matrix[str(j)][i]:
                 max_column = str(j)
         results['id1'][i] = str(i) # I coul put here also matrix['0'][i]
         results['id2'][i] = max_column
         results['max_pos'][i] = matrix[max_column][i]
  
     for i in range(0, len(results)): #now I will check if two or more rows have the same max column
         for ii in range(0, len(results)):
         # if two id1 has their max in the same column, I keep it with the biggest 
         # ... max value and chage the other to "-1" to iterate again
             if (results['id2'][i] == results['id2'][ii]) and (results['max_pos'][i] &lt; results['max_pos'][ii]):
                 matrix[results['id2'][i]][i] = -1
                 next = 1
  </code></pre>
  
  <p>Putting an example:</p>
  
  <pre><code>#consider
  pd.DataFrame({'a':[1, 2, 5, 0], 'b':[4, 5, 1, 0], 'c':[3, 3, 4, 2], 'd':[1, 0, 0, 1]})
  
     a  b  c  d
  0  1  4  3  1
  1  2  5  3  0
  2  5  1  4  0
  3  0  0  2  1
  
  #at the first iterarion I will have the following result
  
  0  b  4 # this means that the row 0 has its maximum at column 'b' and its value is 4
  1  b  5
  2  a  5
  3  c  2
  
  #the problem is that column b is the maximum of row 0 and 1, but I know that the maximum of row 1 is bigger than row 0, so I take the second maximum of row 0, then:
  
  0  c  3
  1  b  5
  2  a  5
  3  c  2
  
  #now I solved the problem for row 0 and 1, but I have that the column c is the maximum of row 0 and 3, so I compare them and take the second maximum in row 3 
  
  0  c  3
  1  b  5
  2  a  5
  3  d  1
  
  #now I'm done. In the case that two rows have the same column as maximum and also the same number, nothing happens and I keep with that values.
  
  #what if the matrix would be 
  pd.DataFrame({'a':[1, 2, 5, 0], 'b':[5, 5, 1, 0], 'c':[3, 3, 4, 2], 'd':[1, 0, 0, 1]})
  
     a  b  c  d
  0  1  5  3  1
  1  2  5  3  0
  2  5  1  4  0
  3  0  0  2  1
  
  #then, at the first itetarion the result will be:
  
  0  b  5
  1  b  5
  2  a  5
  3  c  2
  
  #then, given that the max value of row 0 and 1 is at the same column, I should compare the maximum values
  # ... but in this case the values are the same (both are 5), this would be the end of iterating 
  # ... because I can't choose between row 0 and 1 and the other rows have their maximum at different columns...
  </code></pre>
  
  <p>This code works perfect to me if I have a matrix of 100x100 for example. But, if the matrix size goes to 50,000x50,000 the code takes to much time in finish it. I now that my code could be the most inneficient way to do it, but I don't know how to deal with this. </p>
  
  <p>I have been reading about threads in python that could help but it doesn't help if I put 50,000 threads because my computer doesn't use more CPU. I also tried to use some functions as <code>.max()</code> but I'm not able to get column of the max an compare it with the other max ...</p>
  
  <p>If anyone could help me of give me a piece of advice to make this more efficient I would be very grateful.</p>