title: Multiprocessing with python application to shave running time that is currently 36 hours
body: <p>I am currently working on a data mining project that is creating a similarity matrix that is 18000x18000</p>
  
  <p>Here are the two methods which build the matrix</p>
  
  <pre><code>def CreateSimilarityMatrix(dbSubsetData, distancePairsList):
  
      global matrix
      matrix = [ [0.0 for y in range(dbSubsetData.shape[0])] for x in range(dbSubsetData.shape[0])]
      for i in range(len(dbSubsetData)): #record1
  
          SimilarityArray = []
          start = time.time()
          for j in range(i+1, len(dbSubsetData)): #record2
              Similarity = GetDistanceBetweenTwoRecords(dbSubsetData, i, j, distancePairsList)
  
              #The similarities are all very small numbers which might be why the preference value needs to be so precise. 
              #Let's multiply the value by a scalar 10 to give the values more range. 
              matrix[i][j] = Similarity * 10.0
              matrix[j][i] = Similarity * 10.0
  
          end = time.time()
      return matrix
  
  def GetDistanceBetweenTwoRecords(dbSubsetData, i, j, distancePairsList):
  
      Record1 = dbSubsetData.iloc[i]
      Record2 = dbSubsetData.iloc[j]
      columns = dbSubsetData.columns
      distancer    = 0.0
      distancec    = 0.0
  
      for i in range(len(Record1)):
  
          columnName = columns[i]
          Record1Value = Record1[i]
          Record2Value = Record2[i]
  
          if(Record1Value != Record2Value):
              ob = distancePairsList[distancePairsDict[columnName]-1]
  
              if(ob.attributeType == "String"):
                  strValue = Record1Value+":"+Record2Value
                  strValue2 = Record2Value+":"+Record1Value
  
                  if strValue in ob.distancePairs:
                      val = ((ob.distancePairs[strValue])**2) 
                      val = val * -1
                      distancec = distancec + val
                  elif strValue2 in ob.distancePairs:
                      val = ((ob.distancePairs[strValue2])**2)
                      val = val * -1
                      distancec = distancec + val
  
              elif(ob.attributeType == "Number"):
                  val = ((Record1Value - Record2Value)*ob.getSignificance())**2 
                  val = val * -1
                  distancer = distancer + val
  
      distance = distancer + distancec
      return distance
  </code></pre>
  
  <p>Each iteration is looping 18000x19 times (18000 for each row and 19 times for each attribute). The total number of iterations is (18000x18000x19)/2 since it is symmetric and therefore I only have to do one half of the matrix. This will take around 36 hours to complete, which is a timeframe I obviously want to shave down. </p>
  
  <p>I figured Multiprocessing is the trick. Since each row is independently generating numbers and fitting them to the matrix, I could run multiprocess with CreateSimilarityMatrix. So I created this in the function which will create my processes</p>
  
  <pre><code>matrix = [ [0.0 for y in range(SubsetDBNormalizedAttributes.shape[0])] for x in range(SubsetDBNormalizedAttributes.shape[0])] 
  if __name__ == '__main__':    
      procs = []
      for i in range(4):
          proc = Process(target=CreateSimilarityMatrix, args=(SubsetDBNormalizedAttributes, distancePairsList, i, 4))
          procs.append(proc)
          proc.start()
          proc.join()
  </code></pre>
  
  <p>CreateSimilarityMatrix is now changed to </p>
  
  <pre><code>def CreateSimilarityMatrix(dbSubsetData, distancePairsList, counter=0, iteration=1):
  
      global Matrix
  
      for i in range(counter, len(dbSubsetData), iteration): #record1
  
          SimilarityArray = []
          start = time.time()
          for j in range(i+1, len(dbSubsetData)): #record2
              Similarity = GetDistanceBetweenTwoRecords(dbSubsetData, i, j, distancePairsList)
              #print("Similarity Between Records",i,":",j," is ", Similarity)
  
              #The similarities are all very small numbers which might be why the preference value needs to be so precise. 
              #Let's multiply the value by a scalar 10 to give the values more range. 
              Matrix[i][j] = Similarity * 10.0
              Matrix[j][i] = Similarity * 10.0
  
          end = time.time()
          print("Iteration",i,"took",end-start,"(s)")
  </code></pre>
  
  <p>Currently this goes s-l-o-w. It's really slow. It takes minutes to start one process, then it takes minutes to start the next one. I thought these were supposed to run concurrently? Is my application of the process incorrect? </p>