title: New classification method in machine learning python code optimization
body: <p>I've new method of classification. I've developed it in python. But code is very slow.</p>
  <p>Please, help me to optimize code below.</p>
  <pre><code>from joblib import Parallel, delayed
  from datetime import datetime
  from sklearn.metrics.pairwise import euclidean_distances
  import copy
  import pandas as pd
  import numpy as np    
  </code></pre>
  <p>This function for finding hyperspheres in space</p>
  <pre><code>def find_sphere(ind, df, dist_sq, sidx):
      objectCount = df.shape[0]
      sphere = dict()
      sphere['index'] = ind
      sphere['relatives'] = []
      sphere['class'] = df.at[ind, &quot;Class&quot;]
      indexes = sidx[:, ind]
      sphere['distances'] = dist_sq[ind, :]
      for i in range(objectCount):
          if df.at[ind, &quot;Class&quot;] != df.at[indexes[i], &quot;Class&quot;]:
              sphere['relatives'] = indexes[:i]
              sphere['radius'] = sphere['distances'][indexes[i]]
              break
      sphere['enemies'] = np.where(
          sphere['distances'] == sphere['radius'])[0]
      sphere['coverages'] = set()
      for enemy in sphere['enemies']:
          min_dist = dist_sq[enemy, sphere['relatives']].min()
          sphere['coverages'].update(
              sphere['relatives'][np.where(dist_sq[enemy, sphere['relatives']] == min_dist)[0]])
      sphere['relatives'] = set(sphere['relatives'])
      return sphere
  </code></pre>
  <p>This function is clustering objects in space</p>
  <pre><code>def find_groups(spheres):
      coverages = {x for d in spheres for x in d['coverages']}
      all_coverages = {x for d in spheres for x in d['coverages']}
      notSeenObjects = copy.deepcopy(spheres)
  
      groups = list()
      while (len(coverages) &gt; 0):
          # print(f'\t getting one of {len(coverages)} coverage(s)...')
          obj = coverages.pop()
          group = {d['index'] for d in notSeenObjects if obj in d['relatives']}
          if len(group) == 0:
              continue
          linkerCoverage = set()
          while True:
              linkerCoverage = linkerCoverage | {r for s in notSeenObjects if s['index'] in group
                                                 for r in s['relatives'] if r in all_coverages}
  
              notSeenObjects = [
                  s for s in notSeenObjects if not s['index'] in group]
  
              newObjects = {s['index'] for s in notSeenObjects if len(
                  s['relatives'].intersection(linkerCoverage)) &gt; 0}
  
              if len(newObjects) == 0:
                  break
  
              group = group | newObjects
              coverages = coverages - linkerCoverage
          groups.append(group)
      return groups
  </code></pre>
  <p>This function finding all standart objects. Means, new objects can be classified with this (standart) objects instead of seeing of all objects</p>
  <pre><code>def find_standart_objects(spheres, dists, groups):
      standartObjects = [(s['index'], s['radius'], s['class']) for s in spheres]
      for group in sorted(groups, key=len, reverse=True):
          candidates = [s for s in standartObjects if s[0] in group]
          for candidate in sorted(candidates, key=lambda x: x[1]):
              isRightRecognition = True
              for obj in spheres:
                  result = sorted([(dists[obj['index'], s[0]] / s[1], obj['class'], s[2])
                                   for s in standartObjects if s[0] != candidate[0]], key=lambda x: x[0])
                  mindist = result[0][0]
                  for r in result:
                      if r[0] != mindist:
                          break
                      isRightRecognition = isRightRecognition and r[1] == r[2]
                  if not isRightRecognition:
                      print(f'Standart found {candidate[0]}')
                      break
              if isRightRecognition:
                  standartObjects = [
                      s for s in standartObjects if s[0] != candidate[0]]
      return standartObjects
  </code></pre>
  <p>This part is using that functions</p>
  <pre><code>df = pd.read_csv(&quot;Dry_Bean.txt&quot;, sep='\t') #https://archive.ics.uci.edu/ml/machine-learning-databases/00602/DryBeanDataset.zip
  objectCount = df.shape[0]
  dist_sq = euclidean_distances(df.iloc[:, :-1]).round(15)
  sidx = np.argsort(dist_sq, axis=0)
  spheres = Parallel(n_jobs=6)(delayed(find_sphere)
                               (ind, df, dist_sq, sidx) for ind in range(0, objectCount))
  groups = find_groups(spheres)
  standartObjects = find_standart_objects(spheres, dist_sq, groups)
  </code></pre>
  <p>Bottle neck parts are find_standart_objects and find_sphere functions.</p>