title: Increasing performance of nearest neighbors of rows in Pandas
body: <p>I am given 8000x3 data set similar to this one:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  
  df = pd.DataFrame(np.random.rand(8000,3), columns=list('XYZ'))
  </code></pre>
  
  <p>So for a visual reference, df.head(5) looks like this:</p>
  
  <pre><code>          X         Y         Z
  0  0.462433  0.559442  0.016778
  1  0.663771  0.092044  0.636519
  2  0.111489  0.676621  0.839845
  3  0.244361  0.599264  0.505175
  4  0.115844  0.888622  0.766014
  </code></pre>
  
  <p>I'm trying to implement a method that when given an index from the dataset, it will return similar items from the dataset (in some reasonable way).  For now I have:</p>
  
  <pre><code>def find_similiar_items(item_id):
      tmp_df = df.sub(df.loc[item_id], axis='columns')
      tmp_series = tmp_df.apply(np.square).apply(np.sum, axis=1)
      tmp_series.sort()
      return tmp_series
  </code></pre>
  
  <p>This method takes your row, then subtracts it from each other row in the dataframe, then calculates the norm for each row.  So this method simply returns a series of the nearest points to your given point using the euclidean distance.</p>
  
  <p>So you can get the nearest 5 points, for instance, with:</p>
  
  <pre><code>df.loc[find_similiar_items(5).index].head(5)
  </code></pre>
  
  <p>which yields:</p>
  
  <pre><code>             X         Y         Z
  5     0.364020  0.380303  0.623393
  4618  0.369122  0.399772  0.643603
  4634  0.352484  0.402435  0.619763
  5396  0.386675  0.370417  0.600555
  3229  0.355186  0.410202  0.616844
  </code></pre>
  
  <p>The problem with this method is that  it takes roughly half a second each time I call it.  This isn't acceptable for my purpose, so I need to figure out how to improve the performance of this method in someway.  So I have a few questions:</p>
  
  <p><strong>Question 1</strong> Is there perhaps a more efficient way of simply calculating the euclidean distance as above?  </p>
  
  <p><strong>Question 2</strong> Is there some other technique that will yield <em>reasonable</em> results like this (the euclidean distance isn't import for instance).  Computation time is more important than memory in this problem and pre-processing time is not important; so I would be willing, for instance, to construct a new dataframe that has the size of the Cartesian product (n^2) the original dataframe (but anything more than that might become unreasonable)</p>