What does sklearn's pairwise_distances with metric='correlation' do? I've put different values into this function and observed the output. But I can't find a predictable pattern in what is being outputed.

Then I tried digging through the function itself, but its confusing because it can do a number of different calculations.

Docs

According to the docs: 

Compute the distance matrix from a vector array X and optional Y.


I see it returns a matrix of height and width equal to the number of nested lists inputted, implying that it is comparing each one. 

But otherwise I'm having a tough time understanding what its doing and where the values are coming from.

Examples I've tried:

pairwise_distances([[1]], metric='correlation')
&gt;&gt;&gt; array([[0.]])

pairwise_distances([[1], [1]], metric='correlation')
&gt;&gt;&gt; array([[ 0., nan],
&gt;&gt;&gt;        [nan,  0.]])

# returns same as last input although input values differ
pairwise_distances([[1], [2]], metric='correlation')
&gt;&gt;&gt; array([[ 0., nan],
&gt;&gt;&gt;        [nan,  0.]])

pairwise_distances([[1,2], [1,2]], metric='correlation')
&gt;&gt;&gt; array([[0.00000000e+00, 2.22044605e-16],
&gt;&gt;&gt;        [2.22044605e-16, 0.00000000e+00]])

# returns same as last input although input values differ
# I incorrectly expected more distance because input values differ more
pairwise_distances([[1,2], [1,3]], metric='correlation')
&gt;&gt;&gt; array([[0.00000000e+00, 2.22044605e-16],
&gt;&gt;&gt;       [2.22044605e-16, 0.00000000e+00]])



Computing correlation distance with Scipy 

I don't understand where the sklearn 2.22044605e-16 value is coming from if scipy returns 0.0 for the same inputs.

# Scipy
import scipy
scipy.spatial.distance.correlation([1,2], [1,2])
&gt;&gt;&gt; 0.0

# Sklearn
pairwise_distances([[1,2], [1,2]], metric='correlation')
&gt;&gt;&gt; array([[0.00000000e+00, 2.22044605e-16],
&gt;&gt;&gt;        [2.22044605e-16, 0.00000000e+00]])


I'm not looking for a high level explanation but an example of how the numbers are calculated.
Y = pdist(X, 'correlation')


Computes the correlation distance between vectors u and v. This is

$$ 1-\frac{(u-\overline{u})\dot{}(v-\overline{v})}{||u-\overline{u}||_2*||v-\overline{v}||_2} $$

where $\overline{u}$ is the mean of the elements of vector $u$, and $x\dot{}y$ is the dot product of $x$ and $y$.

The correlation between any vector which have ONLY TWO entries is always 0 (or nearly zero: $2*10^-16$), why? Because correlation distance measures the distance as the linearity between the data.

When I have [1,2] and [1,2] the equation $y=x$ fits perfectly, when I have [1,2] and [1,3] the equation $y=x+1$ also fits perfectly. The correlation distance says wheter a equation can be drawn for the data, in both cases the equation is perfect.

If you want to try getting a different result, try putting 2 vectors of three elements, and you will see changes.

Try with: [[1,2],[2,3],[3,4]] and [[1,4],[3,8],[-5,6]]. But first, plot them and you will comprehen what 'correlation' measures.