<p dir="auto">The documentation for the function, scipy.cluster.hierarchy.linkage, claims that the function can receive either a condensed (1D) or redundant distance matrix. Alternatively, the original observation vectors can also be given as well, and scipy.cluster.hierarchy.linkage will use scipy.spatial.distance.pdist to calculate the distance matrix.</p>
<p dir="auto">It seems to me, from testing as well as reading the code, that scipy.cluster.hierarchy.linkage cannot handle redundant distance matrices. According to the code, if the dimension of the input array is 2, then it goes ahead and treats it as a matrix of original observation vectors.</p>
<p dir="auto">from line 636 in scipy/cluster/hierarchy.py  0.14.0</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="    elif len(s) == 2:
        X = y
        n = s[0]
        m = s[1]
        if method not in _cpy_linkage_methods:
            raise ValueError('Invalid method: %s' % method)
        if method in _cpy_non_euclid_methods:
            dm = distance.pdist(X, metric)
            Z = np.zeros((n - 1, 4))
            _hierarchy_wrap.linkage_wrap(dm, Z, n,
                                       int(_cpy_non_euclid_methods[method]))
        elif method in _cpy_euclid_methods:
            if metric != 'euclidean':
                raise ValueError((&quot;Method '%s' requires the distance metric &quot;
                                 &quot;to be euclidean&quot;) % method)
            dm = distance.pdist(X, metric)
            Z = np.zeros((n - 1, 4))
            _hierarchy_wrap.linkage_euclid_wrap(dm, Z, X, m, n,
                                              int(_cpy_euclid_methods[method]))"><pre class="notranslate"><code class="notranslate">    elif len(s) == 2:
        X = y
        n = s[0]
        m = s[1]
        if method not in _cpy_linkage_methods:
            raise ValueError('Invalid method: %s' % method)
        if method in _cpy_non_euclid_methods:
            dm = distance.pdist(X, metric)
            Z = np.zeros((n - 1, 4))
            _hierarchy_wrap.linkage_wrap(dm, Z, n,
                                       int(_cpy_non_euclid_methods[method]))
        elif method in _cpy_euclid_methods:
            if metric != 'euclidean':
                raise ValueError(("Method '%s' requires the distance metric "
                                 "to be euclidean") % method)
            dm = distance.pdist(X, metric)
            Z = np.zeros((n - 1, 4))
            _hierarchy_wrap.linkage_euclid_wrap(dm, Z, X, m, n,
                                              int(_cpy_euclid_methods[method]))
</code></pre></div>
<p dir="auto">where <code class="notranslate">s</code> is the shape tuple of the input matrix.</p>
<p dir="auto">To fix, the documentation should either be corrected, or perhaps scipy.spatial.distance.is_valid_dm should be used to assess whether the input 2D array is a redundant distance matrix versus a vector of observations.</p>
<p dir="auto">the document for scipy.cluster.hierarchy.linkage() function states that it accepts redundant distance matrix as input:</p>
<p dir="auto"><a href="http://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.linkage.html" rel="nofollow">http://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.linkage.html</a><br>
"y : ndarray<br>
A condensed or redundant distance matrix. A condensed distance matrix is a flat array containing the upper triangular of the distance matrix. This is the form that pdist returns. Alternatively, a collection of m observation vectors in n dimensions may be passed as an m by n array."</p>
<p dir="auto">This feature of accepting redundant distance matrix as input is also introduced by many popular books like "SciPy and NumPy: An Overview for Developers" By Eli Bressert:<br>
<a href="http://books.google.co.uk/books?id=c-xzkDMDev0C&amp;pg=PA48&amp;dq=scipy+linkage+input&amp;hl=en&amp;sa=X&amp;ei=4BzUUZqpNoSbPaj3gLgN&amp;ved=0CDgQ6AEwAQ#v=onepage&amp;q=scipy%20linkage%20input&amp;f=false" rel="nofollow">http://books.google.co.uk/books?id=c-xzkDMDev0C&amp;pg=PA48&amp;dq=scipy+linkage+input&amp;hl=en&amp;sa=X&amp;ei=4BzUUZqpNoSbPaj3gLgN&amp;ved=0CDgQ6AEwAQ#v=onepage&amp;q=scipy%20linkage%20input&amp;f=false</a></p>
<p dir="auto">However, when distance matrix is specified as input to the linkage() function, it is silently considered to be observation vectors. This can be seen by reading the source code :<br>
<a href="https://github.com/scipy/scipy/blob/master/scipy/cluster/hierarchy.py">https://github.com/scipy/scipy/blob/master/scipy/cluster/hierarchy.py</a></p>
<p dir="auto">If the input is a 2-dimensional matrix, it is not checked by predicate scipy.spatial.distance.is_valid_dm(). Rather, it is simply taken as observation vectors for hierarchical clustering. It may be also seen from the following example:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="arr = scipy.array([[0,5],[1,5],[-1,5],[0,-5],[1,-5],[-1,-5], [-1.1,-5]])
pd = pdist(arr)
dm = squareform(pd)
linkage(pd)
linkage(dm)  # result is different from the previous "><pre class="notranslate"><span class="pl-s1">arr</span> <span class="pl-c1">=</span> <span class="pl-s1">scipy</span>.<span class="pl-en">array</span>([[<span class="pl-c1">0</span>,<span class="pl-c1">5</span>],[<span class="pl-c1">1</span>,<span class="pl-c1">5</span>],[<span class="pl-c1">-</span><span class="pl-c1">1</span>,<span class="pl-c1">5</span>],[<span class="pl-c1">0</span>,<span class="pl-c1">-</span><span class="pl-c1">5</span>],[<span class="pl-c1">1</span>,<span class="pl-c1">-</span><span class="pl-c1">5</span>],[<span class="pl-c1">-</span><span class="pl-c1">1</span>,<span class="pl-c1">-</span><span class="pl-c1">5</span>], [<span class="pl-c1">-</span><span class="pl-c1">1.1</span>,<span class="pl-c1">-</span><span class="pl-c1">5</span>]])
<span class="pl-s1">pd</span> <span class="pl-c1">=</span> <span class="pl-en">pdist</span>(<span class="pl-s1">arr</span>)
<span class="pl-s1">dm</span> <span class="pl-c1">=</span> <span class="pl-en">squareform</span>(<span class="pl-s1">pd</span>)
<span class="pl-en">linkage</span>(<span class="pl-s1">pd</span>)
<span class="pl-en">linkage</span>(<span class="pl-s1">dm</span>)  <span class="pl-c"># result is different from the previous </span></pre></div>
<p dir="auto">I think this issue has relatively serious consequence if users do not check their clustering results carefully.</p>
<p dir="auto">I suggest: either update the documentation for linkage() function to reflect the real functionality, or add a predicate check using scipy.spatial.distance.is_valid_dm() if two dimensional matrix is given as input so distance matrix is processed properly in the linkage() function.</p>