title: Triangle vs. Square distance matrix for Hierarchical Clustering Python?
body: <p>I have been experimenting with <code>Hierarchical Clustering</code> and in <code>R</code> it's so simple <code>hclust(as.dist(X),method="average")</code>  . I found a method in <code>Python</code> that is pretty simple as well, except I'm a little confused on what's going on with my input distance matrix.  </p>
  
  <p>I have a similarity matrix (<code>DF_c93tom</code> w/ a smaller test version called <code>DF_sim</code>) that I convert into a dissimilarity matrix <code>DF_dissm = 1 - DF_sim</code>.</p>
  
  <p>I use this as input into <code>linkage</code> from <code>scipy</code> but the documentation says it takes in a square or triangle matrix.  I get a different cluster for inputing a <code>lower triangle</code>, <code>upper triangle</code>, and <code>square matrix</code>.  Why is this? It wants an upper triangle from the documentation but the lower triangle cluster looks REALLY similar.</p>
  
  <p><strong>My question, why are all the clusters different? Which one is correct?</strong> </p>
  
  <p>This is the documentation for the input distance matrix for <code>linkage</code></p>
  
  <pre><code>y : ndarray
  A condensed or redundant distance matrix. A condensed distance matrix is a flat array containing the upper triangular of the distance matrix. 
  </code></pre>
  
  <p>Here is my code:</p>
  
  <pre><code>import matplotlib.pyplot as plt
  import seaborn as sns
  import numpy as np
  import pandas as pd
  from scipy.cluster.hierarchy import dendrogram, linkage
  
  %matplotlib inline
  
  #Test Data
  DF_sim = DF_c93tom.iloc[:10,:10] #Similarity Matrix
  DF_sim.columns = DF_sim.index = range(10) 
  #print(DF_test)
  #           0  1         2         3  4  5  6  7  8  9
  # 0  1.000000  0  0.395833  0.083333  0  0  0  0  0  0
  # 1  0.000000  1  0.000000  0.000000  0  0  0  0  0  0
  # 2  0.395833  0  1.000000  0.883792  0  0  0  0  0  0
  # 3  0.083333  0  0.883792  1.000000  0  0  0  0  0  0
  # 4  0.000000  0  0.000000  0.000000  1  0  0  0  0  0
  # 5  0.000000  0  0.000000  0.000000  0  1  0  0  0  0
  # 6  0.000000  0  0.000000  0.000000  0  0  1  0  0  0
  # 7  0.000000  0  0.000000  0.000000  0  0  0  1  0  0
  # 8  0.000000  0  0.000000  0.000000  0  0  0  0  1  0
  # 9  0.000000  0  0.000000  0.000000  0  0  0  0  0  1
  
  #Dissimilarity Matrix
  DF_dissm = 1 - DF_sim
  
  #Redundant Matrix
  #np.tril(DF_dissm).T == np.triu(DF_dissm)
  #True for all values
  
  #Hierarchical Clustering for square and triangle matrices
  fig_1 = plt.figure(1)
  plt.title("Square")
  Z_square = linkage((DF_dissm.values),method="average")
  dendrogram(Z_square)
  
  fig_2 = plt.figure(2)
  plt.title("Triangle Upper")
  Z_triu = linkage(np.triu(DF_dissm.values),method="average")
  dendrogram(Z_triu)
  
  fig_3 = plt.figure(3)
  plt.title("Triangle Lower")
  Z_tril = linkage(np.tril(DF_dissm.values),method="average")
  dendrogram(Z_tril)
  
  plt.show()
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/Rq0XI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Rq0XI.png" alt="enter image description here"></a></p>