<p dir="auto"><strong>Is your feature request related to a problem? Please describe.</strong><br>
scipy.cluster.hierarchy.linkage uses double (float64) to store and do its computation for hierarchical clustering. However, I have a very large dataset (292000x292000) that I would like to perform hclust on but my computer is RAM limited. I have 252GB RAM and I think the clustering algorithm should be able to work on my dataset when all values are stored and computed using float16s instead.</p>
<p dir="auto">For large datasets on machines with insufficient RAM to store and compute on Arrays of float64s, it would be awesome if computation could be done on a different precision float to reduce the memory footprint.</p>
<p dir="auto">Additionally, adding choices for datatypes could be very useful for almost all scipy functions.</p>
<p dir="auto"><strong>Describe the solution you'd like</strong><br>
Allow for an argument to specify what datatype you'd like to use (e.g. np.float64, np.float32, np.float16)</p>
<p dir="auto">The argument could be like dtype='np.double' by default but changable to whatever datatype is chosen.</p>
<p dir="auto">I try the scipy ward clustering, when calculating linkage, it report follow error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ward_h = linkage(X, method='ward', metric='euclidean')
Python(2557,0x7fff732cc310) malloc: *** mach_vm_map(size=18446744067627675648) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
&lt;ipython-input-10-769ae7c53f7c&gt; in &lt;module&gt;()
----&gt; 1 ward_h = linkage(X, method='ward', metric='euclidean')

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.pyc in linkage(y, method, metric)
    652             Z = np.zeros((n - 1, 4))
    653             _hierarchy_wrap.linkage_euclid_wrap(dm, Z, X, m, n,
--&gt; 654                                               int(_cpy_euclid_methods[method]))
    655     return Z
    656 

MemoryError: out of memory while computing linkage"><pre class="notranslate"><code class="notranslate">ward_h = linkage(X, method='ward', metric='euclidean')
Python(2557,0x7fff732cc310) malloc: *** mach_vm_map(size=18446744067627675648) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
&lt;ipython-input-10-769ae7c53f7c&gt; in &lt;module&gt;()
----&gt; 1 ward_h = linkage(X, method='ward', metric='euclidean')

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/cluster/hierarchy.pyc in linkage(y, method, metric)
    652             Z = np.zeros((n - 1, 4))
    653             _hierarchy_wrap.linkage_euclid_wrap(dm, Z, X, m, n,
--&gt; 654                                               int(_cpy_euclid_methods[method]))
    655     return Z
    656 

MemoryError: out of memory while computing linkage
</code></pre></div>
<p dir="auto">How can I solve this？</p>
<p dir="auto">The data set I use is here: <a href="https://dl.dropboxusercontent.com/u/68126956/df.csv" rel="nofollow">https://dl.dropboxusercontent.com/u/68126956/df.csv</a>.</p>
<p dir="auto">Thanks.</p>