<p dir="auto">Python 2.7.9, scikit-learn 0.16.1<br>
MacBook Pro, 16GB RAM, MacOS 10.10.3</p>
<p dir="auto">When running the following code in an iPython notebook, it runs for a long time producing no output, and then the iPython kernel crashes and has to restart.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
from sklearn.manifold import TSNE

X = np.random.random((100000, 100))
tsne = TSNE(n_components=2, perplexity=40, learning_rate=100, verbose=2).fit_transform(X)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">manifold</span> <span class="pl-k">import</span> <span class="pl-v">TSNE</span>

<span class="pl-v">X</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">random</span>((<span class="pl-c1">100000</span>, <span class="pl-c1">100</span>))
<span class="pl-s1">tsne</span> <span class="pl-c1">=</span> <span class="pl-v">TSNE</span>(<span class="pl-s1">n_components</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">perplexity</span><span class="pl-c1">=</span><span class="pl-c1">40</span>, <span class="pl-s1">learning_rate</span><span class="pl-c1">=</span><span class="pl-c1">100</span>, <span class="pl-s1">verbose</span><span class="pl-c1">=</span><span class="pl-c1">2</span>).<span class="pl-en">fit_transform</span>(<span class="pl-v">X</span>)</pre></div>
<p dir="auto">Running the same code from the command line produces one line of output</p>
<p dir="auto"><code class="notranslate">[t-SNE] Computing pairwise distances...</code></p>
<p dir="auto">and then it dies.</p>
<p dir="auto">The error is pretty much the same as reported in <a href="https://github.com/scikit-learn/scikit-learn/issues/3088" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/3088/hovercard">this older issue</a>. I'll attach a crash log next, but I think it's out of memory because if I reduce the rows of X to 10000, then it runs to completion without error.</p>
<p dir="auto">Some suggested improvements:</p>
<ul dir="auto">
<li>Estimate the memory required to carry out this operation and warn or error out earlier.</li>
<li>Try to recover from OOM errors in libBLAS more gracefully.</li>
<li>If possible, reduce the memory requirements of sklearn.manifold.TSNE or implement out-of-core</li>
<li>Document the memory requirements in sklearn.manifold (and sklearn.decomposition), and perhaps warn about potential OOM errors and what they'll look like to users. You've already nicely documented the runtime complexities of the algorithms but have not documented their space complexities.</li>
</ul>
<p dir="auto">[The bug here is that some BLAS libraries will crash if you pass in &gt; 16 GiB arrays, because they use 32-bit indices internally. The most obvious solution is to break the dot call into multiple calls to dgemm, though there are also <a href="https://github.com/numpy/numpy/issues/5533#issuecomment-114309520" data-hovercard-type="issue" data-hovercard-url="/numpy/numpy/issues/5533/hovercard">other possibilities</a>. Original report follows:]</p>
<hr>
<p dir="auto">On python 2.7.8, numpy 1.9.1, on Mac OS X:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy
numpy.random.seed(1)
X = numpy.random.random((50000,100))
numpy.dot(X, X.T)"><pre class="notranslate"><code class="notranslate">import numpy
numpy.random.seed(1)
X = numpy.random.random((50000,100))
numpy.dot(X, X.T)
</code></pre></div>
<p dir="auto">Results in:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Segmentation fault: 11"><pre class="notranslate"><code class="notranslate">Segmentation fault: 11
</code></pre></div>
<p dir="auto">Segfault doesn't occur on smaller arrays (e.g. 30000x100 is fine).  In case it's useful, some linkage info:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; numpy.__config__.show()
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3)]
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH']
    define_macros = [('NO_ATLAS_INFO', 3)]
openblas_lapack_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; numpy.__config__.show()
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3)]
atlas_blas_threads_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-DAPPLE_ACCELERATE_SGEMV_PATCH']
    define_macros = [('NO_ATLAS_INFO', 3)]
openblas_lapack_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE
</code></pre></div>