<p dir="auto">I've got a machine running a GTX 980 Ti with 32 gb of ram (of which about half is used by the data loading pipeline) and a i7-5930K cpu driving a network. I'm running Ubuntu 14.04, I've got the drivers x86_64-361.45.11 installed, I've installed CUDA 7.5.18_linux with CUDNN linux-x64-v4.0-prod. Every 30 or so iterations the run method hangs and 2 to 5 virtual cores are busy, with the gpu not doing anything more than idling. I thought it hanged indefinitely but when I ran it overnight I saw that it actually completed what it was doing after some time. Pausing and continuing the process using gdb will make it continue immediately. I saw something like this with the initial release of 0.6.0 but it was fixed by 0.6.1 so I just pulled from master and assumed I would never see it again. Inspecting the root process just shows that its waiting on the run method:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(gdb) bt full
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00007f29299964dc in std::condition_variable::wait(std::unique_lock&lt;std::mutex&gt;&amp;) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#2  0x00007f292b46a883 in tensorflow::DirectSession::WaitForNotification(tensorflow::DirectSession::RunState*, long long) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#3  0x00007f292b474800 in tensorflow::DirectSession::Run(tensorflow::RunOptions const&amp;, std::vector&lt;std::pair&lt;std::string, tensorflow::Tensor&gt;, std::allocator&lt;std::pair&lt;std::string, tensorflow::Tensor&gt; &gt; &gt; const&amp;, std::vector&lt;std::string, std::allocator&lt;std::string&gt; &gt; const&amp;, std::vector&lt;std::string, std::allocator&lt;std::string&gt; &gt; const&amp;, std::vector&lt;tensorflow::Tensor, std::allocator&lt;tensorflow::Tensor&gt; &gt;*, tensorflow::RunMetadata*) () from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#4  0x00007f292b549c81 in TF_Run_Helper(TF_Session*, char const*, TF_Buffer const*, char const**, TF_Tensor**, int, char const**, TF_Tensor**, int, char const**, int, TF_Buffer*, TF_Status*) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#5  0x00007f292b54a0f1 in TF_Run () from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#6  0x00007f292a5787b2 in tensorflow::TF_Run_wrapper_helper(TF_Session*, char const*, TF_Buffer const*, tensorflow::gtl::InlinedVector&lt;std::pair&lt;char const*, tagPyArrayObject*&gt;, 8&gt; const&amp;, tensorflow::gtl::InlinedVector&lt;char const*, 8&gt; const&amp;, tensorflow::gtl::InlinedVector&lt;char const*, 8&gt; const&amp;, TF_Status*, tensorflow::gtl::InlinedVector&lt;_object*, 8&gt;*, TF_Buffer*) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#7  0x00007f292a578de1 in tensorflow::TF_Run_wrapper(TF_Session*, TF_Buffer const*, tensorflow::gtl::InlinedVector&lt;std::pair&lt;char const*, tagPyArrayObject*&gt;, 8&gt; const&amp;, tensorflow::gtl::InlinedVector&lt;char const*, 8&gt; const&amp;, tensorflow::gtl::InlinedVector&lt;char const*, 8&gt; const&amp;, TF_Status*, tensorflow::gtl::InlinedVector&lt;_object*, 8&gt;*, TF_Buffer*) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#8  0x00007f292a5666c8 in _wrap_TF_Run () from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
..."><pre class="notranslate"><code class="notranslate">(gdb) bt full
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
No locals.
#1  0x00007f29299964dc in std::condition_variable::wait(std::unique_lock&lt;std::mutex&gt;&amp;) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#2  0x00007f292b46a883 in tensorflow::DirectSession::WaitForNotification(tensorflow::DirectSession::RunState*, long long) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#3  0x00007f292b474800 in tensorflow::DirectSession::Run(tensorflow::RunOptions const&amp;, std::vector&lt;std::pair&lt;std::string, tensorflow::Tensor&gt;, std::allocator&lt;std::pair&lt;std::string, tensorflow::Tensor&gt; &gt; &gt; const&amp;, std::vector&lt;std::string, std::allocator&lt;std::string&gt; &gt; const&amp;, std::vector&lt;std::string, std::allocator&lt;std::string&gt; &gt; const&amp;, std::vector&lt;tensorflow::Tensor, std::allocator&lt;tensorflow::Tensor&gt; &gt;*, tensorflow::RunMetadata*) () from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#4  0x00007f292b549c81 in TF_Run_Helper(TF_Session*, char const*, TF_Buffer const*, char const**, TF_Tensor**, int, char const**, TF_Tensor**, int, char const**, int, TF_Buffer*, TF_Status*) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#5  0x00007f292b54a0f1 in TF_Run () from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#6  0x00007f292a5787b2 in tensorflow::TF_Run_wrapper_helper(TF_Session*, char const*, TF_Buffer const*, tensorflow::gtl::InlinedVector&lt;std::pair&lt;char const*, tagPyArrayObject*&gt;, 8&gt; const&amp;, tensorflow::gtl::InlinedVector&lt;char const*, 8&gt; const&amp;, tensorflow::gtl::InlinedVector&lt;char const*, 8&gt; const&amp;, TF_Status*, tensorflow::gtl::InlinedVector&lt;_object*, 8&gt;*, TF_Buffer*) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#7  0x00007f292a578de1 in tensorflow::TF_Run_wrapper(TF_Session*, TF_Buffer const*, tensorflow::gtl::InlinedVector&lt;std::pair&lt;char const*, tagPyArrayObject*&gt;, 8&gt; const&amp;, tensorflow::gtl::InlinedVector&lt;char const*, 8&gt; const&amp;, tensorflow::gtl::InlinedVector&lt;char const*, 8&gt; const&amp;, TF_Status*, tensorflow::gtl::InlinedVector&lt;_object*, 8&gt;*, TF_Buffer*) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#8  0x00007f292a5666c8 in _wrap_TF_Run () from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
...
</code></pre></div>
<p dir="auto">Inspection of one of the busy threads is also equally uninteresting:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="(gdb) bt full
#0  0x00007f293767e3f7 in sched_yield () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00007f292b76aa60 in Eigen::NonBlockingThreadPoolTempl&lt;tensorflow::thread::EigenEnvironment&gt;::WorkerLoop(unsigned int) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#2  0x00007f292b769f52 in std::_Function_handler&lt;void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function&lt;void ()&gt;)::{lambda()#1}&gt;::_M_invoke(std::_Any_data const&amp;) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#3  0x00007f2929999a60 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4  0x00007f2937980182 in start_thread (arg=0x7f28657fa700) at pthread_create.c:312
        __res = &lt;optimized out&gt;
        pd = 0x7f28657fa700
        now = &lt;optimized out&gt;
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139811478284032, -825179003405916230, 1, 0, 139811478284736, 139811478284032, 782688377649571770, 783080021823268794}, mask_was_saved = 0}}, priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = &lt;optimized out&gt;
        pagesize_m1 = &lt;optimized out&gt;
        sp = &lt;optimized out&gt;
        freesize = &lt;optimized out&gt;
        __PRETTY_FUNCTION__ = &quot;start_thread&quot;
#5  0x00007f29376ad47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals."><pre class="notranslate"><code class="notranslate">(gdb) bt full
#0  0x00007f293767e3f7 in sched_yield () at ../sysdeps/unix/syscall-template.S:81
No locals.
#1  0x00007f292b76aa60 in Eigen::NonBlockingThreadPoolTempl&lt;tensorflow::thread::EigenEnvironment&gt;::WorkerLoop(unsigned int) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#2  0x00007f292b769f52 in std::_Function_handler&lt;void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function&lt;void ()&gt;)::{lambda()#1}&gt;::_M_invoke(std::_Any_data const&amp;) ()
   from /usr/local/lib/python3.4/dist-packages/tensorflow/python/_pywrap_tensorflow.so
No symbol table info available.
#3  0x00007f2929999a60 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4  0x00007f2937980182 in start_thread (arg=0x7f28657fa700) at pthread_create.c:312
        __res = &lt;optimized out&gt;
        pd = 0x7f28657fa700
        now = &lt;optimized out&gt;
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139811478284032, -825179003405916230, 1, 0, 139811478284736, 139811478284032, 782688377649571770, 783080021823268794}, mask_was_saved = 0}}, priv = {
            pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = &lt;optimized out&gt;
        pagesize_m1 = &lt;optimized out&gt;
        sp = &lt;optimized out&gt;
        freesize = &lt;optimized out&gt;
        __PRETTY_FUNCTION__ = "start_thread"
#5  0x00007f29376ad47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
</code></pre></div>
<p dir="auto">Finally the cpu usage revealed by htop doesn't seem to be fully accounted for in the displayed processes, 5 virtual cores might be busy but the processes are only credited with 0-300% cpu usage.</p>
<p dir="auto">There does not appear to be a way to extract a (sparse or dense) slice from a sparse tensor in the same manner as there is from a dense tensor. Will this be a feature in the future, or is it already present and I'm just missing something? Perhaps it should be added to the planned improvements for indexing.</p>