Issue title: Test failure in test_object_transfer_dump in runtest.py.
 Issue body: Running

```
python -m pytest -s -v test/runtest.py38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5test_object_transfer_dump
```

fails with the following. This is also happening on Travis (on **Linux with Python 2.7**).

```
root@actorft-001:~/ray$ python -m pytest -s -v test/runtest.py38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5test_object_transfer_dump
================================================== test session starts ===================================================
platform linux2 -- Python 2.7.15, pytest-3.10.0, py-1.7.0, pluggy-0.8.0 -- /root/miniconda2/bin/python
cachedir: .pytest_cache
rootdir: /root/ray, inifile:
collected 1 item                                                                                                         

test/runtest.py38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5test_object_transfer_dump /root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:1000: Allowing the Plasma store to use up to 1GB of memory.
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:1030: Starting object store with directory /dev/shm and huge page support disabled
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:1000: Allowing the Plasma store to use up to 1GB of memory.
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:1030: Starting object store with directory /dev/shm and huge page support disabled
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:1000: Allowing the Plasma store to use up to 1GB of memory.
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:1030: Starting object store with directory /dev/shm and huge page support disabled
I1107 22:51:08.522545 15209 scheduling_policy.cc:107] The task with ID 00000000f78996f7ee9f452c8ba4d12fd2105e91 requires {2,1.000000},{CPU,1.000000} for execution and {2,1.000000},{CPU,1.000000} for placement, but no nodes have the necessary resources. Check the client table to view node resources.
I1107 22:51:08.523793 15209 scheduling_policy.cc:107] The task with ID 000000006e227615d7013c389799b1afede5948e requires {2,1.000000},{CPU,1.000000} for execution and {2,1.000000},{CPU,1.000000} for placement, but no nodes have the necessary resources. Check the client table to view node resources.
F1107 22:51:08.593224 15209 task_dependency_manager.cc:176]  Check failed: it != task_dependencies_.end() 
*** Check failure stack trace: ***
    @           0x57ea8c  googl38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5LogMessag38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5Fail()
    @           0x57e9d0  googl38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5LogMessag38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5SendToLog()
    @           0x57e312  googl38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5LogMessag38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5Flush()
    @           0x57e10d  googl38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5LogMessag38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5~LogMessage()
    @           0x4d3350  ray38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5RayLog38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5~RayLog()
    @           0x54e57c  ray38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5raylet38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5TaskDependencyManager38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5UnsubscribeDependencies()
    @           0x5105fa  ray38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5raylet38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5NodeManager38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5HeartbeatAdded()
    @           0x4c8fb2  _ZNSt17_Function_handlerIFvPN3ray3gcs14AsyncGcsClientERKNS0_8UniqueIDERKSt6vectorI19HeartbeatTableDataTSaIS8_EEEZNS1_5TableIS4_18HeartbeatTableDataE9SubscribeES6_S6_RKSt8functionIFvS3_S6_RKS8_EERKSH_IFvS3_S6_EERKSH_IFvS3_EEEUlS3_S6_SC_E_E9_M_invokeERKSt9_Any_dataOS3_S6_SC_
    @           0x4b9533  _ZZN3ray3gcs3LogINS_8UniqueIDE18HeartbeatTableDataE9SubscribeERKS2_S6_RKSt8functionIFvPNS0_14AsyncGcsClientES6_RKSt6vectorI19HeartbeatTableDataTSaISB_EEEERKS7_IFvS9_EEENKUlRKSsE_clESP_
    @           0x4ce709  (anonymous namespace)38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5ProcessCallback()
    @           0x4cf689  ray38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5gcs38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5SubscribeRedisCallback()
    @           0x52775d  redisProcessCallbacks
    @           0x4d2c9d  RedisAsioClient38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5handle_read()
    @           0x4d3255  boost38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5sio38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5tail38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5reactive_null_buffers_op<>38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5o_complete()
    @           0x4893b8  boost38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5sio38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5tail38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5poll_reactor38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5scriptor_stat38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5do_complete()
    @           0x489d18  boost38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5sio38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5tail38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5task_io_servi38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5run()
    @           0x47fab0  main
    @     0x7f25b452c830  __libc_start_main
    @           0x483da9  _start
    @              (nil)  (unknown)
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 10
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.128841 15234 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.416245 15253 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.423425 15232 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.608633 15238 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.610317 15229 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.802784 15255 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.812130 15260 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.828364 15259 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.829855 15231 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.906795 15257 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.911777 15228 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:09.928257 15233 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.016029 15242 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.016852 15258 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.018923 15239 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.114563 15252 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.+1-325-532-3351 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.214190 15241 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.236562 15236 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.309136 15248 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.328008 15240 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.334754 15230 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.399689 15235 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.516479 15243 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.519310 15249 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.524260 15227 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.591334 15256 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.608604 15250 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.618911 15237 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.704175 15246 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:10.911255 15254 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1107 22:51:11.004866 15244 local_scheduler_client.cc:71] Connection to socket failed for pathname /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
F1107 22:51:14.134971 15234 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 10
F1107 22:51:14.421813 15253 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:14.430136 15232 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 12
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 13
F1107 22:51:14.614594 15238 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:14.615934 15229 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 15
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 14
F1107 22:51:14.809145 15255 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:14.817134 15260 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:14.834663 15259 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:14.905952 15231 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:14.912945 15257 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:14.917119 15228 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 16
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 17
F1107 22:51:14.934618 15233 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 18
F1107 22:51:15.021909 15242 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.022635 15258 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.024111 15239 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 19
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 20
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 21
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 22
F1107 22:51:15.+1-325-532-3351 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 23
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 25
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 24
F1107 22:51:15.198415 15245 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.219650 15241 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 26
F1107 22:51:15.300004 15236 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 27
F1107 22:51:15.315953 15248 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 28
F1107 22:51:15.334071 15240 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.340764 15230 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.404876 15235 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 29
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 30
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 32
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 31
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 33
F1107 22:51:15.523303 15243 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.532348 15249 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.541057 15227 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.597569 15256 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.615936 15250 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:15.624785 15237 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 34
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 35
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 36
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 37
F1107 22:51:15.717670 15246 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 38
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 39
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 40
F1107 22:51:15.917282 15254 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
F1107 22:51:16.010191 15244 local_scheduler_client.cc:78] Could not connect to socket /tmp/ray/session_2018-11-07_22-51-07_15170/sockets/raylet
*** Check failure stack trace: ***
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 41
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 9
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 7
/root/ray/build/external/arrow/src/arrow_ep/cpp/src/plasma/store.cc:609: Disconnecting client on fd 42
I1107 22:51:18.524951 15495 node_manager.cc:1422] Reconstructing task 000000004896da55a21ce7e4d980610f598c1168 on client 69055c8c42a5f20ae288126f40e6962ed2dfc078
W1107 22:51:18.599422 15189 monitor.cc:46] Client timed out: d2f85914cab231f84702f2a9623f98cf67326eb4

I1107 22:51:28.525182 15495 node_manager.cc:1422] Reconstructing task 000000004896da55a21ce7e4d980610f598c1168 on client 69055c8c42a5f20ae288126f40e6962ed2dfc078
I1107 22:51:28.525348 15495 scheduling_policy.cc:107] The task with ID 000000004896da55a21ce7e4d980610f598c1168 requires {0,1.000000},{CPU,1.000000} for execution and {0,1.000000},{CPU,1.000000} for placement, but no nodes have the necessary resources. Check the client table to view node resources.
```
 Comments: 
 Comment 0: I can reproduce this.
 Comment 1: This looks like it's probably because the tasks removed from the local queue during spillover could come from either the `INFEASIBLE` or the `READY` queue. Dependencies should only be subscribed for tasks in the `READY` queue, so this is most likely failing on a task that came from the `INFEASIBLE` queue.

The check in `TaskDependencyManager38d1:d2b4:3d06:83e8:96a5:8c2:893d:aba5UnsubscribeDependencies` is a little annoying I suppose, but it seems useful to keep. I suggest keeping the check and returning the `TaskState` from `local_queues_.RemoveTask`, so that we can unsubscribe only if it was in `READY`.
 Comment 2: I'm adding this as a release-blocker, since I think this can happen a lot in practice.
 Comment 3: When this issue is fixed, we should make sure to remove the unnecessary call to `cluster.wait_for_nodes` in https://github.com/ray-project/ray/pull/3276.
{'source': 'GithubIssues', 'repo_name_with_owner': 'ray-project/ray', 'issue_id': 378504517, 'repo_stars': 22616.0}