Theano and Celery : Worker exited prematurely: signal 11 (SIGSEGV)
Question: I'm building a web app where I start the training of a neural network
implemented with theano on the server side by sending an ajax request from the
client. Obviously I don't want to wait for the server to train the network
entirely to send an answer back to my client because it would be way to long.

So I came up with celery which enable me to do asynchronous code on the server
side. I run the celery workers with the command `celery -A CBIR worker -l
info`. Unfortunately I get the following message each time a worker is running
my task (which is training my network using theano) :

    
    
    [2015-12-14 19:15:06,790: ERROR/MainProcess] Process 'Worker-3' pid:1610 exited with 'signal 11 (SIGSEGV)'
    [2015-12-14 19:15:07,001: ERROR/MainProcess] Task fit[ac40d4d4-5b56-4278-b270-647ef76f3a49] raised unexpected: WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV).',)
    Traceback (most recent call last):
    File "/Users/leo/anaconda/envs/ImgRet/lib/python3.5/site-packages/billiard/pool.py", line 1175, in mark_as_worker_losthuman_status(exitcode)),
    billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 11 (SIGSEGV).
    

I have been searching why this error might occur and from what I understand
the code I'm running is suffering memory leak. What I don't understand is why
my code has no problem running when not using celery but this error come up
when using Celery ?

And most importantly I have no clue how to resolve this. I used lldb to look
at the dump file generated and here is the backtrace I have :

    
    
    thread #1: tid = 0x0000, 0x00007fff93b4a9b3 libdispatch.dylib`dispatch_group_async + 533, stop reason = signal SIGSTOP
    * frame #0: 0x00007fff93b4a9b3 libdispatch.dylib`dispatch_group_async + 533
    frame #1: 0x00007fff7c5b8d40 libdispatch.dylib`_dispatch_root_queues + 1280
    frame #2: 0x00007fff9519b228 libBLAS.dylib`APL_dgemm + 1100
    frame #3: 0x00007fff951d27aa libBLAS.dylib`cblas_dgemm + 1420
    frame #4: 0x0000000104beeb18 multiarray.cpython-35m-darwin.so`gemm + 200
    frame #5: 0x0000000104bee3b9 multiarray.cpython-35m-darwin.so`cblas_matrixproduct + 3097
    frame #6: 0x0000000104bc01af multiarray.cpython-35m-darwin.so`PyArray_MatrixProduct2 + 207
    frame #7: 0x0000000104bc4808 multiarray.cpython-35m-darwin.so`array_matrixproduct + 264
    frame #8: 0x00000001000671a9 libpython3.5m.dylib`PyCFunction_Call + 281
    frame #9: 0x00000001000f2fbd libpython3.5m.dylib`PyEval_EvalFrameEx + 32029
    frame #10: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #11: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #12: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #13: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #14: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #15: 0x00000001000e95a7 libpython3.5m.dylib`PyEval_CallObjectWithKeywords + 87
    frame #16: 0x00000001042fae3a lazylinker_ext.so`pycall(self=0x0000000108fad3d8, node_idx=13, verbose=0) + 442 at mod.cpp:510
    frame #17: 0x00000001042fa869 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=24, one=0x000000010026cf60, zero=0x000000010026cf40) + 2089 at mod.cpp:704
    frame #18: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=28, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690
    frame #19: 0x00000001042fa16d lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=30, one=0x000000010026cf60, zero=0x000000010026cf40) + 301 at mod.cpp:576
    frame #20: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=33, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690
    frame #21: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=36, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690
    frame #22: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=41, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690
    frame #23: 0x00000001042fa789 lazylinker_ext.so`lazy_rec_eval(self=0x0000000108fad3d8, var_idx=42, one=0x000000010026cf60, zero=0x000000010026cf40) + 1865 at mod.cpp:690
    frame #24: 0x00000001042f83db lazylinker_ext.so`CLazyLinker_call(_self=0x0000000108fad3d8, args=0x0000000100382048, kwds=0x0000000000000000) + 811 at mod.cpp:838
    frame #25: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #26: 0x00000001000ed08c libpython3.5m.dylib`PyEval_EvalFrameEx + 7660
    frame #27: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #28: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #29: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #30: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #31: 0x000000010002a79c libpython3.5m.dylib`method_call + 140
    frame #32: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #33: 0x0000000100080743 libpython3.5m.dylib`slot_tp_call + 67
    frame #34: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #35: 0x00000001000ed08c libpython3.5m.dylib`PyEval_EvalFrameEx + 7660
    frame #36: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #37: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462
    frame #38: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #39: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #40: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #41: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #42: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563
    frame #43: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #44: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #45: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #46: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #47: 0x000000010002a79c libpython3.5m.dylib`method_call + 140
    frame #48: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #49: 0x0000000100080743 libpython3.5m.dylib`slot_tp_call + 67
    frame #50: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #51: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563
    frame #52: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #53: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462
    frame #54: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #55: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #56: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #57: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #58: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563
    frame #59: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #60: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462
    frame #61: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #62: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462
    frame #63: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #64: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #65: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #66: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #67: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #68: 0x000000010002a79c libpython3.5m.dylib`method_call + 140
    frame #69: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #70: 0x0000000100080471 libpython3.5m.dylib`slot_tp_init + 81
    frame #71: 0x000000010007b114 libpython3.5m.dylib`type_call + 212
    frame #72: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #73: 0x00000001000ed08c libpython3.5m.dylib`PyEval_EvalFrameEx + 7660
    frame #74: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #75: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #76: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #77: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #78: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #79: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #80: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563
    frame #81: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #82: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #83: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #84: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #85: 0x000000010002a79c libpython3.5m.dylib`method_call + 140
    frame #86: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #87: 0x0000000100080471 libpython3.5m.dylib`slot_tp_init + 81
    frame #88: 0x000000010007b114 libpython3.5m.dylib`type_call + 212
    frame #89: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #90: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563
    frame #91: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #92: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #93: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #94: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #95: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #96: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #97: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #98: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #99: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #100: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563
    frame #101: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #102: 0x00000001000f4ef7 libpython3.5m.dylib`PyEval_EvalCodeEx + 71
    frame #103: 0x0000000100041d2a libpython3.5m.dylib`function_call + 186
    frame #104: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #105: 0x000000010002a79c libpython3.5m.dylib`method_call + 140
    frame #106: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #107: 0x0000000100080743 libpython3.5m.dylib`slot_tp_call + 67
    frame #108: 0x000000010000d783 libpython3.5m.dylib`PyObject_Call + 99
    frame #109: 0x00000001000eff0b libpython3.5m.dylib`PyEval_EvalFrameEx + 19563
    frame #110: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #111: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462
    frame #112: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #113: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462
    frame #114: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #115: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #116: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462
    frame #117: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #118: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462
    frame #119: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #120: 0x00000001000f3d26 libpython3.5m.dylib`PyEval_EvalFrameEx + 35462
    frame #121: 0x00000001000f4053 libpython3.5m.dylib`PyEval_EvalFrameEx + 36275
    frame #122: 0x00000001000f4df0 libpython3.5m.dylib`_PyEval_EvalCodeWithName + 2400
    frame #123: 0x00000001000f4f51 libpython3.5m.dylib`PyEval_EvalCode + 81
    frame #124: 0x0000000100123d4e libpython3.5m.dylib`PyRun_FileExFlags + 206
    frame #125: 0x0000000100123fef libpython3.5m.dylib`PyRun_SimpleFileExFlags + 447
    frame #126: 0x000000010013c7d7 libpython3.5m.dylib`Py_Main + 3479
    frame #127: 0x0000000100000e92 python3`main + 418
    frame #128: 0x0000000100000cc4 python3`start + 52
    

I don't really know how to interpret this backtrace. Thanks in advance for any
help !


Answer: If anyone run into the same problem turns out a workaround is to import theano
library inline in the task and not at the module level.

This way :

    
    
    import baz
    import bar
    
    @app.task
    def foo():
        import theano
    
        # do something with theano
    

Check
[here](https://github.com/celery/celery/issues/2964#issuecomment-165669659)
for more explanation

