Theano crashing using cuDNN in linux
Question: I am a non-root user on a cluster computer running Scientific Linux release
6.6 (Carbon).

I am experiencing some theano crashes when running code on a GPU with CUDA 7.5
and cuDNN 5. I am using Python 2.7, Theano 0.9, Keras 1.0.7 and Lasange 0.1.

The following crash occurs ONLY when I run the program on a GPU node with
cuDNN enabled. The code completes without issue on a CPU and a GPU with cuDNN
disabled.

    
    
    Traceback (most recent call last):
      File "runner.py", line 306, in <module>
        main()
      File "runner.py", line 241, in main
        queries_exp = __import__(args.exp_model).queries_exp
      File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity-convnet/exp_multi_conv_cosim.py", line 923, in <module>
        queries_exp = EntityVectorLinkExp()
      File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity-convnet/exp_multi_conv_cosim.py", line 51, in __init__
        self._setup()
      File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity-convnet/exp_multi_conv_cosim.py", line 543, in _setup
        on_unused_input='ignore',
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/function.py", line 326, in function
        output_keys=output_keys)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/pfunc.py", line 484, in pfunc
        output_keys=output_keys)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1788, in orig_function
        output_keys=output_keys).create(
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1467, in __init__
        optimizer_profile = optimizer(fgraph)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 102, in __call__
        return self.optimize(fgraph)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 90, in optimize
        ret = self.apply(fgraph, *args, **kwargs)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 235, in apply
        sub_prof = optimizer.optimize(fgraph)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 90, in optimize
        ret = self.apply(fgraph, *args, **kwargs)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 235, in apply
        sub_prof = optimizer.optimize(fgraph)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 90, in optimize
        ret = self.apply(fgraph, *args, **kwargs)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 2262, in apply
        lopt_change = self.process_node(fgraph, node, lopt)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 1825, in process_node
        lopt, node)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 1719, in warn_inplace
        return NavigatorOptimizer.warn(exc, nav, repl_pairs, local_opt, node)
      File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", line 1705, in warn
        raise exc
    AssertionError
    

My .theanorc looks like this:

    
    
    [global]
    floatX = float32
    device = gpu
    
    [lib]
    cnmem = 1
    
    [nvcc]
    fastmath = True
    

And my profile has the following:

    
    
    export LD_LIBRARY_PATH=/home/t/tj/tjb32/cuda/lib64:$LD_LIBRARY_PATH 
    export CPATH=/home/t/tj/tjb32/cuda/include:$CPATH
    export LIBRARY_PATH=/home/t/tj/tjb32/cuda/lib64:$LD_LIBRARY_PATH
    export PATH=/home/t/tj/tjb32/cuda/bin:$PATH
    

When I query theano, the following is returned, which suggests to me that
theano is interacting with CUDA and cuDNN.

    
    
    Using gpu device 0: Tesla K20m (CNMeM is enabled with initial size: 95.0% of memory, cuDNN 5005)
    

I'm fairly sure that I have installed CUDA and cuDNN correctly, if anyone
could suggest any additional configuration steps that I may have missed that
is causing cuDNN to crash the program, that would be greatly appreciated.


Answer: Not sure if this can be the issues but: export
LIBRARY_PATH=/home/t/tj/tjb32/cuda/lib64:$**LD_** LIBRARY_PATH should be?
export LIBRARY_PATH=/home/t/tj/tjb32/cuda/lib64:$LIBRARY_PATH

