title: mxnet - long GPU initialization time
body: <p>A follow-up of this question: <a href="https://stackoverflow.com/questions/57260730/mxnet-slow-array-copy-to-gpu/57261108">Mxnet - slow array copy to GPU</a></p>
  
  <p>Question: mxnet GPU initialization takes around 20 seconds. How can I fix it?</p>
  
  <p>I have the following code:</p>
  
  <pre><code>import mxnet as mx
  import mxnet.ndarray as nd
  
  from mxnet import profiler
  
  profiler.set_config(aggregate_stats=True)
  
  ctx = mx.gpu()
  
  profiler.set_state('run')
  nd.random.uniform(-1, 1, shape=(1, 1), ctx=ctx)
  nd.waitall()
  profiler.set_state('stop')
  print(profiler.dumps(reset=True))
  </code></pre>
  
  <p>And this is the profiler output:</p>
  
  <pre><code>Device Storage
  =================
  Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
  ----                          -----------        ---------    -------------    -------------    -------------
  Memory: gpu/0                           3           0.0080           0.0040           0.0120           0.0040
  
  MXNET_C_API
  =================
  Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
  ----                          -----------        ---------    -------------    -------------    -------------
  MXNDArrayWaitAll                        1           2.0640           2.0640           2.0640           2.0640
  MXNDArrayFree                           1           0.0010           0.0010           0.0010           0.0010
  MXImperativeInvokeEx                    1       22197.0469       22197.0469       22197.0469       22197.0469
  MXNet C API Concurrency                 6           0.0000           0.0000           0.0010           0.0005
  MXNet C API Calls                       3           0.0030           0.0010           0.0030           0.0010
  
  operator
  =================
  Name                          Total Count        Time (ms)    Min Time (ms)    Max Time (ms)    Avg Time (ms)
  ----                          -----------        ---------    -------------    -------------    -------------
  _random_uniform                         2           1.6280           0.8130           0.8150           0.8140
  DeleteVariable                          2           0.0130           0.0060           0.0070           0.0065
  ResourceParallelRandomSetSeed              10          17.9840           0.4670           5.7260           1.7984
  </code></pre>
  
  <p>So, it takes 22 seconds. Any operation after this will be very fast, but the first GPU operation takes 22 seconds (no matter what operation I use). So it's likely that initialization requires a long time. How can I fix it?</p>
  
  <p>I tried this approach <a href="https://github.com/apache/incubator-mxnet/issues/3239" rel="nofollow noreferrer">https://github.com/apache/incubator-mxnet/issues/3239</a>:</p>
  
  <pre><code>export CUDA_CACHE_MAXSIZE=2147483647
  export CUDA_CACHE_DISABLE=0
  export CUDA_CACHE_PATH="my_home_path/.nv/ComputeCache"
  </code></pre>
  
  <p>but it doesn't work.</p>
  
  <p>This page: <a href="https://github.com/apache/incubator-mxnet/issues/13040" rel="nofollow noreferrer">https://github.com/apache/incubator-mxnet/issues/13040</a> also mentions <code>SM_70 arch.wait update</code>, but I wasn't able to understand what it means.</p>
  
  <p>As I understand, the problem is that some libraries have to be loaded to GPU, and it just has to be done. The idea is to cache these libraries (so that there is no load second time), but I don't know how to do this.</p>