title: Best practices to benchmark deep models on CPU (and not GPU) in PyTorch?
body: <p>I am little uncertain about how to measure execution time of deep models on CPU in PyTorch ONLY FOR INFERENCE. I list here some of them but they maybe inaccurate. Please correct them if required and mention more if required. I am running on PyTorch version 1.3.1 and Intel Xeon with 64GB RAM, 3.5GHz processor and 8 cores.</p>
  
  <ol>
  <li><p>Should we use <code>time.time()</code>?</p>
  
  <ul>
  <li>I know that for GPU this is a very bad idea. For GPU I do as follows</li>
  </ul></li>
  </ol>
  
  <pre><code>with torch.no_grad():
      wTime = 0
      start = torch.cuda.Event(enable_timing=True)
      end = torch.cuda.Event(enable_timing=True)
      out = model(input) # JUST FOR WARMUP
  
      start.record()
      for i in range(200):
          input = torch.rand(1,1,2048,2048).to(device)
  
  #        beg = time.time() DO NOT USE FOR GPU
  
          got = net_amplifier(low,for_amplifier)
  
  #        wTime+=time.time()-beg DO NOT USE FOR GPU
      end.record()
      torch.cuda.synchronize()
  
      print('execution time in MILLISECONDS: {}'.format(start.elapsed_time(end)/200))
  
  </code></pre>
  
  <p>For this code execution was done on GPU. If I have to run them on CPU what changes should be made? Will <code>time.time()</code> do?</p>
  
  <ol start="2">
  <li>Should we use <code>volatile</code>?
  
  <ul>
  <li>I think the use if volatile is now discouraged after v0.3. But will it still help if I use the eval mode and no_grad()</li>
  </ul></li>
  </ol>
  
  <pre><code>input = Variable(torch.randn(1, 3, 227, 227), volatile=True) 
   model(input) 
  </code></pre>
  
  <ol start="3">
  <li><p>Should the page cache be cleared?</p>
  
  <ul>
  <li>One way of doing this that I know is using <code>sudo sh -c "/bin/echo 1 &gt; /proc/sys/vm/drop_caches"</code></li>
  </ul></li>
  <li><p>Should I remove <code>nn.Sequential()</code> and directly put in forward part</p>
  
  <ul>
  <li>According to this <a href="https://discuss.pytorch.org/t/execution-time-on-cpu/13782/5" rel="nofollow noreferrer">link</a></li>
  </ul></li>
  </ol>
  
  <blockquote>
    <p>All the methods using copy_ take some time to execute, especially on CPU this might be slow. Also the nn.Sequential() modules are slower than just executing them on the forward pass. I think this is due to some overhead that needs to be created when executing the Sequential module.</p>
  </blockquote>
  
  <p>Another thing which i do not understand on the same link is </p>
  
  <blockquote>
    <p>If you are running into performance issues with these small numbers, you might try to use torch.set_flush_denormal(True) to disable denormal floating point numbers on the CPU.</p>
  </blockquote>
  
  <ol start="5">
  <li><p>Should <code>torch.set_num_threads(int)</code> be used? If yes can a demo code be provided?</p></li>
  <li><p>What does <code>These context managers are thread local, so they won’t work if you send work to another thread using the :module:`threading` module, etc.</code> mean as given in the <a href="https://pytorch.org/docs/1.3.1/torch.html#locally-disabling-gradient-computation" rel="nofollow noreferrer">documentation</a>.</p></li>
  </ol>
  
  <p><strong>Please list any more issues for calculating execution time in CPU.</strong>
  Thankyou</p>