title: Prunning model doesn't improve inference speed or reduce model size
body: <p>I'm trying to prune my model in PyTorch with <code>torch.nn.utils.prune</code>, which provides 2 tensors, </p>
  
  <ol>
  <li>one is the original weight and </li>
  <li>the other is a mask contain 0s and 1s that help us close certain connections in the network. </li>
  </ol>
  
  <p>I have tried both of the solutions, but none improve the inference speed:  </p>
  
  <ol>
  <li>Use the network after pruning to infer which will first close some connections with the mask and then run inference.  </li>
  <li>Zeros out the original weights with the mask and then remove the mask from the state_dict to infer.</li>
  </ol>
  
  <p><strong><em>Is there a way to improve the speed with the model tensor and the mask? Doesn't multiply with a non-zero float number with 0 will faster than multiply 2 floats with each other?</em></strong><br>
  Here is my prune function and the pruning speed calculating procedure:</p>
  
  <pre><code>def prune_net(net):
      """Prune 20% net's weights that have abs(value) approx. 0
      Function that will be use when an iteration is reach
      Args:
  
      Return:
          newnet (nn.Module): a newnet contain mask that help prune network's weight
      """
      if not isinstance(net,nn.Module):
          print('Invalid input. Must be nn.Module')
          return
      newnet = copy.copy(net)
      modules_list = []
  
      for name, module in newnet.named_modules():
          if isinstance(module, torch.nn.Conv2d):
              modules_list += [(module,'weight'),(module,'bias')]
          if isinstance(module, torch.nn.Linear):
              modules_list += [(module,'weight'),(module,'bias')]
  
      prune.global_unstructured(
          modules_list,
          pruning_method=prune.L1Unstructured,
          amount=0.2,)
      return newnet
  </code></pre>
  
  <p>Test inference speed 1st case:</p>
  
  <pre><code>import torch
  from torch import nn
  import torch.nn.utils.prune as prune
  import torch.nn.functional as F
  import time
  from torch.autograd import Variable
  
  
  torch.set_default_tensor_type('torch.cuda.FloatTensor')
  old_net = init_your_net()
  
  new_net = prune_net(old_net)
  new_net = prune_net(new_net)
  
  old_net.eval()
  new_net.eval()
  
  old_net = old_net.cuda()
  new_net = new_net.cuda()
  dataset = load_your_dataset()
  
  for i in range(100):
      x = dataset[i]
      x = x.cuda()
      y = x.cuda()
  
      #new infer
      start_time = time.perf_counter()
      detections = new_net(x).data
      time_new += time.perf_counter() - start_time
  
      #old infer
      start_time = time.perf_counter()
      detections = old_net(y).data
      time_old += time.perf_counter() - start_time
  print('old ',time_old)
  print('new ', time_new)
  </code></pre>
  
  <p>Test inference speed 2nd case:</p>
  
  <pre><code>import torch
  from torch import nn
  import torch.nn.utils.prune as prune
  import torch.nn.functional as F
  import time
  from torch.autograd import Variable
  
  
  torch.set_default_tensor_type('torch.cuda.FloatTensor')
  old_net = init_your_net()
  
  new_net = prune_net(old_net)
  new_net = prune_net(new_net)
  # Apply mask to model tensor and remove mask from state_dict
  for name, module in new_net.named_modules():
      if isinstance(module, torch.nn.Conv2d):
          prune.remove(module,'weight')
          prune.remove(module,'bias')
      if isinstance(module, torch.nn.Linear):
          prune.remove(module,'weight')
          prune.remove(module,'bias')
  
  old_net.eval()
  new_net.eval()
  
  old_net = old_net.cuda()
  new_net = new_net.cuda()
  dataset = load_your_dataset()
  
  for i in range(100):
      x = dataset[i]
      x = x.cuda()
      y = x.cuda()
  
      #new infer
      start_time = time.perf_counter()
      detections = new_net(x).data
      time_new += time.perf_counter() - start_time
  
      #old infer
      start_time = time.perf_counter()
      detections = old_net(y).data
      time_old += time.perf_counter() - start_time
  print('old ',time_old)
  print('new ', time_new)
  </code></pre>
  
  <p><strong>UPDATE</strong><br>
  I found torch have a sparse module that can reduce memory usage if we prune enough parameter but it hasn't support nn.Module yet, only Tensor object. Here are some useful link:<br>
  <a href="https://github.com/pytorch/pytorch/issues/36214#issuecomment-619586452" rel="nofollow noreferrer">https://github.com/pytorch/pytorch/issues/36214#issuecomment-619586452</a><br>
  <a href="https://pytorch.org/docs/stable/sparse.html" rel="nofollow noreferrer">https://pytorch.org/docs/stable/sparse.html</a></p>