Why a larger neural network back-propagate faster than a smaller one
I wrote the following two NN in pytorch for image segmentation:
The smaller one:

class ConvNetV0(nn.Module):

def __init__(self):
    super(ConvNetV0, self).__init__()
    self.conv1 = nn.Conv2d(3, 30, 4, padding=2)
    self.conv2 = nn.Conv2d(30, 50, 16, padding=7, bias=True)
    self.conv3 = nn.Conv2d(50, 20, 2, stride=2)
    self.conv4 = nn.Conv2d(20, 2, 2, stride=2)

def forward(self, x):
    x = self.conv1(x)
    x = F.relu(x)
    x = self.conv2(x)
    x = F.relu(x)
    x = self.conv3(x)
    x = F.relu(x)
    y = self.conv4(x)
    return y


The larger one:

class ConvNetV1(nn.Module):

def __init__(self):
    super(ConvNetV1, self).__init__()
    self.conv0 = nn.Conv2d(3, 50, 4, padding=1, stride=2)
    self.conv_r1 = nn.Conv2d(50, 40, 15, padding=7, bias=True)
    self.conv_r2 = nn.Conv2d(40, 25, 3, padding=1)
    self.conv_r3 = nn.Conv2d(25, 25, 2, stride=2)
    # self.conv_r3 = nn.MaxPool2d(2, stride=2)
    self.conv_b1 = nn.Conv2d(50, 15, 4, padding=1, stride=2)
    self.conv1 = nn.Conv2d(40, 2, 1)

def forward(self, x):
    x = self.conv0(x)
    x = F.relu(x)

    x1 = self.conv_r1(x)
    x1 = F.relu(x1)
    x1 = self.conv_r2(x1)
    x1 = F.relu(x1)
    x1 = self.conv_r3(x1)

    x2 = self.conv_b1(x)

    y = torch.cat([x1, x2], dim=1)
    y = self.conv1(y)
    return y


However during training at mini-batch size = 8. The smaller net take 2s to complete one iteration while the larger one only take 0.3s to complete. 

I also observe that the no of parameters ratio between the two net is around 5:6. However, during training the smaller net only takes 1GB VRAM while the larger takes 3GB. Since My 1050ti have 4GB VRAM. I would like trade off memory for speed. Any idea how I can do so?

I ran a quick benchmark of your models on synthetic data of the size you indicated. At least on my system, the difference isn't actually given by the model forward or backward, but by the computation of the loss. This is likely to be due to the fact that the first model uses more the GPU, therefore the queuing of the operations is slightly longer.

In fact your first model performs ~630 millions operations, while the second one is around ~270 millions. Notice that in the second model you immediately reduce the size of the feature maps from 256x256 to 128x128, while in the first model you reduce the size only in the last two convolutions. This has a big effect in the number of operations performed. 

So if you wish to use a V0-like model and make it faster, you should try to decrease the size of the feature maps right away. With this smaller model (in terms of memory), you'll also be able to increase the batch size.

If you wish to use the V1 instead, there isn't much you can do. You could try to use checkpoints introduced in Pytorch 0.4 to trade compute for memory to the point where you can increase the batch to 16. It may run a bit faster, or not, depending on how much compute you need to trade off.

Another simple thing you can do to make it run faster, if your input size don't change, is set torch.cudnn.benchmark = True. This will look for the fastest set of algorithms for a particular configuration.
