Instruction: Combined GRU and CNN network always returns the same value for all inputs

Input:
I am trying to train a combined CNN and GRU/LSTM to find out the number of objetcs in a series of pictures that move and the number of objects that do not move. For this reason I am using a CNN to process my images and consequently use a GRU.
My problem is that the GRU always returns the same value for each input set. What could be reasons for that?

I have already tried to use different learning rates and adding linear layers after the GRU.

My network:

    class GRU(nn.Module):
        def __init__(self, **kwargs):
            super(GRU, self).__init__()
            self.n_class = int(kwargs.get("n_class"))
            self.seq_length = int(kwargs.get("seq_length"))
            self.input_shape = int(kwargs.get("input_shape"))
            self.n_channels = int(kwargs.get("n_channels"))
            self.conv1 = nn.Conv2d(in_channels=1 * seq_length, out_channels=4 * seq_length, kernel_size=5)
            self.conv2 = nn.Conv2d(in_channels=4 * seq_length, out_channels=8 * seq_length, kernel_size=5)
            self.conv3 = nn.Conv2d(in_channels=8 * seq_length, out_channels=16 * seq_length, kernel_size=5)
            self.rnn = nn.GRU(
                input_size=self.seq_length,
                hidden_size=64,
                num_layers=1,
                batch_first=True)
            self.linear = nn.Linear(64, 2)

        def forward(self, t):
            t = self.conv1(t)
            t = F.relu(t)
            t = F.max_pool2d(t, kernel_size=2, stride=2)
            # second conv layer
            t = self.conv2(t)
            t = F.relu(t)
            t = F.max_pool2d(t, kernel_size=4, stride=4)
            # third conv layer
            t = self.conv3(t)
            t = F.relu(t)
            t = F.max_pool2d(t, kernel_size=3, stride=3)
            t = t.reshape(-1 , self.seq_length, 16 * 20 ** 2)
            t = t.permute(0,2,1)
            t, (h_n) =self.rnn(t)
            t = self.linear(t[:,-1])
            return t

and this is my training:

for epoch in range(number_epochs):
    for batch in get_batch_generator(batch_size, rootdir, seq_length=seq_length):
        current_batch = batch[0].cuda()
        current_labels = batch[1].cuda()
        pre = nw(current_batch)
        loss_func = torch.nn.MSELoss()
        loss = loss_func(pre, current_labels)
        loss.backward()
        optimizer = optim.Adam(nw.parameters(), lr=learning_rate)
        optimizer.step()

Here is an example of the output, actual labels:

tensor([[ 4.,  5.],
        [10.,  0.],
        [10.,  0.],
        [ 2.,  9.],
        [ 5.,  1.],
        [10.,  0.]], device='cuda:0')

Prediction of my network:

tensor([[2.0280, 1.1517],
        [2.0175, 1.1593],
        [2.0323, 1.1434],
        [2.0333, 1.1557],
        [2.0200, 1.1546],
        [2.0069, 1.1687]], device='cuda:0', grad_fn=<AddmmBackward>)

So for both classes the output is the same for both classes (moving and not moving objects), which should not be the case.

Output:
Finally I found out that it is necessary to set the gradients to zero for every batch. For some reason this did not cause problems, when I was training normal CNNs without LSTM.
The command that needs to be added in every training loop before the back-propagation:

optimizer.zero_grad()

or

nw.zero_grad()


Instruction: load pickle file obtained from GPU to CPU

Input:
I meet a problem when I load a pickle file to CPU. I search it on the internet, and they say I need to add map_location parameter. However, after I add this parameter, the problem still exists.

the code is as follows:

torch.__version__
torch.load('featurs.pkl',map_location='cpu')

>>>

'1.0.1.post2'
Attempting to deserialize object on a CUDA device
but torch.cuda.is_available() is False. If you are running
on a CPU-only machine, please use torch.load with map_location='cpu'
to map your storages to the CPU.

I know it is because of the different devices, but I use the instruction in the error message, so I do not know how to solve it in the next step.

Thanks in advance!

Output:
The error message suggests using map_location=torch.device('cpu') but even that doesn't work. One workaround is to use the pickle library and implement a custom unpickler.
import pickle
import io

class CPU_Unpickler(pickle.Unpickler):
    def find_class(self, module, name):
        if module == 'torch.storage' and name == '_load_from_bytes':
            return lambda b: torch.load(io.BytesIO(b), map_location='cpu')
        else:
        return super().find_class(module, name)

#contents = pickle.load(f) becomes...
contents = CPU_Unpickler(f).load()

Source: Github


Instruction: Why does the CrossEntropy loss not go down during training of my network?

Input:
I am fairly new to Pytorch and I am currently trying to implement the network in this paper: https://arxiv.org/pdf/1811.06621.pdf?fbclid=IwAR3Ya9ZfBNN40UO0wct7dGupjlBFEpU47IRHK-wXmejI4U2UQGf03sXHMlw.

I have provided the class for this network and some training code that uses dummy data. The code compiles and runs but the loss that's printed each iteration is always the same (8.371). This leads me to believe that there is something wrong with the way I implemented my network. Is there anything glaringly wrong with my implementation?

import torch
import torch.nn as nn
import torchvision.datasets as dsets
import torchvision.transforms as transforms
from torch.autograd import Variable

torch.manual_seed(1)

# Hyper Parameters
sequence_length = 1
input_size = 320
hidden_size = 2048
recurrent_size = 640
num_layers = 8
num_classes = 10
batch_size = 10
num_epochs = 2
learning_rate = 0.01

# RNNT Model

class RNNTModel(nn.Module):
    def __init__(self, input_size, hidden_size, recurrent_size, bias=True):
        super(RNNTModel, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.recurrent_size = recurrent_size
        self.bias = bias
        self.downsample_fc = nn.Linear(self.recurrent_size * 2, self.recurrent_size)
        self.joint_fc = nn.Linear(self.recurrent_size * 2, self.recurrent_size)
        self.out_fc = nn.Linear(640, 4096)
        self.softmax = nn.LogSoftmax(dim=1)

        self.encoder_1 = nn.ModuleDict({
            'lstm1': nn.LSTM(self.input_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj1': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm2': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj2': nn.Linear(self.hidden_size, self.recurrent_size)
        })

        self.encoder_2 = nn.ModuleDict({
            'lstm3': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj3': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm4': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj4': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm5': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj5': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm6': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj6': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm7': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj7': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm8': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj8': nn.Linear(self.hidden_size, self.recurrent_size)
        })

        self.prediction_net = nn.ModuleDict({
            'fc1': nn.Linear(4096, 76),
            'lstm1': nn.LSTM(76, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj1': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm2': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj2': nn.Linear(self.hidden_size, self.recurrent_size)
        })

    def forward(self, x, ):
        y = [torch.zeros(1, x.size(1), 4096)]
        for i in range(x.size(0) // 2):

            # Unrolled loop of encoder 1
            enc_out, (h1, c1) = self.encoder_1['lstm1'](torch.stack([x[2 * i], x[2 * i + 1]]))
            enc_out = self.encoder_1['proj1'](enc_out)
            enc_out, _ = self.encoder_1['lstm2'](enc_out)
            enc_out = self.encoder_1['proj2'](enc_out)

            # Downsample by halving framrate
            enc_out = enc_out.view(1, -1, 2 * self.recurrent_size)
            enc_out = self.downsample_fc(enc_out)

            # Unrolled loop of encoder 2
            enc_out, _ = self.encoder_2['lstm3'](enc_out)
            enc_out = self.encoder_2['proj3'](enc_out)
            enc_out, _ = self.encoder_2['lstm4'](enc_out)
            enc_out = self.encoder_2['proj4'](enc_out)
            enc_out, _ = self.encoder_2['lstm5'](enc_out)
            enc_out = self.encoder_2['proj5'](enc_out)
            enc_out, _ = self.encoder_2['lstm6'](enc_out)
            enc_out = self.encoder_2['proj6'](enc_out)
            enc_out, _ = self.encoder_2['lstm7'](enc_out)
            enc_out = self.encoder_2['proj3'](enc_out)
            enc_out, _ = self.encoder_2['lstm7'](enc_out)
            enc_out = self.encoder_2['proj3'](enc_out)
            enc_out, _ = self.encoder_2['lstm8'](enc_out)
            enc_out = self.encoder_2['proj8'](enc_out)

            # Unrolled loop of prediction net
            pred_out = self.prediction_net['fc1'](y[i])
            pred_out, _ = self.prediction_net['lstm1'](pred_out)
            pred_out = self.prediction_net['proj1'](pred_out)
            pred_out, _ = self.prediction_net['lstm2'](pred_out)
            pred_out = self.prediction_net['proj2'](pred_out)

            # Unrolled loop of joint layers
            joint_out = torch.cat([enc_out, pred_out], dim=-1)
            joint_out = self.joint_fc(joint_out)
            joint_out = self.out_fc(joint_out)
            joint_out = self.softmax(joint_out)

            y.append(joint_out)

        return(torch.stack(y[1:]))

rnnt = RNNTModel(input_size, hidden_size, recurrent_size, bias=True)
# y = rnnt(torch.rand(batch_size, sequence_length, input_size))

training_data = [(torch.rand(batch_size, sequence_length, input_size), torch.ones(batch_size//2, 1, 4096).long()) for _ in range(100)]

# Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(rnnt.parameters(), lr=learning_rate)

# Train the model
for epoch in range(num_epochs):
    for i, (x, y) in enumerate(training_data):
        x = Variable(x)
        y = Variable(y)
        # Forward + Backward + Optimize
        optimizer.zero_grad()
        outputs = rnnt(x).view(-1, 4096)
        loss = criterion(outputs, torch.max(y, 2)[1].squeeze())
        loss.backward()
        optimizer.step

        if (i+1) % 1 == 0:
            print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f'
                   %(epoch+1, num_epochs, i+1, len(training_data)//batch_size, loss.item()))

Output: I think you want optimizer.step() instead of optimizer.step.


Instruction: Does Tensorflow or Pytorch work on RTX 20xx super series

Input: I'm going to buy a new gpu to learn deep learning. The new Nvidia RTX 2060 Super seems to best fit my needs. But I wonder that is it compatible with CUDA and tensorflow or pytorch now?

Output:
I have bought same VGA a few days ago and using it.
tested it, it works fine in Python (I used numby library) and used CUDA 10.1


Instruction: A strange behavior about pytorch tensor. Any one can explain it clear?

Input:
When I create a PyTorch tensor and tried to explore its type, I found this：

>>> a = torch.rand(3,5)
>>> a
tensor([[0.3356, 0.0968, 0.2185, 0.9843, 0.7846],
        [0.8523, 0.3300, 0.7181, 0.2692, 0.6523],
        [0.0523, 0.9344, 0.3505, 0.8901, 0.6464]])
>>> type(a)
<class 'torch.Tensor'>
>>> a = a.cuda()
>>> a.is_cuda
True
>>> type(a)
<class 'torch.Tensor'>
>>> a.dtype
torch.float32
>>>

Why is type(a) still torch.Tensor rather than torch.cuda.Tensor, even though this tensor is already on GPU?

Output:
You got me digging there, but apparaently type() as built-in method does not work for type detection since 0.4.0 (see this comment and this point in migration guide).

To get proper type, torch.Tensor classes have type() member, which can be simply used:

import torch

a = torch.rand(3, 5)
print(a)
print(a.type())
a = a.to("cuda")
print(a.is_cuda)
print(a.type())

which yields, as expected:

tensor([[0.5060, 0.6998, 0.5054, 0.4822, 0.4408],
        [0.7686, 0.4508, 0.4968, 0.4460, 0.7352],
        [0.1810, 0.6953, 0.7499, 0.7105, 0.1776]])
torch.FloatTensor
True
torch.cuda.FloatTensor

However I am unsure about the rationale standing behind the decision and could not find anything relevant other than that.


Instruction: Why are the images generated by a GAN get darker as the network trains more?

Input:
I created a simple DCGAN with 6 layers and trained it on CelebA dataset (a portion of it containing 30K images).
I noticed my network generated images are dimmed looking and as the network trains more, the bright colors fade into dim ones!

here are some example:
This is how CelebA images look like (real images used for training)  :

and these are the generated ones ,the number shows  the epoch number(they were trained for 30 epochs ultimately) :

What is the cause for this phenomenon?
I tried to do all the general tricks concerning GANs, such as rescaling the input image between -1 and 1, or not using BatchNorm in the first layer of the Discriminator, and for the last layer of the Generator or
using LeakyReLU(0.2) in the Discriminator, and ReLU for the Generator. yet I have no idea why the images are this dim/dark!
Is this caused by simply less training images?
or is it caused by the networks deficiencies ? if so what is the source of such deficiencies?
Here are how these networks implemented :

def conv_batch(in_dim, out_dim, kernel_size, stride, padding, batch_norm=True):
    layers = nn.ModuleList()
    conv = nn.Conv2d(in_dim, out_dim, kernel_size, stride, padding, bias=False)
    layers.append(conv)
 if batch_norm:
        layers.append(nn.BatchNorm2d(out_dim))
 return nn.Sequential(*layers)

class Discriminator(nn.Module):
    def __init__(self, conv_dim=32, act = nn.ReLU()):
        super().__init__()

        self.conv_dim = conv_dim
        self.act = act
        self.conv1 = conv_batch(3, conv_dim, 4, 2, 1, False)
        self.conv2 = conv_batch(conv_dim, conv_dim*2, 4, 2, 1)
        self.conv3 = conv_batch(conv_dim*2, conv_dim*4, 4, 2, 1)
        self.conv4 = conv_batch(conv_dim*4, conv_dim*8, 4, 1, 1)
        self.conv5 = conv_batch(conv_dim*8, conv_dim*10, 4, 2, 1)
        self.conv6 = conv_batch(conv_dim*10, conv_dim*10, 3, 1, 1)

        self.drp = nn.Dropout(0.5)
        self.fc = nn.Linear(conv_dim*10*3*3, 1)

    def forward(self, input):
        batch = input.size(0)
        output = self.act(self.conv1(input))
        output = self.act(self.conv2(output))
        output = self.act(self.conv3(output))
        output = self.act(self.conv4(output))
        output = self.act(self.conv5(output))
        output = self.act(self.conv6(output))

        output = output.view(batch, self.fc.in_features)
        output = self.fc(output)
        output = self.drp(output)

        return output

def deconv_convtranspose(in_dim, out_dim, kernel_size, stride, padding, batchnorm=True):
    layers = []
    deconv = nn.ConvTranspose2d(in_dim, out_dim, kernel_size = kernel_size, stride=stride, padding=padding)
    layers.append(deconv)
    if batchnorm:
        layers.append(nn.BatchNorm2d(out_dim))
    return nn.Sequential(*layers)

class Generator(nn.Module):
    def __init__(self, z_size=100, conv_dim=32):
        super().__init__()
         self.conv_dim = conv_dim
         # make the 1d input into a 3d output of shape (conv_dim*4, 4, 4 )
         self.fc = nn.Linear(z_size, conv_dim*4*4*4)#4x4
         # conv and deconv layer work on 3d volumes, so we now only need to pass the number of fmaps and the
         # input volume size (its h,w which is 4x4!)
         self.drp = nn.Dropout(0.5)
        self.deconv1 = deconv_convtranspose(conv_dim*4, conv_dim*3, kernel_size =4, stride=2, padding=1)
        self.deconv2 = deconv_convtranspose(conv_dim*3, conv_dim*2, kernel_size =4, stride=2, padding=1)
        self.deconv3 = deconv_convtranspose(conv_dim*2, conv_dim, kernel_size =4, stride=2, padding=1)
        self.deconv4 = deconv_convtranspose(conv_dim, conv_dim, kernel_size =3, stride=2, padding=1)
        self.deconv5 = deconv_convtranspose(conv_dim, 3, kernel_size =4, stride=1, padding=1, batchnorm=False)

    def forward(self, input):
        output = self.fc(input)
        output = self.drp(output)
        output = output.view(-1, self.conv_dim*4, 4, 4)
        output = F.relu(self.deconv1(output))
        output = F.relu(self.deconv2(output))
        output = F.relu(self.deconv3(output))
        output = F.relu(self.deconv4(output))
        # we create the image using tanh!
        output = F.tanh(self.deconv5(output))

        return output

# testing nets
dd = Discriminator()
zd = np.random.rand(2,3,64,64)
zd = torch.from_numpy(zd).float()
# print(dd)
print(dd(zd).shape)

gg = Generator()
z = np.random.uniform(-1,1,size=(2,100))
z = torch.from_numpy(z).float()
print(gg(z).shape)

Output:
I think that the problem lies rather in the architecture itself and I would first consider the overall quality of generated images rather than their brightness or darkness. The generations clearly get better as you train for more epochs. I agree that the images get darker but even in the early epochs, the generated images are significantly darker than the ones in the training samples. (At least compared to ones that you posted.)
And now coming back to your architecture, 30k samples are actually enough to obtain very convincing results as achieved by state-of-the-art models in face generations. The generations do get better but they are still far away from being "very good".
I think the generator is definitely not strong enough and is the problematic part. (The fact that your generator loss skyrockets can also be a hint for this.) In the generator, all you do is just upsampling and upsampling. You should note that the transposed convolution is more like a heuristic and it does not provide much learnability. This is related to the nature of the problem. When you are doing convolutions, you have all the information and you are trying to learn to encode but in the decoder, you are trying to recover information that was previously lost :). So, in a way, it is harder to learn because the information taken as input is limited and lacking.
In fact, deterministic bilinear interpolation methods do perform similar or even better than transposed convolutions and these are purely based on scaling/extending with zero learnability. (https://arxiv.org/pdf/1707.05847.pdf)
To observe the transposed convolutions' limits, I suggest that you replace all the Transposedconv2d with UpSampling2D (https://www.tensorflow.org/api_docs/python/tf/keras/layers/UpSampling2D) and I claim that the results will not be much different. UpSampling2D is one of those deterministic methods that I mentioned.
To improve your generator, you can try to insert convolutional layers between upsampling layers. These layers would refine the features/images and correct some of the mistakes that occurred during the up-sampling. In addition to corrections, the next upsampling layer would take a more informative input. What I mean is to try a UNet like decoding that you can find in this link (https://arxiv.org/pdf/1505.04597.pdf). Of course, that would be a primary step to explore. There are many more GAN architectures that you can try and probably perform better.


Instruction: How do I compile a pytorch script into an exe file with small size?

Input:
I have a semantic segmentation model using PyTorch. In order to participate in a competition, I am compiling the test.py to an exe file with PyInstaller and UPX. Although the resulting executable file runs correctly, its size is nearly 800MB. How do I make it smaller?

This is my test.py:

from torch import nn
from torch.autograd import Variable as V
from torch import Tensor
from torch import cuda
from torch import load

import cv2
import os
import numpy as np

from time import time

from networks.unet import Unet
# from networks.dunet import Dunet
# from networks.dinknet import LinkNet34, DinkNet34, DinkNet50, DinkNet101, DinkNet34_less_pool
# from networks.dinkbranch import DinkBranch50, DinkBranch34

BATCHSIZE_PER_CARD = 2

class TTAFrame():
    def __init__(self, net):
        self.net = net().cuda()
        self.net = nn.DataParallel(self.net, device_ids=range(cuda.device_count()))

    def test_one_img_from_path(self, path, evalmode = True):
        if evalmode:
            self.net.eval()
        batchsize = cuda.device_count() * BATCHSIZE_PER_CARD
        if batchsize >= 8:
            return self.test_one_img_from_path_1(path)
        elif batchsize >= 4:
            return self.test_one_img_from_path_2(path)
        elif batchsize >= 2:
            return self.test_one_img_from_path_4(path)

    def test_one_img_from_path_8(self, path):
        img = cv2.imread(path)#.transpose(2,0,1)[None]
        img90 = np.array(np.rot90(img))
        img1 = np.concatenate([img[None],img90[None]])
        img2 = np.array(img1)[:,::-1]
        img3 = np.array(img1)[:,:,::-1]
        img4 = np.array(img2)[:,:,::-1]

        img1 = img1.transpose(0,3,1,2)
        img2 = img2.transpose(0,3,1,2)
        img3 = img3.transpose(0,3,1,2)
        img4 = img4.transpose(0,3,1,2)

        img1 = V(Tensor(np.array(img1, np.float32)/255.0 * 3.2 -1.6).cuda())
        img2 = V(Tensor(np.array(img2, np.float32)/255.0 * 3.2 -1.6).cuda())
        img3 = V(Tensor(np.array(img3, np.float32)/255.0 * 3.2 -1.6).cuda())
        img4 = V(Tensor(np.array(img4, np.float32)/255.0 * 3.2 -1.6).cuda())

        maska = self.net.forward(img1).squeeze().cpu().data.numpy()
        maskb = self.net.forward(img2).squeeze().cpu().data.numpy()
        maskc = self.net.forward(img3).squeeze().cpu().data.numpy()
        maskd = self.net.forward(img4).squeeze().cpu().data.numpy()

        mask1 = maska + maskb[:,::-1] + maskc[:,:,::-1] + maskd[:,::-1,::-1]
        mask2 = mask1[0] + np.rot90(mask1[1])[::-1,::-1]

        return mask2

    def test_one_img_from_path_4(self, path):
        img = cv2.imread(path)#.transpose(2,0,1)[None]
        img90 = np.array(np.rot90(img))
        img1 = np.concatenate([img[None],img90[None]])
        img2 = np.array(img1)[:,::-1]
        img3 = np.array(img1)[:,:,::-1]
        img4 = np.array(img2)[:,:,::-1]

        img1 = img1.transpose(0,3,1,2)
        img2 = img2.transpose(0,3,1,2)
        img3 = img3.transpose(0,3,1,2)
        img4 = img4.transpose(0,3,1,2)

        img1 = V(Tensor(np.array(img1, np.float32)/255.0 * 3.2 -1.6).cuda())
        img2 = V(Tensor(np.array(img2, np.float32)/255.0 * 3.2 -1.6).cuda())
        img3 = V(Tensor(np.array(img3, np.float32)/255.0 * 3.2 -1.6).cuda())
        img4 = V(Tensor(np.array(img4, np.float32)/255.0 * 3.2 -1.6).cuda())

        maska = self.net.forward(img1).squeeze().cpu().data.numpy()
        maskb = self.net.forward(img2).squeeze().cpu().data.numpy()
        maskc = self.net.forward(img3).squeeze().cpu().data.numpy()
        maskd = self.net.forward(img4).squeeze().cpu().data.numpy()

        mask1 = maska + maskb[:,::-1] + maskc[:,:,::-1] + maskd[:,::-1,::-1]
        mask2 = mask1[0] + np.rot90(mask1[1])[::-1,::-1]

        return mask2

    def test_one_img_from_path_2(self, path):
        img = cv2.imread(path)#.transpose(2,0,1)[None]
        img90 = np.array(np.rot90(img))
        img1 = np.concatenate([img[None],img90[None]])
        img2 = np.array(img1)[:,::-1]
        img3 = np.concatenate([img1,img2])
        img4 = np.array(img3)[:,:,::-1]
        img5 = img3.transpose(0,3,1,2)
        img5 = np.array(img5, np.float32)/255.0 * 3.2 -1.6
        img5 = V(Tensor(img5).cuda())
        img6 = img4.transpose(0,3,1,2)
        img6 = np.array(img6, np.float32)/255.0 * 3.2 -1.6
        img6 = V(Tensor(img6).cuda())

        maska = self.net.forward(img5).squeeze().cpu().data.numpy()#.squeeze(1)
        maskb = self.net.forward(img6).squeeze().cpu().data.numpy()

        mask1 = maska + maskb[:,:,::-1]
        mask2 = mask1[:2] + mask1[2:,::-1]
        mask3 = mask2[0] + np.rot90(mask2[1])[::-1,::-1]

        return mask3

    def test_one_img_from_path_1(self, path):
        img = cv2.imread(path)#.transpose(2,0,1)[None]

        img90 = np.array(np.rot90(img))
        img1 = np.concatenate([img[None],img90[None]])
        img2 = np.array(img1)[:,::-1]
        img3 = np.concatenate([img1,img2])
        img4 = np.array(img3)[:,:,::-1]
        img5 = np.concatenate([img3,img4]).transpose(0,3,1,2)
        img5 = np.array(img5, np.float32)/255.0 * 3.2 -1.6
        img5 = V(Tensor(img5).cuda())

        mask = self.net.forward(img5).squeeze().cpu().data.numpy()#.squeeze(1)
        mask1 = mask[:4] + mask[4:,:,::-1]
        mask2 = mask1[:2] + mask1[2:,::-1]
        mask3 = mask2[0] + np.rot90(mask2[1])[::-1,::-1]

        return mask3

    def load(self, path):
        self.net.load_state_dict(load(path))

#source = 'dataset/test/'
import sys
if len(sys.argv) < 2:
    arg1 = r'dataset/504/original'
else:
    arg1 = sys.argv[1]

# source = r'dataset/504/original'
source = arg1
source_path = os.path.join(os.getcwd(), source)

val = os.listdir(source_path)
solver = TTAFrame(Unet)
model_path = '/'
model_path = r'weights/log02_Unet.th'
solver.load(os.path.join(os.getcwd(), model_path))

tic = time()
target = r'submits/log02_baseline504'

target_path = os.path.join(os.getcwd(), target)
if os.path.exists(target_path):
    pass
else:
    os.makedirs(target_path)
for i,name in enumerate(val):
    if i%10 == 0:
        print(i/10, '    ','%.2f'%(time()-tic))
    mask = solver.test_one_img_from_path(os.path.join(source_path, name))
    mask[mask>4.0] = 255
    mask[mask<=4.0] = 0
    mask = np.concatenate([mask[:,:,None],mask[:,:,None],mask[:,:,None]],axis=2)
    cv2.imwrite(target_path+r'/'+name[:-7]+'mask.png', mask.astype(np.uint8))

This is the 'Unet' file:

from torch import autograd, cat
from torch import nn

class Unet(nn.Module):
    def __init__(self):
        super(Unet, self).__init__()

        self.down1 = self.conv_stage(3, 8)
        self.down2 = self.conv_stage(8, 16)
        self.down3 = self.conv_stage(16, 32)
        self.down4 = self.conv_stage(32, 64)
        self.down5 = self.conv_stage(64, 128)
        self.down6 = self.conv_stage(128, 256)
        self.down7 = self.conv_stage(256, 512)

        self.center = self.conv_stage(512, 1024)
        #self.center_res = self.resblock(1024)

        self.up7 = self.conv_stage(1024, 512)
        self.up6 = self.conv_stage(512, 256)
        self.up5 = self.conv_stage(256, 128)
        self.up4 = self.conv_stage(128, 64)
        self.up3 = self.conv_stage(64, 32)
        self.up2 = self.conv_stage(32, 16)
        self.up1 = self.conv_stage(16, 8)

        self.trans7 = self.upsample(1024, 512)
        self.trans6 = self.upsample(512, 256)
        self.trans5 = self.upsample(256, 128)
        self.trans4 = self.upsample(128, 64)
        self.trans3 = self.upsample(64, 32)
        self.trans2 = self.upsample(32, 16)
        self.trans1 = self.upsample(16, 8)

        self.conv_last = nn.Sequential(
            nn.Conv2d(8, 1, 3, 1, 1),
            nn.Sigmoid()
        )

        self.max_pool = nn.MaxPool2d(2)

        for m in self.modules():
            if isinstance(m, nn.Conv2d) or isinstance(m, nn.ConvTranspose2d):
                if m.bias is not None:
                    m.bias.data.zero_()

    def conv_stage(self, dim_in, dim_out, kernel_size=3, stride=1, padding=1, bias=True, useBN=False):
        if useBN:
            return nn.Sequential(
              nn.Conv2d(dim_in, dim_out, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias),
              nn.BatchNorm2d(dim_out),
              #nn.LeakyReLU(0.1),
              nn.ReLU(),
              nn.Conv2d(dim_out, dim_out, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias),
              nn.BatchNorm2d(dim_out),
              #nn.LeakyReLU(0.1),
              nn.ReLU(),
            )
        else:
            return nn.Sequential(
              nn.Conv2d(dim_in, dim_out, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias),
              nn.ReLU(),
              nn.Conv2d(dim_out, dim_out, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias),
              nn.ReLU()
            )

    def upsample(self, ch_coarse, ch_fine):
        return nn.Sequential(
            nn.ConvTranspose2d(ch_coarse, ch_fine, 4, 2, 1, bias=False),
            nn.ReLU()
        )

    def forward(self, x):
        conv1_out = self.down1(x)
        conv2_out = self.down2(self.max_pool(conv1_out))
        conv3_out = self.down3(self.max_pool(conv2_out))
        conv4_out = self.down4(self.max_pool(conv3_out))
        conv5_out = self.down5(self.max_pool(conv4_out))
        conv6_out = self.down6(self.max_pool(conv5_out))
        conv7_out = self.down7(self.max_pool(conv6_out))

        out = self.center(self.max_pool(conv7_out))
        #out = self.center_res(out)

        out = self.up7(cat((self.trans7(out), conv7_out), 1))
        out = self.up6(cat((self.trans6(out), conv6_out), 1))
        out = self.up5(cat((self.trans5(out), conv5_out), 1))
        out = self.up4(cat((self.trans4(out), conv4_out), 1))
        out = self.up3(cat((self.trans3(out), conv3_out), 1))
        out = self.up2(cat((self.trans2(out), conv2_out), 1))
        out = self.up1(cat((self.trans1(out), conv1_out), 1))

        out = self.conv_last(out)

        return out

Output:
pyinstaller is kind of cheated .exe. It does not compile the script, but bundles what's needed (including python interpreter) into one (or many) files.

To really be Python agnostic you should convert your model using torchscript (read about it here). You will be able to run your module using C++ libtorch without Python interpreter.


Instruction: Undefined symbol on torchvision import

Input:
pip install torch==0.4.1 -f https://download.pytorch.org/whl/cu100/stable

cat /usr/local/cuda/version.txt
CUDA Version 10.1.168

python -V
Python 3.5.2

python -c "import torch; print(torch.__version__)"
0.4.1

Get error on :

python -c "import torchvision; print(torchvision.__version__)"

    ...
    from torchvision import _C
    ImportError: /my_env/lib/python3.5/site-packages/torchvision/_C.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E

Versions:

pip freeze | grep torch
torch==0.4.1
torchfile==0.1.0
torchvision==0.3.0

I have tried:

pip install torchvision==0.4.1 --no-dependencies
Collecting torchvision==0.4.1
  ERROR: Could not find a version that satisfies the requirement torchvision==0.4.1 (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.3.0)
ERROR: No matching distribution found for torchvision==0.4.1

How to install older version of torchvision?

Output: Seems pip install torchvision==0.2.0 --no-deps --no-cache-dir helped.


Instruction: Extracting 3D patches from 3D images in both overlapping and nonoverlapping process and recovering the image back

Input:
I am working with 172x220x156 shaped 3D images. To feed the image into the network for output I need to extract patches of size 32x32x32 from the image and add those back to get the image again.
Since my image dimension are not multiples of patch size thus I have to get overlapping patches.
I want to know how to do that.

I am working in PyTorch, there are some options like unfold and fold but I am not sure how they work.

Output:
To extract (overlapping-) patches and to reconstruct the input shape we can use the torch.nn.functional.unfold and the inverse operation torch.nn.functional.fold. These methods only process 4D tensors or 2D images, however you can use these methods to process one dimension at a time.
Few notes:

This way requires fold/unfold methods from pytorch, unfortunately I have yet to find a similar method in the TF api.

We can extract patches in 2 ways, their output is the same. The methods are called extract_patches_3d and extract_patches_3ds where X is the number of dimensions. The latter uses torch.Tensor.unfold() and has less lines of code. (output is the same, except it cannot use dilation)

The methods extract_patches_Xd and combine_patches_Xd are inverse methods and the combiner reverses the steps from the extracter step by step.

The lines of code are followed by a comment stating the dimensionality such as (B, C, D, H, W). The following are used:

B: Batch size
C: Channels
D: Depth Dimension
H: Height Dimension
W: Width Dimension
x_dim_in: In the extraction method, this is the number input pixels in dimension x. In the combining method, this is the number of number of sliding windows in dimension x.
x_dim_out: In the extraction method, this is the number of sliding windows in dimension x. In the combining method, this is the number output pixels in dimension x.

I have a public notebook to try out the code

The get_dim_blocks() method is the function given on the pytorch docs website to compute the output shape of a convolutional layer.

Note that if you have overlapping patches and you combine them, the overlapping elements will be summed. If you would like to get the initial input again there is a way.

Create similar sized tensor of ones as the patches with torch.ones_like(patches_tensor).
Combine the patches into full image with same output shape. (this creates a counter for overlapping elements).
Divide the Combined image with the Combined ones, this should reverse any double summation of elements.
(3D):
We need to use 2 fold and unfold where we first apply the fold to the D dimension and leave the W and H untouched by setting kernel to 1, padding to 0, stride to 1 and dilation to 1. After we review the tensor and fold over the H and W dimensions. The unfolding happens in reverse, starting with H and W, then D.

def extract_patches_3ds(x, kernel_size, padding=0, stride=1):
    if isinstance(kernel_size, int):
        kernel_size = (kernel_size, kernel_size, kernel_size)
    if isinstance(padding, int):
        padding = (padding, padding, padding, padding, padding, padding)
    if isinstance(stride, int):
        stride = (stride, stride, stride)

    channels = x.shape[1]

    x = torch.nn.functional.pad(x, padding)
    # (B, C, D, H, W)
    x = x.unfold(2, kernel_size[0], stride[0]).unfold(3, kernel_size[1], stride[1]).unfold(4, kernel_size[2], stride[2])
    # (B, C, d_dim_out, h_dim_out, w_dim_out, kernel_size[0], kernel_size[1], kernel_size[2])
    x = x.contiguous().view(-1, channels, kernel_size[0], kernel_size[1], kernel_size[2])
    # (B * d_dim_out * h_dim_out * w_dim_out, C, kernel_size[0], kernel_size[1], kernel_size[2])
    return x

def extract_patches_3d(x, kernel_size, padding=0, stride=1, dilation=1):
    if isinstance(kernel_size, int):
        kernel_size = (kernel_size, kernel_size, kernel_size)
    if isinstance(padding, int):
        padding = (padding, padding, padding)
    if isinstance(stride, int):
        stride = (stride, stride, stride)
    if isinstance(dilation, int):
        dilation = (dilation, dilation, dilation)

    def get_dim_blocks(dim_in, dim_kernel_size, dim_padding = 0, dim_stride = 1, dim_dilation = 1):
        dim_out = (dim_in + 2 * dim_padding - dim_dilation * (dim_kernel_size - 1) - 1) // dim_stride + 1
        return dim_out

    channels = x.shape[1]

    d_dim_in = x.shape[2]
    h_dim_in = x.shape[3]
    w_dim_in = x.shape[4]
    d_dim_out = get_dim_blocks(d_dim_in, kernel_size[0], padding[0], stride[0], dilation[0])
    h_dim_out = get_dim_blocks(h_dim_in, kernel_size[1], padding[1], stride[1], dilation[1])
    w_dim_out = get_dim_blocks(w_dim_in, kernel_size[2], padding[2], stride[2], dilation[2])
    # print(d_dim_in, h_dim_in, w_dim_in, d_dim_out, h_dim_out, w_dim_out)

    # (B, C, D, H, W)
    x = x.view(-1, channels, d_dim_in, h_dim_in * w_dim_in)
    # (B, C, D, H * W)

    x = torch.nn.functional.unfold(x, kernel_size=(kernel_size[0], 1), padding=(padding[0], 0), stride=(stride[0], 1), dilation=(dilation[0], 1))
    # (B, C * kernel_size[0], d_dim_out * H * W)

    x = x.view(-1, channels * kernel_size[0] * d_dim_out, h_dim_in, w_dim_in)
    # (B, C * kernel_size[0] * d_dim_out, H, W)

    x = torch.nn.functional.unfold(x, kernel_size=(kernel_size[1], kernel_size[2]), padding=(padding[1], padding[2]), stride=(stride[1], stride[2]), dilation=(dilation[1], dilation[2]))
    # (B, C * kernel_size[0] * d_dim_out * kernel_size[1] * kernel_size[2], h_dim_out, w_dim_out)

    x = x.view(-1, channels, kernel_size[0], d_dim_out, kernel_size[1], kernel_size[2], h_dim_out, w_dim_out)
    # (B, C, kernel_size[0], d_dim_out, kernel_size[1], kernel_size[2], h_dim_out, w_dim_out)

    x = x.permute(0, 1, 3, 6, 7, 2, 4, 5)
    # (B, C, d_dim_out, h_dim_out, w_dim_out, kernel_size[0], kernel_size[1], kernel_size[2])

    x = x.contiguous().view(-1, channels, kernel_size[0], kernel_size[1], kernel_size[2])
    # (B * d_dim_out * h_dim_out * w_dim_out, C, kernel_size[0], kernel_size[1], kernel_size[2])

    return x

def combine_patches_3d(x, kernel_size, output_shape, padding=0, stride=1, dilation=1):
    if isinstance(kernel_size, int):
        kernel_size = (kernel_size, kernel_size, kernel_size)
    if isinstance(padding, int):
        padding = (padding, padding, padding)
    if isinstance(stride, int):
        stride = (stride, stride, stride)
    if isinstance(dilation, int):
        dilation = (dilation, dilation, dilation)

    def get_dim_blocks(dim_in, dim_kernel_size, dim_padding = 0, dim_stride = 1, dim_dilation = 1):
        dim_out = (dim_in + 2 * dim_padding - dim_dilation * (dim_kernel_size - 1) - 1) // dim_stride + 1
        return dim_out

    channels = x.shape[1]
    d_dim_out, h_dim_out, w_dim_out = output_shape[2:]
    d_dim_in = get_dim_blocks(d_dim_out, kernel_size[0], padding[0], stride[0], dilation[0])
    h_dim_in = get_dim_blocks(h_dim_out, kernel_size[1], padding[1], stride[1], dilation[1])
    w_dim_in = get_dim_blocks(w_dim_out, kernel_size[2], padding[2], stride[2], dilation[2])
    # print(d_dim_in, h_dim_in, w_dim_in, d_dim_out, h_dim_out, w_dim_out)

    x = x.view(-1, channels, d_dim_in, h_dim_in, w_dim_in, kernel_size[0], kernel_size[1], kernel_size[2])
    # (B, C, d_dim_in, h_dim_in, w_dim_in, kernel_size[0], kernel_size[1], kernel_size[2])

    x = x.permute(0, 1, 5, 2, 6, 7, 3, 4)
    # (B, C, kernel_size[0], d_dim_in, kernel_size[1], kernel_size[2], h_dim_in, w_dim_in)

    x = x.contiguous().view(-1, channels * kernel_size[0] * d_dim_in * kernel_size[1] * kernel_size[2], h_dim_in * w_dim_in)
    # (B, C * kernel_size[0] * d_dim_in * kernel_size[1] * kernel_size[2], h_dim_in * w_dim_in)

    x = torch.nn.functional.fold(x, output_size=(h_dim_out, w_dim_out), kernel_size=(kernel_size[1], kernel_size[2]), padding=(padding[1], padding[2]), stride=(stride[1], stride[2]), dilation=(dilation[1], dilation[2]))
    # (B, C * kernel_size[0] * d_dim_in, H, W)

    x = x.view(-1, channels * kernel_size[0], d_dim_in * h_dim_out * w_dim_out)
    # (B, C * kernel_size[0], d_dim_in * H * W)

    x = torch.nn.functional.fold(x, output_size=(d_dim_out, h_dim_out * w_dim_out), kernel_size=(kernel_size[0], 1), padding=(padding[0], 0), stride=(stride[0], 1), dilation=(dilation[0], 1))
    # (B, C, D, H * W)

    x = x.view(-1, channels, d_dim_out, h_dim_out, w_dim_out)
    # (B, C, D, H, W)

    return x

a = torch.arange(1, 129, dtype=torch.float).view(2,2,2,4,4)
print(a.shape)
print(a)
b = extract_patches_3d(a, 2, padding=1, stride=1)
bs = extract_patches_3ds(a, 2, padding=1, stride=1)
print(b.shape)
print(b)
c = combine_patches_3d(b, (2,2,2,4,4), kernel_size=2, padding=1, stride=1)
print(c.shape)
print(c)
ones = torch.ones_like(b)
ones = combine_patches_3d(ones, (2,2,2,4,4), kernel_size=2, padding=1, stride=1)
print(torch.all(a==c))
print(c.shape, ones.shape)
d = c / ones
print(d)
print(torch.all(a==d))

Output (3D)
torch.Size([2, 2, 2, 4, 4])
tensor([[[[[  1.,   2.,   3.,   4.],
           [  5.,   6.,   7.,   8.],
           [  9.,  10.,  11.,  12.],
           [ 13.,  14.,  15.,  16.]],

          [[ 17.,  18.,  19.,  20.],
           [ 21.,  22.,  23.,  24.],
           [ 25.,  26.,  27.,  28.],
           [ 29.,  30.,  31.,  32.]]],

         [[[ 33.,  34.,  35.,  36.],
           [ 37.,  38.,  39.,  40.],
           [ 41.,  42.,  43.,  44.],
           [ 45.,  46.,  47.,  48.]],

          [[ 49.,  50.,  51.,  52.],
           [ 53.,  54.,  55.,  56.],
           [ 57.,  58.,  59.,  60.],
           [ 61.,  62.,  63.,  64.]]]],

        [[[[ 65.,  66.,  67.,  68.],
           [ 69.,  70.,  71.,  72.],
           [ 73.,  74.,  75.,  76.],
           [ 77.,  78.,  79.,  80.]],

          [[ 81.,  82.,  83.,  84.],
           [ 85.,  86.,  87.,  88.],
           [ 89.,  90.,  91.,  92.],
           [ 93.,  94.,  95.,  96.]]],

         [[[ 97.,  98.,  99., 100.],
           [101., 102., 103., 104.],
           [105., 106., 107., 108.],
           [109., 110., 111., 112.]],

          [[113., 114., 115., 116.],
           [117., 118., 119., 120.],
           [121., 122., 123., 124.],
           [125., 126., 127., 128.]]]]])
torch.Size([150, 2, 2, 2, 2])
tensor([[[[[  0.,   0.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  0.,   1.]]],

         [[[  0.,   0.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  1.,   2.]]]],

        [[[[  0.,   0.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  2.,   3.]]],

         [[[  0.,   0.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  3.,   4.]]]],

        [[[[  0.,   0.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  4.,   0.]]],

         [[[  0.,   0.],
           [  0.,   0.]],

          [[  0.,   1.],
           [  0.,   5.]]]],

        ...,

        [[[[124.,   0.],
           [128.,   0.]],

          [[  0.,   0.],
           [  0.,   0.]]],

         [[[  0., 125.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  0.,   0.]]]],

        [[[[125., 126.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  0.,   0.]]],

         [[[126., 127.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  0.,   0.]]]],

        [[[[127., 128.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  0.,   0.]]],

         [[[128.,   0.],
           [  0.,   0.]],

          [[  0.,   0.],
           [  0.,   0.]]]]])
torch.Size([2, 2, 2, 4, 4])
tensor([[[[[   8.,   16.,   24.,   32.],
           [  40.,   48.,   56.,   64.],
           [  72.,   80.,   88.,   96.],
           [ 104.,  112.,  120.,  128.]],

          [[ 136.,  144.,  152.,  160.],
           [ 168.,  176.,  184.,  192.],
           [ 200.,  208.,  216.,  224.],
           [ 232.,  240.,  248.,  256.]]],

         [[[ 264.,  272.,  280.,  288.],
           [ 296.,  304.,  312.,  320.],
           [ 328.,  336.,  344.,  352.],
           [ 360.,  368.,  376.,  384.]],

          [[ 392.,  400.,  408.,  416.],
           [ 424.,  432.,  440.,  448.],
           [ 456.,  464.,  472.,  480.],
           [ 488.,  496.,  504.,  512.]]]],

        [[[[ 520.,  528.,  536.,  544.],
           [ 552.,  560.,  568.,  576.],
           [ 584.,  592.,  600.,  608.],
           [ 616.,  624.,  632.,  640.]],

          [[ 648.,  656.,  664.,  672.],
           [ 680.,  688.,  696.,  704.],
           [ 712.,  720.,  728.,  736.],
           [ 744.,  752.,  760.,  768.]]],

         [[[ 776.,  784.,  792.,  800.],
           [ 808.,  816.,  824.,  832.],
           [ 840.,  848.,  856.,  864.],
           [ 872.,  880.,  888.,  896.]],

          [[ 904.,  912.,  920.,  928.],
           [ 936.,  944.,  952.,  960.],
           [ 968.,  976.,  984.,  992.],
           [1000., 1008., 1016., 1024.]]]]])
tensor(False)
torch.Size([2, 2, 2, 4, 4]) torch.Size([2, 2, 2, 4, 4])
tensor([[[[[  1.,   2.,   3.,   4.],
           [  5.,   6.,   7.,   8.],
           [  9.,  10.,  11.,  12.],
           [ 13.,  14.,  15.,  16.]],

          [[ 17.,  18.,  19.,  20.],
           [ 21.,  22.,  23.,  24.],
           [ 25.,  26.,  27.,  28.],
           [ 29.,  30.,  31.,  32.]]],

         [[[ 33.,  34.,  35.,  36.],
           [ 37.,  38.,  39.,  40.],
           [ 41.,  42.,  43.,  44.],
           [ 45.,  46.,  47.,  48.]],

          [[ 49.,  50.,  51.,  52.],
           [ 53.,  54.,  55.,  56.],
           [ 57.,  58.,  59.,  60.],
           [ 61.,  62.,  63.,  64.]]]],

        [[[[ 65.,  66.,  67.,  68.],
           [ 69.,  70.,  71.,  72.],
           [ 73.,  74.,  75.,  76.],
           [ 77.,  78.,  79.,  80.]],

          [[ 81.,  82.,  83.,  84.],
           [ 85.,  86.,  87.,  88.],
           [ 89.,  90.,  91.,  92.],
           [ 93.,  94.,  95.,  96.]]],

         [[[ 97.,  98.,  99., 100.],
           [101., 102., 103., 104.],
           [105., 106., 107., 108.],
           [109., 110., 111., 112.]],

          [[113., 114., 115., 116.],
           [117., 118., 119., 120.],
           [121., 122., 123., 124.],
           [125., 126., 127., 128.]]]]])
tensor(True)


Instruction: How to delete permanently from mounted Drive folder?

Input:
I wrote a script to upload my models and training examples to Google Drive after every iteration in case of crashes or anything that stops the notebook from running, which looks something like this:

drive_path = 'drive/My Drive/Colab Notebooks/models/'
if path.exists(drive_path):
    shutil.rmtree(drive_path)
shutil.copytree('models', drive_path)

Whenever I check my Google Drive, a few GBs is taken up by dozens of deleted models folder in the Trash, which I have to manually delete them.

The only function in google.colab.drive seems to be mount and that's it.

According to this tutorial, shutil.rmtree() removes a directory permanently but apparently it doesn't work for Drive.

Output:
It is possible to perform this action inside Google Colab by using the pydrive module. I suggest that you first move your unwanted files and folders to Trash (by ordinarily removing them in your code), and then, anytime you think it's necessary (e.g. you want to free up some space for saving weights of a new DL project), empty your trash by coding the following lines.

In order to permanently empty your Google Drive's Trash, code the following lines in your Google Colab notebook:

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
my_drive = GoogleDrive(gauth)

After entering authentication code and creating a valid instance of GoogleDrive class, write:

for a_file in my_drive.ListFile({'q': "trashed = true"}).GetList():
    # print the name of the file being deleted.
    print(f'the file "{a_file['title']}", is about to get deleted permanently.')
    # delete the file permanently.
    a_file.Delete()

If you don't want to use my suggestion and want to permanently delete a specific folder in your Drive, it is possible that you have to make more complex queries and deal with fileId, parentId, and the fact that a file or folder in your Drive may have multiple parent folders, when making queries to Google Drive API.

For more information:

You can find examples of more complex (yet typical) queries, here.
You can find an example of Checking if a file is in a specific folder, here.
This statement that Files and folders in Google Drive can each have multiple parent folders may become better and more deeply understood, by reading this post.


Instruction: How to get grads in pytorch after matrix multiplication?

Input:
I want to get the product of matrix multiplication in the latent space and optimize the weight matrix by the optimizer. I use different kinds of ways to do that. While, The value of 'pi_' in the below codes never changes. What should I do?

I've tried different functions to get the product, like torch.mm(), torch.matual() and @. The weight matrix 'pi_' never changed.

import torch
from torch.utils.data import DataLoader
from torch.utils.data import TensorDataset
#from torchvision import transforms
from torchvision.datasets import MNIST

def get_mnist(data_dir='./data/mnist/',batch_size=128):
    train=MNIST(root=data_dir,train=True,download=True)
    test=MNIST(root=data_dir,train=False,download=True)

    X=torch.cat([train.data.float().view(-1,784)/255.,test.data.float().view(-1,784)/255.],0)
    Y=torch.cat([train.targets,test.targets],0)

    dataset=dict()
    dataset['X']=X
    dataset['Y']=Y

    dataloader=DataLoader(TensorDataset(X,Y),batch_size=batch_size,shuffle=True)

    return dataloader

class tests(torch.nn.Module):
    def __init__(self):
        super(tests, self).__init__()

        self.pi_= torch.nn.Parameter(torch.FloatTensor(10, 1).fill_(1),requires_grad=True)
        self.linear0 = torch.nn.Linear(784,10)
        self.linear1 = torch.nn.Linear(1,784)

    def forward(self, data):
        data = torch.nn.functional.relu(self.linear0(data))
#        data = data.mm(self.pi_)
#        data = torch.mm(data, self.pi_)
#        data = data @ self.pi_
        data = torch.matmul(data, self.pi_)
        data = torch.nn.functional.relu(self.linear1(data))
        return data

if __name__ == '__main__':
    DL=get_mnist()
    t = tests().cuda()
    optimizer = torch.optim.Adam(t.parameters(), lr = 2e-3)

    for i in range(100):
        for inputs, classes in DL:
            inputs = inputs.cuda()

            res = t(inputs)
            loss = torch.nn.functional.mse_loss(res, inputs)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        print("Epoch:", i,"pi:",t.pi_)

Output:
TL;DR
You have too many parameters in your neural network, some of them becomes useless and therefore they are no longer being updated. Change your network architecture to reduce useless parameters.

Full explanation:
The weight matrix pi_ does change. You initialize pi_ as all 1, after running the first epochs, the weight matrix pi_ becomes

output >>>
tensor([[0.9879],
        [0.9874],
        [0.9878],
        [0.9880],
        [0.9876],
        [0.9878],
        [0.9878],
        [0.9873],
        [0.9877],
        [0.9871]], device='cuda:0', requires_grad=True)

So, it has changed once. The true reason behind it involves some mathematics. But to put it in non-mathematical ways it means this layer doesn't contribute much to the loss, therefore, the network decided not to update this layer. i.e. The existence of pi_ in this network is redundant.

If you want to observe the change in pi_, you should modify the neural network such that pi_ is not redundant anymore.

One possible modification is to change your reconstruction problem to a classification problem

import torch
from torch.utils.data import DataLoader
from torch.utils.data import TensorDataset
#from torchvision import transforms
from torchvision.datasets import MNIST

def get_mnist(data_dir='./data/mnist/',batch_size=128):
    train=MNIST(root=data_dir,train=True,download=True)
    test=MNIST(root=data_dir,train=False,download=True)

    X=torch.cat([train.data.float().view(-1,784)/255.,test.data.float().view(-1,784)/255.],0)
    Y=torch.cat([train.targets,test.targets],0)

    dataset=dict()
    dataset['X']=X
    dataset['Y']=Y

    dataloader=DataLoader(TensorDataset(X,Y),batch_size=batch_size,shuffle=True)

    return dataloader

class tests(torch.nn.Module):
    def __init__(self):
        super(tests, self).__init__()

#         self.pi_= torch.nn.Parameter(torch.randn((10, 1),requires_grad=True))
        self.pi_= torch.nn.Parameter(torch.FloatTensor(10, 1).fill_(1),requires_grad=True)
        self.linear0 = torch.nn.Linear(784,10)
#         self.linear1 = torch.nn.Linear(1,784)

    def forward(self, data):
        data = torch.nn.functional.relu(self.linear0(data))
#        data = data.mm(self.pi_)
#        data = torch.mm(data, self.pi_)
#        data = data @ self.pi_
        data = torch.matmul(data, self.pi_)
#         data = torch.nn.functional.relu(self.linear1(data))
        return data

if __name__ == '__main__':
    DL=get_mnist()
    t = tests().cuda()
    optimizer = torch.optim.Adam(t.parameters(), lr = 2e-3)

    for i in range(100):
        for inputs, classes in DL:
            inputs = inputs.cuda()
            classes = classes.cuda().float()
            output = t(inputs)
            loss = torch.nn.functional.mse_loss(output.view(-1), classes)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
#         print("Epoch:", i, "pi_grad", t.pi_.grad)
        print("Epoch:", i,"pi:",t.pi_)

Now, pi_ changes every single epoch.

output >>>
Epoch: 0 pi: Parameter containing:
tensor([[1.3429],
        [1.0644],
        [0.9817],
        [0.9767],
        [0.9715],
        [1.1110],
        [1.1139],
        [0.9759],
        [1.2424],
        [1.2632]], device='cuda:0', requires_grad=True)
Epoch: 1 pi: Parameter containing:
tensor([[1.4413],
        [1.1977],
        [0.9588],
        [1.0325],
        [0.9241],
        [1.1988],
        [1.1690],
        [0.9248],
        [1.2892],
        [1.3427]], device='cuda:0', requires_grad=True)
Epoch: 2 pi: Parameter containing:
tensor([[1.4653],
        [1.2351],
        [0.9539],
        [1.1588],
        [0.8670],
        [1.2739],
        [1.2058],
        [0.8648],
        [1.2848],
        [1.3891]], device='cuda:0', requires_grad=True)
Epoch: 3 pi: Parameter containing:
tensor([[1.4375],
        [1.2256],
        [0.9580],
        [1.2293],
        [0.8174],
        [1.3471],
        [1.2035],
        [0.8102],
        [1.2505],
        [1.4201]], device='cuda:0', requires_grad=True)


Instruction: PyTorch loss decreases even if requires_grad = False for all variables

Input:
When I create a neural network with PyTorch, using the torch.nn.Sequential method for defining layers, it seems that the parameters have requires_grad = False by default. However, when I train this network, the loss decreases. How is this possible if the layers are not being updated via gradients?

For example, this is the code for defining my network:

class Network(torch.nn.Module):

def __init__(self):
    super(Network, self).__init__()
    self.layers = torch.nn.Sequential(
        torch.nn.Linear(10, 5),
        torch.nn.Linear(5, 2)
    )
    print('Network Parameters:')
    model_dict = self.state_dict()
    for param_name in model_dict:
        param = model_dict[param_name]
        print('Name: ' + str(param_name))
        print('\tRequires Grad: ' + str(param.requires_grad))

def forward(self, input):
    prediction = self.layers(input)
    return prediction

And this prints out:

Network Parameters:
Name: layers.0.weight
    Requires Grad: False
Name: layers.0.bias
    Requires Grad: False
Name: layers.1.weight
    Requires Grad: False
Name: layers.1.bias
    Requires Grad: False

Then this is the code to train my network:

network = Network()
network.train()
optimiser = torch.optim.SGD(network.parameters(), lr=0.001)
criterion = torch.nn.MSELoss()
inputs = np.random.random([100, 10]).astype(np.float32)
inputs = torch.from_numpy(inputs)
labels = np.random.random([100, 2]).astype(np.float32)
labels = torch.from_numpy(labels)

while True:
    prediction = network.forward(inputs)
    loss = criterion(prediction, labels)
    print('loss = ' + str(loss.item()))
    optimiser.zero_grad()
    loss.backward()
    optimiser.step()

And this prints out:

loss = 0.284633219242
loss = 0.278225809336
loss = 0.271959483624
loss = 0.265835255384
loss = 0.259853869677
loss = 0.254015892744
loss = 0.248321473598
loss = 0.242770522833
loss = 0.237362638116
loss = 0.232097044587
loss = 0.226972639561
loss = 0.221987977624
loss = 0.217141270638
loss = 0.212430402637
loss = 0.207852959633
loss = 0.203406244516
loss = 0.199087426066
loss = 0.19489350915
loss = 0.190821439028
loss = 0.186868071556
loss = 0.183030322194
loss = 0.179305106401
loss = 0.175689414144
loss = 0.172180294991
loss = 0.168774917722
loss = 0.165470585227
loss = 0.162264674902
loss = 0.159154698253

Why is the loss decreasing if all of the parameters have requires_grad = False?

Output:
This is interesting -- there seems to be a difference between state_dict() and parameters():

class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.layers = torch.nn.Sequential(
            torch.nn.Linear(10, 5),
            torch.nn.Linear(5, 2)
        )
        print(self.layers[0].weight.requires_grad) # True
        print(self.state_dict()['layers.0.weight'].requires_grad) # False
        print(list(self.parameters())[0].requires_grad) # True

    def forward(self, input):
        prediction = self.layers(input)
        return prediction

So it appears your loss is decreasing because the net is in fact learning, because requires_grad is True. (In general for debugging, I prefer querying the actual objects (self.layers[0]...).

[EDIT] Ahah - found the problem: there's a keep_vars boolean option that you can pass into state_dict that does the following (among other things): (https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L665)

for name, param in self._parameters.items():
    if param is not None:
        destination[prefix + name] = param if keep_vars else param.data

so, if you want the actual param, use keep_vars=True -- if you want just the data, use the default keep_vars=False.

So:

print(self.layers[0].weight.requires_grad) # True
print(self.state_dict(keep_vars=True)['layers.0.weight'].requires_grad) # True
print(list(self.parameters())[0].requires_grad) # True


Instruction: How to handle entrypoints nested in folders with amazon sagemaker pytorch estimator?

Input:
I am attempting to run a training job on amazon sagemaker using the python-sagemaker-sdk, estimator class.

I have the following

estimator = PyTorch(entry_point='training_scripts/train_MSCOCO.py',
                            source_dir='./',
                            role=#dummy_role,
                            train_instance_type='ml.p3.2xlarge',
                            train_instance_count=1,
                            framework_version='1.0.0',
                            output_path=#dummy_output_path,
                            hyperparameters={'lr': 0.001,
                                             'batch_size': 32,
                                             'num_workers': 4,
                                             'description': description})

role and output_path hidden for privacy.

I get the following error, "No module named training_scripts\train_MSCOCO".

When I run python -m training_scripts.train_MSCOCO the script runs fine. However when I pass entry_point='training_script.train_MSCOCO.py it will not run as, "No file named "training_scripts.train_MSCOCO.py" was found in directory "./"".

I am confused as to how to run a nested training script from the top level of my repository within AWS sagemaker, as they seem to have conflicting path needs, one in python module dot notation, the other in standard filepath slash notation.

Output:
Either one of these will work:

estimator = PyTorch(entry_point='training_scripts/train_MSCOCO.py',
                    role=#dummy_role,
                    ...

estimator = PyTorch(entry_point='train_MSCOCO.py',
                    source_dir='training_scripts',
                    role=#dummy_role,
                    ...


Instruction: Non-reproducible results in pytorch after saving and loading the model

Input:
I am unable to reproduce my results in PyTorch after saving and loading the model whereas the in-memory model works as expected. Just for context, I am seeding my libraries, using model.eval to turn off the dropouts but still, results are not reproducible. Any suggestions if I am missing something.  Thanks in advance.

Libraries that I am using:-

import matplotlib.pyplot as plt
import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import datasets, transforms
import numpy as np
import random

Libraries that I am seeding

manualSeed = 1
np.random.seed(manualSeed)
random.seed(manualSeed)
torch.manual_seed(manualSeed)
random.seed(manualSeed)

Below are the results on in memory and loaded model.

In Memory model Loss : 1.596395881312668, In Memory model Accuracy : tensor(0.3989)
Loaded model Loss : 1.597083057567572, Loaded model Accuracy : tensor(0.3983)

Output:
Since the date that Szymon Maszke posted his response above (2019), a new API has been added, torch.use_deterministic_algorithms().
This new function does everything that torch.backends.cudnn.deterministic did (namely, makes CuDNN convolution operations deterministic), plus much more (makes every known normally-nondeterministic function either deterministic or throw an error if a deterministic implementation is not available). CuDNN convolution is only one of the many possible sources of nondeterminism in PyTorch, so torch.use_deterministic_algorithms() should now be used instead of the old torch.backends.cudnn.deterministic.
The link to the reproducibility documentation is still relevant. However, note that this page has been changed a fair bit since 2019.


Instruction: What is pixel-wise softmax loss?

Input: what is the pixel-wise softmax loss? In my understanding, it's just a cross-entropy loss, but I didn't find the formula. Can someone help me? It's better to have the pytorch code.

Output:
You can read here all about it (there's also a link to source code there).

As you already observed the "softmax loss" is basically a cross entropy loss which computation combines the softmax function and the loss for numerical stability and efficiency.
In your example, the loss is computed for a pixel-wise prediction so you have a per-pixel prediction, a per-pixel target and a per-pixel loss term.


Instruction: How to implement my own ResNet with torch.nn.Sequential in Pytorch?

Input:
I want to implement a ResNet network (or rather, residual blocks) but I really want it to be in the sequential network form.
What I mean by sequential network form is the following:
## mdl5, from cifar10 tutorial
mdl5 = nn.Sequential(OrderedDict([
    ('pool1', nn.MaxPool2d(2, 2)),
    ('relu1', nn.ReLU()),
    ('conv1', nn.Conv2d(3, 6, 5)),
    ('pool1', nn.MaxPool2d(2, 2)),
    ('relu2', nn.ReLU()),
    ('conv2', nn.Conv2d(6, 16, 5)),
    ('relu2', nn.ReLU()),
    ('Flatten', Flatten()),
    ('fc1', nn.Linear(1024, 120)), # figure out equation properly
    ('relu4', nn.ReLU()),
    ('fc2', nn.Linear(120, 84)),
    ('relu5', nn.ReLU()),
    ('fc3', nn.Linear(84, 10))
]))

but of course with the NN lego blocks being “ResNet”.
I know the equation is something like:

but I am not sure how to do it in Pytorch AND Sequential. Sequential is key for me!

Bounty:
I'd like to see an example with a fully connected net and where the BN layers would have to go (and the drop out layers would go too). Ideally on a toy example/data if possible.

Cross-posted:

https://discuss.pytorch.org/t/how-to-have-residual-network-using-only-sequential-blocks/51541
https://www.quora.com/unanswered/How-does-one-implement-my-own-ResNet-with-torch-nn-Sequential-in-Pytorch
https://www.reddit.com/r/pytorch/comments/uyyr28/how_to_implement_my_own_resnet_with/

Output:
You can't do it solely using torch.nn.Sequential as it requires operations to go, as the name suggests, sequentially, while yours are parallel.
You could, in principle, construct your own block really easily like this:
import torch

class ResNet(torch.nn.Module):
    def __init__(self, module):
        super().__init__()
        self.module = module

    def forward(self, inputs):
        return self.module(inputs) + inputs

Which one can use something like this:
model = torch.nn.Sequential(
    torch.nn.Conv2d(3, 32, kernel_size=7),
    # 32 filters in and out, no max pooling so the shapes can be added
    ResNet(
        torch.nn.Sequential(
            torch.nn.Conv2d(32, 32, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.BatchNorm2d(32),
            torch.nn.Conv2d(32, 32, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.BatchNorm2d(32),
        )
    ),
    # Another ResNet block, you could make more of them
    # Downsampling using maxpool and others could be done in between etc. etc.
    ResNet(
        torch.nn.Sequential(
            torch.nn.Conv2d(32, 32, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.BatchNorm2d(32),
            torch.nn.Conv2d(32, 32, kernel_size=3),
            torch.nn.ReLU(),
            torch.nn.BatchNorm2d(32),
        )
    ),
    # Pool all the 32 filters to 1, you may need to use `torch.squeeze after this layer`
    torch.nn.AdaptiveAvgPool2d(1),
    # 32 10 classes
    torch.nn.Linear(32, 10),
)

Fact being usually overlooked (without real consequences when it comes to shallowe networks) is that skip connection should be left without any nonlinearities like ReLU or convolutional layers and that's what you can see above (source: Identity Mappings in Deep Residual Networks).


Instruction: How is it that torch is not installed by torchvision?

Input:
Somehow when I do the install it installs torchvision but not torch. Command I am running as dictated from the main website:

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

then I do conda list but look:

$ conda list
# packages in environment at /home/ubuntu/anaconda3/envs/pytorch_p36:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.10           py36h306e16b_0
anaconda-client           1.6.14                   py36_0
anaconda-project          0.8.2            py36h44fb852_0
argparse                  1.4.0                     <pip>
asn1crypto                0.24.0                   py36_0
astroid                   1.6.3                    py36_0
astropy                   3.0.2            py36h3010b51_1
attrs                     18.1.0                   py36_0
autovizwidget             0.12.7                    <pip>
babel                     2.5.3                    py36_0
backcall                  0.1.0                    py36_0
backports                 1.0              py36hfa02d7e_1
backports.shutil_get_terminal_size 1.0.0            py36hfea85ff_2
bcrypt                    3.1.6                     <pip>
beautifulsoup4            4.6.0            py36h49b8c8c_1
bitarray                  0.8.1            py36h14c3975_1
bkcharts                  0.2              py36h735825a_0
blas                      1.0                         mkl
blaze                     0.11.3           py36h4e06776_0
bleach                    2.1.3                    py36_0
blosc                     1.14.3               hdbcaa40_0
bokeh                     1.0.4                    py36_0
boto                      2.48.0           py36h6e4cd66_1
boto3                     1.9.146                   <pip>
boto3                     1.9.134                    py_0
botocore                  1.12.146                  <pip>
botocore                  1.12.134                   py_0
bottleneck                1.2.1            py36haac1ea0_0
bzip2                     1.0.6                h14c3975_5
ca-certificates           2019.1.23                     0
cached-property           1.5.1                     <pip>
cairo                     1.14.12              h8948797_3
certifi                   2019.3.9                 py36_0
cffi                      1.11.5           py36h9745a5d_0
chardet                   3.0.4            py36h0f667ec_1
click                     6.7              py36h5253387_0
cloudpickle               0.5.3                    py36_0
clyent                    1.2.2            py36h7e57e65_1
colorama                  0.3.9            py36h489cec4_0
contextlib2               0.5.5            py36h6c84a62_0
cryptography              2.3.1            py36hc365091_0
cudatoolkit               10.0.130                      0
curl                      7.60.0               h84994c4_0
cycler                    0.10.0           py36h93f1223_0
cymem                     2.0.2            py36hfd86e86_0
cython                    0.28.2           py36h14c3975_0
cytoolz                   0.9.0.1          py36h14c3975_0
dask                      0.17.5                   py36_0
dask-core                 0.17.5                   py36_0
dataclasses               0.6                        py_0    fastai
datashape                 0.5.4            py36h3ad6b5c_0
dbus                      1.13.2               h714fa37_1
decorator                 4.3.0                    py36_0
defusedxml                0.6.0                      py_0
dill                      0.2.9                    py36_0
distributed               1.21.8                   py36_0
docker                    3.7.2                     <pip>
docker-compose            1.24.0                    <pip>
docker-pycreds            0.4.0                     <pip>
dockerpty                 0.4.1                     <pip>
docopt                    0.6.2                     <pip>
docutils                  0.14             py36hb0f60f5_0
entrypoints               0.2.3            py36h1aec115_2
environment-kernels       1.1.1                     <pip>
et_xmlfile                1.0.1            py36hd6bccc3_0
expat                     2.2.5                he0dffb1_0
fastai                    1.0.52                        1    fastai
fastcache                 1.0.2            py36h14c3975_2
fastprogress              0.1.21                     py_0    fastai
filelock                  3.0.4                    py36_0
flask                     1.0.2                    py36_1
flask-cors                3.0.4                    py36_0
fontconfig                2.13.0               h9420a91_0
freetype                  2.9.1                h8a8886c_1
fribidi                   1.0.5                h7b6447c_0
get_terminal_size         1.0.0                haa9412d_0
gevent                    1.3.0            py36h14c3975_0
glib                      2.56.1               h000015b_0
glob2                     0.6              py36he249c77_0
gmp                       6.1.2                h6c8ec71_1
gmpy2                     2.0.8            py36hc8893dd_2
graphite2                 1.3.11               h16798f4_2
graphviz                  2.40.1               h21bd128_2
greenlet                  0.4.13           py36h14c3975_0
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb453b48_1
h5py                      2.8.0            py36h989c5e5_3
harfbuzz                  1.8.4                hec2c2bc_0
hdf5                      1.10.2               hba1933b_1
hdijupyterutils           0.12.7                    <pip>
heapdict                  1.0.0                    py36_2
html5lib                  1.0.1            py36h2f9c1c0_0
icu                       58.2                 h9c2bf20_1
idna                      2.6              py36h82fb2a8_1
imageio                   2.3.0                    py36_0
imagesize                 1.0.0                    py36_0
intel-openmp              2018.0.0                      8
ipykernel                 4.8.2                    py36_0
ipyparallel               6.2.2                     <pip>
ipython                   6.4.0                    py36_0
ipython_genutils          0.2.0            py36hb52b0d5_0
ipywidgets                7.2.1                    py36_0
ipywidgets                7.4.0                     <pip>
isort                     4.3.4                    py36_0
itsdangerous              0.24             py36h93cc618_1
jbig                      2.1                  hdba287a_0
jdcal                     1.4                      py36_0
jedi                      0.12.0                   py36_1
jinja2                    2.10             py36ha16c418_0
jmespath                  0.9.4                      py_0
jpeg                      9b                   h024ee3a_2
jsonschema                2.6.0            py36h006f8b5_0
jupyter                   1.0.0                    py36_4
jupyter_client            5.2.3                    py36_0
jupyter_console           5.2.0            py36he59e554_1
jupyter_core              4.4.0            py36h7c827e3_0
jupyterlab                0.32.1                   py36_0
jupyterlab_launcher       0.10.5                   py36_0
kiwisolver                1.0.1            py36h764f252_0
krb5                      1.14.2               hcdc1b81_6
lazy-object-proxy         1.3.1            py36h10fcdad_0
libcurl                   7.60.0               h1ad7b7a_0
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3
libpng                    1.6.37               hbc83047_0
libprotobuf               3.5.2                hd28b015_1    conda-forge
libsodium                 1.0.16               h1bed415_0
libssh2                   1.8.0                h9cfc8f7_4
libstdcxx-ng              8.2.0                hdf63c60_1
libtiff                   4.0.9                he85c1e1_1
libtool                   2.4.6                h544aabb_3
libuuid                   1.0.3                h1bed415_2
libxcb                    1.13                 h1bed415_1
libxml2                   2.9.8                h26e45fe_1
libxslt                   1.1.32               h1312cb7_0
llvmlite                  0.23.1           py36hdbcaa40_0
locket                    0.2.0            py36h787c0ad_1
lxml                      4.2.1            py36h23eabaa_0
lzo                       2.10                 h49e0be7_2
markupsafe                1.0              py36hd9260cd_1
matplotlib                2.2.2                     <pip>
matplotlib                3.0.3            py36h5429711_0
mccabe                    0.6.1            py36h5ad9710_1
mistune                   0.8.3            py36h14c3975_1
mkl                       2018.0.3                      1
mkl-service               1.1.2            py36h17a0993_4
mkl_fft                   1.0.6            py36h7dd41cf_0
mkl_random                1.0.1            py36h629b387_0
mock                      3.0.5                     <pip>
more-itertools            4.1.0                    py36_0
mpc                       1.0.3                hec55b23_5
mpfr                      3.1.5                h11a74b3_2
mpi                       1.0                     openmpi    conda-forge
mpmath                    1.0.0            py36hfeacd6b_2
msgpack                   0.6.0                     <pip>
msgpack-numpy             0.4.3.2                  py36_0
msgpack-python            0.5.6            py36h6bb024c_0
multipledispatch          0.5.0                    py36_0
murmurhash                1.0.2            py36he6710b0_0
nb_conda                  2.2.1                    py36_2    conda-forge
nb_conda_kernels          2.2.1                    py36_0    conda-forge
nbconvert                 5.4.1                    py36_3
nbformat                  4.4.0            py36h31c9010_0
ncurses                   6.1                  hf484d3e_0
networkx                  2.1                      py36_0
ninja                     1.8.2            py36h6bb024c_1
nltk                      3.3.0                    py36_0
nose                      1.3.7            py36hcdf7029_2
notebook                  5.5.0                    py36_0
numba                     0.38.0           py36h637b7d7_0
numexpr                   2.6.5            py36h7bf3b9c_0
numpy                     1.15.4           py36h1d66e8a_0
numpy                     1.15.4                    <pip>
numpy-base                1.15.4           py36h81de0dd_0
numpydoc                  0.8.0                    py36_0
nvidia-ml-py3             7.352.0                    py_0    fastai
odo                       0.5.1            py36h90ed295_0
olefile                   0.45.1                   py36_0
onnx                      1.4.1                     <pip>
openmpi                   3.1.0                h26a2512_3    conda-forge
openpyxl                  2.5.3                    py36_0
openssl                   1.0.2r               h7b6447c_0
packaging                 17.1                     py36_0
pandas                    0.24.2                    <pip>
pandas                    0.23.0           py36h637b7d7_0
pandoc                    1.19.2.1             hea2e7c5_1
pandocfilters             1.4.2            py36ha6701b7_1
pango                     1.42.3               h8589676_0
paramiko                  2.4.2                     <pip>
parso                     0.2.0                    py36_0
partd                     0.3.8            py36h36fd896_0
patchelf                  0.9                  hf79760b_2
path.py                   11.0.1                   py36_0
pathlib2                  2.3.2                    py36_0
patsy                     0.5.0                    py36_0
pcre                      8.42                 h439df22_0
pep8                      1.7.1                    py36_0
pexpect                   4.5.0                    py36_0
pickleshare               0.7.4            py36h63277f8_0
pillow                    5.2.0            py36heded4f4_0
pip                       10.0.1                   py36_0
pixman                    0.34.0               hceecf20_3
pkginfo                   1.4.2                    py36_1
plac                      0.9.6                    py36_0
plotly                    2.7.0                     <pip>
pluggy                    0.6.0            py36hb689045_0
ply                       3.11                     py36_0
preshed                   2.0.1            py36he6710b0_0
prompt_toolkit            1.0.15           py36h17d85b1_0
protobuf                  3.5.2            py36hd28b015_0    conda-forge
protobuf3-to-dict         0.1.5                     <pip>
psutil                    5.4.5            py36h14c3975_0
psycopg2                  2.7.5                     <pip>
ptyprocess                0.5.2            py36h69acd42_0
py                        1.5.3                    py36_0
py4j                      0.10.7                    <pip>
pyasn1                    0.4.5                     <pip>
pycodestyle               2.4.0                    py36_0
pycosat                   0.6.3            py36h0a5515d_0
pycparser                 2.18             py36hf9f622e_1
pycrypto                  2.6.1            py36h14c3975_8
pycurl                    7.43.0.1         py36hb7f436b_0
pyflakes                  1.6.0            py36h7bd6a15_0
pygal                     2.4.0                     <pip>
pygments                  2.2.0            py36h0d3125c_0
pykerberos                1.2.1            py36h14c3975_0
pylint                    1.8.4                    py36_0
PyNaCl                    1.3.0                     <pip>
pyodbc                    4.0.23           py36hf484d3e_0
pyopenssl                 18.0.0                   py36_0
pyparsing                 2.2.0            py36hee85983_1
pyqt                      5.9.2            py36h751905a_0
pysocks                   1.6.8                    py36_0
pyspark                   2.3.2                     <pip>
pytables                  3.4.3            py36h02b9ad4_2
pytest                    3.5.1                    py36_0
pytest-arraydiff          0.2                      py36_0
pytest-astropy            0.3.0                    py36_0
pytest-doctestplus        0.1.3                    py36_0
pytest-openfiles          0.3.0                    py36_0
pytest-remotedata         0.2.1                    py36_0
python                    3.6.5                hc3d631a_2
python-dateutil           2.7.3                    py36_0
pytorch                   1.1.0           py3.6_cuda10.0.130_cudnn7.5.1_0    pytorch
pytz                      2018.4                   py36_0
pywavelets                0.5.2            py36he602eb0_0
pyyaml                    3.12             py36hafb9ca4_1
pyzmq                     17.0.0           py36h14c3975_0
qt                        5.9.6                h52aff34_0
qtawesome                 0.4.4            py36h609ed8c_0
qtconsole                 4.3.1            py36h8f73b5b_0
qtpy                      1.4.1                    py36_0
readline                  7.0                  ha6073c6_4
regex                     2018.01.10      py36h14c3975_1000    fastai
requests                  2.20.0                py36_1000    conda-forge
requests-kerberos         0.12.0                    <pip>
rope                      0.10.7           py36h147e2ec_0
ruamel_yaml               0.15.35          py36h14c3975_1
s3fs                      0.1.5                    py36_0
s3transfer                0.2.0                     <pip>
s3transfer                0.2.0                    py36_0
sagemaker                 1.20.1                    <pip>
sagemaker-pyspark         1.2.4                     <pip>
scikit-image              0.13.1           py36h14c3975_1
scikit-learn              0.19.1           py36h7aa7ec6_0
scikit-learn              0.20.3                    <pip>
scipy                     1.1.0            py36hfc37229_0
seaborn                   0.8.1            py36hfad7ec4_0
send2trash                1.5.0                    py36_0
setuptools                39.1.0                   py36_0
simplegeneric             0.8.1                    py36_2
singledispatch            3.4.0.3          py36h7a266c3_0
sip                       4.19.8           py36hf484d3e_0
six                       1.11.0           py36h372c433_1
snappy                    1.1.7                hbae5bb6_3
snowballstemmer           1.2.1            py36h6febd40_0
sortedcollections         0.6.1                    py36_0
sortedcontainers          1.5.10                   py36_0
spacy                     2.0.18          py36hf484d3e_1000    fastai
sparkmagic                0.12.5                    <pip>
sphinx                    1.7.4                    py36_0
sphinxcontrib             1.0              py36h6d0f590_1
sphinxcontrib-websupport  1.0.1            py36hb5cb234_1
spyder                    3.2.8                    py36_0
SQLAlchemy                1.2.11                    <pip>
sqlalchemy                1.2.7            py36h6b74fdf_0
sqlite                    3.23.1               he433501_0
statsmodels               0.9.0            py36h3010b51_0
sympy                     1.1.1            py36hc6d1c1c_0
tblib                     1.3.2            py36h34cf8b6_0
terminado                 0.8.1                    py36_1
testpath                  0.3.1            py36h8cadb63_0
texttable                 0.9.1                     <pip>
thinc                     6.12.1          py36h637b7d7_1000    fastai
tk                        8.6.8                hbc83047_0
toolz                     0.9.0                    py36_0
torchvision               0.2.2                      py_3    pytorch
tornado                   5.0.2                    py36_0
tqdm                      4.31.1                   py36_1
traitlets                 4.3.2            py36h674d592_0
typing                    3.6.4                    py36_0
typing-extensions         3.7.2                     <pip>
ujson                     1.35             py36h14c3975_0
unicodecsv                0.14.1           py36ha668878_0
unixodbc                  2.3.6                h1bed415_0
urllib3                   1.23                     py36_0
wcwidth                   0.1.7            py36hdf4376a_0
webencodings              0.5.1            py36h800622e_1
websocket-client          0.56.0                    <pip>
werkzeug                  0.14.1                   py36_0
wheel                     0.31.1                   py36_0
widgetsnbextension        3.2.1                    py36_0
widgetsnbextension        3.4.2                     <pip>
wrapt                     1.10.11          py36h28b7045_0
xlrd                      1.1.0            py36h1db9f0c_1
xlsxwriter                1.0.4                    py36_0
xlwt                      1.3.0            py36h7b00a1f_0
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zeromq                    4.2.5                h439df22_0
zict                      0.1.3            py36h3a3bf81_0
zlib                      1.2.11               ha838bed_2

I was told I do have pytorch installed but my script keeps giving me this error:

$ cat nohup.out
Traceback (most recent call last):
  File "high_performing_data_point_models_cifar10.py", line 5, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

does that mean that I need to install it as pytroch and not torch? Is this not weird?

Note I am running this on an AWS instance p3.2xlarge. This keeps happening when I log out and then go back in that my torch package gets missing...?!?! :/

original post: https://discuss.pytorch.org/t/torchvision-installed-but-not-torch/51758

The issue persists even if I open just a python interactive and try to import it:

(pytorch_p36) ubuntu@ip-123-12-21-123:~$ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>

It also happens running the script directly:

(pytorch_p36) ubuntu@ip-123-12-21-123:~/project/folder$ python high_performing_data_point_models_cifar10.py
Traceback (most recent call last):
  File "high_performing_data_point_models_cifar10.py", line 5, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

I can't import torchvision either!

$ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchvision
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torchvision'
>>>

Output:
Your conda list command shows that it was run from the environment called automl:

# packages in environment at /home/ubuntu/anaconda3/envs/automl:

However, when you show the commands that you are trying to run, you are doing so from the (pytorch_p36) environment.

You should run your conda install command while inside this pytorch_p36 environment.


Instruction: What is the difference between .flatten() and .view(-1) in PyTorch?

Input:
Both .flatten() and .view(-1) flatten a tensor in PyTorch. What's the difference?

Does .flatten() copy the data of the tensor?
Is .view(-1) faster?
Is there any situation that .flatten() doesn't work?

Output:
In addition to @adeelh's comment, there is another difference: torch.flatten() results in a .reshape(), and the differences between .reshape() and .view() are:

[...] torch.reshape may return a copy or a view of the original tensor. You can not count on that to return a view or a copy.

Another difference is that reshape() can operate on both contiguous and non-contiguous tensor while view() can only operate on contiguous tensor. Also see here about the meaning of contiguous

For context:

The community requested for a flatten function for a while, and after Issue #7743, the feature was implemented in the PR #8578.

You can see the implementation of flatten here, where a call to .reshape() can be seen in return line.


Instruction: saving and loading RNN hidden states in PyTorch

Input:
I am trying to use an RNN network in PyTorch for regression task. In the training phase the model is learned. I want to use the trained model in testing phase. For this purpose I have saved the learned model by:

torch.save(learned_model, "model_path")

Then I can load the model again by:

loaded_model = torch.load("model_path")

For testing phase I must use this loaded model but I want to know what is the value of the first hidden state of the model? I can initialize the first hidden state by zero but I think maybe this is not correct. Is there any function other than torch.save which can return the last hidden state in the learned mode? Then I can restore that hidden state and use it as the first hidden state in the loaded model for testing phase.

Thanks in advance.

Output:
Your question is a bit unclear. As far as I understand you want to know the weights of the last hidden layer in the trained model, i.e. loaded_model. In that case, you can simply use model's state_dict, which is basically a python dictionary object that maps each layer to its parameter tensor. Read more about it from here.

for param in loaded_model.state_dict():
    print(param)

Sample output:

rnn.weight_ih_l0
rnn.weight_hh_l0
rnn.bias_ih_l0
rnn.bias_hh_l0
out.weight
out.bias

After that, you can get the weights of the last hidden layer using below code:

out_weights, out_bias = loaded_model.state_dict()['out.weight'], loaded_model.state_dict()['out.bias']


Instruction: Pytorch dataloader, too many threads, too much cpu memory allocation

Input:
I'm training a model using PyTorch. To load the data, I'm using torch.utils.data.DataLoader. The data loader is using a custom database I've implemented. A strange problem has occurred, every time the second for in the following code executes, the number of threads/processes increases and a huge amount of memory is allocated

    for epoch in range(start_epoch, opt.niter + opt.niter_decay + 1):
        epoch_start_time = time.time()
        if epoch != start_epoch:
            epoch_iter = epoch_iter % dataset_size
        for i, item in tqdm(enumerate(dataset, start=epoch_iter)):

I suspect the threads and memories of the previous iterators are not released after each __iter__() call to the data loader.
The allocated memory is close to the amount of memory allocated by the main thread/process when the threads are created. That is in the initial epoch the main thread is using 2GB of memory and so 2 threads of size 2GB are created. In the next epochs, 5GB of memory is allocated by the main thread and two 5GB threads are constructed (num_workers is 2).
I suspect that fork() function copies most of the context to the new threads.

The following is the Activity monitor showing the processes created by python, ZMQbg/1 are processes related to python.

My dataset used by the data loader has 100 sub-datasets, the __getitem__ call randomly selects one (ignoring the index). (the sub-datasets are AlignedDataset from pix2pixHD GitHub repository):

Output:
torch.utils.data.DataLoader prefetch 2*num_workers, so that you will always have data ready to send to the GPU/CPU, this could be the reason you see the memory increase

https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html


Instruction: How can I use a PyTorch DataLoader for Reinforcement Learning?

Input:
I'm trying to set up a generalized Reinforcement Learning framework in PyTorch to take advantage of all the high-level utilities out there which leverage PyTorch DataSet and DataLoader, like Ignite or FastAI, but I've hit a blocker with the dynamic nature of Reinforcement Learning data:

Data Items are generated from code, not read from a file, and they are dependent on previous actions and model results, therefore each nextItem call needs access to the model state.
Training episodes are not fixed length so I need a dynamic batch size as well as a dynamic total data set size. My preference would be to use a terminating condition function instead of a number. I could "possibly" do this with padding, as in NLP sentence processing, but that's a real hack.

My Google and StackOverflow searches so far have yielded zilch. Anyone here know of existing solutions or workarounds to using DataLoader or DataSet with Reinforcement Learning? I hate to loose access to all the existing libraries out there which depend on those.

Output:
Here is one PyTorch-based framework and here is something from Facebook.

When it comes to your question (and noble quest, no doubt):

You could easily create a torch.utils.data.Dataset dependent on anything, including the model, something like this (pardon weak abstraction, it's just to prove a point):

import typing

import torch
from torch.utils.data import Dataset

class Environment(Dataset):
    def __init__(self, initial_state, actor: torch.nn.Module, max_interactions: int):
        self.current_state = initial_state
        self.actor: torch.nn.Module = actor
        self.max_interactions: int = max_interactions

    # Just ignore the index
    def __getitem__(self, _):
        self.current_state = self.actor.update(self.current_state)
        return self.current_state.get_data()

    def __len__(self):
        return self.max_interactions

Assuming, torch.nn.Module-like network has some kind of update changing state of the environment. All in all it's just a Python structure and so you could model a lot of things with it.

You could specify max_interactions to be almost infinite or you could change it on the fly if needed with some callbacks during training (as __len__ will be called multiple times throughout the code probably). Environment could furthermore provide batches instead of samples.

torch.utils.data.DataLoader has batch_sampler argument, there you could generate batches of varying length. As the network is not dependent on the first dimension, you could return any batch size you want from there as well.

BTW. Padding should be used if each sample would be of different length, varying batch size has nothing to do with that.


Instruction: How to Build Simple LSTM Model for Anomally Detection

Input:
I want to create a LSTM model in PyTorch that will be used for anomally detection, but I'm having trouble understanding the details in doing so.

Note, my training-data consists of sets with 16 features in 80 time-steps. Here is what I've written for the model below:

class AutoEncoder(torch.nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim):
        super(AutoEncoder, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim

        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc1 = torch.nn.Linear(hidden_dim, hidden_dim)
        self.fc2 = torch.nn.Linear(hidden_dim, input_dim)
    def forward(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out = self.fc1(out[:, -1, :])
        out = self.fc2(out)
        return out

input_dim = 16
hidden_dim = 8
layer_dim = 2
model = AutoEncoder(input_dim, hidden_dim, layer_dim)

I don't think I've built the model correctly. How does it know that I'm feeding it 80 time-steps of data? How will the auto-encoder reconstruct those 80 time-steps of data?

I'm having a hard time understanding the material online. What would the final layer have to be?

Output:
If you check out the PyTorch LSTM documentation, you will see that the LSTM equations are applied to each timestep in your sequence. nn.LSTM will internally obtain the seq_len dimension and optimize from there, so you do not need to provide the number of time steps.

At the moment, the line

out = self.fc1(out[:, -1, :])

is selecting the final hidden state (corresponding to time step 80) and then this is being projected onto a space of size input_dim.

To output a sequence of length 80, you should have an output for each hidden state. All hidden states are stacked in out so you can simply use

out = self.fc1(out)
out = self.fc2(out)

I would also note that if you must have two fully connected layers after encoding in your hidden state, you should use a non-linearity in between otherwise this is the equivalent of just one layer but with more parameters.