Instruction: SSH into Docker? or docker on SSH? and I need command

Input:
I'm new to DL, and docker and even not familiar with Linux and internet things (SSH and port.. DNS things.. part of them are only existing in my mind). Thus, I'd be so much happy with "specific explanation + command" (or reference sites).

My basic questions are:

what is superior concept between Docker and SSH? (running SSH on the Docker ? or running Docker on SSH? or both are possible?)
Which specific command should I use if I want to use SSH+Docker+Pytorch+Jupyternotebook+visdom ?

2-1) I run SSH first (which is my lab's server, thus I am not usually root user, so if I want to run python file here, I face with permission denied often), let say SSH address is 123.456.789.999

2-2) use docker after running ssh (however, what I found from many posts are about running docker FIRST and then access to SSH.. how's it different?)

2-2-1) so for this, I now I have to pull a image which includes pytorch, jupyternotebook. I've done it

2-2-2) I need to RUN DOCKER using image with proper COMMAND LINE. What makes me confused is here.

$docker run -it --[name] -p 8888:8888 [docker_image_with_pytorch]

this is what I found. I assume to use jupyter notebook (let's say if I want to use 4444 instead of 8888, and visdom as 5555 instead of 8097) then I need to map port from host to docker for two times and is it right?

$docker run -it --[name] -p 4444:8888 -p 5555:8097 [docker_image_with_pytorch]

Finally I need to link SSH(let's say SSH port num:22 as general, ip: 123.456.789.999, id=heyjude) for SSH, I also found command below.

$ ssh -L <host port>:localhost:<remote port> user@remote

But is it general to use command after running docker instead of adding option when I first run docker? also if I assume to use that command for setting SSH, i'm confused which things I need to input  ;(
(host port=22 ? for SSH? localhost is just literal expression? remote port is arbitrary thing?) Below is my assumption..

$ ssh -L <22>:localhost:<12345> heyjude@123.456.789.999

I know it's so messy and you may catch how my thinking is twisted..
it would be very helpful for me to explain from the scratch..

Thank you.

Output:
Your question is somewhat unclear. I'm going to take a guess at what you're trying to solve.
Assuming (!) that you've a container image that includes PyTorch and Jupyter and all their dependencies, it's likely that Jupyter will serve content to you via a web server (via HTTP and I suspect) over port :8888.
If you docker run -it ... which is equivalent to docker run --interactive --tty ..., you should see log output from the process(es) running in the container. These logs should contain relevant information.
To access the Jupyter Notebook once the container is running on your location workstation, you should be able to just browse http://localhost:8888.
You probably do not need to use SSH if you're running everything locally. If you are running e.g. the docker container, on a remote host, you may SSH into the remote host first, run the commands e.g. docker run... but you may alternatively simply configure your Docker client to access the remote Docker Engine.
Somewhat akin to SSH, when using Docker containers, you may execute commands in the container. But, you are able to do this using docker exec ....; you don't need to use SSH to interact with containers.
A container image has one or more statically defined ports that the container will use to expose its services (over TCP|UDP). When you run the container, you may map the container ports to different ports on your host. This may be for necessity (if the container port is already being used on your host) or just for convenience. To do this you use --publish=[HOST-PORT]:[CONTAINER-PORT]. For a given container image, you cannot change the [CONTAINER-PORT] but you may use any available [HOST-PORT]. In your example --publish=4444:8888 would mean that the Jupyter (?) service is now accessible on your local machine via localhost:4444. Docker port forwards traffic from your host's :4444 to the container's :8888.


Instruction: Withou onnx, how to convert a pytorch model into a tensorflow model manually?

Input:
Since ONNX supports limited models, I tried to do this conversion by assigning parameters directly, but the gained tensorflow model failed to show the desired accuracy. Details are described as follows:

The source model is Lenet trained on MNIST dataset.
I firstly extracted each module and its parameters by model.named_parameters() and save them into a dictionary where the key is the module's name and the value is the parameters
Then, I built and initiated a tensorflow model with the same architecture
Finally, I assign each layer's parameters of pytroch model to the tensorflow model

However, the accuracy of gained tensorflow model is only about 20%. Thus, my question is that is it possible to convert the pytorch model by this method?. If yes, what's the possible issue causing the bad result? If no, then please kindly explain the reasons.

PS: assume the assignment procedure is right.

Output:
As the comment by jodag mentioned, there are many differences between operator representations in Tensorflow and PyTorch that might cause discrepancies in your workflow.

We would recommend using the following method:

Use the ONNX exporter in PyTorch to export the model to the ONNX format.

import torch.onnx

# Argument: model is the PyTorch model
# Argument: dummy_input is a torch tensor

torch.onnx.export(model, dummy_input, "LeNet_model.onnx")

Use the onnx-tensorflow backend to convert the ONNX model to Tensorflow.

import onnx

from onnx_tf.backend import prepare

onnx_model = onnx.load("LeNet_model.onnx")  # load onnx model
tf_rep = prepare(onnx_model)  # prepare tf representation
tf_rep.export_graph("LeNet_model.pb")  # export the model


Instruction: PyTorch: Different Forward Methods for Train and Test/Validation

Input:
I'm currently trying to extend a model that is based on FairSeq/PyTorch. During training I need to train two encoders: one with the target sample, and the original one with the source sample.

So the current forward function looks like this:

def forward(self, src_tokens=None, src_lengths=None, prev_output_tokens=None, **kwargs):
    encoder_out = self.encoder(src_tokens, src_lengths=src_lengths, **kwargs)
    decoder_out = self.decoder(prev_output_tokens, encoder_out=encoder_out, **kwargs)
    return decoder_out

And based on this this idea i want something like this:

def forward_test(self, src_tokens=None, src_lengths=None, prev_output_tokens=None, **kwargs):
    encoder_out = self.encoder(src_tokens, src_lengths=src_lengths, **kwargs)
    decoder_out = self.decoder(prev_output_tokens, encoder_out=encoder_out, **kwargs)
    return decoder_out

def forward_train(self, src_tokens=None, src_lengths=None, prev_output_tokens=None, **kwargs):
    encoder_out = self.encoder(src_tokens, src_lengths=src_lengths, **kwargs)
    autoencoder_out = self.encoder(tgt_tokens, src_lengths=src_lengths, **kwargs)
    concat = some_concatination_func(encoder_out, autoencoder_out)
    decoder_out = self.decoder(prev_output_tokens, encoder_out=concat, **kwargs)
    return decoder_out

Is there any way to do this?

Edit:
These are the constraints that I have, since I need to extend FairseqEncoderDecoderModel:

@register_model('transformer_mass')
class TransformerMASSModel(FairseqEncoderDecoderModel):
    def __init__(self, encoder, decoder):
        super().__init__(encoder, decoder)

Edit 2:
The parameters passed to the forward function in Fairseq can be altered by implementing your own Criterion, see for example CrossEntropyCriterion, where sample['net_input'] is passed to the __call__ function of the model, which invokes the forward method.

Output:
First of all you should always use and define forward not some other methods that you call on the torch.nn.Module instance.

Definitely do not overload eval() as shown by trsvchn as it's evaluation method defined by PyTorch (see here). This method allows layers inside your model to be put into evaluation mode (e.g. specific changes to layers like inference mode for Dropout or BatchNorm).

Furthermore you should call it with __call__ magic method. Why? Because hooks and other PyTorch specific stuff is registered that way properly.

Secondly, do not use some external mode string variable as suggested by @Anant Mittal. That's what train variable in PyTorch is for, it's standard to differentiate by it whether model is in eval mode or train mode.

That being said you are the best off doing it like this:

import torch

class Network(torch.nn.Module):
    def __init__(self):
        super().__init__()
        ...

    # You could split it into two functions but both should be called by forward
    def forward(
        self, src_tokens=None, src_lengths=None, prev_output_tokens=None, **kwargs
    ):
        encoder_out = self.encoder(src_tokens, src_lengths=src_lengths, **kwargs)
        if self.train:
            return self.decoder(prev_output_tokens, encoder_out=encoder_out, **kwargs)
        autoencoder_out = self.encoder(tgt_tokens, src_lengths=src_lengths, **kwargs)
        concat = some_concatination_func(encoder_out, autoencoder_out)
        return self.decoder(prev_output_tokens, encoder_out=concat, **kwargs)

You could (and arguably should) split the above into two separate methods, but that's not too bad as the function is rather short and readable that way. Just stick to PyTorch's way of handling things if easily possible and not some ad-hoc solutions. And no, there will be no problem with backpropagation, why would there be one?


Instruction: pytorch nn.Sequential(*list) TypeError: list is not a Module subclass

Input:
when I use pytorch to train a model, I tried to print the whole net structure

so I packed all the layers in a list
then I use nn.Sequential(*list)

but it doesn't work, and the TypeError: list is not a Module subclass

Output:
Please provide the list of layers that you have created, are you sure you haven' done any error in that. Try checking if your list is actually [] and not [[..]]. The other thing that I noticed is that you have list as a variable name, which isn't a good idea - list is a Python keyword.

I tried writing a sample code of unpacking a list and it works fine for me.

import torch
import torch.nn as nn                                                                           net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))
layers = [nn.Linear(2, 2), nn.Linear(2, 2)]
net = nn.Sequential(*layers)
print(net)

This ran without any error, and the result was:

Sequential(
  (0): Linear(in_features=2, out_features=2, bias=True)
  (1): Linear(in_features=2, out_features=2, bias=True)
)

Hope this helps. :)


Instruction: PyTorch Data Augmentation is taking too long

Input:
For the task that involves regression, I need to train my models to generate density maps from RGB images. To augment my dataset I have decided to flip all the images horizontally. For that matter, I also have to flip my ground truth images and I did so.

dataset_for_augmentation.listDataset(train_list,
                        shuffle=True,
                        transform=transforms.Compose([
                            transforms.RandomHorizontalFlip(p=1),
                            transforms.ToTensor(),
                            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
                        ]),
                        target_transform=transforms.Compose([
                            transforms.RandomHorizontalFlip(p=1),
                            transforms.ToTensor()
                        ]),
                        train=True,
                        resize=4,
                        batch_size=args.batch_size,
                        num_workers=args.workers),

But here is the problem : For some reason, PyTorch transforms.RandomHorizontalFlip function takes only PIL images (numpy is not allowed) as input. So I decided to convert the type to PIL Image.

img_path = self.lines[index]

img, target = load_data(img_path, self.train, resize=self.resize)

if type(target[0][0]) is np.float64:
    target = np.float32(target)

img = Image.fromarray(img)
target = Image.fromarray(target)

if self.transform is not None:
    img = self.transform(img)
    target = self.target_transform(target)

return img, target

And yes, this operation need enormous amount of time. Considering I need this operation to be carried out for thousands of images, 23 seconds (should have been under half a second at most) per batch is not tolerable.

2019-11-01 16:29:02,497 - INFO - Epoch: [0][0/152]  Time 27.095 (27.095)    Data 23.150 (23.150)    Loss 93.7401 (93.7401)

I would appreciate any suggestions to speed up my augmentation process

Output:
You don't need to change the DataLoader to do that. You can use ToPILImage():

transform=transforms.Compose([
    transforms.ToPILImage(),  # check mode assumption in the documentation
    transforms.RandomHorizontalFlip(p=1),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

Anyway, I would avoid converting to PIL. It seems completely unnecessary. If you want to flip all images, then why not to do that using NumPy only?

img_path = self.lines[index]

img, target = load_data(img_path, self.train, resize=self.resize)

if type(target[0][0]) is np.float64:
    target = np.float32(target)

# assuming width axis=1 -- see my comment below
img = np.flip(img, axis=1)
target = np.flip(target, axis=1)

if self.transform is not None:
    img = self.transform(img)
    target = self.target_transform(target)

return img, target

And remove the transforms.RandomHorizontalFlip(p=1) from the Compose. As ToTensor(...) also handles ndarray, you are good to go.

Note: I am assuming the width axis is equal to 1, since ToTensor expects it to be there.

From the docs:

  Converts a PIL Image or numpy.ndarray (H x W x C) ...


Instruction: How to compute pairwise distance between point set and lines in PyTorch?

Input:
The point set A is a Nx3 matrix, and from two point sets B and C with the same size of Mx3 we could get the lines BC betwen them. Now I want to compute the distance from each point in A to each line in BC. B is Mx3 and C is Mx3, then the lines are from the points with correspoinding rows, so BC is a Mx3 matrix. The basic method is computed as follows:

D = torch.zeros((N, M), dtype=torch.float32)
for i in range(N):
    p = A[i]  # 1x3
    for j in range(M):
        p1 = B[j] # 1x3
        p2 = C[j] # 1x3
        D[i,j] = torch.norm(torch.cross(p1 - p2, p - p1)) / torch.norm(p1 - p2)

Are there any faster method to do this work? Thanks.

Output:
You can remove the for loops by doing this (it should speed-up at the cost of memory, unless M and N are small):

diff_B_C = B - C
diff_A_C = A[:, None] - C
norm_lines = torch.norm(diff_B_C, dim=-1)
cross_result = torch.cross(diff_B_C[None, :].expand(N, -1, -1), diff_A_C, dim=-1)
norm_cross = torch.norm(cross_result, dim=-1)
D = norm_cross / norm_lines

Of course, you don't need to do it step-by-step. I just tried to be clear with the variable names.

Note: if you don't provide dim to torch.cross, it will use the first dim=3 which would give the wrong results if N=3 (from the docs):

  If dim is not given, it defaults to the first dimension found with the size 3.

If you are wondering, you can check here why I chose expand instead of repeat.


Instruction: Loss doesn't decrease in Pytorch CNN

Input:
I'm doing a CNN with Pytorch for a task, but it won't learn and improve the accuracy. I made a version working with the MNIST dataset so I could post it here. I'm just looking for an answer as to why it's not working. The architecture is fine, I implemented it in Keras and I had over 92% accuracy after 3 epochs. Note: I reshaped the MNIST into 60x60 pictures because that's how the pictures are in my "real" problem.

import numpy as np
from PIL import Image
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torch.autograd import Variable
from keras.datasets import mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()

def resize(pics):
    pictures = []
    for image in pics:
        image = Image.fromarray(image).resize((dim, dim))
        image = np.array(image)
        pictures.append(image)
    return np.array(pictures)

dim = 60

x_train, x_test = resize(x_train), resize(x_test) # because my real problem is in 60x60

x_train = x_train.reshape(-1, 1, dim, dim).astype('float32') / 255
x_test = x_test.reshape(-1, 1, dim, dim).astype('float32') / 255
y_train, y_test = y_train.astype('float32'), y_test.astype('float32')

if torch.cuda.is_available():
    x_train = torch.from_numpy(x_train)[:10_000]
    x_test = torch.from_numpy(x_test)[:4_000]
    y_train = torch.from_numpy(y_train)[:10_000]
    y_test = torch.from_numpy(y_test)[:4_000]

class ConvNet(nn.Module):

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 32, 3)
        self.conv2 = nn.Conv2d(32, 64, 3)
        self.conv3 = nn.Conv2d(64, 128, 3)

        self.fc1 = nn.Linear(5*5*128, 1024)
        self.fc2 = nn.Linear(1024, 2048)
        self.fc3 = nn.Linear(2048, 1)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.dropout(x, 0.5)
        x = torch.sigmoid(self.fc3(x))
        return x

net = ConvNet()

optimizer = optim.Adam(net.parameters(), lr=0.03)

loss_function = nn.BCELoss()

class FaceTrain:

    def __init__(self):
        self.len = x_train.shape[0]
        self.x_train = x_train
        self.y_train = y_train

    def __getitem__(self, index):
        return x_train[index], y_train[index].unsqueeze(0)

    def __len__(self):
        return self.len

class FaceTest:

    def __init__(self):
        self.len = x_test.shape[0]
        self.x_test = x_test
        self.y_test = y_test

    def __getitem__(self, index):
        return x_test[index], y_test[index].unsqueeze(0)

    def __len__(self):
        return self.len

train = FaceTrain()
test = FaceTest()

train_loader = DataLoader(dataset=train, batch_size=64, shuffle=True)
test_loader = DataLoader(dataset=test, batch_size=64, shuffle=True)

epochs = 10
steps = 0
train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    for images, labels in train_loader:
        optimizer.zero_grad()
        log_ps = net(images)
        loss = loss_function(log_ps, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    else:
        test_loss = 0
        accuracy = 0

        with torch.no_grad():
            for images, labels in test_loader:
                log_ps = net(images)
                test_loss += loss_function(log_ps, labels)
                ps = torch.exp(log_ps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class.type('torch.LongTensor') == labels.type(torch.LongTensor).view(*top_class.shape)
                accuracy += torch.mean(equals.type('torch.FloatTensor'))
        train_losses.append(running_loss/len(train_loader))
        test_losses.append(test_loss/len(test_loader))
        print("[Epoch: {}/{}] ".format(e+1, epochs),
              "[Training Loss: {:.3f}] ".format(running_loss/len(train_loader)),
              "[Test Loss: {:.3f}] ".format(test_loss/len(test_loader)),
              "[Test Accuracy: {:.3f}]".format(accuracy/len(test_loader)))

Output:
First the major issues...

1. The main issue with this code is that you're using the wrong output shape and the wrong loss function for classification.

nn.BCELoss computes the binary cross entropy loss. This is applicable when you have one or more targets which are either 0 or 1 (hence the binary). In your case the target is a single integer between 0 and 9. Since there are only a small number of potential target values, the most common approach is to use categorical cross-entropy loss (nn.CrossEntropyLoss). The "theoretical" definition of cross entropy loss expects the network outputs and the targets to both be 10 dimensional vectors where the target is all zeros except in one location (one-hot encoded). However for computational stability and space efficiency reasons, pytorch's nn.CrossEntropyLoss directly takes the integer as a target. However, you still need to provide it with a 10 dimensional output vector from your network.

# pseudo code (ignoring batch dimension)
loss = nn.functional.cross_entropy_loss(<output 10d vector>, <integer target>)

To fix this issue in your code we need to have fc3 output a 10 dimensional feature, and we need the labels to be integers (not floats). Also, there's no need to use .sigmoid on fc3 since pytorch's cross-entropy loss function internally applies log-softmax before computing the final loss value.

2. As pointed out by Serget Dymchenko, you need to switch the network to eval mode during inference and train mode during train. This mainly affects dropout and batch_norm layers since they behave differently during training and inference.

3. A learning rate of 0.03 is probably a little too high. It works just fine with a learning rate of 0.001 and in a couple experiments I saw the training diverge at 0.03.

To accommodate these fixes a number of changes needed to be made. The minimal corrections to the code are shown below. I commented any lines which were changed with #### followed by a short description of the change.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torch.autograd import Variable
from keras.datasets import mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()

def resize(pics):
    pictures = []
    for image in pics:
        image = Image.fromarray(image).resize((dim, dim))
        image = np.array(image)
        pictures.append(image)
    return np.array(pictures)

dim = 60

x_train, x_test = resize(x_train), resize(x_test) # because my real problem is in 60x60

x_train = x_train.reshape(-1, 1, dim, dim).astype('float32') / 255
x_test = x_test.reshape(-1, 1, dim, dim).astype('float32') / 255
#### float32 -> int64
y_train, y_test = y_train.astype('int64'), y_test.astype('int64')

#### no reason to test for cuda before converting to numpy

#### I assume you were taking a subset for debugging? No reason to not use all the data
x_train = torch.from_numpy(x_train)
x_test = torch.from_numpy(x_test)
y_train = torch.from_numpy(y_train)
y_test = torch.from_numpy(y_test)

class ConvNet(nn.Module):

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 32, 3)
        self.conv2 = nn.Conv2d(32, 64, 3)
        self.conv3 = nn.Conv2d(64, 128, 3)

        self.fc1 = nn.Linear(5*5*128, 1024)
        self.fc2 = nn.Linear(1024, 2048)
        #### 1 -> 10
        self.fc3 = nn.Linear(2048, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.dropout(x, 0.5)
        #### removed sigmoid
        x = self.fc3(x)
        return x

net = ConvNet()

#### 0.03 -> 1e-3
optimizer = optim.Adam(net.parameters(), lr=1e-3)

#### BCELoss -> CrossEntropyLoss
loss_function = nn.CrossEntropyLoss()

class FaceTrain:

    def __init__(self):
        self.len = x_train.shape[0]
        self.x_train = x_train
        self.y_train = y_train

    def __getitem__(self, index):
        #### .unsqueeze(0) removed
        return x_train[index], y_train[index]

    def __len__(self):
        return self.len

class FaceTest:

    def __init__(self):
        self.len = x_test.shape[0]
        self.x_test = x_test
        self.y_test = y_test

    def __getitem__(self, index):
        #### .unsqueeze(0) removed
        return x_test[index], y_test[index]

    def __len__(self):
        return self.len

train = FaceTrain()
test = FaceTest()

train_loader = DataLoader(dataset=train, batch_size=64, shuffle=True)
test_loader = DataLoader(dataset=test, batch_size=64, shuffle=True)

epochs = 10
steps = 0
train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    #### put net in train mode
    net.train()
    for idx, (images, labels) in enumerate(train_loader):
        optimizer.zero_grad()
        log_ps = net(images)
        loss = loss_function(log_ps, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    else:
        test_loss = 0
        accuracy = 0

        #### put net in eval mode
        net.eval()
        with torch.no_grad():
            for images, labels in test_loader:
                log_ps = net(images)
                test_loss += loss_function(log_ps, labels)
                #### removed torch.exp() since exponential is monotone, taking it doesn't change the order of outputs. Similarly with torch.softmax()
                top_p, top_class = log_ps.topk(1, dim=1)
                #### convert to float/long using proper methods. what you have won't work for cuda tensors.
                equals = top_class.long() == labels.long().view(*top_class.shape)
                accuracy += torch.mean(equals.float())
        train_losses.append(running_loss/len(train_loader))
        test_losses.append(test_loss/len(test_loader))
        print("[Epoch: {}/{}] ".format(e+1, epochs),
              "[Training Loss: {:.3f}] ".format(running_loss/len(train_loader)),
              "[Test Loss: {:.3f}] ".format(test_loss/len(test_loader)),
              "[Test Accuracy: {:.3f}]".format(accuracy/len(test_loader)))

Results of training are now...

[Epoch: 1/10]  [Training Loss: 0.139]  [Test Loss: 0.046]  [Test Accuracy: 0.986]
[Epoch: 2/10]  [Training Loss: 0.046]  [Test Loss: 0.042]  [Test Accuracy: 0.987]
[Epoch: 3/10]  [Training Loss: 0.031]  [Test Loss: 0.040]  [Test Accuracy: 0.988]
[Epoch: 4/10]  [Training Loss: 0.022]  [Test Loss: 0.029]  [Test Accuracy: 0.990]
[Epoch: 5/10]  [Training Loss: 0.017]  [Test Loss: 0.066]  [Test Accuracy: 0.987]
[Epoch: 6/10]  [Training Loss: 0.015]  [Test Loss: 0.056]  [Test Accuracy: 0.985]
[Epoch: 7/10]  [Training Loss: 0.018]  [Test Loss: 0.039]  [Test Accuracy: 0.991]
[Epoch: 8/10]  [Training Loss: 0.012]  [Test Loss: 0.057]  [Test Accuracy: 0.988]
[Epoch: 9/10]  [Training Loss: 0.012]  [Test Loss: 0.041]  [Test Accuracy: 0.991]
[Epoch: 10/10]  [Training Loss: 0.007]  [Test Loss: 0.048]  [Test Accuracy: 0.992]

Some other issues that will improve your performance and code.

4. You're never moving the model to the GPU. This means you won't be getting GPU acceleration.

5. torchvision is designed with all the standard transforms and datasets and is built to be used with PyTorch. I recommend using it. This also removes the dependency on keras in your code.

6. Normalize your data by subtracting the mean and dividing by the standard deviation to improve performance of your network. With torchvision you can use transforms.Normalize. This won't make a big difference in MNIST because its already too easy. But in more difficult problems it turns out to be important.

Further improved code is show below (much faster on GPU).

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision.datasets import MNIST
from torchvision import transforms

dim = 60

class ConvNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 32, 3)
        self.conv2 = nn.Conv2d(32, 64, 3)
        self.conv3 = nn.Conv2d(64, 128, 3)

        self.fc1 = nn.Linear(5 * 5 * 128, 1024)
        self.fc2 = nn.Linear(1024, 2048)
        self.fc3 = nn.Linear(2048, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.dropout(x, 0.5)
        x = self.fc3(x)
        return x

net = ConvNet()
if torch.cuda.is_available():
    net.cuda()

optimizer = optim.Adam(net.parameters(), lr=1e-3)

loss_function = nn.CrossEntropyLoss()

train_dataset = MNIST('./data', train=True, download=True,
                      transform=transforms.Compose([
                          transforms.Resize((dim, dim)),
                          transforms.ToTensor(),
                          transforms.Normalize((0.1307,), (0.3081,))
                      ]))
test_dataset = MNIST('./data', train=False, download=True,
                     transform=transforms.Compose([
                         transforms.Resize((dim, dim)),
                         transforms.ToTensor(),
                         transforms.Normalize((0.1307,), (0.3081,))
                     ]))

train_loader = DataLoader(dataset=train_dataset, batch_size=64, shuffle=True, num_workers=8)
test_loader = DataLoader(dataset=test_dataset, batch_size=64, shuffle=False, num_workers=8)

epochs = 10
steps = 0
train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    net.train()
    for images, labels in train_loader:
        if torch.cuda.is_available():
            images, labels = images.cuda(), labels.cuda()
        optimizer.zero_grad()
        log_ps = net(images)
        loss = loss_function(log_ps, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    else:
        test_loss = 0
        accuracy = 0

        net.eval()
        with torch.no_grad():
            for images, labels in test_loader:
                if torch.cuda.is_available():
                    images, labels = images.cuda(), labels.cuda()
                log_ps = net(images)
                test_loss += loss_function(log_ps, labels)
                top_p, top_class = log_ps.topk(1, dim=1)
                equals = top_class.flatten().long() == labels
                accuracy += torch.mean(equals.float()).item()
        train_losses.append(running_loss/len(train_loader))
        test_losses.append(test_loss/len(test_loader))
        print("[Epoch: {}/{}] ".format(e+1, epochs),
              "[Training Loss: {:.3f}] ".format(running_loss/len(train_loader)),
              "[Test Loss: {:.3f}] ".format(test_loss/len(test_loader)),
              "[Test Accuracy: {:.3f}]".format(accuracy/len(test_loader)))

Updated results of training...

[Epoch: 1/10]  [Training Loss: 0.125]  [Test Loss: 0.045]  [Test Accuracy: 0.987]
[Epoch: 2/10]  [Training Loss: 0.043]  [Test Loss: 0.031]  [Test Accuracy: 0.991]
[Epoch: 3/10]  [Training Loss: 0.030]  [Test Loss: 0.030]  [Test Accuracy: 0.991]
[Epoch: 4/10]  [Training Loss: 0.024]  [Test Loss: 0.046]  [Test Accuracy: 0.990]
[Epoch: 5/10]  [Training Loss: 0.020]  [Test Loss: 0.032]  [Test Accuracy: 0.992]
[Epoch: 6/10]  [Training Loss: 0.017]  [Test Loss: 0.046]  [Test Accuracy: 0.991]
[Epoch: 7/10]  [Training Loss: 0.015]  [Test Loss: 0.034]  [Test Accuracy: 0.992]
[Epoch: 8/10]  [Training Loss: 0.011]  [Test Loss: 0.048]  [Test Accuracy: 0.992]
[Epoch: 9/10]  [Training Loss: 0.012]  [Test Loss: 0.037]  [Test Accuracy: 0.991]
[Epoch: 10/10]  [Training Loss: 0.013]  [Test Loss: 0.038]  [Test Accuracy: 0.992]


Instruction: Pytorch Hardware Requirement

Input:
What is the minimum Computation Capability required by the latest PyTorch version?
I have Nvidia Geforce 820M with computation capability 2.1. How can I run PyTorch models on my GPU (if it doesn't support naturally)

Output:
Looking at this page, PyTorch (even the somewhat oldest versions) support CUDA upwards from version 7.5. Whereas, looking at this page, CUDA 7.5 requires minimum Compute Capability 2.0. So, on paper, your machine should support some older version of PyTorch which allows CUDA 7.5 or preferably 8.0 (as of writing this answer, the latest version uses minimum CUDA 9.2).

However, PyTorch also requires cuDNN. So, cuDNN 6.0 works for CUDA 7.5. But cuDNN 6.0 requires Compute Capability of 3.0. So, mostly, PyTorch won't work on your machine. (Thanks for pointing out the cuDNN part Robert Crovella)


Instruction: Can not get pytorch working with tensorboard

Input:
I"m going through this tutorial to set up pytorch (v1.3.0 through conda) with tensorboard https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html#

but on the step

from torch.utils.tensorboard import SummaryWriter

# default `log_dir` is "runs" - we'll be more specific here
writer = SummaryWriter('runs/fashion_mnist_experiment_1')

I keep getting the error

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
C:\ProgramData\Anaconda3\envs\fastai_v1\lib\site-packages\torch\utils\tensorboard\__init__.py in
      1 try:
----> 2     from tensorboard.summary.writer.record_writer import RecordWriter  # noqa F401
      3 except ImportError:

ModuleNotFoundError: No module named 'tensorboard.summary'; 'tensorboard' is not a package

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
c:\Users\matt\Documents\code\playground\tensorboard.py in
----> 1 from torch.utils.tensorboard import SummaryWriter
      2
      3 # default `log_dir` is "runs" - we'll be more specific here
      4 writer = SummaryWriter('runs/fashion_mnist_experiment_1')

C:\ProgramData\Anaconda3\envs\fastai_v1\lib\site-packages\torch\utils\tensorboard\__init__.py in
      2     from tensorboard.summary.writer.record_writer import RecordWriter  # noqa F401
      3 except ImportError:
----> 4     raise ImportError('TensorBoard logging requires TensorBoard with Python summary writer installed. '
      5                       'This should be available in 1.14 or above.')
      6 from .writer import FileWriter, SummaryWriter  # noqa F401

ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed. This should be available in 1.14 or above.

Does anyone have any suggestions?

Output:
The error log says, among other things,

  ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed. This should be available in 1.14 or above.

So, when it tries to import TensorBoard, it's unable to do so because it's missing it in the search path. You can install the latest version (without specifying any version number), as in:

$ conda install -c conda-forge tensorboard

Apart from that, you might also need to install protobuf:

$ conda install -c conda-forge protobuf

These installations should fix the ImportErrors.


Instruction: How do I augment data after spliting traininng datset into train and validation set for CIFAR10 using PyTorch?

Input:
When classifying the CIFAR10 in PyTorch, there are normally 50,000 training samples and 10,000 testing samples. However, if I need to create a validation set, I can do it by splitting the training set into 40000 train samples and 10000 validation samples. I used the following codes

train_transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])
test_transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])

cifar_train_L = CIFAR10('./data',download=True, train= True, transform = train_transform)
cifar_test = CIFAR10('./data',download=True, train = False, transform= test_transform)

train_size = int(0.8*len(cifar_training))
val_size = len(cifar_training) - train_size
cifar_train, cifar_val = torch.utils.data.random_split(cifar_train_L,[train_size,val_size])

train_dataloader = torch.utils.data.DataLoader(cifar_train, batch_size= BATCH_SIZE, shuffle= True, num_workers=2)
test_dataloader = torch.utils.data.DataLoader(cifar_test,batch_size= BATCH_SIZE, shuffle= True, num_workers= 2)
val_dataloader = torch.utils.data.DataLoader(cifar_val,batch_size= BATCH_SIZE, shuffle= True, num_workers= 2)

Normally, when augmenting data in PyTorch, different augmenting processes are used under the
transforms.Compose function (i.e., transforms.RandomHorizontalFlip()). However, if I use these augmentation processes before splitting the training set and validation set, the augmented data will also be included in the validation set. Is there any way, I can fix this problem?

  In short, I want to manually split the
  training dataset into train and validation set as well as I want to
  use the data augmentation technique into the new training set.

Output:
You can manually override the transforms of the dataset:

cifar_train, cifar_val = torch.utils.data.random_split(cifar_train_L,[train_size,val_size])
cifar_val.transforms = test_transform


Instruction: Pytorch BiLSTM POS Tagging Issue: RuntimeError: input.size(-1) must be equal to input_size. Expected 6, got 12

Input:
I have a nlp dataset, and according to the Pytorch official tutorial, I change the dataset to the word_to_idx and tag_to_idx, like:

word_to_idx =  {'I': 0, 'have': 1, 'used': 2, 'transfers': 3, 'on': 4, 'three': 5, 'occasions': 6, 'now': 7, 'and': 8, 'each': 9, 'time': 10}
tag_to_idx =  {'PRON': 0, 'VERB': 1, 'NOUN': 2, 'ADP': 3, 'NUM': 4, 'ADV': 5, 'CONJ': 6, 'DET': 7, 'ADJ': 8, 'PRT': 9, '.': 10, 'X': 11}

I want to complete the POS-Tagging task with BiLSTM. Here is my BiLSTM code:

class LSTMTagger(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTMTagger, self).__init__()
        self.hidden_dim = hidden_dim
        self.word_embeddings = nn.Embedding(vocab_size, tagset_size)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, bidirectional=True)

        # The linear layer that maps from hidden state space to tag space
        self.hidden2tag = nn.Linear(in_features=hidden_dim * 2, out_features=tagset_size)

    def forward(self, sentence):
        embeds = self.word_embeddings(sentence)
        lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
        tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))

        # tag_scores = F.softmax(tag_space, dim=1)
        tag_scores = F.log_softmax(tag_space, dim=1)
        return tag_scores

Then I run the training code in Pycharm, like:

EMBEDDING_DIM = 6
HIDDEN_DIM = 6
NUM_EPOCHS = 3

model = LSTMTagger(embedding_dim=EMBEDDING_DIM,
                   hidden_dim=HIDDEN_DIM,
                   vocab_size=len(word_to_idx),
                   tagset_size=len(tag_to_idx))

loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

# See what the scores are before training
with torch.no_grad():
    inputs = prepare_sequence(training_data[0][0], word_to_idx)
    tag_scores = model(inputs)
    print(tag_scores)
    print(tag_scores.size())

However, it shows error with line tag_scores = model(inputs) and line lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1)).
The error is:

Traceback (most recent call last):
  line 140, in <module>
    tag_scores = model(inputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  line 115, in forward
    lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 559, in forward
    return self.forward_tensor(input, hx)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 539, in forward_tensor
    output, hidden = self.forward_impl(input, hx, batch_sizes, max_batch_size, sorted_indices)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 519, in forward_impl
    self.check_forward_args(input, hx, batch_sizes)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 490, in check_forward_args
    self.check_input(input, batch_sizes)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 153, in check_input
    self.input_size, input.size(-1)))
RuntimeError: input.size(-1) must be equal to input_size. Expected 6, got 12

I don't know how to debug with it. Could somebody help me fix this issue? Thanks in advance!

Output:
The error is here:

self.word_embeddings = nn.Embedding(vocab_size, tagset_size)

Instead of using the embedding dimension, you use the number of tags which is 12 and not 6 which is what the LSTM layer expects.


Instruction: Recommendation for Best Neural Network Type (in TensorFLow or PyTorch) For Fitting Problems

Input:
I am looking to develop a simple Neural Network in PyTorch or TensorFlow to predict one numeric value based on several inputs.

For example, if one has data describing the interior comfort parameters for a building, the NN should predict the numeric value for the energy consumption.

Both PyTorch or TensorFlow documented examples and tutorials are generally focused on classification and time dependent series (which is not the case). Any idea on which NN available in those libraries is best for this kind of problems? I'm just looking for a hint about the type, not code.
Thanks!

Output:
The type of problem you are talking about is called a regression problem. In such types of problems, you would have a single output neuron with a linear activation (or no activation). You would use MSE or MAE to train your network.

If your problem is time series(where you are using previous values to predict current/next value) then you could try doing multi-variate time series forecasting using LSTMs.

If your problem is not time series, then you could just use a vanilla feed forward neural network. This article explains the concepts of data correlation really well and you might find it useful in deciding what type of neural networks to use based on the type of data and output you have.


Instruction: Why am I getting a Pytorch Runtime Error on Test Set

Input:
I have a model that is a binary image classification model with the resnext model.  I keep getting  a run time error when it gets to the test set.  Error message is
RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'

I am sending my test set tensors to my GPU like my train model.  I've looked at the following and I'm doing what was suggested here as stated above.

Here is my model code:

resnext = models.resnext50_32x4d(pretrained=True)
resnext = resnext.to(device)
for param in resnext.parameters():
    param.requires_grad = True
resnext.classifier = nn.Sequential(nn.Linear(2048, 1000),
                                 nn.ReLU(),
                                 nn.Dropout(0.4),
                                 nn.Linear(1000, 2),
                                 nn.Softmax(dim = 1))
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(resnext.classifier.parameters(), lr=0.001)
import time
start_time = time.time()

epochs = 1

max_trn_batch = 5
max_tst_batch = 156

y_val_list = []
policy_list = []

train_losses = []
test_losses = []
train_correct = []
test_correct = []

for i in range(epochs):
    for i in tqdm(range(0, max_trn_batch)):
        trn_corr = 0
        tst_corr = 0

        # Run the training batches
        for b, (X_train, y_train, policy) in enumerate(train_loader):
            #print(y_train, policy)
            X_train = X_train.to(device)
            y_train = y_train.to(device)
            if b == max_trn_batch:
                break
            b+=1

            # Apply the model
            y_pred = resnext(X_train)
            loss = criterion(y_pred, y_train)

            # Tally the number of correct predictions
            predicted = torch.max(y_pred.data, 1)[1]
            batch_corr = (predicted == y_train).sum()
            trn_corr += batch_corr
            # Update parameters
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            # Print interim results
            if b%1 == 0:
                print(f'epoch: {i:2}  batch: {b:4} [{100*b:6}/63610]  loss: {loss.item():10.8f}  \
    accuracy: {trn_corr.item()/(100*b):7.3f}%')

        train_losses.append(loss)
        train_correct.append(trn_corr)

        # Run the testing batches
        with torch.no_grad():
            for b, (X_test, y_test, policy) in enumerate(test_loader):
                policy_list.append(policy)
                X_test.to(device)
                y_test.to(device)
                if b == max_tst_batch:
                    break

                # Apply the model
                y_val = resnext(X_test)
                y_val_list.append(y_val.data)
                # Tally the number of correct predictions
                predicted = torch.max(y_val.data, 1)[1]
                tst_corr += (predicted == y_test).sum()

        loss = criterion(y_val, y_test)
        test_losses.append(loss)
        test_correct.append(tst_corr)

    print(f'\nDuration: {time.time() - start_time:.0f} seconds') # print the time elapsed

Here is the full traceback:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-84-48bce2e8d4fa> in <module>
     60
     61                 # Apply the model
---> 62                 y_val = resnext(X_test)
     63                 y_val_list.append(y_val.data)
     64                 # Tally the number of correct predictions

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

C:\ProgramData\Anaconda3\lib\site-packages\torchvision\models\resnet.py in forward(self, x)
    194
    195     def forward(self, x):
--> 196         x = self.conv1(x)
    197         x = self.bn1(x)
    198         x = self.relu(x)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py in forward(self, input)
    341
    342     def forward(self, input):
--> 343         return self.conv2d_forward(input, self.weight)
    344
    345 class Conv3d(_ConvNd):

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py in conv2d_forward(self, input, weight)
    338                             _pair(0), self.dilation, self.groups)
    339         return F.conv2d(input, weight, self.bias, self.stride,
--> 340                         self.padding, self.dilation, self.groups)
    341
    342     def forward(self, input):

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'

Again, my tensors and the model are sent to the GPU so I'm not sure what is going on. Does anyone see my mistake?

Output:
  [...] my tensors and the model are sent to the GPU [...]

Not the test Tensors. It is a simple mistake:

X_test.to(device)
y_test.to(device)

should be

X_test = X_test.to(device)
y_test = y_test.to(device)


Instruction: How to reduce the dimensions of a tensor with neural networks

Input:
I have a 3D tensor of size [100,70,42] (batch, seq_len, features) and I would like to get a tensor of size [100,1,1] by using a neural network based on linear transformations (nn.Linear in Pytorch).

I have implemented the following code

class Network(nn.Module):
   def __init__(self):
      super(Network, self).__init__()
      self.fc1 = nn.Linear(42, 120)
      self.fc2 = nn.Linear(120,1)

   def forward(self, input):
      model = nn.Sequential(self.fc1,
                            nn.ReLU(),
                            self.fc2)
      output = model(input)
      return output

However, upon training this only gives me an output of the shape [100,70,1], which is not the desired one.

Thanks!

Output:
nn.Linear acts only on last axis. If you want to apply linear over last two dimensions, you must reshape your input tensor:

class Network(nn.Module):
   def __init__(self):
      super(Network, self).__init__()
      self.fc1 = nn.Linear(70 * 42, 120)  # notice input shape
      self.fc2 = nn.Linear(120,1)

   def forward(self, input):
      input = input.reshape((-1, 70 * 42))  # added reshape
      model = nn.Sequential(self.fc1,
                            nn.ReLU(),
                            self.fc2)
      output = model(input)
      output = output.reshape((-1, 1, 1))  # OP asked for 3-dim output
      return output


Instruction: Is there a tensor.item() equivalent for a tensor containing a list in pytorch?

Input:
In pytorch, if I define a one-element tensor as follows:

>>> import torch
>>> target1 = torch.tensor([5])

I'm able to pull out the value of its one element like this:

>>> target1.item()
5

What I would like to know is if when my tensor is defined as:

target2 = torch.tensor([[5], [5], [5], [5]])

Is there some way (similar or not to .item() above) to pull out all of its entries into a list like:

>>> target2.(something)
[5, 5, 5, 5]

I can't seem to find any function in the documentation that supports an operation like this.

Output:
You can use

target2.numpy().ravel() or

target2.view(-1).numpy() or

target2.view(target2.numel()).numpy()

Out[1]: array([5, 5, 5, 5], dtype=int64)


Instruction: runtimeerror size mismatch at tensor

Input:
error message: RuntimeError: size mismatch, m1: [64 x 3200], m2: [512 x 1] at C:/w/1/s/windows/pytorch/aten/src\THC/generic/THCTensorMathBlas.cu:290

The code is the following.:

class Generator(nn.Module):
def __init__(self):
    super(Generator, self).__init__()

    self.label_emb = nn.Embedding(opt.n_classes, opt.latent_dim)

    self.init_size = opt.img_size // 4  # Initial size before upsampling
    self.l1 = nn.Sequential(nn.Linear(opt.latent_dim, 128 * self.init_size ** 2))

    self.conv_blocks = nn.Sequential(
        nn.BatchNorm2d(128),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 128, 3, stride=1, padding=1),
        nn.BatchNorm2d(128, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 64, 3, stride=1, padding=1),
        nn.BatchNorm2d(64, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Conv2d(64, opt.channels, 3, stride=1, padding=1),
        nn.Tanh(),
    )

def forward(self, noise, labels):
    gen_input = torch.mul(self.label_emb(labels), noise)
    out = self.l1(gen_input)
    out = out.view(out.shape[0], 128, self.init_size, self.init_size)
    img = self.conv_blocks(out)
    return img

class Discriminator(nn.Module):
def __init__(self):
    super(Discriminator, self).__init__()

    def discriminator_block(in_filters, out_filters, bn=True):
        """Returns layers of each discriminator block"""
        block = [nn.Conv2d(in_filters, out_filters, 3, 2, 1), nn.LeakyReLU(0.2, inplace=True), nn.Dropout2d(0.25)]
        if bn:
            block.append(nn.BatchNorm2d(out_filters, 0.8))
        return block

    self.conv_blocks = nn.Sequential(
        *discriminator_block(opt.channels, 16, bn=False),
        *discriminator_block(16, 32),
        *discriminator_block(32, 64),
        *discriminator_block(64, 128),
    )

    # The height and width of downsampled image
    ds_size = opt.img_size // 2 ** 4

    # Output layers
    self.adv_layer = nn.Sequential(nn.Linear(128 * ds_size ** 2, 1), nn.Sigmoid())
    self.aux_layer = nn.Sequential(nn.Linear(128 * ds_size ** 2, opt.n_classes), nn.Softmax())

def forward(self, img):
    out = self.conv_blocks(img)
    out = out.view(out.shape[0], -1)
    validity = self.adv_layer(out)
    label = self.aux_layer(out)

    return validity, label

# Loss functions
adversarial_loss = torch.nn.BCELoss()
auxiliary_loss = torch.nn.CrossEntropyLoss()

# Initialize generator and discriminator
generator = Generator()
discriminator = Discriminator()

os.makedirs("../../data/mnist", exist_ok=True)
labels_path = 'C:/project/PyTorch-GAN/ulna/train-labels-idx1-ubyte.gz'
images_path = 'C:/project/PyTorch-GAN/ulna/train-images-idx3-ubyte.gz'
label_name = []

with gzip.open(labels_path, 'rb') as lbpath:
labels = np.frombuffer(lbpath.read(), dtype="int32", offset=8)
with gzip.open(images_path, 'rb') as imgpath:
images = np.frombuffer(imgpath.read(), dtype="int32", offset=16).reshape(len(labels),70,70,1)

hand_transform2 = transforms.Compose([
            transforms.Resize((70, 70)),
            transforms.Grayscale(1),
            transforms.ToTensor(),
            transforms.Normalize([0.5], [0.5])
            ])

#images=cv2.resize(images, (70, 70),1)
dataset1 = datasets.ImageFolder('C:/project/PyTorch-GAN/ulna/ulna', transform=hand_transform2)

dataloader = torch.utils.data.DataLoader(
dataset1,
batch_size=opt.batch_size,
shuffle=True,
)

The Traceback is the following.:

Traceback (most recent call last):
File "acgan.py", line 225, in <module>
real_pred, real_aux = discriminator(real_imgs)
File "C:\Users\S\AppData\Local\conda\conda\envs\venv\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "acgan.py", line 110, in forward
validity = self.adv_layer(out)
File "C:\Users\S\AppData\Local\conda\conda\envs\venv\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "C:\Users\S\AppData\Local\conda\conda\envs\venv\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
input = module(input)
File "C:\Users\S\AppData\Local\conda\conda\envs\venv\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
result = self.forward(*input, **kwargs)
File "C:\Users\S\AppData\Local\conda\conda\envs\venv\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "C:\Users\S\AppData\Local\conda\conda\envs\venv\lib\site-packages\torch\nn\functional.py", line 1370, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [64 x 3200], m2: [512 x 1] at C:/w/1/s/windows/pytorch/aten/src\THC/generic/THCTensorMathBlas.cu:290

What I want to practice is the GAN code. The entire GAN code before modification can be found at the following link: https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/acgan/acgan.py
Input image is an x-ray image adjusted to 70x70, and output image is a fake x-ray image that is newly created by learning input x-ray image. The code worked well when practised with the minist database. I'm afraid I don't have a clue about the problem of code.
please help me! Thank you.

Output:
It seems that opt.img_size might still be set to 32 as if you were using CIFAR. As you resize it to 70, it should be set to 70.

Anyway, another problem will arise because ds_size = opt.img_size // 2 ** 4 is not valid for opt.img_size=70. If you want a hard-coded solution, set ds_size=5. This fixes the Discriminator, but the same thing will happen to the Generator.

If you don't understand how to fix this properly, I would recommend you to take some time to read about how these models work. If you want to use the code as is, I would recommend you to use an img_size that is multiple of 16, e.g., opt.img_size=80 and you would have no problem. To avoid other problems, you may want to use transforms.Resize((opt.img_size, opt.img_size)) instead of hard-coding the img_size there.


Instruction: What "exactly" happens inside embedding layer in pytorch?

Input:
From multiple searches and pytorch documentation itself I could figure out that inside embedding layer there is a lookup table where the embedding vectors are stored. What I am not able to understand:

what exactly happens during training in this layer?
What are the weights and how the gradients of those weights are computed?
My intuition is that at least there should be a function with some parameters that produces the keys for the lookup table. If so, then what is that function?

Any help in this will be appreciated. Thanks.

Output:
That is a really good question! The embedding layer of PyTorch (same goes for Tensorflow) serves as a lookup table just to retrieve the embeddings for each of the inputs, which are indices. Consider the following case, you have a sentence where each word is tokenized. Therefore, each word in your sentence is represented with a unique integer (index). In case the list of indices (words) is [1, 5, 9], and you want to encode each of the words with a 50 dimensional vector (embedding), you can do the following:
# The list of tokens
tokens = torch.tensor([0,5,9], dtype=torch.long)
# Define an embedding layer, where you know upfront that in total you
# have 10 distinct words, and you want each word to be encoded with
# a 50 dimensional vector
embedding = torch.nn.Embedding(num_embeddings=10, embedding_dim=50)
# Obtain the embeddings for each of the words in the sentence
embedded_words = embedding(tokens)

Now, to answer your questions:

During the forward pass, the values for each of the tokens in your sentence are going to be obtained in a similar way as the Numpy's indexing works. Because in the backend, this is a differentiable operation, during the backward pass (training), Pytorch is going to compute the gradients for each of the embeddings and readjust them accordingly.

The weights are the embeddings themselves. The word embedding matrix is actually a weight matrix that will be learned during training.

There is no actual function per se. As we defined above, the sentence is already tokenized (each word is represented with a unique integer), and we can just obtain the embeddings for each of the tokens in the sentence.

Finally, as I mentioned the example with the indexing many times, let us try it out.
# Let us assume that we have a pre-trained embedding matrix
pretrained_embeddings = torch.rand(10, 50)
# We can initialize our embedding module from the embedding matrix
embedding = torch.nn.Embedding.from_pretrained(pretrained_embeddings)
# Some tokens
tokens = torch.tensor([1,5,9], dtype=torch.long)

# Token embeddings from the lookup table
lookup_embeddings = embedding(tokens)
# Token embeddings obtained with indexing
indexing_embeddings = pretrained_embeddings[tokens]
# Voila! They are the same
np.testing.assert_array_equal(lookup_embeddings.numpy(), indexing_embeddings.numpy())


Instruction: Average error and standard deviation of error within epoch not correctly updating - PyTorch

Input:
I am attempting to use Stochastic Gradient Descent but I am unsure as to why my error/loss is not decreasing. The information I am using from the train dataframe is the index (each sequence) and the binding affinity, and the goal is to predict the binding affinity. Here is what the head of the dataframe looks like:

For the training, I make a one-hot of a sequence and calculate a score with another matrix, and the goal is to get this score to be as close to the binding affinity as possible (for any given peptide). How I calculate the score and my training loop is shown in my code below but I don't think an explanation is necessary to solve why my error fails to decrease.

#ONE-HOT ENCODING
AA=['A','R','N','D','C','Q','E','G','H','I','L','K','M','F','P','S','T','W','Y','V']
loc=['N','2','3','4','5','6','7','8','9','10','11','C']
aa = "ARNDCQEGHILKMFPSTWYV"
def p_one_hot(seq):
    c2i = dict((c,i) for i,c in enumerate(aa))
    int_encoded = [c2i[char] for char in seq]
    onehot_encoded = list()
    for value in int_encoded:
        letter = [0 for _ in range(len(aa))]
        letter[value] = 1
        onehot_encoded.append(letter)
    return(torch.Tensor(np.transpose(onehot_encoded)))

#INITALIZE TENSORS
a=Var(torch.randn(20,1),requires_grad=True) #initalize similarity matrix - random array of 20 numbers
freq_m=Var(torch.randn(12,20),requires_grad=True)
freq_m.data=(freq_m.data-freq_m.min().data)/(freq_m.max().data-freq_m.min().data)#0 to 1 scaling
optimizer = optim.SGD([torch.nn.Parameter(a), torch.nn.Parameter(freq_m)], lr=1e-6)
loss = nn.MSELoss()

#TRAINING LOOP
epochs = 100
for i in range(epochs):
    #RANDOMLY SAMPLE DATA
    train = all_seq.sample(frac=.03)
    names = train.index.values.tolist()
    affinities = train['binding_affinity']
    print('Epoch: ' + str(i))
    #forward pass
    iteration_loss=[]
    for j, seq in enumerate(names):
        sm=torch.mm(a,a.t()) #make simalirity matrix square symmetric
        freq_m.data=freq_m.data/freq_m.data.sum(1,keepdim=True) #sum of each row must be 1 (sum of probabilities of each amino acid at each position)
        affin_score = affinities[j]
        new_m = torch.mm(p_one_hot(seq), freq_m)
        tss_m = new_m * sm
        tss_score = tss_m.sum()
        sms = sm
        fms = freq_m
        error = loss(tss_score, torch.FloatTensor(torch.Tensor([affin_score])))
        iteration_loss.append(error.item())
        optimizer.zero_grad()
        error.backward()
        optimizer.step()
    mean = statistics.mean(iteration_loss)
    stdev = statistics.stdev(iteration_loss)
    print('Epoch Average Error: ' + str(mean) + '. Epoch Standard Deviation: ' + str(stdev))
    iteration_loss.clear()

After each epoch, I print out the average of all errors for that epoch as well as the standard deviation. Each epoch runs through about 45,000 sequences. However, after 10 epochs I'm still not seeing any improvement with my error and I'm unsure as to why. Here is the output I am seeing:

Are there any ideas as to what I'm doing wrong? I'm new to PyTorch so any help is appreciated! Thank you!

Output: It turns out that casting the optimizer parameters into torch.nn.Parameter() makes the tensors fail to hold on to updates, and removing this now shows a decreasing error.


Instruction: "no module named torch". But installed pytorch 1.3.0 with conda in Ubuntu 18.04.02 Server Edition

Input:
installed pytorch with conda :

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ conda install
pytorch torchvision cpuonly -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

I updated conda:

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ conda update
conda
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

Installed mkl=2019 :

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ conda install
mkl=2019
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ conda list | grep
torch
cpuonly                   1.0                           0    pytorch
facenet-pytorch           0.1.0                    pypi_0    pypi
pytorch                   1.3.0               py3.7_cpu_0  [cpuonly]  pytorch
torchfile                 0.1.0                    pypi_0    pypi
torchvision               0.4.1                  py37_cpu  [cpuonly]  pytorch

But it still says "no module torch" :

(base) (3.8.0/envs/my_virtual_env-3.8.0) marco@pc:~/facenet_pytorch/examples$ python3
Python 3.8.0 (default, Oct 30 2019, 16:20:23)
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>

I discovered that the problem appears only with python 3.8.0 version

(base) marco@pc:~/facenet_pytorch$ python3
Python 3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>>

Ubuntu 18.04.02 Server Edition

Or, may be, it's just a matter of python environments, as you said.
But I do not understand why just activating conda environment, with "conda activate", it doesn't work

Marco

Output:
First create a Conda environment using:

conda create -n pytorch_env python=3 ( you can create with any python version )

Activate the environment using:

conda activate pytorch_env

Now install PyTorch using:

conda install pytorch-cpu torchvision -c pytorch

Go to python shell and import using the command:

import torch


Instruction: How do I extract indices of non-equivalent entries between two tensors?

Input:
I have a tensor with N predictions of N objects' classes, and I have another tensor with the real N target objects' classes. I would like to pull out the tensor indices where my classifier predictions are wrong.

Consider the two following tensors defined as:

import torch
predictions = torch.tensor([ [0], [1], [1], [0], [0], [1] ])
target      = torch.tensor([ [0], [0], [1], [1], [0], [1] ])

I want to find some function where I can pass these two vectors and have returned a list like index_diff = [1, 3]. Does this function exist? My current thoughts are to cast both of these vectors to numpy arrays and then loop through N times and compare each entry at each index, but this seemed a little circuitous to me. Is there an alternative?

Output:
Something like

index_diff = (predictions.flatten() != target.flatten()).nonzero().flatten()

should work.


Instruction: Using the full PyTorch Transformer Module

Input:
I tried asking this question on the PyTorch forums but didn't get any response so I am hoping someone here can help me. Additionally, if anyone has a good example of using the transformer module please share it as the documentation only shows using a simple linear decoder. For the transformer I'm aware that we generally feed in the actual target sequence. Therefore, my first question is that prior to the transformer I have a standard linear layer to transform my time series sequence along with positional encodings. According to the documentation the transformer module code the src and trg sequence need to be the same dimension.
    from torch.nn.modules.transformer import Transformer
    class TransformerTimeSeries(torch.nn.Module):
       def __init__(self, n_time_series, d_model=128):
          super()._init__()
          self.dense_shape = torch.nn.Linear(n_time_series, d_model)
          self.pe = SimplePositionalEncoding(d_model)
          self.transformer = Transformer(d_model, nhead=8)

So I was wondering can I simply do something like this or will this somehow leak information about the target? I'm still not actually sure how loss.backward() works so I'm not sure if this will cause problems.
    def forward(self, x, t):
       x = self.dense_shape(x)
       x = self.pe(x)
       t = self.dense_shape(t)
       t = self.pe(t)
       x = self.transformer(x, t)

Secondly, does the target sequence need any sort of offset? So for instance if I have the time series [0,1,2,3,4,5,6,7] and I want to feed in [0,1,2,3] to predict [4,5,6,7] (tgt)? Would I simply feed it in like that or is it more complicated? Typically BERT and those models have [CLS] and [SEP] tokens to denote the beginning and end of sentences however, for time series I assume I don't need a separator time step.

Output:
loss.backward() traverses through the gradient graph of the model and updates the gradients of each component in the way. You can see the graph using an auxiliary library named PytorchViz. Here is an example of what you can visualize using this library:

Whether you use it or not, it looks like you are using the same dense layer for both the target and the input. Since the dense layer will keep track of the gradients it will calculate the gradient of the target in addition to the gradients input which will indeed cause the model to learn based on the target sequence.
As for your second question. I think feeding the model [0,1,2,3] in order to predict [4,5,6,7] will work fine depending on what data you are using. If you are using periodic signals (ie ecg time series, sinx etc) I think it will do a great job as is with no further complications needed.
However if you want to predict certain events like end of sentence or price prediction (like end of trading day) then you will need to add tokens to create a robust model (not to say that without them it will fail, but it can definitely help the prediction accuracy) .


Instruction: Very simple optim.SGD training loop not working as expected - PyTorch

Input:
I've been reading through the PyTorch documentation and I've been trying to figure out MSELoss and autograd. I tried creating a very simple training loop that takes two random tensors and updates the values in each tensor so that the sum all values in tensor1 plus the sum of all values in tensor2 add up to some target number. In my example I used 100. Here is how I implemented this:

import torch.nn as nn
import torch
import torch.optim as optim

loss = nn.MSELoss()
tensor1 = torch.randn(3, 5, requires_grad=True)
tensor2 = torch.randn(3, 5, requires_grad=True)
target_sum = torch.FloatTensor(torch.Tensor([100]))
optimizer = optim.SGD([nn.Parameter(tensor1), nn.Parameter(tensor2)], lr=1e-4)
print(str(tensor1.sum().item()) + str(tensor2.sum().item()))
for i in range(100):
    pred = tensor1.sum() + tensor2.sum()
    optimizer.zero_grad()
    loss(pred, target_sum).backward
    optimizer.step()
print(str(tensor1.sum().item()) + str(tensor2.sum().item()))

The sums of the tensors before the training loop and after the training loop are the same, but I should be seeing an increase in the sum and have it approach 100. I'm not sure what I'm missing here. I think it  might have something to do with my optimizer as the training loop is almost straight out of the documentation example. I'm new to PyTorch and I appreciate any help!

Output:
Use abs(pred - target_sum) to calculate loss since pred and target_sum are scalars. Also, remove nn.Parameter() wrapper in optim definition.
Here is what I did.

In [22]: import torch.nn as nn
    ...: import torch
    ...: import torch.optim as optim
    ...:
    ...: tensor1 = torch.randn(3, 5, requires_grad=True)
    ...: tensor2 = torch.randn(3, 5, requires_grad=True)
    ...: target_sum = 100
    ...: optimizer = optim.SGD([tensor1, tensor2], lr=1e-2)
    ...: print(tensor1.sum().item() + tensor2.sum().item())
    ...: for i in range(1000):
    ...:     pred = tensor1.sum() + tensor2.sum()
    ...:     optimizer.zero_grad()
    ...:     loss = abs((pred-target_sum))
    ...:     loss.backward()
    ...:     optimizer.step()
    ...: print(tensor1.sum().item()+ tensor2.sum().item())

This results before and after the loop are as follows

-0.777163028717041
100.02290725708008


Instruction: How to get log probabilities in TensorFlow?

Input:
I am trying to convert a pytorch script to tensorflow and I need to get log probabilities from a categorical distribution. But the tensorflow calculated log probabilities is different from pytorch's log prob even after using same seed. This is what I have done so far

import torch
from torch.distributions import Categorical
import tensorflow as tf
import tensorflow_probability as tfp

torch.manual_seed(1)
tf.random.set_seed(1)

probs =[0.4,0.6]
m = Categorical(torch.tensor(probs))
action = m.sample()

n = tfp.distributions.Categorical(probs)
print("pytorch",m.log_prob(action))
print("Tensorflow", tf.math.log(n.prob(action.item())))

Output:
tfp.distributions.Categorical(probs)

takes logs as the default argument. They are being normalized and resulting probabilities of built distribution are [.45, .55].

You need to build tfp distribution as:

 tfp.distributions.Categorical(probs=probs)


Instruction: Why does the pytorch model perform poorly after setting eval()?

Input:
I used pytorch to build a segmentation model that uses the BatchNormalization layer. I found that when I set model.eval() on the test, the test result will be 0. If I don't set model.eval(), it will perform well.

I tried to search for related questions, but I got the conclusion that model.eval() can fix the parameters of BN, but I am still confused about how to solve this problem.

My batchsize is 1 and this is my model:

import torch
import torch.nn as nn

class Encode_Block(nn.Module):
    def __init__(self, in_feat, out_feat):
        super(Encode_Block, self).__init__()

        self.conv1 = Res_Block(in_feat, out_feat)
        self.conv2 = Res_Block_identity(out_feat, out_feat)

    def forward(self, inputs):
        outputs = self.conv1(inputs)
        outputs = self.conv2(outputs)
        return outputs

class Decode_Block(nn.Module):
    def __init__(self, in_feat, out_feat):
        super(Decode_Block, self).__init__()

        self.conv1 = Res_Block(in_feat, out_feat)
        self.conv2 = Res_Block_identity(out_feat, out_feat)

    def forward(self, inputs):
        outputs = self.conv1(inputs)
        outputs = self.conv2(outputs)
        return outputs

class Conv_Block(nn.Module):
    def __init__(self, in_feat, out_feat):
        super(Conv_Block, self).__init__()

        self.conv1 = nn.Sequential(
            nn.Conv2d(in_feat, out_feat, kernel_size=3, stride=1, padding=1),
            nn.LeakyReLU(),
        )

    def forward(self, inputs):
        outputs = self.conv1(inputs)
        return outputs

def conv3x3(in_planes, out_planes, stride=1):
    """3x3 convolution with padding"""
    return nn.Conv2d(
        in_planes, out_planes, kernel_size=3, stride=stride, padding=1, bias=False
    )

def conv1x1(in_planes, out_planes, stride=1):
    """1x1 convolution with padding"""
    return nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False)

class Res_Block(nn.Module):
    def __init__(self, inplanes, planes, stride=1):
        super(Res_Block, self).__init__()
        self.conv_input = conv1x1(inplanes, planes)
        self.conv1 = conv3x3(inplanes, planes, stride)
        self.bn = nn.BatchNorm2d(planes)
        self.relu = nn.LeakyReLU(inplace=True)
        self.conv2 = conv3x3(planes, planes)
        self.conv3 = conv1x1(planes, planes)
        self.stride = stride

    def forward(self, x):
        residual = self.conv_input(x)

        out = self.conv1(x)
        out = self.bn(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn(out)

        out += residual
        out = self.relu(out)

        return out

class Res_Block_identity(nn.Module):
    def __init__(self, inplanes, planes, stride=1):
        super(Res_Block_identity, self).__init__()
        self.conv1 = conv3x3(inplanes, planes, stride)
        self.bn = nn.BatchNorm2d(planes)
        self.relu = nn.LeakyReLU(inplace=True)
        self.conv2 = conv3x3(planes, planes)
        self.conv3 = conv1x1(planes, planes)
        self.stride = stride

    def forward(self, x):
        residual = x

        out = self.conv1(x)
        out = self.bn(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn(out)

        out += residual
        out = self.relu(out)

        return out

class UpConcat(nn.Module):
    def __init__(self, in_feat, out_feat):
        super(UpConcat, self).__init__()

        self.de_conv = nn.ConvTranspose2d(in_feat, out_feat, kernel_size=2, stride=2)

    def forward(self, inputs, down_outputs):
        outputs = self.de_conv(inputs)
        out = torch.cat([down_outputs, outputs], 1)
        return out

class Res_UNet(nn.Module):
    def __init__(self, num_channels=1, num_classes=1):
        super(Res_UNet, self).__init__()
        flt = 64
        self.down1 = Encode_Block(num_channels, flt)
        self.down2 = Encode_Block(flt, flt * 2)
        self.down3 = Encode_Block(flt * 2, flt * 4)
        self.down4 = Encode_Block(flt * 4, flt * 8)
        self.down_pool = nn.MaxPool2d(kernel_size=2)
        self.bottom = Encode_Block(flt * 8, flt * 16)
        self.up_cat1 = UpConcat(flt * 16, flt * 8)
        self.up_conv1 = Decode_Block(flt * 16, flt * 8)
        self.up_cat2 = UpConcat(flt * 8, flt * 4)
        self.up_conv2 = Decode_Block(flt * 8, flt * 4)
        self.up_cat3 = UpConcat(flt * 4, flt * 2)
        self.up_conv3 = Decode_Block(flt * 4, flt * 2)
        self.up_cat4 = UpConcat(flt * 2, flt)
        self.up_conv4 = Decode_Block(flt * 2, flt)
        self.final = nn.Sequential(
            nn.Conv2d(flt, num_classes, kernel_size=1), nn.Sigmoid()
        )

    def forward(self, inputs):
        down1_feat = self.down1(inputs)
        pool1_feat = self.down_pool(down1_feat)
        down2_feat = self.down2(pool1_feat)
        pool2_feat = self.down_pool(down2_feat)
        down3_feat = self.down3(pool2_feat)
        pool3_feat = self.down_pool(down3_feat)
        down4_feat = self.down4(pool3_feat)
        pool4_feat = self.down_pool(down4_feat)

        bottom_feat = self.bottom(pool4_feat)

        up1_feat = self.up_cat1(bottom_feat, down4_feat)
        up1_feat = self.up_conv1(up1_feat)
        up2_feat = self.up_cat2(up1_feat, down3_feat)
        up2_feat = self.up_conv2(up2_feat)
        up3_feat = self.up_cat3(up2_feat, down2_feat)
        up3_feat = self.up_conv3(up3_feat)
        up4_feat = self.up_cat4(up3_feat, down1_feat)
        up4_feat = self.up_conv4(up4_feat)

        outputs = self.final(up4_feat)

        return outputs

The model completely fails to segmentation after setting model.eval(), but the model is good after model.eval() is removed. I am confused about this, and is model.eval() necessary in the test?

Output:
BatchNorm layers keeps running estimates of its computed mean and variance during training model.train(), which are then used for normalization during evaluation model.eval().

Each layer has it own statistics of the mean and variance of its outputs/activations.
Since you are reusing your BatchNorm layer self.bn = nn.BatchNorm2d(planes) multiple times, the statics get mixed up and don't represent the actual mean and variance.
So you should create a new BatchNorm layer for every time you use it.

EDIT: I just read that your batch_size is 1, which could also be the core of your problem: see Tensorflow and Batch Normalization with Batch Size==1 => Outputs all zeros


Instruction: SDKManager Install - No ARM files

Input:
The l4t-base:r32.2.1 base image (jetson) provided by NVIDIA does not have cuda binaries
Pytorch is looking for. I am unable to download source files from Jetpack 4.2.2 (specifically cuda-l4t-repo-10.x.x-arm64.deb) to include within docker build.

I am building a docker container for jetsons. I used the l4t image from NVIDIA, but it is missing some binaries pytorch is looking for (I understand you're supposed to mount some files for CUDA, but I don't want to). I therefore was going to install CUDA 10 directly into the container.

I am trying to download the source files for Jetpack 4.2.2. From what I have read, there should be a file, "cuda-repo-l4t-10-0-local-_arm64.deb". However, I only see
cuda-repo-cross-aarch64-10-0-local-10.0.326_1.0-1_all.deb and cuda-repo-ubuntu1804-10-0-local-10.0.326-410.108_1.0-1_amd64.deb.

I created an Ubuntu 18 container to run the SDK in. I run the below command and get the following files...
2019_06_23_2352-26660552-NVIDIA_Nomad_2019.2.19174.2352_Release_External-L4T.linux.deb,
libopencv-dev_3.3.1-2-gb3f86dcd5_amd64.deb,
Jetson_Linux_R32.2.1_aarch64.tbz2,
libopencv-python_3.3.1-2-gb3f86dcd5_amd64.deb
NVIDIA_VisionWorks_References.zip,
libopencv-samples_3.3.1-2-gb3f86dcd5_amd64.deb
NsightSystems-linux-public-2019.4.1.10-a76094a.deb
libopencv_3.3.1-2-gb3f86dcd5_amd64.deb
Tegra_Linux_Sample-Root-Filesystem_R32.2.1_aarch64.tbz2
libvisionworks-repo_1.6.0.500n_amd64.deb
cuda-repo-cross-aarch64-10-0-local-10.0.326_1.0-1_all.deb
libvisionworks-sfm-repo_0.90.4_amd64.deb
cuda-repo-ubuntu1804-10-0-local-10.0.326-410.108_1.0-1_amd64.deb
libvisionworks-tracking-repo_0.88.2_amd64.deb
devtools_docs.zip
sdkml3_jetpack_l4t_422_rev1_b30.json

sdkmanager --cli downloadonly --user $NV_USER \
--view live --host --logintype devzone --product Jetson --version 4.2 \
--targetos Linux --target $DEVICE_ID --flash all --license accept \
--downloadfolder /tmp/cuda_pkg

Desired results:

- download jetson jetpack ARM specific cuda packages during docker build
- all cuda binaries are available in container

instead of:

root@24081bcd996b:/usr/local/cuda-10.0/lib64# ls
libcudadevrt.a  libcudart_static.a  stubs

should look more like:

libOpenCL.so            libcufft.so.10.0         libnppc.so             libnppif_static.a     libnpps.so.10.0.176
libOpenCL.so.1          libcufft.so.10.0.176     libnppc.so.10.0         libnppig.so           libnpps_static.a
libOpenCL.so.1.0        libcufft_static.a       libnppc.so.10.0.176     libnppig.so.10.0       libnvToolsExt.so
libOpenCL.so.1.0.0      libcufftw.so            libnppc_static.a       libnppig.so.10.0.176   libnvToolsExt.so.1
libaccinj64.so          libcufftw.so.10.0        libnppial.so           libnppig_static.a     libnvToolsExt.so.1.0.0
libaccinj64.so.10.0      libcufftw.so.10.0.176    libnppial.so.10.0       libnppim.so           libnvblas.so
libaccinj64.so.10.0.176  libcufftw_static.a      libnppial.so.10.0.176   libnppim.so.10.0       libnvblas.so.10.0
libcublas.so            libcuinj64.so           libnppial_static.a     libnppim.so.10.0.176   libnvblas.so.10.0.176
libcublas.so.10.0        libcuinj64.so.10.0       libnppicc.so           libnppim_static.a     libnvblas.so.10.0.480
libcublas.so.10.0.176    libcuinj64.so.10.0.176   libnppicc.so.10.0       libnppist.so          libnvgraph.so
libcublas.so.10.0.480    libculibos.a            libnppicc.so.10.0.176   libnppist.so.10.0      libnvgraph.so.10.0
libcublas_device.a      libcurand.so            libnppicc_static.a     libnppist.so.10.0.176  libnvgraph.so.10.0.176
libcublas_static.a      libcurand.so.10.0        libnppicom.so          libnppist_static.a    libnvgraph_static.a
libcudadevrt.a          libcurand.so.10.0.176    libnppicom.so.10.0      libnppisu.so          libnvrtc-builtins.so
libcudart.so            libcurand_static.a      libnppicom.so.10.0.176  libnppisu.so.10.0      libnvrtc-builtins.so.10.0
libcudart.so.10.0        libcusolver.so          libnppicom_static.a    libnppisu.so.10.0.176  libnvrtc-builtins.so.10.0.176
libcudart.so.10.0.176    libcusolver.so.10.0      libnppidei.so          libnppisu_static.a    libnvrtc.so
libcudart_static.a      libcusolver.so.10.0.176  libnppidei.so.10.0      libnppitc.so          libnvrtc.so.10.0
libcudnn.so             libcusolver_static.a    libnppidei.so.10.0.176  libnppitc.so.10.0      libnvrtc.so.10.0.176
libcudnn.so.7           libcusparse.so          libnppidei_static.a    libnppitc.so.10.0.176  stubs
libcudnn.so.7.6.3       libcusparse.so.10.0      libnppif.so            libnppitc_static.a
libcudnn_static.a       libcusparse.so.10.0.176  libnppif.so.10.0        libnpps.so
libcufft.so             libcusparse_static.a    libnppif.so.10.0.176    libnpps.so.10.0

Output:
You can locate the url in the sdkml3_jetpack_l4t_422_rev1_b30.json to download the debain files that you are looking for .

Hope this helps


Instruction: ModuleNotFoundError: No module named 'tools.nnwrap' (macOS)

Input:
When installing PyTorch using pip install torch I get the following:

from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'

I'm on macOS 10.14.4, Python 3.8.

I read through the thread here but it didn't help.

UPDATE 13/11/19

The accepted answer works.

As a side note, I finally setup my local environment using Anaconda, despite the meaty installation creating an enviroment has been easy:

#create the env
conda create -n mytensorflowenv python=3.7

#activate it
conda activate mytensorflowenv

then run the PyTorch installation for Conda:

conda install pytorch torchvision -c pytorch

All smooth now.

Output:
as per https://github.com/pytorch/pytorch/issues/29090

Python 3.8 binaries are not yet available.

You'll need to downgrade your python version to 3.7


Instruction: GKE - GPU nvidia - cuda drivers dont work

Input:
I have setup a kubernetes node with a nvidia tesla k80 and followed this tutorial to try to run a pytorch docker image with nvidia drivers and cuda drivers working.

I have managed to install the nvidia daemonsets and i can now see the following pods:

nvidia-driver-installer-gmvgt
nvidia-gpu-device-plugin-lmj84

The problem is that even while using the recommendend image nvidia/cuda:10.0-runtime-ubuntu18.04 i still can't find the nvidia drivers inside my pod:

root@pod-name-5f6f776c77-87qgq:/app# ls /usr/local/
bin  cuda  cuda-10.0  etc  games  include  lib  man  sbin  share  src

But the tutorial mention:

  CUDA libraries and debug utilities are made available inside the container at /usr/local/nvidia/lib64 and /usr/local/nvidia/bin, respectively.

I have also tried to test if cuda was working through torch.cuda.is_available() but i get False as a return value.

Many help in advance for your help

Output:
Ok so i finally made nvidia drivers work.

It is mandatory to set a ressource limit to access the nvidia driver, which is weird considering either way my pod was on the right node with the nvidia drivers installed..

This made the nvidia folder accessible, but im'still unable to make the cuda install work with pytorch 1.3.0 .. [ issue here ]


Instruction: How to get probability of each image belonging to a class

Input:
I am new to Pytorch. I trained and tested a linear classifier (nn.Linear) with an image data set that has 8 categories with the batch_size = 35.
While testing, I wanted to see what is the probability of the given image belonging to any of these 8 classes. That is why I printed output.data variable. But these numbers are above 1 and they are not summing into 1. (I attached the testing code)
So, my question is what does those numbers mean?

Thanks!

correct = 0
total = 0
with torch.no_grad():
    for data in dataloaders['test']:
        images, labels = data[0].to(device), data[1].to(device)
        outputs = model(images)
        print(outputs.data)

        _, predicted = torch.max(outputs.data, 1)
        print(predicted)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
print('Accuracy of the network on the 4000 test images: %d %%' % (
100 * correct / total))

Output:
You get logits as output of your neural network.

Use torch.nn.Softmax on output to squash the values into (0,1) range.

BTW. You network should output logits as pytorch's losses (torch.nn.CrossEntropyLoss in this case) are designed to work with them while being numerically stable.


Instruction: How to save intermediate weights when training a neural network

Input:
I am training a neural network using pytorch, and I want to save the weights at every iteration. In other words, I want to create a list that contains all the weights the neural network has had during training.

I did the following:

for i, (images, labels) in enumerate(train_loader):

     (.....code that is used to train the model here.....)

     weight = model.fc2.weight.detach().numpy()
     weights_list.append(weight)

When I then print the entries of the list 'weights_list', I notice that they are all the same, which cannot be true, because I have printed the weights during the training and they do change (and the network actually does learn, so they have to).
My guess is that every entry of the list is actually a pointer to the weights of the network at the moment the list is checked. So:

1) Is my guess correct?
2) How can I solve this problem?

Thank you!

Output:
The functionality to save and load weights is built in. To save to a file you can use

torch.save('checkpoint.pt', model.state_dict())

and to load you can use

model.load_state_dict(torch.load('checkpoint.pt'))

That said, converting to numpy doesn't necessarily create a copy. For example if you have a numpy array y and want to create a copy you could use

x = numpy.copy(y)


Instruction: GKE - Unable to make cuda work with pytorch

Input:
I have setup a kubernetes node with a nvidia tesla k80 and followed this tutorial to try to run a pytorch docker image with nvidia drivers and cuda drivers working.

My nvidia drivers and cuda drivers are all accessible inside my pod at /usr/local:

$> ls /usr/local
bin  cuda  cuda-10.0  etc  games  include  lib  man  nvidia  sbin  share  src

And my GPU is also recongnized by my image nvidia/cuda:10.0-runtime-ubuntu18.04:

$> /usr/local/nvidia/bin/nvidia-smi
Fri Nov  8 16:24:35 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   73C    P8    35W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

But after installing pytorch 1.3.0 i'm not able to make pytorch recognize my cuda installation even with LD_LIBRARY_PATH set to /usr/local/nvidia/lib64:/usr/local/cuda/lib64:

$> python3 -c "import torch; print(torch.cuda.is_available())"
False

$> python3
Python 3.6.8 (default, Oct  7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print ('\t\ttorch.cuda.current_device()    =', torch.cuda.current_device())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 386, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 192, in _lazy_init
    _check_driver()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 111, in _check_driver
    of the CUDA driver.""".format(str(torch._C._cuda_getDriverVersion())))
AssertionError:
The NVIDIA driver on your system is too old (found version 10000).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.

The error above is strange because my cuda version for my image is 10.0 and Google GKE mentions that:

  The latest supported CUDA version is 10.0

Also, it's GKE's daemonsets that automatically installs NVIDIA drivers

  After adding GPU nodes to your cluster, you need to install NVIDIA's device drivers to the nodes.

  Google provides a DaemonSet that automatically installs the drivers for you.
  Refer to the section below for installation instructions for Container-Optimized OS (COS) and Ubuntu nodes.

  To deploy the installation DaemonSet, run the following command:
  kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

I have tried everything i could think of, without success...

Output:
I have resolved my problem by downgrading my pytorch version by buildling my docker images from pytorch/pytorch:1.2-cuda10.0-cudnn7-devel.

I still don't really know why before it was not working as it should otherwise then by guessing that pytorch 1.3.0 is not compatible with cuda 10.0.


Instruction: Pytorch module not found

Input:
I'm using Python 3.6, and trying to use Pytorch. I've uninstalled it using pip3, and re-installed it

Hugos-MacBook-Pro-2:project hugokitano$ pip3 install torch
Requirement already satisfied: torch in /usr/local/lib/python3.7/site-packages (1.3.1)
Requirement already satisfied: numpy in /Users/hugokitano/Library/Python/3.7/lib/python/site-packages (from torch) (1.17.2)

However, when I try to import torch, the module is not found. I've also tried to install via conda, running

  conda install pytorch torchvision -c pytorch

and it was successful, installing to

  environment location: /Users/hugokitano/anaconda

However, "import torch" still doesn't work. Any thoughts? Thanks!

Output:
It is wise to use environments instead of installs packages on your base. Try do the following:

conda create -n deep7 -c pytorch python=3.7 pytorch torchvision
conda activate deep7
python -c "import torch"

We have created an environment with name deep7, we use pytorch channel to install pytorch in Python 3.7. After that we activate the environment and test if the import works. If it did you will see no error.

To use pytorch, you will have to activate your environment:
conda activate deep7 and to deactivate conda deactivate. You can add libraries with conda install -n deep7 <package name>

Happy coding

BTW: if you want Python 3.6, do the same, change all the 7 above to 6 :)


Instruction: What is the algebraic expression for PyTorch's ConvTranspose2d's output shape?

Input:
When using PyTorch's ConvTranspose2d as such:

w = 5 # input width
h = 5 # output height
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=k, stride=s, padding=p)

What is the formula for the dimensions of the output in each channel? I tried a few examples and cannot derive the pattern. For some reason adding padding seems to shrink the output size (example starts with 5 x 5 as above):

# yields an 11 x 11 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=3, stride=2, padding=0)

# yields a 7 x 7 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=3, stride=2, padding=2)

Using a larger kernel or stride both increase (expected) but not at the rate that I expected:

# yields an 11 x 11 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=3, stride=2, padding=0)

# yields a 13 x 13 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=5, stride=2, padding=0)

# yields a 15 x 15 image
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=3, stride=3, padding=0)

I'm sure there's a pretty simple math equation involving w, h, k, s, p but I can't find it in the documentation and I haven't been able to derive it myself. Normally I wouldn't ask for a math equation, but it completely affects the ability of a CNN to compile and generate the correct size. Thanks in advance!

Output:
The formula to calculate ConvTranspose2d output sizes is mentioned on the documentation page:

H_out = (H_in−1)*stride[0] − 2×padding[0] + dilation[0]×(kernel_size[0]−1) + output_padding[0] + 1
W_out = (Win−1)×stride[1] − 2×padding[1] + dilation[1]×(kernel_size[1]−1)  + output_padding[1] + 1

By default, stride=1, padding=0, and output_padding=0.
For example, for
nn.ConvTranspose2d(in_channels, out_channels, kernel_size=3, stride=2, padding=0)

the H_out will be
H_out = (5-1)*2 - 2*0 + 1*(3-1) + 0 + 1 = 11


Instruction: How should I understand the nn.Embeddings arguments num_embeddings and embedding_dim?

Input:
I'm trying to get used to the Embedding class in the PyTorch nn module.

I've noticed that quite a few other people have had the same problem as myself, and therefore posted questions on the PyTorch discussion forum and on Stack Overflow, but I'm still having some confusion.

According to the official documentation, the arguments that are passed are num_embeddings and embedding_dim which each refer to how large our dictionary (or vocabulary) is and how many dimensions we want our embeddings to be, respectively.

What I'm confused about is how exactly I should interpret those. For example, the small practice code that I ran:

import torch
import torch.nn as nn

embedding = nn.Embedding(num_embeddings=10, embedding_dim=3)

a = torch.LongTensor([[1, 2, 3, 4], [4, 3, 2, 1]]) # (2, 4)

b = torch.LongTensor([[1, 2, 3], [2, 3, 1], [4, 5, 6], [3, 3, 3], [2, 1, 2],
                      [6, 7, 8], [2, 5, 2], [3, 5, 8], [2, 3, 6], [8, 9, 6],
                      [2, 6, 3], [6, 5, 4], [2, 6, 5]]) # (13, 3)

c = torch.LongTensor([[1, 2, 3, 2, 1, 2, 3, 3, 3, 3, 3],
                      [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]]) # (2, 11)

When I run a, b, and c through the embedding variable, I get embedded results of shapes (2, 4, 3), (13, 3, 3), (2, 11, 3).

What's confusing me is that I thought of the number of samples we have exceeds the predefined number of embeddings, we should get an error? Since the embedding I've defined has 10 embeddings, shouldn't b give me an error since it is a tensor containing 13 words of dimension 3?

Output:
In your case, here is how your input tensor are interpreted:

a = torch.LongTensor([[1, 2, 3, 4], [4, 3, 2, 1]]) # 2 sequences of 4 elements

Moreover, this is how your embedding layer is interpreted:

embedding = nn.Embedding(num_embeddings=10, embedding_dim=3) # 10 distinct elements and each those is going to be embedded in a 3 dimensional space

So, it doesn't matter if your input tensor has more than 10 elements, as long as they are in the range [0, 9]. For example, if we create a tensor of two elements such as:

d = torch.LongTensor([[1, 10]]) # 1 sequence of 2 elements

We would get the following error when we pass this tensor through the embedding layer:

  RuntimeError: index out of range: Tried to access index 10 out of table with 9 rows

To summarize num_embeddings is total number of unique elements in the vocabulary, and embedding_dim is the size of each embedded vector once passed through the embedding layer. Therefore, you can have a tensor of 10+ elements, as long as each element in the tensor is in the range [0, 9], because you defined a vocabulary size of 10 elements.