Instruction: CoreML: creating a custom layer for ONNX RandomNormal

Input:
I've trainined a VAE that in PyTorch that I need to convert to CoreML. From this thread PyTorch VAE fails conversion to onnx I was able to get the ONNX model to export, however, this just pushed the problem one step further to the ONNX-CoreML stage.

The original function that contains the torch.randn() call is the reparametrize func:

def reparametrize(self, mu, logvar):
    std = logvar.mul(0.5).exp_()
    if self.have_cuda:
        eps = torch.randn(self.bs, self.nz, device='cuda')
    else:
        eps = torch.randn(self.bs, self.nz)
    return eps.mul(std).add_(mu)

The solution is, of course, to create a custom layer, but I'm having problems creating a layer with no inputs (i.e., it's just a randn() call).

I can get the CoreML conversion to complete with this def:

def convert_randn(node):
    params = NeuralNetwork_pb2.CustomLayerParams()
    params.className = "RandomNormal"
    params.description = "Random normal distribution generator"
    params.parameters["dtype"].intValue = node.attrs.get('dtype', 1)
    params.parameters["bs"].intValue = node.attrs.get("shape")[0]
    params.parameters["nz"].intValue = node.attrs.get("shape")[1]
    return params

I do the conversion with:

coreml_model = convert(onnx_model, add_custom_layers=True,
    image_input_names = ['input'],
    custom_conversion_functions={"RandomNormal": convert_randn})

I should also note that, at the completion of the mlmodel export, the following is printed:

Custom layers have been added to the CoreML model corresponding to the
following ops in the onnx model:
1/1: op type: RandomNormal, op input names and shapes: [], op output
names and shapes: [('62', 'Shape not available')]

Bringing the .mlmodel into Xcode complains that Layer '62' of type 500 has 0 inputs but expects at least 1. So I'm wondering how to specify a kind of "dummy" input to the layer, since it doesn't actually have an input -- it's just a wrapper around torch.randn() (or, more specifically, the onnx RandonNormal op). I should clarify that I do need the whole VAE, not just the decoder, as I'm actually using the entire process to "error correct" my inputs (i.e., the encoder estimates my z vector, based on an input, then the decoder generates the closest generalizable prediction of the input).

Any help greatly appreciated.

UPDATE: Okay, I finally got a version to load in Xcode (thanks to @MattijsHollemans and his book!). The originalConversion.mlmodel is the initial output of converting my model from ONNX to CoreML. To this, I had to manually insert the input for the RandomNormal layer. I made it (64, 28, 28) for no great reason — I know my batch size is 64, and my inputs are 28 x 28 (but presumably it could also be (1, 1, 1), since it's a "dummy"):

spec = coremltools.utils.load_spec('originalConversion.mlmodel')
nn = spec.neuralNetwork
layers = {l.name:i for i,l in enumerate(nn.layers)}
layer_idx = layers["62"] # '62' is the name of the layer -- see above
layer = nn.layers[layer_idx]
layer.input.extend(["dummy_input"])

inp = spec.description.input.add()
inp.name = "dummy_input"
inp.type.multiArrayType.SetInParent()
spec.description.input[1].type.multiArrayType.shape.append(64)
spec.description.input[1].type.multiArrayType.shape.append(28)
spec.description.input[1].type.multiArrayType.shape.append(28)
spec.description.input[1].type.multiArrayType.dataType = ft.ArrayFeatureType.DOUBLE

coremltools.utils.save_spec(spec, "modelWithInsertedInput.mlmodel")

This loads in Xcode, but I have yet to test the functioning of the model in my app. Since the additional layer is simple, and the input is literally a bogus, non-functional input (just to keep Xcode happy), I don't imagine it will be a problem, but I'll post again if it doesn't run properly.

UPDATE 2: Unfortunately, the model doesn't load at runtime. It fails with [espresso] [Espresso::handle_ex_plan] exception=Failed in 2nd reshape after missing custom layer info. What I find very strange and confusing is that, inspecting model.espresso.shape, I see that almost every node has a shape like:

"62" : {
  "k" : 0,
  "w" : 0,
  "n" : 0,
  "seq" : 0,
  "h" : 0
}

I have two question/concerns: 1) Most obviously, why are all the values zero (this is the case with all but the input nodes), and 2) Why does it appear to be a sequential model, when it's just a fairly conventional VAE? Opening model.espresso.shape for a fully-functioning GAN in the same app, I see that the nodes are of the format:

"54" : {
  "k" : 256,
  "w" : 16,
  "n" : 1,
  "h" : 16
}

That is, they contain reasonable shape info, and they don't have seq fields.

Very, very confused...

UPDATE 3: I've also just noticed in the compiler report the error: IMPORTANT: new sequence length computation failed, falling back to old path. Your compilation was sucessful, but please file a radar on Core ML | Neural Networks and attach the model that generated this message.

Here's the original PyTorch model:

class VAE(nn.Module):
def __init__(self, bs, nz):
    super(VAE, self).__init__()

    self.nz = nz
    self.bs = bs

    self.encoder = nn.Sequential(
        # input is (nc) x 28 x 28
        nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
        nn.LeakyReLU(0.2, inplace=True),
        # size = (ndf) x 14 x 14
        nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
        nn.BatchNorm2d(ndf * 2),
        nn.LeakyReLU(0.2, inplace=True),
        # size = (ndf*2) x 7 x 7
        nn.Conv2d(ndf * 2, ndf * 4, 3, 2, 1, bias=False),
        nn.BatchNorm2d(ndf * 4),
        nn.LeakyReLU(0.2, inplace=True),
        # size = (ndf*4) x 4 x 4
        nn.Conv2d(ndf * 4, 1024, 4, 1, 0, bias=False),
        nn.LeakyReLU(0.2, inplace=True),
    )

    self.decoder = nn.Sequential(
        # input is Z, going into a convolution
        nn.ConvTranspose2d(     1024, ngf * 8, 4, 1, 0, bias=False),
        nn.BatchNorm2d(ngf * 8),
        nn.ReLU(True),
        # size = (ngf*8) x 4 x 4
        nn.ConvTranspose2d(ngf * 8, ngf * 4, 3, 2, 1, bias=False),
        nn.BatchNorm2d(ngf * 4),
        nn.ReLU(True),
        # size = (ngf*4) x 8 x 8
        nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
        nn.BatchNorm2d(ngf * 2),
        nn.ReLU(True),
        # size = (ngf*2) x 16 x 16
        nn.ConvTranspose2d(ngf * 2,     nc, 4, 2, 1, bias=False),
        nn.Sigmoid()
    )

    self.fc1 = nn.Linear(1024, 512)
    self.fc21 = nn.Linear(512, nz)
    self.fc22 = nn.Linear(512, nz)

    self.fc3 = nn.Linear(nz, 512)
    self.fc4 = nn.Linear(512, 1024)

    self.lrelu = nn.LeakyReLU()
    self.relu = nn.ReLU()

def encode(self, x):
    conv = self.encoder(x);
    h1 = self.fc1(conv.view(-1, 1024))
    return self.fc21(h1), self.fc22(h1)

def decode(self, z):
    h3 = self.relu(self.fc3(z))
    deconv_input = self.fc4(h3)
    deconv_input = deconv_input.view(-1,1024,1,1)
    return self.decoder(deconv_input)

def reparametrize(self, mu, logvar):
    std = logvar.mul(0.5).exp_()
    eps = torch.randn(self.bs, self.nz, device='cuda') # needs custom layer!
    return eps.mul(std).add_(mu)

def forward(self, x):
    # print("x", x.size())
    mu, logvar = self.encode(x)
    z = self.reparametrize(mu, logvar)
    decoded = self.decode(z)
    return decoded, mu, logvar

Output:
To add an input to your Core ML model, you can do the following from Python:

import coremltools
spec = coremltools.utils.load_spec("YourModel.mlmodel")

nn = spec.neuralNetworkClassifier  # or just spec.neuralNetwork

layers = {l.name:i for i,l in enumerate(nn.layers)}
layer_idx = layers["your_custom_layer"]
layer = nn.layers[layer_idx]
layer.input.extend(["dummy_input"])

inp = spec.description.input.add()
inp.name = "dummy_input"
inp.type.doubleType.SetInParent()

coremltools.utils.save_spec(spec, "NewModel.mlmodel")

Here, "your_custom_layer" is the name of the layer you want to add the dummy input to. In your model it looks like it's called 62. You can look at the layers dictionary to see the names of all the layers in the model.

Notes:

If your model is not a classifier, use nn = spec.neuralNetwork instead of neuralNetworkClassifier.
I made the new dummy input have the type "double". That means your custom layer gets a double value as input.
You need to specify a value for this dummy input when using the model.


Instruction: Activation gradient penalty

Input:
Here's a simple neural network, where I’m trying to penalize the norm of activation gradients:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=5)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5)
        self.pool = nn.MaxPool2d(2, 2)
        self.relu = nn.ReLU()
        self.linear = nn.Linear(64 * 5 * 5, 10)

    def forward(self, input):
        conv1 = self.conv1(input)
        pool1 = self.pool(conv1)
        self.relu1 = self.relu(pool1)
        self.relu1.retain_grad()
        conv2 = self.conv2(relu1)
        pool2 = self.pool(conv2)
        relu2 = self.relu(pool2)
        self.relu2 = relu2.view(relu2.size(0), -1)
        self.relu2.retain_grad()
        return self.linear(relu2)

model = Net()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

for i in range(1000):
    output = model(input)
    loss = nn.CrossEntropyLoss()(output, label)
    optimizer.zero_grad()
    loss.backward(retain_graph=True)

    grads = torch.autograd.grad(loss, [model.relu1, model.relu2], create_graph=True)

    grad_norm = 0
    for grad in grads:
        grad_norm += grad.pow(2).sum()

    grad_norm.backward()

    optimizer.step()

However, it does not produce the desired regularization effect. If I do the same thing for weights (instead of activations), it works well. Am I doing this right (in terms of pytorch machinery)? Specifically, what happens in grad_norm.backward() call? I just want to make sure the weight gradients are updated, and not activation gradients. Currently, when I print out gradients for weights and activations immediately before and after that line, both change - so I’m not sure what’s going on.

Output:
I think your code ends up computing some of the gradients twice in each step. I also suspect it actually never zeroes out the activation gradients, so they accumulate across steps.

In general:

x.backward() computes gradient of x wrt. computation graph leaves (e.g. weight tensors and other variables), as well as wrt. nodes explicitly marked with retain_grad(). It accumulates the computed gradient in tensors' .grad attributes.
autograd.grad(x, [y, z]) returns gradient of x wrt. y and z regardless of whether they would normally retain grad or not. By default, it will also accumulate gradient in all leaves' .grad attributes. You can prevent this by passing only_inputs=True.

I prefer to use backward() only for the optimization step, and autograd.grad() whenever my goal is to obtain "reified" gradients as intermediate values for another computation. This way, I can be sure that no unwanted gradients remain lying around in tensors' .grad attributes after I'm done with them.

import torch
from torch import nn
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=5)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5)
        self.pool = nn.MaxPool2d(2, 2)
        self.relu = nn.ReLU()
        self.linear = nn.Linear(64 * 5 * 5, 10)

    def forward(self, input):
        conv1 = self.conv1(input)
        pool1 = self.pool(conv1)
        self.relu1 = self.relu(pool1)
        conv2 = self.conv2(self.relu1)
        pool2 = self.pool(conv2)
        self.relu2 = self.relu(pool2)
        relu2 = self.relu2.view(self.relu2.size(0), -1)
        return self.linear(relu2)

model = Net()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
grad_penalty_weight = 10.

for i in range(1000000):
    # Random input and labels; we're not really learning anything
    input = torch.rand(1, 3, 32, 32)
    label = torch.randint(0, 10, (1,))

    output = model(input)
    loss = nn.CrossEntropyLoss()(output, label)

    # This is where the activation gradients are computed
    # only_inputs is optional here, since we're going to call optimizer.zero_grad() later
    # But it makes clear that we're *only* interested in the activation gradients at this point
    grads = torch.autograd.grad(loss, [model.relu1, model.relu2], create_graph=True, only_inputs=True)

    grad_norm = 0
    for grad in grads:
        grad_norm += grad.pow(2).sum()

    optimizer.zero_grad()
    loss = loss + grad_norm * grad_penalty_weight
    loss.backward()
    optimizer.step()

This code appears to work, in that the activation gradients do get smaller.
I cannot comment on the viability of this technique as a regularization method.


Instruction: My numpy and pytorch codes have totally different results

Input:
I wanted to calculate the sum of 1st to K-th power of an array and equally calculate the sum of 1st to k-th power of a tensor. I found out that the following codes and their results are totally different and I don't know why.

I debugged the code and I know that the results are equal in the first round.

Numpy code:

adj_k_prob = adj_prob
adj_k_pow =  adj_prob
for i in range(K):
    adj_k_pow = np.matmul(adj_prob, adj_k_pow)
    adj_k_prob += adj_k_pow

Pytorch code:

adj_k_prob = adj_prob_tensor
adj_k_pow = adj_prob_tensor
for i in range(K):
    adj_k_pow = torch.matmul(adj_prob_tensor, adj_k_pow)
    adj_k_prob += adj_k_pow

The value of adj_prob_tensor and adj_prob at the start of loop are as follow:

tensor([[0.0000, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429, 0.1429],
    [0.2500, 0.0000, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.2500, 0.2500, 0.0000, 0.2500, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000, 0.2500],
    [0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5000, 0.0000],
    [0.5000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.5000, 0.0000],
    [0.3333, 0.0000, 0.0000, 0.0000, 0.3333, 0.3333, 0.0000, 0.0000],
    [0.2500, 0.2500, 0.2500, 0.2500, 0.0000, 0.0000, 0.0000, 0.0000]])

Is there anything that I should check for it?

Output: The problem was in using assignment. I should have used .clone() for pytorch and .copy() in numpy.


Instruction: Concat tensors in PyTorch

Input:
I have a tensor called data of the shape [128, 4, 150, 150] where 128 is the batch size, 4 is the number of channels, and the last 2 dimensions are height and width. I have another tensor called fake of the shape [128, 1, 150, 150].

I want to drop the last list/array from the 2nd dimension of data; the shape of data would now be [128, 3, 150, 150]; and concatenate it with fake giving the output dimension of the concatenation as [128, 4, 150, 150].

Basically, in other words, I want to concatenate the first 3 dimensions of data with fake to give a 4-dimensional tensor.

I am using PyTorch and came across the functions torch.cat() and torch.stack()

Here is a sample code I've written:

fake_combined = []
        for j in range(batch_size):
            fake_combined.append(torch.stack((data[j][0].to(device), data[j][1].to(device), data[j][2].to(device), fake[j][0].to(device))))
fake_combined = torch.tensor(fake_combined, dtype=torch.float32)
fake_combined = fake_combined.to(device)

But I am getting an error in the line:

fake_combined = torch.tensor(fake_combined, dtype=torch.float32)

The error is:

ValueError: only one element tensors can be converted to Python scalars

Also, if I print the shape of fake_combined, I get the output as [128,] instead of [128, 4, 150, 150]

And when I print the shape of fake_combined[0], I get the output as [4, 150, 150], which is as expected.

So my question is, why am I not able to convert the list to tensor using torch.tensor(). Am I missing something? Is there any better way to do what I intend to do?

Any help will be appreciated! Thanks!

Output:
You could also just assign to that particular dimension.

orig = torch.randint(low=0, high=10, size=(2,3,2,2))
fake = torch.randint(low=111, high=119, size=(2,1,2,2))
orig[:,[2],:,:] = fake

Original Before

tensor([[[[0, 1],
      [8, 0]],

     [[4, 9],
      [6, 1]],

     [[8, 2],
      [7, 6]]],

    [[[1, 1],
      [8, 5]],

     [[5, 0],
      [8, 6]],

     [[5, 5],
      [2, 8]]]])

Fake

tensor([[[[117, 115],
      [114, 111]]],

    [[[115, 115],
      [118, 115]]]])

Original After

tensor([[[[  0,   1],
      [  8,   0]],

     [[  4,   9],
      [  6,   1]],

     [[117, 115],
      [114, 111]]],

    [[[  1,   1],
      [  8,   5]],

     [[  5,   0],
      [  8,   6]],

     [[115, 115],
      [118, 115]]]])

Hope this helps! :)


Instruction: Calculating Variance in Pytorch on Spatial axis

Input:
I am trying to calculate variance in Pytorch but unable to do on multiple axis.

I have similar thing done in Tensorflow but unable to do it on Pytorch as torch.var function takes int as dimension instead of axes.
Below code is channel last code, I expect axes=[2,3]

Lambda(lambda x: tf.nn.moments(x, axes=[1, 2]))

For example, if input_dims = (5, 10, 25, 25) then output_dims should be (5,10, 1, 1).

Output:
One thing you can do is to use tensor.view() to flatten all the dimensions that you want to calculate the variance for into one dimension before you apply the var() method:

torch.var(x.view(x.shape[0], x.shape[1], 1, -1,), dim=3, keepdim=True)

I used keepdim=True to keep the dimension that we calculate the variance for to get the desired output shape.


Instruction: Pytorch: How to create an update rule that doesn't come from derivatives?

Input:
I want to implement the following algorithm, taken from this book, section 13.6:

I don't understand how to implement the update rule in pytorch (the rule for w is quite similar to that of theta).

As far as I know, torch requires a loss for loss.backwward().

This form does not seem to apply for the quoted algorithm.

I'm still certain there is a correct way of implementing such update rules in pytorch.

Would greatly appreciate a code snippet of how the w weights should be updated, given that V(s,w) is the output of the neural net, parameterized by w.

EDIT: Chris Holland suggested a way to implement, and I implemented it. It does not converge on Cartpole, and I wonder if I did something wrong.

The critic does converge on the solution to the function gamma*f(n)=f(n)-1 which happens to be the sum of the series gamma+gamma^2+...+gamma^inf
meaning, gamma=1 diverges. gamma=0.99 converges on 100, gamma=0.5 converges on 2 and so on. Regardless of the actor or policy.

The code:

def _update_grads_with_eligibility(self, is_critic, delta, discount, ep_t):
    gamma = self.args.gamma
    if is_critic:
        params = list(self.critic_nn.parameters())
        lamb = self.critic_lambda
        eligibilities = self.critic_eligibilities
    else:
        params = list(self.actor_nn.parameters())
        lamb = self.actor_lambda
        eligibilities = self.actor_eligibilities

    is_episode_just_started = (ep_t == 0)
    if is_episode_just_started:
        eligibilities.clear()
        for i, p in enumerate(params):
            if not p.requires_grad:
                continue
            eligibilities.append(torch.zeros_like(p.grad, requires_grad=False))

    # eligibility traces
    for i, p in enumerate(params):

        if not p.requires_grad:
            continue
        eligibilities[i][:] = (gamma * lamb * eligibilities[i]) + (discount * p.grad)
        p.grad[:] = delta.squeeze() * eligibilities[i]

and

expected_reward_from_t = self.critic_nn(s_t)
probs_t = self.actor_nn(s_t)
expected_reward_from_t1 = torch.tensor([[0]], dtype=torch.float)
if s_t1 is not None:  # s_t is not a terminal state, s_t1 exists.
    expected_reward_from_t1 = self.critic_nn(s_t1)

delta = r_t + gamma * expected_reward_from_t1.data - expected_reward_from_t.data

negative_expected_reward_from_t = -expected_reward_from_t
self.critic_optimizer.zero_grad()
negative_expected_reward_from_t.backward()
self._update_grads_with_eligibility(is_critic=True,
                                    delta=delta,
                                    discount=discount,
                                    ep_t=ep_t)
self.critic_optimizer.step()

EDIT 2:
Chris Holland's solution works. The problem originated from a bug in my code that caused the line

if s_t1 is not None:
    expected_reward_from_t1 = self.critic_nn(s_t1)

to always get called, thus expected_reward_from_t1 was never zero, and thus no stopping condition was specified for the bellman equation recursion.

With no reward engineering, gamma=1, lambda=0.6, and a single hidden layer of size 128 for both actor and critic, this converged on a rather stable optimal policy within 500 episodes.

Even faster with gamma=0.99, as the graph shows (best discounted episode reward is about 86.6).

BIG thank you to @Chris Holland, who "gave this a try"

Output:
I am gonna give this a try.

.backward() does not need a loss function, it just needs a differentiable scalar output. It approximates a gradient with respect to the model parameters. Let's just look at the first case the update for the value function.

We have one gradient appearing for v, we can approximate this gradient by

v = model(s)
v.backward()

This gives us a gradient of v which has the dimension of your model parameters. Assuming we already calculated the other parameter updates, we  can calculate the actual optimizer update:

for i, p in enumerate(model.parameters()):
    z_theta[i][:] = gamma * lamda * z_theta[i] + l * p.grad
    p.grad[:] = alpha * delta * z_theta[i]

We can then use opt.step() to update the model parameters with the adjusted gradient.


Instruction: Column-dependent bounds in torch.clamp

Input:
I would like to do something similar to np.clip on PyTorch tensors on a 2D array. More specifically, I would like to clip each column in a specific range of value (column-dependent). For example, in numpy, you could do:

x = np.array([-1,10,3])
low = np.array([0,0,1])
high = np.array([2,5,4])
clipped_x = np.clip(x, low, high)

clipped_x == np.array([0,5,3]) # True

I found torch.clamp, but unfortunately it does not support multidimensional bounds (only one scalar value for the entire tensor). Is there a "neat" way to extend that function to my case?

Thanks!

Output:
Not as neat as np.clip, but you can use torch.max and torch.min:

In [1]: x
Out[1]:
tensor([[0.9752, 0.5587, 0.0972],
        [0.9534, 0.2731, 0.6953]])

Setting the lower and upper bound per column

l = torch.tensor([[0.2, 0.3, 0.]])
u = torch.tensor([[0.8, 1., 0.65]])

Note that the lower bound l and upper bound u are 1-by-3 tensors (2D with singleton dimension). We need these dimensions for l and u to be broadcastable to the shape of x.
Now we can clip using min and max:

clipped_x = torch.max(torch.min(x, u), l)

Resulting with

tensor([[0.8000, 0.5587, 0.0972],
        [0.8000, 0.3000, 0.6500]])


Instruction: PyTorch: What's the difference between state_dict and parameters()?

Input:
In order to access a model's parameters in pytorch, I saw two methods:
using state_dict and using parameters()
I wonder what's the difference, or if one is good practice and the other is bad practice.
Thanks

Output:
The parameters() only gives the module parameters i.e. weights and biases.

  Returns an iterator over module parameters.

You can check the list of the parameters as follows:

for name, param in model.named_parameters():
    if param.requires_grad:
        print(name)

On the other hand, state_dict  returns a dictionary containing a whole state of the module. Check its source code that contains not just the call to parameters but also buffers, etc.

  Both parameters and persistent buffers (e.g. running averages) are included. Keys are the corresponding parameter and buffer names.

Check all keys that state_dict contains using:

model.state_dict().keys()

For example, in state_dict, you'll find entries like bn1.running_mean and running_var, which are not present in .parameters().

If you only want to access parameters, you can simply use .parameters(), while for purposes like saving and loading model as in transfer learning, you'll need to save state_dict not just parameters.


Instruction: Input dimension error on pytorch's forward check

Input:
I am creating an RNN with pytorch, it looks like this:

class MyRNN(nn.Module):
    def __init__(self, batch_size, n_inputs, n_neurons, n_outputs):
        super(MyRNN, self).__init__()

        self.n_neurons = n_neurons
        self.batch_size = batch_size
        self.n_inputs = n_inputs
        self.n_outputs = n_outputs

        self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons)

        self.FC = nn.Linear(self.n_neurons, self.n_outputs)

    def init_hidden(self, ):
        # (num_layers, batch_size, n_neurons)
        return torch.zeros(1, self.batch_size, self.n_neurons)

    def forward(self, X):
        self.batch_size = X.size(0)
        self.hidden = self.init_hidden()

        lstm_out, self.hidden = self.basic_rnn(X, self.hidden)
        out = self.FC(self.hidden)

        return out.view(-1, self.n_outputs)

My input x looks like this:

tensor([[-1.0173e-04, -1.5003e-04, -1.0218e-04, -7.4541e-05, -2.2869e-05,
         -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04,
         -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01],
        [-1.1193e-04, -1.6928e-04, -1.0218e-04, -7.4541e-05, -2.2869e-05,
         -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04,
         -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01],
        ...

        [-6.9490e-05, -8.9197e-05, -1.0218e-04, -7.4541e-05, -2.2869e-05,
         -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04,
         -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01]],
       dtype=torch.float64)

and is a batch of 64 vectors with size 15.

When trying to test this model by doing:

BATCH_SIZE = 64
N_INPUTS = 15
N_NEURONS = 150
N_OUTPUTS = 1
model = MyRNN(BATCH_SIZE, N_INPUTS, N_NEURONS, N_OUTPUTS)
model(x)

I get the following error:

File "/home/tt/anaconda3/envs/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 126, in check_forward_args
    expected_input_dim, input.dim()))
RuntimeError: input must have 3 dimensions, got 2

How can I fix it?

Output:
You are missing one of the required dimensions for the RNN layer.

Per the documentation, your input size needs to be of shape (sequence length, batch, input size).

So - with the example above, you are missing one of these. Based on your variable names, it appears you are trying to pass 64 examples of 15 inputs each... if that’s true, you are missing sequence length.

With an RNN, the sequence length is the number of times you want the layer to recur. For example, in NLP your sequence length might be equal to the number of words in a sentence, while batch size would be the number of sentences you are passing, and input size would be the vector size of each word.

You might not need an RNN here if you are just trying to do use 64 samples of size 15.


Instruction: Stacking up of LSTM outputs in pytorch

Input:
I was going through some tutorial about the sentiment analysis using lstm network.
The below code said that its stacks up the lstm output. I Don't know how it works.

lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

Output:
It indeed stacks the output, the comment by kHarshit is misleading here!

To visualize this, let us review the output of the previous line in the tutorial (accessed May 1st, 2019):

lstm_out, hidden = self.lstm(embeds, hidden)

The output dimension of this will be [sequence_length, batch_size, hidden_size*2], as per the documentation. Here, the length of twice the input comes from having a bidirectional LSTM. Therefore, your first half of the last dimension will always be the forward output, and then afterwards the backwards output (I'm not entirely sure on the direction of that, but it seems to me that it is already in the right direction).

Then, the actual line that you are concerned about:

We're ignoring the specifics of .contiguous() here, but you can read up on it in this excellent answer on Stackoverflow. In summary, it basically makes sure that your torch.Tensor is in the right alignment in memory.
 Lastly, .view() allows you to reshape a resulting tensor in a specific way. Here, we're aiming for a shape that has two dimensions (as defined by the number of input arguments to .view(). Specifically, the second dimension is supposedly having the size hidden_dim. -1 for the first dimension simply means that we're redistributing the vector dimension in such a way that we don't care about the exact dimension, but simply satisfy the other dimension's requirements.
So, if you have a vector of, say, length 40, and want to reshape that one into a 2D-Tensor of (-1, 10), then the resulting tensor would have shape (4, 10).

As we've previously said that the first half of the vector (length hidden_dim) is the forward output, and the latter half is the second half, then the resulting split into a tensor of (-1, hidden_dim) will be resulting in a tensor of (2, hidden_dim), where the first row contains the forward output, "stacked" on top of the second row, which equals the reverse layer's output.

Visual example:

lstm_out, hidden = self.lstm(embeds, hidden)
print(lstm_out) # imagine a sample output like [1,0 , 2,0]
                #                      forward out  | backward out

stacked = lstm_out.contiguous().view(-1,hidden_dim) # hidden_dim = 2

print(stacked) # torch.Tensor([[1,0],
               #               [2,0]])


Instruction: How does Module.parameters() find the parameters?

Input:
I noticed that whenever you create a new net extending torch.nn.Module, you can immediately call net.parameters() to find the parameters relevant for backpropagation.

import torch

class MyNet(torch.nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.fc = torch.nn.Linear(5, 5)

    def forward(self, x):
        return self.fc(x)

net = MyNet()
print(list(net.parameters()))

But then I wondered, how is this even possible? I just assigned this Linear layer object to a member variable but it is not recorded anywhere else (or is it?). Somehow MyNet must be able to keep track of the parameters used but how?

Output:
It's simple really, just go through attributes via meta-programming and check their type

class Example():
    def __init__(self):
        self.special_thing = nn.Parameter(torch.rand(2))
        self.something_else = "ok"

    def get_parameters(self):
        for key, value in self.__dict__.items():
            if type(value) == nn.Parameter:
                print(key, "is a parameter!")

e = Example()
e.get_parameters()
# => special_thing is a parameter!


Instruction: Autograd.grad() for Tensor in pytorch

Input:
I want to compute the gradient between two tensors in a net. The input X tensor (batch size x m) is sent through a set of convolutional layers which give me back and output Y tensor(batch size x n).
I’m creating a new loss and I would like to know the gradient of Y w.r.t. X. Something that in tensorflow would be like:
tf.gradients(ys=Y, xs=X)
Unfortunately, I’ve been making tests with torch.autograd.grad(), but I could not figure out how to do it. I get errors like: “RunTimeerror: grad can be implicitly created only for scalar outputs”.
What should be the inputs in torch.autograd.grad() if I want to know the gradient of Y w.r.t. X?

Output:
Let's start from simple working example with plain loss function and regular backward. We will build short computational graph and do some grad computations on it.

Code:

import torch
from torch.autograd import grad
import torch.nn as nn

# Create some dummy data.
x = torch.ones(2, 2, requires_grad=True)
gt = torch.ones_like(x) * 16 - 0.5  # "ground-truths"

# We will use MSELoss as an example.
loss_fn = nn.MSELoss()

# Do some computations.
v = x + 2
y = v ** 2

# Compute loss.
loss = loss_fn(y, gt)

print(f'Loss: {loss}')

# Now compute gradients:
d_loss_dx = grad(outputs=loss, inputs=x)
print(f'dloss/dx:\n {d_loss_dx}')

Output:

Loss: 42.25
dloss/dx:
(tensor([[-19.5000, -19.5000], [-19.5000, -19.5000]]),)

Ok, this works! Now let's try to reproduce error "grad can be implicitly created only for scalar outputs". As you can notice, loss in previous example is a scalar. backward() and grad() by defaults deals with single scalar value: loss.backward(torch.tensor(1.)). If you try to pass tensor with more values you will get an error.

Code:

v = x + 2
y = v ** 2

try:
    dy_hat_dx = grad(outputs=y, inputs=x)
except RuntimeError as err:
    print(err)

Output:

grad can be implicitly created only for scalar outputs

Therefore, when using grad() you need to specify grad_outputs parameter as follows:

Code:

v = x + 2
y = v ** 2

dy_dx = grad(outputs=y, inputs=x, grad_outputs=torch.ones_like(y))
print(f'dy/dx:\n {dy_dx}')

dv_dx = grad(outputs=v, inputs=x, grad_outputs=torch.ones_like(v))
print(f'dv/dx:\n {dv_dx}')

Output:

dy/dx:
(tensor([[6., 6.],[6., 6.]]),)

dv/dx:
(tensor([[1., 1.], [1., 1.]]),)

NOTE: If you are using backward() instead, simply do y.backward(torch.ones_like(y)).


Instruction: Using autograd.grad() as a parameter for a loss function (pytorch)

Input:
I want to compute the gradient between two tensors in a net. The input X tensor is sent through a set of convolutional layers which give me back and output Y tensor.

I’m creating a new loss and I would like to know the MSE between gradient  of norm(Y) w.r.t. each element of X. Here the code:

# Staring tensors
X = torch.rand(40, requires_grad=True)
Y = torch.rand(40, requires_grad=True)

# Define loss
loss_fn = nn.MSELoss()

#Make some calculations
V = Y*X+2

# Compute the norm
V_norm = V.norm()

# Computing gradient to calculate the loss
for i in range(len(V)):
    if i == 0:
        grad_tensor = torch.autograd.grad(outputs=V_norm, inputs=X[i])
    else:
        grad_tensor_ = torch.autograd.grad(outputs=V_norm, inputs=X[i])
        grad_tensor = torch.cat((grad_tensor, grad_tensor_), dim=0)

# Grund truth
gt = grad_tensor * 0 + 1

#Loss
loss_g = loss_fn(grad_tensor, gt)
print(loss_g)

Unfortunately, I’ve been making tests with torch.autograd.grad(), but I could not figure out how to do it. I get teh following error: RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.

Setting allow_unused=True gives me back None which is not an option. Not sure how to compute the loss between the gradients and the norm. Any idea about how to code this loss?

Output:
You are receiving mentioned error because you are trying to feed a slice of tensor X: X[i] to grad(), and it is going to be considered as a separate tensor, outside of your main computational graph. Not sure, but seems it returns new tensor while performing slicing.

But you don't need a for loop to compute gradients:

Code:

import torch
import torch.nn as nn

torch.manual_seed(42)

# Create some data.
X = torch.rand(40, requires_grad=True)
Y = torch.rand(40, requires_grad=True)

# Define loss.
loss_fn = nn.MSELoss()

# Do some computations.
V = Y * X + 2

# Compute the norm.
V_norm = V.norm()

print(f'V norm: {V_norm}')

# Computing gradient to calculate the loss
grad_tensor = torch.autograd.grad(outputs=V_norm, inputs=X)[0]  # [0] - Because grad returs tuple, so we need to unpack it
print(f'grad_tensor:\n {grad_tensor}')

# Grund truth
gt = grad_tensor * 0 + 1

loss_g = loss_fn(grad_tensor, gt)
print(f'loss_g: {loss_g}')

Output:

V norm: 14.54827

grad_tensor:
    tensor([0.1116, 0.0584, 0.1109, 0.1892, 0.1252, 0.0420, 0.1194, 0.1000, 0.1404,
            0.0272, 0.0007, 0.0460, 0.0168, 0.1575, 0.1097, 0.1120, 0.1168, 0.0771,
            0.1371, 0.0208, 0.0783, 0.0226, 0.0987, 0.0512, 0.0929, 0.0573, 0.1464,
            0.0286, 0.0293, 0.0278, 0.1896, 0.0939, 0.1935, 0.0123, 0.0006, 0.0156,
            0.0236, 0.1272, 0.1109, 0.1456])

loss_g: 0.841885

Loss between the grads and the norm

You also mentioned that you want to compute loss between the gradients and the norm, it is possible. And there are two possible options of it:

You want to include your loss calculation to your computational graph, in this case use:

loss_norm_vs_grads = loss_fn(torch.ones_like(grad_tensor) * V_norm, grad_tensor)

You want just to compute loss and you don't want to start backward path from the loss, in this case don't forget to use torch.no_grad(), otherwise autograd will track this changes and add loss computation to your computational graph.

with torch.no_grad():
    loss_norm_vs_grads = loss_fn(torch.ones_like(grad_tensor) * V_norm, grad_tensor)


Instruction: Why aren't torch.nn.Parameter listed when net is printed?

Input:
I recently had to construct a module that required a tensor to be included. While back propagation worked perfectly using torch.nn.Parameter, it did not show up when printing the net object. Why isn't this parameter included in contrast to other modules like layer? (Shouldn't it behave just like layer?)

import torch
import torch.nn as nn

class MyNet(torch.nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.layer = nn.Linear(10, 10)
        self.parameter = torch.nn.Parameter(torch.zeros(10,10, requires_grad=True))

net = MyNet()
print(net)

Output:

MyNet(
  (layer): Linear(in_features=10, out_features=10, bias=True)
)

Output:
When you call print(net), the __repr__ method is called. __repr__ gives the “official” string representation of an object.

In PyTorch's nn.Module (base class of your MyNet model), the __repr__ is implemented like this:

def __repr__(self):
        # We treat the extra repr like the sub-module, one item per line
        extra_lines = []
        extra_repr = self.extra_repr()
        # empty string will be split into list ['']
        if extra_repr:
            extra_lines = extra_repr.split('\n')
        child_lines = []
        for key, module in self._modules.items():
            mod_str = repr(module)
            mod_str = _addindent(mod_str, 2)
            child_lines.append('(' + key + '): ' + mod_str)
        lines = extra_lines + child_lines

        main_str = self._get_name() + '('
        if lines:
            # simple one-liner info, which most builtin Modules will use
            if len(extra_lines) == 1 and not child_lines:
                main_str += extra_lines[0]
            else:
                main_str += '\n  ' + '\n  '.join(lines) + '\n'

        main_str += ')'
        return main_str

Note that the above method returns main_str which contains call to only _modules and extra_repr, thus it prints only modules by default.

PyTorch also provides extra_repr() method which you can implement yourself for extra representation of the module.

  To print customized extra information, you should reimplement this method in your own modules. Both single-line and multi-line strings are acceptable.


Instruction: How to upscale image in pytorch?

Input:
how to upscale an image in Pytorch without defining height and width using transforms?
 ('--upscale_factor', type=int, required=True, help="super resolution upscale factor")

Output:
This might do the Job

transforms.Compose([transforms.resize(ImageSize*Scaling_Factor)])


Instruction: Simple way to load specific sample using Pytorch dataloader

Input:
I am currently training a 3D CNN for binary classification with relatively sparse labels (~ 1% of voxels in label data correspond to target class).

In order to perform basic sanity checks during the training (e.g. does the network learn at all?) it would be handy to present the network with a small, handpicked subset of training examples having an above-average fraction of target class labels.

As suggested by the Pytorch documentation, I implemented my own dataset class (inheriting from torch.utils.data.Dataset) which provides training examples via it's __get_item__ method to the torch.utils.data.DataLoader.

In the pytorch tutorials I found, the DataLoader is used as an iterator to generate the training loop like so:

for i, data in enumerate(self.dataloader):

    # Get training data
    inputs, labels = data

    # Train the network
    # [...]

What I am wondering now is whether there exist a simple way to load a single or a couple of specific training examples (using a the linear index understood by Dataset's __get_item__ method). However, DataLoader does not have a __get_item__ method and repeatedly calling __next__ until I reach the desired index does not seem elegant.

Apparently one possible way to solve this would be to define a custom sampler or batch_sampler inheriting from the abstract torch.utils.data.Sampler. But this seems over the top to retrieve a few specific samples.

I suppose I am overlooking something very simple and obvious here. Any advice appreciated!

Output:
Just in case anyone with a similar question comes across this at some point:

The quick-and-dirty workaround I ended up using was to bypass the dataloader in the training loop by directly accessing it's associated dataset attribute. Suppose we want to quickly check if our network learns at all by repeatedly presenting it a single, handpicked training example with linear index sample_idx (as defined by the dataset class).

Then one can do something like this:

for i, _ in enumerate(self.dataloader):

    # Get training data
    # inputs, labels = data

    inputs, labels = self.dataloader.dataset[sample_idx]
    inputs = inputs.unsqueeze(0)
    labels = labels.unsqueeze(0)

    # Train the network
    # [...]

EDIT:

One brief remark, since some people seem to be finding this workaround helpful: When using this hack I found it to be crucial to instantiate the DataLoader with num_workers = 0. Otherwise, memory segmentation errors might occur in which case you could end up with very weird looking training data.


Instruction: How to change default optimization in spotlight from pytorch e.g. torch.optim.SGD?

Input:
I'm currently using spotlight https://github.com/maciejkula/spotlight/tree/master/spotlight
to implement Matrix Factorization in recommender system. spotlight is based on pytorch, it's a integrated platform implementing RS. In spotlight/factorization/explicit, it uses torch.optim.Adam as optimizer, I want to change it to torch.optim.SGD.
I tried

emodel = ExplicitFactorizationModel(n_iter=15,
                                embedding_dim=32,
                                use_cuda=False,
                                loss='regression',
                                l2=0.00005,
                                optimizer_func=optim.SGD(lr=0.001, momentum=0.9))

but it gives:TypeError: init() missing 1 required positional argument: 'params'
Any suggestions?

Output:
You could use partial from functools to first set the learning rate and momentum and then pass this class to ExplicitFactorizationModel. Something like:

from functools import partial
SDG_fix_lr_momentum = partial(torch.optim.SGD, lr=0.001, momentum=0.9)
emodel = ExplicitFactorizationModel(n_iter=15,
                                embedding_dim=32,
                                use_cuda=False,
                                loss='regression',
                                l2=0.00005,
                                optimizer_func=SDG_fix_lr_momentum)


Instruction: Does pytorch do eager pruning of its computational graph?

Input:
This is a very simple example:

import torch

x = torch.tensor([1., 2., 3., 4., 5.], requires_grad=True)
y = torch.tensor([2., 2., 2., 2., 2.], requires_grad=True)
z = torch.tensor([1., 1., 0., 0., 0.], requires_grad=True)

s = torch.sum(x * y * z)
s.backward()

print(x.grad)

This will print,

tensor([2., 2., 0., 0., 0.]),

since, of course, ds/dx is zero for the entries where z is zero.

My question is: Is pytorch smart and stop the computations when it reaches a zero? Or does in fact do the calculation "2*5", only to later do "10 * 0 = 0"?

In this simple example it doesn't make a big difference, but in the (bigger) problem I am looking at, this will make a difference.

Thank you for any input.

Output:
No, pytorch does no such thing as pruning any subsequent calculations when zero is reached. Even worse, due to how float arithmetic works all subsequent multiplication by zero will take roughly the same time as any regular multiplication.

For some cases there are ways around it though, for example if you want to use a masked loss you can just set the masked outputs to be zero, or detach them from gradients.

This example makes the difference clear:

def time_backward(do_detach):
    x = torch.tensor(torch.rand(100000000), requires_grad=True)
    y = torch.tensor(torch.rand(100000000), requires_grad=True)
    s2 = torch.sum(x * y)
    s1 = torch.sum(x * y)
    if do_detach:
        s2 = s2.detach()
    s = s1 + 0 * s2
    t = time.time()
    s.backward()
    print(time.time() - t)

time_backward(do_detach= False)
time_backward(do_detach= True)

outputs:

0.502875089645
0.198422908783


Instruction: What kind of tricks we can play with to further refine the trained neural network model so that it has lower objective function value?

Input:
I ask this question because many deep learning frameworks, such as Caffe, supports model refining function. For example, in Caffe, we can use snapshot to initialling the neural network parameters and then continue performing training as the following command shows:

./caffe train -solver solver_file.prototxt -snapshot snap_file.solverstate

In order to further train the model, the following tricks I can play with:

use smaller learning rate
change optimisation method. For example, change stochastic gradient descent to ADAM algorithm

Any other tricks I can play with?

ps: I understand that reducing the loss function value of the training samples does not mean that we can get a better model.

Output:
The question is way too broad, I think. However, this is a common practice, especially in case of a small training set. I would rank possible methods like this:

smaller learning rate
more/different data augmentation
add noise to train set (related to data augmentation, indeed)
fine-tune on subset of the training set.

The very last one is indeed a very powerful method to finalize the model that performs poor on some corner cases. You can then make a 'difficult' train subset in order to bias model towards it. I personally use it very often.


Instruction: Selecting second dim of tensor using an index tensor

Input:
I have a 2D tensor and an index tensor.  The 2D tensor has a batch dimension, and a dimension with 3 values.  I have an index tensor that selects exactly 1 element of the 3 values.  What is the "best" way to product a slice containing just the elements in the index tensor?

t = torch.tensor([[1,2,3], [4,5,6], [7,8,9]])
t = tensor([[1, 2, 3],
        [4, 5, 6],
        [7, 8, 9]])

i = torch.tensor([0,0,1], dtype=torch.int64)
tensor([0, 0, 1])

Expected output...

tensor([1, 4, 8])

Output:
An example of the answer is as follows.

import torch

t = torch.tensor([[1,2,3], [4,5,6], [7,8,9]])
col_i = [0, 0, 1]
row_i = range(3)
print(t[row_i, col_i])
# tensor([1, 4, 8])


Instruction: Pytorch resumes training after every training session

Input:
I have a dataset which is partitioned into smaller datasets.

I want to train 3 models for each partition of the dataset, but I need all training sessions to start from the same initialised network parameters.

so it looks like this:

modelList = []
thisCNN = NNet()

for x in range(3):
    train = torch.utils.data.DataLoader(Subset(train_set, indexes[x]), batch_size=32)
    bb = trainMyNet(thisCNN, train, test)
    modelList.append(list(bb.parameters()))

print modelList[0][1]
print modelList[1][1]
print modelList[2][1]

In the printing, I am getting the same exact parameters for every saved model which is strange, and also I noticed that after every iteration the model in fact resumes the training from the previous iteration as it results in lower loss per iteration.

What I am trying to achieve, is, per iteration to get a new model on the current subset x but the training should start with the same initial thisCNN = NNet() weights.

Output:
When you call bb = trainMyNet(thisCNN, train, test) you are not taking a copy of thisCNN, but it is the same model you are updating in each iteration. To get your code working you should probably pass a copy of this model:

from copy import deepcopy
modelList = []
thisCNN = NNet()

for x in range(3):
    train = torch.utils.data.DataLoader(Subset(train_set, indexes[x]), batch_size=32)
    bb = trainMyNet(deepcopy(thisCNN), train, test)
    modelList.append(list(bb.parameters()))

print modelList[0][1]
print modelList[1][1]
print modelList[2][1]

This should initialize all models as thisCNN and make sure that they are different after training.


Instruction: PyTorch version of as simple Keras LSTM model

Input:
Trying to translate a simple LSTM model in Keras to PyTorch code. The Keras model converges after just 200 epochs, while the PyTorch model:

needs many more epochs to reach the same loss level (200 vs. ~8000)
seems to overfit the inputs because the predicted value is not near 100

This is the Keras code:

from numpy import array
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense

X = array([10,20,30,20,30,40,30,40,50,40,50,60,50,60,70,60,70,80]).reshape((6,3,1))
y = array([40,50,60,70,80,90])
model = Sequential()
model.add(LSTM(50, activation='relu', recurrent_activation='sigmoid',  input_shape=(3, 1)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(X, y, epochs=200, verbose=1)
x_input = array([70, 80, 90]).reshape((1, 3, 1))
yhat = model.predict(x_input, verbose=0)
print(yhat)

And this is the equivalent PyTorch code:

from numpy import array
import torch
import torch.nn as nn
import torch.nn.functional as F

X = torch.tensor([10,20,30,20,30,40,30,40,50,40,50,60,50,60,70,60,70,80]).float().reshape(6,3,1)
y = torch.tensor([40,50,60,70,80,90]).float().reshape(6,1)

class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    self.lstm = nn.LSTM(input_size=1, hidden_size=50, num_layers=1, batch_first=True)
    self.fc = nn.Linear(50, 1)

  def forward(self, x):
    batches = x.size(0)
    h0 = torch.zeros([1, batches, 50])
    c0 = torch.zeros([1, batches, 50])
    (x, _) = self.lstm(x, (h0, c0))
    x = x[:,-1,:]  # Keep only the output of the last iteration. Before shape (6,3,50), after shape (6,50)
    x = F.relu(x)
    x = self.fc(x)
    return x

model = Model()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters())

n_epochs = 8000
for epoch in range(n_epochs):
  model.train()
  optimizer.zero_grad()
  y_ = model(X)
  loss = criterion(y_, y)
  loss.backward()
  optimizer.step()
  print(f"Epoch {epoch+1}/{n_epochs}, loss = {loss.item()}")

model.eval()
x_input = torch.tensor([70, 80, 90]).float().reshape((1, 3, 1))
yhat = model(x_input)
print(yhat)

The only possible difference is the initial weight and bias values, but I don't think that slightly different weights and biases may account for such a big difference in behavior.
What am I missing in the PyTorch code?

Output:
The behaviour difference is because of the activation function in the LSTM API. By changing the activation to tanh, I can reproduce the problem in Keras too.

model.add(LSTM(50, activation='tanh', recurrent_activation='sigmoid',  input_shape=(3, 1)))

There is no option to change the activation function to 'relu' in the pytorch LSTM API.
https://pytorch.org/docs/stable/nn.html#lstm

Taking the LSTM implementation from here, https://github.com/huggingface/torchMoji/blob/master/torchmoji/lstm.py
and changing hardsigmoid/tanh to sigmoid/relu, the model converges in pytorch as well.


Instruction: pytorch 0.4.1 - ‘LSTM’ object has no attribute ‘weight_ih_l’

Input:
Simple question. I’d like to see the initialized parameter of LSTM. How do I see it?
Do I need to always put lstm in the model to see the params?

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(1)
torch.__version__

lstm = nn.LSTM(3, 3)

lstm.weight_ih_l

  --------------------------------------------------------------------------- AttributeError                            Traceback (most recent call
  last)  in ()
        9 lstm = nn.LSTM(3, 3)
       10
  ---> 11 lstm.weight_ih_l

  ~/anaconda3/envs/pytorch0.41/lib/python3.6/site-packages/torch/nn/modules/module.py
  in getattr(self, name)
      516                 return modules[name]
      517         raise AttributeError("'{}' object has no attribute '{}'".format(
  --> 518             type(self).name, name))
      519
      520     def setattr(self, name, value):

  AttributeError: 'LSTM' object has no attribute 'weight_ih_l'

Output:
nn.LSTM is implemented with nn.RNNBase which puts all the parameters inside the OrderedDict: _parameters. So to 'see' the initialized parameters, you can simply do:

import torch
torch.manual_seed(1)

import torch.nn as nn

lstm = nn.LSTM(3, 3)

print(lstm._parameters['weight_ih_l0'])

Also, to know what are the keys value in that OrderedDict, you can simply do:
print(lstm._all_weights).


Instruction: tensorflow equivalent of pytorch ReplicationPad2d

Input:
I'm trying to figure out how to do the tensorflow equivalent of the following padding in pytorch:

nn.ReplicationPad2d((1, 0, 1, 0))

I've tried the following, but this only seems to work if the input tensor is actually 2x2:

tf.pad(my_tensor, [[1, 0], [1, 0]], "SYMMETRIC")

Output:
The equivalent for Tensorflow is tf.pad(my_tensor,[[0,0],[0,0],[1,0],[1,0]],"SYMMETRIC").  (This assumes that you are interested in operating on 4D tensors, with the first two dimensions being batch and channel).

In Tensorflow, you need to explicitly give the padding for all of the four dimensions. If you don't want the batch and channel dimensions to be padded (in convolutional networks you typically do not need them padded), you need to explicitly ask for zero padding in both of these dimensions, on both sides of the tensor. This is why I added the [0,0],[0,0] before your [1,0],[1,0].

In Pytorch, an instance of nn.ReplicationPad2d is already assumed to be padding a 4D tensor, without padding the the first two dimensions. That's why you initialize the instance by specifying the padding only in the two additional dimensions.


Instruction: RNN model (GRU) of word2vec to regression not learning

Input:
I am converting Keras code into PyTorch because I am more familiar with the latter than the former. However, I found that it is not learning (or only barely).

Below I have provided almost all of my PyTorch code, including the initialisation code so that you can try it out yourself. The only thing you would need to provide yourself, is the word embeddings (I'm sure you can find many word2vec models online). The first input file should be a file with tokenised text, the second input file should be a file with floating-point numbers, one per line. Because I have provided all the code, this question may seem huge and too broad. However, my question is specific enough I think: what is wrong in my model or training loop that causes my model to not or barely improve. (See below for results.)

I have tried to provide many comments where applicable, and I have provided the shape transformations as well so you do not have to run the code to see what is going on. The data prep methods are not important to inspect.

The most important parts are the forward method of the RegressorNet, and the training loop of RegressionNN (admittedly, these names were badly chosen). I think the mistake is there somewhere.

from pathlib import Path
import time

import numpy as np
import torch
from torch import nn, optim
from torch.utils.data import DataLoader
import gensim

from scipy.stats import pearsonr

from LazyTextDataset import LazyTextDataset

class RegressorNet(nn.Module):
    def __init__(self, hidden_dim, embeddings=None, drop_prob=0.0):
        super(RegressorNet, self).__init__()
        self.hidden_dim = hidden_dim
        self.drop_prob = drop_prob

        # Load pretrained w2v model, but freeze it: don't retrain it.
        self.word_embeddings = nn.Embedding.from_pretrained(embeddings)
        self.word_embeddings.weight.requires_grad = False
        self.w2v_rnode = nn.GRU(embeddings.size(1), hidden_dim, bidirectional=True, dropout=drop_prob)

        self.dropout = nn.Dropout(drop_prob)
        self.linear = nn.Linear(hidden_dim * 2, 1)
        # LeakyReLU rather than ReLU so that we don't get stuck in a dead nodes
        self.lrelu = nn.LeakyReLU()

    def forward(self, batch_size, sentence_input):
        # shape sizes for:
        # * batch_size 128
        # * embeddings of dim 146
        # * hidden dim of 200
        # * sentence length of 20

        # sentence_input: torch.Size([128, 20])
        # Get word2vec vector representation
        embeds = self.word_embeddings(sentence_input)
        # embeds: torch.Size([128, 20, 146])

        # embeds.view(-1, batch_size, embeds.size(2)): torch.Size([20, 128, 146])
        # Input vectors into GRU, only keep track of output
        w2v_out, _ = self.w2v_rnode(embeds.view(-1, batch_size, embeds.size(2)))
        # w2v_out = torch.Size([20, 128, 400])

        # Leaky ReLU it
        w2v_out = self.lrelu(w2v_out)

        # Dropout some nodes
        if self.drop_prob > 0:
            w2v_out = self.dropout(w2v_out)
        # w2v_out: torch.Size([20, 128, 400

        # w2v_out[-1, :, :]: torch.Size([128, 400])
        # Only use the last output of a sequence! Supposedly that cell outputs the final information
        regression = self.linear(w2v_out[-1, :, :])
        regression: torch.Size([128, 1])

        return regression

class RegressionRNN:
    def __init__(self, train_files=None, test_files=None, dev_files=None):
        print('Using torch ' + torch.__version__)

        self.datasets, self.dataloaders = RegressionRNN._set_data_loaders(train_files, test_files, dev_files)
        self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

        self.model = self.w2v_vocab = self.criterion = self.optimizer = self.scheduler = None

    @staticmethod
    def _set_data_loaders(train_files, test_files, dev_files):
        # labels must be the last input file
        datasets = {
            'train': LazyTextDataset(train_files) if train_files is not None else None,
            'test': LazyTextDataset(test_files) if test_files is not None else None,
            'valid': LazyTextDataset(dev_files) if dev_files is not None else None
        }
        dataloaders = {
            'train': DataLoader(datasets['train'], batch_size=128, shuffle=True, num_workers=4) if train_files is not None else None,
            'test': DataLoader(datasets['test'], batch_size=128, num_workers=4) if test_files is not None else None,
            'valid': DataLoader(datasets['valid'], batch_size=128, num_workers=4) if dev_files is not None else None
        }

        return datasets, dataloaders

    @staticmethod
    def prepare_lines(data, split_on=None, cast_to=None, min_size=None, pad_str=None, max_size=None, to_numpy=False,
                      list_internal=False):
        """ Converts the string input (line) to an applicable format. """
        out = []
        for line in data:
            line = line.strip()
            if split_on:
                line = line.split(split_on)
                line = list(filter(None, line))
            else:
                line = [line]

            if cast_to is not None:
                line = [cast_to(l) for l in line]

            if min_size is not None and len(line) < min_size:
                # pad line up to a number of tokens
                line += (min_size - len(line)) * ['@pad@']
            elif max_size and len(line) > max_size:
                line = line[:max_size]

            if list_internal:
                line = [[item] for item in line]

            if to_numpy:
                line = np.array(line)

            out.append(line)

        if to_numpy:
            out = np.array(out)

        return out

    def prepare_w2v(self, data):
        idxs = []
        for seq in data:
            tok_idxs = []
            for word in seq:
                # For every word, get its index in the w2v model.
                # If it doesn't exist, use @unk@ (available in the model).
                try:
                    tok_idxs.append(self.w2v_vocab[word].index)
                except KeyError:
                    tok_idxs.append(self.w2v_vocab['@unk@'].index)
            idxs.append(tok_idxs)
        idxs = torch.tensor(idxs, dtype=torch.long)

        return idxs

    def train(self, epochs=10):
        valid_loss_min = np.Inf
        train_losses, valid_losses = [], []
        for epoch in range(1, epochs + 1):
            epoch_start = time.time()

            train_loss, train_results = self._train_valid('train')
            valid_loss, valid_results = self._train_valid('valid')

            # Calculate Pearson correlation between prediction and target
            try:
                train_pearson = pearsonr(train_results['predictions'], train_results['targets'])
            except FloatingPointError:
                train_pearson = "Could not calculate Pearsonr"

            try:
                valid_pearson = pearsonr(valid_results['predictions'], valid_results['targets'])
            except FloatingPointError:
                valid_pearson = "Could not calculate Pearsonr"

            # calculate average losses
            train_loss = np.mean(train_loss)
            valid_loss = np.mean(valid_loss)

            train_losses.append(train_loss)
            valid_losses.append(valid_loss)

            # print training/validation statistics
            print(f'----------\n'
                  f'Epoch {epoch} - completed in {(time.time() - epoch_start):.0f} seconds\n'
                  f'Training Loss: {train_loss:.6f}\t Pearson: {train_pearson}\n'
                  f'Validation loss: {valid_loss:.6f}\t Pearson: {valid_pearson}')

            # validation loss has decreased
            if valid_loss <= valid_loss_min and train_loss > valid_loss:
                print(f'!! Validation loss decreased ({valid_loss_min:.6f} --> {valid_loss:.6f}).  Saving model ...')
                valid_loss_min = valid_loss

            if train_loss <= valid_loss:
                print('!! Training loss is lte validation loss. Might be overfitting!')

            # Optimise with scheduler
            if self.scheduler is not None:
                self.scheduler.step(valid_loss)

        print('Done training...')

    def _train_valid(self, do):
        """ Do training or validating. """
        if do not in ('train', 'valid'):
            raise ValueError("Use 'train' or 'valid' for 'do'.")

        results = {'predictions': np.array([]), 'targets': np.array([])}
        losses = np.array([])

        self.model = self.model.to(self.device)
        if do == 'train':
            self.model.train()
            torch.set_grad_enabled(True)
        else:
            self.model.eval()
            torch.set_grad_enabled(False)

        for batch_idx, data in enumerate(self.dataloaders[do], 1):
            # 1. Data prep
            sentence = data[0]
            target = data[-1]
            curr_batch_size = target.size(0)

            # Returns list of tokens, possibly padded @pad@
            sentence = self.prepare_lines(sentence, split_on=' ', min_size=20, max_size=20)
            # Converts tokens into w2v IDs as a Tensor
            sent_w2v_idxs = self.prepare_w2v(sentence)
            # Converts output to Tensor of floats
            target = torch.Tensor(self.prepare_lines(target, cast_to=float))

            # Move input to device
            sent_w2v_idxs, target = sent_w2v_idxs.to(self.device), target.to(self.device)

            # 2. Predictions
            pred = self.model(curr_batch_size, sentence_input=sent_w2v_idxs)
            loss = self.criterion(pred, target)

            # 3. Optimise during training
            if do == 'train':
                self.optimizer.zero_grad()
                loss.backward()
                self.optimizer.step()

            # 4. Save results
            pred = pred.detach().cpu().numpy()
            target = target.cpu().numpy()

            results['predictions'] = np.append(results['predictions'], pred, axis=None)
            results['targets'] = np.append(results['targets'], target, axis=None)
            losses = np.append(losses, float(loss))

        torch.set_grad_enabled(True)

        return losses, results

if __name__ == '__main__':
    HIDDEN_DIM = 200

    # Load embeddings from pretrained gensim model
    embed_p = Path('path-to.w2v_model').resolve()
    w2v_model = gensim.models.KeyedVectors.load_word2vec_format(str(embed_p))
    # add a padding token with only zeros
    w2v_model.add(['@pad@'], [np.zeros(w2v_model.vectors.shape[1])])
    embed_weights = torch.FloatTensor(w2v_model.vectors)

    # Text files are used as input. Every line is one datapoint.
    # *.tok.low.*: tokenized (space-separated) sentences
    # *.cross: one floating point number per line, which we are trying to predict
    regr = RegressionRNN(train_files=(r'train.tok.low.en',
                                      r'train.cross'),
                         dev_files=(r'dev.tok.low.en',
                                    r'dev.cross'),
                         test_files=(r'test.tok.low.en',
                                     r'test.cross'))
    regr.w2v_vocab = w2v_model.vocab
    regr.model = RegressorNet(HIDDEN_DIM, embed_weights, drop_prob=0.2)
    regr.criterion = nn.MSELoss()
    regr.optimizer = optim.Adam(list(regr.model.parameters())[0:], lr=0.001)
    regr.scheduler = optim.lr_scheduler.ReduceLROnPlateau(regr.optimizer, 'min', factor=0.1, patience=5, verbose=True)

    regr.train(epochs=100)

For the LazyTextDataset, you can refer to the class below.

from torch.utils.data import Dataset

import linecache

class LazyTextDataset(Dataset):
    def __init__(self, paths):
        # labels are in the last path
        self.paths, self.labels_path = paths[:-1], paths[-1]

        with open(self.labels_path, encoding='utf-8') as fhin:
            lines = 0
            for line in fhin:
                if line.strip() != '':
                    lines += 1

            self.num_entries = lines

    def __getitem__(self, idx):
        data = [linecache.getline(p, idx + 1) for p in self.paths]
        label = linecache.getline(self.labels_path, idx + 1)

        return (*data, label)

    def __len__(self):
        return self.num_entries

As I wrote before, I am trying to convert a Keras model to PyTorch. The original Keras code does not use an embedding layer, and uses pre-built word2vec vectors per sentence as input. In the model below, there is no embedding layer. The Keras summary looks like this (I don't have access to the base model setup).

Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
bidirectional_1 (Bidirectional)  (200, 400)            417600
____________________________________________________________________________________________________
dropout_1 (Dropout)              (200, 800)            0           merge_1[0][0]
____________________________________________________________________________________________________
dense_1 (Dense)                  (200, 1)              801         dropout_1[0][0]
====================================================================================================

The issue is that with identical input, the Keras model works and gets a +0.5 Pearson correlation between predicted and actual labels. The PyTorch model above, though, does not seem to work at all. To give you an idea, here is the loss (mean squared error) and Pearson (correlation coefficient, p-value) after the first epoch:

Epoch 1 - completed in 11 seconds
Training Loss: 1.684495  Pearson: (-0.0006077809280690612, 0.8173368901481127)
Validation loss: 1.708228    Pearson: (0.017794288315261794, 0.4264098054188664)

And after the 100th epoch:

Epoch 100 - completed in 11 seconds
Training Loss: 1.660194  Pearson: (0.0020315421756790806, 0.4400929436716754)
Validation loss: 1.704910    Pearson: (-0.017288118524826892, 0.4396865964324158)

The loss is plotted below (when you look at the Y-axis, you can see the improvements are minimal).

A final indicator that something may be wrong, is that for my 140K lines of input, each epoch only takes 10 seconds on my GTX 1080TI. I feel that his is not much and I would guess that the optimisation is not working/running. I cannot figure out why, though. To issue will probably be in my train loop or the model itself, but I cannot find it.

Again, something must be going wrong because:
  - the Keras model does perform well;
  - the training speed is 'too fast' for 140K sentences
  - almost no improvemnts after training.

What am I missing? The issue is more than likely present in the training loop or in the network structure.

Output:
TL;DR: Use permute instead of view when swapping axes, see the end of answer to get an intuition about the difference.

About RegressorNet (neural network model)

No need to freeze embedding layer if you are using from_pretrained. As documentation states, it does not use gradient updates.
This part:

self.w2v_rnode = nn.GRU(embeddings.size(1), hidden_dim, bidirectional=True, dropout=drop_prob)

and especially dropout without providable num_layers is totally pointless (as no dropout can be specified with shallow one layer network).
BUG AND MAIN ISSUE: in your forward function you are using view instead of permute, here:

w2v_out, _ = self.w2v_rnode(embeds.view(-1, batch_size, embeds.size(2)))

See this answer and appropriate documentation for each of those functions and try to use this line instead:

w2v_out, _ = self.w2v_rnode(embeds.permute(1, 0, 2))

You may consider using batch_first=True argument during w2v_rnode creation, you won't have to permute indices that way.
Check documentation of torch.nn.GRU, you are after last step of the sequence, not after all of the sequences you have there, so you should be after:

_, last_hidden = self.w2v_rnode(embeds.permute(1, 0, 2))

but I think this part is fine otherwise.

Data preparation

No offence, but prepare_lines is very unreadable and seems pretty hard to maintain as well, not to say spotting an eventual bug (I suppose it lies in here).

First of all, it seems like you are padding manually. Please don't do it that way, use torch.nn.pad_sequence to work with batches!

In essence, first you encode each word in every sentence as index pointing into embedding (as you seem to do in prepare_w2v), after that you use torch.nn.pad_sequence and torch.nn.pack_padded_sequence or torch.nn.pack_sequence if the lines are already sorted by length.

Proper batching

This part is very important and it seems you are not doing that at all (and likely this is the second error in your implementation).

PyTorch's RNN cells take inputs not as padded tensors, but as torch.nn.PackedSequence objects. This is an efficient object storing indices which specify unpadded length of each sequence.

See more informations on the topic here, here and in many other blog posts throughout the web.

First sequence in batch has to be the longest, and all others have to be provided in the descending length. What follows is:

You have to sort your batch each time by sequences length and sort your targets in an analogous way OR
Sort your batch, push it through the network and unsort it afterwards to match with your targets.

Either is fine, it's your call what seems to be more intuitive for you.
What I like to do is more or less the following, hope it helps:

Create unique indices for each word and map each sentence appropriately (you've already done it).
Create regular torch.utils.data.Dataset object returning single sentence for each geitem, where it is returned as a tuple consisting of features (torch.Tensor) and labels (single value), seems like you're doing it as well.
Create custom collate_fn for use with torch.utils.data.DataLoader, which is responsible for sorting and padding each batch in this scenario (+ it returns lengths of each sentence to be passed into neural network).
Using sorted and padded features and their lengths I'm using torch.nn.pack_sequence inside neural network's forward method (do it after embedding!) to push it through RNN layer.
Depending on the use-case I unpack them using torch.nn.pad_packed_sequence. In your case, you only care about last hidden state, hence you don't have to do that. If you were using all of the hidden outputs (like is the case with, say, attention networks), you would add this part.

When it comes to the third point, here is a sample implementation of collate_fn, you should get the idea:

import torch

def length_sort(features):
    # Get length of each sentence in batch
    sentences_lengths = torch.tensor(list(map(len, features)))
    # Get indices which sort the sentences based on descending length
    _, sorter = sentences_lengths.sort(descending=True)
    # Pad batch as you have the lengths and sorter saved already
    padded_features = torch.nn.utils.rnn.pad_sequence(features, batch_first=True)
    return padded_features, sentences_lengths, sorter

def pad_collate_fn(batch):
    # DataLoader return batch like that unluckily, check it on your own
    features, labels = (
        [element[0] for element in batch],
        [element[1] for element in batch],
    )
    padded_features, sentences_lengths, sorter = length_sort(features)
    # Sort by length features and labels accordingly
    sorted_padded_features, sorted_labels = (
        padded_features[sorter],
        torch.tensor(labels)[sorter],
    )
    return sorted_padded_features, sorted_labels, sentences_lengths

Use those as collate_fn in DataLoaders and you should be just about fine (maybe with minor adjustments, so it's essential you understand the idea standing behind it).

Other possible problems and tips

Training loop: great place for a lot of small errors, you may want to minimalize those by using PyTorch Ignite. I am having unbelievably hard time going through your Tensorflow-like-Estimator-like-API-like training loop (e.g. self.model = self.w2v_vocab = self.criterion = self.optimizer = self.scheduler = None this). Please, don't do it this way, separate each task (data creating, data loading, data preparation, model setup, training loop, logging) into it's own respective module. All in all there is a reason why PyTorch/Keras is more readable and sanity-preserving than Tensorflow.
Make the first row of your embedding equal to vector containg zeros: By default, torch.nn.functional.embedding expects the first row to be used for padding. Hence you should start your unique indexing for each word at 1 or specify an argument padding_idx to different value (though I highly discourage this approach, confusing at best).

I hope this answer helps you at least a little bit, if something is unclear post a comment below and I'll try to explain it from a different perspective/more detail.

Some final comments

This code is not reproducible, nor the question's specific. We don't have the data you are using, neither we got your word vectors, random seed is not fixed etc.

PS. One last thing: Check your performance on really small subset of your data (say 96 examples), if it does not converge, it is very likely you indeed have a bug in your code.

About the times: they are probably off (due to not sorting and not padding I suppose), usually Keras and PyTorch's times are quite similar (if I understood this part of your question as intended) for correct and efficient implementations.

Permute vs view vs reshape explanation

This simple example show the differences between permute() and view(). The first one swaps axes, while the second does not change memory layout, just chunks the array into desired shape (if possible).

import torch

a = torch.tensor([[1, 2], [3, 4], [5, 6]])

print(a)
print(a.permute(1, 0))
print(a.view(2, 3))

And the output would be:

tensor([[1, 2],
        [3, 4],
        [5, 6]])
tensor([[1, 3, 5],
        [2, 4, 6]])
tensor([[1, 2, 3],
        [4, 5, 6]])

reshape is almost like view, was added for those coming from numpy, so it's easier and more natural for them, but it has one important difference:

view never copies data and work only on contiguous memory (so after permutation like the one above your data may not be contiguous, hence acces to it might be slower)
reshape can copy data if needed, so it would work for non-contiguous arrays as well.


Instruction: Compute maxima and minima of a 4D tensor in PyTorch

Input:
Suppose that we have a 4-dimensional tensor, for instance

import torch
X = torch.rand(2, 3, 4, 4)
tensor([[[[-0.9951,  1.6668,  1.3140,  1.4274],
          [ 0.2614,  2.6442, -0.3041,  0.7337],
          [-1.2690,  0.0125, -0.3885,  0.0535],
          [ 1.5270, -0.1186, -0.4458,  0.1389]],

         [[ 0.9125, -1.2998, -0.4277, -0.2688],
          [-1.6917, -0.8855, -0.2784, -0.6717],
          [ 1.1417,  0.4574,  0.4803, -1.6637],
          [ 0.7322,  0.2654, -0.1525,  1.7285]],

         [[ 1.8310, -1.5765,  0.1392,  1.3431],
          [-0.6641, -1.5090, -0.4893, -1.4110],
          [ 0.5875,  0.7528, -0.6482, -0.2547],
          [-2.3133,  0.3888,  2.1428,  0.2331]]]])

I want to compute the maximum and the minimum values of X over the dimensions 2 and 3, that is, to compute two tensors of size (2,3,1,1), one for the maximum and one for the minimum values of the 4x4 blocks.

I started by trying to do that with torch.max() and torch.min(), but I had no luck. I would expect the dim argument of the above functions to be able to take tuple values, but it can take only an integer. So I don't know how to proceed.

However, specifically for the maximum values, I decided to use torch.nn.MaxPool2d() with kernel_size=4 and stride=4. This indeed did the job:

max_pool = nn.MaxPool2d(kernel_size=4, stride=4)
X_max = max_pool(X)
tensor([[[[2.6442]],
         [[1.7285]],
         [[2.1428]]]])

But, afaik, there's no similar layer for "min"-pooling. Could you please help me on how to compute the minima similarly to the maxima?

Thank you.

Output:
Just calculate the max for both dimensions sequentially, it gives the same result:

tup = (2,3)
for dim in tup:
    X = torch.max(X,dim=dim,keepdim=True)[0]


Instruction: Given list of image filenames for eachset,Split large dataset to train/valid/test directories?

Input:
I am trying to split a large dataset into train/valid/test sets from Food101 dataset  for image classification

and the structure of a dataset is like this and has all images in one folder

'',
'Structure:',
'----------',
'pec/',
'    images/',
'        <class_name>/',
'            <image_id>.jpg',
'    meta/',
'        classes.txt',
'        labels.txt',
'        test.json',
'        test.txt',
'        train.json',
'        train.txt',
'',
'All images can be found in the "images" folder and are organized per class. All',
'image ids are unique and correspond to the foodspotting.com review ids.
'',
'The test/train splitting used in the experiment of our paper can be found in',
'the "meta" directory.', (edited) ```

I want to divide images dataset to train/valid/test  with the list of filenames given in train.txt and test.txt, which author used

the Shape of train,valid, test lists:  (101, 600) , (101, 150) , 25250

In colab ,i run following code

for x in range(train.shape[0]):
    for y in range(train.shape[1]):

     temp = train[x,y] + ".jpg"

     foldername = temp.split('/')[0]

     !mv /content/food-101/images/$temp /content/food101/train/$foldername/

Individually moving images by running nested loop by taking filenames in lists, is  taking forever time to create folders as there are 100100 images total so,

I have a list of filenames for train/valid and test set but how to make them into folders so that we can feed it to image classifier in pytorch image folder format(i mean train / valid / test set are three different folders and each folder has subfolders of each class)

Please Do tell if anyone knows how to do this please, and please I really need your help here,  THanks:smile:

Output:
It seems i have been going all wrong about the solution, i need not move the images all i need to change is path to images in the required format through os module

Below is the code for doing it. Say you have list of filenames in valid list

#for valid set

v = valid.reshape(15150,)

or_fpath = '/content/food-101/images/' #path of original folder
cp_fpath = '/content/food101/valid/'   #path of destination folder

for y in tqdm(v):

 foldername = y.split('/')[0]

 img = y.split('/')[1] +'.jpg'

 ip_path = or_fpath+foldername
 op_path = cp_fpath+foldername

 if not os.path.exists(op_path):
   os.mkdir(op_path)

 os.rename(os.path.join(ip_path, img), os.path.join(op_path, img))

Thanks!

note: if you have even better answer please share Thanks


Instruction: Import error on Windows10 with pytorch0.4

Input:
Discription

I am trying to install pytorch 0.4 on Windows10.
My enviroment settings:
- Windows10
- cuda9.0
- python 3.6
- pytorch 0.4
- anaconda

I tried by using both conda install -n myenv and pip install $path:whl and both failed.

Error

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\Anaconda\envs\py3.6_pytorch0.4\lib\site-packages\torch\__init__.py", line 80, in <module>
    from torch._C import *
ImportError: DLL load failed:

I found the related a issue #4518 under pytorch but answers under that issue do not work for me.

What I have tried

add all conda related path to environment path
change the directory (cd)
install vs_runtime under this conda env

None of those works.

But if I install pytorch under conda's base environment, it works well.

So what's going on here?

Update:

When we tried to install pytorch on windows, a lot of packages need to be installed at the same time. We can simply follow the steps on official website [link](https://pytorch.org/），for python 3.6 and cuda9.0 the installation command is as

    conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

If we want to install former version of pytorch, we can assign the version, eg. 0.4 as

    conda install pytorch=0.4 torchvision cudatoolkit=9.0 -c pytorch

Output:
After you create the new environment with conda, execute conda install -c pytorch pytorch to install pytorch.

pip does not work that well with external, non-Python dependencies. Not unlikely in your case path to the DLL is not set correctly (just a guess).


Instruction: No module named "Torch"

Input:
I successfully installed pytorch via conda:
conda install pytorch-cpu torchvision-cpu -c pytorch

I also successfully installed pytorch via pip:
pip3 install https://download.pytorch.org/whl/cpu/torch-1.0.1-cp36-cp36m-win_amd64.whl
pip3 install torchvision

But, it only works in a jupyter notebook. Whenever I try to execute a script from the console, I get the error message:

No module named "torch"

Output:
Try to install PyTorch using pip:

First create a Conda environment using:

conda create -n env_pytorch python=3.6

Activate the environment using:

conda activate env_pytorch

Now install PyTorch using pip:

pip install torchvision

Note: This will install both torch and torchvision.

Now go to Python shell and import using the command:

import torch
import torchvision


Instruction: PyTorch get all layers of model

Input:
What's the easiest way to take a pytorch model and get a list of all the layers without any nn.Sequence groupings? For example, a better way to do this?
import pretrainedmodels

def unwrap_model(model):
    for i in children(model):
        if isinstance(i, nn.Sequential): unwrap_model(i)
        else: l.append(i)

model = pretrainedmodels.__dict__['xception'](num_classes=1000, pretrained='imagenet')
l = []
unwrap_model(model)

print(l)

Output:
You can iterate over all modules of a model (including those inside each Sequential) with the modules() method. Here's a simple example:
>>> model = nn.Sequential(nn.Linear(2, 2),
                          nn.ReLU(),
                          nn.Sequential(nn.Linear(2, 1),
                          nn.Sigmoid()))

>>> l = [module for module in model.modules() if not isinstance(module, nn.Sequential)]

>>> l

[Linear(in_features=2, out_features=2, bias=True),
 ReLU(),
 Linear(in_features=2, out_features=1, bias=True),
 Sigmoid()]


Instruction: element 0 of tensors does not require grad and does not have a grad_fn

Input:
I am trying to apply reiforcement learning mechanism to classification tasks.
I know it is useless thing to do because deep learning can overperform rl in the tasks. Anyway in research purposes I am doing.

I reward agent if he's correct positive 1 or not negative -1
and computate loss FUNC with predicted_action(predicted_class) and reward.

But I get an error:

  element 0 of tensors does not require grad and does not have a grad_fn

 # creating model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.pipe = nn.Sequential(nn.Linear(9, 120),
                                 nn.ReLU(),
                                 nn.Linear(120, 64),
                                 nn.ReLU(),
                                 nn.Linear(64,2),
                                 nn.Softmax()
                                 )

    def forward(self, x):
        return self.pipe(x)

def env_step(action, label, size):
    total_reward = []

    for i in range(size):
        reward = 0

        if action[i] == label[i]:
            total_reward.append(reward+1)
            continue
        else:
            total_reward.append(reward-1)
            continue

    return total_reward

if __name__=='__main__':
    epoch_size = 100
    net = Net()
    criterion = nn.MSELoss()
    optimizer = optim.Adam(params=net.parameters(), lr=0.01)

    total_loss = deque(maxlen = 50)

    for epoch in range(epoch_size):
        batch_index = 0
        for i in range(13):
            # batch sample
            batch_xs = torch.FloatTensor(train_state[batch_index: batch_index+50])   # make tensor
            batch_ys = torch.from_numpy(train_label[batch_index: batch_index+50]).type('torch.LongTensor')  # make tensor

            # action_prob; e.g classification prob
            actions_prob = net(batch_xs)
            #print(actions_prob)
            action = torch.argmax(actions_prob, dim=1).unsqueeze(1)
            #print(action)
            reward = np.array(env_step(action, batch_ys, 50))
            #print(reward)

            reward = torch.from_numpy(reward).unsqueeze(1).type('torch.FloatTensor')
            #print(reward)
            action = action.type('torch.FloatTensor')

            optimizer.zero_grad()
            loss = criterion(action, reward)
            loss.backward()
            optimizer.step()

            batch_index += 50

Output:
action is produced by the argmax funtion, which is not differentiable. You instead want take the loss between the reward and the responsible probability for the action taken.

Often, the "loss" chosen for the policy in reinfocement learning is the so called score function:

Which is the product of the log of the responsible probablity for the action a taken times the reward gained.


Instruction: List not populated correctly unless use PyTorch clone()

Input:
I'm attempting to add the final weights of each trained model to a list using below code :

%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
import torchvision.datasets as dset
import os
import torch.nn.functional as F
import time
import random
import pickle
from sklearn.metrics import confusion_matrix
import pandas as pd
import sklearn

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])

root = './data'
if not os.path.exists(root):
    os.mkdir(root)
train_set = dset.MNIST(root=root, train=True, transform=trans, download=True)
test_set = dset.MNIST(root=root, train=False, transform=trans, download=True)

batch_size = 64

train_loader = torch.utils.data.DataLoader(
                 dataset=train_set,
                 batch_size=batch_size,
                 shuffle=True)
test_loader = torch.utils.data.DataLoader(
                dataset=test_set,
                batch_size=batch_size,
shuffle=True)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 2)
    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

num_epochs = 2
random_sample_size = 200

values_0_or_1 = [t for t in train_set if (int(t[1]) == 0 or int(t[1]) == 1)]
values_0_or_1_testset = [t for t in test_set if (int(t[1]) == 0 or int(t[1]) == 1)]

print(len(values_0_or_1))
print(len(values_0_or_1_testset))

train_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1,
                 batch_size=batch_size,
                 shuffle=True)

test_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1_testset,
                 batch_size=batch_size,
                 shuffle=False)

train_loader = train_loader_subset

# Hyper-parameters
input_size = 100
hidden_size = 100
num_classes = 2
# learning_rate = 0.00001
learning_rate = .0001
# Device configuration
device = 'cpu'
print_progress_every_n_epochs = 1

model = NeuralNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

N = len(train_loader)
# Train the model
total_step = len(train_loader)

most_recent_prediction = []
test_actual_predicted_dict = {}

rm = random.sample(list(values_0_or_1), random_sample_size)
train_loader_subset = data_utils.DataLoader(rm, batch_size=4)

weights_without_clone = []
weights_with_clone = []

for i in range(0 , 2) :
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(train_loader_subset):
            # Move tensors to the configured device
            images = images.reshape(-1, 2).to(device)
            labels = labels.to(device)

            # Forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)

            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        if (epoch) % print_progress_every_n_epochs == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

    print('model fc2 weights ' , model.fc2.weight.data)
    weights_without_clone.append(model.fc2.weight.data)
    weights_with_clone.append(model.fc2.weight.data.clone())

Output of model :

12665
2115
Epoch [1/2], Step [50/198], Loss: 0.0968
Epoch [2/2], Step [50/198], Loss: 0.0082
model fc2 weights  tensor([[-3.9507e-02, -4.0454e-02,  3.5576e-03,  ...,  6.2181e-03,
          4.1372e-02, -6.2960e-03],
        [ 1.8778e-02,  2.7049e-02, -3.5624e-02,  ...,  2.6797e-02,
          2.2041e-03, -4.2284e-02],
        [ 1.9571e-02, -3.2545e-02,  2.6618e-02,  ..., -1.6139e-02,
          4.1192e-02, -2.3458e-02],
        ...,
        [-4.6123e-03,  2.6943e-02,  3.9979e-02,  ..., -3.3848e-02,
          3.6096e-02,  2.4211e-02],
        [-1.4698e-02,  9.7528e-04, -2.5244e-03,  ..., -3.3145e-02,
          1.0888e-02,  3.1091e-02],
        [-1.7451e-02, -2.1646e-02,  2.5885e-02,  ...,  4.0453e-02,
         -6.5324e-03, -3.5410e-02]])
Epoch [1/2], Step [50/198], Loss: 0.0025
Epoch [2/2], Step [50/198], Loss: 0.0013
model fc2 weights  tensor(1.00000e-02 *
       [[-3.9891, -4.0454,  0.3558,  ...,  0.7168,  4.1902, -0.6253],
        [ 1.8766,  2.7049, -3.5632,  ...,  2.6785,  0.2192, -4.2297],
        [ 2.1426, -3.2545,  2.6621,  ..., -1.6285,  4.1196, -2.2653],
        ...,
        [-0.4930,  2.6943,  3.9971,  ..., -3.2940,  3.6641,  2.4248],
        [-1.5160,  0.0975, -0.2524,  ..., -3.1938,  1.1753,  3.1065],
        [-1.8116, -2.1646,  2.5883,  ...,  4.1355, -0.5921, -3.5416]])

Printing the values of weights_without_clone :

print(weights_without_clone[0])
print(weights_without_clone[1])

outputs :

tensor(1.00000e-02 *
       [[-3.9891, -4.0454,  0.3558,  ...,  0.7168,  4.1902, -0.6253],
        [ 1.8766,  2.7049, -3.5632,  ...,  2.6785,  0.2192, -4.2297],
        [ 2.1426, -3.2545,  2.6621,  ..., -1.6285,  4.1196, -2.2653],
        ...,
        [-0.4930,  2.6943,  3.9971,  ..., -3.2940,  3.6641,  2.4248],
        [-1.5160,  0.0975, -0.2524,  ..., -3.1938,  1.1753,  3.1065],
        [-1.8116, -2.1646,  2.5883,  ...,  4.1355, -0.5921, -3.5416]])
tensor(1.00000e-02 *
       [[-3.9891, -4.0454,  0.3558,  ...,  0.7168,  4.1902, -0.6253],
        [ 1.8766,  2.7049, -3.5632,  ...,  2.6785,  0.2192, -4.2297],
        [ 2.1426, -3.2545,  2.6621,  ..., -1.6285,  4.1196, -2.2653],
        ...,
        [-0.4930,  2.6943,  3.9971,  ..., -3.2940,  3.6641,  2.4248],
        [-1.5160,  0.0975, -0.2524,  ..., -3.1938,  1.1753,  3.1065],
        [-1.8116, -2.1646,  2.5883,  ...,  4.1355, -0.5921, -3.5416]])

Printing the values of weights_with_clone :

print(weights_with_clone[0])
print(weights_with_clone[1])

outputs :

tensor([[-3.9507e-02, -4.0454e-02,  3.5576e-03,  ...,  6.2181e-03,
          4.1372e-02, -6.2960e-03],
        [ 1.8778e-02,  2.7049e-02, -3.5624e-02,  ...,  2.6797e-02,
          2.2041e-03, -4.2284e-02],
        [ 1.9571e-02, -3.2545e-02,  2.6618e-02,  ..., -1.6139e-02,
          4.1192e-02, -2.3458e-02],
        ...,
        [-4.6123e-03,  2.6943e-02,  3.9979e-02,  ..., -3.3848e-02,
          3.6096e-02,  2.4211e-02],
        [-1.4698e-02,  9.7528e-04, -2.5244e-03,  ..., -3.3145e-02,
          1.0888e-02,  3.1091e-02],
        [-1.7451e-02, -2.1646e-02,  2.5885e-02,  ...,  4.0453e-02,
         -6.5324e-03, -3.5410e-02]])
tensor(1.00000e-02 *
       [[-3.9891, -4.0454,  0.3558,  ...,  0.7168,  4.1902, -0.6253],
        [ 1.8766,  2.7049, -3.5632,  ...,  2.6785,  0.2192, -4.2297],
        [ 2.1426, -3.2545,  2.6621,  ..., -1.6285,  4.1196, -2.2653],
        ...,
        [-0.4930,  2.6943,  3.9971,  ..., -3.2940,  3.6641,  2.4248],
        [-1.5160,  0.0975, -0.2524,  ..., -3.1938,  1.1753,  3.1065],
        [-1.8116, -2.1646,  2.5883,  ...,  4.1355, -0.5921, -3.5416]])

Why is 1.00000e-02 * prepended to the final weight value of the second model ?

Why is using clone() required in order to add final weights for each iteration as omitting clone() the same weights are added to each iteration ? :

weights_without_clone.append(model.fc2.weight.data)
weights_with_clone.append(model.fc2.weight.data.clone())

Output:
First of all, I am going to reproduce your case. I will use very simple model:

Code:

import torch
import torch.nn as nn
import torch.optim as optim

torch.manual_seed(42)

# Some dummy data:
X = torch.randn(100, 5, requires_grad=True, dtype=torch.float)
Y = torch.randn(100, 5, requires_grad=True, dtype=torch.float)

class Model(nn.Module):

    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(5, 5, bias=False)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(5, 5, bias=False)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

def train(model, x, y, loss_fn, optimizer, n_epochs=1000, print_loss=True):

    weights = []

    for i in range(n_epochs):
        y_hat = model(x)
        loss = loss_fn(y_hat, y)

        optimizer.zero_grad()

        loss.backward()

        if print_loss:
            print(f'| {i+1} | Loss: {loss.item():.4f}')

        optimizer.step()

        print('W:\n', model.fc2.weight.data)

        weights.append(model.fc2.weight.data)

    return weights

torch.manual_seed(42)

model = Model()

loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
n_epochs = 2

weights = train(model=model,
                      x=X,
                      y=Y,
                      loss_fn=loss_fn,
                      optimizer=optimizer,
                      n_epochs=n_epochs,
                      print_loss=True)

Output:

| 1 | Loss: 1.0285
W:
 tensor([[-0.2052, -0.1257, -0.2684,  0.0425, -0.4413],
        [ 0.4034, -0.3797,  0.3448,  0.0741, -0.1450],
        [ 0.2759,  0.0695,  0.3608,  0.0487, -0.1411],
        [ 0.1201, -0.1213,  0.1881,  0.3990,  0.2583],
        [-0.1956,  0.2581,  0.0798,  0.2270, -0.2725]])
| 2 | Loss: 1.0279
W:
 tensor([[-0.2041, -0.1251, -0.2679,  0.0428, -0.4410],
        [ 0.4030, -0.3795,  0.3444,  0.0738, -0.1447],
        [ 0.2755,  0.0693,  0.3603,  0.0484, -0.1411],
        [ 0.1200, -0.1213,  0.1879,  0.3987,  0.2580],
        [-0.1958,  0.2580,  0.0796,  0.2269, -0.2725]])

Ok, it works well. Let's now look at weights:

Code:

print(*weights, sep='\n')

Output:

tensor([[-0.2041, -0.1251, -0.2679,  0.0428, -0.4410],
        [ 0.4030, -0.3795,  0.3444,  0.0738, -0.1447],
        [ 0.2755,  0.0693,  0.3603,  0.0484, -0.1411],
        [ 0.1200, -0.1213,  0.1879,  0.3987,  0.2580],
        [-0.1958,  0.2580,  0.0796,  0.2269, -0.2725]])
tensor([[-0.2041, -0.1251, -0.2679,  0.0428, -0.4410],
        [ 0.4030, -0.3795,  0.3444,  0.0738, -0.1447],
        [ 0.2755,  0.0693,  0.3603,  0.0484, -0.1411],
        [ 0.1200, -0.1213,  0.1879,  0.3987,  0.2580],
        [-0.1958,  0.2580,  0.0796,  0.2269, -0.2725]])

Ok, it is not what we want, but actually it is expected behavior. If you look once again, you will see that values in the list correspond to weights values from second epoch. That means we were appending not new tensors, but assignments that point to real weights storage, and that's why we just have the same final results.

In other words, you are getting the same values when using regular append, because gradients still propagate to the original weights tensor. And appended "weights tensors" are pointing to the same tensor from model that changes during backprop.

That's why you need to use clone to create a new tensor, BUT it is recommended to use tensor.clone().detach() whereas clone is recorded to computational graph, this means if you backprop through this cloned tensor,

  Gradients propagating to the cloned tensor will propagate to the original tensor. clone docs

So, if you want to append your weights safely use this:

weights.append(model.fc2.weight.data.clone().detach())